Problem statement
Design the service creators use to upload large video files reliably over unreliable networks. A multi-gigabyte upload must survive dropped connections and app restarts, resume exactly where it left off without re-sending acknowledged bytes, verify integrity, and then hand the finished asset to downstream processing.
Operating context. Files reach tens of gigabytes and are uploaded from phones and desktops on flaky links. Chunks may arrive out of order, be retried, or be duplicated. The service must track which byte ranges it has, tell a resuming client the next offset to send, verify the whole file, store it durably, and emit a completion event. Sessions that are started but never finished must not leak storage forever.
Out of scope. The transcode and thumbnail pipelines (assume they subscribe to an upload-complete event), CDN delivery of the finished asset, content moderation, and the creator-facing client UI. Assume separate teams own those.
What to produce. A high-level architecture covering: the upload-session and chunk protocol, offset tracking and resume, per-chunk and whole-file integrity verification, storage and final assembly, deduplication of identical content, the completion event and downstream hand-off, and garbage collection of abandoned sessions. Sketch the components and the flow of one upload; we will probe specifics during checkpoints.
Functional requirements
- Create an upload session for a file, returning an upload target/ID and the accepted chunk size.
- Accept out-of-order or retried chunks and track which byte ranges have been received.
- Resume an interrupted upload by reporting the next expected offset to the client.
- Verify integrity with per-chunk and whole-file checksums before finalizing the asset.
- Emit an upload-complete event carrying the stored object reference for downstream processing.
Non-functional requirements
- Support files up to 20 GB and 100,000 concurrent in-flight upload sessions.
- Sustain 40 Gbps of aggregate ingest throughput.
- A resumed upload never re-sends an already-acknowledged chunk.
- Finalize (assemble and verify) a 5 GB upload within 30 seconds p95 of the last chunk.
- Once a chunk is acknowledged, its data is durable with 99.95% guarantee and not lost.
- Abandoned sessions are garbage-collected within 24 hours to reclaim storage.
Topics
- System Design HLD
- Media Upload
- Storage Object
- Data Integrity
- Scaling Write-Heavy