Problem statement
Design a chat backup-and-export service. A user requests a complete archive of their history, messages, metadata, and attached media across every conversation, and later downloads it as a single package. This backs the product's data-portability feature and its regulatory data-export obligation, so it must be complete, verifiable, and available for a bounded time.
Operating context. An export can be large: an active multi-year account may hold millions of messages and tens of gigabytes of media, so generation is asynchronous and can take minutes to hours. Requests are spiky, since a privacy-policy news cycle can drive a surge, but the steady rate is modest, on the order of a few hundred exports per minute. The archive must be a faithful snapshot: it captures history up to the request time and need not include messages sent after the job starts. Finished archives are downloadable via a time-limited secure link and are deleted after an expiry window. Media lives in object storage; messages live in a separate datastore.
Out of scope. Real-time message delivery, importing or restoring an archive back into the product, end-to-end-encryption key handling, and content moderation. Assume the request is authenticated and the requesting user is authorized to export exactly their own data.
What to produce. A high-level architecture covering: how an export request is accepted and tracked, how a worker gathers a consistent snapshot of messages and media from their separate stores, how the archive is assembled and stored, how the user is notified and given a secure expiring download, how progress and failures are surfaced, and how one enormous account does not starve everyone else. Sketch the components and the job lifecycle; checkpoints will probe snapshot consistency and large-account handling.
Functional requirements
- Accept an authenticated request to export the requesting user's full chat history and media.
- Assemble a downloadable archive containing messages, metadata, and referenced media attachments.
- Notify the user and expose a time-limited, access-controlled download when the archive is ready.
- Report progress and a terminal success or failure status for each export job.
- Automatically delete finished archives after a fixed retention window.
Non-functional requirements
- Handle a few hundred export requests per minute steady, absorbing 10x surges during news-driven spikes.
- Support archives up to tens of gigabytes and tens of millions of messages per account.
- Complete a typical export at p95 under 15 minutes; very large accounts may take hours.
- Download links are valid for 7 days and are single-user, non-transferable, and revocable.
- 99.9% availability for request submission and status query; generation may queue under surge.
- Every archive is a consistent snapshot as of request time, with an integrity checksum for verification.
Topics
- System Design HLD
- Messaging Export
- Infra Object-Store
- Patterns Batch-Job
- Data Snapshot