Problem statement
Design the delivery-retry subsystem of a chat platform: the piece that guarantees a message eventually reaches every recipient even when they are offline, on a flaky network, or their device is asleep. When the messaging core cannot hand a message to a recipient's live connection, it lands here and is retried until delivered or expired.
Operating context. A large fraction of recipients are offline at any moment, so most messages spend time waiting before their first successful delivery. Throughput is high, with hundreds of thousands of messages/sec entering the system, and messages within one conversation must be delivered to a recipient in send order, so a retry cannot reorder a thread. Retries use backoff and must not hammer a recipient that just came online with a thundering herd. Delivery is at-least-once and the client deduplicates, so occasional redelivery is acceptable but loss is not. An undelivered message is kept up to a TTL of 30 days and then dropped.
Out of scope. The live socket and connection layer and presence detection, push-notification content and vendor integration, end-to-end-encryption, and group membership resolution (assume the fan-out already produced one pending entry per recipient). Assume each entry arrives authenticated with a recipient id and a conversation sequence number.
What to produce. A high-level architecture covering: how a pending entry is durably stored per recipient, how delivery is triggered when a recipient reconnects, the retry scheduler and its backoff policy, how per-conversation ordering is preserved across retries, how at-least-once plus client dedup is achieved, and how TTL expiry and cleanup work at scale. Sketch the components and the delivery lifecycle; checkpoints will probe ordering and the reconnect thundering-herd.
Functional requirements
- Durably enqueue a message for a recipient when it cannot be delivered to a live connection.
- Deliver all pending messages to a recipient, in per-conversation order, when they reconnect.
- Retry undelivered messages on a backoff schedule until delivery or TTL expiry.
- Guarantee at-least-once delivery, relying on the client to deduplicate replays.
- Expire and purge messages that remain undelivered past their retention TTL.
Non-functional requirements
- Absorb hundreds of thousands of enqueue operations per second at peak.
- Deliver a pending message within 2 seconds of a recipient reconnecting, at p95.
- Preserve per-conversation send order for every recipient across all retries.
- No message loss on single-node failure; pending entries are durably replicated.
- Retain an undelivered message for up to 30 days before expiry.
- 99.95% availability for enqueue; deliveries may lag under load but must not be dropped.
Topics
- System Design HLD
- Messaging Delivery
- Patterns Queue
- Reliability At-Least-Once
- Data Ordering