Design a Message Delivery Retry Queue

Design the delivery-retry subsystem of a chat platform: the piece that guarantees a message eventually reaches every recipient even when they are offline, on a flaky network, or their device is asleep. When the messaging core cannot hand a message to a recipient's live connection, it lands here and is retried until delivered or expired.

Operating context. A large fraction of recipients are offline at any moment, so most messages spend time waiting before their first successful delivery. Throughput is high, with hundreds of thousands of messages/sec entering the system, and messages within one conversation must be delivered to a recipient in send order, so a retry cannot reorder a thread. Retries use backoff and must not hammer a recipient that just came online with a thundering herd. Delivery is at-least-once and the client deduplicates, so occasional redelivery is acceptable but loss is not. An undelivered message is kept up to a TTL of 30 days and then dropped.

Out of scope. The live socket and connection layer and presence detection, push-notification content and vendor integration, end-to-end-encryption, and group membership resolution (assume the fan-out already produced one pending entry per recipient). Assume each entry arrives authenticated with a recipient id and a conversation sequence number.

What to produce. A high-level architecture covering: how a pending entry is durably stored per recipient, how delivery is triggered when a recipient reconnects, the retry scheduler and its backoff policy, how per-conversation ordering is preserved across retries, how at-least-once plus client dedup is achieved, and how TTL expiry and cleanup work at scale. Sketch the components and the delivery lifecycle; checkpoints will probe ordering and the reconnect thundering-herd.