Design a Scheduled Payout Service

Design the service that pays money out to external bank accounts on a schedule — merchant settlements, payroll, vendor disbursements, scheduled transfers. Users or systems create payout instructions with an execution time; the service must release each one exactly once at the right moment, debit the internal ledger, submit it to the correct external rail, and track it to a terminal state. The unforgiving requirement is that a payout is released exactly once: never twice, and never dropped.

Operating context. Payouts are scheduled minutes to months ahead, cluster heavily at cycle boundaries (month-end, payday), and go out over external rails that are asynchronous, can fail, and can take days to confirm or return. A released payout moves real money, so idempotency and reconciliation with the rail are central. Assume 10 million scheduled payouts/day with sharp peaks, and a hard rule that no payout is ever sent twice or silently lost.

Out of scope. The external rails' internals, fraud/limit checks (assume already cleared), currency conversion, and the UI for creating payouts. Assume separate teams own the rails, risk, FX, and the app.

What to produce. A high-level architecture covering: durable storage of scheduled payouts and how due ones are found and released on time at scale, exactly-once release under retries and worker failure, atomically debiting the ledger and handing off to a rail, tracking async rail confirmations/failures/returns to a terminal state, retry and reversal on rail failure, reconciliation against the rail, and absorbing peak cycle bursts. Sketch the components and lifecycle; we will probe specifics in checkpoints.