Design a Subscription-Billing System

Design a subscription-billing system that runs recurring charges for a SaaS product with multiple plan tiers (think a team-collaboration app with Free, Pro, and Business plans billed monthly or annually). The system owns the billing lifecycle: it decides who to charge and when, drives money movement through an external payment processor, issues invoices, and keeps subscription state truthful even when charges fail or the processor is flaky.

Operating context. ~2M active paid subscriptions across the tiers. Charges are not spread evenly — renewals cluster hard on the 1st of the month and on annual anniversaries, so a daily "billing run" can face a spike of ~500K renewal attempts inside a few hours (peak ~200 charge-attempts/sec). Each attempt calls a third-party processor (p99 ~800 ms, occasionally times out) that is the source of truth for whether money actually moved; the processor confirms outcomes both synchronously and later via webhooks. Reads (a customer opening the billing page) are light: ~300 RPS, p99 < 200 ms.

Out of scope. Tax/VAT computation, the checkout/first-signup flow, the payment-method vault and PCI handling (assume the processor tokenizes cards), fraud scoring, and the general-ledger/accounting export. Assume a separate team owns each, including the customer-facing UI.

What to produce. A high-level architecture covering: the components and the request/job flow; how the recurring billing cycle is scheduled and swept; the data model for plans, subscriptions, invoices, and charge attempts (plus partitioning); how a mid-cycle plan change is prorated; the dunning/retry policy when a charge fails; how the processor's webhooks are ingested and reconciled against your own state; and — the crux — how a billing run stays exactly-once so a retried or double-triggered run never double-charges a customer. Sketch the major components and the flow between them; we will probe specifics at checkpoints.