Problem statement
Design a multi-tenant scheduled-job service that lets internal teams register recurring jobs (cron-style expressions or fixed intervals) and reliably fires each one at its due time by calling the owner's HTTP endpoint. The core difficulty is correctness at the boundary: a job must fire on time, must not silently skip its slot, and — depending on the job's declared policy — must not fire twice for the same slot even when your own fleet is crashing, restarting, or racing.
Operating context. Thousands of tenants register roughly 2,000,000 active job definitions between them, producing about 3,000 due-fires per second at steady state with sharp spikes to 40,000/sec on whole-minute boundaries (cron jobs align to the minute, so most active jobs wake on the same tick, and the 0 * * * * crowd stacks on the top of every hour). A single scheduler node can die at any moment; the fleet is elastic and nodes come and go during deploys. Each fire delivers a small JSON payload to a tenant-owned webhook, whose latency and reliability you do not control. Jobs carry an IANA timezone (America/New_York, Asia/Kolkata, ...), so wall-clock semantics and DST transitions matter.
Out of scope. The tenant's own webhook implementation and its business logic, per-tenant billing/quota metering, the UI for authoring jobs, authn/authz of the management API, and heavy result-payload storage (fires carry a small JSON body only). Assume separate teams own those.
What to produce. A high-level architecture covering: the schedule store and how you index/partition it to find due jobs cheaply at scale; how a due job is claimed by exactly one worker (leader election, leases, or partition ownership) so the same slot is not fired by two nodes; the timezone/DST computation of the next fire time (including skipped and repeated wall-clock hours); the delivery path with retries and how at-least-once vs at-most-once is chosen per job; the catch-up / missed-run policy after the service itself was down (fire every missed slot, coalesce to one, or skip); and the failure behavior when a scheduler node dies mid-slot. Sketch the major components and the request/timer flow between them; we probe specifics in checkpoints.