Design a Distributed Job Scheduler

Design the backend for a distributed job scheduler: a service that accepts jobs from many client teams, durably enqueues them, and hands them out to a fleet of stateless workers that run the actual work (think the async task backbone behind a large SaaS — send-email, video-transcode, nightly-report, webhook-delivery).

Operating context. Around 200 client services submit jobs through one API. Steady state is roughly 20,000 submissions/sec with bursts to 60,000/sec when a big batch job fans out. A worker fleet of ~2,000 processes pulls work; a single job runs from 50 ms to 15 minutes. Jobs carry a priority (0 = interactive, higher = background) and a queue name; no low-priority queue may be able to starve an interactive one indefinitely. Workers are unreliable — a process can crash, hang, or get network-partitioned mid-run — so a leased job that is never acknowledged must become runnable again, and the business wants each job's side effect to happen effectively once even though at-least-once delivery is the honest floor.

Out of scope. The worker business logic itself, the client SDK's language bindings, cron / recurring-schedule expansion (assume every job is already a concrete one-shot instance), authn/authz of submitters, and the metrics/alerting pipeline. Assume separate teams own those.

What to produce. A high-level architecture covering: the components and the request paths for submit, lease (worker pull), acknowledge/complete, and fail/retry; the job and queue data model and how you partition it so one hot queue does not sink the fleet; how a worker leases a job with a visibility timeout so a crashed worker's job reappears without being run twice concurrently; the retry policy with backoff and a dead-letter path for poison jobs; how priorities and fairness coexist so background floods never starve interactive work; and how you push toward exactly-once effect on top of at-least-once delivery. Sketch the major components and the flow between them; we probe specifics in checkpoints.