Design a Batch ETL Scheduler

Design a batch ETL scheduler that runs directed-acyclic-graph (DAG) data pipelines on a schedule and on data availability, dispatching tasks in dependency order with retries, backfills, and SLA tracking.

Operating context. Thousands of pipelines each model an extract-transform-load flow as a DAG of tasks that run on a shared worker fleet. Triggers are both time-based (cron-like) and data-based (a task runs once an upstream dataset partition lands). The system dispatches tens of thousands of task runs per day, must record durable run state, retry transient failures idempotently, and correctly handle missed or late intervals — all without a single scheduler becoming a bottleneck or single point of failure.

Out of scope. The transform logic inside each task, the compute engine (warehouse / Spark) internals, data-quality validation rules, and the dataset metadata catalog.

What to produce. A high-level architecture covering: the scheduling loop that resolves DAG dependencies and time / data triggers per interval, the executor and worker model, durable task-run state tracking, retries with backoff plus idempotent completion, catch-up / backfill of missed intervals under a concurrency cap, SLA-miss detection and alerting, and how scheduler ownership is partitioned so it scales and survives failure. Sketch the components and flow; we will probe specifics at checkpoints.