Problem statement
Design a backfill orchestrator that reprocesses large historical datasets — re-running a pipeline over months of past data after a bug fix or a new derived column — without disrupting the live pipeline that is still processing current data.
Operating context. A production pipeline continuously processes fresh data. Occasionally the team must recompute historical output over a large range — thousands of partitions, petabytes — using updated logic. The backfill runs alongside live processing on shared, finite compute; it must be resumable, must not corrupt or block live output, must replace historical results atomically, and must be throttled so live latency barely moves. Progress tracking and observability are essential over a job that may run for days.
Out of scope. The transformation logic, the compute engine, the dataset catalog, and how downstream consumers are notified of restated data.
What to produce. A high-level architecture covering: chunking the backfill range into independently schedulable, idempotent, resumable units, the orchestration state machine tracking each chunk's progress durably, throttling that yields to live-pipeline priority and is adjustable at runtime, writing output to a staging location and atomically publishing it over the historical output per partition, idempotency and resume-after-failure, and progress / ETA reporting. Sketch the components and flow; we will probe specifics at checkpoints.
Must do 5
Functional requirements
- Split a backfill request (range plus logic version) into independently schedulable, idempotent chunks.
- Execute chunks under a configurable concurrency/throughput cap that yields to live-pipeline priority.
- Persist per-chunk progress so a paused or failed backfill resumes without reprocessing done chunks.
- Write backfilled results to a staging location and atomically publish them over the historical output.
- Report overall progress, per-chunk status, and an estimated completion time for a running backfill.