Design a Deploy Rollout Controller

Design a deploy rollout controller that progressively rolls a new software version across a fleet, watches health and metric signals as it goes, and automatically pauses or rolls back on regression. It must keep deploys safe, resumable after its own failover, and fully auditable while many rollouts run at once.

Operating context. The controller orchestrates rollouts for about 5,000 services across regions. A rollout defines a strategy: deploy a small canary percentage, bake for a while, then advance through progressive waves to 100%, with each step gated on error-rate, latency, and health signals pulled from the metrics and health systems. On a gate breach it auto-pauses and can auto-rollback. Up to 500 rollouts run concurrently; each must be idempotent, resumable after a controller restart, and operator-controllable (pause, resume, abort).

Out of scope. The actual deploy mechanism (image push, orchestrator apply — assume an executor API), the CI/build pipeline, the metric and health signal sources (you consume their APIs), and infrastructure provisioning. Assume those exist.

What to produce. A high-level architecture covering: the rollout state machine and the control loop that drives it; how canary and wave progression is gated by signals; auto-pause and auto-rollback logic and the safe default when signals are missing; how rollout state is made durable and resumable across controller failover; concurrency across many rollouts; and idempotency of executor actions. Sketch the components and the flow; we will probe specifics in checkpoints.