Design a Config Distribution Service

Design a configuration distribution service for a large fleet. Operators publish versioned config for applications; your system reliably propagates each new version to every running instance that subscribes, in a controlled order (canary, then waves), with instant halt and rollback — while every instance keeps running on last-known-good if the control plane is unreachable.

Operating context. About 200,000 application instances across several regions subscribe to config namespaces. When an operator publishes a version it should reach subscribers within seconds, staged: a small canary percentage first, then progressive waves, with the ability to pause or roll back at any point. Config blobs are small (up to a few hundred KB) but read by everything, so availability of the read path matters more than write throughput.

Out of scope. Secret material (a separate rotation service owns that), how the application interprets its config, feature-flag targeting logic, and per-user configuration. Assume those live elsewhere.

What to produce. A high-level architecture covering: how instances subscribe and receive updates (push vs long-poll vs watch stream); versioning and atomic publish; staged rollout control (canary, waves, halt, rollback) and who decides an instance's bucket; how instances cache last-known-good and behave during a partition; and how you fan out to 200,000 subscribers without a thundering herd. Sketch the components and the flow; we will probe specifics in checkpoints.