Problem statement
Design a configuration distribution service for a large fleet. Operators publish versioned config for applications; your system reliably propagates each new version to every running instance that subscribes, in a controlled order (canary, then waves), with instant halt and rollback — while every instance keeps running on last-known-good if the control plane is unreachable.
Operating context. About 200,000 application instances across several regions subscribe to config namespaces. When an operator publishes a version it should reach subscribers within seconds, staged: a small canary percentage first, then progressive waves, with the ability to pause or roll back at any point. Config blobs are small (up to a few hundred KB) but read by everything, so availability of the read path matters more than write throughput.
Out of scope. Secret material (a separate rotation service owns that), how the application interprets its config, feature-flag targeting logic, and per-user configuration. Assume those live elsewhere.
What to produce. A high-level architecture covering: how instances subscribe and receive updates (push vs long-poll vs watch stream); versioning and atomic publish; staged rollout control (canary, waves, halt, rollback) and who decides an instance's bucket; how instances cache last-known-good and behave during a partition; and how you fan out to 200,000 subscribers without a thundering herd. Sketch the components and the flow; we will probe specifics in checkpoints.
Functional requirements
- Store versioned, immutable config per namespace with an atomic pointer to the current version.
- Let instances subscribe to namespaces and promptly receive the current version and subsequent updates.
- Support staged rollout: a canary percentage, then progressive waves, with operator halt and one-click rollback.
- Serve last-known-good so an instance can start and keep running when the control plane is unreachable.
- Report which versions are live on how many instances for rollout observability.
Non-functional requirements
- Fan an update out to 200,000 subscribers with propagation p95 < 10 s and p99 < 30 s.
- Read/subscribe availability 99.99%; instances are never blocked by control-plane downtime.
- Sustain 200,000 concurrent watchers and a publish rate up to 50 versions/min across namespaces.
- Config blob up to 256 KB; retain full version history for 90 days.
- A rollback command reaches all live subscribers within p95 < 10 s.
- No split-brain: a subscriber must never oscillate between two versions it believes are current.
Topics
- System Design HLD
- Observability Config
- Distribution Fanout
- Rollout Staged
- Consistency Versioning