Problem statement
Design a health-check aggregator that continuously probes a large set of services and dependencies from several vantage points, combines the raw probe results into a per-service and per-dependency health verdict, and serves that verdict to dashboards and to automation that gates traffic. The hard part is telling a real outage apart from a single flaky prober, and doing it within seconds.
Operating context. The system probes about 30,000 target endpoints (HTTP, gRPC, TCP) plus dependency edges, at per-target intervals of 5-30 s, from 5 or more regions. A service usually has many instances, and a target is probed from multiple vantage points, so the verdict is a rollup with quorum. Automation queries the current verdict synchronously to decide whether to route to a target, so false positives are expensive.
Out of scope. Remediation and auto-healing actions, the target's own health-endpoint implementation, alert notification routing, and load-balancing or capacity decisions. Assume those consume your verdict or live elsewhere.
What to produce. A high-level architecture covering: how probers are distributed across regions and assigned targets; how many-probers-per-target and many-instances-per-service results are aggregated with quorum; the health state model with hysteresis; storing and serving the current verdict plus a short history; and how you distinguish a prober-side network partition from a true target failure. Sketch the components and the flow; we will probe specifics in checkpoints.
Functional requirements
- Probe registered targets over HTTP, gRPC, and TCP at per-target intervals from multiple regional vantage points.
- Aggregate multi-prober, multi-instance results into a per-service and per-dependency verdict using quorum.
- Apply hysteresis so a single failed probe or one flaky prober does not flip a service to unhealthy.
- Serve the current verdict and a short history for dashboards and for automation to query synchronously.
- Distinguish target-down from this-vantage-point-can't-reach-it using cross-vantage agreement.
Non-functional requirements
- Probe 30,000 targets at 5-30 s intervals — about 1,000,000 probes/min total — across 5 or more regions.
- Reflect a true outage in the aggregated verdict within p95 < 15 s of onset.
- Health-read API p99 < 100 ms while sustaining 50,000 verdict reads/sec from automation.
- Availability 99.99% for the read/serve path; loss of one prober region must not blind the system.
- Low false-positive rate: no unhealthy flip from a single prober or a transient blip under the hysteresis policy.
- Store 7 days of health history at reduced resolution for post-incident review.
Topics
- System Design HLD
- Observability Health
- Probing Distributed
- Aggregation Quorum
- State Hysteresis