Design a Metrics and Monitoring System

Design the backend for a metrics and monitoring platform (think an internal observability service every engineering team at a large company points their fleet at) that ingests numeric time-series from tens of thousands of hosts, stores them cheaply for years, answers dashboard and ad-hoc queries fast, and fires alerts when a series crosses a rule. The whole difficulty is cardinality and volume: a single metric like http.request.latency explodes into millions of distinct series once you tag it by host, endpoint, status code, and region.

Operating context. Roughly 50,000 emitting hosts, each publishing about 2,000 series at a 10-second resolution — on the order of 10 million data points per second at steady state, with 3x bursts during incidents when everything alarms at once. Active cardinality is around 100 million live series and grows as teams add tags. Dashboards issue ~50,000 queries/sec, heavily skewed to the last hour but occasionally scanning 90 days. Alert rules (tens of thousands of them) must evaluate on a fixed cadence and detect a breach within ~30 seconds. Data older than a few days is almost always read at coarse resolution.

Out of scope. The agent/SDK on the host, log and distributed-trace pipelines (this is numeric metrics only), the dashboard-rendering frontend, on-call paging/escalation policy and notification delivery, and anomaly-detection ML. Assume separate teams own those; you expose the APIs they call.

What to produce. A high-level architecture covering: the ingest path and whether hosts push or a collector pulls; how you tame high cardinality (the series-key / label index, cardinality limits, sharding by series); the time-series storage engine and its on-disk shape; the rollup / downsampling pipeline and the retention tiers it feeds; the read path and query engine (recent vs historical, aggregation across many series); the alerting evaluator and how it reads without starving dashboards; and how ingest, storage, and query scale independently. Sketch the major components and the flow between them; checkpoints will probe cardinality and the rollup/retention design specifically.