Design an Alerting Rules Engine

Design the alerting engine of a monitoring platform. Engineers define rules over metric time series (for example, "p99 latency > 500 ms for 5 minutes"). Your engine evaluates each rule on a schedule, tracks per-series alert state through pending, firing, and resolved, deduplicates and groups related alerts, applies silences, and hands notifications to a downstream router without paging someone every few seconds when an alert flaps.

Operating context. There are roughly 50,000 rules; across their label dimensions they expand to up to 500,000 active label-set series in total (about 10 per rule on average, up to a few thousand for the widest). Rules run on a 15-60 s interval and query a metric backend through a read API. A rule has a threshold and a "for" duration that must hold continuously before it fires. Downstream notifiers (email, chat, pager) are separate systems you feed.

Out of scope. The metric storage and query backend itself, the notification delivery channels, the rule-authoring UI, and incident-management or on-call scheduling. Assume those exist.

What to produce. A high-level architecture covering: how rules are scheduled and sharded across evaluator nodes; how per-series alert state (pending → firing → resolved with the "for" window) is tracked and kept durable across restart and rebalance; deduplication and grouping; silences and inhibition; and how notifications are emitted at-least-once with dedup, without a storm on flaps. Sketch the components and the evaluation flow; we will probe specifics in checkpoints.