Design a distributed key-value store — a horizontally scalable service that stores opaque values under string keys and stays available and low-latency while nodes join, leave, and fail (think an internal storage engine that many product teams call as a shared dependency).

Operating context. The cluster runs across three availability zones in one region, on the order of 60–200 commodity nodes, each with local SSD. It holds ~50 TB of data across ~5 billion keys; values are small (median ~1 KB, capped at 256 KB). Traffic is mixed and bursty: ~150k reads/sec and ~40k writes/sec at steady state, spiking 3× during flash events. Callers want a single-key get/put/delete API with a per-call knob to trade latency for freshness. Individual node failures are routine, not exceptional — assume something is always degraded. Latency target: p99 get < 10 ms, p99 put < 20 ms in-region.

Out of scope. Multi-region / cross-region replication, secondary indexes and range scans, transactions across multiple keys, an on-disk storage-engine design (assume a per-node LSM/log-structured store exists), authn/authz, and billing. Assume separate teams own those.

What to produce. A high-level architecture covering: how keys map to nodes and how that mapping survives cluster resize with minimal data movement; the replication scheme and how reads and writes use quorums; the per-request consistency knob and what guarantees each setting does and does not give; how writes to a temporarily-down replica are not lost and how divergent replicas are reconciled later; how nodes discover each other and detect failures; and how a single hot key is prevented from melting one node. Sketch the components and the read and write request flows between them; we will probe specifics at each checkpoint.