Design a general-purpose distributed in-memory cache (think a self-hosted alternative to a managed key/value cache) that application services call in front of a slower, authoritative datastore. The cache holds opaque string/blob values keyed by a caller-chosen key, and its whole reason to exist is to shave datastore load and tail latency off a read-heavy workload.

Operating context. Dozens of stateless app servers across a single region call the cache on nearly every request. The working set is roughly 500 GB — far too large for one machine — so it must be spread across a fleet of ~20 cache nodes, each with tens of GB of RAM. Traffic is heavily read-skewed: about 800k GET/sec at steady state against 40k SET/sec, and a handful of "celebrity" keys can absorb a wildly disproportionate share of reads during a spike. The datastore behind the cache can serve only ~30k reads/sec before it browns out, so a cold or thundering-herd cache is an outage, not just a slowdown. Values are small (median ~2 KB, capped at 256 KB), and callers set a per-entry TTL.

Out of scope. The client SDK's exact wire protocol and serialization format, cross-region / multi-datacenter replication, on-disk durability and crash recovery of cached data (treat the cache as volatile), authentication/authorization of callers, and secondary-index or range-query features (this is a pure key lookup, not a database).

What to produce. A high-level architecture covering: how keys are mapped to nodes (the sharding scheme) and whether routing lives in a smart client or a proxy tier; the per-node eviction and TTL story once RAM fills; the replication model that keeps the cache available when a node dies; the read/write integration pattern with the source of truth (cache-aside vs write-through/write-behind) and how stale entries are invalidated; and how you defend a hot key against a cache stampede. Sketch the components and the GET / SET request flow between them; checkpoints will probe the trade-offs.