Problem statement
Design the platform service that keeps the company's shared caches warm so users rarely hit a cold miss. After a deploy that flushes a cache, a scale-out that adds empty nodes, or a wave of TTL expiries, cold caches send a stampede of traffic to origin databases and latency spikes. This service predicts which keys are about to be needed and pre-populates cache entries ahead of demand, within a strict budget so the warming itself never overwhelms origin.
Operating context. The prefetcher is off the user's critical path — if it fails, requests still work, just slower on a miss — but its whole value is avoiding those misses at the moments that matter. It learns hot keys from recent access patterns, and it must refresh popular entries slightly before they expire to prevent synchronized expiry stampedes. Origin has finite capacity, so warming competes with live traffic and must yield to it under load.
Out of scope. The cache infrastructure and eviction policy itself, the origin databases' design, the deploy/rollout system that triggers flushes, and client-side caching. Assume other teams own those.
What to produce. A high-level architecture covering: how soon-to-be-hot keys are identified from access history, what triggers a warming wave (post-deploy, scale-out, pre-expiry refresh), how warming load is throttled inside an origin budget and yields to live traffic, how work is prioritized when the candidate set exceeds the budget, how the prefetcher scales and avoids duplicate warming across workers, and how its effectiveness (hit-rate lift, coverage) is measured. Sketch the components and flow; we will probe specifics in checkpoints.
Functional requirements
- Identify likely-hot keys from recent cache and origin access patterns.
- Pre-populate cache entries for those keys ahead of demand.
- Trigger warming waves on deploy-driven flush, scale-out of empty nodes, and impending TTL expiry.
- Refresh popular entries before they expire to avoid synchronized-expiry stampedes.
- Prioritize warming within a bounded origin load budget and yield capacity to live traffic under load.
Non-functional requirements
- Warm up to 500,000 keys/minute during a post-deploy wave.
- Cap warming-induced origin load at a stated budget (for example, <= 10% of origin capacity).
- Restore cache hit rate to >= 95% within a bounded window after a flush or scale-out.
- Bounded warm lag: a newly-hot key is populated within a stated few-second target.
- 99.9% availability of the prefetcher; its failure degrades latency but never causes errors.
- Serve a fleet with hundreds of millions of cacheable keys and thousands of cache nodes.
Topics
- System Design HLD
- Platform Caching
- Data Cache
- Scaling Prefetch
- Reliability Cold-Start