Problem statement
Design a pipeline that ingests a firehose of short public posts, each tagged with zero or more topics (hashtags, entities, phrases), and continuously surfaces the top trending topics — globally and per region — over recent sliding windows (think the "what's trending" rail on a large social feed). "Trending" is not raw volume: a perennially-popular hashtag should not dominate, so you must surface topics whose rate of mention is rising relative to their own decayed baseline, and you must suppress coordinated spam and bot floods so a single actor cannot manufacture a trend.
Operating context. You ingest 500,000 posts/sec at steady state, spiking to 1,500,000/sec during live events; the mean post carries ~2 topics, so ~1,000,000 topic-mentions/sec. The topic space is Zipfian: a few thousand topics dominate while a long tail of tens of millions of rarely-seen topics churns hourly. Consumers (feed services, the trending rail) poll the current top-100 per region and the global top-100 for three windows — 5-minute, 1-hour, and 24-hour — and expect answers with p99 < 200 ms and staleness under ~10 s. Traffic and topic popularity are geographically skewed (a cricket final trends in one region, a storm in another).
Out of scope. Topic extraction / NLP from raw text (assume each post arrives pre-tagged), personalized or per-user trending, ad injection, the post storage system itself, and long-term historical analytics beyond the 24-hour window. Assume upstream delivers an ordered-enough, at-least-once event stream.
What to produce. A high-level architecture covering: the ingest and stream-processing topology; how per-window counts are maintained (sliding windows + time decay); how top-K is computed at this cardinality without keeping an exact count for every one of tens of millions of tail topics (approximate counting / heavy-hitters); the per-region vs. global aggregation and how regional partials roll up; the trend-scoring model (rising-rate vs. raw volume); the spam/bot-suppression stage; and the serving/caching path that answers polls within budget. Sketch the major components and the event flow between them; we will probe specifics during checkpoints.
Requirements
This assessment is a Premium feature.
The statement above is free to read. The functional and non-functional requirements, and the graded canvas that scores your design against them, come with Premium.
Topics
- System Design HLD
- Scaling Write-Heavy
- Scaling Fanout
- Data Timeseries
- Geo Sharding