Problem statement
Design a service that aggregates massive geolocated event streams into spatial heatmaps — density by cell — queryable at multiple zoom levels and time windows for operational dashboards and map overlays.
Operating context. Events (demand, incidents, activity) stream in continuously and are geotagged. Dashboards ask for a heatmap tile: give me cell densities for this bounding box, at this zoom, over this time window, optionally filtered to one event type. Both a live view (the last few minutes) and historical views (arbitrary past windows) are needed. The volume forbids scanning raw events per query, so aggregates are precomputed at several resolutions.
Out of scope. Client-side rendering and color mapping of the tile, the raw event producers, per-user personalization of what is shown, and any predictive/forecasting model. Assume producers publish clean geotagged events to your ingest.
What to produce. A high-level architecture covering: event ingestion; the spatial binning scheme (geohash / H3 / S2) and how cells roll up across zoom levels; time-windowed streaming aggregation (tumbling/sliding windows) for the live path versus batch rollups for history; the aggregate storage model keyed by cell, zoom, window, and type; tile query serving with caching and how a bounding box maps to cells; and how the system degrades under load. Sketch the components and the flow from event to a served tile; we will probe specifics in checkpoints.
Functional requirements
- Ingest geotagged events (lat, lng, type, weight, timestamp) at high volume.
- Aggregate event counts and weights into spatial cells at several zoom resolutions.
- Serve a heatmap tile of cell densities for a bounding box, zoom, and time window.
- Support both a live window (last few minutes) and arbitrary historical windows.
- Let a caller filter aggregation to one or more event types before serving the tile.
Non-functional requirements
- Ingest 1,000,000 events/sec; new events reflected in the live view within 10 seconds.
- Heatmap-tile query p99 < 200 ms; serve 50,000 tile requests/sec.
- Support 5+ zoom resolutions and windows from 1 minute to 30 days.
- 99.9% availability of the query path; degrade to a coarser resolution under load.
- Store rolled-up aggregates durably; retain raw events 7 days for recompute.
- Bound storage growth by resolution and TTL policy at ~hundreds of terabytes.
Topics
- System Design HLD
- Geo Spatial-Index
- Analytics Aggregation
- Data Streaming
- Scaling Read-Heavy