Problem statement
Design a streaming join service that joins two or more high-volume event streams in real time — for example ad impressions joined to their clicks — within a bounded time window, emitting enriched joined records downstream.
Operating context. A large left stream (impressions, ~200,000 events/sec) must be joined by key to a smaller right stream (clicks, ~5,000 events/sec) whenever a match occurs inside a window such as 30 minutes. Events arrive late and out of order. The service maintains large keyed window state, emits matched pairs promptly, and — for a left-outer join — emits unmatched left events once their window closes. Hot keys and skew are expected.
Out of scope. The upstream producers, the downstream sink's own delivery guarantees, the attribution / fraud business rules, and any SQL-like query language.
What to produce. A high-level architecture covering: repartitioning both streams by the join key so co-keyed events co-locate, the windowed keyed-state store and how its size is bounded, watermark and late-event handling with a lateness grace period, join semantics (inner versus left-outer) including timeout emission on window close, checkpointing for at-least-once (or exactly-once) output and recovery, and handling of key skew and hot keys. Sketch the components and flow; we will probe specifics at checkpoints.
Requirements
This assessment is a Premium feature.
The statement above is free to read. The functional and non-functional requirements, and the graded canvas that scores your design against them, come with Premium.
Topics
- System Design HLD
- Data Streaming
- Data Stream-Join
- Patterns Windowing
- Scaling Stateful