Design a Streaming Join Service

Design a streaming join service that joins two or more high-volume event streams in real time — for example ad impressions joined to their clicks — within a bounded time window, emitting enriched joined records downstream.

Operating context. A large left stream (impressions, ~200,000 events/sec) must be joined by key to a smaller right stream (clicks, ~5,000 events/sec) whenever a match occurs inside a window such as 30 minutes. Events arrive late and out of order. The service maintains large keyed window state, emits matched pairs promptly, and — for a left-outer join — emits unmatched left events once their window closes. Hot keys and skew are expected.

Out of scope. The upstream producers, the downstream sink's own delivery guarantees, the attribution / fraud business rules, and any SQL-like query language.

What to produce. A high-level architecture covering: repartitioning both streams by the join key so co-keyed events co-locate, the windowed keyed-state store and how its size is bounded, watermark and late-event handling with a lateness grace period, join semantics (inner versus left-outer) including timeout emission on window close, checkpointing for at-least-once (or exactly-once) output and recovery, and handling of key skew and hot keys. Sketch the components and flow; we will probe specifics at checkpoints.