Problem statement
Design a change-data-capture (CDC) service that streams row-level changes from operational databases into heterogeneous downstream consumers — search indexes, caches, and an analytics warehouse — in near-real-time.
Operating context. Several OLTP source databases commit tens of thousands of writes per second in aggregate. Consumers are diverse and independent: a search indexer, a cache invalidator, and a warehouse loader, each progressing at its own pace. The service must deliver ordered, at-least-once change events per row key and let a lagging consumer catch up without touching the source database's hot path. End-to-end capture-to-delivery latency should sit under 5 seconds at p99.
Out of scope. Automated DDL / schema-migration handling, the source databases themselves, consumer-side business logic, and cross-database distributed transactions. Assume separate teams own those.
What to produce. A high-level architecture covering: how changes are captured (log tailing versus triggers versus polling), the reader / connector tier, per-key ordering and offset tracking, the transport that buffers events, how consumers subscribe and persist their read position, handling of source failover and log truncation, and the delivery guarantee (at-least-once versus exactly-once) with its implications for consumers. Sketch the major components and the flow between them; we will probe specifics at checkpoints.
Requirements
This assessment is a Premium feature.
The statement above is free to read. The functional and non-functional requirements, and the graded canvas that scores your design against them, come with Premium.
Topics
- System Design HLD
- Data Cdc
- Data Streaming
- Patterns Event-Driven
- Scaling Throughput