Problem statement
Design a schema registry service that stores, versions, and enforces the message schemas flowing through a streaming platform, so producers and consumers can evolve independently without breaking one another.
Operating context. Hundreds of streaming topics each carry messages under an evolving schema. A producer registers a schema and receives an immutable schema ID that it embeds in every message; a consumer fetches the schema by ID to deserialize. The registry enforces a compatibility policy (backward, forward, or full) before accepting a new version. The workload is overwhelmingly read-heavy — every producer and consumer caches schemas but must resolve an unknown ID quickly — and a registry outage must not halt already-cached serialization.
Out of scope. The serialization format specification, the message broker, the data records themselves, and access-control policy authoring.
What to produce. A high-level architecture covering: schema storage and per-subject versioning, the register / lookup API and how immutable schema IDs are assigned (and identical schemas deduplicated), compatibility-rule enforcement at registration time, client-side caching and why immutable IDs make it safe, high availability and read scaling including a degraded read path during a writer outage, and how a multi-region deployment keeps IDs globally consistent. Sketch the components and flow; we will probe specifics at checkpoints.
Functional requirements
- Register a new schema version under a subject and return a globally unique, immutable schema ID.
- Look up a schema by ID and list all versions of a subject with their compatibility status.
- Enforce a configurable compatibility policy (backward / forward / full) before accepting a version.
- Serve the latest schema for a subject and resolve a specific version on demand.
- Set or change a subject's compatibility policy without mutating any past version.
Non-functional requirements
- Schema-ID lookup p99 < 10 ms at 100,000 reads/sec (mostly served from client caches).
- Registration p99 < 200 ms including compatibility validation, at up to 50 registrations/sec.
- 99.99% availability for reads; a registry outage must not block already-cached producers.
- Schema IDs are immutable and globally unique across all subjects and regions.
- Store up to 1M schema versions (~5 GB) with all version history retained indefinitely.
- Compatibility checks are deterministic and finish under 100 ms for schemas up to 1 MB.
Topics
- System Design HLD
- Data Schema-Registry
- Data Serialization
- Patterns Versioning
- Scaling Read-Heavy