Design a Distributed Tracing Collector

Design the collection backend for a distributed tracing system (think an internal alternative to a hosted APM). Instrumented services across a large engineering org emit spans; your system ingests them, stitches the spans of one request into a single end-to-end trace, decides which traces to keep, and lets engineers pull a trace by id or search for slow and failing traces.

Operating context. Roughly 500 services run a per-host agent that ships spans over OTLP/gRPC. Peak is about 2,000,000 spans/sec, average span 500 bytes, and a single trace fans across dozens of services. Spans for one trace arrive out of order, land on different collector nodes, usually within a few seconds of each other but occasionally minutes late. The org wants tail-based sampling: keep every error or slow trace, sample the rest down. Retention is 7 days hot, 30 days cold.

Out of scope. The client instrumentation and SDK, the metrics and logs pipelines, the query/visualization UI, and any cross-org tenancy or billing. Assume other teams own those.

What to produce. A high-level architecture covering: the ingest path from agent to collector; how spans that share a trace id are brought together when they land on different nodes; where the sampling decision is made (head vs tail) and where spans buffer while you wait; the storage model and partitioning for both trace-by-id lookup and trace search; and how late-arriving spans are handled. Sketch the components and the span flow between them; we will probe specifics in checkpoints.