Design the backend that collects structured and unstructured logs from a large fleet of services, moves them through a resilient pipeline, and lets engineers search them during an incident (think the log tooling behind a big multi-service product). The design tension is volume that swings wildly — a quiet fleet suddenly emits a torrent when something breaks — and the fact that the exact moment logs matter most is the moment the pipeline is under the most pressure.

Operating context. Roughly 40,000 hosts and containers each run a collection agent that tails log files and captures stdout. Steady state is about 2 million log lines/sec, averaging ~250 bytes per raw line (~500 MB/sec raw). During an incident or a bad deploy the rate can spike 8–10x for minutes at a time as error logs and stack traces flood in. Logs arrive out of order and with duplicates (an agent retries after a network blip). Engineers query recent logs constantly (last 15 minutes, filtered by service, level, trace-id) and occasionally reach back days. Hot recent data must be fast; old data is rarely touched but must be retained for a fixed window for audit.

Out of scope. The agent's on-host file-tailing internals and log-rotation handling, the query UI and dashboards, alerting/anomaly-detection rules built on top of logs, metrics and traces as separate pipelines, and access-control/PII redaction policy (assume a redaction hook exists). Assume services emit logs; you own everything from the agent's send call onward.

What to produce. A high-level architecture covering: the collection agents and how they ship logs; the ingest tier and the buffering layer that absorbs bursts; how backpressure propagates when a downstream stalls so the pipeline sheds or slows load instead of falling over or losing everything; the parsing/enrichment stage (structuring raw lines, attaching host/service/trace metadata) and how it stays idempotent given duplicate delivery; the indexing and storage layout that keeps recent data hot and searchable while older data ages into cheaper tiers under a retention policy; the search path over both hot and cold data; and the sampling/shedding strategy for the incident spike. Sketch the components and the end-to-end flow; checkpoints will probe the backpressure behavior and the hot-vs-cold retention split specifically.