Design a Distributed Object Store

Design an internally-hosted distributed object store that engineering teams across the company use as their durable blob backend (think a self-run alternative to a cloud object service). Callers PUT, GET, and DELETE objects addressed by bucket/key, where a single object ranges from a few kilobytes to several hundred gigabytes.

Operating context. The store sits inside one region spanning three availability zones. Ingest is write-heavy from batch pipelines and log shippers — roughly 30,000 PUT/sec at steady state, 90% of them under 1 MB but the remaining 10% carrying the bulk of the bytes (multi-GB media and dataset shards). Reads run around 60,000 GET/sec, sharply skewed: a small hot set of recently-written objects draws most traffic while the long tail is cold. Total footprint is 40 PB and growing 2 PB/month. Read p99 for a full small object must stay under 80 ms; time-to-first-byte for large ranged reads under 200 ms. The stored bytes are the source of truth for downstream systems, so the design target is eleven-nines (99.999999999%) annual object durability.

Out of scope. Cross-region replication and DR, per-object server-side encryption and key management, IAM / bucket policy authorization, lifecycle tiering to cold archival storage, and billing/usage metering. Assume separate teams own each of those.

What to produce. A high-level architecture covering: the request paths for PUT (including multipart upload of large objects), GET (whole-object and byte-range), and DELETE; how objects are split into chunks and how those chunks are placed and replicated across the three AZs to hit the durability target; the split between the metadata service (the bucket/key → chunk-manifest index) and the bulk data plane; the consistency model callers can rely on after a PUT or overwrite; and how the cluster rebalances chunks when nodes are added, drained, or fail. Sketch the major components and the flow between them; we will probe the placement, durability math, and failure behaviour during checkpoints.