Design a data-lake compaction service that continuously rewrites the many small files produced by streaming ingestion into fewer, larger, well-organized files — keeping query performance and storage cost healthy without disrupting concurrent readers or writers.

Operating context. A table-format data lake stores columnar files in object storage with table metadata (manifests and snapshots) providing snapshot isolation. Streaming ingestion writes thousands of small files per hour per table, and queries slow as the small-file count grows. Compaction must merge small files into target-sized ones, commit the result atomically as a new snapshot, tolerate concurrent ingestion commits, and reclaim storage by expiring old snapshots and deleting orphaned files — across thousands of tables and petabytes of data.

Out of scope. The query engine, the ingestion writers, the table-format specification itself, and data-quality checks.

What to produce. A high-level architecture covering: detecting compaction candidates by scanning partition metadata for small files, the job planner and rewrite executor, atomic commit via a new table snapshot so readers see all-or-nothing, conflict detection and retry when ingestion commits to the same partition mid-job, snapshot expiry and orphan-file garbage collection that never deletes a file a live reader needs, and prioritization across many competing tables. Sketch the components and flow; we will probe specifics at checkpoints.