Design a large-scale web crawler that starts from a set of seed URLs, fetches pages across the public web, extracts and follows their outbound links, and hands the fetched content to a downstream indexing pipeline (think the crawler that feeds a general-purpose search engine). The whole system runs continuously and must stay a courteous, well-behaved visitor to every site it touches while still making forward progress at scale.

Operating context. The target corpus is on the order of 5 billion reachable pages spread across roughly 200 million hosts, and you want a full recrawl cadence that keeps high-value pages fresh within a day and the long tail within a few weeks. Steady fetch throughput is around 20,000 pages/sec across a fleet of stateless fetcher workers, with each fetched page averaging ~80 KB compressed. Link density is high: a typical page yields tens of outbound links, so the frontier of known-but-unfetched URLs balloons into the hundreds of billions and cannot live in memory on one node. Two hard realities dominate the design: you must respect each host's robots.txt and a polite per-host request rate (never hammer one server), and the web is adversarial — infinite calendars, session-id URLs, and spider traps will try to trap you in an unbounded crawl of near-identical pages.

Out of scope. The search index / ranking / query serving that consumes your output, JavaScript rendering of client-side pages, the ML that scores page importance (assume it hands you a priority signal), CAPTCHA solving, and login-gated or paywalled content. Assume separate teams own those.

What to produce. A high-level architecture covering: the major components and the end-to-end request flow from seed URL to stored page; the URL frontier design and how it enforces per-host politeness and robots.txt while staying globally high-throughput; how you deduplicate both URLs (already-seen) and content (near-duplicate pages under different URLs); how work is distributed across fetchers and how a host is mapped to a worker; the prioritization and recrawl-scheduling policy; how you detect and escape crawler traps and cap unbounded link expansion; and how fetched pages plus crawl metadata are stored and partitioned. Sketch the major components and the flow between them; checkpoints will probe the frontier, the politeness enforcement, and trap avoidance specifically.