Design a Web Crawler at Scale
Design a web crawler that, given seed URLs and two primitives — fetchPage(url) and parseUrls(html) — visits as much of the web as its budget allows and stores what it fetched.
Cover the frontier of URLs to visit and how it is partitioned, how fetchers are scheduled so no host is overwhelmed, how duplicates are detected by URL and by content, how failures and slow hosts are handled so the crawl never stalls, and how the system resumes after a crash.
Hints
Hint 1
Partition the frontier by host so one worker owns a host's queue and can enforce politeness — a delay between requests and a concurrency of one or two per host.
Hint 2
Dedup URLs after normalisation with a set or a Bloom filter; dedup content by hashing the page so mirrors are stored once.
Hint 3
Retries with backoff per URL, a per-host failure budget that pauses a misbehaving host, and timeouts so a slow host cannot hold fetchers.
Asked at
Sign in to talk to the interviewer — ask what's in scope, what scale to design for, or for a nudge when you're stuck. Conversations save to your account.