How would you design a distributed web crawler?
Take a moment.
Say your answer out loud or write it down first — retrieval is what makes it stick. Reveal when you're ready.
Short answer
A distributed crawler has a URL frontier (priority queue of URLs to visit), fetcher workers that download pages, a deduplication layer (bloom filters or seen-URL store), a parse-and-extract step, and a storage backend. Distribute by partitioning the URL space (hash the domain to a worker), politeness (rate-limit per domain, respect robots.txt), and a scheduler that prioritizes fresh or important pages.
Detailed explanation
Core components: (1) URL frontier — a priority queue (often sharded) holding URLs to crawl, prioritized by importance, freshness, or crawl depth. (2) DNS resolver and fetcher — workers that download pages, ideally partitioned so each worker owns a set of domains (hash domain → worker) to keep per-domain rate limits local. (3) Deduplication — you will encounter the same URL via many paths; use a Bloom filter for fast probabilistic dedup backed by a persistent seen-set (RocksDB or distributed KV). (4) Politeness layer — per-domain rate limiting and robots.txt compliance to avoid getting banned; maintain a per-domain last-access timestamp. (5) Parser — extract text, links, metadata; normalize discovered URLs (resolve relative paths, drop fragments). (6) Storage — raw pages (object store), parsed content (search index), and a link graph. Scaling dimensions: partition the URL space across workers; scale fetchers horizontally; use a message queue (Kafka) between frontier and fetchers for back-pressure and replay. Hard problems: URL explosion (the web is infinite — scope the crawl), politeness at scale (thousands of domains), content freshness (recrawl scheduling), duplicate near-identical pages, and handling JavaScript-rendered content (may need headless browsers). This question tests how you partition work, handle politeness, deduplicate, and scope an unbounded problem.
Example
Architecture:
Seed URLs → Frontier (sharded priority queue by domain hash)
↓
Fetcher workers (each owns a partition of domains)
↓
Dedup (Bloom filter + RocksDB seen-set)
↓
Parser → extract links (back to frontier) + content (to index)
Politeness: per-domain token bucket, robots.txt cache checked per host.Interview tip
Walk the pipeline (frontier → fetcher → dedup → parser → storage) and name one hard problem (politeness, deduplication, or URL explosion). Mentioning a Bloom filter for dedup shows you think about memory efficiency at scale.
Common mistakes
- Not addressing politeness (rate limits, robots.txt).
- Forgetting deduplication — the same URL appears infinitely.
- Treating the web as finite and not scoping the crawl.