How the interviewer opens it
“Design a web crawler that downloads billions of pages so we can build a search index.”
The prompt is vague on purpose. The first few minutes are yours: narrow it to the core features and agree on the numbers before you draw anything.
What to clarify first
- scale
- content
- politeness
- freshness
The answers stay with the interviewer. In a SysRound round, Alex answers them as an interviewer would.
Scope of a strong answer
Core requirements
- Start from seed URLs
- Fetch and store pages
- Extract and enqueue new links
- Avoid re-crawling duplicates
Usually out of scope
- Indexing/ranking
- JavaScript rendering (mention only)
What interviewers probe
- Frontier and politeness
- Throughput vs never hammering one host.
- Dedup and traps
- Billions of URLs, infinite URL spaces, mirrored content.
Common mistakes
- No politeness
- No URL normalization
- Ignoring DNS as a bottleneck
The bar by level
- Mid-level (L4/E4)
- Frontier + fetch + parse loop, dedup set, robots.txt.
- Senior (L5/E5)
- Per-host politeness design, Bloom filters with sizing, failure recovery of in-flight work, traps.
- Staff (L6+/E6+)
- Prioritization/freshness model, global distribution, cost and storage tiering.