Available question text
Design a distributed web crawler using 10,000 machines to crawl 1 billion URLs without any centralized components (no master nodes, databases, queues, or cloud services). Focus on decentralized coordination, load distribution, and handling communication constraints between nodes.
Related questions
Continue with questions that share topics or the same question type.
- Design a concurrent web crawler
System design · Concurrency · System Design
- Design a distributed web crawler
System design · System Design · System Design Fundamentals
- Design a scalable web crawler
System design · Rate Limiting · System Design · System Design Fundamentals
- Design a Scalable, Polite Web Crawler
System design · System Design · System Design Fundamentals
Sources · 1 reference
These records explain where the question appears. A company tag or candidate report does not guarantee a future interview question.
Hello Interview · 2 records
Bloomberg
Design a Web Crawler with Limited Communication
Reported Feb 13, 2025 · 1 source report
JUNIOR
Source snapshot: Nov 12, 2026
View sourceMeta
Design a Web Crawler with Limited Communication
Reported Apr 17, 2026 · 56 source reports
MANAGER, MID_LEVEL, SENIOR, SENIOR_MANAGER, STAFF
Source snapshot: Nov 12, 2026
View source