Overview
Design a distributed crawler that walks the web, obeys robots.txt and per-domain politeness, avoids loops and traps, and stores the pages and links it finds.
Requirements
- 01Crawl and store 1 billion pages a month, about 400 a second
- 02Respect robots.txt and throttle requests per domain
- 03Detect duplicate URLs and near-duplicate page content (hashing, SimHash)
- 04Escape crawl traps such as infinite recursive links and generated calendar pages
- 05Keep the raw HTML and metadata in distributed storage
Concepts in play
URL FrontierBloom FilterPolitenessDNS CacheBigtable