HardAI Evaluated

Design a Distributed Web Crawler

Overview

Design a robust, distributed web crawler that traverses the World Wide Web, respects crawl politeness and robots.txt, avoids spider traps/cycles, and extracts links and page contents at web scale.

Requirements

1Crawl and download 1 billion web pages per month (~400 pages/sec throughput)
2Enforce crawl politeness: respect robots.txt rules and throttle requests per domain
3Deduplicate URLs and detect duplicate page content (fuzzy fingerprinting/SimHash)
4Handle crawl traps (infinite recursive URL loops, dynamic calendar pages)
5Store extracted raw HTML content and metadata in high-capacity distributed storage

Architecture Hints (4)

Target Concepts

#URL Frontier#Bloom Filter#Politeness#DNS Cache#Bigtable

Canvas Editor

Design a Distributed Web Crawler

Loading canvas…