High-performance web crawling
Mercator: A scalable, extensible Web crawler