Web Crawler Architecture.
Mercator: A scalable, extensible Web crawler