Web Crawler Architecture.
UbiCrawler: a scalable fully distributed Web crawler