High performance crawling system
Design and implementation of a high-performance distributed Web crawler