Smart distributed web crawler
MySpiders: Evolve Your Own Intelligent Web Crawlers