The rapid growth of the world-wide web poses unprecedented scaling challenges for general-purpose crawlers and search engines. In this paper we describe a new hypertext information management system called a Focused Crawler. The goal of a focused crawler is to selectively seek out pages that are relevant to a pre-defined set of topics. The topics are specified not using keywords, but using exemplary documents. Rather than collecting and indexing all accessible hypertext documents to be able to answer all possible ad-hoc queries, a focused crawler analyzes its crawl boundary to find the links that are likely to be most relevant for the crawl. It avoids irrelevant regions of the web. This leads to significant savings in hardware and network resources, and helps keep the crawl more up-to-date. We describe a prototype implementation that is comprised of three programs integrated via a relational database: a crawler that stores metadata for documents and links in relational tables and is guided by various attributes in these tables, a hypertext classifier that updates the metadata with topic information from a large taxonomy (such as Yahoo!), and a rating system that updates metadata fields signifying the value of a page as a access point for a large number of relevant pages. The two "mining" modules guide the crawler away from unnecessary exploration and focus its efforts on web regions of interest. We report on extensive focused-crawling experiments using several topics at different levels of specificity. Focused crawling acquires relevant pages steadily while standard crawling quickly loses its way, even though they are started from the same root set. Focused crawling is robust against large perturbations in the starting set of URLs. It discovers largely overlapping sets of resources in spite of these perturbations. It is also capable of exploring out and discovering valuable resources that are dozens of links away from the start set, while carefully pruning the millions of pages that may lie within this same radius. Our anecdotes suggest that focused crawling is very effective for building high-quality collections of web documents on specific topics, using modest desktop hardware.
REF is to specifically find the relevant pages for a defined set of subjects.
Focused crawling: a new approach to topic-specific web resource discovery
Computer Networks
Computer Science