A focused crawler combinatory link and content model based on T-Graph principles
We present the architectural design of a focused Web crawler that combines link-based and content-based approaches to predict the topical focus of an unvisited page. We present a custom method using the Dewey decimal classification system to best classify the subject of an unvisited page into standard human knowledge categories. To prioritize an unvisited URL, we use a dynamic, flexible and updating hierarchical data structure called T-Graph. It helps find the shortest path to get to on-topic pages on the Web. For a background review, the experimental results from several crawlers are presented. The functional and non-functional requirements of a focused Web crawler are elicited and described, as well as standard evaluation criteria.
t-graph focused web crawler html data information retrieval search engine