intelligent web crawler
As the number of Internet users and the number of accessible Web pages grows, it is becoming increasingly difficult for users to find documents that are relevant to their particular needs. Users must either browse through a large hierarchy of concepts to find the information for which they are looking or submit a query to a publicly available search engine and wade through hundreds of results, most of them irrelevant. Web crawlers are one of the most crucial components in search engines and their optimization would have a great effect on improving the searching efficiency. This paper, introduces an intelligent web crawler that uses an ontological engineering concepts for improving its crawling performance. Intelligent crawler estimates the best path for crawling. This is the first crawler that acts intelligently without any relevance feedback or training.
web crawler ontology importance-metrics domain knowledge