what do web crawlers do
A web crawler (also known as a web spider or web robot) is a program or automated script which browses the World Wide Web in a logical, automated manner. This process is usually known as Web crawling or spidering.