Assertion: crawler | /r/CapableOf | follow link
Facets:
value: on those pages
type: LOCATION
value: on the web
type: LOCATION
Source sentences:
text: Crawlers look at webpages and follow links on those pages, much like you would if you were browsing content on the web.
text: If crawlers follow that link, it will lead to the first crawler-session and the first time your site is saved in the index.
text: Once a crawler lands on your website it analyses all your content line by line and follows each of the links you have whether they are internal or external.
text: Crawlers look at webpages and follow links on those pages, much like you would if you were browsing content on the web.
text: They begin with a list of links to scan, and then follow the links they find.
text: The crawler follows the links.
text: A crawler follows the links on the web.