Assertion: crawler | /r/CapableOf | visit site
Facets:
value: by limiting it’s speed
type: MANNER
value: without approval
type: MANNER
value: many times
type: TEMPORAL
value: often
type: TEMPORAL
Source sentences:
text: Crawlers consume resources on visited systems and often visit sites without approval.
text: As our crawlers visit these websites, they use links on those sites to discover other pages.
text: The file can also be used to limit specific spiders access to any or all of the site, and can also be used to control how many times the crawler visits the site, by limiting it’s speed or the times when the crawler can visit.