Assertion: web crawler | /r/CapableOf | obey instruction
Facets:
value: in a file
type: LOCATION
Source sentences:
text: While Googlebot and other respectable web crawlers obey the instructions in a robots.txt file, other crawlers might not.
text: While Googlebot and other respectable web crawlers obey the instructions in a file, other crawlers might not.