Assertion: web crawler | /r/CapableOf | obey restriction
Facets:
Source sentences:
text: Your web crawler must obey the restrictions specified by web site administrators in robots.txt files.