Assertion: page | /r/HasProperty | crawled
Facets:
Source sentences:
text: The robots.txt file is then parsed and will instruct the robot as to which pages are not to be crawled.