Assertion: crawler | /r/CapableOf | respect the page-level robot exclusion
Facets:
Source sentences:
text: The crawler also respects the page-level robot exclusion specified in HTML metatags.