Assertion: web crawler | /r/CapableOf | traverse page
Facets:
value: based on the access policy
type: CAUSE
Source sentences:
text: If robots exclusion is enabled (default), then the Web crawler traverses the pages based on the access policy specified in the Web server robots .