Assertion: web crawler | /r/CapableOf | check a page’s robots.txt file
Facets:
value: always
type: TEMPORAL
value: before crawling the page
type: TEMPORAL
Source sentences:
text: Using Robots.txt Protocols: Webmasters can use robots.txt protocol to communicate with web crawlers, which always check a page’s robots.txt file before crawling the page.