Assertion: crawler | /r/CapableOf | try to download the site's robots.txt file
Facets:
value: before crawling a web site
type: TEMPORAL
Source sentences:
text: Before crawling a web site, your crawler should try to download the site's robots.txt file.