Assertion: crawler | /r/CapableOf | opt out
Facets:
Source sentences:
text: This directive isn’t as definitive because crawlers can opt-out of obeying your robots.txt files, but it’s handy for controlling your crawl budget.