Assertion: web crawler | /r/CapableOf | avoid duplicate downloads
Facets:
Source sentences:
text: To Avoid Duplicate Downloads: Web crawlers sometimes modify and standardize URLs so that they can avoid crawling the same resource multiple times.
text: The number of possible URLs crawled being generated by server-side software has also made it difficult for web crawlers to avoid retrieving duplicate content.