optimal crawling strategies for web search engines .
Analysis and characterization of large‐scale Web server access patterns and performance