an investigation of web crawler behavior : characterization and metrics .
Focused crawling: a new approach to topic-specific web resource discovery