pruning long documents for distributed information retrieval
Query-based sampling is a method of discovering the contents of a text database by submitting queries to a search engine and observing the documents returned. In prior research sampled documents were used to build resource descriptions for automatic database selection, and to build a centralized sample database for query expansion and result merging. An unstated assumption was that the associated storage costs were acceptable.When sampled documents are long, storage costs can be large. This paper investigates methods of pruning long documents to reduce storage costs. The experimental results demonstrate that building resource descriptions and centralized sample databases from the pruned contents of sampled documents can reduce storage costs by 54-93\% while causing only minor losses in the accuracy of distributed information retrieval.
prune documentation distributed information retrieval distributed information retrieval sampling method contention text database queries queries search engine research resource select centrality merging association storage paper experimentation demonstrate accuracy document pruning query-expansion query