grid based federated digital library
With the growing acceptance of the Open Archive Initiative (OAI) [16] framework, a number of digital libraries are becoming OAI compliant. This is making it feasible to build an effective federated digital library, which harvests metadata from the OAI-compliant libraries and provides a unified search service over the aggregated metadata. Arc [10] is an example of such a federated digital library. Assuming that a rapid increase (e.g., several orders of magnitude) in the adoption of OAI-PMH [16] occurs, we now have a different problem: how to efficiently discover, harvest and index the burgeoning OAI-PMH corpus. In this project, we are working on using Grid and cluster technology to address these performance issues. In this paper, we focus on the use of Grid for parallelizing the harvesting task for an OAI-based federated digital library. We propose a Grid-based architecture for parallel harvesting that supports: dynamic allocation of harvesting nodes, scheduling of harvesting tasks to maximize the performance, and uniform load distribution for the indexing node. We have implemented and evaluated the proposed architecture on a Grid based on the GT3 toolkit
grid oai-pmh high performance parallel search