Spacerini: Plug-and-play Search Engines with Pyserini and Hugging Face (2023)

Abstract: We present Spacerini, a tool that integrates the Pyserini toolkit for reproducible information retrieval research with Hugging Face to enable the seamless construction and deployment of interactive search engines. Spacerini makes state-of-the-art sparse and dense retrieval models more accessible to non-IR practitioners while minimizing deployment effort. This is useful for NLP researchers who want to better understand and validate their research by performing qualitative analyses of training corpora, for IR researchers who want to demonstrate new retrieval models integrated into the growing Pyserini ecosystem, and for third parties reproducing the work of other researchers. Spacerini is open source and includes utilities for loading, preprocessing, indexing, and deploying search engines locally and remotely. We demonstrate a portfolio of 13 search engines created with Spacerini for different use cases.

## introduction the commoditization of data has transformed datadriven computer science in general @xcite and machine learning (ml) and natural language processing (nlp) in particular @xcite . the race to train ever-larger language models depends so much on having access to immense amounts of text @xcite ) that the datasets have become, as @xcite claims, both technically @xcite and methodologically (jo and gebru, 2020) "too big to document". in practice, this often leads to an approach of training first and asking questions later @xcite , which again is an example of a convenience experiment @xcite , a research approach that depends on the availability of a resource and the ease of a method rather than its suitability to the problem at hand. in this sense, @xcite believe it is important to distinguish between the availability of data 1 https://github.com/castorini/hf-spacerini 2 https://hf.co/spacerini and its appropriateness, especially in light of the misconception that web data represents all human experience and is immune to the ever-widening digital divide @xcite . they suggest that the divide not only exists, but limits the representativeness of the web, which in turn reinforces the biases of the artifacts that use it @xcite . being unable to easily audit large datasets incentivizes researchers to release models trained on data they do not truly understand @xcite leading to model behaviors that are hard to study, predict or trace @xcite @xcite . this is especially problematic in light of the potential realworld harms that ensue @xcite @xcite . being able to properly understand the limitations of our datasets and qualitatively explore them in an ad-hoc fashion is a necessary first step toward understanding the behavior, harmful biases and failure modes of the artifacts that build upon them. understanding the training data is therefore a critical step in the process of releasing and auditing large language models @xcite . it is from this vantage point that we initially developed spacerini as an open-source tool for the quick indexing and deployment of shareable search engines, but have also since come to realize its potential in being useful for an even wider audience interested in making their text artifacts searchable. indeed this includes ir students, digital humanists, shared tasks organizers, and digital investigative journalists, all of whom seek to quickly deploy ad hoc search engines for their research. we cover these use cases in more detail in section 4. spacerini helps streamline the process of auditing large datasets by allowing users to effortlessly index their text collections and deploy them as interactive search applications that can be easily edited in the browser and shared with all stakeholders. it achieves that by "standing on the shoulthe loading, preprocessing, indexing, and app deployment were made using an end-to-end workflow similar to the one showcased in section 3. as discussed in section 1, in the face of an increased scrutiny of the models trained on datasets in question, the topic of data understanding and governance has been gaining more traction, being accepted as an important part of research. efforts such as those of mitchell et al. (2022) contribute frameworks for more standardised and reproducible metrics and measurements of datasets, and we position ourselves as a complementary continuation of their work, focusing on a more curatorial and qualitative assessment that might not readily fit under the umbrella of "measurements". we therefore aim to fill the gap in the evaluation landscape by facilitating qualitative, rather than quantitative analysis of large scale data collections. similarly to the authors of gradio @xcite , a python package for fast development of machine learning demos, we believe that the accessibility of data and model analysis tools is crucial to building both the understanding of and the trust in the underlying resources. the potential of relevance-based interfaces to massive textual corpora, the creation of which can be facilitated by leveraging toolkits such as pyserini @xcite , has previously been tapped into by the researchers at the allen institute of ai who propose a c4 @xcite search engine foot_0 . similar interfaces have also been found useful in more specialised domains, e.g. in covid-related datasets @xcite , news quotes @xcite , or medical literature @xcite . however, while these solutions are undeniably useful, they remain very contextual: dataset-specific, and project-specific. we believe spacerini to be the first generalizable tool which proposes an endto-end pipeline automating the route from raw text to qualitative analysis. ## spacerini spacerini is a modular framework that integrates pyserini with the hugging face ecosystem to streamline the process of going from any hugging face text dataset (or indeed any text dataset)either local or hosted on the hugging face hubto a search interface driven by a pyserini index that can be deployed for free on the hugging face hub. in what follows, we deconstruct an example script foot_1 to showcase the different features enabled by spacerini. when run end-to-end, the script pulls a dataset from hugging face, pre-processes it, indexes it, creates a gradio-based search interface and deploys that as a hugging face spaces demo. this is only meant as a feature-complete demo, and we don't expect most people to want to integrate every step into their workflows, but rather to cherry-pick and decide what best to use depending on context. ## use cases and demonstrations we envision spacerini to be useful primarily to nlp researchers, students, shared task organizers, data scientists, and data annotators, as well as tech-adjacent and -proficient professionals and laypeople. in what follows, we overview a series of 13 use-cases that we implemented and how they might benefit their respective targeted audience. an overview of all 13 search engines can be found at https://huggingface.co/spacerini ; in following inline links to the selected engines, this part of their url prefix is shortened to 'hf' for brevity. ## limitations and future plans the main limitation of the off-the-shelf variant of spacerini is the disk space limit imposed by hugging face spaces, which is currently set to 50 gb for the free tier. 16 while not enough to accommodate entire corpora such as roots or the pile, such datasets are typically amalgamations of constituent datasets which can each be studied independently. this limit has no bearing on spacerini search apps deployed locally. should users still want to get more disk space for their spaces-hosted indexes, they can either pay for an upgrade to a more appropriate tier or see whether they qualify for a free hardware upgrade through the community grants offered by hugging face, in the settings pane of the relevant space. planned improvements include automating the creation of dataset cards (or rather "index cards") when pushing an index to the hugging face hub, better documentation, as well as more fine-grained tokenization support. please also note that spacerini is currently in active development and that the stability of its current api and subpackages isn't guaranteed not to involve breaking changes as we converge toward the first stable version release. we also look forward to community contributions both to the codebase and to the frontend templates, as well as in the form of actual use cases of the library that culminate in the deployment of search apps. ## conclusion we presented spacerini, a modular framework that enables the quick and free deployment and serving of template-based search indexes as interactive applications for ad-hoc exploration of text datasets. the need for such a tool is especially pressing as large language models have come to consume inordinate amounts of text data, reinforcing the need for a qualitative exploration and understanding of datasets to assess them in a way that is impenetrable to quantitative analyses alone. spacerini leverages features from both the pyserini toolkit and the hugging face ecosystem to facilitate the creation and hosting of userfriendly search systems for text datasets. users can easily index their collections and deploy them as ad-hoc search interfaces, making the retrieval of relevant data points a quick and efficient process. the user-friendly interface enables non-technical users to effectively search massive datasets, making spacerini a valuable tool for anyone looking to audit their text collections qualitatively. the framework is open-source and available on github under gh/castorini/hf-spacerini and demo search apps can be found under hf/spacerini the key advantage of spacerini is its ability to simplify the search process, allowing researchers to conduct quick and efficient audits, while abstracting away all the minutiae of indexing data or hosting services. we believe that this provides an opportunity for collaboration and transparency in ir and nlp research. with the creation and sharing of search indexes publicly, practitioners, researchers and the general public can work together to pinpoint problematic content, find duplicates, and identify biases in datasets. finally, we emphasize that spacerini is a first step in the direction of systematic dataset auditing, and more work is still needed to create standardized structures that leverage tools such as ours to properly document the different axes of interest.