GAIA Search: Hugging Face and Pyserini Interoperability for NLP Training Data Exploration (2023)

Abstract: Noticing the urgent need to provide tools for fast and user-friendly qualitative analysis of large-scale textual corpora of the modern NLP, we propose to turn to the mature and well-tested methods from the domain of Information Retrieval (IR) - a research field with a long history of tackling TB-scale document collections. We discuss how Pyserini - a widely used toolkit for reproducible IR research can be integrated with the Hugging Face ecosystem of open-source AI libraries and artifacts. We leverage the existing functionalities of both platforms while proposing novel features further facilitating their integration. Our goal is to give NLP researchers tools that will allow them to develop retrieval-based instrumentation for their data analytics needs with ease and agility. We include a Jupyter Notebook-based walk through the core interoperability features, available on GitHub: https://github.com/huggingface/gaia . We then demonstrate how the ideas we present can be operationalized to create a powerful tool for qualitative data analysis in NLP. We present GAIA Search - a search engine built following previously laid out principles, giving access to four popular large-scale text collections. GAIA serves a dual purpose of illustrating the potential of methodologies we discuss but also as a standalone qualitative analysis tool that can be leveraged by NLP researchers aiming to understand datasets prior to using them in training. GAIA is hosted live on Hugging Face Spaces: https://huggingface.co/spaces/spacerini/gaia .

## introduction training large language models, or llms @xcite @xcite @xcite @xcite , established itself as the central task of the modern natural language processing (nlp) research. the attempts to understand the scaling laws of llms led researchers to believe that simply increasing the number of parameters may not bring the desired improvements without a simultaneous increase in the size of the llm training data @xcite . these observations only increased an already pressing need for massive textual datasets, fueling the proliferation of web-based corpora of tb-scale created with varying levels of curation and quality control. rather than investing in scraping the web on their own, dataset creators typically turn to common crawl foot_0 as the main source of text to include in their corpora. a repository of web snapshots dating back to 2011, common crawl contains various types of low-quality text @xcite . pre-processing steps commonly introduced by dataset creators aiming to filter out undesired content include removing any documents with words matching a pre-defined, static blacklist, like in the case of c4 @xcite , perplexitybased filtering like in ccnet and roots @xcite , removing malformed text via simple text statistics like in the case of oscar @xcite or through deduplication, studied extensively by @xcite . however, the generated artifacts still tend to contain a multitude of worrying phenomena, such as synthetic data @xcite , private and copyrighted data @xcite or incorrect language codes and translations @xcite . a lack of representation of diversity and socio-cultural and socio-economic biases constitute another big challenge of common crawl and datasets derived from it @xcite @xcite @xcite . aware of the mounting problems with training data for modern llms, and appreciating the value of data exploration for better modeling in general, we focus our current work on building tools that can facilitate the qualitative analysis of nlp datasets. we propose to leverage the extensive experience of the information retrieval community in building relevance-based search indices for large-scale document collections and put it into practice in the context of nlp data exploration work. we follow with a demonstration of ways in which the interoperability between pyserini (lin et al., 2021), a leading toolkit for reproducible ir research on one side, and hugging face 2 , a platform for open ai research on the other, can be leveraged to build tools for easy and effective analysis of textual data. to facilitate the adoption of the proposed methods we provide a collection of jupyter notebooks with step-by-step explanations of explored functionalities available at github.com/ huggingface/gaia/tree/main/notebooks. finally, we release gaia-a simple, yet powerful search engine giving relevance-based interface to four popular, large-scale, textual datasets, namely c4 @xcite , the pile @xcite , roots @xcite and captions from laion-2b-en @xcite . all considered datasets rely to a big extent on data mined from common crawl. gaia benefits from the interoperability between pyserini and hugging face that 2 https://huggingface.co/ we discuss in the first part of the paper, while also constituting a standalone contribution which can benefit the nlp research community by making it easy to study leading corpora qualitatively. gaia is available online at hf.co/spaces/spacerini/ gaia. ## background the ability to analyze large collections of textual data is core in multiple research and engineering disciplines. while the industrial standard is to rely on robust, scalable database and data analytics infrastructure, in the research environment, we typically resort to more local, granular and flexible, if ad-hoc, solutions which leverage toolkits such as numpy @xcite , pandas (pandas development team, 2020; wes mckinney, 2010), scipy @xcite ) and others. a common research approach to data analytics involves using one of the aforementioned packages in combination with jupyter notebooks foot_1 . notebooks make it easy to deploy and share analyses, however, typically they remain essentially non-interactive, requiring at least a basic understanding of programming to be able to work with them efficiently. with the commodification of ai, and nlp in particular, and the expansion of nlp technologies into research areas beyond ai @xcite @xcite , the need for easy to use, no-code tools for understanding ai artifacts arises. this need is partly addressed by python packages such as streamlit foot_2 and gradio foot_3 , designed to facilitate the creation of interactive machine learning (ml) demos. as the authors of the gradio white paper @xcite point out, the accessibility and ease of use of the analysis tools is critical if we want to build an understanding of ai and trust in it. the hugging face spaces platform, providing free hosting of both streamlit, gradio, and docker-based applications, serves this exact purpose. however, it puts emphasis on demonstrating the capabilities of models while paying less attention to the datasets used to train them. even more so than in nlp, the evaluation of ir systems is heavily dependent on the implementation details of the retrieval systems serving the search indices being evaluated. the lack of standardisation of ir evaluation was the main motivator behind creating anserini @xcite , a lucene foot_4 -based toolkit for reproducible ir research, and the follow-up pyserini @xcite -a convenient python api to the underlying javabased implementation of anserini. while it is relatively easy to build and serve search indices backed by pyserini and lucene, the task of building and deploying interactive user interfaces generally comes with a higher engineering barrier of entry. relevance-based search interfaces have been previously explored in the context of nlp-e.g. in the c4 @xcite , in covid-related datasets @xcite or in news quotes @xcite . rather than focusing only on providing finished artifacts, however, we intend our current work to serve as a reference and inspiration for nlp researchers looking to develop and deploy similar applications by themselves. we attempt to bring together the power of pyserini-backed retrieval and the agility of ml demo development within the hugging face ecosystem to serve the goal of building intuitive data exploration tools. we believe that resulting applications will make a great difference for nlp researchers trying to study their data qualitatively, as well as to non-technical researchers looking for tools allowing them to perform dataset analysis in a no-code fashion. we propose our search engine gaia as a compelling case in point. ## pyserini and hugging face: from data to search in the current section we discuss core components which need to be considered when building a search application for textual datasets. we focus on how each step can be facilitated by the use of pyserini, hugging face, or a combination of the two. we also provide hands-on tutorials covering basic concepts and search engine building blocks such as data loading and indexing, tokenization, search, and index analysis. we further release the preprocessing, backend and frontend code that allowed us to index 3.5 billion documents-chunked into 5.8 billion snippets-and serve 5.55tb worth of bm25 indexes. ## case study: gaia search relevance-based search tools have the potential of the largest impact on massive-scale datasets, common in modern nlp. unlike with smaller data collections, where simpler investigation strategies, e.g. via a combination of pandas and jupyter notebooks, may be feasible, huge datasets are generally too cumbersome to process this way. a big benefit of search engines in the form that we propose is also the fact that after being set up, they require no engineering skills or extensive computing resources to operate, expanding the community of potential users. we demonstrate this with gaia search, available online at hf.co/spaces/spacerini/gaia. ## limitations and future plans a major area for consideration when developing data access tools is that of data governance, privacy and data ownership @xcite . in our current work we focus on the technical aspects of giving access to large data collections, however, we urge users to consider data governance principles when designing their own tools. in terms of the infrastructure, the cost and complexity of hosting the retrieval index falls on the creator of the tool, which can be easy to manage for small datasets but becomes more problematic when entering the realm of tb-scale corpora. we are currently investigating a parallel workstream that could address this limitation at least partly. ## conclusions we showcase interoperability between hugging face and pyserini and provide value to the nlp community by demonstrating easy ways to perform high-quality, large-scale retrieval with open-source tools. we also introduce gaia -a search engine for retrieval-based exploration of four major textual datasets. we wish to encourage nlp and ir practitioners to follow our examples and build their own tools to explore both large and smaller-scale textual datasets.