P rime QA : The Prime Repository for State-of-the-Art Multilingual Question Answering Research and Development (2023)

Abstract: The field of Question Answering (QA) has made remarkable progress in recent years, thanks to the advent of large pre-trained language models, newer realistic benchmark datasets with leaderboards, and novel algorithms for key components such as retrievers and readers. In this paper, we introduce PrimeQA: a one-stop and open-source QA repository with an aim to democratize QA research and facilitate easy replication of state-of-the-art (SOTA) QA methods. PrimeQA supports core QA functionalities like retrieval and reading comprehension as well as auxiliary capabilities such as question generation. It has been designed as an end-to-end toolkit for various use cases: building front-end applications, replicating SOTA methods on public benchmarks, and expanding pre-existing methods. PrimeQA is available at: https://github.com/primeqa .

## introduction question answering (qa) is a major area of investigation in natural language processing (nlp), consisting primarily of two subtasks: information retrieval (ir) @xcite and machine reading comprehension (mrc) @xcite @xcite . ir and mrc systems, also referred to as retrievers and readers, respectively, are commonly assembled in an end-to-end open-retrieval qa pipeline (openqa henceforth), which accepts a query and a large document collection as its input and provides an answer as output @xcite @xcite . the retriever first identifies documents or passages (i.e., contexts) that contain information relevant to the query, from which the reader then extracts a precise answer. alternatively, the reader can also * corresponding author: avi@us.ibm.com be generative and leverage large language models (llms) @xcite . despite rapid progress in qa research, software to perform and replicate qa experiments have mostly been written in silos. at the time of this writing, there is no central repository that facilitates the training, analysis and augmentation of state-of-the-art (sota) models for different qa tasks at scale. in view of the above, and with an aim to democratize qa research by providing easy replicability, here we present primeqa: an open-source repository foot_0 designed as an end-to-end toolkit. it offers all the necessary tools to easily and quickly create a custom qa application. we provide a main repository that contains easy-to-use scripts for retrieval, machine reading comprehension, and question generation with the ability to perform training, inference, and performance evaluation. additionally, several sibling repositories offer features for easily connecting various retrievers and readers, as well as for creating a front-end user interface (ui). primeqa has been designed as a platform for qa development and research, and encourages collaboration from all members of the qa community-from beginners to experts. primeqa has a growing developer base with contributions from major academic institutions. the following is a summary of our contributions: recently, transformers @xcite has become one of the most popular repositories among nlp users. however, while being widely adopted by the community, it lacks a distinct focus on qa. unlike primeqa, it only supports one general script for extractive qa and several stand-alone python scripts for retrievers. similarly fairseq @xcite and allennlp @xcite also focus on a wide array of generic nlp tasks and hence do not solely present a qa repository. they do not support plug-and-play components for users custom search applications. several toolkits exist that cater to building customer-specific search applications @xcite or search-based virtual assistants (ibm, 2020). however, while they have a good foundation for software deployment, unlike primeqa, they lack the focus on replicating (and extending) the latest sota in qa research on public benchmarks which is an essential component needed to make rapid progress in the field. ## primeqa primeqa is a comprehensive open-source resource for cutting-edge qa research and development, governed by the following design principles: 2 https://huggingface.co/primeqa ## services and deployment industrial deployment often necessitates running complex models and processes at scale. we use docker to package these components into microservices that interact with each other and can be ported to servers with different hardware capabilities (e.g. gpus, cpus, memory). the use of docker makes the addition, replacement or deletion of services easy and scalable. all components in the primeqa repository are available via rest and/or grpc micro-services. our docker containers are available on the public dockerhub and can be deployed using technologies such as openshift sample deployment via createprimeqa-app repository grpc rest rest ibm® watson discovery third party integrations user grpc / rest primeqa ui / deployment development build primeqa orchestrator / deployment build development / deployment build development reader, retriever and kubernetes. in addition to the main primeqa repository, we provide three sibling repositories for application deployment: figure 2 illustrates how to deploy a qa application at scale using the core primeqa services (e.g. reader and retriever) and our three sibling repositories. we provide this end-to-end deployment for our demo, however users can also utilize primeqa as an application with their own orchestrator or ui. figure 3 shows an openqa demo application built with the primeqa components. in addition to providing answers with evidence, our demo application features a mechanism to collect user 7 https://www.ibm.com/cloud/watson-discovery feedback. the thumbs up / down icons next to each result enables a user to record feedback which is then stored in a database. the user feedback data can be retrieved and used as additional training data to further improve a retriever and reader model. ## community contributions while being relatively new, primeqa has already garnered positive attention from the qa community and is receiving constant successful contributions from both international academia and industry via github pull requests. we describe some instances here and encourage further contributions from all in the community. we provide support for those interested in contributing through a dedicated slack channel foot_5 , github issues and pr reviews. neural retrievers: colbert, one of our core neural retrievers, was contributed by stanford nlp. since primeqa provides very easy entry points into its core library, they were able to integrate their software into the retriever script run_ir.py independently. their contribution to primeqa provides sota performance on openqa benchmarks by performing 'late interaction' search on a variety of datasets. they also contributed col-bertv2 @xcite and its plaid @xcite variant. the former reduces colbert index size by 10x while the latter makes search faster by almost 7x on gpus. table readers: beihang university and microsoft research asia contributed tapex (liu et al., 2021) as the first generative table reader in primeqa. tapex proposes a novel table pre-training strategy based on a neural sql executor and achieves sota on wiki-sql @xcite and wiki-tablequestions @xcite . they utilize the transformers @xcite sequence-to-sequence trainer for seamless integration into primeqa. lti cmu's neulab contributed omnitab @xcite , which employs an efficient pre-training strategy leveraging both real and synthetic data. this integration happened organically as omnitab builds on top of tapex in primeqa. currently, their model yields the best few-shot performance on wiki-tablequestions, making it also an appropriate candidate system under domain shift. custom search for earth science: nasa re-searchers created a custom search application for scientific abstracts and papers related to earth science which received global attention foot_6 . first, using the top level scripts in primeqa, they trained an openqa system on over 100k abstracts by training a colbert retriever and an extractive reader. then, they were able to quickly deploy the search application using the create-primeqa-app and make it available publicly foot_7 . ## conclusion primeqa is an open-source question answering library designed by researchers and developers to easily facilitate reproduciblity and reusability of existing and future work in qa. this is an important contribution to the community, as it provides researchers and end users with easy access to state-ofthe-art algorithms in the rapidly progressing field of qa. primeqa also provides off-the-shelf models that developers can directly deploy for their custom qa applications. primeqa is built on top of the largest open-source nlp libraries and tools, can incorporate llms through external apis, and provides simple python scripts as entry points for easy reuse of its core components. this straightforward access and high reusability has already garnered significant traction in the community, enabling primeqa to grow organically as an important resource for rapid progress in qa.