SF - QA : Simple and Fair Evaluation Library for Open-domain Question Answering (2021)

Abstract: Although open-domain question answering (QA) draws great attention in recent years, it requires large amounts of resources for building the full system and it is often difficult to reproduce previous results due to complex configurations. In this paper, we introduce SF-QA: simple and fair evaluation framework for open-domain QA. SF-QA framework modularizes the pipeline open-domain QA system, which makes the task itself easily accessible and reproducible to research groups without enough computing resources. The proposed evaluation framework is publicly available and anyone can contribute to the code and evaluations.

## introduction open-domain question answering (qa) is the task of answering open-ended questions by utilizing knowledge from a large body of unstructured texts, such as wikipedia, world-wide-web and etc. this task is challenging because researchers have to face issues in both scalability and accuracy. in the last few years, rapid progress has been made and the performance of open-domain qa systems has been improved significantly @xcite @xcite . several different approaches were proposed, including twostage ranker-reader systems @xcite , end-to-end models @xcite and retrievalfree models @xcite . despite people's increasing interest in open-domain qa research, there are still two main limitations in current opendomain qa research communities that makes research in this area not easily accessible: the first issue is the high cost of ranking large knowledge sources. most of the prior research used wikipedia dumps as the knowledge source. for example, the english wikipedia has more than 7 million articles and 100 million sentences. for many researchers, indexing data of this size with a classic search engine (e.g., apache lucene @xcite ) is feasible but becomes impractical when indexing with a neural ranker that requires weeks to index with gpu acceleration and consumes very large memory space for vector search. therefore, research that innovates in ranking mostly originates from the industry. the second issue is about reproducibility. opendomain qa datasets are collected at different time, making it depends on different versions of wikipedia as the correct knowledge source. for example, squad @xcite uses the 2016 wikipedia dump, and natural question @xcite uses 2018 wikipedia dump. our experiments found that a system's performance can vary greatly when using the wrong version of wikipedia. moreover, indexing the entire wikipedia with neural methods is expensive, so it is hard for researchers to utilize others' new rankers in their future research. lastly, the performance of an open-domain qa system depends on many hyperparameters, e.g. the number of passages passed to the reader, fusion strategy, etc., which is another confounding factor to reproduce a system's results. thus, this work proposes sf-qa (simple and fair question-answering), a python library to solve the above challenges for two-stage qa systems. the key idea of sf-qa is to provide preindexed large knowledge sources as public apis or cached ranking results; a hub of reader models; and a configuration file that can be used to precisely reproduce an open-domain qa system for a task. the pre-indexed knowledge sources enable researchers to build on top of the previously proposed rankers without worrying about the tedious work needed to index the entire wikipedia. then the executable configuration file provides a complete snapshot that captures all of the hyperparameters in order to reproduce a result. experiments are conducted to validate the effectiveness of sf-qa. we show that one can easily reproduce previous state-of-the-art open-domain qa results on four qa datasets, namely open squad, open natural questions, open cmrc, and open drcd. more datasets will be included in the future. also, we illustrate several use cases of sf-qa, such as efficient reader comparison, reproducible research, open-source community, and knowledgeempowered applications. sf-qa is also completely open-sourced foot_0 and encourages the research community to contribute their rankers or readers into the repository, so that their methods can be used by the rest of the community. in short, the contributions of this paper include: ## related work existing deep learning open-domain qa approaches can be broadly divided into three categories. ## conclusion in conclusion, this paper presents sf-qa, a novel evaluation framework to make open-domain qa research simple and fair. this framework fixes the gap among researchers from different fields, and make the open-domain qa more accessible. we show the robustness of this framework by successfully reproducing several existing models in opendomain qa research. we hope that sf-qa can make the open-domain qa research more accessible and make the evaluation easier. we expect to further improve our framework by including more models in both ranker and reader side, and encourage community contributions to the project as well.