Open-Domain Question Answering with Pre-Constructed Question Spaces (2021)

Abstract: Open-domain question answering aims at locating the answers to user-generated questions in massive collections of documents. Retriever-readers and knowledge graph approaches are two big families of solutions to this task. A retriever-reader first applies information retrieval techniques to locate a few passages that are likely to be relevant, and then feeds the retrieved text to a neural network reader to extract the answer. Alternatively, knowledge graphs can be constructed and queried to answer users’ questions. We propose an algorithm with a novel reader-retriever design that differs from both families. Our reader-retriever first uses an offline reader to read the corpus and generate collections of all answerable questions associated with their answers, and then uses an online retriever to respond to user queries by searching the pre-constructed question spaces for answers that are most likely to be asked in the given way. We further combine one retriever-reader and two reader-retrievers into a hybrid model called R6 for the best performance. Experiments with two large-scale public datasets show that R6 achieves state-of-the-art accuracy.

## introduction open-domain question answering, abbreviated as openqa in this paper, aims at enabling computers to answer user-submitted natural language questions based on a large collection of documents (a.k.a. a corpus). there are two big families of state-of-the-art openqa algorithms. one family, namely retriever-readers (fig. 1 , left branch), first retrieves from the corpus some documents or paragraphs that are likely to be relevant to the question, and then uses neural networks to read the retrieved passages and locate the answer. another line of work, namely question answering using knowledge bases (abbreviated as qa using kb in this paper; fig. 1 , middle branch), first constructs a knowledge base (kb) from the corpus, then queries the kb with the given question. either family of algorithms has some pros and cons: all retriever-readers face a trade-off between efficiency and accuracy; qa using kb methods are good at answering simple factoid questions within the kb schema but weak at complex or out-of-schema questions. we propose a novel reader-retriever design for openqa (fig. 1 , right branch). first, we use deep neural networks to read the corpus offline, detect named entities, generate questions, and aggregate the results into two collections of questions that are answerable with the corpus. we use question spaces to term the two collections. when users submit queries online, a retriever compares user queries with the pre-constructed question spaces to retrieve the answers that are most likely to be asked in the given way. we combine two reader-retrievers (one for each question space) and one retriever-reader into a hybrid model called r 6 to predict the most likely answer based on the consistency among the three sub-models. experiments with large-scale public datasets show that the pre-constructed question spaces boost the performance for openqa, and r 6 performs better than state-of-the-art methods by a large margin. the source code of r 6 is publicly available at https: //github.com/jinfengxiao/r6 . ## experiments we evaluate the openqa performance of our proposed method r 6 and baseline methods using two public qa datasets, squad @xcite and triviaqa @xcite . we adopt a rather challenging setting that all trainable components of the models are trained on squad, while the final models are tested on triviaqa. furthermore, we use triviaqa in an open-domain setting by removing all annotated associations between questions and documents and enforcing the systems to answer every question with the entire corpus. we write triviaqa-open to distinguish such an opendomain setting from those officially adopted by triviaqa. one may wonder why we choose to use different datasets for training and testing. because our goal of the experiments is to compare the effectiveness of our proposed methods to others, as long as all the methods are evaluated fairly under the same setting, we can achieve the goal. such experimental settings are also used by the authors of drqa @xcite . in addition, using squad for training enables us to utilize pre-trained models and author-suggested hyper-parameters to the greatest extent, so that we can make sure we correctly reproduce others' work and do not put their models into disadvantages when comparing them with ours. more experimental details are available in section 4.2. although not critical to this study, using different datasets for training and testing has one additional benefit that it shows the ability of the systems to adapt to new corpora. ## conclusion we propose r 6 , a novel algorithm that constructs question spaces from corpora and uses them to improve openqa. r 6 consists of two novel readerretriever modules and one classic retriever-reader. experiments on public datasets show that r 6 outperforms state-of-the-art retriever-readers by a large margin. our method has the potential to get further improved if solutions can be proposed in future work to better handle questions about less typical answer types or generate questions with higher quality.