M ulti R e QA : A Cross-Domain Evaluation for R etrieval Question Answering Models (2021)

Abstract: Retrieval question answering (ReQA) is the task of retrieving a sentence-level answer to a question from an open corpus (Ahmad et al.,2019).This dataset paper presents MultiReQA, a new multi-domain ReQA evaluation suite composed of eight retrieval QA tasks drawn from publicly available QA datasets. We explore systematic retrieval based evaluation and transfer learning across domains over these datasets using a number of strong base-lines including two supervised neural models, based on fine-tuning BERT and USE-QA models respectively, as well as a surprisingly effective information retrieval baseline, BM25. Five of these tasks contain both training and test data, while three contain test data only. Performing cross training on the five tasks with training data shows that while a general model covering all domains is achievable, the best performance is often obtained by training exclusively on in-domain data.

## introduction retrieval-based question answering (qa) investigates the problem of finding answers to questions from an open corpus @xcite @xcite @xcite @xcite . there is a growing interest in building scalable end-to-end question answering systems for large scale retrieval @xcite . retrieval question answering (reqa) @xcite , illustrated in recent research has shown promising results on developing neural models for retrieval tasks including reqa, ms marco, and the retrieval part of open domain question qa @xcite @xcite . one challenge of employing neural models is that it usually requires a large amount of training data. while it is possible to get such data from a general domain, it may hard to get similar data for specialized domains, which is a common span @xcite 3 https://books.google.com/talktobooks/ dataset question answer searchqa at age 33 in 1804, he started a new symphony, his 5th, with a da-da-da-duhg this is the first movement of beethoven's 5th symphony. triviaqa from the greek for color, what element, with an atomic number of 24, uses the symbol cr? rubies and emeralds also owe their colors to chromium compounds. hotpotqa lenny young is a collaborator on the stop motion film released in what year? chicken run is a 2000 stop-motion animated comedy film produced by the british studio aardman animations. ## multi-domain reqa (multireqa) the multi-domain reqa (multireqa) test suite is composed of select datasets drawn from the mrqa shared task @xcite . 5 we follow the training, in-domain test, out-of-domain test splits defined in mrqa. the individual datasets are described below: searchqa jeopardy question-answer pairs augmented with text snippets retrieved by google @xcite . triviaqa trivia enthusiasts authored questionanswer pairs. answers are drawn from wikipedia and bing web search results, excluding trivia websites @xcite . hotpotqa wikipedia question-answer pairs. this dataset differs from the others in that the questions require reasoning over multiple supporting documents @xcite . the annotators generate the questions knowing the answers and the supporting contexts. ## baseline models to establish strong baselines for the multireqa test suite, we use two neural models, based on bert @xcite and use-qa @xcite , respectively, as well as an well established term-based information retrieval baseline, bm25. ## analysis 6.1 does context help? candidate answers may be not fully interpretable when taken out of their surrounding context @xcite . in this section we investigate how model performance changes when removing context. we experiment with one bm25 model and one neural model, by picking the best performing models from previous experiments: bm25 wpm and use-qa finetune . recall, use-qa finetune models are fine-tuned on each individual dataset. figure 1 illustrates the change in performance when models are restricted to evaluating candidate answers without context. 20 for the use-qa model, the performance drop by excluding answer context is less than 5% on all datasets. the drop in bm25 performance is larger, supporting the hypothesis that bm25's token overlap heuristic is effective over large spans of text, while the neural model obtains a "deeper" semantic understanding and thus extracts more signal out of a single sentence. 20 we report p@1 here, but observed similar trends in mrr. ## related work open domain qa involves finding answers to questions within large document collections @xcite . the ground-truth answer for many evaluations is a span often containing a word or a short phrase (i.a., kwiatkowski et al. (2019 bm25wpm: [...] introduced tilapia are the main fish that can tolerate the high salinity levels and pollution . other freshwater fish species live in the rivers and canals that feed the salton sea , including threadfin shad , carp, red shiner , channel catfish , white catfish , largemouth bass , mosquitofish , sailfin molly , and the vulnerable desert pupfish . [...] example 2 (from triviaqa): what was invented in the 1940s by percy spencer, an american self-taught engineer from howland, maine, who was building magnetrons for radar sets? correct answer: [...] after experimenting, he realized that microwaves would cook foods quickly -even faster than conventional ovens that cook with heat. the raytheon corporation produced the first commercial microwave oven in 1954; it was called the 1161 radarange. it was large, expensive, and had a power of 1600 watts. [...] use-qa finetune : [...] because of his accomplishments, spencer was awarded the distinguished service medal by the u.s. navy and has a building named after him at raytheon. percy spencer, while working for the raytheon company, discovered a more efficient way to manufacture magnetrons. in 1941, magnetrons were being produced at a rate of 17 per day. [...] bm25wpm: [...] by the end of 1971, the price of countertop units began to decrease and their capabilities were expanded. spencer, born in howland, maine, was orphaned at a young age. although he never graduated from grammar school, he became senior vice president and a member of the board of directors at raytheon, receiving 150 patents during his career [...] ## conclusion in this paper, we convert eight existing qa tasks from the mrqa shared task @xcite ) into sentence-level retrieval tasks, by treating the sentence containing the ground-truth span as the target sentence-level answer. in additional to a new evaluation suite for sentence level retrieval, we provide strong baselines using unsupervised term-based information retrieval methods (bm25), and three neural models, off-the-self use-qa, finetuned use-qa, and bert dual encoders. overall, bm25's classical term-based retrieval approach is a surprisingly strong baseline, and one that could likely be improved further using additional information retrieval techniques such as normalization and synonym matching. the neural models, however, can be trained end-to-end without feature engineering, and perform particularly well on tasks with a low degree of question/answer token overlap, or in situations where context is limited. the neural model performance can also be improved through the addition of in-domain training data. however, we find that qa tasks are not all alike and having training data in the precise target domain is important.