Choose Your QA Model Wisely: A Systematic Study of Generative and Extractive Readers for Question Answering (2022)

Abstract: While both extractive and generative readers have been successfully applied to the Question Answering (QA) task, little attention has been paid toward the systematic comparison of them. Characterizing the strengths and weaknesses of the two readers is crucial not only for making a more informed reader selection in practice but also for developing a deeper understanding to foster further research on improving readers in a principled manner. Motivated by this goal, we make the first attempt to systematically study the comparison of extractive and generative readers for question answering. To be aligned with the state-of-the-art, we explore nine transformer-based large pre-trained language models (PrLMs) as backbone architectures. Furthermore, we organize our findings under two main categories: (1) keeping the architecture invariant, and (2) varying the underlying PrLMs. Among several interesting findings, it is important to highlight that (1) the generative readers perform better in long context QA, (2) the extractive readers perform better in short context while also showing better out-of-domain generalization, and (3) the encoder of encoder-decoder PrLMs (e.g., T5) turns out to be a strong extractive reader and outperforms the standard choice of encoder-only PrLMs (e.g., RoBERTa). We also study the effect of multi-task learning on the two types of readers varying the underlying PrLMs and perform qualitative and quantitative diagnosis to provide further insights into future directions in modeling better readers.

## introduction question answering (qa) is an important task to evaluate the reading comprehension capacity of an intelligent system and can be directly applied to real applications such as search engines @xcite and dialogue systems @xcite . this * work done during internship at salesforce research. both the state-of-the-art extractive and generative readers are based on large pretrained language models (prlms) and show good performance on different datasets. however, a systematic comparison between them has been largely unexplored. such a comparison reveals the strengths and weaknesses of each reader, which in turn can provide more principled guidance on which reader and prlm should be applied in which cases, and also open up future research opportunities grounded on identified concrete challenges to improve reader models. however fair comparisons between these have been difficult to perform mainly because 1) the prlms for extractive and generative are different, i.e., extractive readers are usually built on top of encoder-only prlm while generative ones are based on encoder-decoder prlms, and 2) the size of generative and extractive readers are not the same, which can greatly affect the performance. we design two main set of controlled experiments to address such challenges in comparing extractive and generative readers in a principled manner. in the first set of experiments, we compare extractive and generative readers using the same prlms. specifically, t5 @xcite generative reader is compared with t5 extractive reader and similarly for bart @xcite . this allows a fair comparison of different answer prediction approaches without being affected by different architecture or prior knowledge of prlms. more-over, we challenge the conventional formulation of extractive readers, which are often built upon encoder-only prlms, by leveraging the encoder of encoder-decoder prlms as a variable alternative. more concretely, we use the encoders of t5 and bart models to explore their capacity as an extractive reader to better understand the effect of different pre-training strategies on the final qa performance. while the aforementioned comparison strategy adopts the same prlms, it remains unclear how generative readers compare with the conventional extractive readers that are built upon encoder-only prlms. thus, in the second experiment, we compare different architecture prlms, including t5, bart, electra @xcite and roberta @xcite , to draw more generalizable and grounded conclusions. all models in this suite of experiments have similar sizes, thus reducing the impact of model size on performance. with these two experiments, we present a systematic comparison of extractive and generative readers using nine readers on the mrqa task @xcite , a collection of multiple extractive qa datasets. this evaluation results in five insightful findings: our work presents an in-depth study of extractive and generative readers for qa task, an important nlp task toward building intelligent systems. our findings shed light on key considerations behind reader selection and would be helpful for formulating future research on advancing reader models. ## model we mainly study the qa models based on prlms with extractive and generative approaches. ## results and analysis we first present the study of using different inference length for each model since it guides us to choose the best performance of each model. then, we compare the generative and extractive readers using the same prlms and the different prlms. last, we present a detail analysis to diagnose the difference among extractive and generative reader. f1 is used to measure performance. note that since we test each model on 12 datasets, the observation and conclusion we draw are mostly based on the average across all datasets. ## conclusion and future work we systematically compare the extractive and generative readers for qa tasks. two sets of experiments are designed to control the effects of different prlms and the size of models. by conducting experiments on 12 qa datasets, our findings provide guidelines on how to choose extractive or generative readers given their strength and weakness. while current work investigates the pros and cons of extractive and generative models systematically, there are some hyperparameters that might affect the model performance. for example, it is known that different prompts in the input effect generative model performance @xcite . also, it is worth studying the ood performance of models deeply. @xcite compares multiple ways to improve the ood performance of an extractive model on qa task, and how these methods affect generative models have not been well-studied yet. meanwhile, most of the work including this work evaluate ood performance by averaging the performance across multiple dataset, but as mentioned in @xcite , the evaluation should be more carefully designed. also, diagnosing the performance on each ood dataset can provide more insights. for example, why models perform better on bioasq dataset than most other datasets (see