UNIFIEDQA : Crossing Format Boundaries with a Single QA System (2020)

Abstract: Question answering (QA) tasks have been posed using a variety of formats, such as extractive span selection, multiple choice, etc. This has led to format-specialized models, and even to an implicit division in the QA community. We argue that such boundaries are artificial and perhaps unnecessary, given the reasoning abilities we seek to teach are not governed by the format. As evidence, we use the latest advances in language modeling to build a single pre-trained QA model, UNIFIEDQA, that performs well across 19 QA datasets spanning 4 diverse formats. UNIFIEDQA performs on par with 8 different models that were trained on individual datasets themselves. Even when faced with 12 unseen datasets of observed formats, UNIFIEDQA performs surprisingly well, showing strong generalization from its outof-format training data. Finally, simply finetuning this pre trained QA model into specialized models results in a new state of the art on 10 factoid and commonsense question answering datasets, establishing UNIFIEDQA as a strong starting point for building QA systems.

## introduction question answering is a common tool for assessing how well can computers understand language and reason with it. to this end, the nlp community has introduced several distinct datasets, with four popular qa formats illustrated in fig. 1 . for instance, some datasets expect the answer to be "yes" or "no", or a unique answer span in the associated paragraph (as opposed to multiple or no spans). these differences have motivated their study in silos, often encoding qa format into the model architecture itself. efforts to exploit multiple datasets remain largely restricted to a single format. for example, @xcite limit consideration to figure 1 : four formats (color-coded throughout the paper) commonly used for posing questions and answering them: extractive (ex) , abstractive (ab) , multiple-choice (mc) , and yes/no (yn) . sample dataset names are shown in square brackets. we study generalization and transfer across these formats. multiple-choice datasets, while @xcite focus their generalization study on extractive span prediction models. to the best of our knowledge, no single qa system targets, not to mention excels at, all of these formats. this raises the question: can qa models learn linguistic reasoning abilities that generalize across formats? our intuition is simple: while question format and relevant knowledge may vary across qa datasets, the underlying linguistic understanding and reasoning abilities are largely common. a multiple-choice model may, therefore, benefit from training on an extractive answers dataset. building upon this intuition, we present a single pre-trained qa system, named unifiedqa, that exploits information across 4 different qa formats to achieve strong performance across 20 different factoid and datasets squad11 squad2 newsqa quoref ropes narqa drop natqa race mctest obqa arc qasc cqa wg piqa siqa boolq np-boolq multirc format extractive qa (ex) abstractive qa (ab) multiple-choice qa (mc) yes/no qa (yn) has paragraphs? ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ has explicit candidate ans? ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ # of explicit candidates 4 4 4 4 8 5 2 2 3 para contains ans as substring? ✓ ✓ ✓ ✓ has idk questions? ✓ figure 2: properties of various qa datasets included in this study: 5 extractive (ex), 3 abstractive (ab), 9 multiplechoice (mc), and 3 yes/no (yn). 'idk' denotes 'i don't know' or unanswerable questions. boolq represents both the original dataset and its contrast-sets extension boolq-cs; similarly for ropes, quoref, and drop. in this work, we advocate for a unifying view of qa formats by building a format-agnostic qa system. our work leverages recent progress in text-to-text pre-trained neural models, specifically t5 @xcite and @xcite , but with a strong focus on differing qa formats. this paradigm allows unifying many nlp models, which formerly had task-specific designs, into a single text-to-text framework. previous work uses textual prefixes to explicitly define the task associated with each input instance @xcite ; often such attempts to build a single model for multiple nlp tasks underperform the standard pre-training plus finetuning setup (a model per task) @xcite . our work narrows down the scope to tasks that stay within the boundaries of qa, demonstrating that a unified text-to-text paradigm can, in fact, be successful across different qa tasks and formats. we develop a single pre-trained qa model by training text-to-text models on a set of seed qa datasets of multiple formats, taking natural text as input, without using format-specific prefixes. our experiments show that unifiedqa can be applied as-is to different qa tasks, generalizes well to other unseen datasets (zero-shot), and with further finetuning achieves state-of-the-art results on many qa tasks including commonsense and factual datasets. contributions. this work advocates for a unified view of different qa formats, and for building format-agnostic qa systems. to support this view, we present unifiedqa, a single pre-trained qa system that works well on and generalizes to datasets with different formats ( §6.2), while performing on par with state-of-the-art dedicated systems tailored to each dataset ( §6.1). additionally, fine-tuning unifiedqa into specialized systems sets a new state of the art for 10 datasets ( §6.3), establishing it as a powerful starting point for qa research. our findings demonstrate that crossing qa format boundaries is not only qualitatively de-sirable but also quantitatively beneficial. ## related work several qa efforts have studied generalization across datasets of a single format. for instance, in multiqa, @xcite study generalization and transfer, but only across extractive span selection datasets. further, while they show strong leave-one-out style results, they find a single system performs substantially worse than one tuned to each dataset. in orb, @xcite propose a multi-dataset evaluation benchmark spanning extractive and abstractive formats. however, that study is limited to an evaluation of systems, falling short of addressing how to build such generalized models. the mrqa shared task @xcite focuses on span-prediction datasets. unlike all these efforts, our goal is to investigate transfer and generalization across different qa formats, as well as to build a single system that does this well. exploiting commonality across machine learning tasks has a rich history studied under transfer learning @xcite . @xcite and @xcite study transfer among various nlp tasks by casting them into a single qa format-an elegant transfer learning approach but orthogonal to the goal of this work. as noted earlier, @xcite investigate the transfer between several diverse nlp tasks (machine translation, summarization, etc). their key contribution is a text-to-text framework, and a powerful model called t5, that makes it easier to mix multiple tasks by encoding both inputs and outputs as text. they rely on textual prefixes to explicitly define the task corresponding to each input instance. while we build upon their framework, we narrow our focus to variations of qa. this allows us to achieve strong results while avoiding reliance on any format-specific prefixes. our models learn to infer the format of each input question based on its content (e.g., whether the phrasing of the question demands a yes/no answer). moreover, we are able to demonstrate generalization across qa tasks, which prior work failed to achieve presumably due to its focus on too broad a set of nlp tasks. ## unifiedqa: multi-format training suppose we would like to train a unified qa model that can operate over k formats f 1 , f 2 , . . . , f k . for each format f i , suppose we have ℓ i datasets sets d i 1 , d i 2 , . . . , d i ℓ i where d i @xmath0, e i j ) includes both training and evaluation examples. in some cases, the training set t i j may be empty or we may want to ignore it in order to treat d i j as an 'unseen', evaluation-only dataset and assess a model's generalization to it. we use the text-to-text paradigm to convert each training question q in format f i into a plain-text input representation enc i (q). this conversion uses a natural encoding process that will be described shortly ( §3.1) for four common qa formats, and is easily extensible to other formats as well. we follow a simple approach of creating a mixed training pool consisting of all available training instances: training batches are drawn from this pooled data, t , by including each @xmath1. each batch thus, on average, contains the same number of instances from each training set, regardless of its size. similar treatments of task mixing have also been adopted by @xcite and @xcite . as our experiments will show, our multi-format mixing approach works well. it clearly highlights the value of training on out-of-format data and confirms our intuition that there are strong ties across qa formats in terms of the underlying reasoning abilities. 2 our unified question-answering system is based on the recent text-to-text frameworks, particularly, t5 @xcite and @xcite . we first define a unifying encoding of the instances across various formats ( §3.1). we then introduce unifiedqa ( §3.2) that is a qa system trained on datasets in multiple formats, indicating new state-of-the-art results on 10 datasets and generalization to unseen datasets. ## experimental results we now discuss our main experimental results, evaluating unifiedqa on seed datasets (used for training the system) as well as unseen datasets. ## discussion the key motivation for this work is the observation that nearly all prior efforts on qa research were limited to the boundaries defined by narrow formats. a format-specific design would not generalize across qa datasets with slightly different definitions (e.g., a model built for squad would not work for race). additionally, such a design would prevent us from benefiting from the labeled data available in other formats. we challenge this view by advocating for approaches that combine seemingly different datasets. we believe that developing qa systems targeted to a specific format is a conceptual barrier for progress in the field. ## conclusion the question-answering community has fruitfully explored the design of strong models, but while staying within the boundaries of individual qa formats. we argued that such boundaries are artificial and can even limit the performance of systems, because the desired reasoning abilities being taught and probed are not tied to specific formats. training data in one format should, in principle, help qa systems perform better even on questions in another format. with this intuition in mind, we presented uni-fiedqa, a single pre-trained qa system based on the text-to-text paradigm, seeking to bring unification across four common qa formats. we showed that even with its simple multi-format training methodology, unifiedqa achieves performance on par with 8 dataset-specific expert models ( §6.1), while also generalizing well to many unseen datasets of seen formats ( §6.2). at the same time, we demonstrated that unifiedqa is a strong starting point for building qa systems: it can achieve state-of-the-art performance by simply fine-tuning on target datasets (6.3). we hope this effort will inspire a future line of work in the qa and nlp communities, moving towards more general and broader system designs. we leave extensions of unifiedqa to other formats such as to direct-answer questions @xcite as a promising avenue for future work.