Single-dataset Experts for Multi-dataset Question Answering (2021)

Abstract: Many datasets have been created for training reading comprehension models, and a natural question is whether we can combine them to build models that (1) perform better on all of the training datasets and (2) generalize and transfer better to new datasets. Prior work has addressed this goal by training one network simultaneously on multiple datasets, which works well on average but is prone to over- or under-fitting different sub- distributions and might transfer worse compared to source models with more overlap with the target dataset. Our approach is to model multi-dataset question answering with an ensemble of single-dataset experts, by training a collection of lightweight, dataset-specific adapter modules (Houlsby et al., 2019) that share an underlying Transformer model. We find that these Multi-Adapter Dataset Experts (MADE) outperform all our baselines in terms of in-distribution accuracy, and simple methods based on parameter-averaging lead to better zero-shot generalization and few-shot transfer performance, offering a strong and versatile starting point for building new reading comprehension systems.

## introduction the goal of reading comprehension is to create computer programs that can answer questions based on a single passage of text. many reading comprehension datasets have been introduced over the years, and prior work has explored ways of training one network on multiple datasets to get a model that generalizes better to new distributions @xcite @xcite . our goal is to build a multidataset model that performs well on the training distributions and can also serve as a strong starting 1 our code and models are available at https:// github.com/princeton-nlp/made . figure 1 : made consists of a set of dataset-specific adapters and classifiers {(φ t , ψ t )} |s| @xmath0. they are optimized jointly on a set of training datasets (left). to transfer to a new dataset (right), we either average the parameters of the adapters, or fine-tune the ensemble on the target dataset and take the weighted average at the end (section 3). our idea is to combine the benefits of singleand multi-dataset by training a collection of singledataset experts that share an underlying transformer model (figure 1 ). this system is based on adapters @xcite , lightweight task-specific modules interleaved between the layers of a pre-trained transformer (e.g., bert; @xcite . the standard use of adapters is as a parameter-efficient alternative to fine-tuning: task-specific adapters are trained separately on top of a frozen transformer, which means the adapters cannot directly learn cross-task regularities. we instead first train a shared transformer in a multi-adapter setup before refining adapters for individual datasets, which we call multi-adapter dataset experts (made). our intuition is that the shared parameters encode regularities across different reading comprehension tasks while the adapters model the sub-distributions, resulting in more accurate and robust specialized models that transfer better to a variety of target datasets. we apply this approach to a range of extractive question answering datasets from the mrqa 2019 shared task @xcite , training made on six in-domain datasets and evaluating generalization and few-shot transfer learning to six outof-domain datasets. the resulting system outperforms single-and multi-dataset models in terms of in-domain accuracy, and we find that a simple approach to transfer learning works well: averaging the parameters of the made adapters results in a single model that gets better zero-shot generalization and few-shot transfer performance compared to both baselines as well as a state-of-theart multi-dataset qa model, unifiedqa @xcite . our experiments illustrate the benefits of modeling both cross-dataset regularities and dataset-specific attributes, and the trained models offer a strong and versatile starting point for new question-answering models. ## related work prior work has addressed multi-dataset reading comprehension by fine-tuning a pre-trained transformer language model @xcite simultaneously on examples from multiple datasets @xcite . several works explore different multi-task sampling schedules, as a way of mitigating training set imbalances @xcite . another line of work focuses on training models to answer a wider variety of question types, including unifiedqa @xcite , a t5 model @xcite trained on datasets with different answer formats, such as yes/no and multiple-choice, using a unified text-to-text format. adapters @xcite are task-specific modules interleaved between the layers of a shared transformer. stickland and @xcite trained task adapters and the transformer parameters jointly for the glue benchmark @xcite but achieved mixed results, improving on small datasets but degrading on larger ones. subsequent work has used a frozen, pre-trained transformer and trained task adapters separately. researchers have explored different methods for achieving transfer learning in this setting, such as learning to interpolate the activations of pre-trained adapters @xcite . ## conclusion made combines the benefits of single-and multidataset training, resulting in better in-domain accuracy and transfer performance than either multidataset models or ensembles of single-dataset models, especially in low resource settings. for future work we plan to explore ensembling methods for better zero-shot prediction and interpolating made weights for better transfer learning.