Not to Overfit or Underfit the Source Domains? An Empirical Study of Domain Generalization in Question Answering (2022)

Abstract: Machine learning models are prone to overfitting their training (source) domains, which is commonly believed to be the reason why they falter in novel target domains. Here we examine the contrasting view that multi-source domain generalization (DG) is first and foremost a problem of mitigating source domain underfitting: models not adequately learning the signal already present in their multi-domain training data. Experiments on a reading comprehension DG benchmark show that as a model learns its source domains better—using familiar methods such as knowledge distillation (KD) from a bigger model—its zero-shot out-of-domain utility improves at an even faster pace. Improved source domain learning also demonstrates superior out-of-domain generalization over three popular existing DG approaches that aim to limit overfitting. Our implementation of KD-based domain generalization is available via PrimeQA at: https://ibm.biz/domain-generalization-with-kd.

## introduction domain generalization (dg) seeks to train models on a small number of source domains in a way that maximizes their zero-shot out-of-domain (ood) utility @xcite . many existing dg methods are rooted in the premise that weak generalization under domain shift occurs due to "source domain overfitting": the learning of spurious source domain correlations, i.e., noise, that are unrelated to the core learning task, and are therefore unlikely to be present in novel target domains. the popular domain-invariant learning (dil) paradigm, for example, proposes to limit overfitting by imposing some form of cross-domain regularization on the empirical risk minimization (erm) process over labeled multi-domain training data @xcite . we argue in this paper that this emphasis on not fitting the noise minimizes the importance of actually learning the signal present in the source domains, i.e., not underfitting them, for dg. underfitting occurs when a trained model, due to inadequacies in its capacity or the training procedure, fails to learn training set patterns that are truly representative of the task. in a recent study of dg in computer vision, @xcite find that large models with sufficiently high capacity can demonstrate strong ood generalization even with ordinary erm training-without the need for dil-when properly configured. here we ask a very similar question, but (a) in the nlp context of question answering (qa), and (b) for relatively small models that are more prone to underfitting, e.g., a bert-base qa model @xcite . concretely, we investigate if enabling smaller qa models to learn their source domains better also improves their generalization to new domains and how the results compare to dil. this formulation essentially views ood generalization in low-capacity models as a natural extension of the more familiar and arguably simpler problem of in-domain generalization in a multidomain setting. a key advantage of such an approach is that familiar supervised learning methods such as knowledge distillation (kd) @xcite )-which was specifically designed for the very purpose of training small yet high-accuracy models-can now be leveraged for dg. kd generally provides stronger source domain supervision than erm by minimizing a surrogate risk, which utilizes the soft predictions of a larger teacher model (e.g., a bert-large qa model) as the learning targets for the smaller model being trained, now called the student. additionally, synthesized input in large quantities has been found to further enhance the performance of kd @xcite . here we extend the application of these methods to dg for qa. we evaluate our methods on a multi-dataset reading comprehension benchmark @xcite and compare their accuracy with three popular dil approaches: domain adversarial training @xcite , episodic training @xcite -for which we propose a novel variant suitable for deep transformer models-and meta learning @xcite . we also design experiments to answer more targeted questions such as: (1) are the improvements more prominent on in-domain validation data than on ood test instances, which could be indicative of weak generalization? (2) do the proposed methods falter on input cases where domain-invariant approaches thrive, potentially indicating weakness on extremely distant test cases? in all these different evaluations, our methods exhibit far superior dg than the three existing methods, whereas the latter only marginally outperform erm. while more experiments on additional tasks, datasets and baselines are needed before a firm conclusion can be reached on the superiority of the proposed formulation of dg over dil (or vice versa), our findings do point to a need for a better understanding of optimal source domain learning as an approach to dg. a primary goal of this paper is to motivate future explorations of this important research question. ## methodology this section describes the proposed kd-based dg approaches as we apply them to the problem of reading comprehension. ## experiments here we describe the baseline dil approaches our proposed methods are evaluated against, our experimental setup and results. ## conclusion this paper puts forward the view with empirical evidence for question answering (qa) that contrary to popular belief, multi-source domain generalization (dg) is better modeled as a problem of addressing model underfitting than overfitting. our experimental results show that by simply learning the training domains well via knowledge distillation, even when the number of such domains is relatively small, strong out-of-domain (ood) generalization can be achieved in qa without the need for any cross-domain regularization. these findings point to the need for a re-examination of whether not fitting the noise in training data is indeed a more reasonable path forward for dg than actually learning the signal in it; we hope that our work will inspire future efforts to answer this important question in greater detail with a wider range of tasks and approaches.