The Trade-offs of Domain Adaptation for Neural Language Models (2022)

Abstract: This work connects language model adaptation with concepts of machine learning theory. We consider a training setup with a large out-of-domain set and a small in-domain set. We derive how the benefit of training a model on either set depends on the size of the sets and the distance between their underlying distributions. We analyze how out-of-domain pre-training before in-domain fine-tuning achieves better generalization than either solution independently. Finally, we present how adaptation techniques based on data selection, such as importance sampling, intelligent data selection and influence functions, can be presented in a common framework which highlights their similarity and also their subtle differences.

## introduction neural language models (lms) trained on large generic training sets -over a billion sentences @xcite -have been shown to be effective at adapting to smaller, specific target domains for language modeling and other downstream tasks @xcite . neural lm adaptation is commonly performed via fine tuning @xcite @xcite , data selection @xcite or their combination @xcite @xcite . however, the tradeoffs between fine-tuning and reweighting of pretraining data is not well understood and a theoretical framework for reasoning about the generalization performance of these methods is needed. in this paper, we connect language model adaptation with concepts of machine learning theory. our derivations support past empirical observations: it has been observed that the size of the out-of-domain pre-training set is important for in domain generalization @xcite or that domain adaptation is more effective on domains which are well represented in the the pre-training data @xcite . our study consider a training setup with a large out-of-domain set and a small in-domain set. as a first contribution, we derive how the benefit of training a model on either set depends on the size of the sets and the distance between their underlying distribution. we also expose how fine-tuning can be viewed as a regularization method that can achieve a better trade-off than training only on either set. the research on data selection for lm adaption originates mainly from intelligent selection @xcite . this method examines the out-of-domain training data to emphasize a subset deemed more likely by an in-domain model than by an out-ofdomain model. although intuitive, the connection of this method with statistical estimation is unclear, which makes studying its impact on generalization error difficult. another family of selection methods stems from influence functions @xcite which estimate whether the model updates from out-of-domain training examples are aligned with the in-domain updates. this approach is more principled and its impact on the generalization error is easier to study. in this work, as a second contribution, we show how intelligent selection and influence function methods are linked in the case of neural lms. in particular, we show that they both can be derived from importance sampling @xcite , a classical, well-studied statistical estimation technique. the rest of our paper is organized as follows. we first presents the theoretical trade-offs between in-domain and out-of-domain training. we highlight the importance of the relative sizes of indomain and out-of-domain training sets along with the distance between their underlying distributions. we also present how fine-tuning with a limited number of updates can be seen as a training method regularized with respect to the out-ofdomain prior. finally, we present data selection methods under a unifying framework. ## neural language modeling language modeling refers to the generative modeling of natural language @xcite . commonly, natural language is represented as a sequence of symbols, tokens, from a finite vocabulary. for instance, language can be represented as a sequence of characters, a sequence of words or alternative units. a neural language model (lm) decomposes the estimates the log probability of a text @xmath0, . . . y n ), as the parameter vector θ ∈ θ of a neural lm is identified by maximizing the log likelihood over a training set d sampled from the true distribution d using variants of stochastic gradient descent. the log likelihood of a held-out set, sampled from the same distribution, can evaluate model quality. one often reports perplexity, the exponentiated negative average log likelihood per token. conditional lms model the distribution of a text y given a conditioning input x. this type of model is used for translation where (x, y) pairs are sentences in the source and target language @xcite or summarization where (x, y) pairs are corresponding articles and summaries @xcite . for both conditional and regular lms, the size of the training data is important to achieve a low held-out perplexity. this is an obstacle for domains with limited available training data. this issue has led to various model adaptation approaches. these methods leverage large amounts of generic training data d along with a small amount of target domain training data t from the domain of interest. fine tuning is a popular domain adaptation method which trains a neural language model in two phases, first maximizing the likelihood of the generic set d (pre-training) before optimizing the likelihood of the target domain set t (fine-tuning). as an alternative to fine-tuning, some methods consider leveraging the small target-domain training set to identify and emphasize similar data in the larger generic training set. these emphasis methods can be used individually or in conjunction with fine-tuning. emphasis methods include importance sampling, contrastive data selection and influence functions. this paper shows that these methods -although proposed in different context -can be presented in a unified way which allows light to be cast on their subtle differences. ## training strategies this section first examines in-domain training, i.e. when the training and test data are sampled from the same distribution. it then studies out-ofdomain training, i.e. when the training and test data distribution differs. finally, it examines outof-domain pre-training followed by in-domain fine tuning. for the three cases, we decompose the loss relying on classical concepts from learning theory and study the trade-offs involved in each setup. ## data selection data selection aims to improve out-of-domain training by selecting or giving stronger weights to some data points. the identification of these points aims to emphasize out-of-domain examples which have an impact on the model similar to the impact of the in-domain training examples. we study three independently proposed selection methods, importance sampling, contrastive data selection and influence functions. we show that these methods all train models through weighted log-likelihood training, but introduce their weights w(y; t , d) with different justifications. despite these differences, we show that these methods result in surprisingly similar selection weights in the specific case of neural language models. data selection is particularly suited when the out-of-domain training distribution and the test distribution have a large kl divergence but the out-of-domain training set is large. in that case, the generalization of a model trained on out-ofdomain data is poor due to the large kl divergence between t and d, see eq. ( 2 ). when this kl divergence is large but out-of-domain data is abundant, data selection methods propose to select a subset of the out-of-domain data d t ⊂ d. ideally, the training loss over such a subset l(θ, d t ) would be a better proxy for the generalization loss over t , l(θ, t ), than the training loss over the full set d, l(θ, d). selection involves a delicate trade-off though. one one hand, data selection is attractive since it replaces the training set with another set closer to the test domain. on the other hand, this training set is smaller, which increases the impact of estimation errors. additionally, data selection is imperfect since the target domain distribution t is only known through a small target training set t . this section successively presents importance sampling, contrastive data selection and influence functions and connect them into a single framework. ## conclusions this work focuses on domain adaptation for neural language modeling. it compares the generalization properties of a model trained over a large outof-domain corpus as opposed to a model trained over a small in-domain corpus. it shows how finetuning, the most common approach for neural lm adaptation can achieve better trade-offs than either solution. we then focus on adaptation via data selection techniques, i.e. techniques to emphasize in-domain data in an out-of-domain training set. we show that common techniques, contrastive data selection and influence function selection, can both be derived from importance sampling. our analysis currently assumes a pure language modeling setup, i.e. an auto-regressive model trained aiming high log-likelihood, both for outof-domain and in-domain data. in the future, we want to extend our analysis of domain adaptation techniques to the popular setting @xcite where model training combines language modeling over out-of-domain data and a different final task on in-domain data. our theoretical work also raises empirical questions. the binarization of importance sampling weights in intelligent selection is a simple variance reduction technique and more sophisticated alternative might be beneficial empirically. the link between influence functions and importance sampling suggests that examples with importance sampling weights lower than one have only a negative effect on the in-domain likelihood, which is not a typical observation in practice. this contradiction suggests expanding influence scores to take into account effects beyond a single update.