How Useful is Continued Pre-Training for Generative Unsupervised Domain Adaptation? (2024)

Abstract: Recent breakthroughs in scale have enabled the emergence of powerful generative language models, and the ability to fine-tune these models on various tasks by casting them into prompts or instructions. In this landscape, the problem of Unsupervised Domain Adaptation (UDA), or the problem of leveraging knowledge from a labeled source domain to an unlabeled target domain, has been left behind, with recent UDA methods still addressing discriminative classification. In particular, two popular UDA approaches, involving Continued Pre-Training (CPT) and learning domain invariant representations, have been under-explored in the generative setting, signaling a gap. In this work, we evaluate the utility of CPT for generative UDA. We first perform an empirical evaluation to measure the trade-offs between CPT and strong methods promoting domain invariance. We further evaluate how well the benefits of CPT extend to different architectures, tuning methods and data regimes. We then motivate the use of CPT by studying to what degree it benefits classification performance on the target domain. Finally, we attempt to understand the mechanism behind which CPT improves classification performance on the unlabeled target domain. Our findings suggest that a implicitly learns the downstream task while predicting masked words informative to that task. Our work connects the body of UDA research with that of instruction tuning, enabling an initial step towards a wider applicability of modern language models.

## introduction recent advancements in the pre-training of language models have enabled the widespread use of powerful generative models, which can be leveraged across multiple domains with no training @xcite @xcite . despite these advancements, these autoregressive models are still fragile under certain kinds of data distribution shifts, making their applications across these domains challenging @xcite @xcite . this is addressed, in part, by the concept of instruction tuning with templates @xcite @xcite @xcite , enabling the learning of new tasks without any randomly initialized parameters. the problem of unsupervised domain adaptation (uda) leverages learned knowledge from a labeled source domain to an unlabeled target domain @xcite @xcite . it is useful for adaptation to unlabeled domains with high labeling costs, where supervised instruction tuning does not suffice. despite the pervasive need for models to generalize to such domains, recent uda methods still address discriminative classification, barring the application of these approaches to recent generative models. in particular, continued pre-training and domain invariance-based methods, two widely popular classes of uda approaches @xcite , are completely unexplored for uda in the generative setting. the continued pre-training (cpt) approach involves extended pre-training on a domain or task, followed by supervised training on the downstream task @xcite . this approach has been widely used for adaptation to labeled domains @xcite @xcite , and in the uda setup for unlabeled domains @xcite @xcite @xcite . invariance-based approaches attempt to learn representations that are invariant across domains @xcite @xcite , with the notion that when the learned representa-tions from both domains cannot be distinguished by a classifier and the classifier performs well on the source domain, it will also exhibit strong performance on the target domain. these two classes of methods introduce a trade-off: invariance-based methods suffer from instability issues @xcite , while continued pre-training requires a larger computational budget. but how would this trade-off extrapolate to the generative setting? for example, invariancebased methods are well motivated in discriminative tasks, where there is a clear decision boundary; however, the same does not hold for generative tasks. to address these gaps, we introduce the setting of generative uda, where an autoregressive model is trained to leverage knowledge from a labeled source domain to an unlabeled target domain, using only next word prediction as its objective. we formalize the use of cpt for this setting in section 2, and then attempt to explore and understand the behaviour of cpt for generative uda. we begin by performing an empirical analysis on 40 real-world domain pairs to explore the tradeoff between continued pre-training and invariance-based approaches, and find vanilla cpt to be competitive with and significantly more stable than a state of the art invariance-based approach (section 3). we then stress test cpt, by applying it across varying model architectures and scales, tuning approaches and data regimes. we find that cpt is robust across these settings, unlike our invariance-based approach (section 4). with recent language models being trained across vast corpora which may include domains similar to the target domain, the requirement for continued pre-training may be raised to question. in section 5, we show that continued pre-training is indeed essential for strong downstream performance on the target domain, and this performance rapidly degrades with limited target domain exposure. finally, we attempt to shed light on how masking plays a role in improving classification accuracy on the unlabeled target domain in section 6. we find that the model may implicitly learn the downstream task as it predicts masked words that are informative to the downstream task. our work attempts to connect the body of uda research with recent trends in language modeling, by providing a set of insights into the behaviour of the popular class of continued pre-training ap-proaches, in the generative uda setting. we hope this enables an initial step towards a wider applicability of modern language models. ## why does word prediction aid classification for generative uda? in this section, we examine why predicting masked words of the source and target domains through cpt boosts sentence classification on the unlabeled target domain for generative uda. we hypothesize that by having to predict masked words that are informative to the downstream task during pre-training, the model implicitly learns information about the downstream task. for example, given the masked sentence, "i really _ the movie, it was a fascinating watch.", the masked word is indicative of the downstream task, in this case sentiment analysis. the model can only predict this masked word (which would be a positive word like "loved" or "enjoyed") by using other words informative to the task ("fascinating"). through this process, the model is essentially learning features which are useful to the downstream task, despite having no direct supervision. to test this hypothesis, we quantize the "informativeness" of each word to a classification task: an informative word is highly correlated with any of the labels in the downstream task. 6 specifically, we follow @xcite and use pointwise mutual information (pmi) (fano, 1961) of the word with respect to the class label: pmi(word, class) = log p(word, class) p(word)p(class) , where we count the frequency of a word-class pair on d src to estimate p(word, class), and similarly count a word and a class individually on d src to estimate p(word) and p(class). we use two sets of words from a dataset: those with the top k% (informative) and bottom k% (uninformative) pmi with any inference label (@xmath0. we also filter out low-frequency words from the selection. 7 we compute these sets for the source and target domains individually, assuming access to target labels. we use the t5v1.1 model on the @xmath1. ## discussion the computational trade-off of cpt our results in section 3.2 show that continued pretraining and methods promoting domain invariance are competitive with each other. continued pre-training suffers from the computational drawback of requiring an additional phase of training. conversely, invariance-based methods are difficult to optimize, possibly requiring more runs to achieve a stable optimum, and having a higher amortized computational cost. inspired by @xcite , we introduce a simple single phase variant of continued pre-training which is equivalent in performance to its two phase variant, nullifying the additional computational overhead of the approach (more details in appendix i). uda in the age of llms recent breakthroughs in scale have showcased that large language models (llms) are highly powerful, and can perform various downstream tasks with limited or no training. this may raise question on the relevance of the uda problem as a whole -does a model even require expensive adaptation to a domain it may have already been exposed to during pretraining? in addition to our analysis in section 5, we argue that the requirement to adapt small models to unseen domains still holds in specific cases. small supervised models have been shown to be comparable with, or even outperform, zero-shot general-purpose llms on various downstream tasks @xcite @xcite , serving as lightweight and customizeable (through fine-tuning) alternatives. safety critical domains like healthcare and finance would benefit more from these models than a generalist llm. our study does not address how to better adapt to domains, rather we investigate ways a model may adapt to data unseen during pre-training. this is a question that holds for current llms, and will continue to hold as long as models are unable to access infinite data during pre-training. ## conclusion we introduce the setting of generative uda, and perform an investigation on the utility of continued pre-training in this setting. we compare the approach with the popular class of domain-invariance based methods for uda, showing that cpt is both competitive with, and more stable than invariancebased approaches. our experiments show that the benefits of cpt extend to different architectures, tuning methods and data regimes. we motivate the need for target domain exposure through cpt by showing that performance on the target domain gradually degrades with increasing masking rate. finally, we shed light on the interplay between masking and classification performance, and how this aids uda. our analysis shows that in predicting masked words that are informative to the downstream task, the model implicitly learns about the downstream task, furthering the benefits of directly learning the task. our work connects the body of uda research with that of instruction tuning, enabling an initial step towards a wider applicability of modern language models.