UDALM : Unsupervised Domain Adaptation through Language Modeling (2021)

Abstract: In this work we explore Unsupervised Domain Adaptation (UDA) of pretrained language models for downstream tasks. We introduce UDALM, a fine-tuning procedure, using a mixed classification and Masked Language Model loss, that can adapt to the target domain distribution in a robust and sample efficient manner. Our experiments show that performance of models trained with the mixed loss scales with the amount of available target data and the mixed loss can be effectively used as a stopping criterion during UDA training. Furthermore, we discuss the relationship between A-distance and the target error and explore some limitations of the Domain Adversarial Training approach. Our method is evaluated on twelve domain pairs of the Amazon Reviews Sentiment dataset, yielding 91.74% accuracy, which is an 1.11% absolute improvement over the state-of-the-art.

## introduction deep architectures have achieved state-of-the-art results in a variety of machine learning tasks. however, real world deployments of machine learning systems often operate under domain shift, which leads to performance degradation. this introduces the need for adaptation techniques, where a model is trained with data from a specific domain, and then can be optimized for use in new settings. efficient techniques for model re-usability can lead to faster and cheaper development of machine learning applications and facilitate their wider adoption. especially techniques for unsupervised domain adaptation (uda) can have high real world impact, because they do not rely on expensive and time-consuming annotation processes to collect labeled data for domain-specific supervised training, further streamlining the process. uda approaches in the literature can be grouped in three major categories, namely pseudo-labeling techniques (e.g. @xcite , domain adversarial training (e.g. @xcite and pivot-based approaches (e.g. @xcite . pseudolabeling approaches use a model trained on the source labeled data to produce pseudo-labels for unlabeled target data and then train a model for the target domain in a supervised manner. domain adversarial training aims to learn a domainindependent mapping for input samples by adding an adversarial cost during model training, that minimizes the distance between the source and target domain distributions. pivot-based approaches aim to select domain-invariant features (pivots) and use them as a basis for cross-domain mapping. this work does not fall under any of these categories, rather we aim to optimize the fine-tuning procedure of pretrained language models (lms) for learning under domain-shift. transfer learning from language models pretrained in massive corpora @xcite @xcite @xcite has yielded significant improvements across a wide variety of nlp tasks, even when small amounts of data are used for fine-tuning. fine-tuning a pretrained model is a straightforward framework for adaptation to target tasks and new domains, when labeled data are available. however, optimizing the fine-tuning process in uda scenarios, where only labeled out-ofdomain and unlabeled in-domain data are available is challenging. in this work, we propose udalm, a fine-tuning method for bert @xcite in order to address the uda problem. our method is based on simultaneously learning the task from labeled data in the source distribution, while adapting to the language in the target distribution using multitask learning. the key idea of our method is that by simultaneously minimizing a task-specific loss on the source data and a language modeling loss on the target data during fine-tuning, the model will be able to adapt to the language of the target domain, while learning the supervised task from the available labeled data. our key contributions are: (a) we introduce udalm, a novel, simple and robust unsupervised domain adaptation procedure for downstream bert models based on multitask learning, (b) we achieve state-of-the-art results for the amazon reviews benchmark dataset, surpassing more complicated approaches and (c) we explore how adistance and the target error are related and conclude with some remarks on domain adversarial training, based on theoretical concepts and our empirical observations. our code and models are publicly available foot_0 . ## related work traditionally, uda has been performed using pseudo-labeling approaches. pseudo-labeling techniques are semi-supervised algorithms that either use the same model (self-training) @xcite @xcite or multiple ensembles of models (tri-training) @xcite in order to produce pseudo-labels for the target unlabeled data. @xcite proposed an asymmetric tri-training approach. ruder and plank (2018) introduced a multi-task tri-training method. @xcite and @xcite study pseudolabeling with contextualized word representations. @xcite combine self-training with xlm-r @xcite to reduce the produced label noise and propose cfd, class aware feature self-distillation. another line of uda research includes pivotbased methods, focusing on extracting crossdomain features. structural correspondence learning (scl) @xcite and spectral feature alignment @xcite aim to find domaininvariant features (pivots) to learn a mapping between two domain distributions. @xcite @xcite combine scl with neural network architectures and language modeling. @xcite propose to jointly learn the task and pivots. @xcite learn pivots with hierarchical attention networks. pivot-based methods have also been used in conjunction with bert @xcite . domain adversarial training is a dominant approach for uda @xcite , in-spired by the theory for learning from different domains introduced in @xcite . @xcite ; @xcite propose to learn a task while not being able to distinguish if samples come from the source or the target distribution, through use of an adversarial cost. this approach has been adopted for a diverse set of problems, e.g. sentiment analysis, tweet classification and universal dependency parsing @xcite @xcite . @xcite pose domain adversarial training in the context of bert models. @xcite propose multi-source domain adversarial networks. @xcite propose a mixture-of-experts approach for multi-source uda. @xcite explore distance measures as additional losses and use them to construct dynamic multi-armed bandit controller for the source domains. @xcite learn domain invariant features via wasserstein distance. @xcite introduce domain seperation networks with private and shared encoders. unsupervised pretraining on domain-specific corpora can be an effective adaptation process. for example biobert @xcite and scib-ert @xcite are specialized bert variants, where pretraining is extended on large amounts of biomedical and scientific corpora respectively. @xcite propose continuing the pretraining of bert with target domain data and multitask learning using relevant tasks for bert fine-tuning. @xcite introduce a review reading comprehension task and a post-training approach for bert with an auxiliary loss on a question-answering task. continuing pretraining on multiple phases, from general to domain specific (dapt) and task specific data (tapt), further improves performance of pretrained language models, as shown by @xcite . @xcite propose adaptabert, which includes a second phase of unsupervised pretraining, in order to use bert in a unsupervised domain adaptation context. recent works have highlighted the merits of using language modeling as an auxiliary task during fine-tuning. @xcite use an auxiliary lm loss to avoid catastrophic forgetting in transfer learning @xcite adopt this approach for cross-domain named-entity recognition. we draw inspiration from these approaches and utilize auxiliary language modeling for uda. ## problem definition let x be the input space and y the set of labels. for binary classification tasks @xmath0, 1}. in domain adaptation there are two different distributions over x × y , called the source domain d s and the target domain d t . in the unsupervised setting labels are provided for samples drawn from d s , while samples drawn from d t are unlabeled. the goal is to train a model that performs well on samples drawn from the target distribution d t . this is summarized in eq. 1. ## proposed method fig. 1 gives an overview of the proposed unsupervised domain adaptation through language modeling (udalm). starting from a model that is pretrained in general corpora (fig. 1a ), we keep pretraining it on target domain data using the masked language modeling task (fig. 1b ). on the final fine-tuning step (fig. 1c ) we update the model weights using both a classification loss on the labeled source data and masked language modeling loss on the unlabeled target data. in fig. 1a we see the bert general pretraining phase. bert @xcite is based on the transformer architecture @xcite . during bert pretraining, input tokens are randomly selected to be masked. bert is trained using the masked language modeling (mlm) objective, which consists of predicting the most probable tokens for the masked positions. additionally it uses a next sentence prediction (nsp) loss, which classifies whether the pair of input sentences are continuous or not. if a labeled dataset is available, a pretrained bert model can be fine-tuned for the downstream task in a supervised manner with the addition of an output layer. in fig. 1b we initialize a model using the weights of a generally pretrained bert and continue pretraining on an unsupervised set of in-domain data, in order to adapt to the target domain. this step does not require use of supervised data, since we use the mlm objective. for the final fine-tuning step, shown in fig. 1c we perform supervised fine-tuning on the source data, while we keep the mlm objective on the target data as an auxiliary task. following standard practice, we use the [cls] token representation for classification. the classifier consists of a single feed-forward layer. during this procedure the model learns the task through the classification objective using the labeled source domain samples, and simultaneously it adapts to the target domain data through the mlm objective. the model is trained on the source domain labeled data for the classification task and target domain unlabeled data for the masked language modeling task. we mask only the target domain data. during training we interleave source and target data and feed them to the bert encoder. features extracted from the source data are then used for classification, while target features are used for masked language modeling. the mixed loss used for the fine-tuning step, is the sum of the classification loss l clf and the auxiliary mlm loss l m lm . l clf is a cross-entropy loss, calculated on labeled examples from source domain, while l m lm is used to predict masked tokens for unlabeled examples from target domain. we train the model over mixed batches, that include both source and target data, used for the respective tasks. the mixed loss is presented in eq. 2: we process n labeled source samples s ∼ d s and m unlabeled target samples t ∼ d t on a batch. the weighting factor λ is selected as the ratio of labeled source data over the sum of labeled source and unlabeled target data, as stated in eq. 3: ## conclusions and future work unsupervised domain adaptation of pretrained language models is a challenging problem with direct real world applications. in this work we propose udalm, a robust, plug and play training strategy, which is able to improve performance in the target domain, achieving state-of-the-art results across 12 adaptation settings in the multi-domain amazon reviews dataset. our method produces robust results with little hyper-parameter tuning and the proposed mixed-loss can be used for model validation, allowing for fast model development. furthermore, udalm scales with the amount of available unsupervised data from the target domain, allowing for adaptation in low-resource settings. in our analysis, we discuss the relationship between the a-distance and the target error. we observe that low a-distance may not suggest low target error for high capacity models. additionally, we examine limitations of domain adversarial training and highlight that the adversarial cost may lead to distortion of the feature space and negatively impact performance. in the future we plan to apply udalm to other tasks under domain-shift, such as sequence classification, question answering and part-of-speech tagging. furthermore, we plan to extend our method for temporal and style adaptation, by adding more relevant auxiliary tasks that model language shift over time and over different platforms. finally, we want to investigate the effectiveness of the proposed fine-tuning approach in supervised scenarios.