Feature Adaptation of Pre-Trained Language Models across Languages and Domains with Robust Self-Training (2020)

Abstract: Adapting pre-trained language models (PrLMs) (e.g., BERT) to new domains has gained much attention recently. Instead of fine-tuning PrLMs as done in most previous work, we investigate how to adapt the features of PrLMs to new domains without fine-tuning. We explore unsupervised domain adaptation (UDA) in this paper. With the features from PrLMs, we adapt the models trained with labeled data from the source domain to the unlabeled target domain. Self-training is widely used for UDA, and it predicts pseudo labels on the target domain data for training. However, the predicted pseudo labels inevitably include noise, which will negatively affect training a robust model. To improve the robustness of self-training, in this paper we present class-aware feature self-distillation (CFd) to learn discriminative features from PrLMs, in which PrLM features are self-distilled into a feature adaptation module and the features from the same class are more tightly clustered. We further extend CFd to a cross-language setting, in which language discrepancy is studied. Experiments on two monolingual and multilingual Amazon review datasets show that CFd can consistently improve the performance of self-training in cross-domain and cross-language settings.

## introduction pre-trained language models (prlms) such as bert @xcite and its variants @xcite have shown significant success for various downstream nlp tasks. however, these deep neural networks are sensitive to different cross-domain distributions @xcite and their effectiveness will be much weakened in such a scenario. how to adapt prlms to new domains is important. unlike the most recent work that fine-tunes prlms on the unlabeled data from the new domains @xcite , we are interested in how to adapt the prlm features without fine-tuning. to investigate this, we specifically study unsupervised domain adaptation (uda) of prlms, in which we adapt the models trained with source labeled data to the unlabeled target domain based on the features from prlms. self-training has been proven to be effective in uda @xcite , which uses the model trained with source labeled data to predict pseudo labels on the unlabeled target set for model training. unlike the methods of adversarial learning @xcite and maximum mean discrepancy (mmd) @xcite that learn domain-invariant features for domain alignment, self-training aims to learn discriminative features over the target domain, since simply matching domain distributions cannot make accurate predictions on the target after adaptation @xcite . to learn discriminative features for the target, self-training needs to retain a model's high-confidence predictions on the target domain which are considered correct for training. methods like ensemble learning @xcite @xcite which adopt multiple models to jointly make decisions on pseudolabel selections have been introduced to achieve this goal. though these methods can substantially reduce wrong predictions on the target, there will still be noisy labels in the pseudo-label set, with negative effects on training a robust model, since deep neural networks with their high capacity can easily fit to corrupted labels @xcite . in our work, to improve the robustness of selftraining, we propose to jointly learn discriminative features from the prlm on the target domain to alleviate the negative effects caused by noisy labels. we introduce class-aware feature selfdistillation (cfd) to achieve this goal ( §4.2). the features from prlms have been proven to be highly discriminative for downstream tasks, so we propose to distill this kind of features to a feature adaptation module (fam) to make fam capable of extracting discriminative features ( §4.2.1). inspired by recent work on representation learning @xcite , we introduce mutual information (mi) maximization for feature self-distillation (fd). we maximize the mi between the features from the prlm and the fam to make the two kinds of features more dependent. since fd can only distill features from the prlm, it ignores the cluster information of data points which can also improve feature discriminativeness @xcite . hence, for the features output by fam, if the corresponding data points belong to the same class, we further minimize their feature distance to make the cluster more cohesive, so that different classes will be more separable. to retain high-confidence predictions, we re-rank the predicted candidates and balance the numbers of samples in different classes ( §4.1). we use xlm-r @xcite as the prlm which is trained on over 100 languages. we also extend our method to cross-language, as well as cross-language and cross-domain settings using xlm-r, since it has already mapped different languages into a common feature space. we experiment with two monolingual and multilingual amazon review datasets for sentiment classification: monoamazon for cross-domain and multiamazon for cross-language experiments. we demonstrate that self-training can be consistently improved by cfd in all settings ( §5.3). further empirical results indicate that the improvements come from learning lower errors of ideal joint hypothesis ( §4.3,5.4). ## preliminary in this section, we introduce the problem definition and the model architecture based on which we build our domain adaptation algorithm presented in the next section. ## class-aware feature self-distillation for domain adaptation in this section, we introduce our method for domain adaptation. our domain adaptation loss function takes the form of: in which l s pred is for learning a task-specific classifier with the source labeled set s (eq. 7), l t pred is the self-training loss trained with the pseudolabel set t ( §4.1), and l cf d is to enhance the robustness of self-training by learning discriminative features from the prlm ( §4.2), which is the main algorithm for domain adaptation in this work. ## conclusion in this paper, we study how to adapt the features from the pre-trained language models without tuning. we specifically study unsupervised domain adaptation of prlms, where we transfer the models trained in labeled source domain to the unlabeled target domain based on prlm features. we build our adaptation method based on self-training. to enhance the robustness of self-training, we present the method of class-aware feature self-distillation to learn discriminative features. experiments on sentiment analysis in cross-language and crossdomain settings demonstrate the effectiveness of our method.