Pseudo-Label Guided Unsupervised Domain Adaptation of Contextual Embeddings (2021)

Abstract: Contextual embedding models such as BERT can be easily fine-tuned on labeled samples to create a state-of-the-art model for many downstream tasks. However, the fine-tuned BERT model suffers considerably from unlabeled data when applied to a different domain. In unsupervised domain adaptation, we aim to train a model that works well on a target domain when provided with labeled source samples and unlabeled target samples. In this paper, we propose a pseudo-label guided method for unsupervised domain adaptation. Two models are fine-tuned on labeled source samples as pseudo labeling models. To learn representations for the target domain, one of those models is adapted by masked language modeling from the target domain. Then those models are used to assign pseudo-labels to target samples. We train the final model with those samples. We evaluate our method on named entity segmentation and sentiment analysis tasks. These experiments show that our approach outperforms baseline methods.

## introduction contextualized embeddings have become the foundations of many state-of-the-art natural language processing technologies @xcite @xcite . pretrained contextualized embeddings can be used for many downstream tasks and be incorporated into an end-to-end system, allowing the embeddings to be fine-tuned from task-specific labeled data @xcite @xcite . one of the problems with contextual embedding models is that although fine-tuned models perform well on the samples generated from the same distribution as the training samples, they suffer considerably from unlabeled data when applied to a different domain @xcite @xcite figure 1 : overview of our training framework. we jointly fine-tune general pre-trained model and target domain pre-trained model with labeled source data. then we generate pseudo-labels on target samples. finally, we train the final model with the pseudo-labeled samples. @xcite . for example, a named entity segmentation model trained on a news dataset fails to predict correctly on social media data such as twitter. because collecting many labeled samples in various domains is expensive, it is important to adapt contextual embedding model to different domains in unsupervised setting. many domain adaptation methods of neural networks in nlp have been proposed in the past several years @xcite @xcite @xcite @xcite . our work focuses on unsupervised domain adaptation of contextual embeddings. we aim to fine-tune a pre-trained model that works well on a target domain when provided with labeled source samples and unlabeled target samples. with no access to the labels in target domain data, it is very difficult to adapt when the divergence of the label distribution between the source domain and the target domain is huge. some of current methods propose a simple unsupervised domain-adaptive method, using a masked language modeling objective over unlabeled text in the target domain @xcite @xcite . they first learn discriminative representations for the tar-get domain and then fine-tune a domain-adapted model with labeled source samples. although selfsupervised fine-tuning in the target domain improves the generalization of the pre-trained model for the target domain, an adapted model cannot capture the task-specific pattern of the target domain only using labeled source samples to fine-tune. we expect the adapted model to not only acquire some target-discriminative language representations but also to obtain some task-specific features of target domain. in this paper, we propose a pseudo-label guided method for unsupervised domain adaptation. as shown in figure 1 , two models are jointly finetuned on labeled source samples as pseudo labeling models. we design a multiview constraint loss to encourage those two model to make predictions based on different view-point. to learn representations for the target domain, one of those models is adapted by masked language modeling from the target domain. then those models are used to assign pseudo-labels to target samples. pseudo-labeled target samples will provide target discriminative information to the model. we train the final model with the pseudo-labeled samples. we evaluate our method both on named entity segmentation (nes) and sentiment analysis (sa) tasks. we find that our pseudo-label guided method outperforms baseline methods. moreover, we demonstrate the multiview constraint significantly improves performance of our method. ## conclusion we propose a new unsupervised domain adaptation method guided by pseudo labels. generated by general pre-trained model and target domain pre-trained model with multiview constraint, the pseudo labels of unlabeled target data are more reliable and benefit model performance on the target domain. experiments show that our approach achieves very promising results on different nlp downstream tasks. yftah ziser and roi reichart. 2018. pivot based language modeling for improved neural domain adaptation. in proceedings of the 2018 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long papers), pages 1241-1251, new orleans, louisiana. association for computational linguistics. yftah ziser and roi reichart. 2019. task refinement learning for improved accuracy and stability of unsupervised domain adaptation. in proceedings of the 57th annual meeting of the association for computational linguistics, pages 5895-5906.