Semi-Supervised Domain Adaptation for Emotion-Related Tasks (2023)

Abstract: Semi-supervised domain adaptation (SSDA) adopts a model trained from a label-rich source domain to a new but related domain with a few labels of target data. It is shown that, in an SSDA setting, a simple combination of domain adaptation (DA) with semi-supervised learning (SSL) techniques often fails to effectively utilize the target supervision and cannot address distribution shifts across different domains due to the training data bias toward the source-labeled samples. In this paper, inspired by the co-learning of multiple classifiers for the computer vision tasks, we propose to decompose the SSDA framework for emotion-related tasks into two subcomponents of unsupervised domain adaptation (UDA) from the source to the target domain and semi-supervised learning (SSL) in the target domain where the two models iteratively teach each other by interchanging their high confident predictions. We further propose a novel data cartography-based regularization technique for pseudo-label denoising that employs training dynamics to further hone our models’ performance. We publicly release our code.

## introduction large pre-trained language models @xcite have significantly improved many natural language processing (nlp) task performances with the help of large quantities of labeled training data and have become the de facto model for nlp applications. however, obtaining vast troves of annotated data for training in many real-world scenarios is costly and challenging. for example, reliable large annotated emotion or empathy data might not exist in a computer-assisted therapy session @xcite . on top of that, it is shown that a shift in data distribution can substantially affect the performance of such text classification models @xcite ; a model trained on a source domain does not perform well on a dataset from another domain. this deficiency is due to the domain shift across the datasets @xcite , which is a problem that is commonly encountered in nlp. to relieve the unsupervised domain adaptation (uda) bottleneck for textual tasks, recent works attempt to align distributions between source and target domains by extracting the domain-invariant representations @xcite . however, despite the recent progress, the uda methods are still impractical as they may yield different new domain-sensitive particularities for large-scale language models. recent works showed that the presence of few labeled samples from the target domain in a semisupervised domain adaption (ssda) setup can positively impact and significantly boost the performance of the neural models @xcite @xcite . there exists prior work on supervised domain adaptation @xcite and multi-task learning @xcite for natural language processing (nlp) tasks. however, despite the importance of semisupervised domain adaption, in nlp, only a few studies have focused on this problem @xcite . @xcite expanded an existing fully supervised domain adaptation technique @xcite to semi-supervised domain adaptation settings using co-regularization, which originated in the context of multi-view learning @xcite . @xcite also framed the semi-supervised domain adaptation problem as learning with a transformation function and a prediction model under manifold constraints. given a large number of labeled data provided in the source domain and only a few target labeled data with the inherent distributional difference, it is shown that in an ssda setting, a single classifier may likely be dominated by the source domain @xcite . in this paper, we extend the co-training strategy @xcite , a semi-supervised learning approach for multi-view data, to a single-view setting for nlp tasks to effectively use unlabeled target data. co-training trains two classifiers from each view and employs the most confident predictions of the unlabeled data for the two classifiers to teach each other. inspired by co-training, we propose to decompose the ssda framework and learn two distinct classifiers to teach each other so that both classifiers can excel in the target domain. particularly, we employ an unsupervised domain adaptation setup where we leverage the labeled source data and the unlabeled target data to learn one classifier. furthermore, we employ a semi-supervised learning setup where we learn another classifier using the labeled target data together with the unlabeled target data. we further propose a novel data cartography-based regularization technique for pseudo-label denoising that employs training dynamics @xcite to further hone our models' performance. our preliminary results on three emotion-related nlp tasks show that transferring knowledge between the two classifiers and incorporating training dynamics to help denoising the generated pseudolabels can effectively improve the performance on the target domain. ## conclusion in this work, we extend the co-training strategy as a semi-supervised learning approach for multiview data to a single-view setting for nlp tasks and propose to decompose the ssda framework and learn two distinct classifiers (one in an semisupervised setup and another one in a domain adaptation setup) to teach each other so that both classifiers can excel in the target domain. we further propose a novel data cartography-based regularization technique for pseudo-label denoising that employs training dynamics to further hone our models' performance. our preliminary results show that denoising the pseudo-labels of unlabeled target data using high-quality labeled target data within a cotraining framework yields improvements in performance over multiple baselines.