Mere Contrastive Learning for Cross-Domain Sentiment Analysis (2022)

Abstract: Cross-domain sentiment analysis aims to predict the sentiment of texts in the target domain using the model trained on the source domain to cope with the scarcity of labeled data. Previous studies are mostly cross-entropy-based methods for the task, which suffer from instability and poor generalization. In this paper, we explore contrastive learning on the cross-domain sentiment analysis task. We propose a modified contrastive objective with in-batch negative samples so that the sentence representations from the same class can be pushed close while those from the different classes become further apart in the latent space. Experiments on two widely used datasets show that our model can achieve state-of-the-art performance in both cross-domain and multi-domain sentiment analysis tasks. Meanwhile, visualizations demonstrate the effectiveness of transferring knowledge learned in the source domain to the target domain and the adversarial test verifies the robustness of our model.

## introduction sentiment classification @xcite has been widely studied by both industry and academia @xcite @xcite . for example, the sentiment is positive towards the text 'the book is exactly as pictured/described. cute design and good quality'. early methods rely on labeled data to train models on a specific domain (e.g. dvd reviews, book reviews, and so on), which are labor-intensive and time-consuming @xcite . to address this issue, cross-domain sentiment analysis attracts increasing attention. various neural models have been proposed for cross-domain sentiment analysis in recent years @xcite @xcite @xcite . most methods focus on making the model unable to distinguish the data from which domain by adversarial training or data argumentation, in order to transfer knowledge from source domains to target domains @xcite @xcite and some attempt to learn domain-specific knowledge @xcite @xcite . pre-trained language models @xcite @xcite have achieved stronger performance compared with previous random initialized models such as lstm (long short-term memory) in cross-domain tasks. the state-of-theart models on cross-domain sentiment analysis, such as bert-daat @xcite , use unlabeled data to continually train the pre-trained model bert to transfer knowledge besides adversarial training. in the representation aspect for the cross-domain sentiment analysis, there are two key requirements for the representations of sentences: (1) sentence representations in the same domain with the different/the same sentiments should be far from/close to each other; (2) sentence representations of different domains with the same labels should be close. existing methods are mostly softmax-based method by optimizing cross-entropy loss to achieve the requirements (illustrate in figure 1 (a) ), which suffers from instability across different runs @xcite , poor generalization performance @xcite , reduction of prediction diversity @xcite , lack of robustness to noisy labels @xcite , or adversarial examples @xcite , especially when supervised data are limited in the cross-domain settings. to address the above shortcomings, we explore the effectiveness of contrastive learning on the task. contrastive learning is a similarity-based training strategy, which aims to push the representations from the same class close and those from the different class further apart @xcite @xcite . contrastive learning has been shown effective in solving the problem of anisotropy @xcite , and it has good generalization and robustness @xcite @xcite . previous work relies mostly on pre-training for representations @xcite or multi-task training for semantic textual similarity @xcite , classification @xcite and so on, but little work uses mere contrastive learning for supervised tasks. intuitively, the optimization of contrastive learning is effective in satisfying the requirements of cross-domain sentiment analysis. we explore contrastive learning on bert (cobe) by a modified contrastive loss function with the in-batch negative method on cross-domain sentiment analysis tasks. in the mini-batch, the samples with the same labels are treated as positive pairs, and those with different labels are treated as negative pairs. as shown in figure 1 , the optimization procedure aims to tighten the cluster of samples with the same labels, and push away samples with different labels. after training, the representations of training data and their labels are saved offline as a knowledge base for classifica-tion. when evaluating the model, a knn (k-nearest neighbors) predictor is used to predict the sentiment of test data, i.e. we search for the k data with the largest cosine similarity in the knowledge base and vote for the final prediction using their labels. experiments on two widely used datasets (the cross-domain amazon dataset @xcite and fdu-mtl @xcite ) show that our model can achieve the state-of-the-art performance in both the cross-domain setting and the multidomain setting sentiment classification. visualizations also demonstrate the effectiveness of transferring knowledge learned in the source domain to the target domain. to our knowledge, we are the first to show that contrastive learning outperforms cross-entropy-based models on crossdomain sentiment analysis for both performance and robustness. the code has been released in https://github.com/luoxiaoheics/cobe . contrastive learning. contrastive learning has been widely used in unsupervised learning @xcite @xcite @xcite . @xcite propose to use contrastive learning to learn the representations of both text and images through raw data in unsupervised method, which achieves strong performance on zero-shot task. @xcite propose to use contrastive learning to obtain sentence and code representations and achieve strong performance on downstream tasks such as sentence classification and text search. wang and isola (2020) further identify the key properties for contrastive learning as (1) alignment (closeness) of features from positive pairs and (2) uniformity of induced distribution of representations. @xcite uses contrastive learning to learn the sentence representations and theoretically prove that contrastive learning can solve the anisotropy problem (the learned embeddings occupy a narrow cone in the vector space), which limits the expressiveness of representations. it also achieves better results for the semantic textual similarity task using a supervised dataset of natural language inference. our model differs from the above studies in that we consider contrastive learning in supervised tasks, which uses golden labels to obtain positive/negative pairs for training. recently, some studies attempt to incorporate contrastive learning into cross-entropy-based methods by adding infonce loss @xcite , which aims to solve the shortcomings of cross-entropy loss. @xcite propose a new scl loss based on infonce loss to boost the stability and robustness of fine-tuning pre-trained language models. subsequently, @xcite attempt to incorporate knn predictors to enhance the generalization of prediction in few-shot tasks, using both cross-entropy loss and scl loss. the above work is similar to ours in making use of contrastive loss for classification. however, the difference is that we do not use a standard cross-entropy loss, but rely solely on vector space similarity losses for achieving cross-domain classification. to our knowledge, we are the first to conduct sentiment classification without using a cross-entropy loss in natural language processing. ## experiments we conduct experiments on both the cross-domain settings (train models on source domains and test on another one) and the multi-domain settings (train and test models on the same domains). to verify the effectiveness of our model, we also visualize the representations (section 4.3) and carry out further analysis such as model robustness (section 4.4). ## conclusion we explored the contrastive learning method in the cross-domain sentiment analysis task. we proposed a suitable contrastive loss for the supervised sentiment analysis task with the in-batch negatives method. experiments on two standard datasets showed the effectiveness of our model. visualizations also demonstrated the effectiveness of transferring knowledge learned in the source domain to the target domain. we also showed that our model has stronger robustness than cross-entropy-based models through the adversarial test. kaidi cao, colin wei, adrien gaidon, nikos arechiga, and tengyu ma. 2019. learning imbalanced datasets with label-distribution-aware margin loss. advances in neural information processing systems, 32. a statistics for fdu-mtl.