S enti X : A Sentiment-Aware Pre-Trained Model for Cross-Domain Sentiment Analysis (2020)

Abstract: Pre-trained language models have been widely applied to cross-domain NLP tasks like sentiment analysis, achieving state-of-the-art performance. However, due to the variety of users’ emotional expressions across domains, fine-tuning the pre-trained models on the source domain tends to overfit, leading to inferior results on the target domain. In this paper, we pre-train a sentiment-aware language model (SentiX) via domain-invariant sentiment knowledge from large-scale review datasets, and utilize it for cross-domain sentiment analysis task without fine-tuning. We propose several pre-training tasks based on existing lexicons and annotations at both token and sentence levels, such as emoticons, sentiment words, and ratings, without human interference. A series of experiments are conducted and the results indicate the great advantages of our model. We obtain new state-of-the-art results in all the cross-domain sentiment analysis tasks, and our proposed SentiX can be trained with only 1% samples (18 samples) and it achieves better performance than BERT with 90% samples.

## introduction sentiment analysis has gained widespread attention from both industry and academia, which aims to judge the sentimental polarity of the given text @xcite . most existing works heavily rely on labeled data to train separate sentiment classifiers for each domain, which are both expensive and timeconsuming to obtain @xcite . therefore, cross-domain sentiment analysis has become a promising direction, which transfers (invariant) sentiment knowledge from the source domain to the target domain 1 . the major challenge here is that language expressions for sentimental text usually vary across different domains. for instance, "fast" has a positive sentiment towards "service" in the restaurant domain (figure 1 ), while in the laptop domain, "fast" expresses a negative sentiment for "power consumption". furthermore, models trained on the source domain tend to overfit, since they learn domain-specific knowledge excessively. therefore, many studies @xcite @xcite propose to address this issue by extracting domain-invariant features. recently, pre-trained language models like bert @xcite have achieved the state-of-theart performance on multiple sentiment analysis tasks @xcite @xcite . however, when they are directly applied to cross-domain sentiment analysis @xcite , two problems arise: 1) existing pre-trained models focus on learning the semantic content via self-supervision strategies, while ignoring sentiment-specific knowledge at the pre-training phrase; 2) ˚yuanbin wu and liang he are the corresponding authors of this paper. this work was conducted when jie zhou was interning at alibaba damo academy. laptop domain this computer consumes power so fast :(. it was a bad experience. i will never buy it again. restaurant domain this is a beautiful place (^_^). the service is fast and food is delicious. token-level: sentiment lexicons, emoticons sentence-level: rating figure 1 : an example of sentiment knowledge (e.g., "bad", "beautiful", ":(", "p ^^q"). to address the above-mentioned problems, we propose a sentiment-aware pre-trained model, named sentix, to learn the domain-invariant sentiment knowledge at the pre-training phase, and it does not need to be fine-tuned for the cross-domain tasks. in particular, we observe that many widely available review datasets contain rich sentiment information, which can be utilized to enhance the domain-invariant knowledge acquisition. large-scale review datasets, such as yelp and amazon, consist of 240 million reviews across 30 domains, full of sentiment words, emoticons and ratings. taking figure 1 as an example, these reviews contain opinion words (like "bad", "beautiful") and emoticons (like ":(", "p ^^q"), and their ratings are 1 and 5, respectively. in order to obtain the above domain-invariant sentiment knowledge, we propose several sentimentaware pre-training objectives, including token and sentiment prediction. at the token level, the sentiment words and emoticons are masked with a higher rate than the general words to emphasize the sentiment knowledge, and we pre-train sentix to predict sentiment-aware words, emoticons, and token sentiments. at the sentence level, we introduce a rating prediction strategy to learn the sentiment knowledge based on the whole sentence. we conduct extensive experiments on cross-domain sentiment analysis tasks to evaluate the effectiveness of sentix, and obtain state-of-the-art results on all settings. sentix achieves more than 90% accuracy over all cross-domain sentiment analysis datasets with only 1% samples, outperforming bert trained with 90% samples. through visualization of the feature representation, we observe that sentix significantly reduces the overfitting issue, while the in-domain tests prove that our sentix also obtains significant improvement over bert for both the sentence-level and aspect-based sentiment classification tasks. the main contributions of this paper can be summarized as follows: ## preliminaries reviews contain a lot of semi-supervised sentiment signals, such as sentiment words, emoticons and ratings, and large-scale review data can be obtained from online review websites like yelp. this sentiment knowledge can help learning domain-common sentiment feature for the cross-domain task. emoticon and was . e sentiment masking pre-training objectives pretty , sentiment word [mask] fast [mask] service good rating positive :) fast cheerful food service good and , was pretty . [cls] positive [mask] sentiment-aware word prediction word sentiment prediction emoticon prediction rating prediction figure 2 : the framework of sentix. first, we design three sentiment masking strategies, including swm (e.g., "cheerful", "good"), em (e.g., ":)"), and gwm (e.g., "food"). then, we propose four sentiment-aware prediction objectives from token level and sentence level. to make full use of this rich sentiment knowledge for cross-domain sentiment analysis, we design several pre-training objectives to enhance the model with domain-invariant sentiment knowledge. ## sentix sentix is a sentiment-aware pre-training model for cross-domain sentiment analysis. it learns domaininvariant features from the above domain-invariant sentiment knowledge, including sentiment lexicons, emoticons, and ratings. the framework contains sentiment masking and pre-training objectives, as shown in figure 2 . sentiment masking (section 3.1) recognizes the sentiment information of an input sequence from sentiment knowledge. pre-training objectives require encoder not only reconstruct the masked sentiment tokens, but also distinguish the word sentiment polarity, emoticon and rating (section 3.2). formally, given a sentence x " tx 1 , x 2 , ..., x |x| u, we first obtain a corrupted sentence x " tx 1 , x2 , ..., x|x| u (x p x , where x is the corrupted corpus.) via sentiment masking. the sentimentaware pre-training tasks are proposed to predict word x i , sentiment s i , emoticon e i of in token level and rating r in sentence level. here s i p tp, n, 0u represents the sentiment polarity (positive, negative, others) of word x i , e i p te, 0u indicates whether word x i is an emoticon, and r p t1, 2, 3, 4, 5u is the rating of x. ## results and analysis in this section, we conduct a series of experiments to validate the performance of sentix. first, we test our model on in-domain sentiment analysis tasks (section 5.1) to prove that sentix performs well for sentiment analysis. second, to verify the effectiveness of sentix, we conduct extensive experiments on cross-domain sentiment analysis (section 5.2). third, we also perform an ablation test to investigate the effectiveness of each component (section 5.3). forth, we explore the influence of the number of training samples (section 5.4) and visualize the feature representations of sentix (section 5.5). finally, we investigate the time complexity, space complexity, and convergence analysis of sentix (section 5.6).