Sentiment-Aware Word and Sentence Level Pre-training for Sentiment Analysis (2022)

Abstract: Most existing pre-trained language representation models (PLMs) are sub-optimal in sentiment analysis tasks, as they capture the sentiment information from word-level while under-considering sentence-level information. In this paper, we propose SentiWSP, a novel Sentiment-aware pre-trained language model with combined Word-level and Sentence-level Pre-training tasks.The word level pre-training task detects replaced sentiment words, via a generator-discriminator framework, to enhance the PLM’s knowledge about sentiment words.The sentence level pre-training task further strengthens the discriminator via a contrastive learning framework, with similar sentences as negative samples, to encode sentiments in a sentence.Extensive experimental results show that SentiWSP achieves new state-of-the-art performance on various sentence-level and aspect-level sentiment classification benchmarks. We have made our code and model publicly available at https://github.com/XMUDM/SentiWSP.

## introduction sentiment analysis plays a fundamental role in natural language processing (nlp) and powers a broad spectrum of important business applications such as marketing @xcite and campaign monitoring (sandoval-almazán and valle-cruz, 2020). two typical sentiment analysis tasks are sentence-level sentiment classification @xcite @xcite and aspect-level sentiment classification @xcite . recently, pre-trained language representation models (plms) such as elmo @xcite , gpt @xcite , bert @xcite , roberta @xcite and xl-net @xcite have brought impressive performance improvements in many nlp problems, including sentiment analysis. plms learn a robust encoder on a large unlabeled corpus, through carefully designed pre-training tasks, such as masked token or next sentence prediction. despite their progress, the application of general purposed plms in sentiment analysis is limited, because they fail to distinguish the importance of different words to a specific task. for example, it is shown in @xcite that general purposed plms have difficulties dealing with contradictory sentiment words or negation expressions, which are critical in sentiment analysis. to address this problem, recent sentiment-aware plms introduce word-level sentiment information, such as token sentiments and emoticons @xcite , aspect word @xcite , word-level linguistic knowledge @xcite , and implicit sentiment-knowledge information @xcite . these word-level pre-training tasks, e.g., sentiment word prediction and word polarity prediction, mainly learn from the masked words and are not efficient to capture word-level information for all input words. furthermore, sentiment expressed in a sentence is beyond the simple aggregation of word-level sentiments. however, general purposed plms and existing sentiment-aware plms underconsider sentence-level sentiment information. in this paper, we propose a novel sentimentaware pre-trained language model called sentiwsp, to combine word-level pre-training and sentence-level pre-training. inspired by electra @xcite , which pre-trains a masked language model with significantly less computation resource, we adopt a generatordiscriminator framework in the word-level pre-training. the generator aims to replace masked words with plausible alternatives; and the discriminator aims to predict whether each word in the sentence is an original word or a substitution. to tailor this framework for sentiment analysis, we mask two types of words for generation, sentiment words and non-sentiment words. we increase the portion of masked sentiment words so that the model focuses more on the sentiment expressions. for sentence-level pre-training, we design a contrastive learning framework to improve the encoded embeddings by the discriminator. the query for the contrastive learning is constructed by masking sentiment expressions in a sentence. the positive example is the original sentence. the negative examples are selected firstly from in-batch samples and then from cross-batch similar samples using an asynchronously updated approximate nearest neighboring (ann) index. in this way, the discriminator, which will be used as the encoder for downstream tasks, learns to distinguish different sentiment polarities even if they are superficially similar. our main contributions are in three folds: 1). sentiwsp strengthens word-level pre-training via masked sentiment word generation and detection, which is more sample-efficient and benefits various sentiment classification tasks; 2). sentiwsp combines word-level pretraining with sentence-level pre-training, which has been underconsidered in previous studies. sentiwsp adopts contrastive learning in the pre-training, where sentences are progressively contrasted with in-batch and crossbatch hard negatives, so that the model is empowered to encode detailed sentiment information of a sentence; 3). we conduct extensive experiments on sentence-level and aspect-level sentiment classification tasks, and show that sentiwsp achieves new state-of-the-art performance on multiple benchmarking datasets. in addition to the pre-training models that encode token representations, sentence-level and passage-level representation learning have undergone rapid development in recent years. a surge of work demonstrates that contrastive learning is an effective framework for sentence-and passagelevel representation learning @xcite @xcite . the common idea of contrastive learning is to pull together an anchor and a "positive" sample in the embedding space, and push apart the anchor from "negative" samples. recently, coco-lm @xcite creates positive samples by masking and cropping tokens from sentences. @xcite demonstrate that constructing positive pairs with only standard dropout as minimal data augmentation works surprisingly well on the natural language inference (nli) task. @xcite investigate the impact of different negative sampling strategies for passage representation learning based on the task of passage retrieval and question answering. ance @xcite adopts approximate nearest neighbor negative contrastive learning, a learning mechanism that selects hard negatives globally from the entire corpus, using an asynchronously updated approximate nearest neighbor (ann) index. inspired by coco-lm @xcite and ance @xcite , we construct positive samples by masking a span of words from a sentence, and construct cross-batch hard negative samples to enhance the discriminator at sentence-level pre-training. pre-trained models for sentiment analysis in the field of sentiment analysis, bert-pt @xcite conducts post-training on the corpora which belong to the same domain of the downstream tasks to benefit aspect-level sentiment classification. skep (tian et al., 2020) constructs three sentiment knowledge prediction objectives in order to learn a unified sentiment representation for multiple sentiment analysis tasks. sentix @xcite investigates domain-invariant senti-inferencer trainer ann negatives query [cls] a smart sassy exceptionally charming romantic comedy [sep] [cls] a [mask] sassy exceptionally [mask] romantic comedy [sep] df k-2 df k-1 df k inferencing inferencing index &search index &search checkpoint k-1 checkpoint k checkpoint k+1 sentiment word sentiment masking sample o:original r:replaced generator a smart [mask] exceptionally [mask] romantic [mask] a smart sassy exceptionally charming romantic comedy a smart stylish exceptionally charming romantic book discriminator o o r o o o r positive ... sentence-level pre-training word-level pre-training ment knowledge from large-scale review datasets, and utilizes it for cross-domain sentiment classification tasks without fine-tuning. @xcite proposes a two-level attention mechanism on top of the bert representation to capture phrase-level compositional semantics. senti-lare @xcite devises a new pre-training task called label-aware masked language model to construct knowledge-aware language representation. scapt @xcite captures both implicit and explicit sentiment orientation from reviews by aligning the representation of implicit sentiment expressions to those with the same sentiment label. ## sentiment-aware word-level and sentence-level pre-training the overall framework of sentiwsp is depicted in figure 1 . sentiwsp consists of two pre-training phases, namely word-level pre-training (sec. 3.1, and sentence-level pre-training (sec. 3.2), before fine-tuning (sec. 3.3) on a downstream sentiment analysis task. in word-level pre-training, an input sentence flows through a word-masking step, followed by a generator to replace the masked words, and a discriminator to detect the replacements. the generator and discriminator are jointly trained in this stage. then, the training of discriminator continues in sentence-level pre-training. each input sentence is masked at sentiment words to construct a query, while the original sentence is treated as the positive sample. their embeddings encoded by the discriminator are contrasted to two types of negative samples constructed in an in-batch warm-up training step and a cross-batch approximate nearest neighbor training step. finally, the discriminator is fine-tuned on the downstream task. compared with previous studies, the discriminator in sentiwsp has three advantages. (1) instead of random token replacement and detection, sen-tiwsp masks a large portion of sentiment words, and thus the discriminator pays more attention to word-level sentiments. (2) instead of pure masked token prediction, sentiwsp incorporates context information from all input words via a replacement detection task. (3) sentiwsp combines sentencelevel sentiments with word-level sentiments by progressively contrasting a sentence with missing sentiments to a superficially similar sentence. ## conclusion in this paper, we introduce sentiwsp, which improves pre-training models on sentiment analysis task, by capturing the sentiment information from word-level and sentence-level simultaneously. extensive experimental results on five sentencelevel sentiment classification benchmarks show that sentiwsp establishes new state-of-the-art performance on all of them. we conduct experiment on two aspect-level sentiment classification benchmarks. the results show that sentiwsp beats most existing models on restaurant14 and achieves new state-of-the-art on laptop14. we further analyze several hyper-parameters that may affect the model performance, and show that sentiwsp can achieve satisfying performance with respect to different hyper-parameter settings.