SKEP : Sentiment Knowledge Enhanced Pre-training for Sentiment Analysis (2020)

Abstract: Recently, sentiment analysis has seen remarkable advance with the help of pre-training approaches. However, sentiment knowledge, such as sentiment words and aspect-sentiment pairs, is ignored in the process of pre-training, despite the fact that they are widely used in traditional sentiment analysis approaches. In this paper, we introduce Sentiment Knowledge Enhanced Pre-training (SKEP) in order to learn a unified sentiment representation for multiple sentiment analysis tasks. With the help of automatically-mined knowledge, SKEP conducts sentiment masking and constructs three sentiment knowledge prediction objectives, so as to embed sentiment information at the word, polarity and aspect level into pre-trained sentiment representation. In particular, the prediction of aspect-sentiment pairs is converted into multi-label classification, aiming to capture the dependency between words in a pair. Experiments on three kinds of sentiment tasks show that SKEP significantly outperforms strong pre-training baseline, and achieves new state-of-the-art results on most of the test datasets. We release our code at https://github.com/baidu/Senta .

## introduction sentiment analysis refers to the identification of sentiment and opinion contained in the input texts that are often user-generated comments. in practice, sentiment analysis involves a wide range of specific tasks @xcite , such as sentence-level sentiment classification, aspect-level sentiment classification, opinion extraction and so on. traditional methods often study these tasks separately and design specific models for each task, based on manuallydesigned features @xcite or deep learning @xcite . recently, pre-training methods @xcite @xcite have shown their powerfulness in learning general semantic representations, and have remarkably improved most natural language processing (nlp) tasks like sentiment analysis. these methods build unsupervised objectives at word-level, such as masking strategy @xcite , next-word prediction @xcite or permutation @xcite . such wordprediction-based objectives have shown great abilities to capture dependency between words and syntactic structures @xcite . however, as the sentiment information of a text is seldom explicitly studied, it is hard to expect such pre-trained general representations to deliver optimal results for sentiment analysis @xcite . sentiment analysis differs from other nlp tasks in that it deals mainly with user reviews other than news texts. there are many specific sentiment tasks, and these tasks usually depend on different types of sentiment knowledge including sentiment words, word polarity and aspect-sentiment pairs. the importance of these knowledge has been verified by tasks at different level, for instance, sentence-level sentiment classification @xcite @xcite , aspect-level sentiment classification @xcite , opinion extraction @xcite @xcite and so on. therefore, we assume that, by integrating these knowledge into the pre-training process, the learned representation would be more sentimentspecific and appropriate for sentiment analysis. in order to learn a unified sentiment representation for multiple sentiment analysis tasks, we propose sentiment knowledge enhanced pre-training (skep), where sentiment knowledge about words, polarity, and aspect-sentiment pairs are included to guide the process of pre-training. the sentiment knowledge is first automatically mined from unlabeled data (section 3.1). with the knowledge transformer encoder @xcite came this @xcite and really mined, sentiment masking (section 3.2) removes sentiment information from input texts. then, the pre-training model is trained to recover the sentiment information with three sentiment objectives (section 3.3). skep integrates different types of sentiment knowledge together and provides a unified sentiment representation for various sentiment analysis tasks. this is quite different from traditional sentiment analysis approaches, where different types of sentiment knowledge are often studied separately for specific sentiment tasks. to the best of our knowledge, this is the first work that has tackled sentiment-specific representation during pretraining. overall, our contributions are as follows: in particular, the pair prediction is converted into multi-label classification to capture the dependency between aspect and sentiment. ## skep: sentiment knowledge enhanced pre-training we propose skep, sentiment knowledge enhanced pre-training, which incorporates sentiment knowledge by self-supervised training. as shown in figure 1 , skep contains sentiment masking and sentiment pre-training objectives. sentiment masking (section 3.2) recognizes the sentiment information of an input sequence based on automaticallymined sentiment knowledge (section 3.1), and produces a corrupted version by removing this information. three sentiment pre-training objectives (section 3.3) require the transformer to recover the sentiment information for the corrupted version. formally, sentiment masking constructs a corrupted version x for an input sequence x guided by sentiment knowledge g. x i and x i denote the i-th token of x and x respectively. after masking, a parallel data ( x, x) is obtained. thus, the transformer encoder can be trained with sentiment pre-training objectives that are supervised by recovering sentiment information using the final states of encoder x 1 , ..., x n . ## fine-tuning for sentiment analysis we verify the effectiveness of skep on three typical sentiment analysis tasks: sentence-level sentiment classification, aspect-level sentiment classification, and opinion role labeling. on top of the pre-trained transformer encoder, an output layer is added to perform task-specific prediction. the neural network is then fine-tuned on task-specific labeled data. contextual text. these two parts are combined with a separator [sep], and fed into the transformer encoder. this task also utilizes the final state of the first token [cls] for classification. ## conclusion in this paper, we propose sentiment knowledge enhanced pre-training for sentiment analysis. sentiment masking and three sentiment pre-training objectives are designed to incorporate various types of knowledge for pre-training model. thought conceptually simple, skep is empirically highly effective. skep significantly outperforms strong pre-training baseline roberta, and achieves new state-of-the-art on most datasets of three typical specific sentiment analysis tasks. our work verifies the necessity of utilizing sentiment knowledge for pre-training models, and provides a unified senti-ment representation for a wide range of sentiment analysis tasks. in the future, we hope to apply skep on more sentiment analysis tasks, to further see the generalization of skep, and we are also interested in exploiting more types of sentiment knowledge and more fine-grained sentiment mining methods.