S enti LARE : Sentiment-Aware Language Representation Learning with Linguistic Knowledge (2020)

Abstract: Most of the existing pre-trained language representation models neglect to consider the linguistic knowledge of texts, which can promote language understanding in NLP tasks. To benefit the downstream tasks in sentiment analysis, we propose a novel language representation model called SentiLARE, which introduces word-level linguistic knowledge including part-of-speech tag and sentiment polarity (inferred from SentiWordNet) into pre-trained models. We first propose a context-aware sentiment attention mechanism to acquire the sentiment polarity of each word with its part-of-speech tag by querying SentiWordNet. Then, we devise a new pre-training task called label-aware masked language model to construct knowledge-aware language representation. Experiments show that SentiLARE obtains new state-of-the-art performance on a variety of sentiment analysis tasks.

## introduction recently, pre-trained language representation models such as gpt @xcite , elmo @xcite , and bert @xcite have achieved promising results in nlp tasks, including sentiment analysis @xcite @xcite . these models capture contextual information from large-scale corpora via well-designed pre-training tasks. the literature has commonly reported that pre-trained models can be used as effective feature extractors and achieve state-of-the-art performance on various downstream tasks @xcite . despite the great success of pre-trained models, existing pre-training tasks like masked language model and next sentence prediction @xcite neglect to consider the linguistic knowledge. such knowledge is important for some nlp tasks, particularly for sentiment analysis. for instance, existing work has shown that linguistic knowledge including part-of-speech tag @xcite and word-level sentiment polarity @xcite is closely related to the sentiment of longer texts. we argue that pre-trained models enriched with the linguistic knowledge of words will facilitate the understanding of the sentiment of the whole texts, thereby resulting in better performance on sentiment analysis. there are two major challenges to construct knowledge-aware pre-trained language representation models which can promote the downstream tasks in sentiment analysis: 1) knowledge acquisition across different contexts. most of the existing work has adopted static sentiment lexicons as linguistic resource @xcite , and equipped each word with a fixed sentiment polarity across different contexts. however, the same word may play different sentiment roles in different contexts due to the variety of part-ofspeech tags and word senses. 2) knowledge integration into pre-trained models. since the introduced word-level linguistic knowledge can only reflect the local sentiment role played by each word, it is important to deeply integrate knowledge into pre-trained models to construct sentence-level language representation, which can derive the global sentiment label of a whole sentence from local information. how to build the connection between sentence-level language representation and wordlevel linguistic knowledge is underexplored. in this paper, we propose a novel pre-trained language representation model called sentilare to deal with these challenges. first, to acquire the linguistic knowledge of each word, we label the word with its part-of-speech tag, and obtain the sentiment polarity via a context-aware sentiment attention mechanism over all the matched senses in sentiwordnet @xcite . then, to incorporate linguistic knowledge into pre-trained models, we devise a novel pre-training task called label-aware masked language model. this task involves two sub-tasks: 1) predicting the word, partof-speech tag, and sentiment polarity at masked positions given the sentence-level sentiment label; 2) predicting the sentence-level label, the masked word and its linguistic knowledge including part-ofspeech tag and sentiment polarity simultaneously. we call the first sub-task early fusion since the sentiment labels are aforehand integrated as input embeddings, whereas in the second sub-task, the labels are used as late supervision to the model in the output layer. these two sub-tasks are expected to establish the connection between sentence-level representation and word-level linguistic knowledge, which can benefit downstream tasks in sentiment analysis. our contributions are in three folds: ## for restaurant domain (res14) and semeval2016 task 5 for restaurant domain (res16) @xcite . the statistics of these datasets are reported in we present the results of aspect-level sentiment analysis in ## conclusion we present a novel pre-trained model called senti-lare for sentiment analysis, which introduces linguistic knowledge from sentiwordnet via contextaware sentiment attention, and adopts label-aware masked language model to deeply integrate knowledge into bert-style models through pre-training tasks. experiments show that sentilare outperforms state-of-the-art language representation models on various sentiment analysis tasks, and thus facilitates sentiment understanding.