KESA : A Knowledge Enhanced Approach To Sentiment Analysis (2022)

Abstract: Though some recent works focus on injecting sentiment knowledge into pre-trained language models, they usually design mask and reconstruction tasks in the post-training phase. This paper aims to integrate sentiment knowledge in the fine-tuning stage. To achieve this goal, we propose two sentiment-aware auxiliary tasks named sentiment word selection and conditional sentiment prediction and, correspondingly, integrate them into the objective of the downstream task. The first task learns to select the correct sentiment words from the given options. The second task predicts the overall sentiment polarity, with the sentiment polarity of the word given as prior knowledge. In addition, two label combination methods are investigated to unify multiple types of labels in each auxiliary task. Experimental results demonstrate that our approach consistently outperforms baselines (achieving a new state-of-the-art) and is complementary to existing sentiment-enhanced post-trained models.

## introduction sentence-level sentiment analysis aims to classify the overall sentiment of a sentence, which has received considerable attention in natural language processing @xcite @xcite . recently, pre-trained language models (plms) have achieved state-of-the-art (sota) performance on many natural language processing (nlp) tasks, including sentiment analysis. however, it is still challenging to integrate external knowledge into plms @xcite @xcite @xcite @xcite . recently, sentiment dictionary, a commonly used sentiment knowledge, has been injected into plms @xcite . a common practice is to post-train @xcite , i.e., continue pretraining, self-designed tasks on domain-specific corpora. these tasks include sentiment word prediction task, word sentiment prediction task, or aspect-sentiment pairs prediction @xcite @xcite @xcite @xcite , just to name a few. specifically, they are usually designed according to the paradigm of the mask language model (mlm), where sentiment words are first masked and then recovered (including their polarities) in the output layer. though effective, we argue that these methods can be further boosted by directly injecting sentiment knowledge, e.g., sentiment polarity, into the output layer when fine-tuning the downstream tasks. in this paper, we aim to inject sentiment knowledge into the fine-tuning phase directly, making it complementary to existing methods. for this aim, we propose two novel auxiliary tasks. the first task is sentiment word selection (sws), aiming to select the sentiment words that belong to the input from the given options, which comprises of k + 1 options (i.e., one ground-truth and k negative words). the second task is conditional sentiment prediction (csp), which pushes the model to predict the sentence polarity (i.e., sentiment), with the word (within the sentence) polarity given as prior information. it can be seen as a simplified main task (i.e., sentence-level sentiment analysis). different from existing sentiment polarity prediction task, csp treats the word sentiment (extracted from the sentiment dictionary) as prior information at the input end instead of as the ground-truth label at the output end. intuitively, this transformation can reduce the dependency on the quality of the sentiment dictionary. otherwise, though effective, its interpretability will be impaired. besides, since more than one type of label (e.g., sentence/word polarity label) is included, two label combination methods, i.e., the joint combination and the conditional combination, are therefore investigated. we are the first (earlier than @xcite ) to inject sentiment knowledge in the fine-tuning stage. our method starts by building the sentiment dictionary out of public resources and recognizing all the sentiment words in the input sentence. next, each auxiliary task is added to the task-specific (i.e., output) layer. finally, the auxiliary loss is added to the main loss to achieve the total loss. model pre/post-training tasks bert mlm and nsp albert sentence order prediction ernie knowledge mask sentence reordering bart token mask/deletion sentence permutation skep sentiment word prediction word polarity prediction aspect-sentiment pair prediction sentilare sentiment word prediction word polarity prediction pos label prediction joint prediction sentix sentiment word prediction word polarity prediction emotion prediction rating prediction kesa sentiment word selection conditional sentiment prediction we conduct experiments to demonstrate the further effectiveness of our proposed approach, and run ablation studies to verify the effectiveness of each auxiliary task. analysis studies are also performed to compare the impacts of hyper-parameters or modules. with kesa, the performance further outperforms the state-of-the-art by (0.76%, 0.75%) accuracy on mr and sst5, respectively. knowledge enhanced post-trained language models. external knowledge, including linguistic knowledge (e.g., part-of-speech, hyponym and syn-onym), factual knowledge (including items from wikidata @xcite , conceptnet @xcite and wikipedia) or domain-specific knowledge (e.g., sentiment polarity), can boost the generalization abilities of plms @xcite . several works have recently attempted injecting knowledge into plms, where the input format or model structure is modified, and knowledge-aware tasks are designed @xcite @xcite @xcite . for example, ernie 3.0 @xcite appends triples, e.g., (andersen, write, nightingale), ahead of the original input sentence, and designs tasks to predict the relation "write" in the triple. k-bert @xcite appends triples as branches to each entity (when fine-tuning downstream tasks) involved in the input sentence to form a sentence tree. k-adapter @xcite designs adapters and regards them as a plug-in with knowledge representations. these adapters are decoupled from the backbone plms and pre-trained from scratch by self-designed tasks, e.g., predicting relations in triples and labels of dependency parser. @xcite also considers adding two auxiliary training objectives when fine-tuning the dialogue generation task, including conjecturing the meaning of the masked entity and predicting its hypernym. different from ours, it is also designed according to the paradigm of mlm (i.e., masking entities and predicting their associated attributes in the knowledge base). knowledge enhanced post-trained language models for sentiment analysis. some domainspecific knowledge (including sentiment dictionary) is used for the sentiment analysis task. generally, these methods inject sentiment-related information into plms by designing sentimentaware tasks and then post-train them on large-scale domain-specific corpora @xcite @xcite . for example, skep @xcite designs sentiment word prediction, word polarity prediction, and aspect-sentiment pair prediction task to enhance plms with sentiment words. sentilare @xcite also designs sentiment word prediction, word polarity prediction, and joint prediction tasks. sen-tix @xcite designs sentiment word prediction, word polarity prediction, emoticon and rating prediction tasks. ## methodology figure 1 illustrates the framework of kesa. in order to integrate sentiment-related information when fine-tuning the downstream tasks, we propose two straightforward auxiliary tasks. the subsequent subsections will detail the two proposed auxiliary tasks (section 3.2 and 3.3), two label combination methods (including joint and conditional combination, section 3.4) and a weighted loss function (section 3.5) . for convenience, we first give some notations used in the following subsections. formally, @xmath0, l 2 , • • • , l m } denotes the sentiment dictionary with the size of m (i.e., including m sentiment words), and @xmath1, w 2 , • • • , w n } denotes an input sentence of length n . p @xmath2, respectively, where c is the sentence sentiment polarity label set, and z is the word sentiment label set. @xmath3, 1} denotes the ascription relationship label set between the word and the sentence, e.g., y w,@xmath4. d is the dimension of embeddings. ## experimental results in this section, we will detail the overall results, and the analysis of loss balance weight, label combination and introduced extra parameters. ## conclusion in this paper, we directly integrate sentiment knowledge into the fine-tuning phase. we design two sentiment-aware auxiliary tasks, sws and csp. sws needs to select the correct sentiment words from the given options, while csp predicts the overall sentiment with the word sentiment given as prior knowledge. further, we propose joint and conditional label combination methods to unify considered multiple kinds of labels into a single label. though straightforward and conceptually simple, experiments demonstrate that kesa still further improves over solid baselines, verifying that kesa is complementary to existing sentimentenhanced plms.