Text Generation Model Enhanced with Semantic Information in Aspect Category Sentiment Analysis (2023)

Abstract: Aspect Category Sentiment Analysis (ACSA) is one of the main subtasks of sentiment analysis, which aims at predicting polarity over a given aspect category. Recently, generative methods emerge as an efficient way to utilize a pre-trained language model for solving ACSA. However, those methods fail to model relations of target words and opinion words in a sentence including multiple aspects. To tackle this problem, this paper proposes a method to incorporate Abstract Meaning Representation (AMR), which describes semantic representation of a sentence as a directed graph, into a text generation model. Furthermore, two regularizers are designed to guide cross attention weights allocation over AMR graphs. One is the identical regularizer that constrains attention weights of aligned nodes, the other is the entropy regularizer that helps the decoder generate tokens by heavily considering only a few related nodes in the AMR graph. Experimental results on three datasets show that the proposed method outperforms state-of-the-art methods, proving the effectiveness of our model.

## introduction aspect based sentiment analysis is an important task, which analyzes sentiments regarding an aspect of a product or a service. this task includes many subtasks, such as aspect category detection (acd) and aspect category sentiment analysis (acsa). acd is the task of detecting aspect categories while acsa concentrates on predicting the polarity of given aspect categories. this study focuses on acsa only. figure 1 shows an example of acsa task, where negative and positive are the polarities of the two provided categories service and food. the conventional approaches carry out acsa as a classification task. @xcite and @xcite use an attention mechanism for discovering aspect-related words. to achieve better representations, pre-trained language models such as bert @xcite are used @xcite . although achieving competitive results, the fine-tuning of a pre-trained language model for acsa suffers from two drawbacks: the difference between fine-tuning and pre-training tasks and the gap between newly initialized classification layers and the pre-trained model. such inconsistency is often harmful to the training of an outstanding classifier for acsa. to solve the above problems, @xcite propose to transform the sentiment classification task into text generation, which better leverages the power of pre-trained language models following the seq2seq framework like bart @xcite . however, the naive text generation method cannot fully capture relations between opinion words and target words in sentences containing multiple aspects. abstract meaning representation (amr, @xcite , which is a semantic representation of a sentence in the form of rooted, labeled, directed and acyclic graphs, can model the relations between the target words and associated opinion words. for example, in figure 1 , the relation be-tween "staff " and "rude" is captured by a directed edge between two nodes "staff " and "rude-01". in addition, the amr graph also provides the high level semantic information, which means words with the same meaning, like "but", "although" and "nevertheless", could be represented by the same node "contrast-01". this paper investigates the potential of combining amr with the naive text generation model to perform acsa. furthermore, we also design two regularizers for guiding cross attentions over the amr graph of the decoder. we observe that words (in a sentence) and nodes (in an amr graph) that are semantically similar should be paid the same amounts of attention. therefore, we minimize the difference of two cross attentions over aligned words and amr nodes. moreover, the decoded tokens should only be attentive to related amr nodes. to achieve that, we minimize the entropy of the cross attentions over the amr graph in the decoder layers. we evaluate our model using the rest14, rest14hard and mams benchmark datasets. the results show that our model is better than the baselines and achieves a state-of-the-art performance. our contributions can be summarized as follows: aspect category sentiment analysis numerous attempts have been made to improve acsa. wang et al. (2016) propose an lstm-based model combined with an attention mechanism to attend for suitable words with given aspects. ruder et al. (2016) capture inter-sentence relations within a review using a hierarchical bidirectional lstm model. xue and li (2018) extract features using cnn and output features related to the categories of aspects by using a gated mechanism. xing et al. (2019), liang et al. (2019) and zhu et al. (2019) incorporate aspect category information into a sentence decoder for generating representations specific to both the aspect and its context. sun et al. (2019) construct auxiliary sentences from aspects for performing acsa as a sentence-pair classification task. jiang et al. (2019) propose a new capsule network which captures relationship between the aspects and the contexts. li et al. (2020b) aggregate the sentiments of the words that indicate an aspect category, so as to predict the sentiment of this category. liu et al. (2021) use a template-based method to perform acsa as a generation task; this can leverage the knowledge of a pre-trained language model. shan et al. (2022) use additional syntactic information to enhance the sentiment features. liu et al. (2023) utilize commonsense knowledge graph and data augmentation to overcome the shortage of training data. to avoid error propagation, joint models which perform acsa and acd simultaneously have been proposed. @xcite propose two models with lstm and cnn, which output an aspect category and its corresponding polarity at the same time. @xcite apply orthogonal and sparseness constraints on attention weights. @xcite design an as-capsules model to explore the correlation of aspects with sentiments through share modules. @xcite propose a joint model with a shared sentiment prediction layer. amr with the development of amr parsers, amr-to-text generation models and larger parallel datasets, amr has been applied successfully to many downstream text generation tasks. for example, it has been integrated into a machine translation model as additional information for the source side @xcite @xcite . in text summarization, several researchers transform amr representations of sentences into an amr graph of a summary and generate a text summary from the extracted subgraph @xcite @xcite . however, as explained in section 1, there has been no attempt to apply amr to acsa. ## text generation model for acsa this section presents an overview of the text generation model for acsa proposed by @xcite , since our proposed model is based on it. the model follows the seq2seq framework that converts a review sentence to a target sentence indicating the polarity of the aspect. it fine-tunes the pre-trained bart model for the text generation task. the model takes a review sentence @xmath0, x 2 , ..., x |x| } = x 1:|x| as an input and generates a target sentence @xmath1, y 2 , ..., y |y | } = y 1:|y | , where |x| and |y | are the number of tokens in the source and target sentence respectively. ## analysis 6.1 case study to further examine how the semantic information of amr and two regularizers work well in acsa, a few examples are shown as a case study. and o represent the positive, negative and neutral class respectively. the first example, "i never had an orange donut before so i gave it a shot", has no explicit sentiment expression. with the help of semantic information and two regularizers, our model can correctly predict the true label while bart generation cannot. the second and third examples contain multiple aspects, which can affect each other's predictions. in the second example, the bart generation model may capture the positive sentiment toward the aspect word "atmosphere" for anticipating the sentiment of the different aspect "service', which leads to outputting the wrong label. another incorrect prediction by this baseline is shown in the third example, where the polarities of "food" and "staff " are mistakenly swapped. in contrast, our model pays attention to only the aspect-related amr nodes, resulting in the correct predictions in both examples. however, our model also faces a difficulty in some cases. in the last example, it wrongly predicts the sentiment polarity for "miscellaneous" because it is really hard to capture aspect-related amr nodes for a coarse-grained aspect class like "miscellaneous". ## conclusions in this paper we proposed a model which integrated the semantic information from the abstract meaning representation (amr) and the text generation method for the task of aspect category sentiment analysis (acsa). moreover, to more precisely cap-sentence aspect category bart generation our model label i never had an orange donut before so i {food} (p) (o) (o) the atmosphere was wonderful, however {ambiance, service, food} (p, p, n) (p, n, n) (p, n, n) the service and food were not. there are several specials that change {food, staff} (o, p) (p, o) (p, o) daily, which the servers recite from memory. the place was busy and had a bohemian feel. {place, miscellaneous} (p, p) (n, p) (n, o)