Exploring Graph Pre-training for Aspect-based Sentiment Analysis (2023)

Abstract: Existing studies tend to extract the sentiment elements in a generative manner in order to avoid complex modeling. Despite their effectiveness, they ignore importance of the relationships between sentiment elements that could be crucial, making the large pre-trained generative models sub-optimal for modeling sentiment knowledge. Therefore, we introduce two pre-training paradigms to improve the generation model by exploring graph pre-training that targeting to strengthen the model in capturing the elements’ relationships. Specifically, We first employ an Element-level Graph Pre-training paradigm, which is designed to improve the structure awareness of the generative model. Then, we design a Task Decomposition Pre-training paradigm to make the generative model generalizable and robust against various irregular sentiment quadruples. Extensive experiments show the superiority of our proposed method, validate the correctness of our motivation.

## introduction aspect-based sentiment analysis (absa) has drawn increasing attention in the community, which includes four fine-grained elements: aspect term, opinion term, aspect category, and opinion polarity. the first two terms exist as a raw text span in the review sentence while the remaining two are the classification result of aspect and opinion respectively. each four mapped sentiment elements form an aspect-level sentiment quadruple. for instance, for the given review "the apps are hard to use.", the corresponding quadruple is (apps, hard, software, negative). the joint extraction of quadruples is the most complex and challenging subtask among all the absa tasks, previous work usually formulate it as either sequence-level @xcite @xcite or token-level classification problems @xcite in joint learning or pipeline manner. however, these methods not only require sophisticated and complex modeling of sentiment elements but also suffer severely from error propagation since the overall prediction performance hinges on the accuracy of every step @xcite . more recently, studies tend to tackle the absa problem with a unified generative approach @xcite @xcite . they organize the target sequence in different approaches, namely listing @xcite : "(apps, hard, software, negative)", indexing @xcite : "(1,1,3,3)", paraphrasing @xcite : "(software is good because apps are hard)" or opinion tree @xcite : " @xcite ,( opinion ( negative, hard )))))". however, they ignore the importance of the relationships among elements (e.g. sentiment polarity should be identified based on opinion words, like great identifies a positive polarity and disappointing identifies a negative polarity). ## related work there are four aspect-level sentiment elements in absa, the various combination of these elements form the numerous sub-tasks of absa. the researches on absa generally follow a route from handling single sub-task to complex compositions of them. the starting point usually locates in the prediction of a single sentiment element, which is the target of fundamental sub-tasks, such as extracting the aspect term @xcite @xcite , classifing the aspect category mentioned in the sentence @xcite , and detecting the sentiment polarity for a given aspect @xcite @xcite @xcite . since the sentiment elements are naturally correlated, many studies further focus on exploring the co-extraction of sentiment elements, including aspect and opinion term extraction @xcite ; aspect term extraction and its polarity detection @xcite ; aspect category and polarity detection @xcite . furthermore, recent studies also employed end-toend models to extract all the sentiment elements in triplet or quadruple format @xcite @xcite @xcite . more recently, studies tend to design a unified framework to extract quadruples at one stop with pre-trained encoder-decoder language models, achieving great improvements in absa @xcite . the target sequence of them is formed by either class index @xcite or the desired sentiment element @xcite . otg @xcite addressed the importance of semantic correlations among sentiment elements, proposed a sentiment tree structure called opinion tree, and employed generative model to extract the linearized tree. however, the generative model is pre-trained to solve textual sequence tasks(e.g. masked language model) but finetuned for structure generation, between which exists a huge gap, making generative models sub-optimal for modeling structural knowledge. different from previous studies, we introduce two pre-training paradigms for opinion tree generation without treating it as a plain graph. to our knowledge, we are the first to consider designing methods depending on the intrinsic characteristics of the opinion tree. ## opinion tree generation model in this section, we introduce the basic opinion tree generation model we employed to generate in the pre-train and finetune phases, along with the objective functions and training. ## pre-training paradigms in this study, we introduce two pre-training paradigms for opinion tree generation. as shown in figure 2 , the two paradigms and finetune task share the same input format with a joint input of prompt, encoded text and tree, each method consists of a set of subtasks focus on respective training targets. the combination of subtasks forms the joint pre-training in our work, we will introduce the paradigms first in this section. ## experiments in this section, we introduce the datasets used for evaluation and the baseline methods employed for comparison. we then report the experimental results conducted from different perspectives, and analyze the effectiveness of the proposed model with different factors. ## analysis and discussion in this section, we first give some analysis and discussion to show the influence of element-level graph pre-training (egp) and task-level graph pre-training (tgp) paradigms. after that, we will investigate our search over masking rate, the influence of pre-training subtasks. ## conclusion in this study, we propose two novel pre-train paradigms for opinion tree generation, which are designed depending on the intrinsic characteristics of the opinion tree. specifically, the element-level graph pre-training paradigm abandons the strategy of capturing the complex structure but focuses directly on the core elements. while the task-level graph pre-training explicitly focuses on improving the generalization and robustness against irregular quadruples with an easy-to-hard routine. furthermore, we explore a dynamic masking rate and a cyclical train method for jointly combining the pre-training paradigms in order to bridge the gap between the pre-training and finetuning phases in modeling structural knowledge. experimental results show that our proposed model can achieve state-of-the-art performance in absa. in addition, the results also validate that, for target graphs that contain certain knowledge like opinion tree, the improving strategy should be made based on the intrinsic characteristics of the structure instead of treating it as a plain graph.