GRACE : Gradient Harmonized and Cascaded Labeling for Aspect-based Sentiment Analysis (2020)

Abstract: In this paper, we focus on the imbalance issue, which is rarely studied in aspect term extraction and aspect sentiment classification when regarding them as sequence labeling tasks. Besides, previous works usually ignore the interaction between aspect terms when labeling polarities. We propose a GRadient hArmonized and CascadEd labeling model (GRACE) to solve these problems. Specifically, a cascaded labeling module is developed to enhance the interchange between aspect terms and improve the attention of sentiment tokens when labeling sentiment polarities. The polarities sequence is designed to depend on the generated aspect terms labels. To alleviate the imbalance issue, we extend the gradient harmonized mechanism used in object detection to the aspect-based sentiment analysis by adjusting the weight of each label dynamically. The proposed GRACE adopts a post-pretraining BERT as its backbone. Experimental results demonstrate that the proposed model achieves consistency improvement on multiple benchmark datasets and generates state-of-the-art results.

## introduction aspect terms extraction (ate) and aspect sentiment classification (asc) are two fundamental, fine-grained subtasks in aspect-based sentiment analysis (absa). ate is the task of extracting the aspect terms (or attributes) of an entity upon which opinions have been expressed, and asc is the task of identifying the polarities expressed on these extracted terms in the opinion text @xcite . consider the example in figure 1 , which contains comments that people expressed about the aspect terms "operating system" and "keyboard", and their polarities are all positive. (a) label statistics (b) gradient statistics figure 2: label statistics and gradient distribution on the laptop dataset of semeval-14. the y-axis in (b) uses a log scale. to better satisfy the practical applications, the aspect term-polarity co-extraction, which solves ate and asc simultaneously, receives much attention in recent years @xcite @xcite . a big challenge of the aspect term-polarity co-extraction in a unified model is that ate and asc belong to different tasks: ate is usually a sequence labeling task, and asc is usually a classification task. previous works usually transform the asc task into sequence labeling. thus the ate and asc have the same formulation. there are two approaches of sequence labeling on the aspect term-polarity co-extraction. as shown in figure 1 , one is the joint approach, and the other is the collapsed approach. the preceding one jointly labels each sentence with two different tag sets: aspect term tags and polarity tags. the subsequent one uses collapsed labels as the tags set, e.g., "b-pos" and "i-pos", in which each tag indicates the aspect term boundary and its polarity. except for the joint and collapsed approaches, a pipelined approach first labels the given sentence using aspect term tags, e.g., "b" and "i" (the beginning and inside of an aspect term), and then feeds the aspect terms into a classifier to obtain their corresponding polarities. several related works have been published in these approaches. @xcite and @xcite found that the joint and collapsed approaches are superior to the pipelined approach on named entities and their sentiments co-extraction. @xcite proposed a unified model with the collapsed approach to do aspect term-polarity co-extraction. @xcite solved this task with a pipelined approach. @xcite adopted the joint approach to do such a co-extraction. we follow the joint approach in this paper, and believe that it has a more apparent of responsibilities than the collapsed approach through learning parallel sequence labels. however, previous works on the joint approach usually ignore the interaction between aspect terms when labeling polarities. such an interaction is useful in identifying the polarity. as an instance, in figure 1 , if "operating system" is positive, "keyboard" should be positive due to these two aspect terms are connected by coordinating conjunction "and". besides, almost all of previous works do not concern the imbalance of labels in such sequence labeling tasks. as shown in 2a, the number of 'o' labels is much larger than that of 'b' and 'i', which tends to dominant the training loss. moreover, we find the same gradient phenomenon as @xcite in the sequence labeling task. as shown in figure 2b , most of the labels own low gradients, which have a significant impact on the global gradient due to their large number. considering the above issues, we propose a gradient harmonized and cascaded labeling model (grace) in this paper. the proposed grace is shown in figure 3 . unlike previous works, grace is a cascaded labeling model, which uses the generated aspect term labels to enhance the polarity labeling in a unified framework. specifically, we use two encoder modules shared with lower layers to extract representation. one encoder module is for ate, and the other is for asc after giving the aspect term labels generated by the preceding encoder. thus, the grace could consider the interaction between aspect terms in the asc module through a stacked multi-head attention @xcite . besides, we extend a gradient harmonized loss to address the imbalance labels in the model training phase. our contributions are summarized as follows: in the following, we describe the proposed framework grace in section 2. the experiments are conducted in section 3, followed by the related work in section 4. finally, we conclude the paper in section 5. ## model an overview of grace is given in figure 3 . it is comprised of two modules with the shared shallow layers: one is for ate, and the other is for asc. we will first formulate the co-extraction problem and then describe the framework in detail in this section. ## related work aspect term extraction and aspect sentiment classification are two major topics of aspect-based sentiment analysis. many researchers have studied each of them for a long time. for the ate task, unsupervised methods such as frequent pattern mining @xcite , rule-based approach @xcite , topic modeling @xcite , and supervised methods such as sequence labeling based models @xcite @xcite @xcite are two main directions. for the asc task, the relation or position between the aspect terms and the surrounding context words are usually used @xcite . besides, there are some other approaches, such as convolution neural networks @xcite , attention networks @xcite @xcite , memory networks @xcite , capsule network @xcite , and graph neural networks @xcite . we regard ate and asc as two parallel sequence labeling tasks in this paper. compared with the separate methods, this approach can concisely generate all aspect term-polarity pairs of input sentences. like our work, @xcite and @xcite are also about performing two sequence labeling tasks, but they extract named entities and their sentiment classes jointly. we have a different objective and utilize a different model. @xcite , @xcite and @xcite have the same objective as us. the main difference is that their approaches belong to a collapsed approach, but ours is a joint approach. @xcite use joint approach like ours, they fo-cus on the interaction between two tasks, and some extra objectives are designed to assist the extraction. @xcite consider the ate as a span extraction question, and extract aspect term and its sentiment polarity using a pipeline approach. there are some other approaches to address these two tasks @xcite @xcite @xcite @xcite . however, almost all of previous models do not concern the imbalance of labels in such sequence labeling tasks. to the best of our knowledge, this is the first work to alleviate the imbalance issue in the absa. ## conclusion in this paper, we proposed a novel framework grace to solve aspect term extraction and aspect sentiment classification simultaneously. the proposed framework adopted a cascaded labeling approach to enhance the interaction between aspect terms and improve the attention of sentiment tokens for each term by a multi-head attention architecture. besides, we alleviated the imbalance issue of labels in our labeling tasks by a gradient harmonized method borrowed from object detection. the virtual adversarial training and post-training on domain datasets were also introduced to improve the extraction performance. experimental results on three benchmark datasets verified the effectiveness of grace and showed that it significantly outperforms the baselines on aspect term-polarity co-extraction.