Effective Token Graph Modeling using a Novel Labeling Strategy for Structured Sentiment Analysis (2022)

Abstract: The state-of-the-art model for structured sentiment analysis casts the task as a dependency parsing problem, which has some limitations: (1) The label proportions for span prediction and span relation prediction are imbalanced. (2) The span lengths of sentiment tuple components may be very large in this task, which will further exacerbates the imbalance problem. (3) Two nodes in a dependency graph cannot have multiple arcs, therefore some overlapped sentiment tuples cannot be recognized. In this work, we propose nichetargeting solutions for these issues. First, we introduce a novel labeling strategy, which contains two sets of token pair labels, namely essential label set and whole label set. The essential label set consists of the basic labels for this task, which are relatively balanced and applied in the prediction layer. The whole label set includes rich labels to help our model capture various token relations, which are applied in the hidden layer to softly influence our model. Moreover, we also propose an effective model to well collaborate with our labeling strategy, which is equipped with the graph attention networks to iteratively refine token representations, and the adaptive multi-label classifier to dynamically predict multiple relations between token pairs. We perform extensive experiments on 5 benchmark datasets in four languages. Experimental results show that our model outperforms previous SOTA models by a large margin.

## introduction structured sentiment analysis (ssa), which aims to predict a structured sentiment graph as shown in figure 1 (a), can be formulated into the problem of tuple extraction, where a tuple (h, e, t, p) denotes a holder h who expressed an expression e towards a target t with a polarity p. ssa is a more challenging task, because other related tasks only focus the head-first parsing graph proposed by @xcite , where the arcs related to holder(target)expression linking relations are bold. (c) our proposed essential label set, which has more balanced label distribution for holder, target or expression span prediction and their linking relation prediction. on extracting part of tuple components or the text spans of the components are short. for example, opinion role labeling @xcite does not include the extraction of sentiment polarities, and aspect-based sentiment analysis (absa) @xcite extracts the aspect and opinion terms typically consisting of one or two words. the stateof-the-art ssa model is proposed by @xcite , which casts the ssa task as the dependency parsing problem and predicts all tuple components as a dependency graph (figure 1(b) ). however, their method exists some shortages. taking figure 1 @xcite as example, only 2 arcs (e.g., @xmath0.e., the relations between expressions and holders or targets), while much more other arcs are related to span prediction (e.g., @xmath1. such imbalanced labeling strat-moscow government has expressed the wish to import the mongolian meat span label span label span label rel label rel label rel label rel label span label span label @xcite moscow government has expressed the wish to import the mongolian meat egy will make the model pay more attention on span prediction but less on span relation prediction. furthermore, since the span lengths of sentiment tuple components may be very large in the ssa task, the label imbalanced problem will become more severe. besides, the dependency parsing graph is not able to deal with multi-label classification, since it does not allow multiple arcs to share the same head and dependent tokens. therefore, some overlapped sentiment tuples cannot be recognized. the statistics of span length and multi-label problems are listed in to alleviate the label imbalance problem in @xcite , we propose a novel labeling strategy that consists of two parts: first, we design a set of labels called essential label set (figure 1(c )), which can be considered as the basic label set for decoding ssa tuples, since it only includes the labels to tag the boundary tokens of spans. as seen, the proportion of span prediction labels and span relation prediction labels are relatively balanced, so that we can mitigate the label imbalance problem and meanwhile keep the basic ability of extracting sentiment tuples if the essential label set is learnt in the final prediction layer of our model. however, the labels related to recognize nonboundary tokens of ssa components are also important. for instance, they can encode the relations between the tokens inside the spans, which may benefit the extraction of the holders, expressions or targets with long text spans. to this end, we design another label set called whole label set (figure 2 ), which includes richer labels to fully utilize various information such as the relations among boundary tokens, non-boundary tokens, the tokens within a span, the tokens across different spans. moreover, since the dependency-based method @xcite only considers the local relation between each pair of tokens, we add the labels between [cls] and other tokens related to sentiment tuples into our whole label set, in order to utilize sentencelevel global information. considering that if the whole label set is directly applied on the output label for training, the label imbalance problem may occur again. we instead employ the whole label set in a soft and implicit fashion by applying it on the hidden layer of our model. to well collaborate with our labeling strategy, we also propose an effective token graph model, namely tgls (token graph with a novel labeling strategy), which uses rich features such as word, part-of-speech tags and characters as inputs and yields contextualized word representations by bilstm and multilingual bert @xcite . in the hidden layer, we build a multi-view token graph, which has four views corresponding to different relations in the whole label set and each view is a graph attention network @xcite with token representations as the nodes. in the prediction layer, we introduce a novel adaptive multi-label classifier to extract all the sentiment tuples no matter that they are overlapped or not. we conduct extensive experiments on five benchmarks, including norec fine @xcite , multib eu , multib ca @xcite , mpqa @xcite and ds unis @xcite . the resluts show that our tgls model outperforms the sota model by a large margin. in summary, our main contributions include: ## related work the task of the structured sentiment analysis (ssa) can be divided into sub-tasks such as span extraction of the holder, target and expression, relation prediction between these elements and assigning polarity. some existing works in opinion mining used pipeline methods to first extract spans and then the relations mostly on the mpqa dataset @xcite . for example, @xcite propose a bilstm-crf model which is the first such attempt using a deep learning approach, @xcite propose a transitionbased model which identifies opinion elements by the human-designed transition actions, and @xcite propose a unified span-based model to jointly extract the span and relations. however, all of these works ignore the polarity classification sub-task. in end2end aspect-based sentiment analysis (absa), there are also some attempts to unify several sub-tasks. for instance, @xcite augment the absa datasets with sentiment expressions, @xcite make use of this data and models the joint relations between several sub-tasks to learn common features, and (chen and qian, 2020) also exploit interactive information from each pair of sub-tasks (target extraction, expression extraction, sentiment classification). however, @xcite only annotate sentiment-bearing words not phrases and do not specify the relationship between target and expression, it therefore may not be adequate for full structured sentiment analysis. thus, @xcite propose a unified approach in which they formulate the structured sentiment analysis task into a dependency graph parsing task and jointly predicts all components of a sentiment graph. however, as aforementioned, this direct transformation may be problematic as it may introduce label imbalance in span and relation prediction. thus, we propose an effective graph model with a novel labeling strategy in which we employ a whole label set in the hidden layer to softly affect our model, and an essential label set in the prediction layer to address the imbalance issue. the design of our essential label set is inspired by the handshaking tagging scheme @xcite , which is a token pair tagging scheme for entity and relation extraction. the handshaking tagging scheme involves only the labels related to the boundary tokens and enables a one-stage joint extraction of spans and relations. in our work, we modify the handshaking tagging scheme to use it for ssa. furthermore, since the component span of this task is relatively long, only utilizing the boundary tokens cannot make full use of the annotation information, so we propose a new label set called whole label set, which together with essential label set constitutes our labeling strategy. ## methodology in this section, we formally present our proposed tgls model in detail (figure 3 ), which mainly consists of four parts, the encoder layer, the multiview token graph as the hidden layer, the adaptive multi-label classifier as the prediction layer and the hierarchical learning strategy to train our model. ## analysis in this section we perform a deeper analysis on the models in order to answer three research questions: 6.1 does our modeling strategy mitigate the label imbalance problem in span prediction and span relation prediction? experimental results in however, there lacks a metric to directly quantify the ability in relation extraction and it is still a worthy question to explore how much of the improvement comes from our new model and how much from our new labeling strategy? to answer the question, we replace our labels with the dependency-parsing-based labels in headfinal setting @xcite and experiment on all datasets in terms of a new relation prediction metric, where a true positive is defined as any span pair that overlaps the gold span pair and has the same relation. in this section, we experiment on five datasets to heuristically search for the appropriate value of hyperparameter α (cf. eq.( 13 )). figure 4 shows that all datasets achieve higher sf1 score with α between 0.1 and 0.5. we ended up fixing alpha to 0.25, since most datasets yield optimal results around this value. in addition, it is worth noting that when α is set to 0, which means that the whole labels are completely removed, the performance drops a lot, which once again proves the effectiveness of learning whole labels in the hidden layer. in this section, we experiment on norec fine to further explore whether whole labels contribute to long span identification. figure 5 (a) evaluates the expression f1 scores regarding to different expression lengths, we can find that whole labels helps most on those expressions with longer length. in figure 5 (b), we also report the sf1 scores regarding to different distances, that is, from the leftmost token in a tuple to the rightmost token, which shows a similar conclusion. ## conclusion in this paper, we propose a token graph model with a novel labeling strategy, consisting of the whole and essential label sets, to extract sentiment tuples for structured sentiment analysis. our model is capable of modeling both global and local token pair interactions by jointly predicting whole labels in the hidden layer and essential labels in the output layer. more importantly, our modeling strategy is able to alleviate the label imbalance problem when using token-graph-based approaches for ssa. experimental results show that our model overwhelmingly outperforms sota baselines and improves the performance of identifying the sentiment components with long spans. we believe that our labeling strategy and model can be well extended to other structured prediction tasks.