Structured Sentiment Analysis as Dependency Graph Parsing (2021)

Abstract: Structured sentiment analysis attempts to extract full opinion tuples from a text, but over time this task has been subdivided into smaller and smaller sub-tasks, e.g., target extraction or targeted polarity classification. We argue that this division has become counterproductive and propose a new unified framework to remedy the situation. We cast the structured sentiment problem as dependency graph parsing, where the nodes are spans of sentiment holders, targets and expressions, and the arcs are the relations between them. We perform experiments on five datasets in four languages (English, Norwegian, Basque, and Catalan) and show that this approach leads to strong improvements over state-of-the-art baselines. Our analysis shows that refining the sentiment graphs with syntactic dependency information further improves results.

## related work structured sentiment analysis can be broken down into five sub-tasks: i) sentiment expression extraction, ii) sentiment target extraction, iii) sentiment holder extraction, iv) defining the relationship between these elements, and v) assigning polarity. previous work on information extraction has used pipeline methods which first extract the holders, targets, and expressions (tasks iiii) and subsequently predict their relations (task iv), mostly on the mpqa dataset @xcite . crfs and a number of external resources (sentiment lexicons, dependency parsers, named-entity taggers) @xcite are strong baselines. given the small size of the training data and the complicated task, these techniques often still outperform neural models, such as bilstms @xcite . transition-based end-toend approaches have shown some potential @xcite . however, all of this work ignores the polarity classification subtask. targeted sentiment analysis only concentrates on extracting sentiment targets (task ii) and classifying the polarity directed towards them (task iv) @xcite . recent shared tasks on aspect-based sentiment analysis (absa) @xcite @xcite also include target extraction and polarity classification subtasks. joint approaches perform on par with pipeline methods @xcite and multitask models can perform even better @xcite . finally, pretrained language models @xcite can also lead to improvements on the absa data @xcite . end2end sentiment analysis is a recently proposed subtask which combines targeted sentiment (tasks ii and v) and sentiment expression extraction (task i), without requiring the resolution of relationships between targets and expressions. @xcite augment the absa datasets with sentiment expressions, but provide no details on the annotation process or any inter-annotator agreement. he et al. (2019 ) make use of this data and propose a multi-layer cnn (imn) to create hidden representations h which are then fed to a target and opinion extraction module (ae), which is also a multi-layer cnn. this module predicts ŷae , a sequence of bio tags foot_3 that predict the presence or absence of targets and expressions. after jointly predicting the targets and expressions, a second multi-layer cnn with a final self-attention network is used to classify the polarity, again as sequence labeling task (as). this second module combines the information from h and ŷae by incorporating the predicted probability of a token to be a target in the formulation of self-attention. finally, an iterative message-passing algorithm updates h using the predictions from all the modules at the previous timestep. @xcite instead propose relation-aware collaborative learning (racl). this model creates task specific representations by first embedding a sentence, passing through a shared feed-forward network and finally a task-specific cnn. this approach then models interactions between each pair of sub-tasks (target extraction, expression extraction, sentiment classification) by creating pairwise weighted attention representations. these are then concatenated and used to create the task-specific predictions. the authors finally stack several racl layers, using the output from the previous layer as input for the next. both models perform well on the augmented se-meval data, but it is unlikely that these annotations are adequate for full structured sentiment, as @xcite only provide expression annotations for sentences that have targets, generally only include sentiment-bearing words (not phrases), and do not specify the relationship between target and expression. finally, the recently proposed aspect sentiment triplet extraction @xcite ; ?) attempts to extract targets, expressions and their polarity. however, the datasets used are unlikely to be adequate, as they augment available targeted datasets, but do not report annotation guidelines, procedure, or inter-annotator agreement. graph parsing: syntactic dependency graphs are regularly used in applications, supplying them with necessary grammatical information @xcite @xcite @xcite . the dependency graph structures used in these systems are predominantly restricted to trees. while trees are sufficient to encode syntactic dependencies, they are not expressive enough to handle meaning representations, that require nodes to have multiple incoming arcs, or having no incoming arcs at all @xcite . while much of the early research on parsing these new structures @xcite focused on specialized decoding algorithms, @xcite presented a neural dependency parser that essentially relies only on its neural network structure to predict any type of dependency graph without restrictions to certain structures. using the parser's ability to learn arbitrary dependency graphs, @xcite phrased the task of negation resolution @xcite as a graph parsing task. this transformed the otherwise flat representations to dependency structures that directly encode the often overlapping relations between the building blocks of multiple negation instances at the same time. in a simpler fashion, @xcite exploit the parser of @xcite to predict spans of named entities. ## datasets we here focus on datasets that annotate the full task of structured sentiment as described initially. we perform experiments on five structured sentiment datasets in four languages, the statistics of which are shown in while all the datasets annotate holders, targets, and expressions, the frequency and distribution of these vary. regarding holders, mpqa has the most (2,054) and ds unis has the fewest (94), whereas norec fine has the largest proportion of targets (8,923) and expressions @xcite . the average length of holders (2.6 tokens) and targets (6.1 tokens) in mpqa is also considerably higher than the others. it is also worth pointing out that mpqa and ds unis additionally include neutral polarity. in the case of mpqa the neutral class refers to verbs which are subjective but do not convey polarity, e.g., 'say', 'opt for'. in ds unis , however, the neutral label tends to indicate expressions that could entail mixed polarity or are polar under the right conditions, e.g., 'the classes were not easy' is considered neutral, as it is possible for difficult classes to be desirable at a university. multib eu , and multib ca also have labels for strong positive and strong negative, which we map to positive and negative, respectively. finally, norec fine includes intensity annotations (strong, normal, slight), which we disregard for the purposes of these experiments. ## modeling this section describes how we define and encode sentiment graphs, detail the neural dependency graph models, as well as two state-of-the-art baselines for end-to-end sentiment analysis (target and expression extraction, plus polarity classification). ## evaluation as we are interested not only in extraction or classification, but rather in the full structured sentiment task, we propose metrics that capture the relations between all predicted elements, while enabling comparison with previous state-of-the-art models on different subtasks. the main metrics we use to rank models are targeted f 1 and sentiment graph f 1 . token-level f 1 for holders, targets, and expressions to easily compare our models to pipeline models, we evaluate how well these models are able to identify the elements of a sentiment graph with token-level f 1 . targeted f 1 this is a common metric in targeted sentiment analysis (also referred to as f 1 -i @xcite or absa f 1 (chen and qian, 2020)). a true positive requires the combination of exact extraction of the sentiment target, and the correct polarity. parsing graph metrics we additionally compute graph-level metrics to determine how well the models predict the unlabeled and labeled arcs of the parsing graphs: unlabeled f 1 (uf 1 ), labeled f 1 (lf 1 ). these measure the amount of (in)correctly predicted arcs and labels, as the harmonic mean of precision and recall @xcite . these metrics inform us of the local properties of the graph, and do not overly penalize a model if a few edges of a graph are incorrect. ## experiments all sentiment graph models use token-level mbert representations in addition to word2vec skip-gram embeddings openly available from the nlpl vector repository 8 @xcite . we train all models for 100 epochs and keep the model that performs best regarding lf 1 on the dev set (targeted f 1 for the baselines). we use default hyperparameters from @xcite (see appendix) and run all of our models five times with different random seeds and report the mean (standard deviation shown as well in regarding the graph metrics, the results depend highly on the dataset, with uf 1 and lf 1 ranging from 35.3/31.4 (ds unis head-first) to 66.8/62.1 (multib ca head-first). sentiment graph metrics nsf 1 and sf 1 have a similar, though slightly lower range (24.5/17.7 -62.0/56.8). the graph and sentiment graph metrics do not correlate perfectly, however, as uf 1 and lf 1 on mpqa are relatively good 8 nordic language processing laboratory vector repo.: http://vectors.nlpl.eu/repository/ . we used 300-dimensional embeddings trained on english wikipedia and gigaword for english (model id 18 in the repo.), and 100dimensional embeddings trained on the 2017 conll corpora for all others; basque (id 32), catalan (id 34), and norwegian bokmål (id 58). on average imn is the weakest baseline, followed by racl and then racl-bert. the main improvement that racl-bert gives over racl on these datasets is seen in the targeted metric, i.e., the contextualized representations improve the polarity classification more than the extraction task. the proposed graph-based models are consistently the best models across the metrics and datasets. regarding graph representations, the differences between head-first and head-final are generally quite small. head-first performs better on multib ca and slightly better on multib eu , while for the others (norec fine , mpqa, and ds unis ) head-final is better. this suggests that the main benefit is the joint prediction of all spans and relationships, and that the specific graph representation matters less. ## analysis in this section we perform a deeper analysis of the models in order to answer the research questions. ## conclusion in this paper, we have proposed a dependency graph parsing approach to structured sentiment analysis and shown that these models outperform state-of-the-art sequence labeling models on five benchmark datasets. using parse trees as input has shown promise for sentiment analysis in the past, either to guide a tree-based algorithm @xcite or to create features for sentiment models @xcite . however, to the authors' knowledge, this is the first attempt to directly predict dependencybased sentiment graphs. in the future, we would like to better exploit the similarities between dependency parsing and sentiment graph parsing, either by augmenting the token-level representations with contextualized vectors from their heads in a dependency tree @xcite or by multi-task learning to dependency parse. we would also like to explore different graph parsing approaches, e.g., perin @xcite . h o l d e r f 1 t a r g e t f 1 e x p f 1 t a r g e t e d f 1 u f 1 l f 1 n s f 1 s f 1 dep. labels dep. edges head-final+inlabel head-final head-first+inlabel head-first 4 2 0 2 4 figure 5: average benefit of each graph annotation scheme (y-axis) on the evaluation metrics (x-axis) for multib eu . h o l d e r f 1 t a r g e t f 1 e x p f 1 t a r g e t e d f 1 u f 1 l f 1 n s f 1 s f 1 dep. labels dep. edges head-final+inlabel head-final head-first+inlabel head-first 4 2 0 2 4 figure 6: average benefit of each graph annotation scheme (y-axis) on the evaluation metrics (x-axis) in percentage points for multib ca . h o l d e r f 1 t a r g e t f 1 e x p f 1 t a r g e t e d f 1 u f 1 l f 1 n s f 1 s f 1 dep. labels dep. edges head-final+inlabel head-final head-first+inlabel head-first 4 2 0 2 4 figure 8: average benefit of each graph annotation scheme (y-axis) on the evaluation metrics (x-axis in percentage points) for ds unis . h o l d e r f 1 t a r g e t f 1 e x p f 1 t a r g e t e d f 1 u f 1 l f 1 n s f 1 s f 1 dep. labels dep. edges head-final+inlabel head-final head-first+inlabel head-first 4 2 0 2 4 figure 9: average benefit of each graph annotation scheme (y-axis) on the evaluation metrics (xaxis) in percentage points. the results on norec fine , multib eu , multib ca .