S em E val 2022 Task 10: Structured Sentiment Analysis (2022)

Abstract: In this paper, we introduce the first SemEval shared task on Structured Sentiment Analysis, for which participants are required to predict all sentiment graphs in a text, where a single sentiment graph is composed of a sentiment holder, target, expression and polarity. This new shared task includes two subtracks (monolingual and cross-lingual) with seven datasets available in five languages, namely Norwegian, Catalan, Basque, Spanish and English. Participants submitted their predictions on a held-out test set and were evaluated on Sentiment Graph F1 . Overall, the task received over 200 submissions from 32 participating teams. We present the results of the 15 teams that provided system descriptions and our own expanded analysis of the test predictions.

## introduction affective computing is a fundamental step towards enabling human computer interaction @xcite , as human communication is filled with affective content which conveys a speaker's private state, i.e. their current mood, their emotional experiences, or their attitude towards a certain object of conversation. along with emotion detection, sentiment analysis @xcite @xcite is an important stepping stone towards this goal. on a more practical level, being able to automatically determine what people think about an idea, product, or policy is of interest to companies, governments, and private citizens. in this paper, we describe the semeval-2022 shared task on structured sentiment analysis, which can be thought of as an information extraction problem in which one attempts to find all of the opinion tuples @xmath0, . . . , o n in a text. each opinion o i is a tuple (h, t, e, p) where h is a holder who expresses a polarity p towards a target t through a sentiment expression e, implicitly defining pairwise relationships between elements of the same tuple. @xcite argues that all of these elements are essential to fully resolve the sentiment analysis problem. although early annotation efforts in sentiment analysis annotated for fine-grained sentiment @xcite , most research on modeling sentiment focuses either on a variety of sub-tasks which avoid performing the full task, e.g. targeted @xcite , aspect-based @xcite , or end-2-end sentiment @xcite , or instead relies on simplified and idealized tasks, e.g. sentence-level binary polarity classification @xcite . we argue that the division of fine-grained sentiment into these sub-tasks has become counterproductive, as reported experiments are often not sensitive to whether a given addition to the pipeline improves the overall resolution of sentiment, nor do they take into account the inter-dependencies of the various sub-tasks. motivated by this, we present the semeval-2022 shared task on structured sentiment analysis, which jointly predicts all elements of an opinion tuple and their relations. some others give the new umuc 5 stars -don't believe them . positive negative holder target expression target expression esperientzia ezin hobea eduki genuen . figure 1 : a structured sentiment graph (shown in english and basque) is composed of a holder, target, sentiment expression, their relationships and a polarity attribute. holders and targets can be null. ## related work the conceptual roots of structured sentiment analysis can be found in early computational work on sentiment @xcite . much research in the field has been motivated by the corpus compiled by @xcite , who annotated english news wire documents with sentiment holders, targets, expressions, intensities, and other variables of interest. subsets of these variables have been detected with linear @xcite and neural models @xcite , but the full task has never been performed simultaneously. for instance, various semeval shared tasks on aspect-based sentiment analysis (absa) @xcite @xcite have focused on target extraction and polarity classification, and there have been research efforts to predict sentiment expressions as well @xcite . the models implemented for that purpose, however, neither resolve relations between expressions nor predict their polarity. the combination of targets, expressions and their polarity have been addressed for the recent task of aspect sentiment triplet extraction @xcite , but the resources that are used for this goal, which typically augment existing targeted datasets with polar expressions, suffer from a major limitation: they do not report annotation guidelines, procedures, or inter-annotator agreement, leaving the final quality of the data unclear. to solve these issues and integrate all such perspectives, @xcite proposed a holistic approach to sentiment. they cast the problem of structured sentiment as one of dependency parsing, they introduced specific metrics to evaluate automatic performance on this task, and developed a state-of-the-art structure-aware model. further improvements were reported by @xcite , who proposed a sparse fuzzy attention mechanism to deal with the sparseness of dependency arcs in the dependency models. ## task description the aim of this shared task is to predict all sentiment graphs in a text (see figure 1 ), where a graph includes the elements of an opinion tuple (h, t, e, p). we proposed two subtasks, corresponding to monolingual structured sentiment and crosslingual structured sentiment. participants were free to participate in one or both setups, and they had the opportunity to submit a single run on each dataset. ## data the task contained seven datasets in five languages: basque (multibooked eu ), catalan (multibooked ca ) @xcite , english, with data from opener en @xcite , multi-perspective question answer corpus @xcite and darmstadt universities corpus @xcite , ds unis ), norwegian @xcite , norec fine ) and spanish (opener es ) @xcite . we directly distributed the datasets in json format, with the exception of the last two corpora, as they have more restrictive licensing. for these, we provided links to the data such that participants could acknowledge their respective terms of use and include scripts to preprocess the data in a uniform manner. multibooked is a collection of hotel reviews in basque and catalan, written by users and collected from booking.com. the data was annotated for structured sentiment with polar expressions, targets, and holder labels, as well as polarity and intensity. each dataset contains around 1,500 sentences, making them the smallest datasets in the shared task. however, these sentences are more densely annotated than some of the larger corpora. for guidelines and inter-annotator agreement, see @xcite . opener contains an opinion mining corpus of hotel reviews for six languages (de, en, es, fr, it, nl) 1 . for the purposes of the shared task, we only used the english and spanish data @xcite . the reviews were extracted from different booking sites from november 2012 to november 2013. data collection was designed to ensure that different ratings and languages were included for a given hotel review. the annotations regard opinion expressions, their respective holders and targets, their polarity and opinion strength. the guidelines were based on the work by @xcite , which defines an opinion expression as a word (or combination of words) that expresses an attitude of the opinion 1 https://github.com/opener-project holder towards a target. the corpus also specifies the relations between holders, targets and opinion expressions in opinion triplets foot_0 . mpqa annotates english news wire texts with a complex set of annotation types, i.e. agent, expressive-subjectivity, direct-subjective, objectivespeech-event, attitude, and target. these types are also associated to a number of features and relations between one another. for the purpose of the shared task, we keep only the labels of agent, target, direct-subjective, as well as the polarity feature of direct-subjective, which respectively map to our holder, target, polar expression and polarity. we further normalize the polarities such that we have only positive, negative, and neutral. this is the second largest dataset, with a large number of holders, but relatively fewer targets and expressions. ds unis was initially published as part of the darmstadt service review corpus (dsrc) @xcite . the dsrc contains reviews of online universities and services annotated with sentiment on the sentence level and fine-grained sentiment on the expression level. the ds unis data that is part of the task contains only the university reviews, and discards the sentence-level annotations. @xcite distinguish between polar facts and opinions in their annotation scheme. in order to map the data to the format of the shared task, this distinction is resolved. norec fine annotates a subset of the norwegian review corpus @xcite for finegrained sentiment, i.e., including polar expressions, targets and holders, as well as their polarity and intensity. the corpus contains annotations for more than 11k sentences (both subjective sentences and fact-implied ones) taken from professional reviews from a number of different domains, such as screen, music, literature, products and games @xcite . further details on the annotation procedure, guidelines and inter-annotator agreement can be found in @xcite . ## evaluation the main metric for the task is sentiment graph f 1 (sf 1 ), which attempts to quantify how well a model captures the full sentiment graph (see figure 1 ). for sf 1 each sentiment graph is a tuple of (holder, target, expression, polarity). a true positive is defined as an exact match at graph-level, weighting the overlap between the predicted and gold spans for each element, averaged across all three h, t, e spans. we therefore allow some variability at the token-level (properly weighted), as long as the sentiment graph is predicted. for precision, we weight the number of correctly predicted tokens divided by the total number of predicted tokens (for recall, we divide instead by the number of gold tokens). correctly predicted tokens can also consist of empty holders and targets. ## baselines we provided participants with two strong baselines: 1) a dependency graph prediction model, and 2) a sequence-labeling pipeline. ## summary of participating systems in this section, we summarize the top three approaches and then further discuss some commonalities among the remaining teams.