L y S _ AC oruña at S em E val-2022 Task 10: Repurposing Off-the-Shelf Tools for Sentiment Analysis as Semantic Dependency Parsing (2022)

Abstract: This paper addressed the problem of structured sentiment analysis using a bi-affine semantic dependency parser, large pre-trained language models, and publicly available translation models. For the monolingual setup, we considered: (i) training on a single treebank, and (ii) relaxing the setup by training on treebanks coming from different languages that can be adequately processed by cross-lingual language models. For the zero-shot setup and a given target treebank, we relied on: (i) a word-level translation of available treebanks in other languages to get noisy, unlikely-grammatical, but annotated data (we release as much of it as licenses allow), and (ii) merging those translated treebanks to obtain training data. In the post-evaluation phase, we also trained cross-lingual models that simply merged all the English treebanks and did not use word-level translations, and yet obtained better results. According to the official results, we ranked 8th and 9th in the monolingual and cross-lingual setups.

## introduction sentiment analysis (sa, @xcite deals with the automatic processing of subjective information in natural language texts. early work on sa focused on conceptually simpler tasks, such as polarity classification at the sentence or document level. with the advances in natural language processing (nlp), more fine-grained and complex tasks have been proposed, such as detecting the entity that expresses an opinionated chunk of text, or the entity that was targeted. more particularly, @xcite consider sentiment analysis as a (graph) structured task, and discuss up to five subtasks: (i) sentiment expression extraction, (ii) sentiment target extraction, (iii) sentiment holder extraction, (iv) defining the relationship between these elements, and (v) assigning a polarity label. they discuss that although these tasks have been extensively studied by different authors @xcite @xcite , inter alia), they are not addressed all together. they also discuss that such subdivision into subtasks might have a negative impact in the general analysis of the sentence, and that a joint analysis could translate into a holistic approach. to do so, they propose to encapsulate all these tasks in the form of a sentiment graph. formally, the goal is to find the set of opinion tuples {o 1 , . . . , o i , . . . , o n } in a given text, where each opinion o i is a tuple of the form (h, t, e, p) where h is a holder who expresses a polarity p towards a target t through a sentiment expression e, implicitly defining pairwise relationships between elements of the same tuple. we illustrate an example in figure 1 . figure 1 : an example of sentiment graph as defined by @xcite . the sentence has a holder ('i'), two sentiment expressions ('got' and 'at no cost') and one target ('an upgrade to executive suite') more particularly, for the semeval-2022 task 10 ( @xcite , the organizers proposed both a monolingual foot_0 and a cross-lingual (zero-shot) setup. they considered 5 languages (and 7 treebanks): english, spanish, catalan, basque, and norwegian. for the zero-shot setup basque, catalan, and spanish were the target languages. our approach is based on the idea of viewing this task as semantic dependency parsing @xcite , since both tasks are structurally similar even if the graphs have different meaning. more specifically, we rely on a bi-affine graph-based parser @xcite and different large pre-trained language models (lm), such as bert @xcite , roberta @xcite or xlm-r @xcite . for the monolingual setup we train a semantic parsing model on single and merged treebanks, and compare the performance using different lms. for the cross-lingual setup, we first do a word-level translation of the datasets in a different language than the target treebank, and then proceed similarly to the monolingual setup. overall, the approach relies on off-the-shelf tools already available, but traditionally used for other purposes. we here repurpose them for their use for sentiment analysis as graph-based parsing. polarity classification. since early times, authors have studied the importance of language structure to deal with relevant linguistic phenomena for polarity classification, first focusing on simpler strategies such as the use of n-grams or lexical rules @xcite . later on, more complex syntactic structures were incorporated as well, both for rule-based and machine learning approaches. for instance, for the rule-based paradigm, @xcite used dependency relations for conceptlevel sentiment analysis, so sentiment could flow from one concept to another to better contextualize polarity. @xcite proposed a model to compute the sentiment of sentences that was driven by syntax-based rules to deal with specific relevant phenomena in sa, and that could be easily re-purposed for any language for which a dependency parser was available. @xcite built on top of vilares et al.'s idea, and proposed a multilingual syntax-based system that achieved a high precision for 17 languages. from the machine learning perspective, @xcite ; @xcite used dependency triplets to train data-driven (pre-neural) models and obtain slight improvements over purely lexical approaches. @xcite collected sentiment labels for phrases and sentences that were previously automatically represented as con-stituent (sub)trees, to then train a compositional model that used a recursive neural network. this work has some relevant resemblances with @xcite 's proposal for structured sentiment analysis. socher et al. were among the first to provide tree-shaped annotated sentiment data (in this case just for polarity classification), while most of previous work had focused on using tree knowledge as external information to the models, but with sentiment annotations only associated with plain texts. this publicly available data later encouraged many authors to design models that could exploit tree-shaped annotated data to obtain better performing models @xcite . ## brief overview of the shared task the goal of the task is to produce graph structures that reflect the sentiment of a sentence, as we showed in figure 1 . more particularly, the organizers released 7 treebanks in 5 different languages: opener @xcite , mpqa @xcite , darm-stadt_unis @xcite , multi-booked @xcite , and norec_fine @xcite . 2 the sentiment of a sentence is composed of all the opinions, o i , that make it up. each opinion can have up to four elements: a holder (h) who expresses a polarity (p) towards a target (t) through a sentiment expression (e). these four elements implicitly define the pairwise relationships between the elements of a tuple. the previous example, figure 1 , shows a sentence with two sentiment expressions (got and at no cost) that express the polarity (positive) of the sentiment that a holder (i) has towards one target (an upgrade to executive suite). preprocessing the organizers of the shared task proposed two possible ways to address the task: as a sequence labeling or as graph-based parsing problem. as mentioned above, we opted for the latter. we use the scripts available in the official repository to transform the json files to the conll-u based format and vice versa, and we applied the needed changes to make it compatible with supar 2 for more detailed information see https: //github.com/jerbarnes/semeval22_ structured_sentiment (see §4). 3 under the graph-based paradigm, the problem is approached as a bilexical dependency graph prediction task, with some assumptions. to convert the data, the organizers suggest two possible conversions, namely head-first and head-final. in head-first, it is assumed that the first token of the sentiment expression is a root node, and that the first token of each holder or target spans is the head node of such span, while the other ones are dependents. meanwhile, in head-final, the final token of the holder and target spans is set as the head of the span, and the final token of the sentiment expression as a root node (figure 1 is a head-final example). in this work, we have chosen head-final, which is the default option for the shared task and also delivered better results than head-first in the experiments carried out by @xcite (see subtasks more in detail, the challenge is divided into two subtasks: metrics each subtask is evaluated independently, and the ranking metric was sentiment graph f1 @xcite , where true positives are exact matches at the graph level, weighting the overlap between the predicted and gold spans for each element, and averaged across all three spans. to compute precision, it weights the number of correctly predicted tokens divided by the total number of predicted tokens, while for recall it weights the number of correctly predicted tokens divided by the total number of gold tokens. also, as mentioned earlier, it is possible to have tuples with empty holders and targets. ## our model we rely on the dozat and manning (2018) parser, a widely used state-of-the-art model both for syntactic and semantic dependency parsing. inspired in previous graph-based parsers @xcite , the parser first computes contextualized representations for each word using bidirectional lstms (bilstms; @xcite . after that, the model computes a head and a dependent representation for each term, to establish through a bi-affine attention whether an edge exists between each pair of tokens, and if so, what is the semantic relationship between them. in particular, in this paper we follow the implementation used in the supar 4 package, as it has been widely adopted by the community and it is available for other flavors of parsing as well, such as constituent or dependency parsing. we preferred this implementation over the graph-based baseline provided in the semeval repository, since early experiments showed a superior performance, and it also offered a simpler integration of large language models. we left the parser hyperparameters, except the learning rate, at their default value. pre-trained language models for each language we looked for avaliable monolingual and multilingual pre-trained lms at https:// huggingface.co/ . specifically, for each language, we included: ## results here, we detail and discuss the results that we got for both subtasks (see §5.1 and 5.2) on: (i) the official development sets, and (ii) the official test sets of the shared tasks. ## conclusion this paper describes our participation at the sem-eval shared task 10 on structured sentiment analysis. we participated both in the monolingual and cross-lingual (zero-shot) setups. we applied a simple, but effective approach, relying on off-theshelf tools, traditionally used for other purposes, and used them to predict sentiment graphs instead. more particularly, for the monolingual setup, we linked pre-trained language models with bi-affine graph parsing and training over single and multiple treebanks. in the zero-shot setup, we followed a similar approach, but relied on publicly available translation models to obtain training data, by applying a word-level translation of treebanks, to then train models similarly to the monolingual setup.