SLPL -Sentiment at S em E val-2022 Task 10: Making Use of Pre-Trained Model’s Attention Values in Structured Sentiment Analysis (2022)

Abstract: Sentiment analysis is a useful problem which could serve a variety of fields from business intelligence to social studies and even health studies. Using SemEval 2022 Task 10 formulation of this problem and taking sequence labeling as our approach, we propose a model which learns the task by finetuning a pretrained transformer, introducing as few parameters (~150k) as possible and making use of precomputed attention values in the transformer. Our model improves shared task baselines on all task datasets.

## introduction sentiment analysis has many applications in different fields. from social and political studies @xcite to business intelligence and even health studies @xcite . semeval 2022 task 10 @xcite formulates this problem as to extract a graph of sentiment-related entities and names that, structured sentiment analysis. an example of such graph is depicted in figure2 specifically given an input sentence, one should extract a list of quadruples. each quadruple consists of a sentiment expression, a target expression, a holder expression and the polarity. semeval 2022 task 10 has two subtasks, monolingual and cross-lingual. we participated in the first track. shared task data is consisted of seven datasets in english, spanish, norwegian, catalan and basque. we trained and evaluated our model on each of these datasets. 1 2 related work structured sentiment analysis aims to extract sentiment, target and holder expressions along 1 our code is available at https: //github.com/sadra-barikbin/ novel-solutions-for-sentiment-analysis with their relations and also sentiment polarities. @xcite solves this problem by taking dependency graph parsing approach. it uses bil-stm and multilingual bert @xcite to encode input sentence and a neural dependency parser to jointly predict expressions and relations. every sentiment, target or holder expression in the sentence would become a star-shaped subtree in the output graph, with last token of expression as its root and its edges labelled with type of entity. relation between a sentiment expression and a target or holder expression is also represented with an edge from the former's root to that of the latter. this work evaluates its model on five datasets in four languages namely, norec fine @xcite , multibooked eu , multibooked ca @xcite , mpqa @xcite and darmstadt unis @xcite . its main evaluation metrics are targeted f1 and sentiment graph f1. first one measures exact prediction of target expression and polarity, and the second one measures exact match at graph level, weighting overlap between gold and predicted entities, averaged across three entity types. a less comprehensive formulation of sentiment analysis is end2end sentiment analysis which aims to predict target and sentiment expression along with polarity but does not consider relation between expressions. @xcite uses this formulation. taking sequence labelling as its approach, it predicts target expression together with sentiment expression by one module and polarity by another one. both modules along with a feature extraction module make use of a cnn to make their specific and shared latent vectors respectively. furthermore, polarity prediction module has a self-attention layer which gets predicted probability of tokens' being in sentiment expression from the other module and incorporates it in computing attention values so as to tokens with higher proba-bility get more attention. beside its main task, this work employs two auxiliary document-level tasks to enjoy the benefits of multi-task learning. this work also uses a mechanism called message passing in which, predicted probabilities of all modules is iteratively fed back into model to make the model like a rnn. for data, they use review datasets from @xcite and @xcite which are annotated with sentiment expressions by @xcite . among their evaluation metrics, f1-i is equivalent with targeted-f1 of @xcite . @xcite , similarly attempts to solve end2end sentiment analysis and takes sequence labelling approach. it introduces a layer called racl which consists of three separate but interconnected modules for sentiment and target extraction and polarity prediction. each module makes a module-specific representation of input using a cnn and information is exchanged among them through attention mechanism. the work uses a stack of racl layers as its model in which features in a layer is fed to upper layer and finally prediction is done by average pooling over prediction of all layers. this work has used same data and evaluation metrics as @xcite . finally @xcite solves targeted sentiment analysis which aims to extract only target expression and polarity. to this end, it has used bert @xcite along with a task specific layer. here bert makes a contextual and rich representation of input words and the task-specific layer performs sequence labelling. for this layer, different types such as fully connected, crf @xcite , gru and self-attention layer are examined that the last two outperformed the others. ## datasets and evaluation shared task is on seven datasets in five languages. we trained and evaluated our model on each one separately. general and detailed information of datasets is shown in table1 and table2 respectively. norec fine is norwegian professional reviews in multiple domains. multib ca , multib eu , opener en and opener es @xcite are hotel reviews in catalan, basque, english and spanish respectively. darmstadt unis consists of english online university reviews and finally mpqa contains annotated news articles in english. evaluation metric of the task is sentiment graph f1. prediction and gold answer of prob-lem are a list of quadroples @xmath0, q t , q h , q pol ) in which the first three entities are sets of tokens for sentiment, target, and holder expressions respectively. q p ol is polarity with value among n egative, n eutral, p ositive. match score of two given source and target quadroples namely, score(src, tgt) is as follows. as it can be seen, this is a weighted match over amount of overlap between entities, averaged across three entity types. polarities should be strictly equal. denominator |src e | is for score to be comparable with another one having different source quadrople. as target and holder expressions could be empty, |src e | is replaced with 1 in the case src e is empty. |src e ∩ tgt e | is also set to 1 when |src e | and |tgt e | are empty. given n input sentences, pred n as the list of predicted quadroples for n-th sentence and gold n as its gold counterpart, precision is computed as follows. our model is comprised of a base pre-trained model that is supposed to extract rich contextualized features and two separate modules for extracting expressions and predicting edges. for english datasets (opener en , darmstadt unis and mpqa) we used roberta base @xcite first five datasets are used in @xcite . max and average are measures of tokens' counts. multibooked ca and multibooked eu ), we used labse @xcite ) as model's base. we treat quadruples in a sentence as a graph which its nodes are expressions and its edges connect expressions within a quadruple. we name the module for extracting expression, node extractor and the module for determining edges, edge predictor. system structure is depicted in figure 1 . firstly, input sentence is fed into model base. then node extractor gets the encoded input and for each one of three entity types, predicts expressions in the sentence. polarities are also predicted by this module. edge predictor, then, gets the expressions and for every pair of sentiment-target and sentiment-holder expressions, predicts if there is an edge between them. finally a sentiment expression plus its polarity along with all of target and holder expressions connected thereto would become one of predicted quadroples. ## ablation study in order to assess and compare performance of different variants of our model, we trained each variant four times using different random seeds on each dataset and compared them on development data. results are depicted in table3.