SPDB Innovation Lab at S em E val-2022 Task 10: A Novel End-to-End Structured Sentiment Analysis Model based on the ERNIE - M (2022)

Abstract: Sentiment analysis is a classical problem of natural language processing. SemEval 2022 sets a problem on the structured sentiment analysis in task 10, which is also a study-worthy topic in research area. In this paper, we propose a method which can predict structured sentiment information on multiple languages with limited data. The ERNIE-M pretrained language model is employed as a lingual feature extractor which works well on multiple language processing, followed by a graph parser as a opinion extractor. The method can predict structured sentiment information with high interpretability. We apply data augmentation as the given datasets are so small. Furthermore, we use K-fold cross-validation and DeBERTaV3 pretrained model as extra English embedding generator to train multiple models as our ensemble strategies. Experimental results show that the proposed model has considerable performance on both monolingual and cross-lingual tasks.

## introduction sentiment analysis @xcite is widely used in many aspects of computer science nowadays, such as human computer interaction, lingual feature extraction, etc. affective computing techniques enable us to explore the sentiment message, which depicts the preference, emotion or even idea of people, behind the sentence itself. sentiment analysis task can be classified into text level, sentence level, entity level and opinion tuple level. the goal of text or sentence level sentiment analysis is to predict sentiments of given documents or sentences. on the other hand, entity level sentiment analysis needs to consider sentiments between each entity in given sentence. furthermore, a sentence may contain multiple entities with different sentiments (positive or negative). therefore, in comparison to entity level sentiment analysis, opinion tuple level sentiment analysis requires additional extraction of relations between entities and opinions. structured sentiment analysis @xcite is a task to predict a sentiment graph for given sentences. it can be theoretically cast as an information extraction problem in which one attempts to find all of the opinion tuples @xmath0, ..., o n in a text. as we can see in figure 1 , each opinion o i is a tuple (h, t, e, p) where h is a holder who expresses a polarity p towards a target t through a sentiment expression e, implicitly defining pairwise relationships between elements of the same tuple. the structure of the paper is as follows. section 2 briefly reviews recent works on similar tasks; section 3 describes our model structure in detail. section 4 shows the analysis of the given datasets; section 5 introduces our experimental setting and results. and finally in section 6, we make a conclusion and give the ideas about future works. ## background dividing structured sentiment analysis into multiple subtasks is a traditional approach, by first identifying holders, targets and expressions through named entity recognition (ner) module, then predicting relations among the entities. @xcite are baselines with good performance. on the other hand, the end-to-end sentiment analysis is a straight-forward approach, which directly extracts target and expression without splitting them into sub-tasks. @xcite presents an interactive multi-task learning network(imn) implemented by a series of a multi-layer cnn modules. recently, @xcite cast the structured sentiment problem as the dependency graph parsing problem, and proposed a method that outperforms the sota(state-of-the-art) baselines on extensive experimental results. in this task, 7 small datasets involving 5 different languages are given, which means our method needs to support multiple languages with limited data. therefore, in this work, we use ernie-m pretrained model as word embedding generator, as it shows sota performances in various of nlp tasks in multiple languages. because of the outstanding performance of @xcite , we employ a similar network structure to extract the sentiment information. ## model structure the model structure is similar to the head-final model structure in @xcite , while we use a pretrained model in this work. a bert-style (ernie-m @xcite ) pretrained model takes sentences starting with "[cls]" token as input. we connect "[cls]" token with the last word of the expression, which is the root node of the tuple (h, t, e). the connection type is related to the sentiment polarity, "exp:pos" for positive polarity, "exp:neu" for neutral polarity, and "exp:neg" for negative corresponding to polarity, such that we are able to predict sentence polarity based on the connection type. as shown in figure 1 , a connection from "[cls]" to "stars" and a connection from "[cls]" to "believe" are established, with a connection type of "exp:pos" and "exp:neg" respectively. we describe the model structure in detail below. in this work, we use ernie-m pretrained model as a text feature extractor, which takes subword tokens as the model input. we apply subword-based tokenization on the input words. as shown in figure 2 , we apply subword-based tokenization on the input sentence, getting -→ @xmath1, t 2 , ..., t m ) for any t j (1 ≤ @xmath2. for instance, word "restful" will be split into "rest" and "##ful", where "##" indicates that the token is not the start of a word. and the process can be easily inversed by these special characters, such as we can restore ["great", "and", "rest", "##ful", "place", "to", "stay", "."] to its original status ["great", "and", "restful", "place", "to", "stay" "."]. after that, we input the subword tokens into the ernie-m model and get the embedding of the subword tokens. then, we apply average pooling on the subword embeddings( -→ v in equation 1) which belong to a same original word to get the word representation -→ @xmath3, c 2 , ..., c n ). after obtaining the word representations of the sentence, we perform a position-wise feed-forward networks to obtain the representation of the heads and dependents, where heads represent head nodes, and the dependents represent follower nodes. then we use bilinear attention network @xcite to calculate the pairwise correlation between each two words in the sentence. in equation 4, we obtain a score matrix that indicates the relationship between each word in the input sentence pair-wisely, by passing head and dependent embedding into a bilinear attention network, followed by a softmax layer. we apply crossentropy loss as the loss function during model training. figure 2 : the process of generating the word representations. ## data semeval-2022 task 10 provides a total of 7 datasets: norec fine(øvrelid et al., 2020), multibooked eu @xcite , multibooked ca @xcite , opener es @xcite , opener en @xcite , mpqa @xcite , darmstadt unis @xcite , involving different languages (en, ca, eu, no, es) and a couple of domains. multi-booked ca (catalan), multibooked eu (basque), opener en(english), and opener es(spanish) belong to hotel reviews domain. norec is a norwegian dataset in literature, movies, video games, restaurants, music and theater domains. darmstadt unis is an english dataset in university domain, and mpqa is an english dataset in news domain. our data analysis on the given datasets is shown in we can see that there are three different types of sentiment(positive, neutral, negative) in the opinions of darmstadt unis and mpqa, while others only contain positive and negative sentiments. in addition, we find that the number of expressions is always larger than or equal to the number of holders and targets for all datasets, which means there is at least one expression in each opinion. we can draw a conclusion that in a quadruple (h, t, e, p), both h and t may be missing, but e and p are not. therefore, the expression(more precisely, the last word of expression) is defined as the root node of a (h, t, e) triplet. ## conclusion in order to solve the issues of small dataset and cross language in semeval-2022 task 10, we introduce a multilingual pretrained model ernie-m as a lingual feature extractor to the given baseline model. furthermore, we use multiple strategies such as data augmentation, k-fold cross-validation, focal loss and ensemble to improve the model performance. in the future, we can take other techniques to do further optimization, such as different data augmentation technieques, fine-tuning the pretrained model with the in-domain dataset, and changing the label from word level to subword level to fit the subword representation of the pretrained model.