AMEX AI Labs at S em E val-2022 Task 10: Contextualized fine-tuning of BERT for Structured Sentiment Analysis (2022)

Abstract: We describe the work carried out by AMEX AI Labs on the structured sentiment analysis task at SemEval-2022. This task focuses on extracting fine grained information w.r.t. to source, target and polar expressions in a given text. We propose a BERT based encoder, which utilizes a novel concatenation mechanism for combining syntactic and pretrained embeddings with BERT embeddings. Our system achieved an average rank of 14/32 systems, based on the average scores across seven datasets for five languages provided for the monolingual task. The proposed BERT based approaches outperformed BiLSTM based approaches used for structured sentiment extraction problem. We provide an in-depth analysis based on our post submission analysis.

## introduction in this paper we present the work done by amex ai labs on the structured sentiment analysis monolingual task at semeval-2022 @xcite . structured sentiment analysis (ssa) focuses on performing fine grained analysis and extracting opinion tuples from a given input text @xcite . an example is presented in fig. 1 . ## background structured sentiment analysis (ssa) aims at capturing the natural hierarchical structure of different aspects, their corresponding sentiments in a sentence and the correlation between them. @xcite leveraged a hierarchical tree structure on the aspect terms, analyses the opinions on those aspect terms and connects them to the corresponding evidence for the same. @xcite identify two types of opinion-related entities -expressions of opinions and sources of opinions -along with the linking relation that exists between them. to identify the joint existence of opinion terms, they had utilized integer linear programming approach. over years, this task was recognized further as aspect based sentiment analysis which focuses more on the precise identification of aspects and their corresponding sentiments @xcite , @xcite . other intermediate works also focus on transition based end-to-end opinion extraction approaches @xcite which ignore the polarity classification subtask. target sentiment analysis, which is another modified form of ssa focuses on extracting only sentiment targets and classifying their polarity @xcite , @xcite . yang and cardie (2012) is a strong baseline which suggests that the use of conditional random fields could enhance the aspect terms prediction, given a smaller dataset with annotations, for training. during this time, some prominent research works discussed the advantage of semantic dependency parsing @xcite , @xcite which was leveraged for ssa task and for the establishment of the state-of-theart ssa model which was proposed by @xcite . this work formulates the ssa task as a dependency parsing problem and predicts all tuple components as a dependency graph. we aim to extend this aforementioned work and propose a modified architecture to improve the results across various language datasets. ## model overview for the ssa task, prior works have deployed separate models for detecting various elements (source, target and polar expression) @xcite . however, this results in information regarding prediction of individual elements not being passed on to other element predictions. figure 2 : sample sentence : head-first representation @xcite figure 3 : sample sentence : head-final representation @xcite for leveraging a joint learning technique, @xcite suggest two sets of approaches, viz., head-first and head-final representation. headfirst considers spans of opinion elements using the first token of each element, i.e., head token first, as shown in fig. 2 . head-final considers the last token, i.e., head token last, as shown in fig. 3 . our proposed architecture was inspired by this joint learning technique and is elaborated in section 4. ssa task involves prediction of opinion elements along with the relationship between them, referred to as arc prediction. in this paper, we provide a comparison between two methods -vbert, which uses only bert embeddings and cbert, which uses bert embeddings along with concatenation of head and dependent bilstm blocks, detailed in section 4. a comparison of results from these two approaches are provided in section 6. ## experimental results and analysis the detailed results of final submission are as shown in our system achieved an average rank of 14/32 systems, based on the average scores across seven datasets. an in-depth analysis reveals that although following is a qualitative analysis to identify main challenges in the cbert system. a detailed set of examples are shown in ## conclusion and future scope this paper discusses the proposed system and experimental results on ssa task at semeval-2022. the proposed model called cbert, assumes a joint learning technique using bert-base multilingual model embeddings along with contextualized embeddings created using pretrained stanza vectors and pretrained elmo embeddings. the arc prediction between opinion term elements are achieved using an attention layer. the proposed cbert approach performed relatively better than bilstm approaches and vbert approach. we find that using bert based encoding, along with concatenation of contextualized information with syntactic information is more effective for ssa task. for improving cbert, leveraging 2d crfs @xcite can be explored for better span detection over longer texts. different joint learning architecture on top of cbert's encoding like sdrn @xcite ) can be leveraged. approaches to fine-tune models for specific tasks @xcite is also a potential area for exploration.