Improving Aspect-Based Sentiment with End-to-End Semantic Role Labeling Model (2023)

Abstract: This paper presents a series of approaches aimed at enhancing the performance of Aspect-Based Sentiment Analysis (ABSA) by utilizing extracted semantic information from a Semantic Role Labeling (SRL) model. We propose a novel end-to-end Semantic Role Labeling model that effectively captures most of the structured semantic information within the Transformer hidden state. We believe that this end-to-end model is well-suited for our newly proposed models that incorporate semantic information. We evaluate the proposed models in two languages, English and Czech, employing ELECTRA-small models. Our combined models improve ABSA performance in both languages. Moreover, we achieved new state-of-the-art results on the Czech ABSA.

## introduction in recent years, the pre-trained bert-like models based on the transformer @xcite architecture demonstrated their performance superiority across various natural language processing (nlp) tasks. in this paper, we study the possibility of a combination of two seemingly unrelated nlp tasks: aspect-based sentiment analysis (absa) and semantic role labeling (srl). we believe that the structured semantic information of a sentence extracted from an srl model can enhance the performance of an absa model. we investigate our assumption on the electra @xcite model architecture since it is a lighter and smaller alternative to the popular and commonly used models such as bert @xcite or roberta @xcite . because the elec-tra model is smaller in terms of the number of parameters, it does require less gpu memory and time to be fine-tuned. sentiment analysis (sa) is an essential part of nlp. the most prevalent sa task is the sentiment classification, where the objective is to classify a text fragment (e.g., sentence or review) as positive or negative, eventually as neutral. in this type of task, we assume that there is only one opinion in the text. in reality, as illustrated in figure 1 , this assumption often does not hold true @xcite . @xcite the burger was excellent but the waitress was unpleasant" ce ⇒ food, service cp ⇒ food:positive, service:negative aspect-based sentiment analysis @xcite focuses on detecting aspects (e.g., food or service in the restaurant reviews domain) and determining their polarity, enabling more detailed analysis and understating of the expressed sentiment. as shown by @xcite , the absa task can be further divided into four subtasks: aspect term extraction (te), aspect term polarity (tp), aspect category extraction (ce), and aspect category polarity (cp). we aim at the ce and cp subtasks, foot_0 and we treat them as a single classification task, see section 3.2. as depicted in figure 1 , the goal of the ce subtask is to detect a set of aspect categories within a given sentence, i.e., for a given text @xmath0, w 2 , . . . w n } assign set @xmath1, a 2 , . . . , a m } of m aspect categories, where @xmath2, k], m ⊂ a and a is a set of k predefined aspect categories @xmath3, a 2 , . . . , a k }. the goal of cp is to assign one of the predefined polarity labels p for each of the given (or predicted) aspect categories of the set m for the given text s, where @xmath4, negative, neutral}. the semantic role labeling task @xcite belongs among shallow semantic parsing techniques. the srl goal is to identify and categorize semantic relationships or semantic roles of given predicates. verbs, such as "believe" or "cook", are natural predicates, but certain nouns are also accepted as predicates. the simplified definition of semantic roles is that semantic roles are abstractions of predicate arguments. for example, the semantic roles for "believe" can be agent (a believer) and theme (a statement) and for "cook" agent (a chef), patient (a food), instrument (a device for cooking) -see examples in figure 2 . the theory of predicates and their roles is very well established in several linguistic resources such as propbank @xcite or framenet @xcite . we assume that leveraging the syntax and semantic information extracted from srl can significantly enhance the performance of the aspect category polarity subtask. this assumption is grounded in the notion that the srl information has the potential to unveil valuable and pertinent relations between entities within a given sentence, which play a crucial role in accurate aspect category polarity predictions. this holds particularly true for longer and more complex sentences, where a broader contextual understanding becomes essential. for a concrete illustration, please refer to appendix b. to combine the srl and absa models effectively, we propose three different approaches. through their integration, we demonstrate performance improvements on the absa task for both english and czech languages, employing electra-small models. moreover, we achieved new state-of-the-art (sota) results on the czech absa task. we publicly release our source codes foot_2 . ## related work the early studies @xcite @xcite focusing on the english absa task relied on word n-grams, lexicons, and other feature extraction techniques in combination with supervised machine learning algorithm such as support vector machine classifiers. these approaches were surpassed by deep neural network (dnn) models @xcite @xcite that typically employed recurrent neural network e.g., long short-term memory (lstm) @xcite . recently, the bert-like models were successfully applied to the absa task. @xcite solve the ce and cp subtasks at once by introducing auxiliary sentences and transforming the problem to a sentence-pair classification task. @xcite and @xcite improved results by pre-training the model on the task domain data. @xcite treated the absa task as a text generation task outperforming the previous sota results. @xcite employed graph convolutional networks. another related work can be found in @xcite . in @xcite @xcite the bert-like models were used for sentiment classification and subjectivity classification, to the best of our knowledge, there is no application of bert-like models for absa in the czech language. @xcite introduced the first czech absa dataset from the restaurant reviews domain. they used a maximum entropy classifier and conditional random fields for their baselines. @xcite extended this dataset and improved the baseline by adding semantic features. @xcite applied a convolutional neural network for the cp task and rnn for the cp task to the dataset from @xcite . the pioneered approaches of the srl @xcite task used standard feature engineering methods @xcite . since srl is closely bounded with syntax, adding syntactic information is very helpful. in 2008 conll shared task @xcite syntax-based srl task was proposed. in more recent years (with dnns), the attention was drawn back to standard span-based srl, where we form srl as (linear) tagging. many approaches are based on lstms @xcite . later, @xcite , inspired by the transformer, proposed a self-attention-based model. several end-to-end models for all srl subtasks were also introduced. @xcite abandon the bio tagging scheme, and they are rather predicting predicate-argument span tuples by searching through the possible combinations. they use a multi-layer bi-lstm to produce contextualized representations of predicates and argument spans. the most recent approaches use bert-like pretrained models. @xcite proposed a simple bert approach for argument identification and classification. this means, in their setting, the gold predicates are known. @xcite propose regular-constrained conditional random fields (crf) decoding on top of the same model. there are many other complex deep models @xcite for our experiments, we need an end-to-end srl model which encodes most of the information in the transformer's hidden state. however, to the best of our knowledge, there is no such model. as a result, we introduce our end-to-end model later in this paper to fulfil this need. various approaches have been made to enhance one task through the integration of another, usually using multi-task learning techniques. @xcite proposed a joint model for learning the whole nlp stack (pos tagging, chunking, parsing, semantic relatedness, entailment). they train a single model for all tasks in a sequence (chunking after pos tagging etc.). at each layer (for each task), they use regularization on the difference from previous layer weights. they show that the tasks help each other significantly. @xcite use dependency neighbourhood prediction and part-of-speech tagging as auxiliary tasks for absa. they introduced the new dependency neighbourhood prediction task to utilize the syntactic dependency information to improve the performance of the sentiment classification task. they train the auxiliary tasks together with the main sentiment classification task. the task classifies each token as either in the dependency neighbourhood or not. the dependency neighbourhood for a given token in a sentence is defined as the tokens in the sentence that are linked to the given token through, at most, n-hop dependency relations. @xcite pretrain bert model on semantic role labeling task and show, that the pretraining helps for many natural language understanding tasks. these examples of multi-task learning demonstrate the potential benefits of incorporating additional tasks in nlp models. ## models to find an effective way to combine the models, we first fine-tune the individual models separately to find the optimal set of hyper-parameters for individual tasks. moreover, we need srl fine-tuned model as the input for the combined models. for absa, we adopt the model proposed by @xcite . we propose a new srl end-to-end model, specifically designed for seamless integration with other tasks. ## experiments in our experiments, we aim to verify our idea that injected srl information can improve the results of the absa task, particularly the cp subtask. ## conclusion in this work, we introduce a novel end-to-end srl model that we use to improve the aspect category polarity task. our contribution lies in proposing several methods to integrate srl and absa models, which ultimately lead to improved performance. the experimental results validate our initial assumption that leveraging semantic information extracted from an srl model can significantly enhance the aspect category polarity task. importantly, the approaches we propose are versatile and can be applied to combine transformer-based models for other related tasks as well, extending the scope of their applicability. moreover, we believe that our approaches hold even greater potential in addressing other absa subtasks, namely term extraction and term polarity classification. these subtasks could benefit from the integration of srl and absa models in a similar manner. further, we would like to validate our approach on larger models, for example, bert or roberta. figure 7 : example of syntactic and semantic parse tree of the following sentence "this place is really trendi but they have forgotten about the most important part of a restaurant, the food".