Investigating the Saliency of Sentiment Expressions in Aspect-Based Sentiment Analysis (2023)

Abstract: We examine the behaviour of an aspect-based sentiment classifier built by fine-tuning the BERT BASE model on the SemEval 2016 English dataset. In a set of masking experiments, we examine the extent to which the tokens identified as salient by LIME and a gradient-based method are being used by the classifier. We find that both methods are able to produce faithful rationales, with LIME outperforming the gradient-based method. We also identify a set of manually annotated sentiment expressions for this dataset, and carry out more masking experiments with these as human rationales. The enhanced performance of a classifier that only sees the relevant sentiment expressions suggests that they are not being used to their full potential. A comparison of the LIME and gradient rationales with the sentiment expressions reveals only a moderate level of agreement. Some disagreements are related to the fixed length of the rationales and the tendency of the rationales to contain content words related to the aspect itself.

## introduction saliency approaches to understanding the output of a model attempt to locate the parts of the input that contribute most to the model's decision. these salient words (or rationales) are often evaluated by measuring their faithfulness to the model, i.e. the extent to which the model relies on them to arrive at a prediction. the jury is still out regarding the best saliency method, whether it is a black-box, model-independent method such as lime @xcite or a model-dependent method such as those which use the training loss function gradient to obtain the rationale @xcite . @xcite compare saliency methods on three text classification tasks and find that gradient-based rationales perform the best overall. @xcite , on a different set of tasks, find that lime produces more faithful rationales than gradient or attention-based rationales. rationales produced by saliency methods are often compared to human rationales. @xcite introduce the eraser benchmark of human rationales for datasets covering the tasks of fact verification, movie review sentiment polarity classification, evidence inference, natural language inference and various forms of qa. we build on this work by sourcing a set of human rationales for the task of aspect-based sentiment analysis. aspect-based sentiment analysis (absa) is a form of fine-grained sentiment analysis that attempts to determine the opinion about some aspect of a topic. for example, given a restaurant review example 1. i love where it is located but the service leaves much to be desired a system should return the polarity positive for the "location" aspect and negative for the "service" aspect. much work in absa has taken place within the context of the semeval 2014-2016 shared tasks which use online consumer reviews of laptops, restaurants and hotels @xcite @xcite . state-of-the-art approaches @xcite are underpinned by pretrained models such as bert @xcite . we carry out a novel interpretability study for aspect-based sentiment analysis, focusing on the semeval 2016 english dataset and a polarity classifier built by fine-tuning bert. our study attempts to answer the following questions: in answer to our first research question, we conduct a masking study and find that, while both methods are useful for selecting rationales, lime produces more faithful rationales. in order to answer our second research question, we need human rationales for the dataset we are using. it turns out that such data already exists, in the form of manually annotated sentiment expression spans in the semeval 2016 data. these annotations were created by @xcite in the context of an effort to boost classifier performance rather than to produce human rationales for an interpretability study. however, they are well suited as human rationales as the sentiment expressions are the words the annotators judged to be expressing the sentiment towards the aspect. we show that the performance of our bert model improves substantially when all but the sentiment expression tokens are masked (during training and testing), suggesting that they are not currently being used to their full potential. this experiment also reveals that sentiment expressions that are not referring to the aspect in focus are also being used. for our third research question, we measure the alignment of the lime and gradient rationales with these human rationales or sentiment expressions. measures of the alignment between automatic and human rationales can serve as an evaluation measure of rationale extraction methods, providing a different view than existing automatic evaluation measures. at the same time, we should not expect the two to fully agree since they are not designed to reflect exactly the same thing. human rationales are the words that human annotators judged to be contributing to the sentiment towards the aspect, and the automatic rationales are designed to reflect what the classifier used regardless of whether they are sentiment-bearing, related to the aspect or neither. we find only moderate agreement between the two types of rationales. the highest overlap with the human rationales (approx 60%) is obtained by an automatic rationale length of 50% of the input. how much of the difference can be explained? some of the words in the lime and gradient rationales that are not in the human rationales are words that are used to identify the focus of the sentiment, i.e. those that are related to the aspect, e.g. service in ex. 1. it is reasonable to expect these to be used by the classifier in order to locate the sentiment expressions. we also note structural differences -the human rationales have been mostly annotated as continuous spans (containing function words)whereas the automatic rationales do not have this property; the automatic rationales are a fixed proportion of the input length whereas the human rationales are not. the fixed length of the automatic rationales is an important factor because agreement with the human rationales jumps to over 80% when an oracle length is employed. ## dataset we use the semeval 2016 english absa dataset, focusing on subtask b foot_0 where the label to be predicted is the sentiment polarity (positive, negative or neutral), the text granularity is sentence-level and the aspect category is supplied with the input sentence. the aspect category is of the form en-tity#aspect, e. g. laptop#battery. there are a total of 2000 training sentences and 676 test in the restaurant domain, with 2500 training and 808 test sentences in the laptop domain. the number of training and test instances is higher as approximately 15% of sentences are annotated with multiple aspects as in ex. 1. @xcite add an additional layer of annotation foot_1 to this dataset by marking the spans of the sentiment expressions (ses) in each sentence. in ex. 2, the se is the phrase good quality. example 2. the display is good quality . for sentences with neutral polarity where no opinion is expressed, the se is empty -see ex. 3 from @xcite . example 3. we had lunch in that restaurant last week. we use these ses as the human rationales in our experiments. we reserve 5% of the training data as development data, keeping the same distribution of domains and target labels as in the full training data. we jointly train on the laptop and restaurant domains, concatenating the respective training and development sets, prefixing the question with a domain label ("laptop:" or "restaurant:") to help the model to adjust to domain-specific patterns. ## sentiment classifier we train twelve classifiers for ten epochs, selecting the epoch with highest accuracy according to development data. the same set of twelve random seeds is used in all settings. the seed controls the randomisation of the training-development split, initialisation of the classification head and any other randomised parts of training, e. g. training batch creation. hyperparameters are detailed in the appendix. ## saliency methods and rationales we employ a gradient-based saliency method and a black-box explanation method from which we derive a saliency score. we do not include attentionbased saliency methods as recent discussion suggests caution with these @xcite and as results of @xcite show superior faithfulness of gradient-based rationales over attention-based rationales for seven of eight settings tested. we also do not include methods that modify a classifier to produce a rationale as a sideproduct or as an intermediate step of the classification and that would produce classifiers that are not comparable to our baseline classifier @xcite @xcite . once each token of the input sequence "b" with length n has received a saliency score, we select the top @xmath0.5 + nl tokens as the rationale, where l is the relative rationale length. we try @xmath1.25, 0.5, 0.75}. 3 this means that in each setting the relative rationale length is fixed. ## conclusion using automatic rationales determined by a blackbox and a gradient-based saliency method, and sourcing human rationales for a popular english absa dataset, we have thoroughly explored the behaviour of a bert model fine-tuned for this task. in answer to our first research question, we find that both saliency methods produce rationales with similar faithfulness but lime produces more comprehensive rationales. in answer to our second research question, we find that the words in the human rationales are not being used to their full potential in our bert-based classifier. when they are used in isolation, performance improves by 5.5%. in answer to our third research question, we do not find a high level of agreement between the human and the automatic rationales. some of the differences can be accounted for by the fixed length of the rationales, content words related to the aspect and the continuous nature of the human rationales. although @xcite report negative results with joint learning of polarities and sentiment expressions on this dataset, the promising classifier results when all but the relevant ses are masked suggest that absa systems should try to learn these prior to or in parallel with learning the polarities. the improved agreement between the human rationales and the oracle length automatic rationales suggests that future work should also explore instance-specific rationale lengths, e. g. using methods of @xcite .