Non-Compositionality in Sentiment: New Data and Analyses (2023)

Abstract: When natural language phrases are combined, their meaning is often more than the sum of their parts. In the context of NLP tasks such as sentiment analysis, where the meaning of a phrase is its sentiment, that still applies. Many NLP studies on sentiment analysis, however, focus on the fact that sentiment computations are largely compositional. We, instead, set out to obtain non-compositionality ratings for phrases with respect to their sentiment. Our contributions are as follows: a) a methodology for obtaining those non-compositionality ratings, b) a resource of ratings for 259 phrases – NonCompSST – along with an analysis of that resource, and c) an evaluation of computational models for sentiment analysis using this new resource.

## introduction in nlp, the topics of the compositionality of language and neural models' capabilities to compute meaning compositionally have gained substantial interest in recent years. yet, the meaning of linguistic utterances often does not adhere to strict patterns and can be surprising when looking at the individual words involved. this affects how those utterances behave in downstream tasks, such as sentiment analysis. given a phrase or sentence, that task involves predicting the polarity as positive, negative or neutral. sentiment largely adheres to compositional principles @xcite : "if the meaning of a sentence is a function of the meanings of its parts then the global polarity of a sentence is a function of the polarities of its parts." modelling sentiment as a compositional process is, therefore, often mentioned as a design principle for computational sentiment models (e.g. by @xcite @xcite . nonetheless, one can think of examples where the sentiment of a phrase is unexpected given the these examples were obtained from our newly annotated stimuli. ## related work over the course of years, sentiment analysis systems went from using rule-based models and sentiment lexicons (e.g. @xcite to using recursive neural networks @xcite @xcite , to abandoning the use of structure altogether by finetuning pretrained large language models (e.g. @xcite @xcite , and recently, to abandoning training, via zero-shot generalisation @xcite . crucial to the development of these systems has been the introduction of benchmarks, such as sst @xcite and se-meval's twitter benchmarks (e.g. @xcite @xcite . although the vast majority of related work focused on simply improving on benchmarks, there have been studies more closely related to ours, asking questions such as: how do phrases with opposing sentiment affect each other @xcite ? what is the role of negations @xcite , modals and adverbs (kiritchenko and mohammad, 2016c)? do idioms have non-compositional sentiment @xcite ? which linguistic phenomena are still problematic for sota sentiment systems @xcite ? can we incorporate compositional and non-compositional processing in one system @xcite ? and can we computationally rank sentences according to their sentiment compositionality @xcite ? gaining a better understanding of the contexts in which sentiment functions non-compositionally and is challenging to predict is crucial for the evaluation of sentiment models in an age where sentiment benchmarks may appear saturated @xcite . 1 we position our work in this latter group of articles, of which that of hwang and hidey is most closely related. 1 as a concrete example, consider the widely-used binary sst sentiment analysis task contained in the glue benchmark @xcite : at the time of writing, sota performance for this task matches humans' performance. step 2: obtain the sentiment of subphrases. step 3: collect sentiment for the phrase and its constructed control phrases. step 4: use control phrases to establish a default mapping to the sentiment. the difference between the default and observed sentiment yields a non-compositionality metric. control default mapping observed sentiment vs. study 1 study 2 step 1: select phrases to annotate: preprocessing postprocessing control control control control control control control subphrase subphrase control control control control control control subphrase subphrase subphrase subphrase subphrase subphrase from the stanford sentiment treebank (sst) good agreement in sst subphrases of length 3 -8 no named entities select control stimuli for subphrases: same grammatical category same sentiment figure 2: the methodology summarised. steps 1 and 4 consist of data pre-and postprocessing; steps 2 and 3 involve collecting data from participants via prolific. ## collecting non-compositionality ratings compositional processing of sentiment involves applying a function to the polarity of subphrases to obtain the polarity of the phrase. turning this notion into a quantifiable metric requires us to measure the polarities and determine the composition function. non-compositional phrases are then simply phrases whose sentiment deviates from what is expected. how do we implement this? we first select data ( §3.1) and then obtain sentiment labels for phrases through data annotation studies ( §3.2). we consider the composition function to be the default mapping from two subphrases with a specific sentiment to their combined sentiment. we obtain the default mapping by replacing subphrases with control stimuli and annotating sentiment for those modified phrases. using those results, we can compute the non-compositionality ratings ( §3.3). figure 2 summarises the full procedure. ## analysis of the ratings what patterns can we identify in these ratings? we examine sentiment composition types, phrase lengths and syntactic categories of subphrases in appendix c; only the sentiment composition type subphrase a subphrase b rating sentiment human roberta a nearly terminal case of the cutes 4.11 5.33 2.00 the franchise's best years are long past -4.11 0.33 1.00 all the excitement of eating oatmeal -3.00 2.00 2.33 a pressure cooker of horrified awe -2.56 1.00 3.67 fans of the animated wildlife adventure show will be in warthog heaven 1.56 5.67 5.00 a real human soul buried beneath a spellbinding serpent's smirk 1.56 4.67 5.00 everyone involved with moviemaking is a con artist and a liar -1.33 0.00 1.00 the modern master of the chase sequence returns with a chase to end all chases 1.11 6.00 5.00 other atypical sentiment patterns that we observe require common-sense reasoning about terms that act as contextual valence shifters @xcite , e.g. to understand that "are long past" implies something about current times, or that "eating oatmeal" relates to the blandness of that experience. lastly, we also observe discourse relations between subphrases that modify the sentiment in a non-compositional manner. in "fans of the animated wildlife adventure show will be in warthog heaven", the parallel between 'wildlife' and 'warthog heaven' amplifies the positive sentiment in a way that would not have happened for fans of a fashion show. similarly, 2 we include them in appendix c, in ## evaluating sentiment models how can we employ the non-compositionality ratings to better understand the quality of sentiment systems? we illustrate this by recreating the ratings using state-of-the-art pretrained neural models and comparing them to the humans' ratings. experimental setup to obtain the ratings from models, we adapt sst foot_0 to use the 7-point scale and exclude the phrases of interest from the training data. per model type, we fine-tune three model seeds that we evaluate on the sst test set using f 1 -score, and on noncompsst using a) the correlation of the models' and humans' noncompositionality ratings (pearson's r), and b) the f 1 -score of noncompsst phrases, using the humans' sentiment scores as labels. to obtain models' non-compositionality ratings, we average sentiment predictions from the three model seeds and apply the same postprocessing as applied to the human-annotated data (see §3.3). we evaluate roberta-base and -large @xcite fine-tuned on sentiment of comments from social media posts; and finally roberta-base fine-tuned on sentiment from imdb movie reviews (maas et al., 2011). 4 results the results in finally, we also inspect the noncompsst f 1scores for all 259 phrases and the 67 phrases with the highest non-compositionality ratings according to the human annotators, for which results are included in the final two columns of ## conclusion sentiment and compositionality go hand-in-hand: success in sentiment analysis is often attributed to models' capability to 'compose' sentiment. indeed, the sentiment of a phrase is reasonably predictable from its subphrases' sentiment, but there are exceptions due to the ambiguity, contextuality and creativity of language. we made this explicit through an experimental design that determines non-compositionality ratings using humans' sentiment annotations and obtained ratings for 259 phrases ( §3). even though most phrases are fairly compositional, we found intriguing exceptions ( §4), and have shown how the resource can be used for model evaluation ( §5). for future sentiment analysis approaches, we recommend a multi-faceted evaluation setup: to grasp the nuances of sentiment, one needs more than compositionality.