A Diagnostic Dataset for Sentiment and Negation Modeling for N orwegian (2023)

Abstract: Negation constitutes a challenging phenomenon for many natural language processing tasks, such as sentiment analysis (SA). In this paper we investigate the relationship between negation and sentiment in the context of Norwegian professional reviews. The first part of this paper includes a corpus study which investigates how negation is tied to sentiment in this domain, based on existing annotations. In the second part, we introduce NoReC-NegSynt, a synthetically augmented test set for negation and sentiment, to allow for a more detailed analysis of the role of negation in current neural SA models. This diagnostic test set, containing both clausal and non-clausal negation, allows for analyzing and comparing models’ abilities to treat several different types of negation. We also present a case-study, applying several neural SA models to the diagnostic data.

## introduction in sentiment analysis, negation is an instance of the more general category of valence shifters @xcite , i.e., expressions that modify the polarity of a sentiment expression. they can shift polarity entirely, or reduce or increase it, functioning as diminishers or intensifiers, respectively. negation is a well-known challenge for the correct treatment of sentiment analysis (sa) related tasks, and several previous studies have discussed negation as a source of error for sa, such as @xcite and @xcite . in this paper we present new data on the relationship between sentiment and negation in norwegian, based on the norwegian review corpus (norec) @xcite , a collection of professional reviews spanning several different domains, where the same subset of documents have been annotated for both structured sentiment (norec fine ; @xcite , and negation (norec neg ; @xcite . to be able to more precisely test the sensitivity of a model to the correlations between these two phenomena, we introduce a new diagnostic dataset, norec negsynth , foot_0 which contains synthetically constructed minimal pairs that illustrate more fine-grained negation phenomena. the dataset is created by extending norec fine by negating existing sentences in the test set, and then annotating them for sentiment and negation. these new sentences have then been annotated for unnaturalness. we further illustrate the use of this synthetic dataset by reporting diagnostics for three models, showing the potential of our dataset. this paper is organized as follows. in section 2 we describe relevant previous research on sentiment and negation. in section 3 we analyze the relationship between sentiment and negation as it can be found in the two original datasets, discussing relevant corpus statistics. in section 4 we turn to describe the creation of the new diagnostic dataset. in section 5 we briefly introduce the models used to showcase the diagnostic data, also discussing the results. we propose some directions for future work in section 6, before concluding in section 7. several studies have focused on how diagnostic datasets can be used for sa, including @xcite , who present a challenge dataset for sa where instances that several sa-models get wrong are annotated for a range of different phenomena, noting that negation is one of the phenomena that affects sa the most. @xcite who create diagnostic tests to analyze robustness in multimodal sa. ## corpus study on negation and sentiment while a synthetically created diagnostic dataset can give us detailed insight into the performance of different models, it does not necessarily tell us anything about the relationship between sentiment and negation in actual language use. in order get a better understanding of the possible interactions between these two phenomena, we perform a corpus study on the norec fine and norec neg datasets. this preliminary study helped inform both the creation of our synthetic dataset and later the interpretation of the results obtained when evaluating models on it. ## synthetic data for diagnostics of negation and sentiment modeling based on the corpus study described above, we have gained more insight into the relation between negation and sentiment. we now turn to describe the creation of a synthetic diagnostic dataset for evaluation of sentiment analysis models that measures specifically the effect of negation. the diagnostic dataset was created based on existing norec annotations, by manually augmenting the test set using cues found in norec neg , or by removing already existing negation cues. the complete process was as follows: 1) we first extracted sentences containing at least one polar expression, before 2) manually inspecting to see which negation cues could be used to negate the polar expressions in the sentence. 3) if a negation was applicable, we ascertain whether the cue can be added without significant syntactic changes. 4) the new sentences were given a new sentence id, and each newly negated polar expression was mapped to its corresponding polar expression in the original sentence. 5) once the dataset was completed, each sentence was annotated for negation, polarity, intensity and naturalness. the process was identical for the removal of cues. examples 4 to 7 below illustrates the process. here the original sentence has no negation, and then the cues ikke 'not', ingen 'no;none;no one' and på ingen måte 'in no way' were used to negate it. one benefit of this method is that it allows us to align the synthetic negated sentences to the preexisting annotations to create minimal negation pairs. furthermore, non-manual methods such as rule-based negation insertion is difficult, as negation is a complex phenomenon that relies not only on syntactic constraints, but also on semantics. not all polar expressions can be negated, and not all negations can be used in all cases. the large number of existence negators such as strippe 'stripped;bare', fravaer 'absence', savne 'miss' and blotte 'void (of)' understandably have limited use, as they require an existing existential expression to negate. another example is the negator la vaere 'refrain from', which requires a verb with an agentive subject, vastly restricting its distribution. in the synthetic sentence in example 8, the original verb gjort 'done' has been substituted by latt vaere å gjøre 'refrained from doing', as gjøre 'do' allows for agentive subjects. examples 4 to 7 are good examples of some of these restrictions. we see that the original sentence in example 4 cannot be negated with an existential negator, as there is no expression of existance to negate. the elided copula also does not take an agentive subject, leaving out possible negation with la vaere 'refrain from'. another problem arises when a polar expression restricts the context in a way that hinders a negation from sounding natural. expressions such as ulempen er at ..., 'the disadvantage is that ...' and heldigvis ... 'fortunately' require a polar expression to have negative and positive polarity, respectively, in order to sound natural. ## benchmarking in order to illustrate the use of our diagnostic dataset in the analysis of sentiment models we apply three different models to it. the two first models are the baseline models for the 2022 se-meval shared task on structured sentiment analysis @xcite , while the third is a graph model presented in @xcite . for all models we focus on polar expressions and the effect the different types of negation has on the interpretation of these if they are in the scope of a cue. although some of the models have the capacity to treat both targets and holders in addition to polar expressions, we have chosen to ignore these expressions for the purpose of this dataset. ## conclusion we have performed basic statistic checks of the relationship between negation and sentiment in norwegian review texts and seen cases where certain negators co-occur more frequently with certain types of polarity and intensity. this motivated the creation of a synthetic dataset, which was used to evaluate three models. we see that this dataset reveals differences in how different machine learning models treat different cues, and how they differ in their ability to correctly identify polarity in minimal negation pairs, when the non-negated sentence is correctly identified. furthermore, we see that although it is still difficult to include low-frequency cues, due to their limited syntactic and semantic flexibility, the increased number of common cues allow us to observe differences with greater confidence. we also note that this allows us to investigate cues that were not present in the original test set. we observe that having minimal negation pairs allows us to gain insight into the capabilities of the model which would not have been possible without these data. as the number of cues in the diagnostic dataset is comparable to the full test set, we believe that this type of diagnostic set can also save future annotation efforts where applicable.