A Multilingual Benchmark for Probing Negation-Awareness with Minimal Pairs (2021)

Abstract: Negation is one of the most fundamental concepts in human cognition and language, and several natural language inference (NLI) probes have been designed to investigate pretrained language models’ ability to detect and reason with negation. However, the existing probing datasets are limited to English only, and do not enable controlled probing of performance in the absence or presence of negation. In response, we present a multilingual (English, Bulgarian, German, French and Chinese) benchmark collection of NLI examples that are grammatical and correctly labeled, as a result of manual inspection and reformulation. We use the benchmark to probe the negation-awareness of multilingual language models and find that models that correctly predict examples with negation cues, often fail to correctly predict their counter-examples without negation cues, even when the cues are irrelevant for semantic inference.

## introduction negation is a fundamental concept of human cognition, for asserting the falsity of a proposition @xcite . the linguistic markers of negation enable us, for example, to deny that events happened, or to express the absence of objects @xcite . hence, correctly processing markers of negation is a key building block of language comprehension, often acquired by children in multiple stages @xcite . there is ample evidence that natural language processing (nlp) models struggle with processing negation: negation has been identified as a frequent source of error for various nlp tasks, such as sentiment analysis @xcite , statistical and neural machine translation @xcite @xcite , and question answering @xcite . to complement insights from anecdotal error analysis, several diagnostic datasets have recently been designed to explicitly investigate the negation processing capabilities of pretrained language models, either directly on the encoder @xcite or after fine-tuning the model for downstream tasks such as natural language inference (nli) @xcite @xcite @xcite or sentiment analysis @xcite . these datasets have been useful to shed light on models' negation processing capabilities, but the diagnostic datasets so far only cover english. with our work, we extend the efforts of analyzing a model's negation awareness to a multilingual setup, by deriving an analysis dataset from the multilingual xnli dataset @xcite . the goal of our work is to provide multilingual datasets to determine if models adequately process negation. we not only want to know if they make correct inferences in the presence of negation but also if correct inferences are a result of adequately modeling the semantics of the negation, or if they are a result of merely exploiting shallow heuristics. in our datasets, we measure this negation awareness by quantifying the extent to which: (i) models correctly change their prediction when a labelchanging negation is inserted/removed from an input sentence; and (ii) models correctly keep their predictions unchanged when a label-preserving negation is inserted/removed from an input sentence. to this end, we create minimal pairs of nli examples, that only differ in the presence/absence of negation (see h: it was certain that the rabbis saw the signs. contributions we create five new targeted probing datasets in english, french, german, bulgarian and chinese. 1 our probing datasets consist of minimal pairs of nli examples, with manual annotations that allow us to measure the extent of a model's negation awareness. in contrast to other negation probing datasets, our minimal pairs allow for a detailed comparison of a model's performance on important (label-changing) and unimportant (label-preserving) negations. we find that in a large amount of cases, the model cannot correctly adjust its prediction when the negation is removed, pointing towards the model's inability to model the negation's effect on sentence semantics. ## related work the processing of negation has been investigated in several works, either by creating dedicated diagnostic datasets, or by investigating the phenomenon as part of a more general analysis. an overview over existing datasets can be found in crafting adversarial nli examples based on unimportant negations several works have established that negation cues in nli examples are a strong indicator for a contradiction label (gururangan et al., 2018; dasgupta et al., 2018; poliak et al., 2018). hence, negation can be used to craft adversarial nli examples, i.e., to apply a label-preserving change that makes the model incorrectly change its prediction, by either inserting or removing unimportant negations. such adversarial examples have been derived from the multinli (mnli) dataset (williams et al., 2018), a large english dataset with nli examples from multiple genres. naik et al. (2018) automatically build such adversarial examples by adding a tautology (and false is not true) to the end of every hypothesis in the mnli data, leading to a drop in model accuracies caused by a large amount of false positives (fps) for the neutral class, rather than an expected increase in fps for the contradiction class. the authors hypothesize that this pattern stems from decreasing lexical overlap by adding the tautology, and as low lexical overlap is indicative for the neutral class, the model mispredicts the example as neutral. aspillaga et al. (2020) confirm that also transformer-based models perform poorly on this negation stress test. on their challenge dataset for non-entailed subsequences based on the mnli data, mccoy and linzen (2019) find that models exploit a mismatch between negation cues in premise and hypothesis to predict non-entailment, and that removing unimportant negation cues decreases model accuracy to almost 0. measuring negation awareness based on important negations instead of focusing on unimportant negations in order to uncover lexical biases in the data, other works focus on processing important negations, i.e., negations that induce a label change when added/removed. kim et al. (2019) build an nli dataset based on mnli premise-hypothesis pairs that contain antonyms. they build combinations of syntactically negated and non-negated versions of the premise and hypothesis, and re-label the resulting examples. their results confirm that models perform worse on this analytic dataset than on the original dataset, and that models are worse in predicting pairs with syntactic negation than pairs with antonyms. hossain et al. (2020b) create a dataset containing challenging negations by adding the syntactic negation cue not to the main verb of the premise and/or the hypothesis of mnli training examples. they find that their newly created examples are more challenging for the model than the original negated examples. authors task base dataset data creation langs ettinger (2020) mlm (cloze) psycholinguistic stimuli -en kassner and schütze (2020) mlm (cloze) -template filling en naik et al. (2018) nli mnli adding tautology en kim et al. (2019) nli mnli inserting/removing negation, swapping antonyms en hossain et al. (2020b) nli mnli inserting negation en richardson et al. (2020) nli -template filling en geiger et al. (2020) nli snli replacing single words en ours nli xnli removing negation to build minimal pairs en, bg, de, fr, zh @xcite and @xcite directly probe pre-trained encoders using a cloze language modeling task. they find that a language model makes the same predictions in negative and assertive contexts, but it is unclear to what extent we can expect a language model to learn the semantics of negation from an unsupervised pretraining task (see @xcite ). our multilingual benchmark differs from previous work on probing negation-awareness, not only in being multilingual, but also in using minimal pairs. minimal pairs have been used for other diagnostic datasets, e.g. to check if language models assign higher probability to grammatical sentences @xcite . @xcite suggest to augment existing datasets with minimally different examples to test a model's decision boundaries. with our multilingual benchmark, we contribute to a line of research that probes a model's handling of linguistic phenomena in multiple languages, including features of sentence representations @xcite , tenses @xcite , gender bias (gonzález et al., 2020), numerical understanding @xcite , and lexical semantics @xcite . ## approach our goal is to verify if a model adequately processes negation: in the presence of negation, the model should make a correct prediction. the most naive way to look at this is to check model performance on examples that contain negations. however, even if the model correctly predicts such examples, it might do so for the wrong reasons: it might (1) ignore the negation completely or not properly model its effect on the inference category. in such cases, it would have predicted the same class if the negation was absent. in order to check for this deficiency, we need minimal pairs with relevant or important negations, i.e. the presence/absence of negation changes the correct inference label. on the other hand, it might (2) predict an inference label based on the presence/absence of negation in the sentence, no matter if the negation is relevant to the inference relation or not. this could either be because of the established biases related to negation cues, or because the model does not correctly model the scope of the negation (assign a wrong scope, or does not have a notion of scope at all). in order to check for this deficiency, we need minimal pairs with irrelevant or unimportant negations, i.e. the presence/absence of the negation does not change the correct inference label. considering minimal pairs with both types of negation allows us to gain a more complete insight into model behaviour and sheds light on the questions if the model ( 1 ) is aware of negation and its effect on sentence semantics and (2) models the effect of negation beyond its mere presence/absence in a sentence. overall, our datasets contribute to newsweek has never written anything about the hamptons. → newsweek has written something about the hamptons. sie werden für nichts im voraus zahlen. my friend is deaf, so he cannot listen to music my friend is deaf, so he can listen to music ## data analysis our new multilingual benchmark comprises nli examples containing a range of different negation cues. the five most frequent cues are listed in ## probing negation-awareness we now use our new multilingual benchmark to probe the negation awareness of a multilingual language model fine-tuned for the nli task. multilingual language model for nli we finetune the cased version of mbert on the english mnli training data using the standard sequence classification approach for sentence pairs @xcite and follow hossain et al. ( 2020b ) in training the model for 3 epochs, with a batch size of 32 and a learning rate of 2e-5. we then evaluate model performance on the minimal pairs. for the languages other than en, the evaluation is considered a zero-shot setup, as the model has not seen any nli training data in the target language. performance on minimal pairs our benchmark is designed to draw conclusions about a model's reaction to the phenomenon of negation, by evaluating model predictions on both parts of the minimal pair. 5 to understand if and under which circumstances a model adequately processes negation, we are interested in cases where the model seemingly correctly processes a negation, i.e. it correctly labels an example containing a negation cue, but mispredicts once the negation is removed. we quantify this behaviour as the percentage change between the correct predictions on the original example (m -), and the correct predictions on the original example and the modified example (m - @xmath1. while we want the model to perform well on all minimal pairs, considering minimal pairs that differ in important and unimportant negations separately allows us to gain better insights into model behaviour. the main results that we discuss in the following are shown in figure 4 . @xmath2. the dotted bars refer to performance on the subset of minimal pairs that does not contain any negation cue in the modified example (m - m + o ). we do not include @xmath3, as we have very few pairs for this category (< 5 pairs). figure 5 : confusions on minimal pairs with important negation. we show model predictions on the modified example, given that the corresponding original example was predicted correctly, with gold labels on the y-axis and predicted labels on the x-axis. the black frames indicate the percentage of correctly handled minimal pairs. the first five plots show results on the subset of our minimal pairs without negation mismatches (m -m + o ); the last plot is on the minimal pairs derived from hossain et al. (2020b) (m + m - h ). unimportant negations performance on minimal pairs with unimportant negations can reveal if the model uses negations cues as shallow statistical cues without correctly modelling their effect on sentence semantics. here, the model should stick with its prediction when the negation is removed. if the model changes its prediction, the initial correct prediction was due to the presence of the negation cue in the data (even though the negation cue is irrelevant for inferring the correct label). possible explanations for the misprediction are (a) the model exploits the negation as a statistical cue without modeling its effect on sentence semantics; performance drops on minimal pairs with unimportant negations are shown on the left of figure 4 . we first focus on the filled bars that indicate performance on the complete set of minimal pairs m - @xmath4. we observe the largest performance drop for the contradiction class, across all languages (except en), indicating that the model incorrectly changes its prediction after negation removal more often than for entailment or neutral. this is in line with the previously reported bias of negation cues indicating contradiction @xcite . interestingly, in contrast to the other languages, for en the largest performance drop is observed for the neutral class. the confusion matrices for this experiment (see appendix d) show a high number of fps for the contradiction class, consistently across all languages, and most pronounced for en. this seems counter-intuitive, but might be in line with the negation mismatch bias @xcite : recall that in case an example in m -contains a negation cue in premise and hypothesis, its counterpart in @xmath5, i.e. a negation mismatch. this applies to around 29% of the modified examples across languages. confirming negation mismatch bias to confirm this bias, we re-compute results on the subset of minimal pairs that does not contain any negation cues in the modified example (m -m + o ). results are shown in figure 4 as dotted bars. we see that the performance drop for @xmath6, indicating that the model makes more mistakes on the minimal pairs when it cannot rely on the negation mismatch in the data. this confirms the previously reported bias for en: negation mismatch makes a model predict contradiction, and by removing examples with negation mismatch we make it harder for the model to predict this class. ## conclusion in this work, we derived a multilingual benchmark for testing negation awareness using minimal pairs, and demonstrated its use to gain insights into a model's prediction behaviour. using our benchmark, we confirm that biases previously reported for english do transfer in a zero-shot transfer setup, and that the problem of exploiting shallow statistical cues might be larger in the latter setup.