An Analysis of Natural Language Inference Benchmarks through the Lens of Negation (2020)

Abstract: Negation is underrepresented in existing natural language inference benchmarks. Additionally, one can often ignore the few negations in existing benchmarks and still make the right inference judgments. In this paper, we present a new benchmark for natural language inference in which negation plays a critical role. We also show that state-of-the-art transformers struggle making inference judgments with the new pairs.

## introduction natural language understanding remains an elusive goal except in limited scenarios. it is arguably the ultimate problem in natural language processing: to empower machines to understand language as generated by humans. the state of the art has seen tremendous progress in recent years, and has moved from symbolic representations @xcite to distributional representations often learned from massive datasets @xcite . recognizing entailments @xcite , identifying paraphrases @xcite , determining semantic textual similarity @xcite , and sentiment analysis @xcite are but a few problems that require natural language understanding to a lesser or greater degree. there are many benchmarks targeting the problems above, and they usually cast them as classification problems. a couple of popular evaluation platforms, glue @xcite and super-glue @xcite , aggregate benchmarks for some of the problems above and provide a single score for many tasks under the umbrella of natural language inference. state-of-the-art models are close to or even surpass human performance @xcite . this fact, however, is true only when evaluating models and humans with existing benchmarks. indeed, researchers have pointed out weaknesses in benchmarks suggesting that we are evaluating models with examples that are much simpler than what humans are capable of (section 3). source text selection, annotation artifacts @xcite , and asking annotators-either experts or crowd workers-to write examples as opposed to retrieving real examples from previously generated language are a few of the culprits. in this paper, we investigate the role of negation in a core natural language understanding task: natural language inference-in its most basic form, determining whether a text entails a hypothesis. recognizing entailments has many applications including question answering @xcite , summarization @xcite and machine translation evaluation @xcite . negation relates an expression e to another expression with a meaning that is in some way opposed to the meaning of e @xcite , thus it plays an important role in natural language understanding. additionally, negation is ubiquitous in regular english texts: approximately 25% of english sentences contain negation depending on the domain and genre (section 4). despite these facts, negation is underrepresented and mostly irrelevant in existing benchmarks-one can literally disregard the negations and still make correct inference judgments in popular datasets. the work presented here addresses these shortcomings and makes the following contributions: 1 1. we show that negation is underrepresented and often irrelevant in existing benchmarks. 2. we create new benchmarks for natural language inference in which negation plays a critical role to make inference judgments. 3. we demonstrate that state-of-the-art trans-formers trained with the original benchmarks are not robust when negation is present. 4. we provide empirical evidence that transformers may be unable to learn the intricacies of negation in the most challenging benchmark, which includes longer texts from many genres. ## background the task of natural language inference or recognizing textual entailment consists in determining whether a hypothesis is true given a text. the original task considers two labels: entailment or no entailment @xcite , and a newer formulation considers three labels: entailment, contradiction or neutral @xcite . for example, the text "a person on a horse jumps over an airplane" entails hypothesis "a person is outdoors, on a horse," contradicts "a person is at a diner, ordering an omelette," and is neutral with respect to "a person is training his horse for a competition." we work with three existing benchmarks: a collection of rte datasets @xcite @xcite , @xcite and mnli @xcite . the rte datasets are smaller (5,767 text-hypothesis pairs) than snli and mnli (569,033 and 431,997 pairs). mnli is more challenging than rte and snli: texts are longer and were selected from 10 genres including fiction and non-fiction as well as conversation transcripts. on the other hand, the texts in snli were selected from image captions. the hypotheses in snli and mnli were crowdsourced, i.e., manually generated by non-experts. tables 2 and 4 show examples in the rte, snli and mnli benchmarks. we work with the formatted versions of these datasets in the glue @xcite and superglue @xcite benchmarks for convenience. ## previous work previous work has revealed weaknesses with the benchmarks we work with and that adversarial examples can break models for many natural language processing tasks. adversarial examples consist of arguably trivial modifications to inputs that trick computational models. some of them include misspellings @xcite , syntactically controlled paraphrases @xcite , lexical substitutions @xcite , and more elaborate substitutions @xcite . more re-cently, @xcite propose checklist, a task-agnostic strategy for testing nlp models. their strategy can be used to identify which linguistic capabilities a model lacks. for example, they show that commercial systems for sentiment analysis are not robust when negation is present. regarding natural language inference, @xcite show that models taking into account only hypotheses significantly outperform majority baselines, and @xcite discuss annotation artifacts, e.g., negation cues (not, never, etc.) are a strong indicator of contradictions. @xcite show that models trained with snli fail to resolve new pairs that require simple lexical substitution, e.g., holding a saxophone contradicts holding an electric guitar. @xcite conclude that models are not robust to negation, but their only test is concatenating the tautology "and false is not true" to hypotheses. @xcite introduce universal triggers and show that concatenating negation cues to snli hypotheses decreases accuracy to almost zero when the gold label is entailment or neutral. the task of identifying paraphrases consists in determining whether two sentences have the same meaning, and can be casted-at least from a definitional perspective-as recognizing bidirectional entailments. @xcite show that computational models underperform in mrpc @xcite with adversarial misspellings, and @xcite present a qualitative analysis of 11 state-of-the-art models (overall accuracies: 68-84%). when negation is present, however, accuracies drop to 33% (6 models) 67% (4 models) and 1% (1 model). finally, @xcite present a dataset for paraphrase identification including adversarial sentence pairs that are not paraphrases but have high word overlap. the new pairs helps training models robust to word scrambling. the aforecited works do not investigate the role of negation in depth. regarding paraphrase identification, previous work only has shown that models underperform with negation. regarding natural language inference and negation, previous work considers negations only in the hypotheses-not the texts. additionally, they only work with unrealistic negations that do not require models to do anything but ignore the negations. indeed, they concatenate tokens including negations cues that are label-preserving and unrelated to the original texts and hypotheses. unlike them, we (a) show #sents. ## negation in english and natural language inference benchmarks negation is pervasive in english @xcite , although there is limited empirical evidence from previous work @xcite . in order to conduct a large-scale analysis and compare how often negation is present in english and existing natural language inference benchmarks, we employ a negation cue detector using a bi-lstm neural architecture with an additional crf layer @xcite . ## a benchmark for natural language understanding with negation we create new benchmarks in which negation plays an important role for natural language inference. the starting points are the original benchmarks, more specifically, we selected at random 500 texthypothesis pairs from rte, snli and mnli (1,500 text-hypothesis pairs total). we work with pairs from the training and development splits as glue and superglue do not include gold labels for some test splits. then, we follow three steps for each of the selected original pairs. in the remaining of the paper, we use t and h to refer to texts and hypotheses in rte, snli and mnli. these steps result in 4,500 new pairs and their judgments (3 per original pair, 1,500 from each rte, snli and mnli). note that the negations are rather simple-adding not to the main verb, and adding auxiliaries and fixing verb tense if neededbut are realistic in the sense that the resulting texts and hypotheses follow proper english grammar. additionally, the new pairs including negation are not more difficult than the original pairs except for the presence of negation. in particular, they do not require additional lexical inference and the overall topic described does not change. t neg : thursday's judge, the honorable charles adams of the coconino county superior court, did not agree, but highly discouraged self-representation. h: self-representation was encouraged by the honorable charles adams. h neg : self-representation was not encouraged by the honorable charles adams. judgments: t-h: contradiction, t neg -h: contradiction, t-h neg : entailment, t neg -h neg : entailment annotation process and agreements. three annotators and an additional adjudicator did the annotations described above in two phases. in the first phase, the three annotators added negation to the main verbs of texts and hypotheses (step 1). after a short training session, we decided to have only one annotator add negation in each original pair as the task is relatively straightforward. any issues in this phase were detected during phase 2. text-hypothesis pairs with issues were discarded (only 5%) and additional pairs were collected to account for the discarded pairs (and still have 1,500 text-hypothesis pairs including negation and generated from each of the three benchmarks, 4,500 new text-hypothesis pairs in total). in the second phase, the three annotators read the new pairs including negation (automatically generated in step 2: t neg -h, t-h neg and t neg -h neg ) and manually labeled them with inference judgments (step 3). in this phase, each pair was annotated by two annotators independently, and the adjudicator resolved any disagreements. we calculated inter-annotator agreement prior to adjudication using cohen's κ @xcite . κ coefficients were 0.85 (rte), 0.81 (snli) and 0.72 (mnli). κ coefficients between 0.6 and 0.8 are considered substantial, and between 0.8 and 1.0 nearly perfect @xcite . label distributions. the original rte, snli and mnli benchmarks contain, by design, texthypothesis pairs with roughly uniform judgment distributions. thus, the majority baseline obtains roughly 50% accuracy in rte (2 labels) and 33% in snli and mnli (3 labels). our new benchmarks including negation do not have a uniform judgment distribution ( the role of negation. ## experiments and results in order to assess whether state-of-the-art systems can solve the task of natural language inference when negation is present, we experiment with three state-of-the art transformers: bert @xcite , xlnet @xcite and roberta @xcite . we use the implementation and pretrained models by @xcite , and tune them to solve each benchmark. the supplemental materials provide details about (a) the hyperparameter settings we use to fine-tune these transformers, and (b) other implementation decisions. we conduct two experiments. first, we assess whether these transformers tuned with the original train splits in rte, snli and mnli are capable of solving our new benchmarks including negation (section 6.1). second, we investigate if tuning with the new text-hypothesis pairs including negation improves the results (section 6.2). ## conclusions negation is ubiquitous in english and critical to understand language and make inferences, as it denies or inverts meaning. despite these facts, negation is underrepresented in some natural language inference benchmarks (rte and snli). additionally, one can ignore negation and still make the correct inference judgment with many text-hypothesis pairs in existing natural language inference benchmarks (rte, snli and mnli). in this paper, we have presented a new benchmark of text-hypothesis pairs containing negation (4,500 pairs). we generate and annotate these pairs after systematically adding negation to the main verb of the texts and hypotheses-either one or both-from rte, snli and mnli thus they are as difficult to solve as the original pairs except for the presence of negation. state-of-the art transformers trained with the original training splits from rte, snli and mnli obtain much worse results results with the new benchmark than with the original pairs-including the few original texthypothesis pairs that do contain negation. in addi-tion, our experimental results show that transformers struggle even after fine-tuning with new pairs containing negation.