An Analysis of Negation in Natural Language Understanding Corpora (2022)

Abstract: This paper analyzes negation in eight popular corpora spanning six natural language understanding tasks. We show that these corpora have few negations compared to general-purpose English, and that the few negations in them are often unimportant. Indeed, one can often ignore negations and still make the right predictions. Additionally, experimental results show that state-of-the-art transformers trained with these corpora obtain substantially worse results with instances that contain negation, especially if the negations are important. We conclude that new corpora accounting for negation are needed to solve natural language understanding tasks when negation is present.

## introduction natural language understanding (nlu) is an umbrella term used to refer to any task that requires text understanding. for example, question answering @xcite , information extraction @xcite , coreference resolution @xcite , and machine reading @xcite , among many others, are tasks that fall under natural language understanding. the threshold for claiming that a system understands natural language is ever-moving. new corpora are often justified by pointing out that state-of-the-art models do not obtain good results. after years of steady improvements, more powerful models eventually obtain so-called human performance, and at that point new, more challenging corpora are created. many corpora for natural language understanding tasks contain language generated by annotators rather than retrieved from texts written independently of the corpus creation process. these corpora are certainly useful and have facilitated tremendous progress. annotator-generated examples, however, carry the risk of evaluating systems with synthetic language that is not representative of language in the wild. for example, annotators are likely to use negation when asked to write a text that contradicts something despite contradictions in the wild need not have a negation @xcite . recently, @xcite present a large corpus for question answering that consists of natural questions (i.e., asked by somebody with a real information need) in order to encourage research in a more realistic scenario. this contrasts with previous corpora, where the questions were written by annotators after being told the answer @xcite . in this paper, we explore the role of negation in eight corpora for six popular natural language understanding tasks. our goal is to check whether negation plays the role it deserves in these tasks. to our surprise, we conclude that negation is virtually ignored by answering the following questions: foot_0 1. do nlu corpora contain as many negations as general-purpose texts? (they don't); 2. do the (few) negations in nlu corpora play a role in solving the tasks? (they don't); and 3. do state-of-the-art transformers trained with nlu corpora face challenges with instances that contain negation? (they do, especially if the negation is important). ## background and related work we work with the eight corpora covering six tasks summarized below and exemplified in for textual similarity and paraphrasing, we select qqp foot_1 and sts-b @xcite . qqp consists of pairs of questions and the task is to determine whether they are paraphrases. sts-b consists of pairs of texts and the task is to determine how semantically similar they are with a score from 0 to 5. we select one corpus for the remaining tasks. for inference, we work with qnli @xcite , which consists in determining whether a text is a valid answer to a question. we use wic (pilehvar and camacho-collados, 2019) for word sense disambiguation. wic consists in determining whether two instances of the same word (in two sentences; italicized in for convenience, we work with the formatted versions of these corpora in the glue @xcite and superglue @xcite benchmarks. the only exception is commonsenseqa, which is not part of these benchmarks. related work previous work has shown that snli @xcite and mnli @xcite have annotation artifacts (e.g., negation is a strong indicator of contradictions) @xcite . the literature has also shown that simple adversarial attacks including negation cues are very effective @xcite . @xcite analyze 11 paraphrasing systems and show that they obtain substantially worse results when negation is present. more recently, @xcite show that negation is one of the linguistic phenomena commercial sentiment analysis struggle with. several previous works have investigated the (lack of) ability of transformers to make inferences when negation is present. for example, ettinger (2020) conclude that bert is unable to complete sentences when negation is present. bert also faces challenges solving the task of natural language inference (i.e., identifying entailments and contradictions) with monotonicity and negation @xcite (2019) show the limitations of bert making acceptability judgments with sentences that contain negative polarity items. most related to out work, hossain et al. ( 2020 ) analyze the role of negation in three natural language inference corpora: rte @xcite @xcite , snli and mnli. in this paper, we present a similar analysis, but we move beyond natural language inference and work with eight corpora spanning six natural language understanding tasks. ## research questions and analysis q1: do natural language understanding corpora contain as many negations as generalpurpose english texts? in order to automatically identify negation cues, we train a negation cue detector with the largest corpus available, conandoyle-neg @xcite . the cue detector is based on the roberta pretrained language model ( liu et al., 2019 ); we provide details about the architecture and training process in appendix a. our cue detector obtains the best results to date: f1: 93.79 vs. 92.94 (khandelwal and sawant, 2020). conandoyle-neg (and thus our cue detector) identifies common negation cues such as no, not, n't and never, affixal negation cues such as impossible and careless, and lexical negations such as deny and avoid. example important? cmmsnsqa [. . . ] he (john) never saw the lady before. they were what? c ✓ a) pay debts, b) slender, c) unacquainted, d) free flowing, e) sparse when you travel you should what in case of unexpected costs? e ✗ a) go somewhere, b) energy, c) spend frivilously, d) fly in airplane, e) have money qqp what are some not-so-boring baby shower games ? yes ✓ what are some baby shower games that are actually fun? who was philosophical guru of shivaji maharaj? no ✗ what are the unknown facts of shivaji maharaj? sts-b colin powell, the secretary of state, said contacts with iran would not stop. 4.3 ✓ secretary of state colin powell said yesterday that contacts with iran would continue. well for one a being could have a non-physical existance and yet not even be in your mind. ## conclusions we have analyzed the role of negation in eight natural language understanding corpora covering six tasks. our analyses show that (a) all but wsc contain almost no negations or around 31%-54% of the negations found in general-purpose texts, (b) the few negations in these corpora are usually unimportant, and (c) roberta obtains substantially worse results when negation is important. our analyses also provide some evidence that creating models to properly deal with negation may require both new corpora and more powerful models. the need for new corpora stems from the answers to questions 1 and 2. the justification for powerful models is more subtle. we point out that the percentage of unimportant negations (section 3) is only a weak indicator of the drop in results with important negations ( negation reverses truth values thus solutions to any natural language understanding task should be robust when negation is present and important. to this end, our future work includes two lines of research. first, we plan to create benchmarks for the six tasks consisting of instances containing negation (50/50 split important/unimportant). second, we plan to conduct probing experiments to investigate whether (and where) pretrained transformers capture the meaning of negation. doing so may help us discover potential solutions to understand negation and make inferences.