Not another Negation Benchmark: The N a N - NLI Test Suite for Sub-clausal Negation (2022)

Abstract: Negation is poorly captured by current language models, although the extent of this problem is not widely understood. We introduce a natural language inference (NLI) test suite to enable probing the capabilities of NLP methods, with the aim of understanding sub-clausal negation. The test suite contains premise–hypothesis pairs where the premise contains sub-clausal negation and the hypothesis is constructed by making minimal modifications to the premise in order to reflect different possible interpretations. Aside from adopting standard NLI labels, our test suite is systematically constructed under a rigorous linguistic framework. It includes annotation of negation types and constructions grounded in linguistic theory, as well as the operations used to construct hypotheses. This facilitates fine-grained analysis of model performance. We conduct experiments using pre-trained language models to demonstrate that our test suite is more challenging than existing benchmarks focused on negation, and show how our annotation supports a deeper understanding of the current NLI capabilities in terms of negation and quantification.

## introduction negation is an important linguistic phenomenon which denotes non-existence, denial, or contradiction, and is core to language understanding. nlp work on negation has mostly focused on detecting instances of negation @xcite @xcite , and the effect of negation on downstream or probing tasks @xcite @xcite . a consistent finding in recent work on pre-trained language models (plms) is that they struggle to correctly handle negation, but also that existing nlp benchmarks are deficient in terms of their relative occurrence and variability * equal contribution of negation @xcite @xcite . in this work, we address the problem of evaluating the ability of models to handle negation in the english language using a systematic, linguisticallybased approach. specifically, we adopt the typology proposed by @xcite whereby negation is classified based on both form (verbal and non-verbal; analytic and synthetic) and meaning (clausal and sub-clausal; ordinary and meta-linguistic). based on this typology, we observe that most negation instances occurring in existing benchmarks are analytic, verbal, and clausal, which is arguably more straightforward to handle than non-verbal, synthetic, and sub-clausal negation. for instance, the dataset proposed by @xcite is constructed by adding the syntactic negation cue not to the main verb of the premise and/or the hypothesis of mnli @xcite training examples, resulting almost exclusively in verbal, analytic, and clausal negation. motivated by this, we construct a new evaluation dataset with a focus on sub-clausal negation, where it is non-trivial to determine the correct negation scope. for instance, the negation in they saw not one but three dolphins only scopes over the modifier one, and thus carries a positive meaning (they saw three dolphins). we choose nli as the probing task based on the intuition that a complete grasp of negation is required to make correct inference judgements. moreover, we adopt the test suite framework @xcite instead of naturally-occurring text corpora, to elicit a full range of linguistic constructions that denote sub-clausal negation. this facilitates systematic evaluation of model performance along controlled dimensions. we collect examples for each construction from @xcite to use as premises, and then construct corresponding hypotheses by introducing minimum changes to premises which highlight their possible interpreta-tions. we manually annotate the constructed pairs in terms of negation types, negation constructions, and the operations used to construct the hypotheses. in summary, our main contributions are: ## related work to investigate the abilities of plms to assign the correct interpretation to negation, many probing tasks have been proposed. for instance, kassner and schütze (2020); ettinger (2020) formulated a cloze-style fill-in-the-blank task where bert is asked to predict words for two near-identical but contrasting sentences (e.g. a bird can vs. a bird cannot ). hossain et al. (2020) constructed an nli dataset where negations essential to correctly judge the label for a premise-hypothesis pair were manually added to existing nli benchmarks. hartmann et al. (2021) constructed a multilingual dataset with minimal pairs of nli examples to analyze model behavior in the presence/absence of negation. most recently, hossain et al. (2022) the idea of using a test suite to measure the performance of nlp models was introduced by @xcite , where the authors propose general guidelines for test suite construction. adopting this methodology for a domain-specific task, @xcite constructed a dataset for benchmarking ontology concept recognition systems. most recently, ribeiro et al. (2020) proposed a task-agnostic testing methodology which closely follows the idea of behavioral testing from software engineering to comprehensively test the linguistic capabilities of nlp models. the main advantages of test suites over datasets made up of naturallyoccurring examples are: (1) control over the precise composition of the data: we can undertake a targeted evaluation of specific criteria (e.g. linguistic features); (2) systematicity: a test suite has specific structure, with samples classified into welldefined categories; and (3) control of redundancy: we can remove samples with similar properties or over-sample rare occurrences. ## discussion we further investigate the results of the best performing model roberta-mnli-negnli in detail to explore potential patterns in the model's predictions on our test suite. ## conclusion in this work, we proposed a new test suite, dubbed nan-nli, for probing the performance of nlp models on data containing sub-clausal negation. in addition to standard nli labels, we also annotated the test suite using a systematic linguistic framework. nan-nli facilitates extensive analysis of negation instances based on their negation and construction type. extensive experiments show that our test suite is significantly harder for existing models than existing benchmarks, and reveal the limited capabilities of pretrained language models in dealing with this type of negation. detailed analysis of the results reveals a class of negations that are particularly challenging, namely those involving quantifiers, showing that our test suite can also be used as a resource to evaluate the upper and lower bounds of quantification.