Making Language Models Robust Against Negation (2025)

Abstract: Negation has been a long-standing challenge for language models.Previous studies have shown that they struggle with negation in many natural language understanding tasks.In this work, we propose a self-supervised method to make language models more robust against negation.We introduce a novel task, Next Sentence Polarity Prediction (NSPP), and a variation of the Next Sentence Prediction (NSP) task.We show that BERT and RoBERTa further pre-trained on our tasks outperform the off-the-shelf versions on nine negation-related benchmarks.Most notably, our pre-training tasks yield between 1.8% and 9.1% improvement on CondaQA, a large question-answering corpus requiring reasoning over negation.

## introduction negation is a semantic phenomenon that alters an expression to convey the opposite meaning. it is a ubiquitous linguistic construct used in various forms across all languages. negation is present in approximately 25% of english sentences @xcite . language models (lms) achieve remarkable performance in a wide range of natural language understanding tasks but have been shown to struggle when negation is involved @xcite @xcite . @xcite showed that larger lms such as gpt-3 @xcite and instructgpt @xcite are also insensitive to the presence of negation and struggle to reason over it. although larger state-ofthe-art models might be better at handling negation, it is still important to address this issue in smaller language models that are more computationally efficient and cheaper to deploy. @xcite showed that bert @xcite often predicts the same token when negation is added to a sentence. for example, in the sentence a beagle is a type of sentence (s1): the computer screen stayed blank. next sentence original (s2): it didn't display any images. reversed polarity (s ′ next sentence prediction (nsp) input: (s1, s2), label: yes, s2 is the next sentence. input: (s1, s ′ 2 ), label: no, s ′ 2 is not the next sentence. figure 1 : an example of the training data for our self-supervised tasks. the tasks are: (a) given a sentence, predict whether the next sentence will contain negation (nspp) and (b) given two sentences, predict whether the second sentence is a coherent continuation of the first one (nsp). the main contributions of this paper are: foot_0 • introducing two novel self-supervised tasks for pre-training lms for negation. ## related work negation understanding studies have focused on various aspects of negation, including negation detection @xcite @xcite , scope detection @xcite @xcite @xcite , and focus detection @xcite @xcite . recent studies have also focused on improving the performance of lms when negation is present. @xcite introduced affirmative interpretations, i.e., paraphrasing negated sentences without using negation cues. @xcite and @xcite showed that incorporating affirmative interpretations improves the performance of lms on negation-related benchmarks. in this paper, we show that our pre-training strategies outperform affirmative interpretations on condaqa. unlike theirs, our pre-training does not add any complexity at prediction time. @xcite propose to augment negation understanding with bert by unlikelihood training and synthetic data generation. @xcite repeats the pre-training procedure of bert (i.e., mlm and nsp) modified to improve negation understanding. they add negation cues to the original next sentences rather than choosing a random sentence. unlike them, we do not always add negation; instead, we reverse the polarity of the second sentence for the nsp task. that is, half the time we add negation to the next sentence and half the time we remove it. we also introduce a novel task, next sentence polarity prediction (nspp), which is not explored by previous work. as we shall see, our pre-training outperforms previous work. ## empowering language models against negation we propose a self-supervised method to make lms more robust against negation. our approach is to further pre-train lms on two tasks that involve negation. these tasks are the next sentence polarity prediction (nspp) task and a variation of the well-known next sentence prediction (nsp) task. none of these tasks require labeled data; any text corpora are suitable. also, they are not specific to any domain or downstream task. ## evaluation benchmarks we evaluate off-the-shelf lms and versions further pre-trained with our strategies on benchmarks that require reasoning over negation. we primarily evaluate on condaqa @xcite and provide results on eight other relevant benchmarks. all the corpora that we work with are in english. ## experiments first, we further pre-train transformers on the tasks described in section 3. then, evaluate the models on the benchmarks described in section 4. ## results and discussion we first present the results on condaqa and then on the eight other corpora introduced in section 4. ## conclusion in this work, we proposed a self-supervised method to make language models more robust against negation. we introduced two tasks to further pre-train lms: (a) the novel task of next sentence polarity prediction (nspp) and (b) a variation of the next sentence prediction (nsp) task that involves reversing the polarity of the second sentence instead of selecting a random sentence. pre-training data for these tasks can be generated from any text corpus, and the tasks do not require any human annotations. our experiments showed that further pre-training bert and roberta models on these tasks consistently improves their performance on a range of negation-related benchmarks, particularly on the challenging condaqa corpus. notably, while both nspp and nsp tasks are beneficial, the nsp task yields consistently stronger improvements than the challenging nspp task. perhaps surprisingly, training on both tasks jointly does not provide further benefits. future research could explore additional model architectures and pre-training corpora beyond wikipedia or include more contextual information (more than two sentences) in the pre-training tasks.