Understanding by Understanding Not: Modeling Negation in Language Models (2021)

Abstract: Negation is a core construction in natural language. Despite being very successful on many tasks, state-of-the-art pre-trained language models often handle negation incorrectly. To improve language models in this regard, we propose to augment the language modeling objective with an unlikelihood objective that is based on negated generic sentences from a raw text corpus. By training BERT with the resulting combined objective we reduce the mean top 1 error rate to 4% on the negated LAMA dataset. We also see some improvements on the negated NLI benchmarks.

## introduction negation is an important property in many language understanding tasks, such as sentiment analysis, question answering, knowledge base completion and natural language inference @xcite . while pretrained language models (plms) such as bert pushed the state-of-the-art on these tasks @xcite , they fail dramatically on instances that require understanding negation. @xcite show that current plms cannot correctly distinguish between the negated and non-negated forms of fill-in-the-blank tests. for instance, when asked to predict the @xcite token in sentences such as "the capital of cuba is [mask]" and "the capital of cuba is not [mask]", bert often generate the same answer "havana", indicating that it may not be appropriately modeling the distribution of negative sentences. additional evidence is given by the fact that, when fine-tuned on natural language inference tasks, plms tend to mis-classify examples which figure 1 : an overview of the unlikelihood objective. a generic sentence is negated using our data augmentation method and an unlikelihood token is chosen and replaced with @xcite . this new sentence is concatenated with the original sentence and fed into the model. the unlikelihood loss is computed using p(improvements) from the language modeling head of bert. in this work, we investigate whether we can alleviate the modeling bias of plms on negated sentences. our approach is composed of two core contributions: i) a syntactic data augmentation scheme to automatically generate negated sentences; ii) a new training paradigm, dubbed unlikelihood training with reference (fig. 1 ), based on the recently proposed unlikelihood training @xcite . at first, we generate a large number of negated sentences by negating sentences mined from an openly available text corpus (wikipedia). our sentence negator uses the dependency parse of the sentence, part of speech tags, and morphological features of each word in the sentence and deterministically negates the sentence. given a negated version of a sentence, we replace its object with the [mask] token and use unlikelihood training to make the object unlikely under the plm distribution (e.g. we minimize the probability of "improvements" as depicted in fig. 1 ). importantly, in order to ensure that the negated sentence is factually false, we use the positive sentence as context (i.e., as a reference) for the unlikelihood prediction task. concretely, we provide the concatenation of the positive and the masked negated sentence as input to the plm. our method can be thought of a type data augmentation, which has be shown to be effective at improving robustness across many tasks in language, such as text classification @xcite , natural language inference @xcite and semantic parsing @xcite . for our negation experiments, we fine-tune pretrained bert with our new objective and a knowledge distillation objective. we test our model on the negated lama dataset @xcite , which is the negated version of knowledge probing dataset lama, introduced in @xcite . our model achieves a mean error rate of 4% (a improvement of 5 points) on the negated lama dataset while maintaining the performance on the original lama dataset without any direct training on the negated lama sentences. we also finetune bert on rte @xcite @xcite , @xcite and mnli @xcite tasks and achieve better results on the language inference benchmark including negation from @xcite . ## related work pre-trained language models have shown impressive results across many tasks, such as question answering @xcite and natural language inference @xcite . these models are also known to encode factual and common-sense knowledge @xcite @xcite . despite these abilities, @xcite found that these models fail at understanding negation through analysing negated factual statements. extensive literature looks at the linguistic knowledge learned by language models @xcite @xcite @xcite @xcite . recent work has also studied the shortcomings in negation scope detection @xcite @xcite @xcite @xcite and focus detection @xcite @xcite . @xcite al. (2018) and mccoy et al. (2019) systematically study the linguistic abilities of these models using nli, and show that these models rely on erroneous syntactic heuristics. our work is in this spirit for negations. @xcite propose taking advantage of negative examples and unlikelihood in the training of language models to increase their syntactic abilities. similarly, @xcite show the effectiveness of syntactic data augmentation in the case of robustness in nli. neither of these works focus on negations. ## syntactic negation augmentation we generate the negated versions of sentences using a syntactic augmentation method. the method gets as input the dependency parse of the sentence, pos tags and morphological information of each word and negates the sentence using a set of rules. each rule has a dependency tree regular expression pattern (semgrex; @xcite . we use semgrex patterns to identify different syntactic templates, and then transform the sentence based on a list of actions defined in the rule. these actions can be move, replace, insert and lemmatize. the unlikelihood token which will be discussed later is also chosen using semgrex patterns (see appendix c for some examples). we use stanza @xcite to get the dependency parse of the sentences, parts of speech tags, lemma, and morphological features of the words. we also filter out sentences with more than 20 words. to test the coverage of our semgrex patterns, we randomly sampled 930 sentences from wikipedia. only 31 of them did not match any of our semgrex patterns (see ## experiments we report our main results on lama and negated lama for knowledge base completion. the cloze statements from lama are facts or commonsense knowledge generated from either subject-relationobject triples (x, rel, y) or question-answers pairs. the cloze statements for the triples are generated using a template for each relation which includes the placeholders x and y (e.g. "x is located in y"). x is replaced for the subject and y is replaced with the [mask] token to be predicted by the model. in the question-answer pairs, the answer is replaced with [mask] token. the facts in the lama dataset are from multiple sources: 1) google-re relations, namely "place of birth", "date of birth" and "place of death"; 2) t-rex, a subset of wikidata triples with 41 relations @xcite ; 3) conceptnet with 16 relations @xcite ; 4) squad, a subset of 305 contextinsensitive questions manually rephrased as clozestyle questions @xcite . negated lama was created by manually negating the templates or questions @xcite . following @xcite we use mean precision at k (p @ k) for lama. for negated lama we report mean top 1 error rate. ## conclusion in this work, we propose a combination of the unlikelihood objective with a reference based setup for input sentences to model negation. this allows us to utilize generic sentences, and negate them with our data augmentation method to be used as examples for the unlikelihood objective. our method notably improves the error rate on the negated lama dataset while keeping the same performance on the original lama queries. we also test our method on the original development sets and new splits containing negation from hossain et al. (2020b) of rte, snli and mnli tasks. we see large improvements on the negated splits in low-resource setting (rte) and slight improvements in high-resource setting (snli and mnli), while also maintaining similar results as bert on original splits.