The Self-Contained Negation Test Set (2023)

Abstract: Several methodologies have recently been proposed to evaluate the ability of Pretrained Language Models (PLMs) to interpret negation. In this article, we build on Gubelmann and Handschuh (2022), which studies the modification of PLMs’ predictions as a function of the polarity of inputs, in English. Crucially, this test uses “self-contained” inputs ending with a masked position: depending on the polarity of a verb in the input, a particular token is either semantically ruled out or allowed at the masked position. By replicating Gubelmann and Handschuh (2022) experiments, we have uncovered flaws that weaken the conclusions that can be drawn from this test. We thus propose an improved version, the Self-Contained Neg Test, which is more controlled, more systematic, and entirely based on examples forming minimal pairs varying only in the presence or absence of verbal negation in English. When applying our test to the roberta and bert base and large models, we show that only roberta-large shows trends that match the expectations, while bert-base is mostly insensitive to negation. For all the tested models though, in a significant number of test instances the top-1 prediction remains the token that is semantically forbidden by the context, which shows how much room for improvement remains for a proper treatment of the negation phenomenon.

## introduction the treatment of negation by plms has recently been the subject of various works whose conclusions are fairly contradictory. on the one hand, kassner and schütze (2020) and @xcite compare the predictions of transformer-based language models @xcite in minimal pairs varying in polarity (1). noting that changes of polarity in the model's inputs result in little or no change for both top-1 predictions and the entire vocabulary distribution, these authors conclude that the models are insensitive to negation. however, it has been established that the presence of negation can be detected in contextual representations. @xcite thus find "traces" of negation on the negated verb, its subject, its object. moreover, the extent to which negation is diffused in contextual representations follows syntactic constraints: @xcite show that the presence of negation in contextual representations is stronger for tokens within the scope of negation, this effect being visible even when controlling for the distance between the token and negation. as pointed out by @xcite , this apparent contradiction can be explained by the fact that @xcite study the factual knowledge of models, and therefore use contexts involving world knowledge, such as (1). the inability of models not to predict bird in the negated case could be explained by stored factual knowledge taking precedence over the ability to capture that negation reverses the truth value of a proposition. especially as there is an asymmetry in the number of acceptable words to replace the mask: only a few are possible for the positive version, but a huge number are for the negative one, which can't be a favorable situation when only the top-1 prediction is studied. @xcite have thus proposed a test where the inputs supplied to the models are self-contained (in our terminology): a context sentence is followed by a target sentence containing a masked position. the context sentence is either negative or affirmative. in the negative case, it renders semantically impossible a certain token at the masked position (sail in example (2)), which is itself plausible in the positive case (see section 2.2). jessica is an architect who doesn't like to sail. however, she does like to @xcite . @xcite observe a variable sensitivity to negation depending on the models tested, suggesting that the truth-value inversion effect of negation is more or less captured depending on the model. in this article, we build on @xcite (hereafter gh22), taking up the idea of self-contained inputs, allowing us to target understanding of the semantics of negation independently of world knowledge. our contributions are the following: in particular, averaged results for different input patterns mask significant sensitivity to factors other than negation (e.g., an intensifier really or does). finally, this test enables us to make a detailed assessment of four models, and to conclude that among these, only roberta-large reasonably meets the defined criteria. crucially, a number of models like bert-large seemed reasonably sensitive to negation in gh22, do pass our baseline test, but don't pass the control tests at all, calling into question the positive interpretation of the baseline test. this highlights the many limitations to plms' understanding of negation for english, and the need for highly controlled tests to reach solid conclusions. ## additional controls: forcing non-coreference in the results analyzed in the previous section, the drop can only be interpreted as an understanding of negation if the model has resolved the co-reference between the proper noun in c and the pronoun in t. in the absence of such a resolution, a repetition of the act is neither forbidden nor required. while the ability of bert to resolve coreference has been evidenced by @xcite , we need to ensure that this resolution is effective in the case of our patterns. to do this, instead of directly testing coreference resolution, we build a set of alternative examples to the base examples, in which non-coreference is forced. in practice, we replace the pronoun in t by a proper noun other than the one used in c, with two variants, depending on whether or not these two proper nouns have the same gender (cf. examples 1 and 2 we also consider cases where we help the model establish a co-reference, so as to test only the impact of negation, independently of the models' ability to establish the co-reference between the proper noun and the pronoun in the basic examples. to this end, we use the same proper noun in c and t (cf. example 3 triplets are selected using the same procedure as in section 3.2, namely retaining only triplets leading to a top-1 repetition for the cptp pattern, and at most 20 verbs for a given [name,prof] pair. results for roberta models are provided in an efficient model is expected to obtain small drops for the non-coref columns, while retaining we can see this is not the case for the roberta-base model: the drops in the upper left part of it is as if the model interpreted the same two names as non-coreferent. it is though impossible to conclude whether this is the case (in which case the smaller drops do not mean that negation is misunderstood), or whether the model interpreted the coreference correctly, but failed to interpret negation. on the other hand, the trends observed for roberta-large ( ## conclusion in this paper we propose a methodology and a dataset to study plms' abilities to correctly interpret the semantics of negation, more precisely verbal negation in english. we were inspired by @xcite , who proposed self-contained examples, consisting of two sentences, the first serving as a context that favors or hinders the repetition of a certain verb in the second sentence. after critically analyzing this test, we propose an improved version, which is more controlled, more systematic, and entirely based on examples forming minimal pairs varying only in the presence or absence of verbal negation. we have sought to minimize the interpretations that the models have to make in addition to the negation interpretation, so that the observed results can be more reliably interpreted as the model's good or bad "understanding" of negation. we applied our test to four pretrained transformer-based language models. a detailed analysis of the results shows a continuum of situations: bert-base is globally unable to take verbal negation into account, bert-large is a little better at first sight, but the control tests we added show its limitations. roberta-base partially passes the basic test, but is disappointing when it comes to controlling co-reference resolution. only the roberta-large model shows trends in line with expectations, for both base and control patterns, clearly showing some ability to capture the semantics of verbal negation in english. however, for all the models we tested, a significant number of examples get a top-1 prediction that is exactly the token semantically forbidden by the context. this shows how much room for improvement remains for this type of models. we chose to focus on verbal negation, being the most frequent form of negation in english, but we plan to extend our test to other forms of negation. extension to other languages is also considered.