Probing structural constraints of negation in Pretrained Language Models (2023)

Abstract: Contradictory results about the encoding of the semantic impact of negation in pretrained language models (PLMs) have been drawn recently (e.g. Kassner and Schütze (2020); Gubelmann and Handschuh (2022)).In this paper we focus rather on the way PLMs encode negation and its formal impact, through the phenomenon of the Negative Polarity Item (NPI) licensing in English.More precisely, we use probes to identify which contextual representations best encode 1) the presence of negation in a sentence, and 2) the polarity of a neighboring masked polarity item. We find that contextual representations of tokens inside the negation scope do allow for (i) a better prediction of the presence of “not” compared to those outside the scope and (ii) a better prediction of the right polarity of a masked polarity item licensed by “not”, although the magnitude of the difference varies from PLM to PLM. Importantly, in both cases the trend holds even when controlling for distance to “not”.This tends to indicate that the embeddings of these models do reflect the notion of negation scope, and do encode the impact of negation on NPI licensing. Yet, further control experiments reveal that the presence of other lexical items is also better captured when using the contextual representation of a token within the same syntactic clause than outside from it, suggesting that PLMs simply capture the more general notion of syntactic clause.

## introduction negation has recently been the focus of various works aiming at determining the abilities of pre-trained language models (plms) to capture linguistic knowledge. some works investigate the 'semantic impact' of negation, namely its impact in terms of truth values, by interpreting how the presence of negation impacts the probability distribution at a masked position. the rationale is that negating a verb reverses the truth value of its clause, which should be reflected in the probability distribution at certain positions. @xcite ; @xcite use factual statements such as (1), and report that models output similar distributions for the positive and negative variants of (1), and conclude that models largely ignore negation. gubelmann and handschuh (2022) chose to avoid factual statements and to focus rather on multi-sentence self-contained examples, such that, given the context provided by the first sentence, one particular word is either likely (in positive items) or ruled out (in negative items) at a masked position in the second sentence. because this particular word is substantially less often the top-1 prediction in the negative items than in the positive items, the authors draw the opposite conclusion that plms do show sensitivity to negation. a different line of works focused on finding out to what extent negation is encoded in plm embeddings. @xcite train classifiers taking as input the contextual embedding of a verb or its subject or direct object, and predicting whether the verb is negated or not. the resulting high accuracy allows them to conclude that these tokens' embeddings do contain "traces" of not. more generally, several authors have investigated whether the contextual representation of a token encodes information about surrounding tokens. to ease further reading, we will talk of a classifier taking as input an input embedding, i.e. the contextual representation of an input token, and predicting some target information about another token in the sentence. for instance, @xcite study how input embeddings encode animacy, gender, and number of surrounding words in a specific svo context. @xcite target the number feature of french participles in the context of object-past participle agreement. they show that the performance of the classifier depends on the syntactic position of the input token in the sentence. we will build on their idea to compare performance at predicting target information depending on the syntactic zone the input token belongs to. in this paper, one of the probed target information will be the presence or absence of a given word within the sentence, which we call the target token. more precisely, our aim is to study plms' ability to capture and encode structural information concerning negation (namely negation scope). to do so we first probe whether input embeddings can serve to accurately predict the presence or absence of a target not. foot_0 moreover, we wish to test plms' ability to actually mobilize this encoding to capture phenomena that are direct consequences of the presence of negation. to do so, we focus on the licensing of negative polarity items (npi) by not modifying a verb. polarity items (pi), either positive (e.g. some), or negative (e.g. any), are words or expressions that are constrained in their distribution @xcite . a npi will require that a word or a construction, called the licensor, be in the vicinity. more precisely, the licensor itself grammatically defines a zone of the sentence, called the licensing scope, in which the npi can appear. the adverb not modifying a verb is one such licensor. while any is licensed by negation in (2-a) vs. (2-b), it is not licensed in (2-c), even though the verb is negated, arguably because it is not in the licensing scope 2 . note that our methodology differs from that of @xcite , who, given an input token, predict the zone this token belongs to. we instead predict the polarity of a neighboring masked polarity item and then compare accuracies depending on the input token's zone. our motivation is that the polarity, being a lexical information, requires less linguistic preconception, and hence our probing method is a more direct translation of the npi licensing phenomenon: we study whether and where the information of "which pis are licit where?" is encoded, in the context of sentence negation. this method also allows us to better control the confounding factor of distance between the input embedding and the licensor not. in the following, we define the linguistic notions of negation scope and npi licensing scope in section 2, and show how we actually identified them in english sentences. in section 3, we describe our probing experiments and discuss their results, both for the encoding of not (section 3.1), and the encoding of npi licensing (section 3.2). we then study the more general ability of plms to deal with clause boundaries (section 4), and conclude in section 5. ## (vp (vb*) ( rb not ) np/pp/adjp ) since it is kind of this fairy-tale land, there aren't any rules of logic so you can do anything, she says. ## probing for the scopes our objective is to study how a transformerbased plm (i) encodes the presence of a negation (the "traces" of negation) and (ii) models lexicosyntactic constraints imposed by negation, such as the modeling of a npi licensing scope. us-8 https://parser.yzhang.site/en/latest/index.html ing the terminology introduced in section 1, we probe whether input embeddings encode as target information (i) the presence of not elsewhere in the sentence, and (ii) the polarity of a masked pi. the former focuses on a plain encoding of negation, whereas the latter focuses on whether the encoding of negation can be mobilized to reflect a property (npi licensing) that is directly imposed by negation. to investigate whether such an encoding matches linguistic notions of scopes, we contrast results depending on the zone the input token belongs to (among the four zones defined for not licensing a npi, namely pre, pre-in, in, post) and its distance to not. we studied four plms : bert-base-case, bert-large-case @xcite and roberta-base and roberta-large @xcite . all our experiments were done with each of these models, and for a given model, each experiment was repeated three times. all the sentences we used for training, tuning and testing were extracted from the coca corpus. ## probing clause boundaries we have seen that plms are able to encode negation scope, however this notion of scope often simply corresponds to the notion of syntactic clause. so it might be the case that plms are mainly sensitive to clause boundaries and that this sensitivity is the unique/main source of plms ability to encode negation scope. in this section we report a number of experiments designed to assess plms ability to encode clause boundaries in general. we chose to use the same setting as the one we used with the neg-classifiers (section 3.1.1). instead of using not as a target token, we chose various tokens with a similar number of occurrences, but other poss: often, big, house, wrote. we trained classifiers to predict whether the target token is in the neighborhood of the input token. this time, the objective is to compare these classifiers' accuracies depending on whether the input token is or isn't in the same clause as the target token (instead of whether the input token is within or outside the negation scope). and just as we did for the neg-classifiers, we will control for distance to the target token by breaking down the accuracies according to the distance between the target and the input tokens. ## conclusion in this paper, we studied the way negation and its scope are encoded in contextual representations of plms and to what extent this encoding is used to model npi licensing. we trained classifiers to predict the presence of not in a sentence given the contextual representation of a random input token. we also trained classifiers to predict the polarity of a masked polar item given the contextual representation of a random token. a test set of sentences was designed with not licensing an npi, inside which we identified the negation scope , and the licensing scope. for these sentences, we found that the contextual embeddings of tokens within the scope of a negation allow a better prediction of the presence of not. these embedding also allow a better prediction of the (negative) polarity of a masked pi. these results hold even when controlling for the distance to not. the amplitude of this trend though varies across the four plms we tested. while this tends to indicate that plms do encode the notion of negation scope in english, and are able to further use it to capture a syntactic phenomena that depends on the presence of not (namely the licensing of a negative polarity item), further experiments tend to show that what is captured is the more general notion of clause boundary. indeed, negation scope is closely related and often amounts to negation scope. using alternative target tokens with varied parts-of-speech, we find that classifiers are better able to predict the presence of such target tokens when the input token is within the same syntactic clause than when it is outside from it. these results lead us to conclude that knowledge of the negation scope might simply be a special case of knowledge of clause boundaries. moreover, distance to the target token is way stronger a factor than the "being in the same clause" factor. we leave for further work the study of other factors, such as the pos of the input token, as well as the study of the differences in amplitudes observed between the plms we tested.