NPI s Aren’t Exactly Easy: Variation in Licensing across Large Language Models (2023)

Abstract: We examine the licensing of negative polarity items (NPIs) in large language models (LLMs) to enrich the picture of how models acquire NPIs as linguistic phenomena at the syntax-semantics interface. NPIs are a class of words which have a restricted distribution, appearing only in certain licensing contexts, prototypically negation. Unlike much of previous work which assumes NPIs and their licensing environments constitute unified classes, we consider NPI distribution in its full complexity: different NPIs are possible in different licensing environments. By studying this phenomenon across a broad range of models, we are able to explore which features of the model architecture, properties of the training data, and linguistic characteristics of the NPI phenomenon itself drive performance.

## introduction negative polarity items (npis) are words or phrases that must be licensed by another element, often negation, that occurs in a syntactically appropriate context @xcite . determining the contexts in which such elements are possible has proven to be a difficult problem for language models. @xcite report that lstm language models fail to systematically distinguish grammatical from ungrammatical occurrences of npis across a range of difficult cases. @xcite study bert's ability to determine the possibility of an npi occuring in a masked position and find improved performance, though success depends upon the mode of evaluation. @xcite report even better results with gpt-2 and gpt-2-xl. finally, @xcite show that with sufficiently many parameters and enough training data, as found in roberta-large @xcite , a transformer language model can achieve nearhuman levels of performance on npi licensing. though this might be thought of as a success story for llms, we may still wonder why this particular grammatical regularity has posed such difficulty, as compared to subject-verb agreement, where smaller lstms trained on less data achieved quite good performance @xcite . we suspect that the reasons for this are threefold. first, unlike grammatical subjects that condition agreement on their corresponding verbs in reasonably predictable ways, the set of contexts that license npis and the range of npis are both rather diverse. further, while every instance of a finite clause will include a subject and agreeing verb, many contexts that could license npis do not include one. finally, npi licensing and subjectverb agreement are both dependencies that are unbounded by linear distance; however, the structural distance between a licensing context and an npi can grow without bound (cf. i don't see anyone and i don't want to try to see anyone), unlike the dependency that determines subject-verb agreement. previous studies of llm performance on npis, including @xcite and @xcite , have examined the first of these factors: the variability of the licensing environment. specifically, these studies explored the degree to which the licensing properties of distinct environments are encoded uniformly, with what look like reasonably promising results. such work assumes implicitly that different environments should be treated identically (though see @xcite for work that does not make this assumption). similarly, llm evaluations on npis have assumed that different npis are licensed in identical environments. however, as we will discuss in section 2, these assumptions are false: different environments license different npis. learning the distribution of npis is thus more complex than previous llm evaluations have assumed. we aim to develop an approach to evaluate llms' knowledge of npis in a way that is sensitive to their unique distributional patterns, and to uncover what factors lead to greater success in a model's ability to correctly determine the possibility of an npi in a given context. ## variability in npi licensing as already noted, npis are expressions that are only grammatical in a restricted set of contexts, prototypically understood to be negative. canonical examples of such contexts include the negative quantifiers no or none of the, or sentential negation not. as seen in (1), the english npi ever is possible when it is in the scope of such an element, and ungrammatical otherwise. no/none of the packages had ever arrived at the yellow house. b. packages had not ever arrived at the yellow house. c. * packages had ever arrived at the yellow house. as seen in (2), the npi ever is also licensed by other contexts, including (indirect) yes/no questions, the restrictor of superlatives, and under the scope of only, among many others. these are the greatest packages that had ever arrived at the yellow house. c. only packages had ever arrived at the yellow house. a major step forward in our understanding of the distribution of npis came from attempts to characterize these licensing environments in a uniform fashion @xcite . however, it was quickly observed that not all npis are licensed by the same contexts. for example, the english npi (adverbial) any is licensed by negation and indirect yes/no questions but not by superlatives. no masons build cathedrals any better than that. b. * these are the greatest masons that build cathedrals any better than that. c. i wonder whether the masons have built a cathedral any better than that. npis like exactly are even more restrictive, occurring only with negation: none of the students have exactly been getting good grades. b. * these are the smartest students that have exactly been getting good grades. recent research in formal semantics has aimed to understand this variation. under the proposal of zwarts (1998), which was further refined in @xcite , licensing contexts are characterized according to their semantic properties. zwarts and giannakidou provide four increasingly demanding semantic criteria for characterizing contexts, each of which entails the previous one. such a semantic classification allows us to characterize the distribution of different npis. each npi is associated with a certain minimal requirement on its licensing context, and will therefore be allowed in all more restrictive contexts. this creates a hierarchy of npis, ranging from superweak npis, which require environments satisfying only the weakest condition, to superstrong npis, which are require environments satisfying the strongest condition. elegant as this approach is, hoeksema (2012) shows that this classification is not completely adequate, as it does not capture the full complexity and diversity in the distribution of different npis. ## discussion our results paint a complex picture, where both model size and the number of licensing contexts of a given npi contribute to higher correlations with human judgments. nevertheless, we observe substantial variation in the correlations between model and human preferences across npis and contexts. for the results by npi, we see relatively consistent positions of npis across different model architectures when considering their relative relationships with the correlations, indicating that interesting structural patterns exist within the class of npis. while some aspects of traditional npi theory are reflected, namely the number of licensing contexts for a given npi (which is informative about its relationship to other npis), there is still much complexity that does not fit in with the specifics of npi theory. for example, exactly is consistently among the npis on which the models perform best, despite being licensed in only one of the contexts we consider. this indicates that the behavior of the llms we considered does not fully capture the distinctions relevant to the licensing of npis proposed in traditional linguistic theories. it is also possible that because exactly is not licensed by any contexts other than negation, its licensing conditions are easier to learn. in other words, exactly is such a strong npi in a prototypical sense that models may find it easier to distinguish the context that licenses it from the contexts that do not in comparison to npis that are subject to greater complexity. although much contemporary research has shown that larger llms trained on larger datasets tend to exhibit better performance compared to smaller models trained on smaller datasets, our findings make us less sanguine about the prospect of big models and big data leading to fully human-like linguistic behavior. while the models do acquire a degree of knowledge pertaining to npi licensing contexts, many of the subtleties are lost. it is plausible that within this domain, the notion that larger llms are inherently superior may not hold true at the level of detail we investigate. additionally, it is worth noting that the context frequency in natural text seems to be related to model performance in some way, though a more extensive investigation may better distinguish its effects from the effects of licensing number and model features. the elevated performance observed in conditional contexts across all three model types may be plausibly attributed to the syntactic characteristics of this licensing environment. specifically, the use of the word if serves as a distinguishing marker for a conditional, while other contexts may be identified only by more abstract structural properties. this easy-to-identify distinguishing feature may render proficiency in this licensing construction relatively more obtainable. in future research, more robust npi theory could provide additional explanatory power for understanding the relationship that llms learn about npis and their licensing environments. ## conclusion we investigated npi licensing in llms by analyzing the similarities between model and human judgments and their relationship with certain linguistic and model features. analysis by npi reveals a significant positive relationship between both model size and model performance, as well as between licensing number and model performance. however, analysis by licensing context reveals that larger llms may not be inherently better than smaller llms at particular levels of granularity and that model performance may not be influenced by all of the anticipated factors in the most intuitive way. additionally, we have determined hierarchies among npis and licensing contexts, which provide a broader perspective on npi licensing across model tasks. several patterns emerged: while traditional semantic classifications of npis were not reflected, a key feature, namely the number of contexts that license a given npi, does appear to have an impact on the hierarchies, though with some clear exceptions. similarly, hierarchies among licensing contexts may be influenced by the syntactic characteristics of the environments. this complex situation seems to reflect the complexity of npis, which are linguistically heterogeneous.