Probing Quantifier Comprehension in Large Language Models: Another Example of Inverse Scaling (2023)

Abstract: With their increasing size, large language models (LLMs) are becoming increasingly good at language understanding tasks. But even with high performance on specific downstream task, LLMs fail at simple linguistic tests for negation or quantifier understanding. Previous work on quantifier understanding in LLMs show inverse scaling in understanding few-type quantifiers. In this paper, we question the claims of of previous work and show that it is a result of inappropriate testing methodology. We also present alternate methods to measure quantifier comprehension in LLMs and show that LLMs are able to better understand the difference between the meaning of few-type and most-type quantifiers as their size increases, although they are not particularly good at it. We also observe inverse scaling for most-type quantifier understanding, which is contrary to human psycho-linguistic experiments and previous work, where the model’s understanding of most-type quantifier gets worse as the model size increases. We do this evaluation on models ranging from 125M-175B parameters, which suggests that LLMs do not do as well as expected with quantifiers. We also discuss the possible reasons for this and the relevance of quantifier understanding in evaluating language understanding in LLMs.

## introduction large language models (llms) are getting increasingly better at understanding language @xcite @xcite @xcite which can be seen by their improving performance on various language understanding benchmarks @xcite . auto-regressive llms including encoder-decoder models like bart @xcite and t5 @xcite and decoder-only models like * this work was done while at ai research, jpmorgan. gpt @xcite @xcite @xcite have been scaled to billions of parameters to improve their language understanding capabilities. with increasing model sizes, the models also gets increasingly better at learning from context and can just be prompted with few examples rather than fine-tuning to do downstream task @xcite . even with this unprecedented yet implicit evidence of increasing language understanding capability of llms, these models still fail simple linguistic tests on understanding negation and quantifiers @xcite @xcite . understanding negation and quantifiers is challenging for language models because the presence of a single negating or quantifying word can drastically change the meaning of the sentence. also, such sentences are infrequently used in pre-training text corpora @xcite , which makes it hard for the models to account for such situations. due to this, actual comprehension of negation or quantifier words is overpowered by the larger context of the sentence, which makes it challenging for language models to deal with these situations. we focus on one specific linguistic phenomenon, which is the use of quantifiers. quantifiers are words that usually occur before a noun to express the quantity of an object @xcite . the presence of different quantifiers can make statements semantically very different from each other. it can be seen from the following example: in polar opposite meanings as can be seen on the right side of respective equations. all ps are qs means that all objects in the set p belong to the set q, whereas no ps are qs means that p and q are mutually exclusive sets. this minor distinction in the sentence has a drastic effect on the relationship between p and q. in this work, we aim to test and quantify the ability of llms to understand quantifiers and how this understanding changes as the models scale. we build upon the work of @xcite , who test understanding and sensitivity of llms for most-type and few-type quantifiers. they do these tests on a dataset of 960 sentences created using a previously published study on human response (measured using n400 amplitude) to different quantifiers (urbach and kutas, 2010). they find that while llms do increasingly well on understanding most-type quantifiers, while their understanding of few-type quantifiers diminishes as the size of these language models increase. this is an example of an inverse-scaling law @xcite , where the model gets worse at doing a task as the model size increases. inverse scaling laws are rare in natural language processing and important to identify, yet they must be cautiously evaluated @xcite . in this paper, we first show that conclusions about the inverse-scaling of few-type quantifier comprehension in llms @xcite need to be revisited because of a possibly faulty methodology, thus leading to a wrong conclusion about inverse-scaling. we discuss the reasons for this in detail later in the paper. we then propose our own method of measuring quantifier comprehension in llms. we find that llms are able to differentiate between sentences that contain mosttype versus few-type quantifiers quite well and this understanding improves as the model size increases. we measure this by quantitatively evaluating if the models react differently for different types of quantifiers. although, when we evaluate if the model takes into account the meaning of a quantifier, we find that llms comprehend few-type quantifiers much better than most-type quantifiers. we also find that contrary to the results of (michaelov and bergen, 2022), most-type quantifier comprehension gets worse with increasing model size, thus showing an inverse-scaling law in most-type quantifier comprehension. in this study, we evaluate a number of different language model families, with models ranging from a size of 125 million parameters to 175 billion parameters, and find that the results are consistent for all llms. ## dataset and models the models and dataset used in this paper are identical to the ones used in @xcite . this work uses the log probabilities produced by different language models to calculate a quantity called surprisal, which is introduced later in the paper. we do not make additional api calls or query models. we simply use the log probabilities released by @xcite , thus mitigating differences due to experimental conditions. this paper aims to provide an alternative way of interpreting the output logits produced by different llms compared to @xcite . ## quantifier comprehension in llms in this section, we first present how (michaelov and bergen, 2022) measure quantifier comprehension in llms. specifically, we present two ideas of surprisal and quantifier accuracy and ways to measure both properties as proposed by @xcite . alongside, we also highlight shortcomings of these quantifier comprehension evaluation methods. ## proposed evaluation of quantifier comprehension in llms in this section, we present a more robust way of measuring quantifier comprehension in llms. measuring quantifier comprehension in llms in the setting defined by @xcite has to be grounded in the principle that the typical and atypical words chosen in the dataset are not unique, and hence to measure the effect of presence of quantifier in a context, we should do measurements on the same critical word. we propose two tests do this. ## discussion the above two tests for evaluating quantifier comprehension in llms show that these models are far from perfect. the underlying premise of the method used in this paper and @xcite is that the presence of a quantifier should increase or decrease the probability of a critical word depending on its typicality (michaelov and bergen, 2022). but both tests described in section 4 show that this is not ubiquitously observed. the accuracy numbers for both tests are around 50-60%, which means that the probability distributions do not incorporate quantifier meaning for a large majority of sentences. a test like this makes a fundamental assumption that understanding of meaning can be measured by studying the relative ranking of tokens in the generated word logit. while this is a fair assumption, we think it is necessary to explicitly point this out incorporating quantifier meaning in this way is not a necessary condition for models to perform well, as can be seen by their consistent improvement across different benchmark @xcite @xcite . also, it has been shown in previous studies that humans are not that great at quantifier comprehension as well (urbach and kutas, 2010), and continue to have a preference towards the more typical word in a context irrespective of the quantifier. these observations suggest two things. firstly, that llms are not good at quantifier comprehension. secondly, we also observe this lack of sensitivity to quantifier meaning in humans. this combined with the fact that despite lack of quantifier comprehension, llms get increasingly better at language understanding, we can argue that quantifier comprehension is not as necessary of a task in language processing and understanding as we thought it was. ## related work inverse scaling laws were introduced as a competition @xcite to incentivize research towards finding scenarios where language models get worse as their size increases. as the field of nlp moves towards scaling models to larger and larger sizes, it is important to know the scenarios where this scaling becomes detrimental @xcite . as language models get increasingly better, some common linguistic tests that they are put through revolve around understanding negation and quantifiers. studying the affects of negation has been the subject of focus for many studies (kassner and schütze, 2019; @xcite for different encoder-based masked language models. these studies find that these language models are not sensitive to negations. studies on quantifiers @xcite also seem to show similar results for masked language models. (michaelov and bergen, 2022) was the first work to study the quantifier understanding in decoder-based llms. ## conclusion in this paper, we conduct a study to evaluate how well large language models understand quantifiers. specifically, we study two types of quantifiersmost-type and few-type quantifiers. we present a set of experiments to evaluate quantifier comprehension of large language models and show that these models are able to differentiate between mosttype and few-type quantifiers as they scale. we also show that llms struggle incorporate the meaning of most-type quantifier comprehension when com-pared to few-type quantifiers. we also show that most-type quantifier comprehension demonstrates an inverse-scaling law and their understanding of most-type quantifiers get worse as the model size increases. this study indicates that llms do not take into account the meaning of quantifiers that strongly, as shown by low accuracy scores in figures 3 and 4. even so, these models get increasingly better at language understanding tasks, thus indicating that quantifier understanding might not be the best test to evaluate language understanding in llms.