Quantifying Generalizations: Exploring the Divide Between Human and LLM s’ Sensitivity to Quantification (2024)

Abstract: Generics are expressions used to communicate abstractions about categories. While conveying general truths (e.g., “Birds fly”), generics have the interesting property to admit exceptions (e.g., penguins do not fly). Statements of this type help us organizing our knowledge of the world, and form the basis of how we express it (Hampton, 2012; Leslie, 2014).This study investigates how Large Language Models (LLMs) interpret generics, drawing upon psycholinguistic experimental methodologies. Understanding how LLMs interpret generic statements serves not only as a measure of their ability to abstract but also arguably plays a role in their encoding of stereotypes. Given that generics interpretation necessitates a comparison with explicitly quantified sentences, we explored i.) whether LLMs can correctly associate a quantifier with the generic structure, and ii.) whether the presence of a generic sentence as context influences the outcomes of quantifiers. We evaluated LLMs using both Surprisal distributions and prompting techniques.The findings indicate that models do not exhibit a strong sensitivity to quantification. Nevertheless, they seem to encode a meaning linked with the generic structure, which leads them to adjust their answers accordingly when a generalization is provided as context.

## introduction generic generalizations, or simply generics, are statements such as birds fly , dogs are mammals or clocks are round, that convey information about categories. they are a powerful way through which language allows us to communicate and learn abstract knowledge that extends beyond present context and direct experience. we use generic sentences to express our knowledge about the world, including stereotypes or prejudices (e.g., men are better at math than women). generics are fundamental to human cognition because they help us conceptualize the properties we associate with dif-ferent categories, organizing our experience of the world @xcite . the most notable feature of generics is that they allow for exceptions @xcite . for example, birds fly is considered true by speakers even if there are birds that cannot fly (e.g., penguins): in this case, therefore, the corresponding universal statement (all birds fly) is false. different generalizations tolerate exceptions to varying degrees, according to their different semantic content: dogs are mammals requires for its truth that all dogs be mammals; ducks lay eggs is judged true even if only mature female ducks lay eggs, while mosquitoes carry malaria refers to an even smaller minority (about 1 percent of mosquitoes carry malaria). there are generics that might be better paraphrased with all, others with most, and others with some; however, unlike quantified statements, they do not explicitly convey information about how many category members possess the predicated property. given this property, the meaning of generics can be considered "underspecified": humans' correct interpretation is derived through world knowledge and pragmatic abilities @xcite . the main questions that cognitive and psycholinguistic studies conducted on generics seek to answer are whether generics are a default mechanism, whether there exists a generic bias, and what is the relationship between genericity and prevalence, i.e., to what proportion of category members the property predicted by the generic applies @xcite @xcite . these studies investigate the nature of generalizations by contrasting generic and overtly quantified sentences @xcite ; in this sense, quantifiers are used to make explicit the underspecified meaning of generics. the present paper investigates the interpretation of generalizations in different large language models (llms). since psycholinguistics experiments conducted on humans involve the compari-son with quantified expressions, we also used quantifiers as a means of unraveling generics comprehension to directly compare humans' and models' interpretations. we aim to explore the capability of llms to interpret generalizations that differ in semantic content as humans do. as we mentioned, this ability in people is closely related to world knowledge, which allows us to interpret 'underspecified' and implicitly quantified sentences by making use of our prior information. for this reason, investigating what is encoded in models with regard to generic form and its relation to quantification is crucial to comprehend whether they effectively understand the level of inclusiveness of a conceptual category, based on the context in which it is used, and whether they can leverage the power of quantifiers to replicate human-like distinctions, thereby enhancing their capacity to comprehend and interpret natural language accurately. the capability of llms to interpret generic sentences is not only an index of their capacity for abstraction but is also arguably involved in their encoding of stereotypes, since generics are one of the most powerful ways through which language convey them @xcite . in what follows, we present two experiments that try to answer our research questions (rqs): rq1: are llms capable of interpreting generic sentences according to their semantic content? generalizations can have different implicit quantificational values depending on their semantic content. humans are able to derive their correct meaning thanks to their world knowledge. in our first experiments (cf. 4), we investigate if llms are able to do the same through two different methodologies. rq2: do llms have a linguistic default interpretation associated with the generic form? people seem to have a default interpretation associated with the form of generics @xcite . in our second experiment (cf. 5), we conduct an exploratory analysis of how models interpret generalizations aside from their content, i.e., whether they seem to have encoded linguistic knowledge associated with the generic form. ## related works most of the nlp literature dealing with genericity in language has focused on the building of re-sources geared towards distinguishing generic expressions that refer to whole categories from their non-generic counterparts that refer to specific exemplars @xcite @xcite . more recently, the usefulness of generic sentences as a resource to retrieve common sense knowledge, exploitable to boost performance in various nlp applications, has been proposed and demonstrated by @xcite . however, to the best of our knowledge, there are no studies investigating the interpretation of linguistic generalizations by llms, except for the recent works by @xcite , which addresses the generic overgeneralization effect, and @xcite , which investigates generics interpretation, building on psycholinguistic experimental designs. both works, however, only focus on masked language models such as bert and roberta. we will use quantifiers to investigate generics comprehension, placing them at the beginning of bare generic sentences to explicitly specify their quantificational value. the most relevant studies that have evaluated model predictions following quantifiers are @xcite , which focus on logical quantifiers such as all, every, and some, and @xcite and @xcite , which focus on few and most-type quantifier; the other few works involving quantifiers focus on predicting the quantifier itself @xcite . the present work will contrast llms' predictions on generic sentences and sentences quantified by no, few, some, most, and all, investigating which quantifiers seem to best approximate the meaning of the generic form. therefore, our work aims not only to understand llms' generics interpretation but also to contribute to the exploration of llms' knowledge of quantifiers, adding the systematic comparison with generic sentences as a novel element. ## materials and methods dataset for this study, we created a dataset in which each generic sentence is paired with the correct quantifier, i.e., the quantifier that humans would prefer to make explicit the implicit quantification value of the generic sentence 1 . from now on, we will refer to this quantifier as human quantifier, while we will use the label llm quantifier to indicate each quantifier when paired with the sentences for the llms evaluation. to assemble our dataset, we employed sentences from different existing resources. in the first place, we looked at the herbelot and vecchi (2016) dataset, consisting of concept-feature pairs from @xcite , such as airplane hasengines or ant is-black, labeled by native speakers through quantifiers. for each pair in the norms, annotators were asked to provide a label expressing how many members of the category possess the property in question, choosing among the natural language quantifiers no, few, some, most, all. we selected only those pairs on which all three annotators agreed on the same quantifier. from this dataset, we sampled 500 sentences annotated with some and all, plus 97 sentences annotated with most. sentences annotated by humans with some, most, and all are those that can be considered as true generalizations, in their generic form. however, in order to better understand whether they are correctly interpreted by llms, we decided to add to the dataset also sentences quantifiable with few and no, that are characterized by implausible or impossible category-property pairs. in this case, the effect of the quantifier is to reverse the truth value of the sentence (from implausible to plausible and from impossible to possible): because of this feature, these sentences will be useful as a touchstone to evaluate the others. first, we included a sample of 500 conceptproperty pairs extracted from the comps dataset @xcite . we selected 500 cases in which negative-sample-type is equal to random (i.e., for which the similarity between the acceptable and unacceptable concept for a certain property is equal to 0), and used the unacceptable concept to form our sentences. in these cases, the human quantifier would always be no because the predicated property is unacceptable, as in unicycles clean dishes. additionally, we selected 240 stimuli originally constructed by urbach and kutas (2010) for a psycholinguistic task and recently used by @xcite and @xcite for llms evaluation. these stimuli consist of 120 typical subject-verb pairs (called "backbone phrases") completed by both a typical and an atypical object, as in postmen carry mail vs. postmen carry oil. in the original psycholinguistic experiment of urbach and kutas (2010), these sentences were alternatively modified by most-type and few-type quantifiers in order to collect offline plausibility ratings and record brain activity (using eeg) for the different conditions. following the plausibility ratings of the original experiment, we included these stimuli by annotating sentences with a typical object with most quantifier, while sentences with an atypical object are annotated with few. our final dataset consists of 1,837 sentences. even if the dataset is not completely balanced, we believe that the stimuli should be sufficient to observe tendencies in intra-and inter-conditions. models we conducted our experiments on bertlarge-uncased @xcite , a bi-directional masked language model, gpt2-xl @xcite , and 2 open-source pre-trained generative llms and their instruction-tuned variants: llama-2 @xcite and mistral @xcite with 7 billion parameters. foot_1 4 are llms capable to interpret generic sentences according to their semantic content? ## general discussion this paper offers both quantitative and qualitative insights into how llms interpret generics, employing experimental designs that utilized quantified expressions to probe the comprehension of generic statements. our two experiments were conceived to evaluate two related but separate abilities: first, the models' capacity to accurately recognize the common knowledge implied in generic statements (i.e., they can generalize a property to the right level of inclusiveness of categories); secondly, their ability to comprehend generalizations irrespective of their content, specifically, whether they incorporate any linguistic cues linked to the generic form. to the best of our knowledge, we are the first to perform this investigation with recent llms, including their instruction-tuned variant, and testing them with prompting methodologies. the experiment illustrated in 4 was designed to investigate whether llms are capable of interpreting generic sentences according to their semantic content through quantifiers (rq1). we observed that surprisals do not seem to be particularly sensitive to the effect of quantifiers on sentence meaning, thus preventing us from using them as an explicit marker of the interpretation of generic sentences that differ in semantic content. however, it is possible that this outcome is not due to a complete insensitivity of the models to the meaning of quantifiers as much as to the method employed. in fact, the measurement of surprisals could be more akin to measurements of human online processing (such as recording of brain activity) rather than offline judgments (such as the annotations we have on our dataset). interestingly, the surprisal of the models with respect to the effect of quantification does indeed seem to follow a similar pattern to that emerging from comparable studies on human n400 potentials @xcite @xcite . therefore, we investigated the behavior of the models through prompting, which mirrors offline human judgments. the analyzed outcomes suggest that the instruction-tuned models have encoded some kind of meaning associated with quantifiers and the generic form, although not particularly refined. llms judge the choice of most and all, as well as of the generic form over the others, as more suitable as the semantic content of the sentence goes from impossible/implausible category-property pairs (no/few classes) to plausible category-property pairs (some/most classes), to necessary category-property pairs (all class). however, the comprehension of the meaning of generic and quantified sentences with respect to their semantic content does not seem to be particularly accurate. llms tend to take a very 'conservative' stance, preferring the intermediate quantifier some when given only quantified sentences as options, and the generic sentence itself (inherently vague) when this is added among the options foot_5 . this could be due to the fact that explicit quantification is actually a relatively rare phenomenon in naturally occurring text, on which llms are exclusively trained, while underspecified constructions like generic sentences are much more frequent @xcite . moreover, the different quantifiers all appear in the same syntactic positions and in superficially very similar contexts; the choice of one or the other is inextricably linked to our extralinguistic knowledge of the categories and the properties predicated on them, something llms do not possess. for this reason, we conducted a last study to explore the models' interpretation of generalization and quantification aside from the semantic content of the predications, i.e., whether they seem to have encoded linguistic knowledge associated with the generic form (rq2). we found that, when a generic sentence is provided as a premise in an entailment condition, instruction-tuned models tend to reshape the distributions of the different quantifiers in their responses (cf. figure 3 vs. figure 4 ), skewing their preferences toward "strong" positive quantifiers (most and all). people behave similarly (interpreting a property predicated by a generic as applicable to virtually all members of the category) when tested on novel categories, for which they have no prior understanding. however, in a real language setting humans undeniably modulate their interpretations of generalizations through their knowledge of real categories. ## conclusions in conclusion, this study observed that i) llms seem not to have the world knowledge necessary for the comprehension of the meaning of generic and quantified sentences with respect to their semantic content in a human-like way; ii) llms overextend the truth of a generic sentence when this is presented as an assumption, on most/all members of real-world categories, regardless of the meaning of the predication. this behavior could play a role in their encoding of stereotypes, which could be a potentially harmful bias. overall, we believe that further investigations are needed to clarify the interpretation of generics in language models and, more generally, the role that this phenomenon has in their behavior.