VAQUUM : Are Vague Quantifiers Grounded in Visual Data? (2025)

Abstract: Vague quantifiers such as “a few” and “many” are influenced by various contextual factors, including the number of objects present in a given context. In this work, we evaluate the extent to which vision-and-language models (VLMs) are compatible with humans when producing or judging the appropriateness of vague quantifiers in visual contexts. We release a novel dataset, VAQUUM, containing 20,300 human ratings on quantified statements across a total of 1089 images. Using this dataset, we compare human judgments and VLM predictions using three different evaluation methods. Our findings show that VLMs, like humans, are influenced by object counts in vague quantifier use. However, we find significant inconsistencies across models in different evaluation settings, suggesting that judging and producing vague quantifiers rely on two different processes. We release our dataset and code at https://github.com/hughmee/vaquum.

## introduction everyday conversations are replete with statements containing vague quantifiers, such as "there are many horses" (figure 1 ). despite the fact that they are vague, they cause surprisingly little misunderstanding among interlocutors @xcite . vague quantifiers, unlike crisp quantifiers, allow for borderline cases in which it is unclear whether the quantifier applies or not, and where we can also expect some variation in the extent to which speakers would use it. for example, all does not allow for borderline cases, but it is unclear when a quantity ceases to be a few or how many many is. although vague quantifiers have long been a subject of investigation among formal semanticists (see e.g. @xcite and (psycho)linguists (e.g. @xcite , they have received relatively little attention within the field of natural language processing (nlp). figure 1 : experiments in this work. we (1) ask human participants to rate, using a slider, the appropriateness of statements containing vague quantifiers in relation to images. we (2) extract vlm generation probabilities for those same statements, (3) prompt the models to generate an accuracy score for them, and (4) evaluate probabilities assigned to these statements in a multiplechoice setup. the image above is originally from the fsc-147 dataset @xcite . in visually grounded settings, the use of vague quantifiers can be influenced by factors related to the scene itself, such as the number of entities observed (e.g. @xcite ; their sizes @xcite ; as well as information like the speaker's and hearer's personal beliefs and attitudes @xcite . this broad range of factors, coupled with their vagueness, raises the question of how well multimodal language models can capture human patterns in the grounded comprehension and use of such expressions. in this paper, we explore this question with vision-and-language models (vlms) in multimodal settings involving quantified statements about images, using a novel dataset. studying alignment between human and vlm judgments can reveal the extent to which such models grasp the semantics of natural language. the inclusion of a vision modality allows us to provide context in the form of both visual and textual information @xcite . we examine to what extent visual cues influence state-of-the-art vlms' understanding and production of expressions containing vague quantifiers, and how this compares to human linguistic intuitions (figure 1 ). by zooming in on such visual and contextual variables, our work follows the spirit of recent research exploring the grounding capabilities of vlms (e.g. @xcite @xcite @xcite @xcite . the contributions of this paper are as follows. the use and judgment of vague quantifiers have been studied extensively in formal semantics and psycholinguistics. recent years have also seen a growing but relatively limited interest in studying (v)lm behavior with linguistic quantifiers. vague quantifiers in formal semantics work on vague quantifiers in formal semantics sits at the crossroads of generalized-quantifier theory and degree semantics. the foundational work of @xcite drew attention to properties of natural language quantifiers such as monotonicity and entailment; this serves as the basis for subsequent work extending generalized quantifier theory to vague quantifiers (e.g. @xcite @xcite . a parallel line of work has focused on the treatment of vagueness in natural language, e.g. in gradable adjectives. here, an important insight is that vague predicates are frequently interpreted with respect to a standard that is set by some salient frame of reference (e.g. @xcite @xcite . @xcite shows that comparison classes (frames of reference) also play an important role in the interpretation of vague quantifiers such as many and few. vague quantifiers in psycholinguistics it has been suggested that humans make use of an approximate number system @xcite @xcite , where vague terms might not refer to exact numbers but rather approximations thereof. related to this, subitizing is the ability to instantly and accurately recognize a small number of items without going through the process of counting. humans are generally able to do this for object counts up to 4 @xcite . above this subitizability threshold, they tend to use quantifiers @xcite . however, it has also been shown that quantifier comprehension and use go beyond (approximations of) the cardinality of the targeted object. factors include object size @xcite ; the number and proportions of other objects in the scene @xcite @xcite ; set size (e.g. the answer to a question such as: "several marbles from a set of 12 marbles would be marbles"; @xcite ; the functionality of objects in the scene @xcite ; and object grouping and spacing @xcite . in conversations and texts, the choice of quantifier influences the expected rhetorical impact of a statement, and vice versa. @xcite show that the choice of quantifier can reveal a speaker's prior expectations regarding the frequency of the object in the scene. moreover, several works have outlined the different perspectives that a few and few convey: while "a few people were at the party" focuses on those who were present, "few people were at the party" puts the emphasis on those who did not attend @xcite . moving beyond the gold label approach, @xcite demonstrate that models using both audio and visual input to select appropriate quantifiers can achieve results that align with human distributions reported by @xcite . @xcite compare human and large language model (llm) responses on questions such as "there are 500 balls. 234 of them are yellow. are many balls yellow?" they find that responses generated by llms align more closely with human judgments on crisp quantifiers than on vague ones. @xcite find that llms can map uncertainty expressions such as probably and unlikely to probabilistic (numerical) responses in a human-like fashion. more akin to our experiments, @xcite evaluate three vlms on their abilities to assign appropriate quantifiers to visual scenes, prompting models to select one out of nine quantifiers in response to questions such as "how many animals are there in the image?", with synthetic images generated by @xcite . our approach diverges from theirs on several points. first, we use natural images rather than artificial ones, offering a more realistic setting for evaluating vlms. additionally, we use a wider range of methods to provide a more comprehensive assessment of model behavior. ## the vaquum dataset we construct and release the vaquum dataset: vague quantifiers with human judgments. images we utilize annotated datasets used for object counting in computer vision. fsc-147 @xcite ) contains 6146 images across 147 object types, with annotated object counts ranging from 7 to 3731. hobley and prisacariu (2023) refine and deduplicate this dataset to release fsc-133 (containing 133 object types). we sample images from fsc-133 and exclude a total of 22 object categories for several reasons, such as their uncountable nature (e.g. fresh cut), obscurity (e.g. carrom board pieces) or simply because the images do not depict the object from the label. we also remap 37 categories to either their plural form, where necessary, or their basic-level category (e.g. mapping crows to birds; cf. @xcite . since the lowest count in fsc-133 is 7, we complement this dataset with samples from the test set of tallyqa @xcite , which includes images and annotated counts sourced from visual genome @xcite and vqa2 @xcite . here, we use images classified as "simple" in tallyqa, which have counts between 1 and 15. from this set, we exclude images for which the labelled object is not in the set of remapped fsc-133 labels. we discard all counts below 2 (from tal-lyqa) and above 100 (from fsc-133). we include three types of object features in our dataset. ## experiment 1: production probabilities our first series of experiments studies the predicted production probabilities of quantified statements by sota vlms. we prompt the models with "how would you describe the amount of [object] in the image?" we extract log probabilities, conditioned on this prompt and the image, for the quantified statements in vaquum, as well as the unquantified version. all extracted scores are normalized by token length. we consider the following models. blip-2 @xcite . we use the checkpoint powered by opt-6.7b @xcite connected to a eva-clip vit-g @xcite image encoder via a lightweight query transformer. instructblip @xcite . we use the checkpoint with a vicuna-13b @xcite language backbone, instruction-tuned on blip-2. llava-next @xcite . we use the 7b checkpoint with a mistral @xcite model few a few some many a lot of language backbone. it integrates a clip-vit through an mlp vision-language connector. llava-onevision @xcite . we utilize the 7b checkpoint, which integrates a siglip @xcite vision encoder with a qwen2 @xcite language decoder. molmo @xcite . we use the 7b-d checkpoint, which connects a vit image encoder to qwen2.7b via a connector mlp. figure 3 displays predicted log probabilities as a function of count, segmentation area and size norm and alignment with humans of the vlms tested, the two llava models exhibit the highest correlation with the human data in vaquum. for these models, we observe in figure 3 patterns similar to those of vaquum in figure 2 . probabilities for many and a lot of increase as a function of count, while few and a few show a downward trend. given that the question in the prompt focused explicitly on the amount of objects, the unquantified sentence is expected to be generally dispreferred. the trends in figure 3 suggest that the llava models can indeed draw this distinction between quantified and unquantified statements, as the unquantified expression displays the lowest-ranking log probabilities across count, segmentation and size norm. however, other models do not reveal that same ability. this is most pronounced for instructblip and molmo, which generally tend to favor the unquantified statement as a response to the question. these models also show the same pattern across all quantifiers, further confirming their inability to differentiate among them. while the behavior of blip-2 is seemingly random, figure 3 shows an upward trend for all quantifiers as a function of count. linear mixed model in prompts should target amounts for most models, we find that simply changing the question from "how would you describe the amount of [object] in the image?" to "how would you describe the image?" yields different patterns in the results (see appendix c). most notably, we find that the observed similarity between trends in human judgments and model predictions disappears once the prompt does not focus on amounts. interim conclusion in §3.1, most estimates of the lmm fit to participant data were statistically significant. moreover, object count made the biggest difference across all quantifiers. for llava-onevision, the model displaying the high-est pearson's correlation with human data in we now evaluate the instruction-tuned vlms using an approach that is more akin to the way vaquum was constructed in §3. that is, we prompt the models to explicitly rate the acceptability of quantified statements. we experimented with 10 different prompts that are variations on the question shown to human participants in §3.1. drawing inspiration from prompts used by @xcite , we center our analyses in the remainder of this section around the following prompt: "on a scale of 0 (completely inaccurate) to 100 (completely accurate), how accurate is the following statement for the image? please respond with one of the following options: 0, 5, 10, @xcite [statement]", where statement is an expression from the vaquum dataset. we refer to appendix d for the complete list of prompts tested. for vlms, appropriateness is not gradable we find that in this evaluation setup, blip-2 and instructblip generally fail to generate numerical responses to the prompts we tested, despite some prompts explicitly encouraging them to only respond with a number. the two llava models and molmo consistently provide numerical responses to most of the prompt templates tested. however, while we construct the prompts in such a way that vlms are encouraged to provide a response that falls between a certain range, the vast majority of model responses tend towards the extremes (i.e. on the lower or upper bound of the specified range; see appendix d for a distribution of responses). some is generally appropriate when numeric answers to prompts tend towards the extremes of a scale, it can be informative to aggregate generated scores, which is virtually the same as calculating the relative frequency of a vlm dis/agreeing with the statements. we report this in figure 4 for ob- interim conclusion experiment 1 showed that object count influences model predictions for many and a lot of. similar patterns emerge in figure 4 , where average scores for these quantifiers increase with count. discrepancies between results from experiments 1 and 2 show that in a setting where models are explicitly required to judge statements (exp 2), the outcomes are unrelated to the models' log probabilities for the same statements (exp 1). in experiment 1, probabilities are extracted using an autoregressive method compatible with the pretraining objective of the llm backbone. in contrast, experiment 2 relies on model abilities acquired during post-training, which further modifies model parameters. the discrepancies we observe align with independent observations that posttraining can negatively impact model calibration @xcite . in figure 5 , we report the predicted log probabilities of instruction-tuned vlms as a function of count. ## discussion alignment with humans in this paper, we explore how vision-and-language models produce and evaluate simple expressions containing vague quantifiers. we constructed the vaquum dataset and used this to investigate whether object count, segmentation area and size norm affect vlms to the same extent as they do humans. we showed that in particular for object count, the patterns found in some vlms show striking similarities with the human data in vaquum. this result appears to contradict the observation that vlms perform poorly on counting tasks @xcite . however, our findings with vague quantifiers could be accounted for in terms of an approximate number system, which cognitive scientists have posited to account for the human ability to rapidly estimate quantities @xcite @xcite @xcite . in the context of vague quantifiers, it has been argued that there exists a mapping between exact and approximate number systems @xcite . the extent to which vlms rely on something akin to an ans is a topic for future work. to gather data that better aligns with experiment 1, one possible approach is to leverage existing human-annotated datasets from image captioning studies. however, commonly used datasets such as mscoco @xcite typically feature low count ranges, leading to a scarcity of relevant images as counts increase. moreover, while some captions do include the quantifiers central to our study, such instances are rare: most images have at most one caption containing a relevant quantifier, and annotators often avoid using them altogether. nevertheless, we see potential in incorporating naturalistic datasets that include human-generated descriptions. expanding our work to include such resources (potentially through re-annotation with a focus on quantifiers) could offer additional answers to the question of human-model alignment on vague quantifiers. self-consistency our experiments relied on paradigms incorporating production (experiment 1) and judgment (experiments 2 and 3). we find that vlms are not self-consistent across these evaluation paradigms. that is, when a vlm is set to judge the use of a quantifier-a meta-linguistic task-its judgment is not necessarily rooted in the log probabilities that govern the model's generation of the quantifier. questions of calibration and consistency such as these go beyond the domain of quantifiers and are an active area of ongoing research (e.g. @xcite @xcite . perspectivism in this paper, we follow the standard practice of aggregating human judgments through taking the average and focusing on the general trends. this, however, might overlook meaningful variability that emphasizes the complexity of human judgments on vague expressions. while developed with our specific research questions in mind, the vaquum dataset captures not only quantifier judgments in visual contexts, but also the variation and disagreement among humans. modeling disagreement among annotators is increasingly recognized as an important focus for nlp, as it reveals the shortcomings of assuming a single ground truth (such as the opinion of the majority). this view is one of the cornerstones of perspectivist approaches to nlp @xcite @xcite . we see our dataset as a useful potential resource for this community, as well as any community that studies linguistic phenomena where variation is bound to play a role. vagueness is a case in point, as it inherently gives rise to disagreements-not as noise, but as a result of the context-sensitive nature of language. outlook psycholinguistics has shown that vague quantifiers do not depend exclusively on the count and size of target objects. this is further confirmed by the residual variance (49.7%) in vaquum that cannot be explained by the linear mixed effects model (lmm) on human judgments. while the lmm analysis yields a better fit for vlm log probabilities, we find that there, too, the lmm cannot explain all the variance (leaving a residual variance of 8.8% for llava-onevision). future work could focus on other contextual factors, such as the number of other objects present, the object density in the image, as well as the role of scene semantics and other objects in the image background. in combination with visual grounding capabilities, it is worthwhile to investigate the role of commonsense and world knowledge in vague quantifier usage: while seeing 20 people at a conference will most likely not be reason for one to exclaim that there are many, the same amount of toddlers at such an event might be.