Naming, Describing, and Quantifying Visual Objects in Humans and LLM s (2024)

Abstract: While human speakers use a variety of different expressions when describing the same object in an image, giving rise to a distribution of plausible labels driven by pragmatic constraints, the extent to which current Vision & Language Large Language Models (VLLMs) can mimic this crucial feature of language use is an open question. This applies to common, everyday objects, but it is particularly interesting for uncommon or novel objects for which a category label may be lacking or fuzzy. Furthermore, similar patterns of variation are observed among human speakers for highly context-sensitive expressions, such as the quantifiers ‘few’ or ‘most’. In our work, we evaluate VLLMs (FROMAGe, BLIP-2, LLaVA) on three categories (nouns, attributes, and quantifiers) where humans show great subjective variability concerning the distribution over plausible labels, using datasets and resources mostly under-explored in previous work. Our results reveal mixed evidence on the ability of VLLMs to capture human naming preferences at generation time: while some models are good at mimicking human distributions for nouns and attributes, all of them fail to assign quantifiers, a task that requires more accurate, high-level reasoning.

## introduction recent years have witnessed increasing popularity in the development of large language models (llms) given their notable performance in following instructions, answering questions, and in many reasoning tasks, serving as general-purpose assistants @xcite . in parallel, a new generation of powerful vision and language llms (vllms) with excellent visual understanding and generation capabilities have emerged @xcite . rapidly, these models have outperformed previous approaches in many downstream tasks. in our work, we focus on the natural language generation skills of powerful vllms by analyzing an important but under-explored problem, namely, their ability to capture human production variability (in terms of distribution over plausible labels/descriptions) in naming tasks. previous work highlighted that speakers display a wide range of variability when asked to utter sentences, resulting in inter-speaker variability but also variability over time for the same speaker @xcite @xcite . in particular, in object naming, speakers may refer to objects appearing in a visual scene in many different ways @xcite . objects generally belong to multiple categories/super-categories, and all the lexicalized labels of such categories are valid @xcite . however, although multiple labels are valid, humans pragmatically adapt their naming preferences depending on the context @xcite , resulting in some labels being more frequently uttered than others. for instance, 'mammal' is a correct label to describe a gold retriever, but pragmatically less likely than 'dog'. similarly, speakers tend to prefer sub-ordinate words like 'car' instead of the potentially ambiguous super-ordinate word 'vehicle' in case multiple vehicles appear in the image. in our work, we are interested in capturing both these two features: while many labels are equally valid and acceptable when naming or describing entities, these labels distribute according to a certain likelihood distribution. in our work, we investigate this issue, which has recently entered the nlp research community @xcite , in three different production conditions. first of all, we consider the manynames dataset @xcite , where annotators assign labels to describe common objects in images in a referential expression generation setting @xcite . we also explore two additional resources that have not received much attention within the nlp community and that allow us to broaden the horizons of this phenomenon. first, we analyze the noun dataset (horst and hout, 2016), where speakers describe uncommon and novel objects: we focus on both the choice of the adjectives and how they distribute in the across-subject distribution. finally, we investigate human production variability arising from the context-sensitive nature of non-numerical quantifiers using the data collected by @xcite . we evaluate three vllms (fromage, blip-2, llava) on the above-mentioned tasks in a zeroshot setting. we sample multiple times from the model using nucleus sampling, mimicking various human speakers, and compare the generated samples against human production patterns using different metrics (jensen-shannon divergence and pearson's correlation, depending on the task at hand). our results show that models weakly to moderately mimic human distributions in naming common and uncommon objects. instead, all of them fail to mimic human distributions when selecting quantifiers, as highlighted by our in-depth analyses. we use the images and corresponding human labels or descriptions from three datasets in english, that we briefly describe below. noun the novel object and unusual name (noun) dataset (horst and hout, 2016) contains 64 images of multipart, multicolored, and threedimensional uncommon and novel objects. the dataset was originally created for behavioral studies on word learning and, to the best of our knowledge, it has not been used for nlp research. we focus on the naming task, where participants were asked to answer the question "what would you call this object?". the answers are sentences like: 'a plastic object with red stuff on top'. for each object, the proportion of colors (e.g., 'red', 'bronze') and textures (e.g., 'soft', 'rough') was calculated as the number of attributes given the number of responses. an example from the dataset is reported in figure 1 (left), together with the ratio of colors and textures in human responses. in noun, we examine human production preferences on a high level, by looking at the frequency according to which certain adjectives (related to color and texture attributes) are used. manynames in manynames @xcite , the authors collected names for 25k objects appearing in real-world images from visu-algenome @xcite by asking human annotators to generate a name for them. each object (highlighted by a red box in the image) is associated with an average number of 35.3 annotations. more than 90% of the objects are associated with more than one unique label (5.7 average name types per object). an example is shown in figure 1 (middle). when describing the object in the 1) the image features a green toy with an orange nose and two orange wheels. quant to study how quantifiers are used when referring to quantities grounded in images, @xcite introduced a dataset of visual abstract scenes containing a variable number of animals and artifacts and asked human participants to answer the question "how many of the objects are animals?". participants could select the answer from a list of nine pre-selected quantifiers: 'none', 'almost none', 'the smaller part', 'few', 'some', 'many', 'most', 'almost all', and 'all'. the authors used images with 17 different proportions of animals and artifacts (ranging from 0% to 100%). in our work, we tested 50 images for each of the 17 proportions in the dataset, resulting in a total number of 850 images. foot_0 ## the curious case of quantifiers we run some analyses to investigate the poor performance of all models in the quant task. first of all, we acknowledge that the multiplechoice prompting used in quant is different and more complex than the prompts used for the other datasets. still, it is unlikely that this is the main reason behind the poor performance of all models. figure 4 shows the correlation results broken down by the proportion of animals in the image. we observe that even though the overall correlation results are similar across models (figure 3 ), they perform quite differently depending on the proportion of animals in the scene. while blip-2 performs relatively well on the 'extreme' proportions (no animals or all animals in the image, when speakers generally choose the quantifiers 'none' and 'all', respectively), llava excels at intermediate proportions, and fromage performs better on proportions above 50%. can we conclude that models properly handle the task of assigning the most likely quantifiers for some proportions? these results evoke two hypotheses: (a) the models are capable of selecting plausible quantifiers only for some proportions or, vice versa, they understand only some of the quantifiers analyzed; (b) the models have a bias towards some specific quantifiers, regardless of the proportion of targets in the scene, leading to a decent perform on some proportions as a side effect. our additional analyses, reported in figures 8 and 9 in the appendix, support hypothesis (b): fromage has a strong bias towards selecting the quantifier 'many'; blip-2 frequently selects the extreme quantifiers 'none' and 'all', and its selection is not influenced by the proportion of targets; llava has a bias towards selecting the quantifier 'some', regardless of the proportion of targets. to further shed light on this result, we qualitatively assess the 'counting' skills of the models, a crucial skill to succeed in assigning quantifiers. as the examples in figure 11 in the appendix illustrate, all models struggle to successfully count how many animals appear in the image. we hypothesize that the reason for the poor performance in assigning quantifiers lies in the quantity estimation and comparison skills of the models. this observation is in line with recent research investigating the poor 'counting' skills of current models @xcite . ## conclusion while human speakers exhibit a wide range of human production variability in naming tasks, mirroring pragmatic constraints and subjective preferences, it is not clear to what extent vllms can mimic this peculiar trait of language use. in our work, we investigate this issue in three tasks: naming common objects, naming novel objects, and assigning quantifiers. our results reveal that bestperforming models achieve a moderate correlation with human patterns in some tasks (object names and color terms). however, all models dramatically fail when assigning quantifiers, the only production setup that requires some form of reasoning, i.e., the ability to reason over sets of objects and process quantities. based on our analyses, we hypothesize that the reason behind this failure stems from the poor "counting" skills of the models.