Are Multimodal Large Language Models Pragmatically Competent Listeners in Simple Reference Resolution Tasks? (2025)

Abstract: We investigate the linguistic abilities of multimodal large language models in reference resolution tasks featuring simple yet abstract visual stimuli, such as color patches and color grids. Although the task may not seem challenging for today’s language models, being straightforward for human dyads, we consider it to be a highly relevant probe of the pragmatic capabilities of MLLMs. Our results and analyses indeed suggest that basic pragmatic capabilities, such as context-dependent interpretation of color descriptions, still constitute major challenges for state-of-the-art MLLMs.

## introduction the advent of large language models (llms) and their expansion in scale, variety, and availability over the past decade has led to considerable interest -both in the research community as well as in the public at large -in understanding their capabilities and limitations. a lot of research focuses on their general abilities (reasoning, math, professional examinations, . . . ) in order to answer questions about the level or category of 'intelligence' these models exhibit (e.g., @xcite or which practical tasks they might be suitable for. in contrast to this, research in (computational) linguistics is also interested in their basic linguistic abilities @xcite . in this paper, we investigate the linguistic abilities of multimodal large language models (mllms) on the level of language use, i.e., linguistic pragmatics, in the well-known reference resolution paradigm. more specifically, we examine whether off-the-shelf mllms (llava-next, qwen2-vl, and janus-pro) are able to resolve references to abstract visual stimuli (color patches and color grids; @xcite ; see fig. 1 ) given in director-matcher-style dyadic reference games @xcite . although the task may not seem challenging for today's language models because it is straightforward and easy for human dyads, we consider it to be a highly relevant probe of the pragmatic capabilities of mllms. being ubiquitous in all of language use, reference (generation and resolution) is highly context-dependent (here visual context). the same can be said about the color references at the center of the task (where color is a main distinguishing attribute between potential referents; @xcite . the complexity of the visual context varies between the two types of stimuli we consider, but can generally be considered simple. their abstractness, however, poses demands on the basic visual perception capabilities of mllms. thus, we investigate how well mllms perform basic reference resolution tasks requiring contextualized pragmatic reasoning about color and simple spatial arrangements in two abstract visual domains. 1 our results show that models with sufficient capacity achieve promising results for color patches. however, even the best-performing models struggle with the complex structure of color grids. ## background the semantics and pragmatics of references to color have long played a special role in work on reference games @xcite @xcite . the categorization of colors is well-known to be subject to complex interactions between semantic and perceptual information @xcite . the naming of colors in interactive games is also well-known to be subject to pragmatic reasoning and negotiation between interaction partners @xcite @xcite . finally, color references have been studied from the perspective of figurative language and creativity @xcite . color perception and understanding tasks have also been used in recent work on probing language models. @xcite explored the alignment between the perceptual color spaces of humans and text-based llms. their findings indicate moderate alignment for basic color terms, which decreased as color descriptions become complex and subjective. similarly, @xcite found that alignment improves with model size. @xcite examined the sensitivity of mllms to sensorimotor features by testing their ability to identify images that match textual implied features, showing that the effect for color emerges only in the largest model. rahmanzadehgervi et al. (2024)'s study on vision language models shows that even state-of-the-art models, such as gpt4-o and gemini-1.5 pro, perform surprisingly poorly at a range of low-level tasks, which humans are expected to complete with ease, e.g., determining whether two circles overlap. together, these studies warrant further investigations and analyses into multimodal language understanding tasks. in this paper, we investigate tasks for analyzing low-level language grounding of color stimuli in combination with pragmatic capabilities, focusing on referring expressions produced by human players of reference games. ## results we evaluate the mllms' performance with accuracy, i.e., how often the models identify the visual target correctly. we compare results between (a) visual complexity condition (far/close/split, see section 3.2) (b) humans and models; and (c) different models and model sizes (see section 3.1). ## conclusion in this paper, we investigated how well mllms perform basic reference resolution tasks that require contextualized pragmatic reasoning about color in two abstract visual domains of different complexity. we found that models with sufficient capacity achieve promising results in the simpler domain of color patches, but that even the best-performing models struggle with the more complex structure of color grids.