S cene G ram: Conceptualizing and Describing Tangrams in Scene Context (2025)

Abstract: Research on reference and naming suggests that humans can come up with very different ways of conceptualizing and referring to the same object, e.g. the same abstract tangram shape can be a “crab”, “sink” or “space ship”. Another common assumption in cognitive science is that scene context fundamentally shapes our visual perception of objects and conceptual expectations. This paper contributes SceneGram, a dataset of human references to tangram shapes placed in different scene contexts, allowing for systematic analyses of the effect of scene context on conceptualization. Based on this data, we analyze references to tangram shapes generated by multimodal LLMs, showing that these models do not account for the richness and variability of conceptualizations found in human references.

## introduction reference to visual objects is an elementary component of language and situated interaction. almost always, we can refer to something in many different ways, and the linguistic choices we make reflect the ways we categorize or conceptualize it. more often than not, we can have multiple conceptual perspectives @xcite on the same things: for example, the same person could be referred to as a "human", "woman", "engineer", "mom" or using their proper name, depending on what is deemed relevant, appropriate or useful in a given situation (e.g. @xcite . even fewer limitations exist for abstract shapes such as tangrams: in figure 1 , the same shape can be seen as representing e.g., a "crab", "sink", or "space ship", showcasing the flexibility and richness of human conceptualization and interpretation at the intersection of visual and semantic processing. despite recent progress in multimodal language modeling 1 data and code for this project are available at: github.com/clause-bielefeld/scenegram @xcite , current systems show mixed results in reproducing human variation in object naming @xcite and figurative descriptions for abstract stimuli remain a major challenge in vision and language (v&l) research @xcite . how can we navigate this complex web of manyto-many relationships between visual stimuli and possible conceptualizations and descriptions? a common assumption in linguistics, cognitive science, and psychology is that our cognition is highly "tuned" to the everyday contexts and situations we interact in. a well-researched instance of this is our visual perception of objects which is known to be fundamentally shaped by high-level, conceptual expectations on the level of scenes @xcite @xcite , among many others). for example, at the beach, we would rather expect to see certain animals and plants than household items, whereas in a bathroom, it would be the other way around. it is well-researched that these processes facilitate, e.g., visual object recognition (e.g. @xcite , and it seems plausible that scene context also affects descriptions of objects which can be conceptualized in different ways. however, existing v&l datasets with real-world images make it difficult to study the same objects in different scenes: as objects often occur in typical contexts, the same regularities that are exploited for visual processing obstruct investigation with natural data. in this paper, we contribute (i) a dataset of human references to tangram shapes placed in different scene contexts, allowing for systematic analyses of the effect of scene context on the conceptualization of the same shape, and (ii) analysis of references to tangram shapes generated by multimodal llms, showing that these models do not account for the richness and variability of conceptualizations found in human references. our findings show that scene context affects tangram descriptions in a way, that speakers tend to verbalize conceptualizations that human: sink (5); bowl (2); crab (2); bathtub shape (1) human: crab (7); bathtub (1); bowl (1); bull (1) human: crab (4); bowl (2); dog (1); seal (1); letter c (1); space ship (1) llava 7b: bathtub (6); rectangle (2); bathroom (2) llava 7b: sun (3); bird (2); diamond (2); boat (1); wave (1); house (1) llava 7b: house (3); square (2); diamond (2); triangle (1); parallelogram (1); box (1) llava 72b: house (8); boat (1); bathtub (1) llava 72b: sailboat (4); house (3); boat (3) llava 72b: house (7); boat (3) figure 1 : a single tangram in bathroom, beach and baseline contexts with counts for labels in annotated or predicted descriptions. human annotations commonly include labels which are coherent with scenes ("sink" and "crab"). llava 7b and 72b show similar patterns, but also issues in differentiating between tangrams and scenes. ## background human scene understanding research on human vision and perception has shown that when viewing a scene, humans perceive it as a coherent whole instead of mere collections of objects @xcite . capturing the gist of a scene is a rapid process @xcite , and while incongruent context can also be misleading @xcite , scene-level information has been demonstrated to facilitate e.g., visual object recognition in both human cognition @xcite @xcite and computer vision systems @xcite ; galleguillos and belongie 2010, see wang and zhu 2023 for a survey). for this, humans and machines can exploit learned knowledge about regularities of the visual word for visual processing @xcite @xcite @xcite , e.g. semantic rules that certain objects tend to occur in some contexts rather than others @xcite @xcite . context, conceptual perspective and referential choice verbal reference to visual objects requires making linguistic choices, as the same things can be called and described in many different ways @xcite @xcite . research in referring expression generation has modeled these choices as a function of context @xcite , i.e., objects co-occurring with the target are factored in to determine which properties have to be realized to make a description unambiguous in a given scenario (see krahmer and van deemter 2012 for a survey). more generally, however, speakers can take on different conceptual perspectives on referents, highlighting different (often not mutually exclusive) facets and aspects of referents, guided by principles beyond pragmatic informativeness @xcite @xcite @xcite . importantly, different conceptualizations can be reflected in object labels or names @xcite , which have been shown to be highly varied and flexible @xcite @xcite @xcite . recent work in vision and language research has started to model this variation @xcite , but general questions remain about how visual context affects conceptualization and naming in humans and generation systems. tangrams in linguistic research tangrams are abstract figures, which are constructed from a small set of geometric primitives and can be more or less nameable, i.e., easy or hard to describe @xcite . unlike natural objects, tangrams lack established naming conventions, triggering diverse figurative descriptions and making them suitable as stimuli to investigate linguistic ref-erence in humans @xcite @xcite @xcite @xcite @xcite and vision-language systems @xcite @xcite . shore et al. (2018); ji et al. (2022) released crowdsourced datasets using tangram figures as stimuli, we use tangrams from the latter for our work. research gap while e.g. manynames (silberer et al., 2020a,c) quantifies naming variation for objects in photographs, object types are often bound to certain contexts (reflecting real-life patterns), and most objects are highly nameable and easy to identify, limiting the range of different conceptualizations. in contrast to this, tangram datasets like kilogram (ji et al., 2022) offer rich variation in conceptualizations, but do not account for contextual influences, as items are described in isolation. in this work, we take a different approach and pair abstract tangram shapes with generated images representing a taxonomy of scene contexts. in this way, we collect diverse descriptions of visual items, which we subsequently analyze for context effects and compare with the predictions of multimodal llms. ## data collection we combine tangram figures with images depicting different types of scenes, and crowdsource annotations to investigate how context affects the conceptualization of inherently ambiguous shapes. formally, each item in our dataset @xmath0, i.e. a tuple @xmath1, s i ⟩. for each item, we collect a set of d @xmath2,s i descriptions in english. from each annotated description, we extract an object label and the corresponding wordnet @xcite synset, to reduce onomasiological variation and facilitate taxonomy-based analyses. ## modelling experiments analyzing human annotations has shown that scene context affects tangram descriptions, i.e., tangrams are often interpreted in ways that align with the scenes they are placed in. in this section, we explore the tangram descriptions in context generated by off-the-shelf multimodal llms. for this, we generate tangram descriptions using the 7b, 13b, 34b and 72b parameter variants of llava-next @xcite foot_2 , and test the outputs for variation, alignment with human data and conceptual shifts, using methods from the preceding analysis. ## discussion and conclusion the role of context in the conceptualization of visual objects and its interactions with variability and creativity in referring are notoriously hard to grasp for (computational) linguistic research. scene-gram addresses this gap, proposing a controlled paradigm that elicits descriptions of conceptually ambiguous tangram shapes in scene context. our results underpin a common theoretical assumption that has, however, been rarely tested "in the wild", especially not in language & vision research: scene context can fundamentally shape speakers' conceptual perspectives when describing a visual stimulus. scenegram shows that tangram descriptions in context remain highly diverse while becoming conceptually more coherent with the scene context. experiments with off-the-shelf multimodal llms indicate that the systems cannot reproduce human variance and conceptualizations of tangrams, but demonstrate general effects of scene context. overall, our results highlight the importance of scene context on object naming at the level of conceptu-alization, pointing to weaknesses of current multimodal language models in this regard. for future work, our data can be used to, e.g., probe more general mechanisms of visuo-linguistic processing in multimodal llms, similar to work in cognitive science or psycholinguistics, where using abstract visual stimuli is a well-established paradigm. at the same time, our approach also raises new questions and directions. more work is needed to understand how exactly context modulates the variance of conceptualizations. data in scenegram suggests that context could prime humans for certain interpretations, effectively reducing variance, or could evoke entirely new interpretations without blocking preexisting ones, pointing to interesting connections to creativity. further work is also needed to understand how conceptual perspective and scene context interact when the communicative effectiveness of object descriptions is at stake, e.g. in a reference game. here, the creative use of language is a necessity in the first place. combining abstract stimuli with contextual information or communicative demands could be a valuable tool for future research to study creativity and linguistic variability in humans and language models.