CIVET : Systematic Evaluation of Understanding in VLM s (2025)

Abstract: While Vision-Language Models (VLMs) have achieved competitive performance in various tasks, their comprehension of the underlying structure and semantics of a scene remains understudied. To investigate the understanding of VLMs, we study their capability regarding object properties and relations in a controlled and interpretable manner. To this scope, we introduce CIVET, a novel and extensible framework for systemati C evaluat I on V ia controll E d s T imuli. CIVET addresses the lack of standardized systematic evaluation for assessing VLMs’ understanding, enabling researchers to test hypotheses with statistical rigor. With CIVET, we evaluate five state-of-the-art VLMs on exhaustive sets of stimuli, free from annotation noise, dataset-specific biases, and uncontrolled scene complexity. Our findings reveal that 1) current VLMs can accurately recognize only a limited set of basic object properties; 2) their performance heavily depends on the position of the object in the scene; 3) they struggle to understand basic relations among objects. Furthermore, a comparative evaluation with human annotators reveals that VLMs still fall short of achieving human-level accuracy.

## introduction recent advancements have shown that vision-language models (vlms) have achieved competitive performance on several vision-language tasks. however, data used to train and evaluate these models is limited in size, and may suffer from annotation errors @xcite , label imbalance @xcite , visual scene biases (e.g. ## literature review several works have evaluated vlms across a wide range of tasks, including vqa @xcite @xcite , reasoning with external knowledge @xcite , counting @xcite , and understanding object relations @xcite @xcite . these works focus on specific tasks without assessing vlms' understanding. indeed, their evaluation is based on real-world visual scenes paired with human-annotated ground truths, which often suffer from issues such as label imbalance @xcite , positional biases (e.g., relevant objects appearing centrally @xcite ), and unknown or uncontrolled scene complexity, including occlusions, distractors, and ill-posed questions. to address some of these limitations, other frameworks have been proposed to assess the performance on visual-language tasks in a controlled setting. while spec @xcite proposed to study vlms' understanding of objects' properties and relations by generating realistic visual scenes, it uses diffusion models, which are known to suffer from hallucination @xcite . unlike civet, spec provides no guarantee of being free of annotation error, hindering the interpretability of results. on the other hand, earlier deterministic approaches, such as shapes @xcite and clevr @xcite , eliminate annotation errors but do not ensure uniform distributions of visual scenes, precluding systematic evaluation. additionally, since the (pre-)training data of vlms is often undisclosed, they may have been partially included in the pretraining. this makes them unsuitable to answer our research questions, as good performance may not reflect vlms' scene understanding. furthermore, the 3d scenes of clevr include additional complexity (e.g., occlusions, reflections, and shadows) that may confound our evaluation. ## civet framework we introduce civet, a framework designed to address the lack of standardized, systematic evaluation to assess vlms' understanding. civet enables systematic investigation of open research questions in the field by leveraging an exhaustive set of controlled visual scenes and natural language inputs. we formalize the framework to make it extensible, allowing researchers to adapt it to diverse evaluation objectives. following the definitions given in ontology @xcite , an object is defined as an instance of an entity foot_0 and is characterized by a given set of properties. a property is a characteristic of an object, such as its shape or color. moreover, the way objects stand to each other is called a relation, such as their relative position (e.g., on top, or in front), or relative size (e.g., smaller, or larger). we define a world as a set of objects and their relations, where objects are characterized by property-value pairs. each world is a structured representation of a stimulus, which can be used to generate a scene (i.e., a visual representation of the world), and a set of natural language questions about its objects, properties, and relations. to evaluate the understanding of an aspect of the scene, we fix that aspect and marginalize over all combinations of the remaining variables. for example, to assess recognition of a particular shape like a star, we consider all scenes that contain a star, and marginalize over variations in color and position. this allows us to isolate the model's understanding of shape by averaging out the influence of other factors. ## experimental settings to rigorously answer our research questions, we need tailored sets of stimuli that are free from annotation error and visual biases, and are balanced in terms of position, property values, and labels. using civet, we systematically evaluate the understanding of five vlms by generating these controlled sets of stimuli. ## evaluation we assess the capability of five state-of-the-art vlms to predict the underlying structure and semantics of a scene. for each model, we report its performance when recognizing the property of an object (rq1), evaluate its robustness to variations in the object positioning (rq2), and measure its ability to identify relations among objects (rq3). finally, we compare the model's performance with that of humans, and the human performance with the ground truth of the stimuli. following preliminary experiments on image and object sizes (see appendix a.3), we select the best setting, i.e., regular size objects with 672 × 672 images for we report results on vlms' recognition of basic properties and absolute positions when analyzing a single, elementary object. to consolidate our findings and avoid overestimating vlms performance, we extend our study by testing these models on real-world objects. single object among our set of properties, shape was the easiest to recognize, with models obtaining at least 95%. conversely, vlms achieved the worst performance when predicting the sheen of the object, with the best accuracy reaching 64%. regarding the color, the llava-next models obtained the worst results, with the smaller 7b model being 12% more accurate than its larger 13b counterpart, suggesting that scaling the text decoder does not always improve performance. when considering colors individually ( ## conclusion in this work, we studied whether state-of-the-art vlms understand the underlying structure and semantics of a visual scene. to respond to the lack of standardized and systematic evaluation, we introduce civet, a framework to systematically assess vlm's understanding via controlled stimuli. our study reveals that 1) vlms are only capable of recognizing certain properties, 2) their performance heavily depends on the position of the object, and 3) they struggle to identify basic relations among objects. moreover, a comparative analysis with human annotators shows that vlms fall short of human-level accuracy. our findings indicate that vlms have limited understanding, which limits their generalization in learning. we encourage further community engagement to extend civet and promote novel training paradigms that are pedagogical rather than utility driven, i.e. fine-tuning for downstream tasks.