Evaluating Compositional Generalisation in VLM s and Diffusion Models (2025)

Abstract: A fundamental aspect of the semantics of natural language is that novel meanings can be formed from the composition of previously known parts.Vision-language models (VLMs) have made significant progress in recent years, however, there is evidence that they are unable to perform this kind of composition. For example, given an image of a red cube and a blue cylinder, a VLM such as CLIP is likely to incorrectly label the image as a red cylinder or a blue cube, indicating it represents the image as a ‘bag-of-words’ and fails to capture compositional semantics. Diffusion models have recently gained significant attention for their impressive generative abilities, and zero-shot classifiers based on diffusion models have been shown to perform competitively with CLIP in certain compositional tasks. We explore whether the generative Diffusion Classifier has improved compositional generalisation abilities compared to discriminative models. We assess three models—Diffusion Classifier, CLIP, and ViLT—on their ability to bind objects with attributes and relations in both zero-shot learning (ZSL) and generalised zero-shot learning (GZSL) settings. Our results show that the Diffusion Classifier and ViLT perform well at concept binding tasks, but that all models struggle significantly with the relational GZSL task, underscoring the broader challenges VLMs face with relational reasoning. Analysis of CLIP embeddings suggests that the difficulty may stem from overly similar representations of relational concepts such as left and right. Code and dataset are available at [link redacted for anonymity].

## introduction compositionality is a fundamental part of how humans learn @xcite . it allows us to take familiar concepts and combine them in new ways to interpret novel situations, learn from limited examples, and build increasingly complex ideas. within formal semantics figure 1 : we evaluate the compositional generalisation of vision-language models (vlms) by assessing their ability to bind colours to objects and relations to objects in both zero-shot and generalised zero-shot settings across single-object, two-object, and relational scenarios as in e.g. @xcite , compositionality is assumed in the formalism. however, vision-language models (vlms) fall short in tasks requiring compositional understanding @xcite @xcite . even with advances in attention mechanisms @xcite and positional encoding @xcite , vlms are unable to match the compositional reasoning skills of humans @xcite . vlms such as clip @xcite have been shown to treat captions as a bagof-words @xcite and are not able to bind concepts to objects in the same way humans can. for example, given an image of a red cube and a blue cylinder, a vlm may misinterpret the image as containing a blue cube or a red cylinder (see figure 1 ). additionally, a vlm should be able to generalise learned concepts to new unseen combinations of attributes and objects: if a model learns the colour cyan through images of cyan cone and the shape cube through images of green cubes, it should also be able to recognise images of cyan cubes or green cones. in formal semantics, given correct representations of the words green, cyan, cube, and cone, this property would naturally occur. diffusion models have gained significant interest in recent years for their state-of-the-art performance on image generation @xcite and editing tasks @xcite . their performance as zero-shot classifiers in vision tasks is a recent topic of exploration @xcite . on compositional benchmarks such as winoground @xcite or the concept binding benchmark from @xcite , their performance has been shown to be comparable to that of clip @xcite . however, winoground has been argued to require commonsense and world knowledge rather than purely testing for compositional abilities @xcite , and performance on the concept binding benchmark can be at chance. in this paper, we contribute to the understanding of the compositional abilities of diffusion modelbased classifiers by comparing with transformerbased classifiers on compositional tasks. specifically, we explore how these two types of models are able to compose attributes and relations-tasks vlms particularly struggle with. we aim to assess whether diffusion classifier can offer new insights or improvements in handling these challenging aspects of compositional semantics. we consider two settings for our experimentszero-shot learning (zsl) and generalised zero-shot learning (gzsl). in zsl, the aim is to recognise only unseen classes whereas gzsl aims to train models that are able to discriminate between both seen and unseen classes during test time @xcite . the gzsl setting is particularly important for real world scenarios as there may only be labelled data for a small number of classes and capturing every possible class in the training set is often impossible. therefore, it is important for models to be able to generalise to unseen classes in the presence of labels that have previously been seen. to probe these abilities, we extend the concept binding benchmark from @xcite , which evaluates model performance on attributeobject binding and relational composition. we evaluate the performance of diffusion classifiera classifier built from stable diffusion (rombach et al., 2022)-comparing it with clip and vilt @xcite . despite the dataset being lightweight, it still proves challenging for the models, particularly in the important gzsl setting. the main contributions of this work are three-fold: (1) we compare diffusion classifier, clip, and vilt on compositional tasks. diffusion classifier generalises best in single-object settings, however, vilt has by far the best two-object performance. all models struggle to reliably compose relations with objects. ## related work benchmarking compositionality in vlms there is a growing interest in the ability of vlms to reason compositionally, with several benchmarks being proposed in recent years @xcite @xcite @xcite @xcite @xcite . compositional generalisation is an important ability for vlms to have because it encourages the interpretability and data efficiency of models @xcite . however, it has been argued @xcite that various compositionality benchmarks are 'hackable', showing that in some cases it is possible to solve the benchmark simply by comparing prompts @xcite and ignoring the image. sugarcrepe @xcite is designed to deal with this problem, but is still prone to the issue that the correct caption is statistically more likely in the training corpus. unlike benchmarks that use complex real-world images, we use simple, synthetic images to ensure no spurious correlations and to directly test compositional understanding. we argue that vlms should be able to handle these simpler reasoning tasks before advancing to more complex, real-world images. improving compositionality in vlms methods have been proposed to improve the compositional abilities of vlms @xcite . several works use hard negative sampling to fine-tune clip on batches of similar images e.g. "a black cat sitting on a desk" and "a black desk sitting on a cat" which force the model to learn more detailed representa-tions of the data @xcite @xcite . other methods include different representations for objects within images such as trees or graphs @xcite and adaptations to the contrastive loss function of clip to include more compositional supervision @xcite . despite advancements, vlms still struggle with compositional reasoning @xcite . our benchmark aims to investigate why vlms struggle with compositional tasks by testing in gzsl settings using in-distribution and out-of-distribution images to identify potential biases. diffusion model classifiers recently, methods have been proposed to leverage diffusion models as zero-shot classifiers @xcite @xcite . li et al. (2023) propose diffusion classifier, a model built from stable diffusion, which achieves a higher accuracy than clip on tasks requiring compositional reasoning such as concept binding. krojer et al. (2023) use a similar method for using stable diffusion (rombach et al., 2022) as a classifier but include a normalising value based on the noise prediction error calculated with no text guidance. he et al. (2023) use the attention scores between the image and text representations of stable diffusion to adapt it for image-text matching tasks. clark and jaini (2023) also propose a zero-shot classifier created from google's imagen, which shows some ability to bind attributes such as shape, size and colour where clip fails to do so. for our experiments we use the diffusion classifier from li et al. (2023) as stable diffusion is open source with easily accessible fine-tuning methods. ## experiments we base the design of our benchmark on the experiments from @xcite where three datasets were created for exploring composition of attributes and relations with objects. while this setup reveals that models often struggle even with simple object compositions, our aim is to extend this evaluation to include both zero-shot learning (zsl) and generalised zero-shot learning (gzsl) settings. to enable this, we adapt and expand the original benchmark to support systematic and rigorous testing in both settings. the images are generated using the generation script for the clevr dataset @xcite to render 3d shapes. the original code included only three shapes cubes, cylinders, and spheres which we extend with an additional shape, cones, to increase the diversity across the dataset splits. for the single and two-object datasets, we consider the following colours: blue, brown, cyan, gray, green, purple, red, and yellow. we define the label sets for the single and two-object datasets as follows: let c be the set of colours and s the set of shapes. for object classification, each object is identified by its colour-shape pair, and the label set is defined as: each element of y represents a unique object (e.g., red square, blue circle). in the two-object dataset, labels consist of two such tuples, e.g., ((c 1 , s 1 ), (c 2 , s 2 )). for the relational dataset, we define a set of spatial relations @xmath0, right}. we exclude the relations front and behind which were included in lewis et al. @xcite as we found these to be too ambiguousdistinguishing which shape is further forward is often difficult even for humans. the relational label set is then defined as: this setup enables evaluation both within the training distribution and on novel combinations, to assess generalisation. we give the structure of our single and two-object datasets within figure 2 . the label red cube is in the test set, meaning that it is not seen during training, but red (e.g. in red sphere) and cube (e.g. in gray cube) have both been seen during training in other combinations. the structure of the relational dataset is given in figure 3 . for both zsl and gzsl tasks, models are finetuned on images and labels from the training split of the data. in the zsl setting, at test time, models must pick the correct label for an image from a set s of unseen labels, i.e. s ⊆ y oodtest . in the gzsl task, at test time, models must pick the correct label for an image from a set of both seen and unseen labels, i.e. s ⊆ y or s ⊆ y rel . this setup evaluates the ability of models to generalise colours or relations learned during fine-tuning to new unseen shape combinations. because of this, the single and two-object train split contains at least one class containing each shape and each colour. similarly, the relational train split contains at least one of each shape. we only use positive examples when fine-tuning clip rather than both positive and negative examples to keep consistent with the dreambooth fine-tuning method for stable diffusion which only accepts positive training examples. in addition, to further align with dreambooth, we fine-tune clip with a small number of samples from each class (20-40 per class). ## results we conduct experiments comparing frozen and finetuned clip, vilt, and diffusion classifier (dc) on three datasets: single-object, two-object, and relational. experiments are carried out in a linux environment using an rtx 2080 gpu for both training and inference. ## model understanding stable diffusion images we compare images generated by frozen and fine-tuned stable diffusion to evaluate what features diffusion classifier is able to learn from fine-tuning on each dataset. we use a guidance scale of 7 and 50 inference steps. examples using prompts from each dataset are shown in figure 5 . frozen stable diffusion is generally very poor at generating images in alignment with the specified prompt, except in the singleobject case. interestingly, the two-object and relational fine-tuned stable diffusion generate three objects fairly frequently showing some pre-training bias and knowledge is still preserved. the rela-tional fine-tuned model fails to understand the difference between the left and right relations with the prompts "a cylinder to the left of a cone" and "a cylinder to the right of a cone" both resulting in images of a cylinder on the left-the class seen during training. clip embeddings we show t-sne visualisations of image and text embeddings from relational dataset examples for frozen and fine-tuned clip. for images, we show the embeddings of 5 samples from each class and only consider classes containing left since corresponding classes containing right use the same images. figure 6a shows the text embeddings which are clearly clustered into quadruples corresponding to prompts where the object shapes are the same, with no clear separation between prompts corresponding to different arrangements of objects. for example, the closest neighbours of cube left sphere are sphere left cube, cube right sphere and sphere right cube. fine-tuning (right-hand plot) fails in most cases to overcome this clustering of similar prompts. an exception is the cluster of prompts containing sphere left cube and cube right sphere, which have been moved closer together, and are visibly distinct from sphere right cube and cube left sphere. other groups of prompts tend to cluster according to ordering of nouns (e.g. cube left cone and cube right cone), or by bag-of-words similarity (e.g. cube left cylinder and cylinder left cube). this inability to distinguish prompts corresponding to different arrangements of objects likely contributes towards clip's inability to correctly caption images with the same shapes but different relations. the t-sne visualisation of image embeddings presented in figure 6b shows that images belonging to the same class are mostly well-clustered. however, there are a few instances of classes in the wrong cluster e.g. a cube left cylinder sample appears within the sphere left cube cluster. notably, we observe that embeddings of images with reversed relational structures tend to occupy similar regions in the space-for instance, cylinder left cone and cone left cylinder appear close together at the bottom of the plot, while cylinder left sphere and sphere left cylinder are both near the left side of the plot. this spatial overlap may contribute to clip's difficulty with relational reasoning. ood classes such as cube left cylinder and sphere left cylinder, which are not directly fine-tuned, appear slightly less well clustered. ## discussion we extend the concept binding benchmark from @xcite to assess concept binding in zero-shot (zsl) and generalised zero-shot (gzsl) settings. using this extended framework, we compare the performance of the discriminative models clip and vilt against a generative model, diffusion classifier, on single-object, two-object, and relational compositional tasks. diffusion classifier shows the highest generalisation accuracy on the single-object task. vilt achieves state-of-theart performance on both zsl and gzsl two-object tasks, demonstrating strong compositional ability in binding attributes to objects even in gzsl settings. diffusion classifier shows some capacity to generalise in the two-object gzsl setting, however, it falls short of vilt's performance. on the relational composition task, all models perform poorly, showing considerable drops in performance on the gzsl from the zsl task showing that hard distractors such as cube left sphere versus cube right sphere are a particular problem. despite initial hopes that diffusion classifier's generative approach might better handle compositionality, relational reasoning remains a major challenge for all models tested. on all our experiments, our fine-tuned clip model consistently outperforms the model from @xcite . we attribute this to our fine-tuning strategy of only using positive examples unlike lewis et al. who use both positive and negative examples. we hypothesise that the inclusion of negative examples exacerbates overfitting. this is due to prompts appearing as negative training examples which then appear as positive examples in the ood splits, causing clip to suppress their prediction. therefore our positive-only approach appears to lead to better generalisation and reduced overfitting. the low performance on the gzsl relational task suggests current vlms may rely too heavily on shortcuts such as object recognition rather than developing structured, compositional representations. our analysis of image and text embeddings in clip further supports this: relational concepts (e.g., left vs. right) are not sufficiently disentangled, especially in the text embedding space, limiting the models' capacity to reason about spatial relationships. potential avenues to address this are training on datasets with more explicit compositional objectives and developing better prompting or fine-tuning strategies that encourage attribute and relation disentanglement. further work in this area also includes analysis of where exactly the models fail: do they fail in forming correct representations of individual words, or do they fail in how these words are combined? this could be tackled using a formal semantic approach that has been integrated with a vector-based semantics, such as in @xcite or @xcite . however, while these routes to improved compositional understanding are important, we argue that our results highlight an important limitation of the tested models as they stand: at present compositional understanding is clearly limited. since there may be a number of aspects of composition that we require models to perform, these should be considered at the pre-training stage rather than expecting users to fine-tune for these fundamental semantic abilities.