Does CLIP Bind Concepts? Probing Compositionality in Large Image Models (2024)

Abstract: Large-scale neural network models combining text and images have made incredible progress in recent years. However, it remains an open question to what extent such models encode compositional representations of the concepts over which they operate, such as correctly identifying ‘red cube’ by reasoning over the constituents ‘red’ and ‘cube’. In this work, we focus on the ability of a large pretrained vision and language model (CLIP) to encode compositional concepts and to bind variables in a structure-sensitive way (e.g., differentiating ‘cube behind sphere’ from ‘sphere behind cube’). To inspect the performance of CLIP, we compare several architectures from research on compositional distributional semantics models (CDSMs), a line of research that attempts to implement traditional compositional linguistic structures within embedding spaces. We benchmark them on three synthetic datasets – single-object, two-object, and relational – designed to test concept binding. We find that CLIP can compose concepts in a single-object setting, but in situations where concept binding is needed, performance drops dramatically. At the same time, CDSMs also perform poorly, with best performance at chance level.

## introduction good semantic representations are generally assumed to require, at a minimum, compositionality and groundedness. that is, meanings of sentences should be functions of the words they contain and the syntax via which those words are combined @xcite , and such meanings should be at least in part responsible for reference to the real world, e.g., via truth conditions (groundedness). the current state-of-the-art of semantic representation consists of vectors extracted from very large neural networks trained either on text alone (devlin et al., 2019; brown et al., 2020; * equal contribution @xcite or a mix of text and images @xcite . it remains a wide-open question whether such models constitute good semantic representations @xcite , with empirical evidence and in-principle arguments simultaneously supporting claims that models are and are not compositional @xcite , and that they are and are not grounded @xcite @xcite . in this paper, we focus on vision-and-language models foot_0 (specifically clip and fine-tuned variants of clip), and seek to answer, in a controlled setting, whether such models meet basic tests of grounded compositionality. specifically, we consider three basic types of linguistic compositions: combining a single adjective and noun (red cube), combining two adjectives with respective nouns (red cube and blue sphere), and relating two nouns (cube behind sphere). all three of these settings require some degree of compositionality and groundedness, with the latter two exemplifying a more abstract type of compositionality (pervasive in language) which depends not only on recognizing a conjunction of constituents but an ability to bind meaning representations to abstract syntactic roles. recently, there has been a significant interest in the community to benchmark the compositional capabilities of clip and other vlms @xcite @xcite . however, @xcite shows that these datasets are 'hackable' as the incorrect labels may not be meaningful and do not require the image to predict the correct label. for example, an image of a horse eating the grass can have the distractor the grass eating a horse. in contrast, we are less prone to such "hackable" artifacts as we include meaningful distractors that require both the image and the labels for the final prediction. we therefore provide a controlled setting for benchmarking compositionality in clip. we situate our work within the tradition of research on compositional distributional semantics models (cdsms) @xcite @xcite @xcite , which seek to bridge the gap between distributional models and formal semantics by building architectures which operate over vectors yet still obey traditional theories of linguistic composition. formal semantics approaches such as @xcite describe how the meaning of a sentence can be built from its component parts. this approach to meaning representation accounts for how a wide variety of expressions can be produced by speakers, and how we can understand sentences that we have never heard before by composing their component parts. phenomena such as inference are also easily accounted for -although there are still difficulties @xcite . distributional semantics approaches represent word meanings according to their distribution in large text corpora. these have been extremely successful in encoding lexical meaning @xcite , as well as in a variety of applications @xcite . cdsms unify these approaches by representing the symbolic, compositional structure of formal semantic models within vector spaces. this allows for the principled compositional approaches seen in formal semantics to be applied within the distributional setting, using lexical meaning representations from the latter arena. cdsms are intrinsically compositional, and because of this, they have the potential to model concept binding effectively. cdsms also have the capacity to capture a range of linguistic and cognitive phenomena @xcite , and lend themselves to modeling the truth value as well as the meaning of sentences @xcite , or accounting for polysemy @xcite . because of their formal background, they are also potentially more interpretable than current large language models. we adapt several cdsms to the grounded lan-guage setting, and compare the performance of clip's text encoder (tuned in various settings) to the performance of these explicitly compositional models. overall, we see that on single adjectivenoun compositions (red cube), clip performs better than any of the more explicitly compositional cdsms. in the other settings, which rely on the ability to bind variables, we see that using cdsms for the text encoder sometimes improves performance, but not always, and that, across all models, performance is essentially at chance in the best case. these results suggest that clip's representation of the visual world is poorly suited for compositional semantics, and suggest that future work on improving these representations is a necessary next step in advancing work on grounded compositional distributional semantics. in summary, we make the following contributions: ## models in this work, we are interested in comparing contemporary "end to end" methods for training neural networks with explicitly compositional models of the type developed in compositional distributional semantics @xcite @xcite @xcite (henceforth cdsms for "compositional distributional semantics models"). below, we describe the models we compare, including baselines, explicitly compositional models, and contemporary vision-and-language models. ## concept binding benchmark we introduce the concept binding benchmark to evaluate the compositional generalization capabilities of vlms. in this benchmark, we introduce three datasets: single-object, two-object, and relational (see figure 1 ). following @xcite , we use community (2018) to generate synthetic datasets with objects of simple shapes and colors. each dataset contains train, validation, and generalization sets with no overlap in the true class labels. class labels are of the form adjective-noun or subject-relation-object. all individual nouns, adjectives, and relations are included in the training sets such that we can train models on the training set and test for compositional generalization on held-out classes in the validation and generalization set. unlike prior work that introduces datasets with a focus on concept binding @xcite @xcite , our synthetically generated datasets contain both semantically meaningful and hard labels and provide a controlled setting to evaluate the compositional capabilities of vlms. ## experiments and results to understand the compositional capabilities of clip, we benchmark clip and the compositional models from section 2 on the three datasets described in section 3. detailed training setup and parameters are given in appendix a. we have released code and datasets for all experiments. foot_1 ## discussion our work highlights the limitations of clip as a basis for compositional language representations. we show that clip is capable of disassociating objects and adjectives, enabling it to behave compositionally in the single-object setting. however, it appears to lack a richer structure necessary for compositions that require more abstraction, such as syntax-sensitive variable binding. we find that fine-tuning clip or training composition-aware models (cdsms) does not help the model generalize better on the unseen classes for two-object and relation settings. our results show that among the clip variants, clip-ft overfits to the training set and achieves high training accuracy while hurting the generalization accuracy. csp can show improved training accuracy over clip and sometimes show increases in validation and generalization accuracy but not always. among the syntax insensitive models, we see that add, mult, and conv improve on the training accuracy on the singleobject and the two-object settings but only add generalizes to held-out classes in the single-object setting. as expected, these models cannot represent order and achieve accuracy close to chance on the relational dataset. our results with type-logical models (tl) have high training accuracy but validation and generalization accuracy are usually close to 0. finally, rf can learn to generalize to classes in the single-object dataset but achieves chance on the two-object and the relational dataset. our experiments focus only on clip, and thus should be interpreted conservatively. newer visual encoders trained with different training objectives may produce better results, even with the same text encoders we use in the paper. or, perhaps, progress on compositionality both in visual and text encoding will be necessary to alleviate the problems highlighted here. overall, our results motivate the need for pretraining methods in vlms that account for binding for better compositionality. we also shed light on the benchmarking datasets used in compositional zero-shot learning. typical benchmarking datasets for this task are mit-states @xcite , ut-zappos (yu and grauman, 2014), and c-gqa @xcite . clip and csp show strong performance compared to several existing methods on these datasets (see section 5 in nayak et al. ( 2023 )). however, these datasets do not explicitly test for binding of adjectives to nouns, i.e., they are restricted to a singleobject setting. while this setting captures one important aspect of composition, it does not require models to encode an abstract, order-aware syntax, a critical component of linguistic composition. in our experiments, we find that clip and csp show high accuracy on the single-object dataset (section 3) but the performance drops dramatically on the twoobject dataset (section 4.2) and relational dataset (section 4.3). challenging datasets like aro @xcite show that fine-tuning clip with harder negative images and captions can improve clip's accuracy on the relational split that accounts for the order of objects. our training setup shares similarities as we include hard negative captions for each image. however, we do not see improved performance after fine-tuning. recent work @xcite shows that the aro benchmark includes test examples that can be solved without the visual encoder which could explain the possible improvement in performance. these findings motivate the need for more realistic and challenging benchmarks that test for binding and order. ## related work compositionality in language our work contributes to the extensive body of work in compositionality and language spanning several decades @xcite @xcite @xcite @xcite . key models of composition used in language include simple elementwise composition @xcite , neural models of composition @xcite , type-logical models of composition @xcite , and role-filler modes of composition @xcite @xcite . we focus on type-logical and role-filler models of composition. in the area of type-logical models, our work extends models from @xcite ; @xcite ; nagarajan and grauman (2018) to learn from both images and text and to handle a wider range of compositions. within the area of role-filler approaches, recent work has looked at approaches to reasoning @xcite , mathematics @xcite , and whether recurrent neural networks can be emulated using role-filler approaches @xcite . in particular, @xcite use tensor product representations to show that sentence encoders @xcite can be well approximated by a "bag of words" model. in this work, we show that clip image embeddings behave like a "bag of concepts". ## conclusion we investigate the ability of clip and variants and cdsms in a controlled environment to perform compositional visual reasoning tasks. our results show that clip performs well on the single adjective-noun compositions but struggles on compositional tasks that rely on the ability to bind variables. some of the cdsms perform well on single adjective-noun composition but show performance closer to chance in the two-object and relational tasks. our work not only sheds light on the limitations of clip but also suggests that the pretraining of vlms should account for binding and order for better compositional generalization.