W HEN AND W HY V ISION -L ANGUAGE M ODELS B EHAVE LIKE B AGS -O F -W ORDS , AND W HAT TO D O A BOUT I T ?
A BSTRACT
Despite the success of large vision and language models (VLMs) in many downstream applications, it is unclear how well they encode the compositional relationships between objects and attributes.
Here, we create the Attribution, Relation, and Order (ARO) benchmark to systematically evaluate the ability of VLMs to understand different types of relationships, attributes, and order information.
ARO consists of Visual Genome Attribution , to test the understanding of objects’ properties; Visual Genome Relation , to test for relational understanding; and COCO-Order & Flickr30k-Order , to test for order sensitivity in VLMs.
ARO is orders of magnitude larger than previous benchmarks of compositionality, with more than 50, test cases.
We present the settings where state-of-the-art VLMs behave like bagsof-words—i.e. when they have poor relational understanding, can blunder when linking objects to their attributes, and demonstrate a severe lack of order sensitivity.
VLMs are predominantly trained and evaluated on large scale datasets with rich compositional structure in the images and captions.
Yet, training on these datasets has not been enough to address the lack of compositional understanding, and evaluating on these datasets has failed to surface this deficiency.
To understand why these limitations emerge and are not represented in the standard tests, we zoom into the evaluation and training procedures.
We demonstrate that it is possible to perform well on image-text retrieval over existing datasets without using the composition and order information.
This further motivates the value of using ARO to benchmark VLMs.
Given that contrastive pretraining optimizes for retrieval on large datasets with similar shortcuts, we hypothesize that this can explain why the models do not need to learn to represent compositional information.
This finding suggests a natural solution: composition-aware hard negative mining.
We show that a simple-to-implement modification of contrastive learning significantly improves the performance on tasks requiring an understanding of order and compositionality.
I NTRODUCTION
Vision and language models (VLMs) have demonstrated high performance on dozens of wellestablished benchmarks (Radford et al., 2021;
Li et al., 2022; Singh et al., 2022; Alayrac et al., 2022; Wang et al., 2022a;b; Zhai et al., 2022).
Yet it is unclear whether high performance on these benchmarks indicates rich compositional understanding of either text or images.
For example, does
CLIP distinguish between “the man is eating the sandwich” and “the sandwich is eating the man”?
Natural scenes are complex, composed of many objects and attributes, in relationships with one another.
While there have been important efforts to test compositional representations of objects, attributes, and relations (Thrush et al., 2022), such efforts are based on small sets of hand-crafted examples, often combined with testing many other types of knowledge.
This makes it hard to evaluate the role of relational and attributional knowledge in isolation and lacks the statistical power to quantify how well VLMs perform on granular subtypes of compositions.
Here, we provide a large-scale test bed to evaluate VLMs’ attribution, relation, and order understanding.
Using the test bed we create, we find significant deficiencies: many models fail to perform beyond the chance level at simple tasks requiring compositional understanding.
Many VLMs are pretrained and tested on large datasets with complex scenes and detailed captions with rich compositional structure.
Yet, training on these datasets has not been enough to address the lack of compositional understanding, and evaluating on these datasets has failed to surface this deficiency.
In the recent literature, the dominant VLM training paradigm is image-text contrastive pretraining (Jia et al., 2021;
Radford et al., 2021; Zhang et al., 2020) over these large pretraining datasets.
Contrastive pretraining optimizes for the task of image-text retrieval, and naturally many
VLMs are tested in the retrieval task.
In this work, we provide an analysis of retrieval, as an evaluation and objective.
We propose experiments to analyze how these models are evaluated and trained, to understand the underlying issues.
Our main contributions are three-fold:
Introducing the Attribution, Relation, and Order benchmark (ARO) for fine-grained evaluation of VLMs’ relation, attribution, and order understanding.
We present four new tasks: Visual Genome Attributions and Visual Genome Relations to test the understanding of objects’ attributes and relations in complex natural scenes; and COCO(Order) and Flickr30k(Order) , to test the models’ ability to identify the correct ordering of the words in a caption (Section 2).
Using these evaluations
we show that state-of-the-art VLMs fail to represent simple relations such as “to the right of” and “behind”, fail to represent the attributive difference between “the black jacket and the blue sky” versus “the blue jacket and the black sky”
and fail to represent the difference between correct and permuted captions.
We provide fine-grained insights into the types of attributions and relations that models most frequently fail to understand.
A critique of retrieval and contrastive pretraining.
Given we find VLMs exhibit poor compositional understanding, why have these issues not surfaced in many previous evaluations?
Existing retrieval datasets are equipped with complex scenes and detailed descriptions as captions, likely full of rich compositional structure.
Intriguingly, the models can perform well on retrieval without having a good compositional understanding.
Our experiments (Section 3) show that models can achieve a high performance on retrieval even when the order and composition cues are removed from captions or images.
Hence, it is natural that models with compositional deficiencies can still perform well on the standard evaluations.
This suggests that standard retrieval tasks are limited in their ability to assess compositional understanding of the model , further motivating the need for our comprehensive ARO benchmark.
Since contrastive pretraining optimizes for retrieval, our findings also show that models can perform well on contrastive pretraining without learning compositional information.
Given our results, we argue that not learning the compositional information is a valid shortcut strategy (Geirhos et al., 2020), and VLMs have little incentive to learn to encode compositionality during contrastive pretraining.
Composition-aware hard negatives can go a long way.
We propose a simple fix: mining of composition-aware hard negatives (Section 4).
First, we introduce hard negatives consisting of the nearest neighboring images into each batch, to force models to represent fine-grained differences between very similar scenes.
Second, we introduce hard negative captions into each batch, consisting of the true captions with word order perturbed, to force models to distinguish between correct and incorrect order.
Finally, we show that this simple finetuning modification provides significant improvements in model understanding of attributes and relations.
A TTRIBUTION , R ELATION , AND O RDER (ARO) B ENCHMARK : W HEN DO MODELS BEHAVE LIKE A BAG - OF - WORDS ?
Whereas humans effortlessly parse natural scenes containing rich objects in relation to one another, it is unclear whether machines understand the complexity of these scenes.
To do so, models must be able to correctly represent objects, their attributes, and the relations between objects.
Recent research has started to probe VLMs’ for such information.
Thrush et al.
(2022) proposed Winoground, a dataset of test cases documenting a clear lack of compositional and pragmatic understanding in
VLMs.
The dataset is high quality but relatively small scale; its 400 test cases cover a wide range of linguistic phenomena (e.g., relation, pragmatics, world knowledge), making it hard to render statistically significant results about fine-grained relational and attributive abilities.
In concurrent work, Diwan et al.
(2022) suggest that Winoground has further challenges beyond compositionality [Figure] such as requiring common-sense reasoning or strong world knowledge.
This makes it harder to render specific conclusions about compositional understanding.
Here we introduce a large dataset for a fine-grained and targeted evaluation of VLMs’ ability to encode relations, attributes and order
. 2. N EW B ENCHMARKS FOR A SSESSING R ELATIONAL AND A TTRIBUTIVE U NDERSTANDING
We leverage Visual Genome (VG) (Krishna et al., 2017) – a large-scale dataset with over 100, images, annotated with objects, attributes, and relations – and the high-quality GQA annotations (Hudson & Manning, 2019) established in past work.
Building upon these, we generate two novel datasets for probing relation and attribution understanding:
• Visual Genome Relation.
Given an image and a constituent relation of the form X relation Y , we test whether the model can pick the correct order.
Specifically, we probe models to pick between X relation Y and Y relation X with test cases from various relations, such as prepositional relations (e.g. “the man is behind the tree ’ vs “the tree is behind the man ”) and verbs (e.g. “the man eating the sandwich ” vs “the sandwich is eating the man )”).
• Visual Genome Attribution.
We test the ability to attribute the properties of objects appropriately.
For instance, we ask the model to pick between “the crouched man and the open door ” and “the open man and the crouched door ”.
We extract images with 48 relations including eating, sitting on, inside , and below , with 23 , 937 test cases in total; and 117 unique attribute pairs including ‘gray vs wood’, ‘open vs white’ , and ‘small vs brown’ , with 28 , 748 test cases in total.
The details of the dataset generation procedure are presented in Appendix A, along with the full list of relations with the count statistics.
In brief, we mined
Visual Genome for cases in which both of the constituent objects for the relations/attributes took a meaningfully large space in the image, and extracted the smallest bounding box containing both of the constituents, presenting this cropped image alongside the correct and swapped relation/attribute annotation.
Each test case is thus made of an image (e.g., the image of a man eating a sandwich) a correct caption (e.g., “the man is eating the sandwich”) and a grammatically correct, but swapped, caption (e.g., “the sandwich is eating the man”).
For each of the test cases in these datasets, we quantify the performance of each model in identifying the correct caption from the two alternatives, i.e. chance level performance here is 50%.
Examples of these tests can be seen in Figure 1.
2. N EW B ENCHMARKS FOR A SSESSING O RDER S ENSITIVITY
Whereas Visual Genome Relation and Visual Genome Attribution assess the model’s ability to understand order and compositionality related to attributes and relations , we also wish to discern whether this is connected to a broader inability to represent word order in general .
With this motivation, we specifically want to test the order sensitivity of models.
Do models broadly exhibit any preference towards the correct ordering of the words in a scene description, or are they indifferent towards any permutation, even for those that are unreasonable?
We propose an additional stress test, to test the models’ ability to pick the correct ordering of the words within a caption.
Given an image, we probe the models to pick between the correct ordering of a caption versus alternatives where the words are reordered in systematic ways.
Given four systematic permutations of a given caption and the caption itself, can the model ‘pick the right caption’?
We augment existing retrieval datasets to derive COCO - Order , and Flickr30k - Order (Lin et al., 2014;
Young et al., 2014).
To generate these datasets, we utilize four different perturbations of a caption, provided in Table 1.
These largely follow prior work that evaluates language models (O’Connor &
Andreas, 2021).
We use Spacy (Honnibal & Montani, 2017) for part-of-speech tagging to perform the perturbations.
2. E VALUATING VLM S ON ARO
We evaluate four state-of-the-art VLMs: CLIP (Radford et al., 2021), BLIP (Li et al., 2022),
Flava (Singh et al., 2022), and X-VLM (Zeng et al., 2022b).
More details on these models can be found in Appendix B.
Models exhibit deficiencies in compositional understanding: In Figure 1, we present model performance on the Visual Genome Relation and Visual Genome Attribution evaluations.
In relation tests, we observe that most models are near or below the chance level, indicating severe deficiencies in relational understanding.
In Appendix Table 2, we provide the performance on each relation separately.
For instance, while BLIP is relatively accurate at understanding positional relations, its performance is generally near the chance level for verbs, such as ‘eating’ or ‘watching’; whereas CLIP generally performs at the chance level on positional relations.
Quantitatively, while BLIP obtains 66% macro accuracy for spatial relations, it obtains 56% accuracy in verbs.
In contrast, CLIP achieves 56% in spatial relations and 61% in verbs.
In attribution tests, although BLIP ( 88% ) and XLVM ( 87% ) perform remarkably well, CLIP ( 62% ) is again close to the chance level.
While Flava is reasonably good at attribution ( 73% ), its performance is below-chance for relations ( 25% ).
Overall, VLMs exhibit significant deficiencies in compositional understanding, particularly relational understanding.
These deficiencies motivate our interest in probing whether these models are failing to represent compositional information in particular — e.g. failing to represent the order of relation constituents — or whether models are in fact failing to represent word order more broadly.
Perturbation Type
[Table]
Models have little to no preference toward correctly formed sentences:
In Figure 1, we present the COCO/Flickr30k order task and the performance of the tested VLMs.
Given an image, the
VLM must pick between the original caption and four alternative captions, augmented with the 4 perturbations listed in Table 1 respectively; thus, chance-level performance is 20%.
Given the randomness over the permutations of captions, we repeat the experiment with 5 different seeds and present the mean performance and error bars.
All numerical values can be found in Appendix 5.
Overall, models exhibit different levels of preference toward the correct ordering.
For instance, while BLIP performs relatively well on the earlier tasks compared to CLIP, we observe here that its performance is much closer to the chance level.
Similarly, while Flava obtains a good performance with the Attribution task, its performance is below the chance level for the COCO-Flickr30k Order task.
Connection to prior evaluations with text-condition image generation: We observe that CLIP cannot identify the correct ordering of constituents in a relation.
We take the view that this is in line with prior observations, where it is shown that text-conditioned image generators that use CLIP as the text encoder, struggle with generating images that are faithful to the relations in the descriptions (Conwell & Ullman, 2022).
We hypothesize that the problem may lie with CLIP’s inability to encode order.
Given that Imagen (Saharia et al., 2022) has better results on compositionality tests, we speculate that this can be because they use T5 (Raffel et al., 2020), a language model, as the text encoder.
We believe our results suggest there is a promising potential for having language model priors in VLMs.
W HY DO MODELS BEHAVE LIKE BAG - OF - WORDS ? A CRITIQUE OF RETRIEVAL AND CONTRASTIVE PRETRAINING
Given that VLMs exhibit poor compositional and order understanding, why have these issues not surfaced in many previous evaluations?
Most VLMs are consistently evaluated over image-to-text retrieval, and demonstrate impressive performance.
In this section, we demonstrate why retrieval can be problematic, both as an evaluation and as an objective.
We first show that models can perform well in the text-image retrieval evaluations on existing large-scale datasets, without using order or composition information.
It is thus natural that the issues of lack of compositional and order information have been masked by the celebrated performance.
Next, we discuss the connection between contrastive pretraining on large datasets to the task of retrieval and argue that models may not have an incentive to learn composition and order.
3. L IMITATIONS OF RETRIEVAL AS AN EVALUATION
Even though existing retrieval datasets are equipped with complex scenes and detailed descriptions, it is unclear how much complexity the models need to understand to perform well on this task.
In particular, do the models have to use compositional information to perform well in a large-scale retrieval task?
To understand what it does and does not take to perform well on retrieval, we propose evaluations with modified datasets.
First, we propose augmentations to the existing datasets, where we remove the order and composition cues.
Next, we evaluate models on these augmented versions of these datasets to understand whether it is possible to perform well without these cues.
Datasets: We zoom into two of the standard cross-modal text-image retrieval datasets, namely
COCO (Lin et al., 2014) and Flickr30k (Young et al., 2014).
Following prior work (Li et al., 2022), we use Karpathy splits (Karpathy & Fei-Fei, 2015) for both of the datasets.
We test the models on the test splits; where COCO contains 5k images, and Flickr30k contains 1k images.
We report Recall@ and Recall@5 for both datasets.
Experimental Protocol:
Our goal is to understand whether models need order information / compositional understanding to perform well on existing text-image retrieval datasets.
We test our hypothesis on the augmented versions of the existing datasets.
We propose two augmented setups (see Figure 2):
Perturbing the order and composition information in the captions:
To understand if models need the compositional information in captions to perform well, we aim to remove these, and test whether models can perform well without this information.
In this case, we take the permutations of the words in a given caption, using the strategies specified in Table 1.
For instance, we take the COCO dataset, shuffle all of the words in all of the captions and compute the retrieval performance over this caption-modified dataset.
Given that all words are shuffled, the compositional structure within the captions is generally altered, e.g. “the sandwich is eating the man” becomes “eating the sandwich man a”.
Perturbing the order and composition information in the images : Similar to captions, to understand if models need the compositional information in the images to perform well, we aim to test the models in the absence of these.
As it is harder to manipulate entities in an image, we resort to a more severe method and take the permutations over the patches of an image.
For instance, we split images into 9 equally sized patches and take a permutation of those patches to form the new images.
Qin et al.
(2021) used a similar strategy to show the insensitivity of vision transformers to patch-based augmentations, in the context of image classification.
Again, this should alter most compositional structures within an image; for instance, an object that is “below” another can move into an arbitrary point in an image.
We then compute the retrieval performance with this image-modified dataset.
We explore three such strategies: splitting the image into either 4 equally-sized rows, 4 equally-sized columns, or 9 equally-sized patches and then shuffling.
[Figure]
Models can achieve high performance even when order information is inaccessible. In Figure 2, we present the retrieval performance of existing models under different augmentation strategies, and a detailed table can be found in Appendix 3,4.
For each perturbation, we augment the dataset and compute the retrieval performance 3 times with different random seeds, and report standard error bars.
Notably, across all of the mentioned perturbation strategies, most models lose marginal performance in performing the retrieval task with the perturbed caption, or the perturbed image.
These results demonstrate that it is possible to obtain high performance in a retrieval task without utilizing the compositional structure, even in the case of these arguably large-scale datasets.
3. L IMITATIONS OF RETRIEVAL AND CONTRASTIVE PRETRAINING AS AN OBJECTIVE
To understand why these deficiencies emerge in the first place, we discuss the training procedure of VLMs.
Most state-of-the-art VLMs, at their core, are trained with a contrastive loss (Radford et al., 2021;
Chen et al., 2020; Zeng et al., 2022a; Li et al., 2021; 2022; Zhang et al., 2020) on a large pretraining dataset scraped from the web.
We hypothesize that the lack of compositional understanding can be attributed to the way the models are trained.
Here, we discuss the connection between contrastive pretraining to retrieval, to better understand the underlying phenomenon.
Quoting from Radford et al.
(2021): “CLIP pre-trains for the task of image-text retrieval on our noisy web-scale dataset”.
The goal in contrastive text-image pretraining is to optimize the model to identify the matching pairs of caption and text; namely retrieval .
Simple as it is, training models with a retrieval objective, on large pretraining datasets, has demonstrated remarkable off-the-shelf performance, for instance in tasks requiring single object recognition.
Although the existing large retrieval/pretraining datasets can have long and detailed captions, our results demonstrate that it is possible to perform well on these datasets without using compositional structure.
Accordingly, these results provide evidence that models can achieve high performance in retrieval objectives, thus also obtaining low contrastive loss, without order information, unless the datasets are carefully designed .
These datasets are designed to cover a large conceptual and semantic space, in order to make models broadly useful for downstream tasks; yet these datasets are not designed to contain many images with captions containing similar words that must be differentiated.
Without such alternatives in the dataset, the task can be solved without taking order information into account — and behaving like a bag-of-words becomes a high-reward strategy.
There is a large body of evidence showing that neural networks are prone to exploiting shortcut strategies (Geirhos et al., 2020; 2019), and our findings demonstrate that not learning the order information is a valid shortcut strategy, for general-purpose retrieval/captioning datasets.
We thus argue that it is unclear what should incentivize models trained with a contrastive loss to learn to pay attention to order structure unless the datasets or algorithms are carefully designed with this consideration.
A SIMPLE FIX : C OMPOSITION - AWARE HARD NEGATIVES
Our analysis of the retrieval objective and datasets leads to a natural solution: hard negatives for contrastive learning (Robinson et al., 2021; Kalantidis et al., 2020).
We propose a natural extension of CLIP’s contrastive objective to alleviate some of the issues discovered in previous sections.
To make CLIP sensitive to word order and better at capturing compositions, we use strong alternatives:
Generation of negative captions : For each image-caption pair, we generate a negative caption by swapping different linguistic elements: noun phrases, nouns, adjectives, adverbs, verb phrases .
For example, the caption “The man is eating the sandwich and the woman is watching the television” either becomes “The woman is eating the sandwich and the man is watching the television” (noun swapping) or “The man is watching the sandwich and the woman is eating the television” (verb phrase swapping).
Sampling strong alternative images: To generate images that are strong alternatives to the images in a batch, we first use CLIP to compute the pairwise similarity between all images in the dataset.
During training, for each image in the batch, we sample one of the K = 3 nearest neighbors as the strong alternative image.
The sampled alternative images (and the respective captions and negative captions) are added to the batch.
We first briefly describe the standard contrastive methodology used in CLIP: Let f i : X image → R d be the image encoder, f t : X text → R d be the text encoder for a VLM.
In the CLIP objective, given a batch of N image-caption pairs, the goal is to predict the actual pairings that occurred.
Given a batch of N images I N =
{ I 1 , I 2 , ..., I N } and N captions T N =
{ T 1 , T 2 , ..., T N } , CLIP first computes the matrix of cosine similarities, denoted by S ∈ R N × N , where each item S j,k computes the cosine similarity between the image j and caption k , namely S j,k = f i ( I j ) ·
f t ( T k ) ||
f i ( I j ) ||·||
f t ( T k ) || .
Using these similarities, the row-wise and column-wise cross-entropy losses are computed.
For composition-aware mining of hard negatives, we propose a simple modification to the CLIP objective.
In Figure 3 we give an overview of the procedure.
Given a batch of images I N , captions
T N , we first generate negative captions T − N , and then concatenate the two matrices to obtain ( ˜T 2 N ).
Next, we again compute the similarity matrix ˜S ∈ R N × 2 N .
Here, the row-wise and column-wise cross-entropy losses are computed as in CLIP, with the difference that we do not compute the loss for the negative captions column-wise (as there is no matching image for a negative caption).
Experimental protocol:
Due to the computational cost of training CLIP from scratch, we only focused on finetuning experiments.
Specifically, we finetune the ViT-B/32 variant of CLIP on the
COCO dataset with hard negatives (NegCLIP).
As an ablation, we also perform finetuning on COCO, without the sampled hard negatives to disentangle the effect of finetuning.
The details of finetuning, hyperparameter selection, and ablation are provided in Appendix C.
Evaluation: We propose two main sets of evaluations.
First, we evaluate models on four of the order and composition-sensitive tasks, namely Visual Genome Relation , Visual Genome Attribution , COCO & Flickr30k Order .
In addition to these, to ensure that the model is still comparable to the original
CLIP, we perform evaluations on five downstream tasks: CIFAR10, 100 (Krizhevsky et al., 2009) and ImageNet (Deng et al., 2009) for image classification; and Flickr30k and COCO for retrieval.
Results:
In Figure 3, we provide a comparison of CLIP to NegCLIP with a radar plot for an overview; numerical values can be found in Appendix Table 6 with the additional ablation on a CLIP model fine-tuned on MSCOCO without negative samples.
NegCLIP does not suffer in downstream tasks, and it improves the performance on VG-Relation from 63% to 81% , on VG-Attribution from 62% to 71% , on COCO-Order from 46% to 86% , and on Flickr30k-Order from 59% to 91% . Further,
[Figure] in VG-Relations, NegCLIP becomes the best model, in comparison to all other models, and in
VG-Attribution it becomes comparable to X-VLM and BLIP.
Overall, we observe that NegCLIP does not have a substantial loss in performance on the downstream tasks, yet provides substantial gains on the order-sensitive tasks.
Our results highlight that mining targeted and cheap negatives can lead to substantial improvements in compositional tasks, without causing a loss of performance in important downstream tasks.
While contrastive learning provided substantial improvements in representation learning, it is unclear if simply scaling the size of pretraining datasets will be as efficient as exploring algorithmic improvements, especially in learning the compositional structure.
We believe our results provide evidence that seeking such modifications can bring various improvements.
We do not suggest that ours is the best possible model for encoding relations; rather, in order to represent these bits of compositional information, our addition of targeted negatives should be considered as a promising option for improving contrastive learning.
R ELATED W ORK
Visio-linguistic compositionality: Understanding what aspects of language and vision VLMs capture is the main objective of several recent papers.
Frank et al.
(2021) suggest that information sharing between the text and vision modalities is not balanced, as the representations from the text encoder are more influenced by the vision modality than the other way round.
Parcalabescu et al.
(2021b) show that VLMs have difficulties in counting objects in images.
In terms of the evaluation part of our paper, Winoground (Thrush et al., 2022) presents the nearest neighbor to our work.
Winoground is a carefully curated dataset of 400 test cases that aims to evaluate compositional and pragmatics language understanding of VLMs where VLMs perform around chance level over a set of image-text matching tasks.
Diwan et al.
(2022) suggest that Winoground has challenges beyond compositionality that requires complex reasoning with visually/textually difficult or unusual examples.
Our main goal is to propose an isolated fine-grained evaluation of relation and attribution understanding with a large scale dataset; hence we believe our dataset complements Winoground.
While Winoground proposes a great breadth; we attempt to focus on the depth of testing for relation and attributive understanding.
Bogin et al.
(2021) utilizes VG to test VQA systems for compositional generalization; in comparison here our purpose is to implement tasks that can probe any VLM.
Closer in spirit to our VG - Relations dataset, Conwell & Ullman (2022) demonstrate the lack of competency of a text-to-image model,
DALL-E, in generating images faithful to the relationships described in the textual prompt.
Zerroug et al.
(2022) design a benchmark to evaluate vision models’ understanding of simple compositional attributes of abstract shapes.
GQA (Hudson & Manning, 2019) is a visual question-answering dataset derived from Visual Genome (Krishna et al., 2017) to test visual question answering systems for scene understanding with compositional question answering.
Other VQA datasets such as
NVLR (Suhr et al., 2017; 2018) cover a broad range of linguistic phenoma including compositionality.
VALSE (Parcalabescu et al., 2021a) is a dataset that tests whether VLMs can identify the correct linguistic phenomenon appearing in an image.
Recently, Saharia et al.
(2022) released a set of prompts, collectively called DrawBench, as a benchmark for text-to-image models’ ability to generate images faithful to the given challenging prompts, involving a set of compositional tests.
Previous evaluations, while informative, were limited by small data size; Winoground has 400 examples, limiting the statistical strength of the fine-grained conclusions.
Our ARO dataset (50,000 test cases) is 2 orders of magnitude larger, enabling us to characterize the performance for different types of compositions.
Moreover, previous works did not quantify how using composition-aware hard negatives in contrastive learning could affect VLMs’ performance for compositional tasks.
Order information in language and vision: Several works have highlighted the lack of word order sensitivity in large language models Hessel & Schofield (2021); O’Connor & Andreas (2021); Pham et al.
; Sinha et al.
Sinha et al.
(2021) show that pre-training BERT with sentences with shuffled words, marginally affect the performance on down stream tasks.
Pham et al.
(2021) show that some of the tasks in GLUE (Wang et al., 2018) can be solved even when disregarding word order.
Our analysis of image retrieval leans in a very similar direction, suggesting the need for more careful benchmarks.
O’Connor & Andreas (2021) show that for long-range contexts, models use content words and local co-occurrence statistics to make predictions.
Ettinger (2020) uses a set of psycholinguistic tasks to evaluate the linguistic and contextual information used by BERT, showing
BERT’s insensitivity to elements like negation.
On the vision side, Brendel & Bethge (2019) show that a bag-of-local-features model performs almost as well as their state-of-the-art counterparts.
Closer to our experiments, the work of Tejankar et al.
(2021) shows that training contrastive vision language models using only bag-of-words in place of the caption does not significantly hurt performance on zero-shot classification.
Our work generalize these results, showcasing the general limits of vision language models when dealing with relations, attributes and shuffled captions.
Negative mining and contrastive learning : Using hard negatives has been successful in representation learning (Harwood et al., 2017; Wu et al., 2017; Ge, 2018).
Furthermore, hard negatives have also been shown to improve contrastive learning (Kalantidis et al., 2020;
Robinson et al., 2021) or used with a contrastive loss to improve ViTs (Qin et al., 2021).
Our work differs in that we propose to explore hard negatives with contrastive learning, particularly in the context of vision-language models and compositional abilities.
Note that Li et al.
(2021) and Li et al.
(2022) use negative mining by selecting pairs of items with high similarity.
However, in light of our results, this strategy alone does not seem to be enough to effectively train the model to deal with relationships and word order.
C ONCLUSION
In this work, we evaluate VLMs for the ability to encode composition and order structure, introducing large-scale test beds to generate fine-grained and statistically strong insights.
We show that models struggle with relation, attribution, and order understanding, and our datasets revealed different limitations of models.
We show that models can achieve high performance on the task of crossmodal retrieval without needing to learn order and composition information.
Given that contrastive pretraining optimizes models for the task of retrieval, we argue that this can explain why
VLMs need not learn to encode order and compositional cues.
Using these insights, we proposed a simple modification to the training procedure, namely composition-aware hard negative mining.
Through several evaluations, we demonstrate that by generating composition-aware hard negatives during model training, we can improve the compositional and order understanding of VLMs.
Overall, our work demonstrates the importance of the interaction between the pretraining objective and large datasets VLMs are trained on.
In this work, we focused on finetuning of VLMs for demonstrating the use of the composition-aware negative mining.
For future work, we are interested in further exploring composition-aware contrastive pretraining of VLMs.
Given that VLMs are trained with large datasets with rich text corpora, the limited language understanding of these models are intriguing.
While here we specifically focused on contrastive learning, in light of our findings, studying the interaction between different pretraining objectives and compositional understanding is an interesting future avenue.
Our results further highlight the importance of rich evaluations of
VLMs.
We hope future VLMs will release results on these fine-grained evaluations in addition to standard tasks, and more fine-grained evaluations will be developed. A CKNOWLEDGMENTS
We would like to thank Adarsh Jeewajee, Candace Ross, Duygu Yilmaz, Edward Chen, Kyle
Swanson, Rishi Bommasani, Tristan Thrush, Tuomas Oikarinen, and Weixin Liang for their support and comments on the manuscript, and all members of the Zou Lab, Jurafsky Lab, and Guestrin
Lab for helpful discussions.
We thank the anonymous reviewers for their suggestions to improve the paper.
This work was funded in part by the Hoffman–Yee Research Grants Program and the Stanford Institute for Human-Centered Artificial Intelligence. P.K. is supported in part by the Open Philanthropy AI Fellowship.
J.Z. is supported by NSF CAREER 1942926 and the Chan-Zuckerberg Biohub. E THICAL S TATEMENT There is a large body of literature illuminating contrastive models like CLIP and BLIP are enmeshed in a variety of biases (see, for example, the work by Wolfe & Caliskan (2022)).
Our work uses standard datasets in the literature and we do not foresee additional issues created by our-fine tuning or methodology.
Nonetheless, our dataset, based on Visual Genome, inherits stereotypical patterns present in the original data (e.g., the original dataset has more images of men than women (Krishna et al., 2017)).
R EPRODUCIBILITY S TATEMENT
The code to reproduce all experiments, along with the code to generate the datasets and tasks we propose are released at https://github.com/mertyg/ vision-language-models-are-bows Experiments on caption perturbation have been run with three different seeds to take into account the randomness of the permutation methods.
All of the models that are used were obtained from the checkpoints released with the respective paper.
Particularly, we obtained the checkpoints released at BLIP 2 , X-VLM 3 , CLIP 4 , and Flavarepositories.
Experiments with CLIP and NegCLIP have not been run multiple times due to the computational requirements.
However, the experiments have been run using fixed seed, so they can be replicated by other researchers.
In addition to this, our NegCLIP is implemented as a fork of the open clip project 6 .
The code will be released with the same license.
A P ROPOSED D ATASETS A. G ENERATING THE D ATASETS
We follow the below steps while generating the datasets:
We go through the scene graphs annotated in GQA (Hudson & Manning, 2019).
First, we identify candidate objects in all scenes.
Namely, we require the objects to be large enough such that they are recognizable in an image.
We enforce a heuristical criterion, where we discard all objects that have a width lower than 14 of the width of the entire image, or with a height lower than 14 of the height of the entire image.
For VG-Relation, we identify all pairings of objects, where we make sure that the pair of objects are not from the same category (say, both are not simultaneously “dogs”).
Similarly, for VG-Attribution, we identify all pairs of objects that are modified by at least 1 attribute, where both objects and attributes are different from each other.
After identifying the pairs of objects, we extract the smallest bounding box containing both of the objects from the scene.
This is to minimize the distraction in the rich scenes in Visual Genome.
Finally, for each identified pairs of objects, we fill the preset templates.
For relations, we fill the templates of “the [object 1] is [relation] [object 2]” for the true caption, and “the [object 2] is [relation] [object 1]” for the false caption.
For attributes, we fill the templates of “the [attribute 1] [object 1] and the [attribute 2] [object 2]” and “the [attribute 2] [object 1] and the [attribute 1] [object 2]”.
For Visual Genome Relations, we post-process the relations to remove symmetric relations; such as “near” or “next to”.
Overall, this process results in a set of 23 , 937 test cases for relations, and 28 , 748 test cases for attributes.
[Table]
12 covered by 0.
36 covered in 0.
14 covered with 0.
16 covering 0.
33 cutting 0.
12 eating 0.
21 feeding 0.
10 grazing on 0. 10 hanging on 0.
14 holding 0.
142 leaning on 0.
12 looking at 0.
31 lying in 0.
15 lying on 0.
60 parked on 0.
21 reflected in 0.
14 resting on 0.
13 riding 0.
51 sitting at 0.
26 sitting in 0.
23 sitting on 0.
175 sitting on top of 0.
10 standing by 0.
12 standing in 0.
59 standing on 0.
52 surrounded by 0.
14 using 0.
19 walking in 0.
10 walking on 0.
19 watching 0.
22 wearing 0.
B M ODELS
[Table]
[Table]
No
[Table]
BLIP
[Table]
only
[Table]
BLIP
[Table]
[Table] words [Table] [Table]
[Table] [Table] [Table] [Table]
BLIP
[Table] [Table] [Table] [Table]
[Table]
[Table]
BLIP
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table]
within
[Table]
BLIP
[Table] [Table]
[Table] [Table]
001 No
[Table]
CLIP 0 .
[Table] [Table] [Table] [Table]
only
[Table]
and
[Table]
CLIP 0 .
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
all words
CLIP 0 .
587 ± 0 .0 .
854 ± 0 .0 .
423 ± 0 .0 .
694 ± 0 .
002 Shuffle all but
[Table]
and
[Table]
CLIP 0 .
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
002 Shuffle trigrams
CLIP 0 .
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
BLIP (Li et al., 2022):
We report results for the ‘Base’ variants of BLIP.
For experiments with COCO, we use the version finetuned on COCO; and for experiments with Flickr30k, we use the version finetuned on Flickr.
For Visual Genome experiments, we use the variant that gives better results, that is the one finetuned on COCO.
We use the checkpoints released at https://github.com/salesforce/BLIP .
X-VLM (Zeng et al., 2022b): Similar to BLIP, we report results with the ‘Base’ variants of XVLM.
For experiments with COCO, we use the version finetuned on COCO; and for experiments with Flickr30k, we use the version finetuned on Flickr.
We use the checkpoints released at https://github.com/zengyan-97/X-VLM/ .
[Table] [Table]
FLAVA (
et al.,
et
[Table]
[Table]
C N EGATIVE M INING C. N EGATIVE T EXT M INING
Starting from the MSCOCO image dataset
[Table]
we use spacy to swap the position of two elements of the caption.
These elements can be either nouns, adjectives, adverbs, verb phrases and noun phrases (only if the noun phrase is composed by three tokens or more, to not overlap with noun swapping).
For each caption we thus build a set of 5 possible negative captions (note that, a negative caption might have less then 5 negative captions if there are not enough elements to swap).
If a caption has zero negative captions, we remove it from the dataset.
During training, at each epoch for each caption we sample one of its negative captions as additional element of the batch.
C.
[Table]
M INING
Starting from the MSCOCO image dataset, we compute the pairwise similarity between all the images.
Then, for each image we collect the 3 most similar images.
During training, at each epoch for each image we sample one of its negative images as additional element of the batch.
C. F INE - TUNING D ETAILS
For finetuning models, we build our code on https://github.com/mlfoundations/ open_clip/ .
We finetune all models on the training split of the COCO dataset, and validate on the validation split.
We finetune both CLIP-FT and NegCLIP for 5 epochs, with sweeping learning rates in { 1 e − 5 , 5 e − 6 , 1 e − 6 } and picking the models based on the retrieval performance in the
COCO validation set.
We use 50 steps of warmup and AdamW optimizer with a cosine-annealing learning rate schedule with N = 1024 batch size using a single NVIDIA RTX 2080 Ti GPU.
Limitations: We note that we were not able to train an entire model from scratch due to computational resources.
We expect future work further to verify the benefits of composition-aware negative mining in contrastive pretraining.
Furthermore, note that Radford et al.
(2021) use N = 32 , 000 as the batch size, while we only used a single gpu with N = 1024 .
Similarly, we expect to gain further improvements from larger batch sizes.