When are Lemons Purple? The Concept Association Bias of Vision-Language Models (2023)

Abstract: Large-scale vision-language models such as CLIP have shown impressive performance on zero-shot image classification and image-to-text retrieval. However, such performance does not realize in tasks that require a finer-grained correspondence between vision and language, such as Visual Question Answering (VQA). We investigate why this is the case, and report an interesting phenomenon of vision-language models, which we call the Concept Association Bias (CAB), as a potential cause of the difficulty of applying these models to VQA and similar tasks. We find that models with CAB tend to treat input as a bag of concepts and attempt to fill in the other missing concept crossmodally, leading to an unexpected zero-shot prediction. We demonstrate CAB by showing that CLIP’s zero-shot classification performance greatly suffers when there is a strong concept association between an object (e.g. eggplant) and an attribute (e.g. color purple). We also show that the strength of CAB predicts the performance on VQA. We observe that CAB is prevalent in vision-language models trained with contrastive losses, even when autoregressive losses are jointly employed. However, a model that solely relies on autoregressive loss seems to exhibit minimal or no signs of CAB.

## introduction recent large-scale vision-language models such as clip @xcite and align @xcite have shown remarkable performance on zero-shot classification and text-image retrieval tasks. these models are trained via cross-modal contrastive learning on web-scale image-text pairs and obtain powerful multimodal representations. encouraged by these strong zero-shot capabilities, several recent papers explored clip for more complicated vision-language tasks. the initial attempt made by @xcite reports near chance ac-figure 1 : when we ask clip the color of the lemon in the above image, clip answers "purple". the text prompt we use is "in this picture, the color of the lemon is [mask]", where clip picks one from @xcite . however, the zero-shot performance of clip on vqa-v2 is still not state-of-the-art @xcite . in this paper, we report a phenomenon, which we call concept association bias (cab), as one of the reasons why clip struggles with vqa. to describe this phenomenon, we present a simple image containing a "lemon" and an "eggplant" to clip, and ask what color the lemon is, as shown in figure 1 . surprisingly, clip predicts "purple" with high confidence. when we instead ask for the color of the eggplant, clip answers "yellow". to cross-check this phenomenon, we formulate a binary zero-shot image classification task on the same image where the two labels are "yellow lemon" and "purple lemon", and find that clip predicts "purple lemon" with high confidence. we hypothesize that this phenomenon comes from the discrepancy between what is described in the image and text input, where clip attempts to fill in the missing concept. the association between "purple" and "eggplant" is strong, so when asked to fill in the mask in "[mask] lemon", predicting "purple" instead of "yellow" makes more sense for clip, because the text description of "purple lemon" is aligned with the image that contains both a lemon and an eggplant more faithfully than "yellow lemon", which only describes the lemon in the image. in fact, when we randomize the color of the lemon and eggplant (e.g. "red" for the lemon and "green" for the eggplant), this bias disappears, and clip picks the color almost randomly between the two. we also find that cab exists for more general object-attribute relationship such as the part-whole relationship (e.g. "humans" tend to have "clothes" on, and "trees" tend to have "leaves".) does cab exist in other vision and language models as well? to answer this question, we also test blip @xcite , blip-2 @xcite , and ofa @xcite . we find that cab exists in both blip and blip-2, but not in ofa, which is trained solely with autoregressive loss. finally, we demonstrate that enabling deeper interaction between modalities in clip can mitigate cab. in particular, we show that extending clip with an additional transformer layer on top and fine-tuning it on vqa is particularly helpful. across such variants of clip, we report that the lower the degree of cab, the higher a model performs on visual question answering. however, we also find that this fine-tuning method may not be a comprehensive solution for the more general binding problem @xcite , such as accurately connecting attribute and object representations, which leaves room for further research. ## the concept association bias the zero-shot image classification of clip is remarkable for images that contain a single concept. however, when there are multiple concepts in the image but the text input does not cover all of them, the zero-shot classification of clip can be significantly biased towards the missing concept(s). we call this bias the concept association bias (cab). we first showcase this bias using color recognition tasks. 1 for this analysis, we use the natural-color @xcite , modified for our color recognition tasks so that each image contains two different objects. dataset (ncd) @xcite , which is a dataset of vegetables and fruits with a white background. we take the following objects: banana, brinjal, broccoli, carrot, cherry, corn, cucumber, lemon, orange, plum, pomegranate, strawberry, tomato. we then randomly sample two images with different vegetable types and place the two objects side-by-side, resulting in 494 images in total. examples are shown in figure 2 . for zero-shot transfer from clip to our color recognition task, we ask for the color of one of the objects in the image. the labels we use are "red", "yellow", "purple", "green", and "orange", so it is a 5-way color recognition task. when there is a single object in the image, we use the following text prompt: "in this picture, the color of the object is @xcite ." when there are two objects in the image, we specify one of these objects in the prompt. for example, if there is a lemon and another object in the image, the prompt takes the following format: "in this picture, the color of the lemon is @xcite ." the results are shown in figure 3 . we first note that the zero-shot performance of clip on our color recognition task is almost perfect when there is a single object per image ("single object" in figure 3 ). however, the classification performance degrades to below chance when there are two objects per image ("two objects" in figure 3 ). how does this happen? we suggest that clip does not have a mechanism that stores objectcentric representation that correctly binds the object's name and its attribute. in another words, clip processes its input as a "bag of concepts". to inspect this possibility, we look at what kind of mistakes clip makes when there are two objects a and b. we find that many mistakes are derived from a common source. that is, when asked for the color of object a, clip often predicts the color of object b in the image. in fact, when we measure the accuracy of predicting the color of the object b cab gap figure 3 : zero-shot performance of clip on color recognition tasks using ncd @xcite . clip achieves almost perfect accuracy when there is a single object in the image, but the accuracy significantly drops with two objects. "two object*" refer to the case in which we instead measure the accuracy of predicting the color of the object b when it is asked for the color of the object a, where we see 80% zero-shot accuracy. we claim this gap between two objects and two objects* is a result of the concept association bias (cab). when in reality it is asked to predict the color of the object a, we see that the zero-shot transfer performance of clip is much higher ("two objects*" in figure 3 ), approaching the single object accuracy. to understand this phenomenon, we find it helpful to consider two variables per object, where each variable represents the object's name in the image and the color attribute of the object, as shown in figure 4 . when the colors are natural (figure 4 (a)), both the object "lemon" and its attribute "yellow" in the image are fully explained by the word "lemon" in the text prompt, resulting in the concept of the eggplant remaining. when clip performs zero-shot color recognition, we see that placing the color "purple" in the prompt can most faithfully explain the remaining concept of the eggplant in the image (figure 4 (b) ). the above explanation suggests that there is a strong association between the color "purple" and the object "eggplant" in clip to the point where "purple" can partially explain the concept of the eggplant. what if we break this strong association? does the gap between two objects and two objects* disappear? to test this, we generate images of fruit and vegetable in unnatural color using stable diffusion 2.0 @xcite with a prompt format '[color name] [fruit/vegetable name]', and filter bad images by ourselves. examples are shown in figure 5 . we call this dataset unnatural-color color), respectively. we suggest that the text prompt and the image are represented as two separate "bags of concepts" in clip. when a pair of object-attribute concepts are naturally associated with each other, both concepts can be accounted for by including in the prompt either of the object or the attribute. when only some of the concepts in the image are included in the text, this leaves other concepts in the image unaccounted for. dataset (uncd). we repeat the same experiment on uncd. the results are shown in figure 6 . we see that the zero-shot performance for a single object is still high, suggesting that clip can pick up the color attribute even if the color is not strongly associated with the object itself. however, for the two object cases, we see that there is almost no difference between two objects and two objects* tasks. in other words, clip predicts the two nonassociated colors in the image with almost equal chance. we also create a version of ncd, which we call uncd-v2, where we artificially change the color of each fruit and vegetable of ncd to non-associated color. as shown in appendix, we see a similar pattern of cab as uncd. why does the cab gap disappear when objects are paired with random attributes in images? this result arises from a common mechanism that impacts both the two objects and two objects* tasks. to see this, we go back to our diagram in figure 4 (c). when the colors are unnatural (e.g., a lemon in red color and an eggplant in green color), then the remaining bag of concepts that are yet to be explained by the text include "red", "green", and "eggplant". this is because the color "red" is not associated with the concept of "lemon", and therefore the word "lemon" in the text prompt cannot explain the color "red", unlike the case that uses natural color. as a result, clip can choose either "red" or "green" for color recognition. and indeed, surprisingly, clip randomly chooses between the two -it does not associate the concept of "red" with the lemon, even though in the image the lemon unambiguously appears in red. likewise, for the two objects* task (in which the correct prediction is defined as the color of object b when asked for object a), clip essentially randomly picks one of the two colors present in the image, despite the fact that each object has their own very distinct color. ## cab widely exists in vision-language models in section 3, we demonstrate cab using clip. in this section, we extend our experiments to other vision-language models, including blip @xcite , blip-2 @xcite , and ofa @xcite . blip and blip-2 are both multi-modal, multitask model that has three heads and pre-training objectives: the clip-like image-text contrastive loss, the binary image-text matching loss that classifies whether the text corresponds to the image, and the language modeling loss, which autoregressively generates the caption for an image. for blip-2, there is the second stage of pre-training for visual conditioned language generation, bootstrapped from pre-trained llm. in our experiment, we treat these models with different heads separately and abbreviate them as "contrast", "match", and "caption". ofa unifies various vision and language tasks via a simple sequence-to-sequence framework and employs autoregressive loss for its objective. for comparison across models, we define the cab score as: in the networks trained with contrastive or matching losses (i.e. clip, blip-contrast/match, blip-2-contrast/match) have stronger cab than those trained with autoregressive loss (i.e. blip-caption, blip-2-caption, and ofa). moreover, in comparison with blip, blip-2 uses a large language model as the final decoder on top of its sub-networks, making them more associated with the autoregressive loss and having less cab scores than their counterparts in blip. furthermore, we observe that matching losses have lower cab than contrastive losses for both blip and blip-2. although the two losses are similar, the matching task uses cross-attention that jointly processes texts and images, whereas the contrastive task only uses unimodal self-attention. based on this observation, we hypothesize that the deeper cross-modal interaction helps mitigate the cab. we further investigate this in section 6. it is worth noting that the amount of cab in autoregressive models is substantial. for example, two objects*'s accuracy for blip-caption, blip-2caption, and blip-2-flant5 are 0.471, 0.483, and 0.377 respectively. this means that when joitnly trained with contrastive losses, even autoregressive models are biased by concept association, resulting in two objects*'s accuracy being much higher than random chance. the only model with minimal or almost no cab is ofa, which sorely relies on autoregressive loss. in the size varies greatly, the cab score stays around the same level for these different clip models. in fine-tuning helps reduce cab in the last section, we show that cab can be seen in visionlanguage models, and it is especially prominent for purely contrastively trained models. in this section, we test our hypothesis from section 5 that a deeper modality interaction helps mitigate cab in the idea of using deep modality interaction on top of image and text embeddings has been explored before in @xcite . however, in @xcite , the image and text encoders are shallow unlike clip. in @xcite , instead of using clip as is, they employ the architecture that uses the image encoder of clip, the bert text embeddings, and a transformer as an additional modality interaction module. they apply this model for vision and language tasks such as visual question answering, visual entailment, and v&l navigation tasks. the goal of @xcite was to demonstrate that the image encoder of clip is more helpful than imagenetpretrained image encoders. for our architecture, we use both image and text encoders from clip, and also a transformer for modality interaction on top of clip, as shown in figure 11 . we conducted experiments in two settings: 1. freezing clip and only fine-tuning the transformer head, and 2. fine-tuning both clip and the transformer head. following @xcite , we use vqa-v2 to fine-tune these model variants. we follow the standard pre-processing of vqa-v2, where we filter less common answers and select vision encoder text encoder modality interaction image text image text transformer text encoder vision encoder clip clip+transformer head ncd two objects two objects* cab score vqa-v2 clip(frozen) 0.352 0.094 0.371 0.542 clip(finetuned) 0.328 0.168 0.420 0.390 fine-tuning alone may not necessarily solve the binding problem the last section demonstrates that by introducing deeper modality interaction and fine-tuning, we can mitigate cab on ncd. can 6 , respectively. in fact, we find that these fine-tuned models also have lower accuracy for two objects* ( ## conclusion every object has a set of concepts that are roughly associated with it. for instance, the object "lemon" can be associated with "yellow", "fruit", and so on. such concept association is automatically learned in vision-language models, to the point where the word "yellow" can partially explain the object "lemon" in certain cases. we establish that the concept association bias (cab) exists for visionlanguage models through a series of experiments, and find that the models trained with contrastive loss are especially affected. furthermore, we verify that the lower the degree of cab is, the higher the performance of vqa-v2 is. contrastive models like clip is increasingly popular in both computer vision and natural language processing. we hope our work raises awareness of the brittleness of contrastive objectives as we develop new vision and language models.