C olor F oil: Investigating Color Blindness in Large Vision and Language Models (2025)

Abstract: With the utilization of Transformer architecture, large Vision and Language (V&L) models have shown promising performance in even zero-shot settings. Several studies, however, indicate a lack of robustness of the models when dealing with complex linguistics and visual attributes. In this work, we introduce a novel V&L benchmark - ColorFoil, by creating color-related foils to assess the models’ perception ability to detect colors like red, white, green, etc. We evaluate seven state-of-the-art V&L models including CLIP, ViLT, GroupViT, and BridgeTower, etc. in a zero-shot setting and present intriguing findings from the V&L models. The experimental evaluation indicates that ViLT and BridgeTower demonstrate much better color perception capabilities compared to CLIP and its variants and GroupViT. Moreover, CLIP-based models and GroupViT struggle to distinguish colors that are visually distinct to humans with normal color perception ability.

## introduction vision and language models (v&l) have exhibited improved performance for many v&l tasks in recent years @xcite @xcite @xcite . thus, the current paradigm has now been shifting towards zero-shot learning, where models are evaluated without fine-tuning for specific tasks @xcite . large-scale v&l models, in particular, show promise for taskindependent zero-shot evaluation @xcite . several studies have been conducted to perform comprehensive evaluations of v&l models on a variety of tasks to identify their strengths and weaknesses @xcite @xcite @xcite . for instance, the valse evaluation benchmark has been proposed to assess the state-of-art v&l models for challenging linguistic constructs @xcite . therefore, five distinct tasks, including existence, plurality, counting, relations, actions, and coreference, have been introduced. in this benchmark, foils are generated from the existing v&l datasets for each of the tasks. a foil is referred to as a distractor or slightly incorrect example that is passed along with the correct example to the v&l model to assess the model's ability to correctly distinguish them @xcite . although the existing v&l benchmarks like valse help the community to test the capabilities of v&l models, there is still much work to be done to evaluate the robustness and generalizability of the models on numerous other tasks. it remains unknown how well the large v&l models can perceive colors from the visual content. color perception requires a human-like understanding of visual content. thus, by evaluating the v&l models on color attributes, we can determine how closely the large v&l models perceive colors to humans. a v&l model can be biased towards detecting particular colors and perform poorly with others. therefore, it is essential to investigate it in order to improve the explainability and interpretability of the models. by assessing the v&l models with their color-perception ability, we can ensure robustness in real-life applications. in this study, we aim to shed light on the following research question: how well can the stateof-the-art large-scale v&l models perceive colorrelated attributes, such as red, green, yellow, etc.? our contributions are mainly twofold: the outline of this paper is as follows. we provide a background study in section 2. in section 3, we describe the process of constructing colorfoil from the ms coco dataset. experiments and results are discussed in section 4. in section 5, we discuss the limitations of our work. ethical considerations are provided in 5. a conclusion and future scope is presented in section 5. clip contrastive language-image pre-training (clip) is a v&l model that is pre-trained with 400m image-text pairs with a contrastive objective @xcite . the model jointly trains a text encoder and an image encoder to maximize the cosine similarity of the image-text embeddings of real pairing while minimizing the cosine similarity of the embeddings of the incorrect pairings within a multi-modal embedding space. each of the encoders are based on transformers. clip demonstrates the ability to perform zero-shot visual classification, object detection, and image generation tasks. vilt vision-and-language transformer (vilt) is pre-trained using a transformer with more than 4m images with two objectives such as image text matching and masked language modeling @xcite . the text embedding and the image features are concatenated into a sequence and then fed into the transformer. thus, vilt is a single stream model. vilt achieves competitive or better performance than other v&l models on downstream tasks while being 10 times faster due to simpler processing of visual inputs. bridgetower there is a visual encoder, a textual encoder and a cross-modal encoder with multiple lightweight bridge layers in the bridgetower architecture @xcite . the top layers of the unimodal encoders and each layer of the crossmodal encoder are connected with the bridge layers, thus enabling extensive interactions at each layer of the cross-modal encoder. each of visual, textual and cross-modal encoders is transformer-based encoders. the model is pre-trained with 4m images with two common objectives: masked language modeling and image text matching. the model is found to outperform in all downstream v&l tasks with negligible additional computational cost. vit a vision transformer (vit) is designed for image classification tasks, adapting the transformer architecture from natural language processing @xcite . it divides an image into fixed-size patches, linearly embeds each patch, and treats these embeddings as sequences akin to word tokens in text. using self-attention mechanisms, the vit captures global image context more effectively than convolutional networks, allowing for superior performance on large-scale image datasets. vits leverage transfer learning and pretraining for enhanced accuracy and efficiency. groupvit (group vision transformer) is a variant of the vision transformer designed to improve efficiency and scalability in image classification tasks @xcite . it enhances the standard vit by introducing a group-wise processing mechanism, where the input image is divided into smaller groups of patches. each group is processed independently through parallel self-attention layers, reducing computational complexity. the results from these groups are then aggregated to form a cohesive representation. groupvit aims to retain the global context modeling capabilities of vits while optimizing resource usage, making it more suitable for large-scale and real-time applications. related work several v&l tasks include visual question answering @xcite , visual reasoning @xcite , image retrieval @xcite , etc. foiling is an approach that slightly edits the original captions to evaluate the robustness of the v&l models @xcite . similar to our work, @xcite foiled the ms coco dataset, and constructed the foil-coco dataset. however, their work did not focus on the perception of colors of the v&l models. following the work of @xcite , several studies have been performed that evaluated the v&l models @xcite @xcite @xcite . ## construction of the colorfoil benchmark the colorfoil benchmark is automatically derived from the ms coco (microsoft common objects in context) and flickr30k dataset, which serves as a resource for studying image understanding, object recognition, image captioning, and visual question-answering tasks @xcite . in the ms coco dataset, textual annotations are provided solely for the train and validation (val) sets. to construct the colorfoil, we obtain the images and annotations from the 2017 ms coco validation set, resulting in a total of 5,000 image-text pairs. among these instances, each of 2,511 pairs includes at least one word related to color. for flickr30k dataset, we use the standard val and test sets to prepare the colorfoil benchmark. our aim is to foil only the color name from the textual input, leaving the original image and the rest of the text input as it is. for example, given a caption like a blue bus driving down a street past a park. we foil the color-related word, resulting in a modified sentence like -a brown bus driving down a street past a park. if there are multiple color attributes in a caption, we foil all of them. we utilize the webcolors 1.3 python package to determine whether a substring within a caption corresponds to a color @xcite . this package encompasses a total of 147 colors. our filtering process involves excluding captions that lack color names and selecting solely those containing at least one color name. when replacing the original color name with a foiled alternative, we consider the most widely used colors. the chosen target colors for foiling consist of "blue", "black", "red", "pink", "yellow", "grey", "orange", "white", "green", and "brown." so, rather than utilizing the complete list of 147 colors from the webcolors package, we opt for a narrower selection of common colors for foiling. this decision is based on the fact that numerous colors in the package have limited practical usage (e.g. medium blue, mint cream, etc.). the target color for foiling is selected randomly from the 10 common colors. if the original color in the caption is one of the common colors, we randomly select any other common color for foiling except for the one found in the caption. after excluding four instances of twodimensional grayscale images due to compatibility issues with certain models, our resulting dataset comprises 2,507 pairs of rgb images along with their captions and foils from mscoco and 2500 pairs of rgb image-caption pairs from flickr30k. to ensure data integrity, we conduct manual validation on a significant number of image-text pairs randomly selected from the benchmark and find no anomalies. examples of original captions and corresponding foils are illustrated in figure 1 . ## experiments experimental setup: we pass the original caption, foil as well as the corresponding image to a v&l model. the model provides the logits for each of the caption and foil corresponding to the image. we take the softmax of the logits. our hypothesis is that a model with a well-perceivable ability to distinguish colors is supposed to provide a higher probability for the original caption and a lower probability for the foil. we evaluate all the models in a zero-shot setting. we utilize the huggingface transformer library to load the models @xcite . these models are chosen due to the fact that they represent different architectural variants. clip has a text encoder and an image encoder, which are jointly trained with a contrastive loss. vilt is a single-stream model. bridgetower contains multiple bridge layers that connect the uni-modal encoders with the cross-modal encoder. the evaluation metric employed in our study is accuracy and f1-score, which are widely used in similar contexts. to elaborate, if the model accurately identifies the foil in comparison to the original caption, the accuracy of that particular example is incremented. results: the relatively poor performance of clip is also evident in its variants, including altclip @xcite and @xcite . while the align model outperforms clip, it still lags behind bridgetower and vilt. groupvit, similar to clip, struggles to achieve high performance. this performance trend is consistent across both mscoco and flickr30k datasets, reinforcing our observations. when presented with more foils alongside the original caption, the models exhibit performance degradation. nonetheless, bridgetower and vilt maintain strong performance even under these challenging conditions with more foils. we present several examples for which the clip model incorrectly assigns higher probabilities to the foils (see figure 2 ). these examples demonstrate that the clip model is unable to distinguish between blue-brown, black-red, and redwhite pairs, despite the fact that they are visually distinct to most humans. ## conclusion and future work in this work, we introduce a novel benchmark, col-orfoil, derived from the ms coco and flickr30k datasets, to assess the perception ability of the cutting-edge v&l models to detect colors. to this end, we foil the colors from the original captions and feed both caption and foil along with the corresponding image to the model to observe whether it can provide a higher probability for the caption or not. seven state-of-the-art v&l models, including clip, vilt, vit, and bridgetower, have been benchmarked using the colorfoil. while all models outperform a random classifier, vilt and bridgetower are much more capable to perceive colors compared to clip and vit. this intriguing finding is seen using both ms coco and flickr30k datasets, which strengthens our analysis. as part of our future work, we would like to evaluate the robustness of v&l models on additional tasks by constructing foils that swap gender (man -> woman), size (small -> large), emotions (smiling -> crying), and sentence negation (playing football -> not playing football), etc.