Can CLIP Count Stars? An Empirical Study on Quantity Bias in CLIP (2024)

Abstract: CLIP has demonstrated great versatility in adapting to various downstream tasks, such as image editing and generation, visual question answering, and video understanding. However, CLIP-based applications often suffer from misunderstandings regarding user intent, leading to discrepancies between the required number of objects and the actual outputs in image generation tasks. In this work, we empirically investigate the quantity bias in CLIP. By carefully designing different experimental settings and datasets, we comprehensively evaluate CLIP’s understanding of quantity from text, image, and cross-modal perspectives. Our experimental results reveal a quantity bias in CLIP embeddings, impacting the reliability of downstream tasks.

## introduction the contrastive language-image pre-training (clip) model @xcite , trained on large-scale image-text pairs, has shown significant success in various downstream visionlanguage tasks, including editing @xcite , generation @xcite , and quality evaluation @xcite . it is crucial to maintain a reliable clip model at the core to ensure the development of trustworthy applications built upon it @xcite . however, several factors potentially hinder the interpretability and trustworthiness of clip, including the black-box nature of the learning process, uneven distributions of the training data, and the difficulty in accurately learning specific data distributions. such issues may lead to unintended systematic errors like spurious correlations @xcite and subgroup biases @xcite . these drawbacks not only degrade clip's performance in learning reliable latent representations for image and text translation, but also pose a risk of propagating unexpected biases to models that utilize clip for downstream tasks, thereby resulting in more challenging bugs to fix @xcite . for instance in fig. 1 , when using stable-diffusion (sd) @xcite , which leverages the relationship between image and text learned from clip @xcite , to generate an image of five pandas, most of the output consistently depicts seven pandas. we query the reason from the foundational clip model rather than the sd model at the top layer, as debugging becomes significantly more difficult and challenging if the foundational models are already biased. this intriguing phenomenon raises an important question: can clip count stars? to address this question, our work investigates the quantity bias in clip. specifically, we empirically evaluate clip at two levels: uni-modal (text & image), and cross-modal interactions. for each level, we set up tasks of varying difficulty to ensure a comprehensive evaluation. our findings highlight the need for addressing these biases to enhance the reliability and effectiveness of clip in real-world applications. ## related work there have been many efforts working on the model bias. kotek et al. (2023) investigate the behavior of large language models on gender bias. liu et al. (2022) measure the political bias in language models and propose a reinforcement learning-based method to mitigate the bias. zhang et al. (2024a) identify the existence of subgroup bias in image classifiers and use a supervised decomposition method to discover unknown bias from the joint information from the model and inputs. hosseini et al. (2018) find the shape bias learning by convolutional neural networks. khayatkhoei and elgammal (2022) discover generative models can easily learn the spatial bias from the data. heinert et al. (2024) and hönig et al. (2024) research on the texture bias of deep learning models and downstream tasks. different from previous task-specific and application-driven studies on model bias, we study the bias of the embedding in clip. there are mainly two reasons motivating us: first, various studies have identified that there is fruitful semantic information in the embedding, where the existence of bias could have a great impact on the whole model. second, as clip serves as a visionlanguage foundation model in many downstream tasks and model developments, it can help us understand the model behavior by studying the bias issue of the used foundation model. in this work, we study the existence of quantity bias from the clip embedding level for a better understanding of the failure of generative models. ## conclusion in this work, we study an interesting problem: can clip count stars? through extensive empirical studies on different modalities, we conclude that the clip models cannot understand the concept of quantity well. in the future, we will delve deeper into this quantity bias and design novel methods for efficient bias mitigation.