Uncovering Bias in Large Vision-Language Models at Scale with Counterfactuals (2025)

Abstract: With the advent of Large Language Models (LLMs) possessing increasingly impressive capabilities, a number of Large Vision-Language Models (LVLMs) have been proposed to augment LLMs with visual inputs. Such models condition generated text on both an input image and a text prompt, enabling a variety of use cases such as visual question answering and multimodal chat. While prior studies have examined the social biases contained in text generated by LLMs, this topic has been relatively unexplored in LVLMs. Examining social biases in LVLMs is particularly challenging due to the confounding contributions of bias induced by information contained across the text and visual modalities. To address this challenging problem, we conduct a large-scale study of text generated by different LVLMs under counterfactual changes to input images, producing over 57 million responses from popular models. Our multi-dimensional bias evaluation framework reveals that social attributes such as perceived race, gender, and physical characteristics depicted in images can significantly influence the generation of toxic content, competency-associated words, harmful stereotypes, and numerical ratings of individuals.

## introduction large vision-language models (lvlms) have gained popularity recently for their ability to extend the conversational abilities of llms to the multimodal domain. specifically, lvlms condition generation on both a text prompt and an image, enabling a user to ask questions and engage in a conversation about visual inputs. these capabilities have been popularized in recently-introduced models such as gpt-4 @xcite and @xcite . while lvlms have exhibited impressive capabilities, a critical question remains regarding the extent to which they possess harmful social biases. prior studies have extensively investigated the social biases in the context of language models and various nlp tasks @xcite @xcite @xcite . lvlms, which combine a language model with a visual encoder such as clip @xcite , have the potential to introduce additional bias beyond that encoded in the llm through the incorporation of visual inputs. here, we conceptualize bias as a tendency of the model to produce different output when prompted with inputs referencing different social attributes (e.g., race, gender) 1 such that the output leads to representational harms and the perpetuation of stereotypes @xcite . to address this important question, we evaluate english text generated by recently-proposed lvlms on both open-ended and close-form text prompts, varying only the model's visual input using synthetic counterfactual image sets that are highly similar in their depiction of people in various occupations while differing only in the person's perceived race, gender, and physical characteristics. crucially, our use of counterfactual images allows us to isolate the influence of perceived social attributes depicted in images on text generated by lvlms because other image details (e.g., image background) are held constant. we conduct the largest study to-date of bias in lvlms under this evaluation framework, using five open-source models of varying architectures and sizes plus one commercial model. our work makes the following contributions: (1) we investigate intersectional bias in lvlms along the attributes of race, gender, and physical characteristics, rather than focusing on a single attribute at a time. (2) our use of counterfactuals ensures that differing outputs can be attributed to the variables of interest and not spurious differences in image context, unlike when using real images sourced from the web. (3) while previous research relied on manually curated image datasets, which are necessarily quite small, we conduct our analysis using 171k images (45x larger than the largest dataset previously used) and produce over 57 million lvlm generations. this allows more robust bias estimation and also provides a window into the "long-tail" behavior of lvlms. (4) we conduct a multi-dimensional analysis of bias through evaluation of toxicity, competency-related words, stereotypes, and numeric ratings rather than relying only on a single measure of bias estimation. (5) we investigate the relationship between bias in lvlms and their constituent llms. (6) finally, we explore the effectiveness of inference time mitigation strategies. while lvlms do not generate harmful content most of the time, our results surprisingly show that they can generate highly offensive text at the tails of the distribution when deployed at a large scale. this characterization of bias in lvlms is unique to our work and addresses an important gap in the existing literature, particularly in light of the scale at which lvlms are being deployed today. we make our code 2 and our dataset 3 of lvlm generations publicly available to support future research. ## related work several datasets have been proposed for probing social biases in vision-language models, including visogender (690 images) @xcite ) to 2 our code is available via github 3 our dataset of generations is available via hugging face detect gender bias in multimodal coreference resolution, mmbias (3800 images) (janghorbani and de melo, 2023) to assess biases based on religion, disability, and sexual orientation, and vlstereoset (1028 images) @xcite which extended the text-only stereoset dataset into a multimodal benchmark using photographs from google. these datasets have primarily been used to measure bias in vision-language models such as clip rather than lvlms. therefore, they differ from our work in that (1) they do not specifically evaluate lvlms, which combine a vision encoder with an llm; and there has been relatively little prior research on the nature of social biases in lvlms. @xcite generated 1120 images of gender-neutral robots performing various tasks and asked five lvlms to deduce the gender based on the image context. fraser and kiritchenko (2024) introduced the pairs dataset, containing 200 synthetic images of different people in highly similar visual contexts (e.g., wearing scrubs in a hospital). they measured gender and racial biases in lvlm responses to questions like is this a doctor or a nurse? as well as open-ended generation tasks. our work differs from these prior studies as follows: (1) we evaluate intersectional bias in lvlms rather than focusing on a single attribute at a time. ## conclusion our study reveals how lvlms can generate harmful and offensive content when deployed at scale. even when generations are not explicitly offensive, our lexical analysis shows that lvlms often rely on stereotypes when producing open-ended descriptions of individuals from different social groups. while our investigation of inference-time mitigation strategies show that bias can sometimes be reduced via prompt engineering, further research is needed into robust methods for debiasing lvlms across a broad range of generation scenarios. additionally, the investigation of other types of social biases in lvlms beyond race, gender, and physical characteristics would be a promising direction for future studies.