A Unified Framework and Dataset for Assessing Societal Bias in Vision-Language Models (2024)

Abstract: Vision-language models (VLMs) have gained widespread adoption in both industry and academia. In this study, we propose a unified framework for systematically evaluating gender, race, and age biases in VLMs with respect to professions. Our evaluation encompasses all supported inference modes of the recent VLMs, including image-to-text, text-to-text, text-to-image, and image-to-image. We create a synthetic, high-quality dataset comprising text and images that intentionally obscure gender, race, and age distinctions across various professions. The dataset includes action-based descriptions of each profession and serves as a benchmark for evaluating societal biases in vision-language models (VLMs). In our benchmarking of popular vision-language models (VLMs), we observe that different input-output modalities result in distinct bias magnitudes and directions. We hope our work will help guide future progress in improving VLMs to learn socially unbiased representations. We will release our data and code.

## introduction numerous studies have underscored the existence of social biases within large models. these biases frequently emerge as artifacts resulting from the models' pretraining on vast web-scale corpora, which predominantly consist of unmoderated usergenerated content @xcite @xcite . this paper focuses on assessing gender, race and age bias within widely adopted large-scale vision and language models (vlms) like llava @xcite , vipllava (cai † equal contribution ‡ work done when the author was an intern at msr et al., 2024), gpt4v @xcite , geminipro vision @xcite , codi @xcite , @xcite , dall-e-2, dall-e-3 @xcite , stable diffusion xl (sdxl) @xcite and others @xcite . these cutting-edge models, particularly codi, demonstrate remarkable versatility by seamlessly handling diverse input and output modalities. we expect a proliferation of similar models in the future. hence, conducting a comprehensive evaluation of bias across all inference dimensions becomes essential. this assessment allows us to gain deeper insights into the origins of bias, facilitating the design of more effective bias mitigation strategies. we employ three tasks for bias evaluation of vlms: question answering (qa) task (text-totext, image-to-text), image generation task (text-toimage) and image editing task (image-to-image). for each task, we utilize bias-bleached @xcite input to study respective societal bias in generated output. for example to assess gender bias in text-to-text direction, we use genderbleached input text, that uses gender neutral language and avoid adjectives that are associated with a particular gender. this is important because bias in the input can propagate to the output, impacting the overall fairness evaluation of the model. to generate gender bleached images, previous works proposed different pre-processing methods such as blurring or occluding pixels corresponding to people @xcite @xcite . however, these are unnatural forms of image that the model was not exposed to during training and may result in unintended spurious correlations, and hence are not suitable for societal bias evaluation of vlms. to overcome this limitation, we advocate an alternative approach: utilizing bias-bleached images that depict robots in lieu of human professionals. in contrast to prior approaches @xcite , our method generates realistic bias neutral images that also emphasize professional actions rather than relying solely on individual portraits. by directing attention to observable behaviors, the dataset enable the vlms to enhance their contextual understanding of presented images and help in detecting any inherent biases in model, in a given situation. in this work we focus on building a unified framework for societal bias evaluation of vlm models. the two key considerations of the framework include: (1) comprehensive evaluation of model inference: the method systematically assesses the vlm model's inference across all four input-output modalities: text-to-image, image-totext, image-to-image, and text-to-text. unlike prior approaches that only partially evaluate the model in specific dimensions, our method provides a more accurate depiction of bias within the model. (2) input bias independence: the method must guarantee that the system's output is not influenced by the bias in textual or visual input, focusing solely on the task at hand. we list our contributions below: ## action-based dataset to measure profession bias across gender, race and age in a vlm model, we use action-based descriptions of a profession instead of the appearance or other characteristics of a professional. this is because action-based descriptions provide a visual representation of the tasks and responsibilities associated with the profession, which can help gain a better understanding of the skills and knowledge required for a particular profession. an image of a professional's actions is more indicative of their profession than their appearance or other characteristics. for instance, images of doctors performing actions specific to their profession (like surgery) are more informative than images of them wearing scrubs and stethoscopes. this is because the former type of images can help understand the tasks and responsibilities associated with the profession. it is also worth noting that scrubs and stethoscopes are not unique to the medical profession, as other professions such as veterinarians and nurses also wear scrubs and use stethoscopes. therefore, images of doctors wearing scrubs and stethoscopes may not be as informative or representative of the profession as images that depict doctors performing actions specific to their profession. hence in this work we generate action based images vs portraits of professionals. to the best of our knowledge this is the first dataset of this kind. providing additional image details to generative models, improves the quality of generated images. ## vlm evaluation framework we propose to evaluate biases in vlms by prompting them with neutral inputs and checking if they demonstrate a preference towards certain racial or gender classifications. in particular, our proposed framework works in all the 4 possible directions vlms can operate i.e. image-to-text, text-to-text, text-to-image and image-to-image. on any-to-any ("omni") models such as codi @xcite , this gives us a holistic understanding of vlm capabilities and limitations. to evaluate vlm bias in a particular bias dimension (we consider gender, race and age in this work) and direction (one out of text-to-image, text-to-text, image-to-image and image-to-text), we consider a dataset of "neutral" text and image prompts. each neutral text/image in this dataset depicts an action performed by some profession e.g. "a doctor is performing an open heart surgery". given this neu-tral text/image, we prompt the model in various ways to elicit bias in the interested dimension. details on constructing such a dataset are presented in sec. 4.1. a neutral text prompt has description of a neutral human subject (we refer as "human") performing some action. a neutral image is the image corresponding to the neutral prompt but the "human" replaced with a "humanoid robot". such neutral text-image pairs ensure that the vlms cannot rely on any visual or textual queues when responding to our probes. in image-to-text and text-to-text settings, we give neutral {text, image} and {text} as inputs to each model respectively to see if model shows any preference to our bias probes. in image-to-image and text-to-image, we give neutral {text, image} and {text} as inputs to each model respectively and ask the model to generate a human performing the same task. we then use blip-2 @xcite to identify various attributes of the human in the generated image to evaluate bias similar to @xcite . ## experiments in this section, we discuss how our neutral textimage pairs can be used to evaluate biases in various aspects of vlms. the full breakdown of the models we evaluate across all dimensions is shown in figure 2 . in the figure, proprietary models are denoted by a star or a dot, while the remaining models are open source. ## discussion data contamination is an essential consideration in machine learning, especially when working with large-scale vision language models. our findings emphasize the robustness of our results against data contamination. this resilience arises from conducting experiments on a freshly generated dataset. furthermore, we underscore the straightforward process of constructing such datasets, which facilitates the creation of additional versions and an expanded corpus for future research. our gender/race/age-profession dataset generation technique and experimental framework can be readily extended to study more societal bias (in context of profession) and even intersectional biases. this extensibility allows for a more comprehensive examination of biases across multiple dimensions, contributing to a deeper understanding of societal disparities and informing equitable practices. we perform manual validation of the quality of images generated, ensuring no leakage of sensitive attribute and the images indeed represent the action. we will do a more thorough large scale evaluation of the images generated from dalle3 model as future work. note that the clarity of the images is crucial for conveying the intended action representing a profession. previously used images that were blurred or blacked-out compromised the quality of input images, whereas our robot-based images provide clear representations of actions related to a profession. while it's true that the distribution of synthetic images differs from real-world images used in model training, synthetic image datasets still hold significant value. lately, ai-generated images have been increasingly used on various websites. if ai content filters are to be deployed, understanding the bias of vision language models (vlms) towards generated images becomes essential. in light of these arguments, it's clear that while realworld image consistency is important, the benefits and applications of synthetic datasets cannot be overlooked. note that this is a preliminary study to measure bias in vlms, such that bias in input image doesn't impact the output. to the best of our knowledge, no such real-dataset exist. but building such a real-dataset is part of a larger exploration and is a potential future work. our approach, while limited by budget constraints, offers a promising direction for future work to explore more diverse subject variations (like stick image, slime) to evaluate bias in vlms. ## conclusion to the best of our knowledge we are the first to examine gender/race/age-profession bias across all dimensions of vlms in a comprehensive manner. our key contributions include a unified approach to systematically analyze bias in various dimensions, ensuring a holistic understanding of genderrelated biases. our curated dataset facilitates unbiased measurement of bias across all possible vlm dimensions. it employs action-based profession descriptions, closely resembling real-world perceptions. using our defined metric, we demonstrate that several vlms exhibit different amounts of gender, race and age bias across all dimensions. fine-grained analysis of gender-profession-wise bias reveals discrepancies between perceived and actual gender bias, emphasizing the need for nuanced evaluation.