Examining Gender and Racial Bias in Large Vision–Language Models Using a Novel Dataset of Parallel Images (2024)

Abstract: Following on recent advances in large language models (LLMs) and subsequent chat models, a new wave of large vision–language models (LVLMs) has emerged. Such models can incorporate images as input in addition to text, and perform tasks such as visual question answering, image captioning, story generation, etc. Here, we examine potential gender and racial biases in such systems, based on the perceived characteristics of the people in the input images. To accomplish this, we present a new dataset PAIRS (PArallel Images for eveRyday Scenarios). The PAIRS dataset contains sets of AI-generated images of people, such that the images are highly similar in terms of background and visual content, but differ along the dimensions of gender (man, woman) and race (Black, white). By querying the LVLMs with such images, we observe significant differences in the responses according to the perceived gender or race of the person depicted.

## introduction when openai announced gpt-4, one of the most intriguing claims was that the model would be multimodal: accepting both image and text as input. 1 while this functionality has not yet been released to the public at the time of writing, several independent research groups have since trained instructiontuned conversational large vision-language models (lvlms) using open-source resources @xcite @xcite . extending the capabilities of ai chatbots to describe, discuss, and analyze images offers an exciting array of new use cases. however, it is also important to understand how such systems may perpetuate harmful social stereotypes when presented with ambiguous images and/or text prompts. humans instantly (and often, subconsciously) make judgments about other people based on their appearance, mentally categorizing them into particular social groups based on perceived characteristics of gender, race, age, and so on @xcite . when we then make assumptions about an individual based on their perceived membership in a particular social group, this is known as stereotyping. here, we are interested in the question of whether lvlms make similar assumptions based on the visual information present in input images. one example of gender stereotyping that has been widely-reported is role incredulity, in which women, particularly in the workplace, are assumed to be in a stereotypically-female subordinate or care-based role, as opposed to a stereotypicallymale leadership role @xcite . consider the example of when a female doctor is assumed to be a nurse, or a woman in a boardroom is assumed to be a secretary, rather than a ceo. in this paper, we explore the question of whether lvlms also show gender-related role incredulity bias when presented with images of people of different genders foot_2 in workplace settings. we also explore the stereotypical association between race and socioeconomic status that is seen in western culture @xcite . biased associations with poverty and criminality can lead to black and white people being treated very differently in the same situation. black runners have described the experience of "running while black," such that they must take special precautions not to be mistaken for a criminal running from police @xcite . black academics have reported being targeted by security at conferences or on university campuses @xcite . similarly to the gender-based discrimination described above, such cases involve stereotypical assumptions about a person's likely role in an environment, given their perceived membership in a demographic group. in this paper, we take a first step towards examining the presence of such biases in lvlms by presenting four different models with images of black and white men and women, and asking questions to probe the models' underlying assumptions about the people depicted in the images. crucially, for this approach to work, all other visual information in the image must be controlled. we employ a novel methodology of generating images using the text-to-image model midjourney, such that the dimensions of interest (gender and race) are variable, while the background scenario (e.g., a hospital, boardroom, or university) is fixed. we can then measure any differences in the text output with respect to the demographic features of the subject of the image. our main contributions then are as follows: ## related work similar to both text-only and image-only datasets, multimodal (text-image) datasets collected from the web (such as image-caption pairs) contain social biases, particularly in the representations of minority and marginalized groups. for example, the popular microsoft coco dataset @xcite has been shown to have unbalanced representations of male and female subjects in certain contexts (e.g., a person cooking) as well as biases in the manual annotations, reflecting the underlying stereotypical beliefs of the human annotators @xcite . as a result, systems trained on such data tend to rely on context (e.g., a person cooking must be a woman) rather than the person's appear-ance perpetuating and amplifying bias @xcite . @xcite examined another popular dataset, laion-400m @xcite , containing image-alt-text pairs parsed from the commoncrawl web data. they found that it contains pornography, malignant stereotypes, racist and ethnic slurs, and other problematic content. several studies investigated the presence of bias and various techniques for its mitigation in general vision-language representations like clip @xcite @xcite , as well as in downstream applications such as image captioning @xcite , image retrieval @xcite , visual question answering @xcite , and text-to-image generation @xcite @xcite @xcite . multiple text-image datasets for bias evaluation were created @xcite @xcite . typically, such datasets comprise images scraped from the web representing members of specific social groups and textual descriptions corresponding to stereotypical or anti-stereotypical associations. we continue this line of work and aim to evaluate bias in the emerging technology of large visionlanguage models. our work is most similar in spirit to that of @xcite , in which the authors used a measure of visual similarity to identify pairs of images that depicted similar scenarios, but containing subjects with lighter or darker skin tones. they then analyzed any differences in the captions that were produced for these image pairs, under the assumption that an unbiased system should produce very similar captions for each image in the pair. our work extends this idea in two important ways: (1) we use the newly-available image generation tools to generate images that are extremely similar, except for the demographic characteristic in question, therefore addressing one of the previous limitations that the difference in captions could be due to actual differences in the images. (2) using lvlms, we are able to extend the task beyond image captioning, to a number of different generation tasks. ## parallel image dataset to uncover subtle biases in lvlms, we present a novel dataset of images portraying people in every-day situations (e.g., cooking, studying, working). we call this dataset the pairs dataset (parallel images for everyday scenarios). for each scenario, we generate four parallel images: where the subject was dark-skinned woman, dark-skinned man, light-skinned woman, or light-skinned man. the scenarios are somewhat ambiguous, in that the subjects' occupation, social status, or intentions can be interpreted in different ways. the critical characteristic of this dataset is that the four images for each scenario are very similar, except the demographic characteristics of the portrayed person. here, we draw a comparison with the framework proposed by @xcite for handling underspecification in text-to-image models, where two approaches are possible: ambiguity-in, ambiguity-out, and ambiguity-in, diversity-out. in our (text+image)-to-text setting, we suggest that there are similarly two appropriate responses from the system. (1) ambiguity-in, ambiguity-out: if we prompt the system with a question (e.g., is this person high-status or low-status?) and there is insufficient information in the image to make such a determination, the system should answer to that effect. (2) ambiguity-in, consistency-out: if we prompt the system with the same question, but there is some cue in the image which increases the probability of one answer over the other (e.g., the presence of a business suit or expensive jewellery), then the system should provide the same distribution of output labels for all four images, regardless of gender or skin colour. the scenarios are grouped in three categories. the first group comprises occupation scenarios that are (a) visually ambiguous (e.g., an image of a person wearing scrubs in a hospital could realistically be either a doctor or a nurse), but (b) can be stereotypically associated with either male or female gender (e.g., a male wearing scrubs is more likely to be identified as a doctor, and a female wearing scrubs is more likely to be identified as a nurse). we use data from the u.s. bureau of labor statistics from 2022 to generate such pairs of male-and female-dominated occupations (see table a .1 in the appendix for the complete list). we generate images for 20 ambiguous occupation pairs; examples are shown in figure 1 . the second group contains images portraying neutral scenarios of day-to-day life (cooking, riding the bus, pushing a baby stroller, etc.). these situations can occur in everybody's life and should not form the basis for determining the person's social status. there are 20 scenarios in this group. the third group is inspired by the distressing pattern reported by black americans of being mistaken for criminals while undertaking normal daily activities. it comprises potentially crime-related scenarios where the subject's actions or intentions can be interpreted as either criminal or sociallyacceptable activities (e.g., a person in a ski-mask can be a skier or a robber, a person opening a window can be a home-owner or a burglar). there are 10 scenarios in this group. in total the dataset includes 50 scenarios (200 images). the images were created using midjourney (versions 4 and 5) between may and august 2023. 3 the basic methodology involved prompting for an image of a person in a particular scenario, e.g. a photo portrait of a person cooking dinner. when an acceptable image was produced, it was then varied using midjourney's "variation" command (in v5, "subtle variation") to generate visually similar images but with a different combination of gender and skin tone, e.g. a photo portrait of a black woman cooking dinner. this process was repeated in an iterative manner until four images were obtained for each scenario, covering the space of {black man, black woman, white man, white woman}. we then performed a manual verification to ensure that all parallel images were highly similar, except for the variables (gender/race) in question. minor differences in the background and details of the image were unavoidable in the generation process, and were considered acceptable if judged that they were irrelevant to the interpretation of the image by a human observer. while we do acknowledge the possibility that these subtle differences may have unanticipated effects on an lvlm's judgement about the image, we also contend that if the bias in the model's decision is increased by the presence of small perturbations in the input, then this is also a problematic result. all images are available for download at https: //github.com/katiefraser/pairs . in the first row, we ask whether the person is a doctor or a nurse; in the second row, we ask whether the person is a pilot or a flight attendant; and in the third row we ask whether the person is an architect or an event planner. ## results we now present the results of the three experiments outlined above. we first review our expectations for what an unbiased system should output in response to our ambiguous inputs: (1) ambiguity-in, ambiguity-out: either refuses to choose between the two labels offered (association @xmath0, or randomly assigns a label (average association @xmath1; or (2) ambiguity-in, consistency-out: makes a decision based on cues in the image other than race or gender (non-zero but equal association scores for all demographic groups). we begin by examining the number of times the system refuses to answer, and then present the overall scores for each experiment. ## discussion we have evaluated four publicly available state-ofthe-art lvlms and found that, although in many cases the output is not problematic, all of the models exhibit some degree of gender and racial bias in certain situations. our hypothesis for what an unbiased output should be for the first three binary-choice question experiments was straightforward: decisions about whether a person is a doctor versus a nurse, high-status versus low-status, or criminal versus innocent bystander should not be made on the basis of perceived gender or skin colour. therefore, for highly-similar inputs, we expect highly-similar outputs (or, a refusal to answer). although we observed significant differences based on gender for the occupation dataset, and based on race for the status dataset, we were encouraged by the lack of bias observed in the criminality dataset. however, our subsequent lexical analysis did uncover harmful trends for black people, and specifically men, to be associated with crime, violence, gangs, and guns. this finding aligns with previous work showing that the intersection between race and gender-based social categories results in complex and meaningful differences in how the groups are perceived @xcite . in addition to this clearly undesirable output, however, we also noticed more subtle differences in the words that were produced in response to images of black subjects: explicit markers of race, words that have come to act as euphemisms for race, and references to "low-income familes" and "overcoming obstacles." in thinking about these kinds of phenomena, the hypothesis for what an ideal, unbiased output should look like becomes harder to define. in some sense, for the closed-ended questions, we are advocating a version of racial colorblindness: that race is not a relevant characteristic in the context of the decision and should be ignored. however, when we prompt instead for things like stories and emotions, it is harder to support such an assumption. do we really want generated output such that we cannot determine which of the four images it describes? in the real world, racial colourblindness has been criticized as an insufficient and naive approach to combating racism, which can in fact result in reinstating the existing social hierarchies, denying systemic racism, and ignoring manifestations of discrimination @xcite . from that perspective, perhaps it is reasonable for an lvlm to describe the challenges and discrimination faced by a black subject in an image, and not a white subject. such an approach would be more in line with the idea of multiculturalism, the view that our differences should be acknowledged and celebrated, rather than ignored. on the other hand, it is important to consider the limitations of these models and their usefulness. while it may be empowering and inspiring to hear a real person's story of resilience and survival, the usefulness of an artificially-generated story about an artificially-generated image is less obvious, and any benefit might be outweighed by the risk of the further perpetuation of social stereotypes. ultimately, the answers to these questions are likely highly dependent on the context in which the model is being used, and must be considered carefully to ensure positive social impact of these emerging technologies. ## conclusion we investigate the presence of gender-and racerelated bias in four publicly available, state-of-the art large vision-language models. for this, we created a unique dataset of parallel images depicting persons of different race and gender in identical surroundings. the pairs dataset opens new avenues for evaluating large pre-trained vision-language models for the presence and extent of gender and racial biases, as well as other research questions. in the current work, using both direct questions and open-ended prompts, we were able to reveal gender, race as well as intersectional biases in all four models. these results underline the need for improved bias mitigation strategies to ensure the safety and fairness of large multimodal models. a first step towards the development of such strategies will require a better understanding of where in the model pipeline the bias originates. it is reasonable to assume that the degree of bias in the output is affected by bias in the base llm, bias in the base vision encoder, as well as the details of the multimodal training process and datasets. the four models examined here differ in all of these respects. untangling the specific channels of bias propagation to develop a set of best practices for combining pretrained black-box components into a single lvlm will be a challenging undertaking for the field going forward.