VLS tereo S et: A Study of Stereotypical Bias in Pre-trained Vision-Language Models (2022)

Abstract: In this paper we study how to measure stereotypical bias in pre-trained vision-language models. We leverage a recently released text-only dataset, StereoSet, which covers a wide range of stereotypical bias, and extend it into a vision-language probing dataset called VLStereoSet to measure stereotypical bias in vision-language models. We analyze the differences between text and image and propose a probing task that detects bias by evaluating a model’s tendency to pick stereotypical statements as captions for anti-stereotypical images. We further define several metrics to measure both a vision-language model’s overall stereotypical bias and its intra-modal and inter-modal bias. Experiments on six representative pre-trained vision-language models demonstrate that stereotypical biases clearly exist in most of these models and across all four bias categories, with gender bias slightly more evident. Further analysis using gender bias data and two vision-language models also suggest that both intra-modal and inter-modal bias exist.

## introduction recently there has been much interest in adapting foundation models such as albert @xcite ,roberta @xcite , t5 @xcite , gpt-3 @xcite and clip @xcite for different downstream tasks. these models demonstrate powerful transfer capabilities largely because they have acquired the rich body of knowledge contained in their pre-training data. however, their pre-training data may also contain social biases and stereotypes, especially when the data are crawled from the internet without cleaning. as a result, pre-trained models may "inherit" these biases and stereotypes, affecting the fairness of systems derived from these foundation models for downstream tasks. previous work mainly focused on measuring biases and stereotypes in a single modality. for example, in nlp, people studied social biases in word embeddings (e.g., @xcite and language models (e.g., @xcite , and in computer vision, people studied social biases in unsupervised vision models (e.g., @xcite . however, there has been little work to understand social biases in multi-modal or cross-modal settings. in particular, although there has been fast progress recently in developing large-scale pre-trained vision-language models (e.g., @xcite @xcite , because these models are relatively new, little work has been done to understand biases and stereotypes in them. it is important to measure biases and stereotypes in pre-trained vision-language models because they are used for a wide range of downstream vision-language tasks, many directly involving human users, such as automatic caption generation, visual question answering and multimodal hate speech detection. in this work, we study the problem of measuring stereotypical bias in pre-trained vision-language models. we regard the problem as a probing task. since there is no suitable existing dataset with a good coverage of different biases for our purpose, we first construct a new dataset called vlstereoset, built on top of the recently released stereoset designed for stereotypical bias in language models and has a wide coverage @xcite . we note that the key to measuring stereotypical bias is to measure the degree of association between a target social group (e.g., sister) and some potentially stereotypical or anti-stereotypical attributes (e.g., caring or rude). however, unlike text where we can use words to represent the target social group and the attributes separately, it is usually not easy to disentangle a target social group from an attribute in an image (e.g., an image of a sister may inevitably reveal her facial expression and body language, which may imply whether she is caring or rude). we therefore cannot directly replicate the context association test designed by @xcite in our vision-language settings. observing this challenge, we propose a different approach. our vlstereoset consists of images showing stereotypical or anti-stereotypical scenarios. each image is accompanied by three candidate captions (taken from stereoset), where one is stereotypical, one is anti-stereotypical and the third is semantically meaningless. one of these captions is labeled as the correct caption for the image, and the probing task is to identify this correct caption given the image. in particular, to assess whether a model contains stereotypical bias, we can present an anti-stereotypical image to the model and check which caption the model would pick. an example is shown in figure 1 where the image shows an anti-stereotypical scenario, with option 2 as the correct caption. if a pre-trained vision-language model prefers option 1 (a stereotypical statement) instead, it exhibits stereotypical behavior. based on our constructed vlstereoset and following the metrics introduced by @xcite , we define three metrics, one to measure a model's capability to pick meaningful captions, another to measure a model's tendency to pick stereotypical captions, and the third combining the first two. while an ideal model should have a high value for the first metric and a low value for the second metric, empirically we find that the two metrics are positively correlated. therefore, the third combined metric offers a balanced way to assess pre-trained models. furthermore, inspired by @xcite , we note that when a model picks a stereotypical caption, the bias may come from either (i) a biased association within the caption itself, between the word(s) representing the target group and the word(s) representing the stereotypical attribute, or (ii) a biased association between the visual representation of the target group in the image and the textual representation of the stereotypical attribute in the caption. we therefore further design two fine-grained metrics to separately measure the intra-modal bias and the cross-modal bias. we conduct experiments on six representative pre-trained vision-language models using our vl-stereoset and our designed metrics. we find that while most of these pre-trained models generally do not pick semantically meaningless captions (e.g., my sister is hi), most of these models also exhibit a high degree of stereotypical behaviors, picking a stereotypical caption when presented with an antistereotypical image. we also find that such stereotypical behaviors are observed in all categories of stereotypical biases in the dataset, including gender, profession, race and religion, with gender stereotypes more evident. we further conduct experiments using two pre-trained models and the subset of our data covering gender stereotypes to separately measure intra-modal bias and cross-modal bias, and we find clear evidence to show that both sources of bias exist. ## methodology in this section, we first introduce our vlstereoset and the associated caption selection probing task. we then describe how we use the dataset to probe pre-trained vision-language models (pt-vlms). we further define a vision-language relevance score (vlrs) and a vision-language bias score (vlbs) that are used jointly used to assess a pt-vlm. finally, inspired by a recent study by @xcite , we define two fine-grained metrics to disentangle intra-modal bias and inter-modal bias. ## conclusion in this work, we constructed a vlstereoset dataset and proposed a caption selection probing task for measuring stereotypical bias in pre-trained visionlanguage models. using the metrics we defined, we showed that several representative pre-trained vision-language models exhibit strong stereotypical bias on vlstereoset, and further experiments with two models on gender bias data showed clear evidence to suggest that there are both intra-modal and inter-modal bias in these models. we hope that vlstereoset will spur further research in the important direction of fairness in nlp and vision.