Multi-Modal Bias: Introducing a Framework for Stereotypical Bias Assessment beyond Gender and Race in Vision–Language Models (2023)

Abstract: Recent breakthroughs in self-supervised training have led to a new class of pretrained vision–language models. While there have been investigations of bias in multimodal models, they have mostly focused on gender and racial bias, giving much less attention to other relevant groups, such as minorities with regard to religion, nationality, sexual orientation, or disabilities. This is mainly due to lack of suitable benchmarks for such groups. We seek to address this gap by providing a visual and textual bias benchmark called MMBias, consisting of around 3,800 images and phrases covering 14 population subgroups. We utilize this dataset to assess bias in several prominent self-supervised multimodal models, including CLIP, ALBEF, and ViLT. Our results show that these models demonstrate meaningful bias favoring certain groups. Finally, we introduce a debiasing method designed specifically for such large pretrained models that can be applied as a post-processing step to mitigate bias, while preserving the remaining accuracy of the model.

## introduction the recent emergence of large pretrained visionlanguage models has revolutionized many multimodal tasks previously considered impractical to solve. although architectures capable of jointly addressing computer vision and nlp tasks using a single unified model have been around for a while @xcite @xcite , recent advances in self-supervised training methods have amplified the significance and applicability of such models. the sheer power of these methods is highly dependent on the scale of the model and the diversity and distributional properties of the dataset on which they are trained. due to their wide range of diverse applications @xcite , it is of utmost importance to be aware of the shortcomings of vision-language pretrained (vlp) models as well as their capabilities. one such limitation is that, like any other machine learning system, multimodal models may be prone to exhibiting human-like stereotypical biases such as gender or race-related stereotypes @xcite . for instance, pretrained language models have been shown to associate male-gendered phrases and sentences to a greater extent with certain high-paying professions and even with individual traits such as intelligence, in comparison to female-gendered phrases @xcite . similarly, it has been found that hispanic and african american names may be tied to words representing danger and crime more often than caucasian names @xcite . certain biases have also been identified in computer vision models as well @xcite . such biases are discriminatory towards affected population groups and can be extremely harmful to society the more these models are deployed in real-world applications. while there has been some research aimed at identifying and addressing biases in visionlanguage models, most such studies have focused on gender and racial biases, while other notable groups such as religious minorities, national minorities, lgbtq people, and people with disabilities have received much less attention, despite their legal status as protected groups in the us. this is alarming considering the fact that the potentially affected groups together constitute a considerable part of the global population. for instance, the us census bureau reported approximately 40 million people identifying as immigrants in the us and 244 million world-wide as of 2015. foot_0 furthermore, approximately 40 million people in the us and about 1 billion people in the world suffer from some sort of disability. 2 one of the main obstacles for bias analysis of these relevant population groups has been the lack of standardized benchmark datasets that specifically enable an analysis of how they may be affected. in this paper, we attempt to address this problem by gathering and releasing a visual and textual bias benchmark called mmbias, consisting of approximately 3,500 images and 350 phrases covering over 14 minority subgroups. furthermore, we utilize the dataset to measure stereotypical bias in several prominent self-supervised multimodal vlp models that have attracted significant attention recently, namely openai clip @xcite , albef @xcite , and vilt @xcite . in our experiments, we quantify the bias present in these models, including both cross-modal and intra-modal bias. our results confirm that these models harbor meaningful biases favoring certain groups. finally, we introduce a novel debiasing method designed for such large pretrained models that can be applied as a postprocessing step to mitigate bias, and we show that this step does not adversely affect the performance in a substantial way. ## related work the majority of work on language models only focuses on gender and racial bias assessment @xcite . however, there have been some studies that consider bias with regard to other categories such as profession, religion, and disability as well @xcite . however, these forms of bias are not just exclusive to the language domain, and image classifiers as well as multimodal models have also been shown to demonstrate such biases @xcite . thus far, there has been rather limited work on multimodal bias assessment of self-supervised models such as clip, and prior work considers only gender and racial biases. wang et al. (2021b) measures the gender and racial bias in clip's image classification module using the fairface dataset, while @xcite further show that clip associates male-gendered phrases to high-paying professions more than female-gendered phrases. @xcite provide insights towards potential applications of the clip model and further study and evaluate its gender/racial bias as well as measuring the misclassification differences between different subgroups. @xcite measure and propose solutions to gender bias in several image captioning systems. the only work that addresses other relevant groups such as religion, sexual orientation, and disability in the image space is @xcite . however, the data considered in the study is limited, consisting of only around 600 images, out of which around 500 again correspond to gender and racial biases. this leaves only around 100 for other protected groups, i.e., fewer than 20 images for each protected group study. @xcite use this limited dataset to measure bias in several self-supervised visual models but the authors do not explore multimodal models such as clip. an orthogonal line of research has been pursued in @xcite , where several multimodal vision-language models are analyzed to measure these models' tendency to pick stereotypical statements as captions for anti-stereotypical images in pre-trained vision-language models. with mm-bias, we thus hope to enable further research on diverse forms of bias in vision-language models. 3 methodology bias and fairness. in conventional social studies, one of the most well-established and widelyinvestigated forms of biases is what is known as implicit bias, or as social stereotypes, defined and investigated in @xcite . this type of bias is usually measured using implicit associate testing (iat), introduced in the seminal work of @xcite , and has so far been widely used to describe and account for a wide range of implicit prejudices @xcite . iat experiments quantify human implicit bias by measuring response times differences when human subjects are asked to pair similar concepts and different concepts. in its original form, iat was used to to measure the degree of pleasantness (a.k.a. valence in psychology), of entities such as "flowers" and "insects" by pairing them with abstract attributes such as pleasant and unpleasant @xcite . @xcite showed that a similar iat testing paradigm can be applied to bias measurement in deep embeddings. in this approach, instead of subject reaction time, the proximity of embeddings of a basket of words that represent a concept is measured. furthermore, word sentiment is usually used to represent valence, due to well-established studies linking word sentiment with the psychological concept of valence @xcite . the experimental methodology used in our study follows similar principles. more generally, a machine learning system may be deemed unbiased or fair when its predictions do not favor members of any relevant population group or discriminate against any other @xcite . for instance, suppose that the class under consideration is religion and we are evaluating pleasantness / unpleasantness scores a system would assign to each considered religious subgroup. a machine learning system is assessed as fair if and only if the scores it assigns to different religious subgroups do not differ substantially. more formally, in a bias study, the two subgroups under study, also known as target entities, may be represented as sets of instances @xmath0, x 2 , ..., x n } and @xmath1, y 2 , ..., y n }. for example, x may be images corresponding to islam and y to christianity. furthermore, the attributes towards which the bias is being measured may be given as sets @xmath2, a 2 , ..., a m } and @xmath3, b 2 , ..., b m }. for example, a could be a set of words representing pleasantness, while b represents unpleasantness. similarly, many gender-bias studies consider sets for high paying vs. low paying professions as attribute sets. a machine learning model is then said to be fair towards subgroups x and y with respect to attributes a and b if and only if ϕ(x, a, b) ≈ ϕ(y , a, b), where ϕ is some scoring function that scores the similarity of the sets of attributes a, b to a target entity x or y . ## the mmbias dataset the majority of the work on societal bias analysis so far focuses on unimodal language models. although there has been some limited work on multimodal models, these studies mainly focus on gender and racial disparities. as a result, bias with regard to other classes, including religion, nationality, sexual orientation, and disability have largely been unexplored. this has been mainly due to the lack of standardized benchmark datasets that specifically target these minority groups. to address this concern, we gather and release the first multimodal dataset of this size in this line of research that spans over a wider range of groups. we hope that this dataset can serve as a benchmark in future research. our dataset, referred to as mmbias, contains 3,500 target images and 350 english phrases corresponding to different target concepts. each target target concept x target values {x 1 , ..., x n } religion islam, christianity, judaism, buddhism, hinduism nationality american, arab, chinese, mexican disability physical disability, mental disability, no disability sexual orientation homosexual, heterosexual our dataset also contains 20 textual phrases related to each target concept, used for bias experiments in the textual domain. data compilation for gathering the image data, we invoke the flickr api and retrieve 1,000 most relevant images for each target concept using 10-12 search keywords for each. the keywords are chosen to be as diverse as possible to minimize any potential bias in the data gathering process to the extent possible. then human annotators are used to filter out noisy images. annotators were instructed to manually eliminate irrelevant or explicit content as well as images that contained private information or names/addresses. in order to balance the dataset, 250 images were randomly chosen for each concept out of the filtered images, and we only consider images with a creative commons license. the processing pipeline and quality control is similar to the one used for the creation of flickr30k @xcite . furthermore, the textual part of the dataset contains phrases such as "this is x." replacing x with a "muslim person", "christian person", etc. the same aforementioned keywords were used to retrieve textual data for each concept using the relatedwords site foot_2 followed by a similar data cleaning and noise filtering process. finally, in order to conduct intra-visual bias studies, mmbias also contains two sets of images corresponding to visual pleasantness and unpleasantness, called the valence dataset. these sets were constructed by following a similar method to @xcite , retrieving images corresponding to pleasant concepts such wealth, peace, babies, love, butterflies, etc. and unpleasant concepts such as death, injury, prison, fear, etc. analysis. fig. 2 provides some sample images taken from the dataset. each row shows a different target class. fig. 3 provides deeper insights into the sizes of the crawled images. the x-axis reflects the surface area of the image in pixels, scaled by a factor of 1,000. as can be seen, image sizes in most classes follow a normal distribution with a mean of around 110,000 pixels, translating to approximately 340x340 images, with the exception of images corresponding to the nationality class, which have a slightly higher mean of around 350x350. the height and width of images does not vary substantially across the dataset. furthermore, we analyze the separability of our dataset with regards to image classes. the images are fed into the clip model and the first 100 principal components are extracted from the resulting embeddings, and then t-sne is applied. fig. 4 shows the t-sne representation of the images. we observe that the dataset can be well-separated forming clearly-defined clusters. for instance, we notice that different religions form well-separated clusters. interestingly the clusters that are more intertwined correspond to correlated subjects such as the religion islam and the arab nationality designation. this is not surprising given islam is particularly prevalent in arab countries and thus many of the images share similar features. ## experimental evaluation we have conducted three sets of experiments to assess and quantify the bias in the aforementioned models: clip, albef, and vilt. the following sections explain the details of each setting. ## bias mitigation algorithm bias mitigation methods typically fall into one of three categories: data augmentation (fair resampling), model adjustment, and embedding post processing algorithms. each of these alternatives have their own benefits and drawbacks, but a major limitation of the first two is that they require retraining the models. this can be burdensome in many cases. in particular, we often lack access to the dataset, the model's training procedure, or in the case of large pretrained models, retraining may also be computationally infeasible on typical hardware and cost budgets. post-processing methods, on the other hand, may be invoked as a fast and efficient plug-and-play method to modify learned embeddings without the need for retraining. since vision-language tasks are complex, vlp models usually have large architectures to be able to capture all the complex dynamics. however, this can cause them to learn redundant or highly correlated features, since they are not optimally compressed. these features are not only computationally wasteful but can also amplify model bias. due to the high correlation among some features, we can remove some without affecting performance, while simultaneously reducing bias. in order to identify those features, we directly optimize for the objective in eq. 1 by removing features in a greedy manner, pruning n dimensions that cause the largest decrease in bias effect size. however, we only consider the features that exhibit a small mutual information with respect to classification labels. θ can be set empirically and this ensures only redundant dimensions are removed. algorithm 1 details the steps of our technique. the results of this debiasing method are presented in regarding the choice of n (number of features removed) in the bias mitigation algorithm, a larger n will affect the performance of the model more negatively, as previously observed in other dimensionality reduction algorithm. in order to find a reasonable n we can plot the bias reduction as well as performance reduction as a function of n . inspecting this graph allows us to consider the trade-off between greater bias removal and the loss of accuracy, allowing us to choose an n that decreases the bias in a meaningful way while not affecting performance significantly. ## conclusion most bias analysis studies focus on gender and racial biases, which is primarily due to a lack of suitable data to consider further important forms of bias. in this study, we have compiled a new multimodal bias assessment dataset called mm-bias enabling the study of bias affecting population groups largely neglected in prior studies. our dataset consists of around 3,500 images and hundreds of phrases covering over 14 minority subgroups. furthermore, based on a formulation of the bias-fairness problem, we draw on this data to assess the level of bias in several prominent selfsupervised multimodal models, including clip, albef, and vilt. our results show that these models demonstrate meaningful bias towards certain groups. finally, we introduce a novel bias mitigation technique designed specifically for large pretrained models that can be applied as a postprocessing step to reduce bias, and show that it has negligible effects on classification performance as well as data separability. our data and code is available at github.com/sepehrjng92/mmbias.