A Multi-dimensional study on Bias in Vision-Language models (2023)

Abstract: In recent years, joint Vision-Language (VL) models have increased in popularity and capability. Very few studies have attempted to investigate bias in VL models, even though it is a well-known issue in both individual modalities. This paper presents the first multi-dimensional analysis of bias in English VL models, focusing on gender, ethnicity, and age as dimensions. When subjects are input as images, pre-trained VL models complete a neutral template with a hurtful word 5% of the time, with higher percentages for female and young subjects. Bias presence in downstream models has been tested on Visual Question Answering. We developed a novel bias metric called the Vision-Language Association Test based on questions designed to elicit biased associations between stereotypical concepts and targets. Our findings demonstrate that pre-trained VL models contain biases that are perpetuated in downstream tasks.

## introduction computer vision (cv) and natural language processing (nlp) have entered a new era as a result of the development of large pre-trained models based on the transformer architecture @xcite . this advancement has also involved their multi-modal integration into vision-language (vl) models, reaching significant progress @xcite @xcite @xcite @xcite . as these technologies are used in more and more applications, with real-world consequences @xcite , a new branch of research has begun to investigate their fairness and discrimination issues. while many studies have shown that bias, defined in this paper as human prejudice @xcite , exists in uni-modal cv and nlp @xcite et @xcite @xcite , it is concerning that bias research in multi-modal vl applications is still in its infancy @xcite @xcite , as combining those two complex applications are likely to produce even more issues @xcite . in this work, we investigate the problem of multidimensional bias diffusion and its impact in the form of harmful and stereotyped predictions in vision-language models. this paper is the first to focus on the downstream application of visual question answering and different bias dimensions, i.e., gender, ethnicity, and age. first, we analyzed pre-trained vl models' harmful completions, extending existing metrics proposed for uni-modal nlp models @xcite to multi-modal ones ( ## related work while studied individually, bias is still an understudied problem in vision and language models. bias has been demonstrated to perpetuate in natural language processing models in a variety of languages and tasks both in word and contextualized embeddings @xcite @xcite @xcite @xcite @xcite @xcite . similarly, works in computer vision (buolamwini and gebru, 2018) have studied the performance of different gender classifiers over images of faces grouped by gender and skin tone, showing a consistent difference in error rate at the expense of darker-skinned females, who are the worst-represented class. the recent advancement in both vision-language models has made it possible to design new architectures @xcite @xcite for various crossmodal tasks, e.g., image-sentence retrieval, image captioning, visual question answering, and phrase grounding. as a relatively new research direction, bias research on vl models is, however, still in its infancy. @xcite constructed a dataset of counterfactual template-based image-text pairs for measuring gender bias in pre-trained vl models. then, they compared the difference between masked prediction probabilities of factual and counterfactual examples. e.g., the difference of p ([m ask] = "shopping") for the sentence the gender is [m ask] between male and female inputs. @xcite demonstrated that vl models prefer to reinforce a stereotype over faithfully describing the visual scene. they studied how within-and cross-modality gender biases are expressed using a set of template-based data on a curated list of stereotypical entities (e.g., suitcase vs. purse). @xcite presented an extensive study on investigating gender and racial bias in vqa datasets. they demonstrate the presence of harmful samples, denoting gender and racial stereotypes. @xcite measured stereotypical bias in pre-trained vl models by extending stereoset, a text-only dataset proposed for detecting stereotypes in language models @xcite . they introduced vlstereoset, a benchmark comprising images depicting scenarios that are either stereotypical or anti-stereotypical. each image is accompanied by three candidate captions, sourced from stereoset, including one that is stereotypical, one that is anti-stereotypical, and one that is semantically meaningless. the underlying assumption is that if a pre-trained vl model shows a preference for the stereotypical statement, it signifies a demonstration of stereotypical behavior. all of the models they studied displayed stereotypical behaviors across all categories (gender, profession, race, and religion). finally, @xcite demonstrated the extent of stereotypes and complex biases present in image generation models and the images generated by them. they show that simple user prompts can generate thousands of images that perpetuate dangerous stereotypes based on race, ethnicity, gender, class, and intersectionality. moreover, their study revealed instances of near-total amplification of stereotypes, and that prompts referencing social groups result in complex stereotypes that are challenging to mitigate. similar to our work, @xcite explored bias metrics to measure gender and racial bias in facial images on contrastive pretraining vl model such as clip @xcite . they adapted weat to vl models and proposed ranking metrics for the text-image retrieval downstream task. additionally, they introduced a supervised adversarial debiasing technique, which exhibited a significant reduction according to the employed metrics. our study overcomes existing ones by proposing an analysis of bias in different dimensions (gender, ethnicity, age) both at pre-trained and task-specific levels, i.e., visual question answering. ## conclusions this paper presents the first investigation on bias in vision-language models that focus on multiple dimensions (i.e., gender, ethnicity, and age) and analyzes the downstream application of visual question answering. this work extends the methodologies of state-of-the-art bias evaluation metrics @xcite to the multi-modal vision and language framework. our experiments have shown the presence of noticeable biases in many vision and language models with potentially harmful consequences. in future work, we aim to broaden both the model and the language coverage, as well as to develop a bias detection pipeline that can be automatically run whenever a new vl model is released @xcite .