Evaluating Bias and Fairness in Gender-Neutral Pretrained Vision-and-Language Models (2023)

Abstract: Pretrained machine learning models are known to perpetuate and even amplify existing biases in data, which can result in unfair outcomes that ultimately impact user experience. Therefore, it is crucial to understand the mechanisms behind those prejudicial biases to ensure that model performance does not result in discriminatory behaviour toward certain groups or populations. In this work, we define gender bias as our case study. We quantify bias amplification in pretraining and after fine-tuning on three families of vision-and-language models. We investigate the connection, if any, between the two learning stages, and evaluate how bias amplification reflects on model performance. Overall, we find that bias amplification in pretraining and after fine-tuning are independent. We then examine the effect of continued pretraining on gender-neutral data, finding that this reduces group disparities, i.e., promotes fairness, on VQAv2 and retrieval tasks without significantly compromising task performance.

## introduction as shown by @xcite and @xcite , inductive biases are essential for learning algorithms to outperform random guessing. these task-specific biases allow algorithms to generalize beyond training data but, necessarily, they should not be conflated with prejudicial or unwanted biases. unwanted bias, such as bias against demographic groups, can be found in many applications, from computer vision systems to natural language processing (nlp). vision-and-language (v&l) models lie at the intersection of these areas, where one of the key challenges is deploying robust models to perform high-level reasoning based on the multimodal context instead of exploiting biases in data @xcite . multiple studies @xcite @xcite have shown that v&l models leverage co-occurrences between objects is further pretrained on gender-neutral multimodal data d n , resulting in a gender neutral v&l model (m n d ). both models can then be used in a two-phase analysis: 1) bias amplification is measured on the intrinsic bias of pretrained models, and 2) bias amplification, task performance and fairness are evaluated on the extrinsic performance of fine-tuned models. and their context to make predictions, and thus are susceptible to unwanted biases. however, these authors do not explore the broad landscape of v&l models and focus on biases in common visual datasets @xcite , only on pretrained models @xcite or only focus on one application, e.g., image captioning @xcite or semantic segmentation @xcite . in this work, we investigate to what extent the unwanted bias in a v&l model is caused by the pretraining data. to answer this question, we focus on one important aspect of bias encoded in v&l models, namely bias amplification. bias amplification occurs when a model exacerbates unwanted biases from the training data and, unlike other forms of bias, it is not solely attributed to the data, yet it can vary greatly during training @xcite . we explore bias amplification in two encoderonly v&l models: lxmert @xcite and albef @xcite , and the encoder-decoder model blip @xcite . specifically, we quantitatively and qualitatively analyse the relationship between the bias encoded in pretrained models, and after fine-tuning on downstream tasks including visual question answering, visual reasoning and image-text retrieval. while bias can be studied with respect to any protected attribute, the majority of nlp research has focused on (binary) gender @xcite @xcite . we also use gender bias as our case study but different to previous work, we advocate for the inclusion of gender-neutral terms @xcite and consider three gender categories based on visual appearance: male, female and genderneutral (e.g., @xcite . the use of both visual and grammatical gender information across v&l tasks is needed for identifying the target of, for example, a question. but the demographics of the subject should not solely influence the outcome of the model. otherwise, the model may reinforce harmful stereotypes resulting in negative consequences for certain group identities @xcite . motivated by this argument, we investigate the effect of shifting the projection of gender-marking to a gender-neutral space by continued pretraining on gender-neutral multimodal data-a form of domain adaptation @xcite )-and how it reflects on task performance after fine-tuning. figure 1 depicts an overview of our full workflow. ## problem formulation we characterize the gender bias encoded in v&l models in a two-phase analysis: i) intrinsic bias: first, we investigate the bias encoded after the v&l pretraining phase. these investigations will be performed using a set of original, pretrained models m d , and models that have been further pretrained on gender-neutral data m n d in order to mitigate any biases learned during pretraining ( §4.4). we hypothesize that this bias mitigation technique will decrease both intrinsic and extrinsic biases encoded in the models. data our analysis relies on data where the gender of the main actor of the image is known. this is, to some degree, annotated in the crowdsourced text, e.g., image captions or questions. 3 following @xcite and @xcite , images are labelled as 'male' if the majority of its 3 zhao et al. (2021) annotated samples from the coco dataset @xcite with the perceived attributes (gender and skin-tone) of the people in the images. however, their gender labels agree on 66.3% of the images compared to caption-derived annotations. to be consistent across all datasets used in our project, we will not use their humancollected annotations for analysing gender bias on coco. ## conclusion this paper presented a comprehensive analysis of gender bias amplification and fairness of encoderonly and encoder-decoder v&l models. the intrinsic bias analysis shows consistent results -in terms of bias mitigation -in models trained on gender-neutral data, even if these models reflect biases present in data instead of diminishing them (as we observed with lxmert). in line with previous findings in language models @xcite @xcite , intrinsic bias in v&l models does not necessarily transfer to extrinsic bias on downstream tasks. similarly, we find that the bias in a model and its empirical fairness -group disparity on task performanceare in fact independent matters, which is in line with the nlp literature @xcite . intrinsic bias can potentially reinforce harmful biases, but these may not impact the treatment of groups (or individuals) on downstream tasks. we believe that bias and fairness should always be carefully evaluated as separate matters. one of they key findings of our work is that the extra pretraining steps on gender-neutral data are beneficial to reduce the group disparity in every model architecture tested on vqav2, and in the majority of models for both retrieval tasks. crucially, there is no penalty to pay for this fair outcome: the overall task performance of gender-neutral models is similar or better than their original versions.