Worst of Both Worlds: Biases Compound in Pre-trained Vision-and-Language Models (2022)

Abstract: Numerous works have analyzed biases in vision and pre-trained language models individually - however, less attention has been paid to how these biases interact in multimodal settings. This work extends text-based bias analysis methods to investigate multimodal language models, and analyzes intra- and inter-modality associations and biases learned by these models. Specifically, we demonstrate that VL-BERT (Su et al., 2020) exhibits gender biases, often preferring to reinforce a stereotype over faithfully describing the visual scene. We demonstrate these findings on a controlled case-study and extend them for a larger set of stereotypically gendered entities.

## introduction pre-trained contextualized word representations @xcite @xcite @xcite have been known to amplify unwanted (e.g. stereotypical) correlations from their training data @xcite @xcite . by learning these correlations from the data, models may perpetuate harmful racial and gender stereotypes. the success and generality of pre-trained transformers has led to several multimodal representation models @xcite @xcite which utilize visual-linguistic pre-training. these models also condition on the visual modality, and have shown strong performance on downstream visual-linguistic tasks. this additional input modality allows the model to learn both intra-and inter-modality associations from the training data -and in turn, gives rise to unexplored new sources of knowledge and bias. for instance, we find (see figure 1 ) the word purse's female association can override the visual evidence. while there are entire bodies of work surrounding bias in vision @xcite and language @xcite , there are relatively few works at the intersection of the two. as we build models that include multiple input modalities, each containing their own biases and artefacts, we must be cognizant about how each of them are influencing model decisions. in this work, we extend existing work for measuring gender biases in text-only language models to the multimodal setting. specifically, we study how within-and cross-modality biases are expressed for stereotypically gendered entities in vl-bert @xcite , a popular visuallinguistic transformer. through a controlled case study ( §4), we find that visual-linguistic pretraining leads to vl-bert viewing the majority of entities as "more masculine" than bert @xcite does. additionally, we observe that model predictions rely heavily on the gender of the agent in both the language and visual contexts. these findings are corroborated by an analysis over a larger set of gendered entities ( §5). ## bias statement we define gender bias as undesirable variations in how the model associates an entity with different genders, particularly when they reinforce harm- ## methodology we identify three sources of learned bias when visual-linguistic models are making masked word predictions -visual-linguistic pre-training, the visual context, and the language context. the former refers to biases learned from image-text pairs during pre-training, whereas the latter two are biases expressed during inference. ## case study in this section, we present a case study of our methodology by examining how gender bias is expressed in each bias source for several entities. the case study serves as an initial demonstration of our methodology over a small set of gendered entities, whose findings we expand upon in section 5. ## comparing model bias with human annotations of stereotypes to test if the trends in the case study match human intuitions, we curate a list of 40 entities, which are considered to be stereotypically masculine or feminine in society. 4 we analyze how the gendered-ness of these entities is mirrored in their vl-bert language bias scores. to evaluate the effect of multimodal training on the underlying language model, we remove the visual input when extracting language model probabilities and compare how the language bias varies between textonly vl-bert and the text-only bert model. for the language input, we create template captions similar to those described in ## future work and ethical considerations this work extends the bias measuring methodology of @xcite to multimodal language models. our case study shows that these language models are influenced by gender information from both language and visual contexts -often ignoring visual evidence in favor of stereotypes. gender is not binary, but this work performs bias analysis for the terms "male" and "female" -which are traditionally proxies for cis-male and cis-female. in particular, when images are used of male and female presenting individuals we use images that self-identify as male and female. we avoid guessing at gender presentation and note that the biases studied here in this unrealistically simplistic treatment of gender pose even more serious concerns for gender non-conforming, nonbinary, and trans-sexual individuals. a critical next step is designing more inclusive probes, and training (multi-modal) language models on more inclusive data. we welcome criticism and guidance on how to expand this research. our image based data suffers from a second, similar, limitation on the dimension of race. all individuals self-identified as "white" or "black", but a larger scale inclusive data-collection should be performed across cultural boundaries and skintones with the self-identification and if appropriate prompts can be constructed for llms. ilar to suits above) are associated with business, which we noted is customarily a male occupation. if a model tends to associate purses with women, in the presence of contrary visual evidence, it could reinforce heteronormative gender associations. similarly, associating briefcases with primarily men undermines the efforts of women to enter the workforce. wine vs beer alcoholic drinks also contain gendered stereotypes that could be perpetuated by visual-linguistic models. beer is typically considered to be a masculine drink @xcite , whereas wine is associated with feminine traits @xcite .