Controlling for Stereotypes in Multimodal Language Model Evaluation (2022)

Abstract: We propose a methodology and design two benchmark sets for measuring to what extent language-and-vision language models use the visual signal in the presence or absence of stereotypes. The first benchmark is designed to test for stereotypical colors of common objects, while the second benchmark considers gender stereotypes. The key idea is to compare predictions when the image conforms to the stereotype to predictions when it does not. Our results show that there is significant variation among multimodal models: the recent Transformer-based FLAVA seems to be more sensitive to the choice of image and less affected by stereotypes than older CNN-based models such as VisualBERT and LXMERT. This effect is more discernible in this type of controlled setting than in traditional evaluations where we do not know whether the model relied on the stereotype or the visual signal.

## introduction the center of gravity of nlp research has shifted to the development of language models (lms) for representation and generation of text, and most recent high-impact research contributions describe new lms. for some tasks, a model needs to take into account not only a text but also some non-textual information, and a wide range of multimodal lms have been developed that allow the representation of a text jointly with some external modality. most of this work focuses on visual tasks where nlp models need to be integrated with computer vision models; examples of tasks in this area include visual question answering and caption generation. a range of combined language-and-vision lms have been developed using different approaches for integrating representations of text and of images or videos. but can we be sure that a multimodal model actually uses the provided visual information instead figure 1 : an example of a controlled test of a masked language model for a color stereotype. we compute the output from the mlm head when providing an image of an object with a stereotypical color (a yellow banana) and compare it to the output when the object has an unusual color (green). if the mlm is strongly affected by a stereotype bias, the predictions change little. it has been widely noted that representations of language are affected by several kinds of stereotypes, which we loosely define as any type of phenomenon that has a highly skewed prior probability distribution. in these cases, the skewed distribution may cause a model to simply go with the default choice and ignore contextual information that would suggest an unusual analysis. most of the discussion in the field has been about stereotypes relating to various demographic attributes @xcite , but in this work, we use the term "stereotype" in the more general sense mentioned above. this issue is likely to affect multimodal lms as well, although we are aware of no previous work that investigates this phenomenon systematically; for instance, if some object is often associated with some visual property (e.g. a color or shape), this property may be predicted by the model even in cases where it is not present. this effect may also have methodological implications in benchmarks for the evaluation of lms: if a model predicted the correct answer, did it do so because of the stereotype or because it actually used the available visual information? in this work, we propose a methodology and develop two benchmark sets for stress-testing multimodal lms to determine to what extent they are affected by problems related to stereotypes. the key idea is to look at predictions of a language/vision lm with different visual inputs and compare the behavior of the lm in the presence or absence of stereotypes. for cases when a stereotype is present, we compare model outputs when the image does correspond to the stereotype to when it does not. the rest of the paper is organized as follows. section 2 discusses the design of the benchmark sets and how we use them to investigate multimodal lms for stereotypes. details about the multimodal lms we have used are covered in section 3, and section 4 describes how they are applied for the benchmarks, while section 5 presents the figures achieved on the benchmarks and discusses their implications. in section 6, we discuss related research. finally, section 7 summarizes the main points and discusses limitations and possible extensions. ## design of benchmark datasets we have collected two datasets consisting of textual templates and corresponding images. these datasets were selected because in these cases it was relatively easy to collect images exemplifying some visual property, and where on the one hand we could find images corresponding to a stereotype, but on the other hand also control images not corresponding to the stereotype. these datasets also contain subsets we call "neutral" where stereotypes are not present. the purpose of these images is to investigate whether lms are more sensitive to the choice of images in the cases when they cannot rely on stereotypes. ## multimodal language models the transformer @xcite is a sequence-based model that is now the standard architecture in nlp for devising representation and generation components in neural models. pretrained language models such as bert @xcite based on the architecture of transformers, have proven capable of learning powerful representations applicable to a wide range of tasks. they have yielded state-of-the-art performance in many downstream tasks. multimodal models fusing the textual and visual modalities have been devised by researchers after looking at the huge success of pre-trained language models. in such models, multiple modalities are considered, and data for the training of the models is in multiple modalities. as our research problem revolves around the aspect of multimodality, we will focus on two modalities: a textual and a visual signal. the visual signal is in the form features in the form of 36 roi (region of interest) boxes for each image, and these roi boxes are used as features. each of these 36 roi boxes are vectors of size 2048. the boxes with highest probability/confidence are chosen. the visual representations are appended at the end of the sequence of word embeddings. lxmert this model is a dual stream multimodal model, where the inputs are processed through two transformers, for natural language and vision signals respectively. text is processed in the same manner as of visualbert, based on bert's tokenizer. the image features for the lxmert are extracted by the faster-rcnn, in the same way as of visualbert, but we also feed the normalized boxes alongside features, which are locations of these bounding boxes. at last, the transformers are fused. flava flava has a text encoder, an image encoder, and a multimodal encoder. it is a dual stream multimodal model. the text encoder, has an architecture of vit (visual transformers) to extract single-modal text representations. for the images, an image encoder based on vit architecture extracts single-modal image representations. a separate transformer, multimodal encoder, is then applied. the unimodal representations are passed through the fusion encoder which fuses two modalities, and thus obtaining cross-modal representations. ## methodology of analysis our benchmarking method uses a cloze-style fillin-the-blank approach @xcite , which has previously been applied in experiments investigating the interaction between visual and linguistic representation @xcite . this approach is easy to apply to bert-style models that include a masked language model (mlm) as part of their pre-training pipeline. when applying the mlm in our experiment, the model is provided with an image and a text prompt, where the visual property to be predicted by the model has been replaced by the mask dummy token. we then investigate how well the missing token is predicted under different circumstances. since the nature of the two benchmarks is different, we had to apply different methodologies to get the results. we discuss these details below. ## results we evaluated the three selected models on the two benchmarks. in both cases, we compare the predictions when a stereotype is present and the image corresponds to the stereotype to the case where the image does not correspond to the stereotype. we also evaluate cases where there is no stereotype and we carry out similar comparisons in this case. additionally, we look at the model's predictions when provided with a black dummy image. ## related work this work falls in the broad category of model analysis @xcite of transformer models @xcite . @xcite divide previous approaches to model analysis into several methodological categories; in the current work, we use an approach based on behavioral testing of a specific model behavior. specifically, our analysis is based on the outputs of the masked language model head of bert-like models, similarly to how @xcite and @xcite tested bert models for basic encyclopedic and commonsense knowledge. the methodology based on targeted behavioral testing has also been used to investigate a number of research questions in the analysis of languageand-vision transformer models. in particular, a number of investigations look at what type of generalizations happen between the visual and textual modalities. @xcite claimed that when considering attention scores, the effect of the visual modality is limited and that the textual modality dominates. @xcite investigated the effect of multimodal training on textual representations, and concluded that the degree of transfer between the representations of the respective modalities is limited, at least for cnn-based models; @xcite drew similar conclusions based on more extensive experiments that also include the flava model. @xcite considered the task of predicting numbers and arrived at a conclusion similar to ours: frequently occurring numbers are predicted more often by the model. the previous work that is most closely related to our in terms of research questions and methodology is that by @xcite . they designed ablation tests where parts of the image or the text are hidden; as we have discussed, this setup is comparable to our experiments where black and white-noise images are used. @xcite introduced the idea of "foils": texts that differs minimally from the one describing the image. our use of adversarially selected images can be seen as similar to the idea of foils, but focused on the visual modality. ## conclusions in this work, we have proposed a methodological framework based on controlled tests designed to tease out the influence of stereotypes on the predictions of visually augmented language models. the key idea is that we expect common evaluation benchmarks to include many stereotypical cases that can easily be predicted simply by relying on language statistics. in order to disentangle the effect of the stereotype and the contribution of the visual representations we compare the model's output in cases where the provided image adheres to the stereotype to cases where it does not. we also consider the model's behavior in cases where there are no stereotypes, that is when the prior distribution of outputs is more evenly distributed. as an application of this framework, we created two datasets to facilitate the investigation of stereotypes for two properties: the color of objects and the gender of people. each dataset contains a set of text prompts and corresponding image pairs, where one image in the pair corresponds to the stereotype and the other is a control where the stereotypical property is not present. this allows comparisons to be carried out in a controlled fashion. using the two benchmark sets, we evaluated three mlm-based visually augmented transformer models: visualbert, lxmert, and flava. there are clear differences between the models, and in particular some of these differences emerge much more clearly in the controlled setting. for instance, the cnn-based lxmert and transformerbased flava achieve similar scores in terms of raw accuracy scores for predicting the color of objects in images. however, if we consider the control images where the objects do not have the stereotypical color, the flava outperforms lxmert by a wide margin, since lxmert keeps predicting the stereotypical color. this means that we can see clear differences among the models with respect to how sensitive they are to the choice of images. for the gender stereotypes experiments, the results were somewhat unexpected since it turned out that the older cnn-based models almost consistently assigned higher probabilities to malerelated words, where we had expected at least the lxmert model to be somewhat affected by stereotypes suggested by the textual prompt. the newer flava model on the other hand again predicts more consistently with the input image in this experiment, and only falls back on stereotypes when the input images are uninformative.