Measuring Social Biases in Grounded Vision and Language Embeddings (2021)

Abstract: We generalize the notion of measuring social biases in word embeddings to visually grounded word embeddings. Biases are present in grounded embeddings, and indeed seem to be equally or more significant than for ungrounded embeddings. This is despite the fact that vision and language can suffer from different biases, which one might hope could attenuate the biases in both. Multiple ways exist to generalize metrics measuring bias in word embeddings to this new setting. We introduce the space of generalizations (Grounded-WEAT and Grounded-SEAT) and demonstrate that three generalizations answer different yet important questions about how biases, language, and vision interact. These metrics are used on a new dataset, the first for grounded bias, created by augmenting standard linguistic bias benchmarks with 10,228 images from COCO, Conceptual Captions, and Google Images. Dataset construction is challenging because vision datasets are themselves very biased. The presence of these biases in systems will begin to have real-world consequences as they are deployed, making carefully measuring bias and then mitigating it critical to building a fair society.

## introduction since the introduction of the implicit association test (iat) by @xcite , we have had the ability to measure biases in humans. many iat tests focus on social biases, such as inherent beliefs about someone based on their racial or gender identity. social biases have negative implications for the most marginalized people, e.g., applicants perceived to be black based on their names are less likely to receive job interview callbacks than their white counterparts @xcite . @xcite introduce an equivalent of the iat for word embeddings, called the word embedding association test (weat), to measure word associations between concepts. the results of testing bias in word embeddings using weat parallel those seen when testing humans: both reveal many of the same biases with similar significance. @xcite extend this work with a metric called the sentence encoder association test (seat), that probes biases in embeddings of sentences instead of just words. we take the next step and demonstrate how to test visually grounded embeddings, specifically embeddings from visuallygrounded bert-based models by extending prior work into what we term grounded-weat and grounded-seat. the models we evaluate are vil-bert @xcite , @xcite , lxmert @xcite and vl-bert @xcite . grounded embeddings are used for many consequential tasks in natural language processing, like visual dialog @xcite and visual question answering @xcite . many realworld tasks such as scanning documents and interpreting images in context employ joint embeddings as the performance gains are significant over using separate embeddings for each modality. it is therefore important to measure the biases of these grounded embeddings. specifically, we seek to answer three questions: do joint embeddings encode social biases? since visual biases can be different from those in language, we would expect to see a difference in the biases exhibited by grounded embeddings. biases in one modality might dampen or amplify the other. we find equal or larger biases for grounded embeddings compared to the ungrounded embeddings reported in @xcite . we hypothesize that this may be because visual datasets used to train multimodal models are much smaller and much less diverse than language datasets. can grounded evidence that counters a stereotype alleviate biases? the advantage to having multiple modalities is that one modality can demon-strate that a learned bias is irrelevant to the particular task being carried out. for example, one might provide an image of a woman who is a doctor alongside a sentence about a doctor, and then measure the bias against women doctors in the embeddings. we find that the bias is largely not impacted, i.e., direct visual evidence against a bias helps little. to what degree are biases encoded in grounded word embeddings from language or vision? it may be that grounded word embeddings derive all of their biases from one modality, such as language. in this case, vision would be relevant to the embeddings, but would not impact the measured bias. we find that, in general, both modalities contribute to encoded bias, but some model architectures are more dominated by language. vision could have a more substantial impact on grounded word embeddings. we generalize weat and seat to grounded embeddings to answer these questions. several generalizations are possible, three of which correspond to the questions above, while the rest appear unintuitive or redundant. we first extracted images from coco @xcite and conceptual captions @xcite ; the images and english captions in these datasets lack diversity, making finding data for most existing bias tests nearly impossible. to address this, we created an additional dataset from google images that depicts the targets and attributes required for all bias tests considered. this work does not attempt to reduce bias in grounded models. we believe that the first critical step to doing so, is having metrics and a dataset to understand grounded biases, which we introduce here. the dataset introduced along with the metrics presented can serve as a foundation for future work to eliminate biases in grounded word embeddings. in addition, they can be used as a sanity check before deploying systems to understand what kinds of biases are present. the relationship between linguistic and visual biases in humans is unclear, as the iat has not been used in this way. our contributions are: 1. grounded-weat and grounded-seat answering three questions about biases in grounded embeddings, 2. a new dataset for testing biases in grounded systems, 3. demonstrating that grounded word embeddings have social biases, 4. showing that grounded evidence has little impact on social biases, and 5. showing that biases come from a mixture of language and vision. ## related work models that compute word embeddings are widespread @xcite @xcite . given their importance, measuring the presence of harmful social biases in such models is critical. caliskan et al. (2017) introduce the word embedding association test, weat, based on the implicit association test, iat, to measure biases in word embeddings. weat measures social biases using multiple tests that pair target concepts, e.g., gender, with attributes, e.g., careers and families. may et al. (2019) generalize weat to biases in sentence embeddings, introducing the sentence encoder association test (seat). @xcite generalize seat to contextualized word representations, e.g., the encoding of a word in context in the sentence; @xcite also evaluated gender bias in contexutal embeddings from elmo. these advances are incorporated into the grounded metrics developed here, by measuring the bias of word embeddings, sentence embeddings, as well as contextualized word embeddings. blodgett et al. (2020) provide an in-depth analysis of nlp papers exploring bias in datasets and models and also highlight key areas for improvement in approaches. we point the reader to this paper and aim to draw from key suggestions from this work throughout. ## the grounded weat/seat dataset existing weat/seat bias tests @xcite al. (2017), may et al. (2019) and tan and celis (2019) ) contain sentences for categories and attributes; we augment these tests to a grounded domain by pairing each word/sentence with an image. visual-bert and vilbert were trained on coco and conceptual captions respectively, so we use the images in these datasets' validation splits by querying the captions for the keywords. to compensate for their lack of diversity, we collected another version of the dataset where the images are top-ranked hits on google images. results on coco and conceptual captions are still important for the bias tests that can be collected, for two reasons. first, it gives us an indication of where datasets are lack ## methods existing weat/seat bias tests @xcite base the word embedding association test (weat) on an iat test administered to humans. two sets of target words, x and y , and two sets of attribute words, a and b, are used to probe systems. the average cosine similarity between pairs of word embeddings is used as the basis of an indicator of bias, as in: by incorporating both target word classes x and y , this distance can be used to measure bias. the space of embeddings may encode social biases by making some targets, e.g., men's names or women's names, closer to one profession than another. in this case, bias is defined as one of the two targets being significantly closer to one set of embedding index word 1 man 2 woman 3 lawyer 4 teacher (a) possible embeddings for an ungrounded model embedding index word what the image shows 1 man any man 2 man any woman 3 woman any man 4 woman any woman 5 lawyer man lawyer 6 lawyer man teacher 7 lawyer woman lawyer 8 lawyer woman teacher 9 teacher man lawyer 10 teacher man teacher 11 teacher woman lawyer 12 teacher woman teacher (b) possible embeddings for a visually grounded model the effect size, i.e., the number of standard deviations in which the peaks of the distributions of embedding distances differ, of this metric is computed as: @xcite extend this test to measure sentence embeddings, by using sentences in the target and attribute sets. @xcite extend the test to measure contextual effects, by extracting the embedding of single target and attribute tokens in the context of a sentence rather than the encoding of the entire sentence. we demonstrate how to extend these notions to a grounded setting, which naturally adapts these two extensions to the data, but requires new metrics because vision adds new degrees of freedom to what we can measure. to explain the intuition behind why multiple grounded tests are possible, consider a trivial hypothetical dataset that measures only a single property; see with these additional degrees of freedom, we can formulate many different grounded tests in the spirit of eq. ( 2 ). we find that three such tests, described next, have intuitive explanations and measure different but complementary aspects of bias in grounded word embeddings. these questions are relevant to both bias and to the quality of word embeddings. for example, attempting to measure the impact of vision separately from language on grounded word embeddings can indicate if there is an over-reliance on one modality over another. we evaluate bias tests on embeddings produced by transformer-based vision and language models which take as input an image and a caption. models are used to produce three kinds of embeddings (of 1 an alternate way to construct such a dataset might have ambiguity about which of two agents a sentence is referring to, more closely mirroring how language is used. this would require images that simultaneously depict both targets, e.g., both a man and woman who are teachers. finding such data is difficult and may be impossible in many cases, but it would also be a less realistic measure of bias. in practice, systems built on top of grounded embeddings will not be used with balanced images, and so while in a sense more elegant, this construction may completely misstate the biases one would see in the real world. ## results we evaluate each model on images from the dataset used for pretraining and our collected images from google image search. pretraining datasets are ms-coco for visualbert @xcite and lxmert @xcite and conceptual captions for vilbert @xcite and vl-bert @xcite foot_0 . image features are computed in the same manner as in the original publications. we compute p-values using the updated permutation test described in @xcite . in each case, we evaluate the task-agnostic, pretrained base model without task-specific fine tuning. the effect of task-specific training on biases is an interesting open question for future work. overall, the results are consistent with prior work on biases in both humans and with ungrounded models such as bert. following @xcite , each experiment examines the bias in three types of embeddings: word embeddings, sentence embeddings, and contextualized word embeddings. while there is broad agreement between these different ways of using embeddings, they are not identical in terms of which biases are discovered. it is unclear which of these methods is more sensitive, and which finds biases that are more consequential in predicting the results of a larger system constructed from these models. methods to mitigate biases will hopefully address all three embedding types and all of the three questions we restate below. do joint embeddings encode social biases? see experiment 1, section 4.1. the results presented in can grounded evidence that counters a stereotype alleviate biases? see experiment 2, section 4.2. the results presented in number of statistically significant tests out of 7 total race bias tests these results, with, large effect sizes (@xmath2.42 for vilbert and @xmath3.467 for visualbert with 12% of tests being statistically significant) demonstrating that language contributes more than vision. it could be that the biases in language are so powerful that vision does not contribute to them given that in any one example it appears unable to override the existing biases (experiment 2). it is encouraging that models do consider vision, but the differing biases in vision and text do not appear to help. ## discussion visually grounded embeddings have biases similar to ungrounded embeddings and vision does not appear to help eliminate them. at test time, vision has difficulty overcoming biases, even when presented counter-stereotypical evidence. this is worrisome for deployed systems that use such embeddings, as it indicates that they ignore visual evidence that a bias does not hold for a particular interaction. overall, language and vision each contribute to encoded bias, yet the means of using vision to mitigate is not immediately clear. we enumerated the combinations of inputs possible in the grounded setting and selected three interpretable questions that we answered above. other questions could potentially be asked using the dataset we developed, although we did not find any others that were intuitive or non-redundant. while we discuss joint vision and language embeddings, the methods introduced here apply to any grounded embeddings, such as joint audio and language embeddings @xcite . measuring bias in such data would require collecting a new dataset, but could use our metrics, grounded-weat and grounded-seat, to answer the same three questions. many joint models are transferred to a new dataset without fine-tuning. we demonstrate that going out-of-domain into a new dataset amplifies biases. this need not be so: out-of-domain models have worse performance which might result in fewer biases. we did not test task-specific finetuned models, but intend to do so in the future. humans clearly have biases, not just machines. although, initial evidence indicates that when faced with examples that go against prejudices, i.e., counter-stereotyping, there is a significant reduction in human biases @xcite . straightforward applications of this idea are far from trivial, as @xcite show that merely balancing a dataset by a certain attribute is not enough to eliminate bias. perhaps artificially manipulating visual datasets can debias shared embeddings. we hope that these datasets and metrics will lead to understanding human biases in grounded settings as well as the development of new methods to debias representations.