Unequal Representations: Analyzing Intersectional Biases in Word Embeddings Using Representational Similarity Analysis (2020)

Abstract: We present a new approach for detecting human-like social biases in word embeddings using representational similarity analysis. Specifically, we probe contextualized and non-contextualized embeddings for evidence of intersectional biases against Black women. We show that these embeddings represent Black women as simultaneously less feminine than White women, and less Black than Black men. This finding aligns with intersectionality theory, which argues that multiple identity categories (such as race or sex) layer on top of each other in order to create unique modes of discrimination that are not shared by any individual category.

## introduction word embeddings are ubiquitous components of modern natural language processing systems, and have enabled performance increases on a wide variety of nlp tasks @xcite @xcite . they provide vector representations of words, and are trained on a large corpus of text in order to capture the distributional semantics of lexical items. many studies have shown that these embeddings also encode a variety of dangerous social biases, as these biases are present in naturally-occurring english text (see @xcite for a survey of such work). the present work studies intersectional biases in both contextualized and non-contextualized word embeddings. the theory of intersectionality states that multiple different aspects of a person's identity often combine to create unique modes of discrimination @xcite . for example, specific biases against black women are not necessarily shared by either white women or black men. furthermore, discourses on discrimination and public health typically focus on sexism or racism, each implicitly excluding the other @xcite . it is further argued that these discourses are focused on the most privileged members of a group @xcite . thus, we might expect that the ostensibly race-neutral category 'woman' is implicitly white, and that the ostensibly non-gendered category 'black' is implicitly male. using representational similarity analysis (rsa; @xcite ), we find that both non-contextualized glove embeddings @xcite and contextualized bert embeddings @xcite display exactly these biases. we also find that the bert embeddings of the names of black women are semantically impoverished, compared to the embeddings of the names of white women. related work: many recent studies have sought to understand social biases in nlp systems. notably, @xcite showed that the human biases revealed through the implicit association test @xcite are also present in word embeddings, prompting other studies that seek to detect bias in word embeddings @xcite . other studies have attempted to detect bias in other nlp systems @xcite , to automatically discover and measure intersectional biases in word embeddings @xcite , to analyze the distributions of gendered words in training sets @xcite , and to automatically mitigate the effects of social biases on nlp systems @xcite . another study has produced empirical evidence for the claims of intersectionality theory by leveraging distributional representations @xcite . perhaps most relevant, some prior work has attempted to understand whether intersectional differences are present in contextualized word embeddings by trying to detect whether these embeddings encoded intersectional stereotypes @xcite . however, detecting these stereotypes is only a proxy for detecting intersectional biases in embeddings: an embedding might encode intersectional differences without encoding more complex interesectional stereotypes. this work presents a method that leverages representational similarity analysis to directly probe word embeddings for intersectional biases. foot_0 ## methods: representational similarity analysis representational similarity analysis (rsa) is a technique first used in cognitive neuroscience for analyzing distributed activity patterns in the brain @xcite , but recent work has used it to analyze artificial neural systems @xcite @xcite @xcite . rsa analyzes the representational geometry of a system, which is defined by the pairwise dissimilarities between representations of a set of stimuli. specifically, we use rsa to compare the representational geometry of the word embeddings under study to the representational geometries of well-understood hypothesis models. if the representational geometries are similar, you can infer that the word embeddings represent the particular information expressed by a hypothesis model. applying rsa: the approach proceeds as follows: first, we create a corpus that consists of three types of items: group 1 items, g 1 , group 2 items, g 2 , and concept items, c. next, we define a reference model m ref , which consists of the set of embeddings that we are interested in. then, we define two hypothesis models, m hyp 1 and m hyp 2 . m hyp 1 instantiates the hypothesis that: 'group 1 is associated with the concept under study, while group 2 is not'. likewise m hyp 2 instantiates the hypothesis that 'group 2 is associated with the concept under study, while group 1 is not'. we then draw a sample c, consisting of n 1 items from g 1 , n 2 items from g 2 , and n 3 items from c, and calculate the (n we define @xmath0. for m hyp 1 , the dissimilarity metric is equal to 1 if one item is from g 2 and the other item is from g 1 or c, and 0 otherwise. likewise, the dissimilarity metric for m hyp 2 is equal to 1 is one item is from g 1 and the other item is from g 2 or c, and 0 otherwise. clearly, these hypothesis models are not expected to provide a very good fit to the representational geometry of a set of word embeddings. however, if one hypothesis model exhibits a significantly better fit than the other, then there is evidence for a problematic bias. finally, we calculate the similarity, s, between the representational geometries of our hypothesis models and our reference model using a similarity metric, sim. because the r matrices are symmetric, sim only operates on the upper triangle of each r matrix. we define @xmath1. we then repeat the process on 100 samples from our corpus in order to create two 100-length vectors of representational similarities, s hyp 1 and s hyp 2 . we can then apply a nonparametric sign test to the difference of these vectors, s hyp 1 -s hyp 2 , in order to test for a consistent difference between measurements of s hyp 1 and s hyp 2 . see figure 1 for a visual summary of this approach. ## data in order to perform the analyses described in section 2, we need to create sets of group items for all of the groups under study, as well as concept items for the identity concepts under study. for analyses involving glove embeddings, these glossaries can only contain single-word entries. for analyses involving bert embeddings, these entries can be full sentences. see appendix a for more details about all of the following datasets. ## limitations one notable limitation of this work is that we study artificial binary gender and race distinctions. unfortunately, these binary distinctions exclude many individuals from our analysis. future work should expand upon the methodological framework presented here in order to assess biases against groups that are not represented by these distinctions. furthermore, @xcite describes several limitations of relying on name data to represent racial categories. perhaps most importantly, that work demonstrates that 'black names' are not one monolithic category, that perceptions of the blackness of names are correlated with other confounding variables (such as education status), and that the associations between identity categories and proper names are only loosely reflected in real-world naming practices. furthermore, we rely on an incomplete set of country names provided by the census bureau to generate concept sets for the black identity category. we acknowledge that this excludes many nationalities from our analysis, and also acknowledge that any attempt to exhaustively list nations associated with the black identity category is destined to fail. we hope that our results inspire future work that aspires to more robust coverage of the black identity group. finally, it is important to note that this method can only provide positive evidence: if this method does not detect intersectional biases for a particular system, that does not provide evidence that the system does not contain such biases. ## conclusion both contextualized and non-contextualized word embeddings learn to represent human-like biases. we apply rsa to the task of detecting these biases, and find that both glove embeddings and bert embeddings reproduce (and thus perpetuate) the dual marginalization of black women within both the black community and female community, as predicted by the theory of intersectionality. specifically, we showed that word embeddings represent the female identity category as implicitly white, and the black identity category as implicitly masculine. to our knowledge, this is the first method that can probe word embeddings for these intersectional biases directly. aside from addressing the issues discussed in section 5, future work should investigate whether this method can be used to detect other sorts of intersectional biases, particularly those facing the lgbtq community.