Interdependencies of Gender and Race in Contextualized Word Embeddings (2020)

Abstract: Recent years have seen a surge in research on the biases in word embeddings with respect to gender and, to a lesser extent, race. Few of these studies, however, have given attention to the critical intersection of race and gender. In this case study, we analyze the dimensions of gender and race in contextualized word embeddings of given names, taken from BERT, and investigate the nature and nuance of their interaction. We find that these demographic axes, though typically treated as physically and conceptually separate, are in fact interdependent and thus inadvisable to consider in isolation. Further, we show that demographic dimensions predicated on default settings in language, such as in pronouns, may risk rendering groups with multiple marginalized identities invisible. We conclude by discussing the importance and implications of intersectionality for future studies on bias and debiasing in NLP.

## introduction in recent years, the rapid growth of natural language processing (nlp) has been accompanied by increasing attention to biases in nlp systems, with several studies investigating the demographic biases inherited by word embeddings from human text corpora. the vast majority of these studies have focused on measuring or mitigating bias with respect to gender, while fewer have concentrated on stereotypes with respect to race @xcite . even fewer of these studies have considered the intersection of gender and race. intersectionality, however, is a framework as critical as ever. coined by black feminist scholar kimberlé crenshaw in 1989, intersectionality has since gained momentum as a vital framework to highlight and understand the powerful ways that different dimensions of an individual's identity combine and interact to create unique experiences of discrimination; crenshaw elucidates the intersectional experience associated with race and sex as greater than the sum of racism and sexism, such that "any analysis that does not take intersectionality into account cannot sufficiently address the particular manner in which black women are subordinated" @xcite . one consequence illuminated by this framework is that, because "people with multiple subordinategroup identities (e.g., ethnic minority women) do not fit the prototypes of their respective identity groups (e.g., ethnic minorities, women)," they and their unique experiences are often erased from the conversation -they experience 'intersectional invisibility' @xcite . in the field of psychology, due to systematic underrepresentation of certain groups in participant samples, "much of what is known about women in psychology is based on responses from women who are white and often middle class" @xcite . at an institutional level, "law enforcement, the government, and research institutions measure 'gender' as 'white women' and 'race' as 'african-american men'," and because of the prevalence of this "dualistic" approach in the social sciences, african-american women receive weaker attention in studies focusing solely on race or gender @xcite . we find this gap to be true of the social and linguistic properties characterizing contextual word embeddings as well. in this case study, we investigate the concepts of intersectionality inherent in the relationships and geometries of contextualized word embeddings. specifically, this paper: ## related work the vast majority of studies of bias in word embeddings and contextual representations have focused on binary gender in isolation @xcite @xcite @xcite . this is likely due to the advantage that gender pronouns are marked in language in a visible way that other demographic attributes are not. 'bias', in these studies, is often measured as the similarity of words that are by definition gender-neutral -typically profession words -to words that are explicitly male or female gendered, such as pronouns. one popular method proposed for mitigating bias in word embeddings involves reducing the projection of gender-neutral words in the direction of a gender dimension vector defined by the difference between the vectors of 'he' and 'she' @xcite . another measure of bias in word embeddings, the word embedding association test (weat), was introduced by @xcite , adapted from the implicit association test (iat) of humans' reaction times to pairs of words @xcite . more recently, studies have begun to factor intersectionality into their analyses of bias. studying bias in sentence encoders, @xcite generalized the weat to sentence embeddings, using bleached sentence templates to test a number of hypotheses. though several of their hypotheses are centered on gender, they also include a test of the "angry black woman" stereotype and find significant evidence confirming its presence @xcite . tan and celis (2019) similarly build on the weat, extending it to contextualized word embeddings and measuring intersectional bias as the association of european american and african american male figure 1: african american (aa) and european american (ea) male and female names projected onto their first and second principal components (aa female names in red, ea female names in yellow, aa male names in green, ea male names in blue) and female names with pleasant and unpleasant words; they find significant evidence of bias, noting that the highest measured difference in bias is that between the names from the group with multiple privileged identities -european american men -against the names from the group with multiple marginalized identities -african american women. further, @xcite propose a method for identifying words that are unique to an intersectional group, and find numerous words pertaining to negative stereotypes associated significantly more strongly with an intersectional identity than with its constituent identities @xcite . to our knowledge, however, to date there has not been any systematic study or exploration of the conceptualization of gender and race learned by word embeddings, how these demographic dimensions relate or interact, or how bias may be embedded in gendered words such as pronouns themselves; the objective of this paper is to begin to fill in these gaps. ## learning gender and race in word embeddings from names to understand how and to what extent the concepts of gender and race are learned by contextualized word embeddings, in this case study we use contextualized embeddings of african american and european american male and female names from the pretrained bert base-cased model @xcite , with the sets of popular american given names borrowed from previous studies (tan and celis, 2019; @xcite . these sets consist of 13 names for each race-gender pair, and were selected from greenwald et al. (1998)'s original set of names used in introducing the iat @xcite . we base our analysis on the principal component analysis (pca) of these names, a common approach to identifying, visualizing, and understanding dimensions of gender and race in a word embedding vector space @xcite @xcite . figure 1 plots these names projected onto the first and second principal components of the pca of all the names in the four sets. on inspection, the first principal component decidedly distinguishes the race of the names; all of the european american names have negative projections onto the component and all african american names, with the exception of a few male names for which racial association may have been more ambiguous, have positive projections. the second principal component appears to distinguish gender, cleanly separating the european american female names from the european american male names along the axis. however, though the pca was performed on an equal number of name embeddings from each group, this gender component is significantly less successful in capturing the gender of african american names. african american female names, and male names to a lesser extent, have projections on this gender dimension close to zero, and there is substantial overlap in that region between the male and female names in a way that is in visible contrast to that of the european american names. figure 2a plots the projections of the contextualized embeddings for the european american and african american female names on the first and second principal components of the pca performed on names solely from those two sets, and the subsequent subfigures likewise plot the projections for european american and african american male names, african american male and female names, european american male and female names, african american male names and european american female names, and european american male names and african american female names. through the results of pca on these subsets of the names, a few key themes emerge. first, it is immediately apparent that the concepts of gender and race are not learned uniformly across demographic groups; the juxtaposition of figure 2c with figure 2d illuminates this clearly, as the first principal component of the european american names linearly separates them by gender while neither the first nor the second principal component of the african american names results in a clean separation. to treat gender as a concept independent of race, then, risks the erasure of african americans. further, we find that these concepts are not learned uniformly even within a single demographic group. that a small set of african american male names associate more closely with european american names as in figure 2b is revealing of the heterogeneity of race, but even among the african american names that project onto the same direction of the first principal component -the race dimension -there is much more dispersion along that axis. it is worth noting that even given an equal number of names of each gender-racial group, the first principal components of the names' contextualized embeddings -the components that explain the most variance in the embeddings -are dimensions of race and gender that are better adapted to european american names. finally, across these experiments we find that the influence of race dominates that of gender in the embedding subspace. in figure 1 , the first principal component definitively distinguishes race, while the second principal component, more ambiguously for african american female names, distinguishes gender. likewise, in figure 2 , the comparative ease with which the first principal components distinguish african american names from european american names is significant, as african american women's names associate much more closely with african american men's names than with european american women's names. this is striking, particularly in light of sociological studies on the centrality of race in understanding oppression, and the primacy of the role that race plays, even with the influence of factors such as race and class @xcite . moreover, the weakness of the embedding model to recognize the femininity of african american women's names mirrors the biases in society that systematically marginalize african american women. taken together, these themes underscore that the concepts and dimensions of race and gender in the bert embedding space are not independent, and that ignoring their interaction may have harmful consequences for vulnerable populations. ## default settings and intersectional invisibility studies of gender bias in word embeddings typically use either male and female names or pronouns for measurement and debiasing. as we have found, however, the understanding of gender learned by the model and applied to names depends significantly on the race of the names. in this section, we investigate whether pronouns -by definition neutral with respect to race -are, in fact, independent of race. figure 3a shows the pca of the contextualized embeddings of the gender pronouns, visibly separated by gender along the first principal component axis. more illustratively, figure 3b shows the european american and african american male and female names projected onto the first and second principal component dimensions obtained from the pca of the gender pronouns in figure 3a . we find that while european american names and african american men's names align definitively with the directions of their corresponding gendered pronouns, the names of african american women do not project significantly onto that gender dimension in the direction of the female pronouns. since the default setting of race learned by the model is european american, 'she' is taken to mean european american women, and as the default setting of gender learned by the model is male, african american men become race-prototypical. however, because the race and gender dimensions are not independent, and the race dimension dominates the gender dimension such that the african american women's names associate more closely with the african american men's names, african american women -at the intersection of (a) male (blue) and female (red) pronouns projected onto their first and second principal components (b) aa and ea male and female names projected onto the first and second principal components of the gender pronouns from plot (a) above (aa female names in red, ea female names in yellow, aa male names in green, ea male names in blue) figure 3 : projections of pronouns and names on gender dimension from pca of gender pronouns marginalized gender and racial identities -experience intersectional invisibility to the language model as it is unable to associate their names with female pronouns. in sum, we find strong evidence that the concept and biases of race not only are not independent of the language model's understanding of gender as captured by pronouns, but are in fact embedded in that gender dimension itself. ## discussion and conclusion above all, our findings reveal that language models cannot consider race and gender completely in isolation; since the word embedding space mirrors the reality of a society where racial and gender dimensions are intertwined, it is critically important to consider the significant interactive influences of gender and race. to fail to do so may be to fail to fully recognize both the unique experiences of those with intersecting identities and their positions as members of their constituent groups. further, the existing definitions, measures, and methods handling bias may be too narrow to encapsulate the nuances of diverse and multifaceted identities. we find that the human biases inherited by word embeddings are not merely manifested in the association of gendered words with gender-neutral terms relating to people, professions, or pleasantness and unpleasantness -as typically used in the measurement and analysis of bias -but are also able to fundamentally shape the relationships between the language, the labels, and the dimensions of identity themselves, as well as what these dimensions learn or fail to learn about groups of people. this has profound implications for the way that bias is studied and measured, as well as mitigated. while previous studies have shown that the word embeddings for a set of names representing multiple marginalized identities are indeed shaped by biases in a unique and potentially more potent way compared to the embeddings for sets of names pertaining to each of their constituent identities, our finding that the racial and gender dimensions within word embedding subspaces are not independent implies that those most marginalized and vulnerable may be least seen and protected by methods that 'debias' by operating on isolated dimensions of race or gender built on default settings. future work may extend our case study and analysis to specific demographic biases rooted in human studies, to other intersectional identities, considering potential biases with respect to demographic attributes such as religion, nationality, or age, to other words beyond proper names that may also carry biases, such as professions and adjectives, as well as to other embeddings and language models across different languages and cultures. ultimately, the weakness of word embeddings or language models to adequately, fairly, or uniformly learn the concept of gender for different racial groups and the dearth of research on intersecting identities and resulting biases concerning african american women and other multiply marginalized groups reveal an important blind spot in nlp. these manifestations of bias not only highlight the underlying complexities of inequality inherent in these models but also shed light on the parallel nuances of inequality in society and in the lived experiences of those individuals, further validating the intersectional framework as a crucial piece to understanding and unraveling the biases embedded in technology and society.