Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020)

Abstract: We survey 146 papers analyzing “bias” in NLP systems, finding that their motivations are often vague, inconsistent, and lacking in normative reasoning, despite the fact that analyzing “bias” is an inherently normative process. We further find that these papers’ proposed quantitative techniques for measuring or mitigating “bias” are poorly matched to their motivations and do not engage with the relevant literature outside of NLP. Based on these findings, we describe the beginnings of a path forward by proposing three recommendations that should guide work analyzing “bias” in NLP systems. These recommendations rest on a greater recognition of the relationships between language and social hierarchies, encouraging researchers and practitioners to articulate their conceptualizations of “bias”—i.e., what kinds of system behaviors are harmful, in what ways, to whom, and why, as well as the normative reasoning underlying these statements—and to center work around the lived experiences of members of communities affected by NLP systems, while interrogating and reimagining the power relations between technologists and such communities.

## introduction a large body of work analyzing "bias" in natural language processing (nlp) systems has emerged in recent years, including work on "bias" in embedding spaces (e.g., @xcite @xcite as well as work on "bias" in systems developed for a breadth of tasks including language modeling @xcite , coreference resolution @xcite , machine translation ( @xcite , sentiment analysis @xcite , and hate speech/toxicity detection (e.g., @xcite , among others. although these papers have laid vital groundwork by illustrating some of the ways that nlp systems can be harmful, the majority of them fail to engage critically with what constitutes "bias" in the frst place. despite the fact that analyzing "bias" is an inherently normative process-in which some system behaviors are deemed good and others harmful-papers on "bias" in nlp systems are rife with unstated assumptions about what kinds of system behaviors are harmful, in what ways, to whom, and why. indeed, the term "bias" (or "gender bias" or "racial bias") is used to describe a wide range of system behaviors, even though they may be harmful in different ways, to different groups, or for different reasons. even papers analyzing "bias" in nlp systems developed for the same task often conceptualize it differently. for example, the following system behaviors are all understood to be self-evident statements of "racial bias": (a) embedding spaces in which embeddings for names associated with african americans are closer (compared to names associated with european americans) to unpleasant words than pleasant words @xcite ; (b) sentiment analysis systems yielding different intensity scores for sentences containing names associated with african americans and sentences containing names associated with european americans @xcite ; and (c) toxicity detection systems scoring tweets containing features associated with african-american english as more offensive than tweets without these features @xcite . moreover, some of these papers focus on "racial bias" expressed in written text, while others focus on "racial bias" against authors. this use of imprecise terminology obscures these important differences. we survey 146 papers analyzing "bias" in nlp systems, fnding that their motivations are often vague and inconsistent. many lack any normative reasoning for why the system behaviors that are described as "bias" are harmful, in what ways, and to whom. moreover, the vast majority of these papers do not engage with the relevant literature outside of nlp to ground normative concerns when proposing quantitative techniques for measuring or mitigating "bias." as a result, we fnd that many of these techniques are poorly matched to their motivations, and are not comparable to one another. we then describe the beginnings of a path forward by proposing three recommendations that should guide work analyzing "bias" in nlp systems. we argue that such work should examine the relationships between language and social hierarchies; we call on researchers and practitioners conducting such work to articulate their conceptualizations of "bias" in order to enable conversations about what kinds of system behaviors are harmful, in what ways, to whom, and why; and we recommend deeper engagements between technologists and communities affected by nlp systems. we also provide several concrete research questions that are implied by each of our recommendations. ## method our survey includes all papers known to us analyzing "bias" in nlp systems-146 papers in total. we omitted papers about speech, restricting our survey to papers about written text only. to identify the 146 papers, we frst searched the acl anthology 1 for all papers with the keywords "bias" or "fairness" that were made available prior to may 2020. we retained all papers about social "bias," and discarded all papers about other defnitions of the keywords (e.g., hypothesis-only bias, inductive bias, media bias). we also discarded all papers using "bias" in nlp systems to measure social "bias" in text or the real world (e.g., @xcite . to ensure that we did not exclude any relevant papers without the keywords "bias" or "fairness," we also traversed the citation graph of our initial set of papers, retaining any papers analyzing "bias" in nlp systems that are cited by or cite the papers in our initial set. finally, we manually inspected any papers analyzing "bias" in nlp systems from leading machine learning, human-computer interaction, and web conferences and workshops, such as icml, neurips, aies, facct, chi, and www, along with any relevant papers that were made available in the "computation and language" and "computers and society" categories on arxiv prior to may 2020, but found that they had already been identifed via our traversal of the citation graph. we provide a list of all 146 papers in the appendix. in once identifed, we then read each of the 146 papers with the goal of categorizing their motivations and their proposed quantitative techniques for measuring or mitigating "bias." we used a previously developed taxonomy of harms for this categorization, which differentiates between so-called allocational and representational harms @xcite . allocational harms arise when an automated system allocates resources (e.g., credit) or opportunities (e.g., jobs) unfairly to different social groups; representational harms arise when a system (e.g., a search engine) represents some social groups in a less favorable light than others, demeans them, or fails to recognize their existence altogether. adapting and extending this taxonomy, we categorized the 146 papers' motivations and techniques into the following categories: in ## findings categorizing the 146 papers' motivations and proposed quantitative techniques for measuring or mitigating "bias" into the six categories listed above enabled us to identify several commonalities, which we present below, along with illustrative quotes. 2 we grouped several types of representational harms into two categories to refect that the main point of differentiation between the 146 papers' motivations and proposed quantitative techniques for measuring or mitigating "bias" is whether or not they focus on stereotyping. among the papers that do not focus on stereotyping, we found that most lack suffciently clear motivations and techniques to reliably categorize them further. ## a path forward we now describe how researchers and practitioners conducting work analyzing "bias" in nlp systems might avoid the pitfalls presented in the previous section-the beginnings of a path forward. we propose three recommendations that should guide such work, and, for each, provide several concrete research questions. we emphasize that these questions are not comprehensive, and are intended to generate further questions and lines of engagement. our three recommendations are as follows: ## case study to illustrate our recommendations, we present a case study covering work on african-american english (aae). foot_2 work analyzing "bias" in the context of aae has shown that part-of-speech taggers, language identifcation systems, and dependency parsers all work less well on text containing features associated with aae than on text without these features @xcite @xcite , and that toxicity detection systems score tweets containing features associated with aae as more offensive than tweets without them @xcite . these papers have been critical for highlighting aae as a language variety for which existing nlp systems may not work, illustrating their limitations. however, they do not conceptualize "racial bias" in the same way. the frst four of these papers simply focus on system performance differences between text containing features associated with aae and text without these features. in contrast, the last two papers also focus on such system performance differences, but motivate this focus with the following additional reasoning: if tweets containing features associated with aae are scored as more offensive than tweets without these features, then this might (a) yield negative perceptions of aae; (b) result in disproportionate removal of tweets containing these features, impeding participation in online platforms and reducing the space available online in which speakers can use aae freely; and (c) cause aae speakers to incur additional costs if they have to change their language practices to avoid negative perceptions or tweet removal. more importantly, none of these papers engage with the literature on aae, racial hierarchies in the u.s., and raciolinguistic ideologies. by failing to engage with this literature-thereby treating aae simply as one of many non-penn treebank varieties of english or perhaps as another challenging domain-work analyzing "bias" in nlp systems in the context of aae fails to situate these systems in the world. who are the speakers of aae? how are they viewed? we argue that aae as a language variety cannot be separated from its speakersprimarily black people in the u.s., who experience systemic anti-black racism-and the language ideologies that reinforce and justify racial hierarchies. even after decades of sociolinguistic efforts to legitimize aae, it continues to be viewed as "bad" english and its speakers continue to be viewed as linguistically inadequate-a view called the defcit perspective @xcite . this perspective persists despite demonstrations that aae is rule-bound and grammatical @xcite , in addition to ample evidence of its speakers' linguistic adroitness (e.g., @xcite . this perspective belongs to a broader set of raciolinguistic ideologies @xcite , which also produce allocational harms; speakers of aae are frequently penalized for not adhering to dominant language practices, including in the education system @xcite , when seeking housing @xcite , and in the judicial system, where their testimony is misunderstood or, worse yet, disbelieved @xcite . these raciolinguistic ideologies position racialized communities as needing linguistic intervention, such as language education programs, in which these and other harms can be reduced if communities accommodate to dominant language practices @xcite . in the technology industry, speakers of aae are often not considered consumers who matter. for example, @xcite recounts an apple employee who worked on speech recognition for siri: "as they worked on different english dialects -australian, singaporean, and indian english -[the employee] asked his boss: 'what about african american english?' to this his boss responded: 'well, apple products are for the premium market."' the reality, of course, is that speakers of aae tend not to represent the "premium market" precisely because of institutions and policies that help to maintain racial hierarchies by systematically denying them the opportunities to develop wealth that are available to white americans @xcite an exclusion that is reproduced in technology by countless decisions like the one described above. engaging with the literature outlined above situates the system behaviors that are described as "bias," providing a foundation for normative reasoning. researchers and practitioners should be concerned about "racial bias" in toxicity detection systems not only because performance differences impair system performance, but because they reproduce longstanding injustices of stigmatization and disenfranchisement for speakers of aae. in re-stigmatizing aae, they reproduce language ideologies in which aae is viewed as ungrammatical, uneducated, and offensive. these ideologies, in turn, enable linguistic discrimination and justify enduring racial hierarchies @xcite . our perspective, which understands racial hierarchies and raciolinguistic ideologies as structural conditions that govern the development and deployment of technology, implies that techniques for measuring or mitigating "bias" in nlp systems will necessarily be incomplete unless they interrogate and dismantle these structural conditions, including the power relations between technologists and racialized communities. we emphasize that engaging with the literature on aae, racial hierarchies in the u.s., and raciolinguistic ideologies can generate new lines of engagement. these lines include work on the ways that the decisions made during the development and deployment of nlp systems produce stigmatization and disenfranchisement, and work on aae use in practice, such as the ways that speakers of aae interact with nlp systems that were not designed for them. this literature can also help researchers and practitioners address the allocational harms that may be produced by nlp systems, and ensure that even well-intentioned nlp systems do not position racialized communities as needing linguistic intervention or accommodation to dominant language practices. finally, researchers and practitioners wishing to design better systems can also draw on a growing body of work on anti-racist language pedagogy that challenges the defcit perspective of aae and other racialized language practices (e.g. @xcite @xcite , as well as the work that we described in section 4.3 on reimagining the power relations between technologists and communities affected by technology. ## conclusion by surveying 146 papers analyzing "bias" in nlp systems, we found that (a) their motivations are often vague, inconsistent, and lacking in normative reasoning; and (b) their proposed quantitative techniques for measuring or mitigating "bias" are poorly matched to their motivations and do not engage with the relevant literature outside of nlp. to help researchers and practitioners avoid these pitfalls, we proposed three recommendations that should guide work analyzing "bias" in nlp systems, and, for each, provided several concrete research questions. these recommendations rest on a greater recognition of the relationships between language and social hierarchies-a step that we see as paramount to establishing a path forward.