Social Biases in NLP Models as Barriers for Persons with Disabilities (2020)

Abstract: Building equitable and inclusive NLP technologies demands consideration of whether and how social attitudes are represented in ML models. In particular, representations encoded in models often inadvertently perpetuate undesirable social biases from the data on which they are trained. In this paper, we present evidence of such undesirable biases towards mentions of disability in two different English language models: toxicity prediction and sentiment analysis. Next, we demonstrate that the neural embeddings that are the critical first step in most NLP pipelines similarly contain undesirable biases towards mentions of disability. We end by highlighting topical biases in the discourse about disability which may contribute to the observed model biases; for instance, gun violence, homelessness, and drug addiction are over-represented in texts discussing mental illness.

## introduction this paper focuses on the representation of persons with disabilities through the lens of technology. specifically, we examine how nlp models classify or predict text relating to persons with disabilities (see while previous studies have studied unintended biases in nlp models against other historically marginalized groups @xcite @xcite @xcite @xcite @xcite @xcite , bias with respect to different disability groups has been relatively under-explored. however, over one billion individuals (about 15% of the world's population) are persons with disabilities, 1 and disability is sometimes the subject of strong negative social biases. for example, a 2007 study found implicit and explicit preferences against people with disabilities compared to people without disabilities across the social group domains @xcite . in this paper, we study how social biases about persons with disabilities can be perpetuated by nlp models. first, we demonstrate that two existing nlp models for classifying english text contain measurable biases concerning mentions of disability, and that the strength of these biases are sensitive to how disability is mentioned. second, we show that language models that feed nlp systems for downstream application similarly contain measur-able biases around disability. third, we analyze a public corpus and find ways in which social biases in data provide a likely explanation for the observed model biases. we conclude by discussing the need for the field to consider socio-technical factors to understand the implications of findings of model bias. ## linguistic phrases for disabilities our analyses in this paper use a set of 56 linguistic expressions (in english) for referring to people with various types of disabilities, e.g. a deaf person. we partition these expressions as either recommended or non-recommended, according to their prescriptive status, by consulting guidelines published by three us-based organizations: anti-defamation league, acm sigaccess and the ada national network @xcite @xcite . we acknowledge that the binary distinction between recommended and non-recommended is only the coarsest-grained view of complex and multi-dimensional social norms, however more input from impacted communities is required before attempting more sophisticated distinctions @xcite . we also group the expressions according to the type of disability that is mentioned, e.g. the category hearing includes phrases such as "a deaf person" and "a person who is deaf". ## biases in text classification models following @xcite , we use the notion of perturbation, whereby the phrases for referring to people with disabilities, described above, are all inserted into the same slots in sentence templates. we start by first retrieving a set of naturally-occurring sentences that contain the pronouns he or she. 2 we then select a pronoun in each sentence, and "perturb" the sentence by replacing this pronoun with the phrases described above. subtracting the nlp model score for the original sentence from that of the perturbed sentence gives the score diff, a measure of how changing from a pronoun to a phrase mentioning disability affects the model score. we perform this method on a set of 1000 sentences extracted at random from the reddit sub ## biases in language representations neural text embedding models @xcite are critical first steps in today's nlp pipelines. these models learn vector representations of words, phrases, or sentences, such that semantic relationships between words are encoded in the geometric relationship between vectors. text embedding models capture some of the complexities and nuances of human language. however, these models may also encode undesirable correlations in the data that reflect harmful social biases @xcite @xcite . previous studies have predominantly focused on biases related to race and gender, with the exception of @xcite , who considered physical and mental illness. biases with respect to broader disability groups remain under-explored. in this section, we analyze how the widely used bidirectional transformer (bert) @xcite 3 model represents phrases mentioning persons with disabilities. following prior work @xcite studying social biases in bert, we adopt a templatebased fill-in-the-blank analysis. given a query sentence with a missing word, bert predicts a ranked list of words to fill in the blank. we construct a set of simple hand-crafted templates '<@xmath0.', @xmath1. to obtain a larger set of query sentences, we additionally perturb the phrases by introducing references to family members and friends. for example, in addition to 'a person', we include 'my sibling', 'my parent', 'my friend', etc. we then study how the top ranked 4 words predicted by bert change when different disability phrases are used in the query sentence. in order to assess the valency differences of the resulting set of completed sentences for each phrase, we use the google cloud sentiment model @xcite . for each bert-predicted word w, we obtain the sentiment for the sentence 'a person @xmath2. we use the neutral a person instead of the original phrase, so that we are assessing only the differences in sentiment scores for the words predicted by bert and not the biases associated 3 we use the 1024-dimensional 'large' uncased version, available at https://github.com/google-research/ . ## biases in data nlp models such as the ones discussed above are trained on large textual corpora, which are analyzed to build "meaning" representations for words based on word co-occurrence metrics, drawing on the idea that "you shall know a word by the company it keeps" @xcite . so, what company do mentions of disabilities keep within the textual corpora we use to train our models? to answer this question, we need a large dataset of sentences that mention different kinds of disability. we use the dataset of online comments released as part of the jigsaw unintended bias in toxicity classification challenge @xcite , where a subset of 405k comments are labelled for mentions of disabilities, grouped into four types: physical disability, intellectual or learning disability, psychiatric or mental illness, and other disability. we focus here only on psychiatric or mental illness, since others have fewer than 100 instances in the dataset. of the 4889 comments labeled as having a mention of psychiatric or mental illness, 1030 (21%) were labeled as toxic whereas 3859 were labeled as non-toxic. 5 our goal is to find words and phrases that are statistically more likely to appear in comments that mention psychiatric or mental illness compared to those that do not. we first up-sampled the toxic comments with disability mentions (to @xmath3, by repetition at random), so that we have equal number of toxic vs. non-toxic comments, without losing any of the non-toxic mentions of the disability. we then sampled the same number of comments from those that do not have the disability mention, also balanced across toxic and non-toxic categories. in total, this gave us 15436 (=4*3859) comments. using this 4-way balanced dataset, we calculated the log-odds ratio metric @xcite for all unigrams and bi-grams (no stopword removal) that measure how over-represented they are in the group of comments that have a disability mention, while controlling for co-occurrences due to chance. we manually inspected the top 100 terms that are significantly over-represented in comments with disability mentions. most of them fall into one of the following five categories: foot_2 ## implications of model biases we have so far worked in a purely technical framing of model biases-i.e., in terms of model inputs and outputs-as is common in much of the technical ml literature on fairness @xcite . however, normative and social justifications should be considered when applying a statistical definition of fairness @xcite . further, responsible deployment of nlp systems should also include the socio-technical considerations for various stakeholders impacted by the deployment, both directly and indirectly, as well as voluntarily and involuntarily @xcite , accounting for long-term impacts @xcite and feedback loops @xcite @xcite . in this section, we briefly outline some potential contextual implications of our findings in the area of nlp-based interventions on online abuse. following @xcite and cao and daumé iii (2020), we use three hypothetical scenarios to illustrate some key implications. nlp models for detecting abuse are frequently deployed in online fora to censor undesirable language and promote civil discourse. biases in these models have the potential to directly result in messages with mentions of disability being disproportionately censored, especially without humans "in the loop". since people with disabilities are also more likely to talk about disability, this could impact their opportunity to participate equally in online fora @xcite , reducing their autonomy and dignity. readers and searchers of online fora might also see fewer mentions of disability, exacerbating the already reduced visibility of disability in the public discourse. this can impact public awareness of the prevalence of disability, which in turn influences societal attitudes (for a survey, see @xcite . in a deployment context that involves human moderation, model scores may sometimes be used to select and prioritize messages for review by moderators @xcite . are messages with higher model scores reviewed first? or those with lower scores? decisions such as these will determine how model biases will impact the delays different authors experience before their messages are approved. in another deployment context, models for detecting abuse can be used to nudge writers to rethink comments which might be interpreted as toxic @xcite . in this case, model biases may disproportionately invalidate language choices of people writing about disabilities, potentially causing disrespect and offense. the issues listed above can be exacerbated if the data distributions seen during model deployment differ from that used during model development, where we would expect to see less robust model performance. due to the complex situational nature of these issues, release of nlp models should be accompanied by information about intended and non-intended uses, about training data, and about known model biases @xcite . ## discussion and conclusion social biases in nlp models are deserving of concern, due to their ability to moderate how people engage with technology and to perpetuate negative stereotypes. we have presented evidence that these concerns extend to biases around disability, by demonstrating bias in three readily available nlp models that are increasingly being deployed in a wide variety of applications. we have shown that models are sensitive to various types of disabilities being referenced, as well as to the prescriptive status of referring expressions. it is important to recognize that social norms around language are contextual and differ across groups @xcite @xcite . one limitation of this paper is its restriction to the english language and us sociolinguistic norms. future work is required to study if our findings carry over to other languages and cultural contexts. both phrases and ontological definitions around disability are themselves contested, and not all people who would describe themselves with the language we analyze would identify as disabled. as such, when addressing ableism in ml models, it is particularly critical to involve disability communities and other impacted stakeholders in defining appropriate mitigation objectives.