Unpacking the Interdependent Systems of Discrimination: Ableist Bias in NLP Systems through an Intersectional Lens (2021)

Abstract: Much of the world’s population experiences some form of disability during their lifetime. Caution must be exercised while designing natural language processing (NLP) systems to prevent systems from inadvertently perpetuating ableist bias against people with disabilities, i.e., prejudice that favors those with typical abilities. We report on various analyses based on word predictions of a large-scale BERT language model. Statistically significant results demonstrate that people with disabilities can be disadvantaged. Findings also explore overlapping forms of discrimination related to interconnected gender and race identities.

## background and related work there is a growing body of sociology literature that examines bias against people with disabilities and its relationship with cultural and socio-political aspects of societies @xcite . sociological research has also moved from drawing analogies between ableism and racism to examining their intersectionality @xcite . disability rights movements have stimulated research exploring the gendered marginalization and empowerment of people with disabilities. the field of computing is still lagging behind. work on identifying and measuring ethical issues in nlp systems has only recently turned to ableist bias-largely without an intersectional lens. while ableist bias differs, prior findings on other bias motivate inves-tigation of these issues for people with disabilities @xcite . there is a need for more work that deeply examines how bias against people with disabilities manifest in nlp systems through approaches such as critical disability theory @xcite . however, a growing body of research on ethical challenges in nlp reveals how bias against protected groups permeate nlp systems. to better understand how to study bias in nlp, we focus on prior work in three categories: (1) observing bias using psychological tests, (2) analyzing biased subspaces in text representations such as word embeddings, and (3) comparing performance differences of nlp systems across various protected groups. research has sought to quantify bias in nlp systems using psychological tests, such as the implicit association test (iat) @xcite , which can reveal influential subconscious associations or implicit beliefs about people of a protected group and their stereotypical roles in societies. some work has studied correlations between data on gender and professions and the strengths of these conceptual linkages in word embeddings @xcite . findings suggest that word embeddings encode normative assumptions, or resistance to social change, which can have implications for computational systems. analyzing subspaces in text representations like word embeddings can reveal insights about nlp systems that use them @xcite . for example, @xcite developed a support vector machine to identify gender subspace in word embeddings and then identified gender directions by making "gender-pairs (man-woman, his-her, shehe)". they identified eigenvectors that capture prominent variance in the data. this work has been extended to include non-binary gender distinctions @xcite . researchers have also explored contextualized word embeddings bias at the intersection of race and gender. @xcite proposed methods for automatically identifying intersectional bias in static word embeddings. but debiasing has limitations. for example, @xcite pointed out that even after attempting to reduce the projection of words on a gender direction, biased/stereotypical words in the neighbors of a given word embedding remain @xcite . other work has measured performance bias of nlp systems when used by someone from a protected group or when the input data mentions a protected group. unfortunately, state-of-the-art systems pass on bias to other tasks. for example, a recent study found that bert can perpetuate gender bias in contextualized word embeddings @xcite . some work has explored the effect on performance measures in nlp systems after replacing (swapping) majority-minority lexicons @xcite @xcite . additionally, standard evaluation metrics usually fail to take bias into account, nor are datasets carefully designed to reveal bias effects. researchers have explored the utility of performance metrics for capturing differences due to bias and proposed new metrics @xcite . a recent systematic review raised this concern and pointed to datasets that probe gender bias @xcite . there is a pressing need to develop metrics, evaluation processes, and datasets able to quantitatively assess ableist biases in nlp systems. as a first step, we critically assess how ableist biases manifest in nlp models and examine intersections of bias. ## methods we build on the work of @xcite which used a fill-in-the-blank analysis-originally proposed by @xcite -to study ableist bias in pre-trained bert representations. we used bert large model (uncased), a pretrained english language model @xcite . we adjusted their analysis method to examine ableist bias together with gender and racial bias. our analysis method involves creating template sentence fragments of the form the [blank1] [blank2] [blank3] person [connecting verb] <predicted using bert>. the slots (blank1, blank2, blank3) were filled in based on three lists with referents related to disability, gender, and race. the disability list was provided by @xcite . 1 the list for race included the five categories in the u.s. census (census, 2021) foot_1 , and the list for gender was based on guidelines for gender inclusiveness in writing @xcite the referents used are in analysis was restricted to the 5 sets of sentences in following @xcite and @xcite , bert was trained to predict the masked word. each sentence fragment was input ten times, resulting in 10 predicted words (without replacement) per stimulus. given the added number of referents and connecting words, a three step filtering process was performed where bert context. this sometimes resulted in fewer than 10 words for stimuli. in our final set of results, 83,268 out of 211,820 (21,182 sentences times 10 predicted words) remained. the dataset of sentences has been made available for research. 3 each predicted word not filtered out was added in a carrier sentence template the person [connecting verb] <bert predicted @xmath0. the average sentiment score for each of the five combinations of sets of referents to disability, gender, race or no referent ( ## conclusion and future work our findings reveal ableist biases in an influential nlp model, indicating it has learned undesirable associations between mentions of disability and negative valence. this supports the need to develop metrics, tests, and datasets to help uncover ableist bias in nlp models. the intersectionality of disability, gender, and race deserves further study. this work's limitations are avenues for future research. we only studied the intersections of disability, gender, and race. we did not explore race and gender, or their combination, without disability. studies can also look at other sources of bias such as ageism and expand the connecting verbs. our sentiment analysis was also limited to template carrier sentences with one word predicted by bert. future work can allow bert or other language models to predict multiple words and analyze the findings. we focused on a small number of manually selected verbs while comparing averaged sentiment. future work could investigate a greater variety of verbs, and it could analyze more specifically how particular combinations of identity characteristics and verbs may reveal forms of social bias. for our analysis, we primarily used an averaged sentiment score. future research can consider using other approaches to examine bias as well. finally, future work can modify or improve different state-of-the-art debiasing approaches to remove intersectional ableist bias in nlp systems.