Applying the Stereotype Content Model to assess disability bias in popular pre-trained NLP models underlying AI -based assistive technologies (2022)

Abstract: Stereotypes are a positive or negative, generalized, and often widely shared belief about the attributes of certain groups of people, such as people with sensory disabilities. If stereotypes manifest in assistive technologies used by deaf or blind people, they can harm the user in a number of ways, especially considering the vulnerable nature of the target population. AI models underlying assistive technologies have been shown to contain biased stereotypes, including racial, gender, and disability biases. We build on this work to present a psychology-based stereotype assessment of the representation of disability, deafness, and blindness in BERT using the Stereotype Content Model. We show that BERT contains disability bias, and that this bias differs along established stereotype dimensions.

## introduction pre-trained natural language processing (nlp) models are becoming more commonly deployed in pipelines for consumer tools, including those that fall under the umbrella of assistive technologies. models such as bert are used in tools that utilize automatic text simplification (ats) for reading assistance @xcite , where complex words get replaced with simpler alternatives. bert is also used in natural language understanding tools such as automatic speech recognition @xcite . in addition to a continuing increase in the use cases and complexity of ai-based assistive technologies, there is also growing interest in using them. @xcite found that the deaf community expressed strong interest in ats-based reading assistance tools. to achieve fair and inclusive experiences for deaf and blind people, it is important to understand how they may be represented by the models underlying the assistive technologies that are designed for them @xcite . if an ai-based consumer tool perpetuates existing biases and stereotypes in society, it can inadvertently cause and reinforce structural stigma, or "societal level conditions, cultural norms, and institutional policies that constrain the opportunities, resources, and well-being of the stigmatized" @xcite . the bias against deafness-or audism-is prevalent in both mainstream society @xcite and in the deaf community @xcite . audism has been linked to discrimination in multiple real-world scenarios, including the job application process @xcite . in @xcite , 100% of highly qualified psychology internship applications that mentioned deafness were rejected, whereas 100% of those that didn't mention deafness were invited for an interview. causing or reinforcing structural stigma can lead to allocational and representational harms @xcite . allocational harms arise if assistive technologies distribute resources or opportunities unfairly to disabled people. with representational harms, if assistive technologies represent these people unfairly, disabled people may experience alienation, decreased quality of service, stereotypes, denigration and stigmatization, erasure, and/or decreased public participation. despite recent ballooning of research in nlp fairness @xcite , there has been little investigation into how ai models represent disabled people, who comprise at least 12.5% of the global population @xcite . there has been even less of a focus on how people with sensory disabilities are represented in nlp models. @xcite provided preliminary evidence that disability-mentioning text may be accidentally flagged as toxic. @xcite detected signs of disability bias in bert using sentiment analysis, and they investigated how this bias might shift when applying an intersectional lens to the analysis. to further investigate sensory disability bias in nlp models, we build upon prior work in association bias in bert. our contributions include adapting @xcite 's sentence templates to examine associations between disability qualifiers and stereotype traits, drawing from the stereotype content model (scm), an established approach in social psychology to defining stereotyped bias @xcite . specifically, we answer these research questions: ## related work we review previous work in examining stereotypes in nlp models, and then we briefly describe the scm and its relevance to measuring bias. @xcite first observed that gender stereotypes are present in static word embeddings (e.g. word2vec and glove) using subspace analysis. @xcite found that word embeddings capture a spectrum of implicit biases, using lexicons developed for the implicit association test, or the iat @xcite , and calculated associations within static word embeddings. @xcite extended this approach to work with contextualized embedding models such as bert. however, using word lists pulled from the iat is limiting when it comes to assessing disability bias, since the relevant tests incorporate images instead of words. for this reason, there has been more work in downstream tasks such as sentiment analysis and topic modelling @xcite , and less in direct association analysis. ## results and discussion figure 1 displays the bias score between each pair of targets (abled/disabled antonyms, e.g. "hearing" and "deaf") for each stereotype subdimension in the scm. here we can see certain patterns in how disability is represented in bert, compared to ability. the first takeaway from this figure is that there is a bias, or a difference, in the representations, confirming rq1. the bias is significant at varying levels across all subdimensions except the unable subdimension. correlation in language usage may have contributed to the lack of bias in the unable subdimension. mentions of disability are often accompanied by words referring to ability, and often in a negative, medical context where disability is if the score is above zero, this means that bert positively associates the target with the stereotype subdimension. conversely, if the score is below zero, bert negatively associates the target with the stereotype subdimension. these results reveal patterns in how bert's representations of ability and disability align to known stereotype subdimensions. the second takeaway is that bert is generally more likely to associate the abled target to all stereotype subdimensions, except the unable subdimension for all three pairs of targets, and the immoral and unsociable subdimensions for blindness. this partiality toward ability may been caused by higher frequencies of abled targets in the training data @xcite . people with disabilities are an underrepresented population and are thus mentioned less in mainstream text; there is an ongoing project to improve one of the training datasets to create more text related to disability (wikipedia contributors, 2022). it is also less common to use an abled target to describe a person without a disability (beukeboom and burgers, 2019), and this in addition to these words' increased frequency may have led bert to "understand" them better but in different contexts. the third takeaway is that the bias is stronger if the sentence includes a positive warmth (moral, sociable) or competence (able, independent) context, presenting a high-level insight into rq2. given a positive stereotype context, bert is more likely to predict the abled target than the disabled target in the fill-in-the-blank task. in other words, bert is less likely to associate disability to warmth and competence. this bias is significant for ability, deafness, and blindness at @xmath0.001. on the other hand (or the other side of the figure), the bias between abled/disabled antonym tar-get pairs is weaker if the sentence includes a negative warmth (immoral, unsociable) or competence (dependent) context. this smaller difference in representation is still significant for deafness at @xmath1.001, significant for general ability at varying levels, and significant for blindness with only the dependent subdimension at @xmath2.01. to investigate rq2 in more depth, we show in figure 2 the mean association scores for each combination of target (an abled or disabled antonym) and stereotype subdimension. this figure reveals more nuanced patterns in bert's representation of disability and how this representation aligns to stereotype subdimensions from the scm. one pattern that stands out is that almost all of the mean association scores are negative, regardless of target or subdimension. a negative association score indicates that bert is less likely to predict the target given the stereotype content and the syntactic structure of the sentence template. these negative association scores provide further support for bert having limited knowledge about abled targets' range of usage, and/or the under-representation of disabled targets in the model. figure 2 also sheds additional light on the weaker bias shown in figure 1 for negative subdimensions. although bert may have an overall preference for abled targets, the disabled targets' associations to these negative subdimensions are strong enough to appear nearly on par with the abled targets' associations to the same subdimen-figure 3 : bias scores for pairs of targets, when the attribute is predicted in the presence of the target. for interpretation details, please refer to figure 1 . these results show evidence that bert is less likely to predict any attribute given an accompanying disability context. bert contains significantly stronger associations between all stereotype attribute subdimensions and the abled target. a third takeaway from figure 2 is that disabled targets are less associated with able, independent, moral, and sociable contexts, compared to all other associations. this is especially pronounced with "disabled" and "deaf". in figure 4 , the bias scores from the inverse analysis present evidence that predicting different attributes given the same target do not lead to different biases. different stereotype subdimensions are not any closely combined with different targets, when the target context is already present in the sentence. however, bert shows a general preference for predicting any attribute in the presence of abled targets, since the bias scores are all significantly positive, especially for ability. we want to note that, despite semantic bleaching, syntactic differences in the sentence templates affected the strength of the association scores, but not the patterns. when using identity-first templates to predict a target given stereotype content, bert more strongly associated "abled" and "hearing" to all subdimensions, whereas "sighted", "disabled", "blind", and "deaf" had stronger associations to all attribute subdimensions using person-first templates. this is interesting, because identity-first and person-first language are known to carry cultural connotations. furthermore, some common identityfirst disability qualifiers, such as "disabled" and "deaf", and "blind" are used in contexts outside of social identity categories, e.g. as metaphors: "deaf as a post," "deaf and blind to [insert situation]". this may have impacted how they were understood by the model, and subsequently how they are predicted in identity-first or person-first language contexts. ## conclusions and future work regardless of how biases manifest, the first step toward ensuring harmless use of ai-based assistive technologies is to understand how target users are represented in the underlying models. by applying the stereotype content model to evaluate representational differences, we present evidence of disability association bias in a popular pre-trained nlp model that is used in state-of-the-art ai-based assistive technologies such as text simplification and speech recognition. we also present a breakdown of this bias along stereotype dimensions, which uncovers nuanced patterns in undesirable associations between disability and stereotypes, the most notable being that disabled people are significantly less likely to be associated to warmth and competence. our results emphasize the need to work toward more fair and inclusive assistive technologies, especially since disabled people are the target population for these tools. there are a number of limitations with our study. first, we explored these associations through a broad lens, looking at only ability versus disability. it is important to recognize that disability is not a siloed, unitary concept @xcite . future work should investigate the associations through an intersectional lens @xcite , to better understand how disability bias is affected by the interconnected nature of social categorizations. a second limitation of our study is our usage of sentence templates. despite attempts to semantically strip a sentence to provide a neutral context, bert still draws on the syntactic structure of the sentence itself to help make its predictions @xcite . we took this into consideration by varying the structure. however, we observed that association strengths appear to be influenced to a degree by syntactic differences. future work can investigate stabilizing the bias evaluation metrics by including more templates and a wider range of sentence structure, or randomly sampling a natural sentence dataset. it would also be interesting to further differentiate between identity-first and person-first language, as well as to explore question-answering templates. third, we examined a limited number of targets and only in one model, bert. future work can extend our approach to evaluate additional disabled targets in additional models, such as gpt-2 (radford et al., 2018) and gpt-3 @xcite , to get a fuller picture of disability representation in a wider range of popular pre-trained nlp models underlying ai-based assistive technologies. future work can also draw on debiasing approaches to mitigate bias in these models. we want to note that it is important in this work to also take into consideration the specific model deployment context, because enforcing fairness in an inappropriate context can result in the unintended erasure of a marginalized population @xcite . we provided an array of possible causes of the stereotype patterns that we observed, and these can be avenues for exploring debiasing solutions.