A Robust Bias Mitigation Procedure Based on the Stereotype Content Model (2022)

Abstract: The Stereotype Content model (SCM) states that we tend to perceive minority groups as cold, incompetent or both. In this paper we adapt existing work to demonstrate that the Stereotype Content model holds for contextualised word embeddings, then use these results to evaluate a fine-tuning process designed to drive a language model away from stereotyped portrayals of minority groups. We find the SCM terms are better able to capture bias than demographic agnostic terms related to pleasantness. Further, we were able to reduce the presence of stereotypes in the model through a simple fine-tuning procedure that required minimal human and computer resources, without harming downstream performance. We present this work as a prototype of a debiasing procedure that aims to remove the need for a priori knowledge of the specifics of bias in the model.

## introduction it is well established that large language models (llms) such as bert @xcite , gpt2 @xcite and related contextualised word embeddings such as elmo @xcite are biased against different demographic groups @xcite @xcite , in that they often reflect stereotypes in their output. for example, given the prompt "naturally, the nurse is a", these systems will typically output "woman" @xcite . given the common practice of adapting pre-trained language models for a range of tasks through fine-tuning, upstream bias mitigation may prove to be the most efficient solution @xcite (though cf. @xcite . in this paper, we demonstrate the success of modifying an existing debiasing algorithm to be grounded in a psychological theory of stereotypes -the scm @xcite , to efficiently reduce biases in llms across a range of identities. our proposed debiasing pipeline has the benefit of minimising the time spent researching identity terms and associated stereotypes. being a fine-tuning procedure, this also reduces the amount of computational resources needed compared to training an unbiased model from scratch. this renders our approach efficient and widely applicable. we demonstrate using bert, but this same procedure could easily be adapted to other llms. we adapt the fine-tuning procedure from @xcite . they reduce gender bias in a range of llms by fine-tuning using a data set of sentences containing (binary) gendered terms (like "he, man" or "she, lady") (which they call attributes) or stereotypes associated with different genders ("assertive, secretary") (which they call targets). the training objective is to remove associations with gender in the contextualised embeddings of the targets whilst maintaining these associations for the gendered attributes. crucially, rather than relying on stereotypes specific to a particular demographic such as men and women (as in @xcite ) we plan to use the scm to inform our production of fine-tuning data, inspired by work by @xcite . the scm states that our stereotyped perception of different demographics can be conceptualised as lying in a vector space with axes of warmth/coldness and competence/incompetence @xcite . we tend to consider our own identity group to be warm and competent, and stereotype disfavoured groups such as people experiencing homelessness as cold and/or incompetent @xcite . in the terminology of @xcite , our attributes are terms relating to warmth and competence taken from @xcite (as in @xcite , a paper on stereotypes in static embeddings), our targets are demographic identity terms. because the scm is designed to encompass many different minority groups, this avoids the need to generate lists of stereotypes unique to each minority group, reducing work load and making the tool easy to adapt to different targets. therefore, the procedure should be effective for all identity terms we use. we demonstrate this technique for black/white ethnicity and also the intersectional power dynamic between white men and mexican american women, but this could easily be expanded to other aspects of identity such as disability and sexuality. further, whilst we focus on english language and american identities, there is evidence that the scm may hold relatively well cross-culturally @xcite , so this approach may be transferable to other llms. we adapt the contextualised embedding association test (ceat) @xcite using the vocabulary from @xcite in order to measure stereotypes in contextualised word embeddings. the ceat provides a robust measure of bias in contextualised word embeddings for target words, and is suited for use with the scm terms. in addition to using the ceat to test for bias, we also measure the performance of the model on the language modeling benchmark glue @xcite , to ensure the fine-tuning procedure does not adversely impact the quality of the model, an issue @xcite identify as affecting several debiasing techniques. the main contributions of this paper are to demonstrate: several contributions have been made towards measuring and mitigating bias in nlu models with minimal a priori knowledge. @xcite demonstrated the validity of the scm for static word embeddings, in that the embeddings of words associated with traditionally oppressed minority groups such as mexican americans or africans tend to lie in the cold, incompetent space, as determined by cosine similarity. note that, unlike @xcite , we focus on the embeddings of the identity terms themselves, not of words associated with those identities, as we explicitly want to identify whether there is bias in the embeddings. @xcite looked to establish if the embeddings of associated terms followed the scm's predictions, not whether the word embeddings were biased in a way as to reflect these stereotypes. ## methodology we use the 'bert-base-cased' model from the hugging face library foot_3 ), henceforth bert, although this same procedure should be applicable to any llm with minimal modification. we fine-tune the model following an adapted version of the procedure in @xcite . namely, through a training objective that looks to minimise unwanted projection into the attribute category dimensions for the target words through an orthogonal projection, whilst also staying close to the contextualised embeddings of the pre-trained model to preserve semantics. we visualise this orthogonal projection in figure 1 . adjusting the embeddings of the target terms to lie orthogonal to the warmth dimension (equidistant from the attribute terms) should ensure less negatively biased representations for minority groups (in the visualisation, aa names). crucially, we modified the original algorithm in @xcite as we wish to remove unwanted projections into two dimensions, not just one: warmth/coldness and competence/incompetence. the first component of the loss function for layer i of our model is: the second component of the loss function is: the final loss function is a weighted sum: ## discussion we found that our approach to bias measurement, informed by the scm, proved to be an effective method for detecting bias in an llm. we found that compared to using another list of generic, non-demographic specific attribute terms related to pleasantness, our approach seemed to give a more accurate measure of the level of bias in the model -our terms allow us to capture a stronger association between a minority group and negative stereotypes. it is possible that our approach exaggerates the level of bias in the model and in fact is less accurate. however, the effect sizes from our approach are closer to the effect size for association with demographic specific terms for the intersectional pair, suggesting it paints an accurate picture of negative bias in the model. further, given how often bert has been found to produce offensive content, it seems more likely that use of pleasant-ness terms is underestimating the level of bias in the model, rather than our approach overestimating it. the pleasantness terms were only slightly associated with ea male names compared to ma female names, yet bert has been shown to consistently produce more favourable content about such individuals @xcite . our finding that the intersectional bias terms were actually more strongly associated with the black/white demographic groups highlights how the selection of demographic specific stereotypes for use in measuring bias and debiasing models can be challenging. that these stereotypes are actually more strongly associated with aa/ea names could suggest that the stereotyping captured by the model does not reflect the attitudes of the group of undergraduates responsible for generating these stereotypes @xcite . it could also be that the model has not been exposed to sufficient (stereotyped) data to capture the category of ma females and the associated stereotypes. the results might suggest that these demographic specific terms are actually rather "demographic agnostic", hence they are able to capture bias against aa people. however, intuitively, "sexy" and "feisty" (two ma female specific stereotypes) are not associated with people experiencing homelessness (and studies on public attitudes towards homelessness to our knowledge confirm this intuition), but the stereotype content model is able to predict the contempt they experience due to being perceived as cold and incompetent @xcite , which is likely reflected in language use and thus in an llm. after debiasing using the scm informed approach, we were able to reduce bias in all instances. not only did we reduce the association between competence, warmth and ethnicity, but we also reduced the association with pleasantness. intuitively, this is likely a reflection of the semantic association between warmth and pleasantness -reducing projection in the warmth dimension may have impacted projection in the pleasantness dimension. crucially, we were able to reduce the association between the intersectional groups and their specific stereotypes, using a demographic agnostic approach that did not require prior knowledge of group specific stereotypes. although we only ran the debiasing procedure for warmth and competence dimensions, there was a positive "knock on" effect, supporting our belief that debiasing at the more abstract level will reduce more specific bias associations as well, as these can be thought of as subcategories of these more generic stereotype concepts. we were able to successfully debias the model without impeding performance on benchmark nli tasks, suggesting language modelling abilities have not been negatively impacted, and in two instances performance was actually improved, possibly due to the reduction in bias.