Debiasing Isn’t Enough! – on the Effectiveness of Debiasing MLM s and Their Social Biases in Downstream Tasks (2022)

Abstract: We study the relationship between task-agnostic intrinsic and task-specific extrinsic social bias evaluation measures for MLMs, and find that there exists only a weak correlation between these two types of evaluation measures. Moreover, we find that MLMs debiased using different methods still re-learn social biases during fine-tuning on downstream tasks. We identify the social biases in both training instances as well as their assigned labels as reasons for the discrepancy between intrinsic and extrinsic bias evaluation measurements. Overall, our findings highlight the limitations of existing MLM bias evaluation measures and raise concerns on the deployment of MLMs in downstream applications using those measures.

## introduction text representations produced by mlms have revolutionised nlp by improving the performance of numerous downstream applications @xcite @xcite @xcite . unfortunately, large-scale pretrained mlms demonstrate worrying levels of social biases when applied to downstream tasks @xcite @xcite . given that real-world natural language processing (nlp) systems such as machine translation systems, dialogue systems, etc. are used by millions of users world-wide @xcite , it remains an important responsibility to accurately evaluate the social biases in mlms prior to deployment. two types of social bias evaluation measures have been proposed for mlms in prior work @xcite : (a) task-agnostic intrinsic evaluation measures @xcite @xcite that use the likelihood scores assigned by an mlm under evaluation for sentences representing various social biases, and (b) task-specific extrinsic evaluation measures that use the data from downstream nlp tasks such as predicting the occupation of a person from their biographies @xcite . in contrast to intrinsic measures that directly probe into the biases in mlms, extrinsic measures holistically evaluate an nlp system that uses an mlm. much prior work has decoupled the intrinsic and extrinsic bias evaluations for simplicity, and have assumed that an mlm unbiased according to an intrinsic measure will remain to be so under extrinsic evaluations too. however, this unverified hypothesis begs the following question: can we reliably predict the social biases of an mlm when it is applied to a particular downstream task using only intrinsic evaluation measures? to answer this question, we conduct a comprehensive study using 7 pretrained mlms and their debiased versions using three different debiasing methods. however, only weak correlations are found between intrinsic and extrinsic bias evaluation measures. this is a worrying proposition because intrinsic measures are often used to decide whether an mlm should be deployed in a downstream application. we further investigate on why mlms learn social biases during downstream task fine-tuning and identify two sources: instance-related biases and label-related biases. the training instances for downstream tasks contain unfair associations that are learnt by the masked language modelling objective, which we name instance-related biases. we find that debiased mlms gradually learn such biases during fine-tuning, leading to disagreements between intrinsic and extrinsic evaluation measures. moreover, the labels assigned to downstream train-ing instances can also represent biased ratings, which are also learnt by the mlms when their parameters are updated in an end-to-end manner when predicting task labels. overall, we find weak correlations between intrinsic and extrinsic measures even after the models are fine-tuned on downstream task data. based on our findings, we recommend that intrinsic bias evaluation measure alone must not be used to determine whether an mlm should be deployed in a downstream nlp application. the code and data used in this paper will be publicly released to facilitate reproduction of results upon paper acceptance. ## related work social biases have been reported in models trained for numerous downstream tasks. @xcite examined 219 automatic sentiment analysis systems that participated in the semeval-2018 task 1 @xcite and found that many systems show statistically significant gender or racial biases. @xcite investigated the age-related biases in sentiment classification and found that across 15 sentiment analysis tools, sentences containing adjectives describing the youth were 66% more likely to be scored positively than those describing the elderly. the problem is also observed in mlms trained for many languages @xcite . numerous bias mitigation approaches have been proposed for mlms such as fine-tuning @xcite , counterfactual data augmentation @xcite and parameter dropout @xcite . @xcite proposed a method to debias the sentence embeddings created from bert @xcite and elmo @xcite inspired by hard debiasing @xcite , a method originally proposed for static word embeddings. however, not all biases are adequately mitigated by the current proposals @xcite . our focus in this paper is the evaluation and not debiasing methods. intrinsic bias evaluation measures @xcite @xcite evaluate the social biases in a given mlm standalone, independently of any downstream applications. pseudo log-likelihood scores assigned to stereotypical vs. antistereotypical examples (either manually written or automatically generated using templates) have been used to evaluate the social biases in an mlm as further detailed in § 3. considering that mlms are used to represent input texts in various downstream tasks, several prior work have argued that their social biases must be evaluated with respect to those tasks @xcite @xcite . however, decoupling tasks and mlms makes the bias evaluation simpler and taskindependent, which is attractive because mlms are designed as generic text representations for a widerange of downstream tasks. we further discuss extrinsic bias evaluation measures in §3. goldfarb-tarrant et al. ( 2021 ) studied social biases in static word embeddings using the word embedding association test (weat) as an intrinsic measure and coreference detection and hate speech detection as extrinsic evaluation tasks. they found the correlations between intrinsic and extrinsic measures to be weak or negative. although we share the motivation with @xcite , our focus is contextualised embeddings produced by mlms, which have outperformed static word embeddings in numerous tasks, hence more widely used in real-world nlp applications than static word embeddings. cao et al. (2022) studied the correlation between intrinsic bias scores (i.e. contextualised embedding association test @xcite and increased log probability score @xcite ) and extrinsic bias scores for three tasks: toxicity (jigsaw, 2019), hate-speech @xcite and sentiment @xcite detection. similar to @xcite they also found weak correlations. as reasons for this lack of correlation they highlight misalignment between the metrics such as the notion of bias, protected groups, and noise in the evaluation datasets. our contributions are complementary to theirs because we consider different intrinsic and extrinsic measures, not covered by them. moreover, our extrinsic tasks are directly related to social biases. we observe weak correlations between intrinsic and extrinsic evaluation measures, supporting their claims. ## bias evaluation measures several measures have already been proposed in prior work for evaluating the social biases encoded in mlms. these measures can be broadly categorised into two groups depending on whether they evaluate social biases in mlms with respect to a particular downstream task or not. we refer to measures that evaluate social biases in mlms on their own right, independently of any downstream tasks as intrinsic mlm bias scores and describe three such measures. in contrast, we refer to the measures that evaluate social biases in mlms when they are applied to solve a specific downstream task such as natural language inference (nli), semantic textual similarity (sts) or predicting occupations from biographies as extrinsic mlm bias scores and describe those measures. to describe the intrinsic measures, let us consider a test sentence @xmath0, w 2 , . . . , w |s| , containing length |s| sequence of tokens w i , where part of s is modified to create a stereotypical (or lack of thereof) example for a particular social bias. for example, let's consider the sentence-pair "john completed his phd in machine learning" vs. "mary completed her phd in machine learning". here, {john, his} are the modified tokens for the first sentence, whereas for the second sentence they are {mary, her}. the unmodified tokens between the two sentences are {completed, phd, in, machine, learning}. let us denote the list of modified tokens in a given sentence s by m and the list of unmodified tokens by u , so that @xmath1. foot_0 given an mlm with pretrained parameters θ to evaluate for social biases, let us denote the probability p mlm (w i |s \w i ; θ) that the mlm assigns to a token w i conditioned on the remainder of the tokens, s \w i . @xcite demonstrated that pll(s), the pseudo-loglikelihood (pll) score of sentence s given by (1), can be used to evaluate the preference expressed by an mlm for s, similarly to using log-probabilities for evaluating the naturalness of sentences using conventional language models. pll scores for mlms can be computed directly and are more uniform across sentence lengths (no left-to-right bias), allowing us to recognize natural sentences in a language @xcite . as we will see later, pll can be used to define bias evaluation scores for mlms in a variety of ways. intrinsic: stereoset score the probability of generating the modified tokens given the unmodified tokens in s was expressed as p (m |u ; θ) by @xcite . we refer to this as stereoset score (sss) and it is given by ( 2 ). intrinsic: crows-pairs score the crows-pairs score (cps) is a scoring formula provided by (3) that was defined as p (u |m ; θ) by @xcite to address the frequency-bias in sss. normalization is not performed here because the length of unmodified tokens is the same. however, by masking and predicting one token w at a time from u , we are effectively changing the context (u \w , m ) used as input by the mlm. this has two disadvantages. first, by removing w from the sentence, the mlm loses information that it can use to predict w. as a result, the prediction accuracy of w may decrease, rendering bias evaluations untrustworthy. second, the remaining tokens {u \w , m } may still be biased even if we remove one token w at a time from u . moreover, the context on which we condition the probabilities continuously varies across predictions. intrinsic: all unmasked likelihood with attention weights @xcite proposed all unmasked likelihood with attention weights (aula) to overcome the above-mentioned disfluencies in previously proposed mlm bias evaluation measures. to begin, rather than masking out tokens from s, aula provides the entire sentence to the mlm. second, all tokens in s that appear in between the start and end of sentences are predicted by aula. furthermore, aula computes the likelihood using attention weights to evaluate social biases based on the relative importance of words in a sentence, as given by ( 4 ). here, α i represents the average of all multi-head attentions associated with w i . given a score function @xmath2, cps, aula}, we use the percentage of stereotypical (s st ) test sentences preferred by the mlm over anti-stereotypical (s at ) ones to define the corresponding bias evaluation measure (bias score) as follows: for this purpose, they first filter lines that begin with a name-like-pattern (i.e. a sequence of two capitalised words) followed by the string "is a(n) (xxx) title", where title is an occupation from the bls standard occupation classification system. 2 they used twenty-eight frequent occupations in common crawl to collect 397,340 biographies. the task is to predict the people's occupations, taken from the first sentence of their biographies, given the remainder of their biographies. for example, given the hypothetical biography of a nurse: she graduated from lehigh university, with honours in 1998. nancy has years of experience in weight loss surgery, patient support, education, and diabetes., the goal is to predict the occupation. an mlm under evaluation is used to create an embedding for a biography excluding the title sentence, and a softmax classifier is trained using those 2 https://www.bls.gov/soc biography embeddings to predict the occupation of the person described in each biography. if the mlm is unbiased with respect to gender, we would expect it to have approximately similar true positive rates (tprs) averaged over all twenty-eight occupations for both male and female gendered persons. the tpr gap between the tprs for classifying occupations in female and male biographies is used as an extrinsic mlm bias score. an unbiased mlm is expected to predict occupations with similar tprs for both genders. extrinsic: sts-bias semantic textual similarity (sts) is a task which evaluates the degree of semantic similarity between two sentences. webster et al. (2020) proposed sts-bias to evaluate gender bias by adapting sts formulation. specifically, to create the sts-bias dataset, they collect the 276 sentences from the sts-b test set, which start with a man or a woman, and discard the sentences with multiple gendered words. 3 for each template, they formed two sentence pairs per profession from @xcite , one representing the male gender and the other for the female gender. for example, given a pair "a man is walking" and "a nurse is walking" from sts-b dataset, they create a new pair "a woman is walking" and "a nurse is walking". if not relying on gendered correlations, a model should assign equal estimates of similarity to the two pairs. therefore, following @xcite , as a score we use the difference between similarities of such two pairs to measure how similar model predictions actually are regarding gender. they compute the pearson correlation between the scores per profession and the representation in that profession listed in the us bureau of labor statistics @xcite . sts-bias does not consider the directionality of the gender bias. therefore, we report absolute values of all bias scores. extrinsic: nli-bias @xcite used natural language inference (nli) for evaluating gender bias. this dataset contains sentence triples where a premise must be neutral in entailment w.r.t. two hypotheses. for example, "the accountant ate a bagel" should neither entail nor contradict the sentences "the man ate a bagel" and "the woman ate a bagel". given a set containing d test instances, let the entailment predictor's probabilities for the d-th instance for entail, neutral and contradiction labels be respectively e d , n d and c d . then, they proposed the measure fraction neutral (fn): for an ideal (bias-free) embedding, fn measure would be 1. they compute the fn for each female and male sentences and use the difference between them as the bias score in the nli-bias. ## conclusion we investigated social biases in 28 mlms and found that there exist weak correlation between intrinsic and extrinsic bias evaluation measures. moreover, debiased mlms re-learn biases during task-specific fine-tuning due to instance and label related biases in downstream task data. our findings highlight limitations in existing bias evaluation measures and raise serious concerns regarding their implications.