Intrinsic Bias Metrics Do Not Correlate with Application Bias (2021)

Abstract: Natural Language Processing (NLP) systems learn harmful societal biases that cause them to amplify inequality as they are deployed in more and more situations. To guide efforts at debiasing these systems, the NLP community relies on a variety of metrics that quantify bias in models. Some of these metrics are intrinsic, measuring bias in word embedding spaces, and some are extrinsic, measuring bias in downstream tasks that the word embeddings enable. Do these intrinsic and extrinsic metrics correlate with each other? We compare intrinsic and extrinsic metrics across hundreds of trained models covering different tasks and experimental conditions. Our results show no reliable correlation between these metrics that holds in all scenarios across tasks and languages. We urge researchers working on debiasing to focus on extrinsic measures of bias, and to make using these measures more feasible via creation of new challenge sets and annotated test data. To aid this effort, we release code, a new intrinsic metric, and an annotated test set focused on gender bias in hate speech.

## introduction awareness of bias in natural language processing (nlp) systems has rapidly increased as more and more systems are discovered to perpetuate societal unfairness at massive scales. this awareness has prompted a surge of research into measuring and mitigating bias, but this research suffers from lack of consistent metrics that discover and measure bias. instead, work on bias is "rife with unstated assumptions" @xcite and relies on metrics that are easy to measure rather than metrics that meaningfully detect bias in applications. this untested assumption leaves nlp bias research in a precarious position. research into the semantics of word embeddings has already shown that intrinsic metrics (e.g. using analogies and semantic similarity, as in @xcite do not correlate well with extrinsic metrics @xcite . research into the bias of word embeddings lacks the same type of systematic study, and thus as a field we are exposed to three large risks: 1) making misleading claims about the fairness of our systems, 2) concentrating our efforts on the wrong problem, and most importantly, 3) feeling a false sense of security that we are making more progress on the problem than we are. our bias research can be rigorous and innovative, but unless we understand the limitations of metrics we use to evaluate it, it might have no impact. in this paper, we ask: does the commonly used intrinsic metric for embeddings (weat) correlate with extrinsic metrics of application bias? to answer this question, we analyse the relationship between intrinsic and extrinsic bias. our study considers two languages (english and spanish), two common embedding algorithms (word2vec and fasttext) and two downstream tasks (coreference resolution and hatespeech detection). while we find a moderately high correlation between these metrics in a handful of conditions, we find no correlation or even negative correlation in most conditions. therefore, we recommend that the ethical scientist or engineer does not rely on intrinsic metrics when attempting to mitigate bias, but instead focuses on the harms of specific applications and test for bias directly. as additional contributions to these findings, we release new weat metrics for spanish, and a new gender-annotated test set for hatespeech detection for english, both of which we created in the course of this research. ## bias metrics in all of our experiments, we compute correlations between commonly-used metrics, both intrinsic and extrinsic. ## methodology each of our experiments measures the correlation between a specific instance of weat and a specific extrinsic bias metric. in each experiment, we train an embedding, measure the bias according to weat, and measure the bias in the downstream task that uses that embedding. we then modify the embeddings by applying an algorithm to either debias them, or -by inverting the algorithm's behavior -to overbias them. again we measure weat on the modified embedding and also the downstream bias in the target task. when we have done this multiple times until we reach a stopping condition (detailed below), we compute the correlation between the two metrics (via pearson correlation and analysis with scatterplots). rather than draw conclusions from a single experiment, we attempt to draw more robust conclusions by running many experiments, which vary along several dimensions. we consider two common embedding algorithms, two tasks, and two languages. a full table of experiment conditions can be found in ## results figure 3 displays data for all tasks: one scatterplot per triple of experimental variables: an intrinsic metric, an extrinsic metric, an embedding algorithm. if we want to be able to broadly use weat metrics for any given bias research, these graphs should each show a clear and a positive correlation. none of them do. there are no trends in correlation between the metrics that hold in all cases regardless of experimental detail, for any of the tasks. we have additionally examined whether there are correlations within one bias modification method (pre or postprocessing) in case a difference in the way these methods modify embeddings causes differences in trends. in most cases this breakout tells the same story. the select cases where positive (and negative) correlations are present are discussed below. further breakout graphs and combinations are included in appendix d. coreference (en): gender the coreference task (figure 3 , rows 1-3) doesn't display a clear correlation in all cases, and yet it has the clearest relationship of all three tasks, with a significant moderate positive correlation for both predictive parity (precision) and equality of opportunity (recall) for word2vec (columns 3 & 4). the overall trends are muddied by the data for fasttext, which does not have a significant correlation under any conditions. both are expected: that coreference would display the strongest trends, and that fast-text would display more unpredictable or weaker trends. the winobias coreference task is as directly matched to the weat tests as it is possible to be -since both use common career words to measure bias. so the relationship between the two metrics is clearest here: moving female terms closer to certain career terms most directly helps a system resolve anti-stereotypical coreference chains. however, we still only see a correlation in wod2vec, not fast-text. fasttext may behave less predictably because of its use of subwords; when subwords are used, precision recall precision recall weat 6 weat 7 weat 8 weat 6 weat 7 weat 8 weat gender weat migrant english coreference english hate speech detection spanish hate speech detection fasttext embeddings word2vec embeddings word representations are more interconnected. 14 we can debias with regard to a specific word, but that word's embedding will still be influenced by all other words that share its character ngrams. it is difficult to predict how changing the composition of a training corpus will affect all words that contain a certain ngram (e.g. ch) in them. for this reason, fasttext may be initially more resistant to encoding biases than word2vec, as was found in @xcite , but may also be more complex to debias. this has implications for extending this work to contextual models, which always use some form of subword unit. hatespeech (en): gender hatespeech (en) has fewer and more restricted correlations than coreference, as can be seen in figure 3 , rows 4-6. these plots show no relationship at all between intrinsic and extrinsic metrics. when data is broken out by bias modification method (see figure 4b in appendix d), it becomes clear that there is a moderate positive correlation for postprocessing for recall, and the aggregate appears this way because there is a moderate negative correlation for preprocessing. this holds for both embedding algorithms, though both positive and negative correlations are stronger for fasttext. precision displays no correlation. note that the absolute variance in recall is much smaller than for precision, but this is still significant for each embedding algorithm individually and for both grouped together. of interest for future bias research is that the baseline level of bias (premodification, from raw twitter data) in english hatespeech differs by embedding type, but only for precision. initial models (with unmodified embeddings) using fasttext have 10 additional points of precision for maletargeted hatespeech than for female-targeted. however initial models using word2vec have the opposite bias and have 4 fewer points of precision for male-targeted than female targeted hatespeech. for recall, the two embedding algorithms are equivalent, with 6 fewer points for male-targeted hatespeech. in fact, in the recall metric there is an early indication of unreliability of the relationship we are examining between weat and extrinsic bias, because there is a spread of different weat results that map to nearly the same difference in recall. hatespeech (es): gender and migrant for hatespeech in spanish, we examine two kinds of bias separately -gender bias and bias against migrants, in figure 3 , rows 7,8. neither gender bias nor migrant bias show positive correlations in any experimental conditions. gender bias in our models is in an absolute sense never present, since in spanish hatespeech targeted against women is easier to identify than against others (with f1 in the high 80s). 15 but there are no overall trends when this is bias is modified to be more or less extreme, and there are no positive correlations in any conditions. there is a moderate negative correlation for precision only when looking at fasttext embeddings. migrant bias similarly has no trends save in very restricted conditions broken out by bias modification type. in contrast to the gender case, hatespeech against migrants is clearly challenging to identify, with much lower f1 in the low 60s. there is a positive correlation between migrant bias and performance gap for recall with preprocessing in fasttext only. this fits the expectation that fasttext may be more sensitive to preprocessing than postprocessing due to subwords, as discussed above, though in the gender bias case with negative correlation it is equally sensitive to both, so it is hard to draw conclusions. given the smaller number of datapoints for spanish (discussed below) this is likely just noise. to confuse the situation further, the only trends in precision are present in word2vec, and are negative correlations. note that all graphs for spanish display central clusters, because it was more difficult to get an even spread of bias measures, and because spanish has fewer data points than english. this is for a number of reasons that compound and underscore the difficulty of expanding supposedly languageagnostic techniques beyond english, even to high resourced languages like spanish. we have only one weat test for each type of bias, since we made our own that carefully balanced grammatical gender, after rectifying the issues with the existing translated versions (see section 4.3). bias modification is also more difficult -the richer agreement system in spanish means that there are more surface forms of what would be one word in english. in addition, the language model used for nearest neighbour expansion of wordlists (see sec-tion 4.2) produces predominantly formal register words from news or scientific articles, due to a less varied makeup of its training data than the english model. this makes them less well suited to debiasing twitter data specifically, and there were no readily available models that had more casual register. for bias against migrants, there is the additional challenge that wordlists are predominantly based on proper names, which are much rarer in twitter (which tends to use @ mentions instead) than in other media. ## discussion the broad result of this research is that changes in weat do not correlate with changes in application bias, and therefore that weat should not be used to measure progress in debiasing algorithms. we have found that even when we maximally target bias modification of an embedding, we cannot produce a reliable change in bias direction downstream. there was no pattern or correlation between tasks, for the same task in different languages, or even in most cases within one task. and we have chosen one of the simplest possible setups, with fullword embeddings and a single type of bias at a time. real world scenarios can easily be more complicated and involve multiple types of bias or subword embeddings. our findings also indicate that additional complexity may muddy the relationship further. for example, fasttext behaved less predictably than word2vec across experiments, suggesting that if we were to expand to larger models that are fully reliant on subwords the patterns may become even less clear. the implication of this finding is that an nlp scientist or engineer has limited options when investigating and mitigating bias. they must a) find the specific set of wordlists, embedding algorithms, downstream tasks, and bias modification methods that are together predictive of bias for the given task, language, and model or b) implement full systems to test application bias directly, even if their work focuses on embeddings. while the latter may seem onerous, it may not be more so than exhaustively searching for a configuration where intrinsic bias metrics are predictive. this underscores the importance of making good downstream bias measures available, as either approach will require these. more datasets that are collected need to be annotated with subgroup demographic and identity information -there are very few available. more research needs to focus on creating good challenge sets to measure application bias. additional research on more broad usage of unsupervised methods @xcite would also be valuable, though those also would benefit from subgroup identity annotation to make their results more interpretable. it is only when more of these things are readily available that we can see the true measure of the efficacy of our debiasing efforts. we do note a limitation of this study in that all downstream tasks are discriminative classification tasks. bias in classification is more straightforward to measure, with well established metrics, but covers allocational harms (performance disparity), whereas the inclusion of generative models could better cover representational harms (misleading or harmful representations/portrayals) @xcite . concurrent research on causal mediation analysis for bias has shown that the embedding layer in open-domain generation has the strongest effect on gender bias (as compared to other layers of the network) @xcite . further work could investigate whether generation tasks have display the same or different relationship to intrinsic metrics. ## conclusion we have examined the relationship of the intrinsic bias metric weat to the extrinsic bias metrics of equality of opportunity and predictive parity, for multiple tasks and languages, and determined that positive correlations between them exist only in very restricted settings. in many cases there is either negative correlation or none at all. while intrinsic metrics such as weat remain good descriptive metrics for computational social science, and for examining bias in human texts, we advise that the nlp community not rely on them for measuring model bias. we instead advise that they focus on careful consideration of downstream applications and the creation of datasets and challenge sets that enable measurement at this stage. in each plot, the x-axis represents weat, and the y-axis shows performance gap between groups (male-female, female-other, migrant-other). original embeddings (before modification) shown in black. there is no correlation that holds independently of experimental conditions (embedding type, bias modification method, weat test). figure 6 : coreference (en) results broken out by type of winobias challenge, type 1 is more difficult as there are only semantic cues to correct coreference, type 2 has also syntactic cues.