Choose Your Lenses: Flaws in Gender Bias Evaluation (2022)

Abstract: Considerable efforts to measure and mitigate gender bias in recent years have led to the introduction of an abundance of tasks, datasets, and metrics used in this vein. In this position paper, we assess the current paradigm of gender bias evaluation and identify several flaws in it. First, we highlight the importance of extrinsic bias metrics that measure how a model’s performance on some task is affected by gender, as opposed to intrinsic evaluations of model representations, which are less strongly connected to specific harms to people interacting with systems. We find that only a few extrinsic metrics are measured in most studies, although more can be measured. Second, we find that datasets and metrics are often coupled, and discuss how their coupling hinders the ability to obtain reliable conclusions, and how one may decouple them. We then investigate how the choice of the dataset and its composition, as well as the choice of the metric, affect bias measurement, finding significant variations across each of them. Finally, we propose several guidelines for more reliable gender bias evaluation.

## introduction a large body of work has been devoted to measurement and mitigation of social biases in natural language processing (nlp), with a particular focus on gender bias @xcite @xcite . these considerable efforts have been accompanied by various tasks, datasets, and metrics for evaluation and mitigation of gender bias in nlp models. in this position paper, we critically assess the predominant evaluation paradigm and identify several flaws in it. these flaws hinder progress in the field, since they make it difficult to ascertain whether progress has been actually made. gender bias metrics can be divided into two groups: extrinsic metrics, such as performance difference across genders, measure gender bias with respect to a specific downstream task, while intrinsic metrics, such as weat @xcite , are based on the internal representations of the language model. we argue that measuring extrinsic metrics is crucial for building confidence in proposed metrics, defining the harms caused by biases found, and justifying the motivation for debiasing a model and using the suggested metrics as a measure of success. however, we find that many studies on gender bias only measure intrinsic metrics. as a result, it is difficult to determine what harm the presumably found bias may be causing. when it comes to gender bias mitigation efforts, improving intrinsic metrics may produce an illusion of greater success than reality, since their correlation to downstream tasks is questionable @xcite . in the minority of cases where extrinsic metrics are reported, only few metrics are measured, although it is possible and sometimes crucial to measure more. additionally, gender bias measures are often applied as a dataset coupled with a measurement technique (a.k.a metric), but we show that they can be separated. a single gender bias metric can be measured using a wide range of datasets, and a single dataset can be applied to a wide variety of metrics. we then demonstrate how the choice of gender bias metric and the choice of dataset can each affect the resulting measures significantly. as an example, measuring the same metric on the same model with an imbalanced or a balanced dataset foot_0 may result in very different results. it is thus difficult to compare newly proposed metrics and debiasing methods with previous ones, hindering progress in the field. to summarize, our contributions are: bias statement this paper examines metrics and datasets that are used to measure gender bias, and discusses several pitfalls in the current paradigm. as a result of the observations and proposed guidelines in this work, we hope that future results and conclusions will become clearer and more reliable. the definition of gender bias in this paper is through the discussed metrics, as each metric reflects a different definition. some of the examined metrics are measured on concrete downstream tasks (extrinsic metrics), while others are measured on internal model representations (intrinsic metrics). the definitions of intrinsic and extrinsic metrics do not align perfectly with the definitions of allocational and representational harms (kate crawford, 2017). in the case of allocational harm, resources or opportunities are unfairly allocated due to bias. representative harm, on the other hand, is when a certain group is negatively represented or ignored by the system. extrinsic metrics can be used to quantify both allocational and representational harms, while intrinsic metrics can only quantify representational harms, in some cases. there are also other important pitfalls that are not discussed in this paper, like the focus on highresource languages such as english and the binary treatment of gender @xcite @xcite . inclusive research of non-binary genders would require a new set of methods, which could benefit from the observations in this work. ## the importance of extrinsic metrics in defining harms in this paper, we divide metrics for gender bias to three groups: it is crucial to define how measured bias harms those interacting with the biased systems @xcite @xcite . extrinsic metrics are important for motivating bias mitigation and for accurately defining "why the system behaviors that are described as 'bias' are harmful, in what ways, and to whom" (blodgett et al., 2020), since they clearly demonstrate the performance disparity between protected groups. for example, in a theoretical cv-filtering system, one can measure the tpr gap between female and male candidates. a gap in tpr favoring men means that, given a set of valid candidates, the system picks valid male candidates more often than valid female candidates. the impact of this gap is clear: qualified women are overlooked because of bias. in contrast, consider an intrinsic metric such as weat @xcite , which is derived from the proximity (in vector space) of words like "career" or "family" to "male" or "female" names. if one finds that male names relate more to career and female names relate more to family, the consequences are unclear. in fact, @xcite found that weat does not correlate with other extrinsic metrics. however, many studies report only intrinsic metrics (a third of the papers we reviewed, §5). the developer argued with the designer because she did not like the design. the developer argued with the designer because he did not like the design. ## coupling of datasets and metrics in this section, we discuss how datasets and metrics for gender bias evaluation are typically coupled, how they may be decoupled, and why this is important. we begin with a representative test case, followed by a discussion of the general phenomenon. ## effect of dataset on measured results the choice of data to measure bias has an impact on the calculated bias. many researchers used sentence templates that are "semantically bleached" (e.g., "this @xmath0.", "<@xmath1.") to adjust metrics developed for static word embeddings to contextualized representations @xcite @xcite . @xcite found that the choice of templates significantly affected the results, with little correlation between different templates. additionally, @xcite reported that templates are not as semantically bleached as expected. another common feature of bias measurement methods is the use of hand-curated word lexicons by almost every bias metric in the literature. @xcite reported that the lexicon choice can greatly affect bias measurement, resulting in differing conclusions between different lexicons. ## different metrics cover different aspects of bias in this section, we explore how the choice of bias metrics influences results. although extrinsic bias metrics are useful in defining harms caused by a gender-biased system, we find that most studies on gender bias use only intrinsic metrics to support their claims. we surveyed a representative list of papers presenting bias mitigation techniques that appeared in the survey by @xcite , as well as recent papers from the past year. in total, we examined 36 papers. the majority of papers do not measure extrinsic metrics. even when downstream tasks are measured, only a very small subset of metrics (three or less) is typically measured, as shown in figure 4 . furthermore, in these studies, typically no explanation is provided for choosing a particular metric. the exceptions are de vassimon manela et al. (2021) and @xcite , who measured six and nine or ten metrics on downstream tasks, respectively. orgad et al. showed that different extrinsic metrics behave differently under various debiasing methods. additionally, in §4 we saw that subsampling the test set increased one bias metric and decreased others, which would not have been evident had we only measured a small number of metrics. measuring multiple metrics is also important for evaluating debiasing. when kaneko and bollegala compared their proposed debiasing method to that of @xcite , the new method outperformed the old one on two of the three metrics. as the examples above illustrate, different extrinsic metrics are not necessarily consistent with one another. furthermore, it is possible to measure more extrinsic metrics, although it is rarely done. when it is not feasible to measure multiple metrics, one should at least justify why a particular metric was chosen. in a cv-filtering system, for example, one might be more forgiving of fpr gaps than of tpr gaps, as the latter leaves out valid candidates for the job in one gender more than the other. however, more extrinsic metrics are likely to provide a more reliable picture of a model's bias. ## conclusion and proposed guidelines the issues described in this paper concern the instabilities and vagueness of gender bias metrics in nlp. since bias measurements are integral to bias research, this instability limits progress. we now provide several guidelines for improving the reliability of gender bias research in nlp. focus on downstream tasks and extrinsic metrics. extrinsic metrics are helpful in motivating bias mitigation ( §2). however, few datasets can be used to quantify extrinsic metrics, especially extrinsic performance metrics, which require gold labels ( §3). more effort should be devoted to collecting datasets with extrinsic bias assessments, from more diverse domains and downstream tasks. stabilize the metric or the dataset. both the metrics and the datasets could have significant effects over the results: the same dataset can be used to measure many metrics and yield different conclusions ( §4), and the same metric can be measured on different datasets and show bias in one instance but not in another ( §5). if one wishes to measure gender bias in an nlp system, it is better to hold one of these variables fixed: for example, to focus on a single metric and measure it on a set of datasets. of course, this can be repeated for other metrics as well. this will produce much richer, more consistent, and more convincing results. neutralize dataset noise. as a result of altering a dataset's composition, we observed very different results ( §4). this is caused by the way various fairness metrics are defined and computed on the entire dataset. to ensure a more reliable evaluation, we recommend normalizing a dataset when using it for evaluation. in the case of occupation prediction, normalization can be obtained by balancing the test set. in other cases it could be by anonymizing the test set, removing harmful words, etc., depending on the specific scenario. motivate the choice of specific metrics, or measure many. most work measures only a few metrics ( §5). a comprehensive experiment, such as to prove the efficacy of a new debiasing method, is more reliable if many metrics are measured. in some situations, a particular metric may be of interest; in this case one should carefully justify the choice of metric and the harm that is caused when the metric indicates bias. the motivation for debiasing this metric then follows naturally. define the motivation for debiasing through bias metrics. @xcite found that papers' motivations are "often vague, inconsistent, and lacking in normative reasoning". we propose to describe the motivations through the gender bias metrics chosen for the study: define what is the harm measured by a specific metric, what is the behavior of a desired versus a biased system, and how the metric measures it. this is where extrinsic metrics will be particularly useful. we believe that following these guidelines will enhance clarity and comparability of results, contributing to the advancement of the field. a list of gender bias metrics, as presented in