Is Your Classifier Actually Biased? Measuring Fairness under Uncertainty with Bernstein Bounds (2020)

Abstract: Most NLP datasets are not annotated with protected attributes such as gender, making it difficult to measure classification bias using standard measures of fairness (e.g., equal opportunity). However, manually annotating a large dataset with a protected attribute is slow and expensive. Instead of annotating all the examples, can we annotate a subset of them and use that sample to estimate the bias? While it is possible to do so, the smaller this annotated sample is, the less certain we are that the estimate is close to the true bias. In this work, we propose using Bernstein bounds to represent this uncertainty about the bias estimate as a confidence interval. We provide empirical evidence that a 95% confidence interval derived this way consistently bounds the true bias. In quantifying this uncertainty, our method, which we call Bernstein-bounded unfairness, helps prevent classifiers from being deemed biased or unbiased when there is insufficient evidence to make either claim. Our findings suggest that the datasets currently used to measure specific biases are too small to conclusively identify bias except in the most egregious cases. For example, consider a co-reference resolution system that is 5% more accurate on gender-stereotypical sentences – to claim it is biased with 95% confidence, we need a bias-specific dataset that is 3.8 times larger than WinoBias, the largest available.

## introduction nlp models have drawn criticism for capturing common social biases with respect to gender and race @xcite @xcite . these biases can be quantified by applying some metric to an embedding space @xcite , but it is unclear how bias in text embeddings affects decisions made by downstream classifiers. this is because bias is not propagated deterministically: it is possible for mini-mally biased embeddings to be fed into a classifier that makes maximally biased predictions @xcite . moreover, recent work has found that weat @xcite , the most popular test of embedding bias, can be easily manipulated to claim that bias is present or absent @xcite . unlike measuring embedding bias, measuring classification bias is difficult: most nlp datasets are not annotated with protected attributes, precluding the use of standard fairness measures such as equal opportunity @xcite . however, manually annotating a large dataset with a protected attribute is slow and expensive. in response to this problem, some have created small datasets annotated with a single protected attribute -typically gender -that is used to estimate bias on tasks such as co-reference resolution @xcite @xcite . this can be done by creating new data or annotating a subset of an existing dataset with the protected attribute. intuitively, the less data we annotate, the less certain we are that our sample bias is close to the true bias (i.e., what we would get by annotating the entire population). we propose using bernstein bounds to express our uncertainty about the sample bias as a confidence interval. first, we show that for standard fairness measures such as equal opportunity and equalized odds @xcite , we can define a cost function such that the fairness measure is equal to the difference in expected cost incurred by the protected and unprotected groups. we treat the contribution of each annotated example to the bias as a random variable. using bernstein's inequality, we can thus estimate the probability that the true bias is within a constant t of our sample bias. working backwards, we then derive a confidence interval for the true bias. treating the "genres" of examples in mnli @xcite as the protected groups and the rate of annotator disagreement as the cost, we offer empirical evidence that our 95% confidence interval consistently bounds the true bias. in quantifying the uncertainty around bias estimates, bernstein-bounded unfairness helps prevent classifiers from being deemed biased or unbiased when there is insufficient evidence to make either claim. for example, even when the sample bias is positive, it is possible that the true bias between groups is zero. conversely, a sample bias of zero does not ensure the absence of bias at the population level. moreover, our findings suggest that most bias-specific datasets in nlp are too small to conclusively identify bias except in the most egregious cases. for example, consider a co-reference resolution system that is 5% more accurate on gender-stereotypical sentences. for us to claim that this system is gender-biased with 95% confidence, we would need a bias-specific dataset that is 3.8 times larger than winobias @xcite , the largest such dataset currently available. not only does the nlp community need more bias-specific datasets, but it also needs datasets that are much larger than the ones it currently has. in this section, we present the core idea of our paper: bernstein-bounded unfairness (bbu). in practice, we estimate the bias -which we call the groupwise disparity -using a small sample of annotated data. given that this estimate deviates from the true bias (i.e., at the population level), bbu helps us express our uncertainty about the bias estimate using a confidence interval. definition 2.1. let c : (y, ŷ) → [0,c] denote the cost of predicting ŷ when the true label is y, where @xmath0. definition 2.2. let f : @xmath1, 0, +1} denote an annotation function that maps an example to the protected group a (+1), the unprotected group b (-1), or neither (0). the groupwise disparity δ ( f ; c) between groups a and b is the difference in expected cost incurred by each group: definition 2.3. the amortized disparity δ (x i , f ; c) for an example x i , given an annotation function f and cost function c, is: the amortized disparity of x i is an estimate of the groupwise disparity based solely on x i . the expectation over all amortized disparities is the groupwise disparity: δ ( f ; c) = e x [ δ (x, f ; c)]. in practice, given n i.i.d. examples x, we can take a monte carlo estimate of δ ( f ; c) by partitioning x into the protected and unprotected groups using f and then calculating the difference in mean cost. an equivalent way of framing this is that we have n random variables δ (x 1 , f ; c), ..., δ (x n , f ; c) and we are taking their mean to estimate δ ( f ; c). because examples x are i.i.d., so are the random variables. this means that we can use bernstein's inequality to calculate the probability that the sample mean δ deviates from the true groupwise disparity δ by some constant @xmath2. where [-m, m] bounds each random variable δ (x i , f ; c) and σ 2 = 1 @xmath3, by bernstein's inequality: since the interval [-m, m] is defined by the frequency of protected and unprotected examples (2.3), if we want it to strictly bound the random variable, it should be [-nc, nc], where n is the population size and we assume that there is at least one protected example. however, if this were the interval, (1) could be criticized for being too loose a bound and effectively useless. therefore we assume that the proportion of the population that is protected and unprotected is bounded and that the lower bounds on these proportions are known. definition 2.4. let γ a , γ b denote the lower bounds of the proportion of the population that is protected and unprotected respectively. let γ = min(γ a , γ b ). note that the protected group does not necessarily have to be the smaller of the two groups in this setup. we set γ to be the lesser of γ a and γ b to reflect this: if the unprotected group is smaller than the protected group, then [-m, m] will be bounded in [-c/γ b ,c/γ b ]. proposition 2.5. under (2.4), [-m, m] ⊆ [-c γ , c γ ] for any random variable. using this interval, (1) can be rewritten as: proposition 2.6. for a given confidence ρ ∈ [0, 1) that the true groupwise disparity δ falls in the inter-val [ δt, δ + t], we can derive @xmath4: this can be derived by rearranging (2) after setting both sides to be equal and then applying the quadratic formula to find the solution to t. note that the width of the confidence interval grows as: (a) the desired confidence ρ increases; (b) the sample size n decreases; (c) γ decreases. to our knowledge, bernstein bounds are the tightest that can be applied here, as they consider the variance of the random variables. we also validated empirically that they are a better candidate than hoeffding bounds, another common choice. standard fairness measures how can we use bernstein-bounded unfairness to derive confidence intervals when the bias metric is demographic parity, equal opportunity, or equalized odds? it is thus possible to define the cost and annotation functions such that the groupwise disparity is equivalent to the bias defined by a common fairness measure. because of our framing of the problem, we treat the cost as something to be minimized. for example, for equal opportunity, the groupwise disparity was defined as the difference in false negative rates. however, we could set c(y, ŷ) = ŷ for equal opportunity as well, such that the groupwise disparity is the difference in true positive rates. both perspectives are equivalent, but one may be more intuitive depending on the use case. ## proof-of-concept experiments we begin by providing empirical evidence that a 95% bbu confidence interval consistently bounds the true bias (i.e., population-level groupwise disparity). we conduct our experiments on the mnli dev set @xcite , used for testing natural language inference. we treat the genres of examples in mnli as the "protected groups". since the genre annotations are given, we calculate the true bias as the difference in annotator disagreement rates for in-genre versus out-genre examples, effectively treating the human annotators as the classifier whose bias we want to measure. we then use bbu and check whether the true bias falls within the 95% confidence interval when we estimate the bias using a subset of the data. the experiments on mnli do not measure an important social bias. rather, they are meant to be a proof-of-concept. we treat the mnli genres as "protected groups" because the protected attribute -the genre -is clearly annotated. we use mnli over smaller datasets annotated with attributes such as gender because this setup -where the cost is the rate of annotator disagreement -does not require any model training, making our results easy to replicate. moreover, this use case illustrates that our conception of bias need not be restricted to social biases -it can be the difference in cost incurred by any arbitrarily defined groups. lastly, we examine how large a bias-specific dataset needs to be in order to conclude that a given classifier is biased. specifically, we consider a co-reference resolution system that is more accurate on sentences containing stereotypical gender roles. fixing the confidence level at ρ = 0.95, we show that as the magnitude of the sample bias δ decreases, we need a larger bias-specific dataset (i.e., larger n) in order to make a bias claim with 95% confidence. ## conclusion we first showed that many standard measures of fairness (e.g., equal opportunity) can be expressed as the difference in expected cost incurred by protected and unprotected groups. given that most bias estimates are made using small samples, we proposed bernstein-bounded unfairness (bbu) for quantifying the uncertainty about a bias estimate using a confidence interval. using mnli, we provided empirical evidence that 95% bbu confidence intervals consistently bound the true populationlevel bias. in quantifying this uncertainty, bbu helps prevent classifiers from being deemed biased or unbiased when there is insufficient evidence to make either claim. although datasets currently used to estimate classification bias (e.g., winobias) are undoubtedly a step in the right direction, our findings suggest that they need to be much larger in order to be a useful diagnostic.