Measuring Fairness of Text Classifiers via Prediction Sensitivity (2022)

Abstract: With the rapid growth in language processing applications, fairness has emerged as an important consideration in data-driven solutions. Although various fairness definitions have been explored in the recent literature, there is lack of consensus on which metrics most accurately reflect the fairness of a system. In this work, we propose a new formulation – accumulated prediction sensitivity, which measures fairness in machine learning models based on the model’s prediction sensitivity to perturbations in input features. The metric attempts to quantify the extent to which a single prediction depends on a protected attribute, where the protected attribute encodes the membership status of an individual in a protected group. We show that the metric can be theoretically linked with a specific notion of group fairness (statistical parity) and individual fairness. It also correlates well with humans’ perception of fairness. We conduct experiments on two text classification datasets – Jigsaw Toxicity, and Bias in Bios, and evaluate the correlations between metrics and manual annotations on whether the model produced a fair outcome. We observe that the proposed fairness metric based on prediction sensitivity is statistically significantly more correlated with human annotation than the existing counterfactual fairness metric.

## introduction ongoing research is increasingly emphasizing the development of methods which detect and mitigate unfair social bias present in machine learningbased language processing models. these methods come under the umbrella of algorithmic fairness which has been quantitatively expressed with numerous definitions @xcite . these fairness definitions are broadly categorized into two types, i.e, individual fairness and group fairness. individual fairness (e.g., counter-factual fairness @xcite ) is aimed at evaluating whether a model gives similar predictions for individuals with similar personal attributes (e.g., age or race). on the other hand, group fairness (e.g., statistical parity @xcite ) evaluates fairness across cohorts with same protected attributes instead of individuals @xcite . although these two broad categories of fairness define valid notions of fairness, human understanding of fairness is also used to measure fairness in machine learning models @xcite . existing studies often consider only one or two these verticals of measuring fairness. in our work, we propose a formulation based on models sensitivity to input features -the accumulated prediction sensitivity, to measure fairness of model predictions. we establish its theoretical relationship with statistical parity (group fairness) and individual fairness @xcite metrics. we then demonstrate the correlation between the proposed metric and human perception of fairness using empirical experiments. researchers have proposed metrics to quantify fairness based on a model's sensitivity to input features. specifically, @xcite ; @xcite propose a prediction sensitivity metric that attempts to quantify the extent to which a single prediction depends on a protected attribute. the protected attribute encodes the membership status of an individual in a protected group. prediction sensitivity can be seen as a form of feature attribution, but specialized to the protected attribute. in our work, we extend their concept of prediction sensitivity to propose accumulated prediction sensitivity. akin to the metric proposed by @xcite , our metric also relies on model output's sensitivity to changes in input features. our metric generalizes their notion of sensitivity, where the model sensitivity to various input features can be weighted non-uniformly. we show that the formulation follows certain properties for the chosen definitions of group and individual fairness and also present several methodologies to select weights assigned to sensitivity of model's output to input features. for each selection, we present the correlation between the accumulated prediction sensitivity and human assessment of the model-output fairness. we define our metric in section 3 and present bounds on it (under settings when a classifier follows the selected group fairness or individual fairness constraints) in sections 4 and 5, respectively. next, given that the human perception of fairness is not theoretically defined, we present an empirical study on two text classification tasks in section 6. we request a group of annotators to annotate whether they think that model output is biased against a specific gender and observe that the proposed metric correlates positively with more biased outcomes. we then observe correlations between our metric and the stated human understanding of fairness. we find that not only the proposed accumulated prediction sensitivity metric correlates positively with human perception of bias, but also beats an existing baseline based on counterfactual fairness. ## related work multiple efforts have looked into defining, measuring, and mitigating biases in nlp models @xcite @xcite . @xcite and @xcite al. (2017 multiple recent works also focus on developing new dataset and associated metrics to capture various types of biases. for example, @xcite and @xcite propose dataset and metrics to measure social biases and stereotypes in language model generations, @xcite ; @xcite ; @xcite define metrics to access gender and race biases in word vector representations, and @xcite define metrics to quantify and mitigate biases in visual recognition task. @xcite propose bernstein bounds to represent uncertainty about the bias. majority of these bias metrics are automatically computed, for example, using a regard classifier @xcite , sentiment classifier @xcite , toxicity classifier @xcite or true positive rate difference between privileged and underprivileged groups @xcite . a few works additionally validate the alignment of these automatically computed bias metrics with human understanding of biases by collecting annotations of biases on a subset of test data from crowdworkers @xcite . @xcite discuss the limitations of several of these bias datasets and measurements. however, the majority of existing bias metrics are specific to the model type and the application domain used, they may not be tested for correlation with human judgement of biases, and their relationship to existing definitions of fairness has not been explored. additionally, metrics such as true positive or error difference between groups requires ground truth labels, thereby making their computation in real-time systems difficult. @xcite have attempted to present unified approach to measuring group and individual fairness via inequality indices, however we note that such metrics are non-trivial to extend to unstructured data such as text. for example, gender information in a text may be subtle (e.g. mention of softball) and it is unclear whether presence of this word should be considered to impact the genderness of the text. accumulated prediction sensitivity metric, presented in this paper, attempts to address all the above limitations of existing bias metrics. we acknowledge that the proposed metric is yet to be associated with other notions of fairness (e.g. preference based notion of fairness @xcite ). ## accumulated prediction sensitivity below, we define accumulated prediction sensitivity, a metric that captures the sensitivity of a model to protected attributes. ## relation to group fairness: statistical parity given a set of protected features (e.g. gender), a model satisfies statistical parity if model outcome is independent of the protected features (we note that identifying protected features may not always be feasible in the real world). we represent the feature vector @xmath0, x l ], where x p is the set of protected features and x l is the remainder. accordingly, we choose v to be a vector such that the entries that sum | ∂f k (xp) ∂x i |∀x @xmath1; and zero otherwise. this choice is intuitive as then we sum the gradients in j that correspond to protected features and measure model's sensitivity to them. the predictor f (x) will satisfy statistical parity if f (x p , x l ) = f (x p , x l )∀x @xmath2. given this, we state the following theorem. theorem 1. given a vector v with non-zero entries corresponding to x p and zero entries for x l , if the predictor f (x) satisfies statistical parity with respect to x p , accumulated prediction sensitivity will be zero. proof: if f (x) satisfies statistical parity with respect to x p , the values ∂f k (x) ∂xp ∀x @xmath3. this is due to the fact that the function f k (x) can not be defined based on entries x @xmath4. therefore, for every multiplication in the product j v, either the entry ∂f k (x) ∂xp will be 0 or the entry in v corresponding to x l will be 0. hence, p will be 0. appendix a presents empirical results in computing p on a synthetic dataset. we construct a dataset where a feature (hair length) correlates with a protected attribute (gender). we show that if the modeler unintentionally uses the correlated feature while attempting to build a classifier with statistical parity, our metric can be used for evaluation. it is natural to choose an lp norm @xcite for d and d. for a classifier f that is trained with the above constrained optimization and the choice of distance metrics d, d is an lp norm, we state the following. theorem 2. if the predictor f (x) is trained with the constrained optimization stated in eq. ( 2 ), the accumulated prediction sensitivity will be upper bounded by l. proof: we restate the constraint in eq. ( 2 ) as (note that the inequality sign does not change as distance metrics d, d are required to be positive for given the inequality holds for any pair of x, x , it must also hold for an x of the following choice. we can rewrite eq. ( 4 ) as: we can further chose ∆x i such that it is small perturbation, leading to the following. therefore, each entry in j is upper bounded by l. as vectors v, w are stochastic and they compute weighted averages of bounded entries in j , p (defined in eq. ( 1 )) must be less than or equal to l. we also note that as l becomes larger, the constraint in the eq. ( 2 ) becomes looser. therefore, a higher value of l during optimization is expected to loosen the fairness constraint as well as the bound on fairness sensitivity. this aligns with our intuition of lower values of p for fairer models. we compute value of l on a synthetically generated classification data, optimized with the individual fairness constraint in equation 2. the results are presented in appendix b. ## correlations with human perception of fairness while the conditional statistical parity and individual fairness establish theoretical constraints on the model behaviour (e.g. independence from protected features and similarity in prediction outcomes for similar data-points), humans may carry a different notion of fairness for model outcomes on individual data-points. this notion may be based on their understanding of cultural norms, which in turn effect their decisions in identifying which model outputs could be considered biased. in this section, we present experiments that correlate accumulated prediction sensitivity with human perception of fairness. ## results ## considerations for accumulated prediction sensitivity metric while the results showcase the promise of our metric, we draw the attention of the reader to the following considerations: (1) we observed that the metric quality depends on choice of the hyperparameters w and v. in this regard, our metric is not different from other metrics that also depend on a hyper-parameter choice. for example, any classifier based metric has a threshold parameter and counterfactual fairness metrics rely on hyperparameters such as the selected gendered words. (2) our metric only works for models for which gradients can be computed. most modern deep learning based models carry this property. (3) lastly, we note that it is hard to interpret the absolute value of the proposed metric. the metric value should be used for relative comparison of two models which share input feature space and label space. in addition, we note two considerations for relying on a psm classifier. first, training it requires access to gender labels. second, the psm model itself could be biased. given that gender labels may not always be available for the dataset used to train model at hand, we study the impact of transferring a psm model trained on a different dataset on computing our metric. we also evaluate the effect of bias in psm model on the overall metric value and present results in the appendix d. we make observations such as the quality of the metric degrades as psm becomes more biased. based on these observations, we recommend that if modeler is not able to obtain high performance psm models, they fall back to using sources such as gendered words for computing the vector v. ## conclusion evaluating fairness is a challenging task as it requires selecting a notion of fairness (e.g. group or individual fairness) and then identifying metrics that can capture these notions of fairness while evaluating a classifier. additionally, certain notions of fairness may not be well defined and can change based upon social norms (e.g. "volleyball" being closely associated with females); that may seep into the dataset at hand. in this work, we define an accumulated prediction sensitivity metric that relies on the partial derivatives of model's class probabilities with respect to input features. we establish properties of this metric with respect to the three verticals of fairness metrics: group, individual and human-perception based. we provide bounds on the metric's value when a predictor is expected to carry statistical parity or is trained with individual fairness. we also evaluate this metric with fairness as perceived through human evaluation of model outputs. we test variants of the proposed metric against an existing baseline derived from counter-factual fairness and observe better mutual information and correlation. specifically, a variant of the metric that relies on a protected status model (that identifies tokens that carry gender information but may not conventionally be considered gendered) yields the best correlation with the human evaluation. in the future, one can associate the proposed formulation with other categories of group and in-dividual fairness @xcite . we also aim to test the metric on other datasets with other protected attributes (e.g. race, nationality). finally, we can compare the metric across these datasets to compare trends across protected groups.