LOGAN : Local Group Bias Detection by Clustering (2020)

Abstract: Machine learning techniques have been widely used in natural language processing (NLP). However, as revealed by many recent studies, machine learning models often inherit and amplify the societal biases in data. Various metrics have been proposed to quantify biases in model predictions. In particular, several of them evaluate disparity in model performance between protected groups and advantaged groups in the test corpus. However, we argue that evaluating bias at the corpus level is not enough for understanding how biases are embedded in a model. In fact, a model with similar aggregated performance between different groups on the entire data may behave differently on instances in a local region. To analyze and detect such local bias, we propose LOGAN, a new bias detection technique based on clustering. Experiments on toxicity classification and object classification tasks show that LOGAN identifies bias in a local region and allows us to better analyze the biases in model predictions.

## introduction machine learning models such as deep neural networks have achieved remarkable performance in many nlp tasks. however, as noticed by recent studies, these models often inherit and amplify the biases in the datasets used to train the models @xcite @xcite @xcite . to quantify bias, researchers have proposed various metrics to study algorithmic fairness at both individual and group levels. the former measures if a model treats similar individuals consistently no matter which groups they belong to, while the latter requires the model to perform similarly for protected groups and advantaged groups in the cor-pus. 1 in this paper, we argue that studying algorithmic fairness at either level does not tell the full story. a model that reports similar performance across two groups in a corpus may behave differently between these two groups in a local region. for example, the performance gap of a toxicity classifier for sentences mentioning black and white race groups is 4.8%. 2 this gap is only marginally larger than the performance gap of 2.4% when evaluating the model on two randomly split groups. however, if we evaluate the performance gap on the sentences containing the token "racist", the performance gap between these two groups is as large as 19%. similarly, @xcite report that a visual semantic role labeling system tends to label an image depicting cooking as woman cooking than man cooking. however, the model is, in fact, more likely to produce an output of man cooking when the agent in the image wears a chef hat. we call these biases exhibited in a neighborhood of instances local group bias in contrast with global group bias which is evaluated on the entire corpus. to detect local group bias, we propose logan, a local group bias detection algorithm to identify biases in local regions. logan adapts a clustering algorithm (e.g., k-means) to group instances based on their features while maximizing a bias metric (e.g., performance gap across groups) within each cluster. in this way, local group bias is highlighted, allowing a developer to further examine the issue. our experiments on toxicity classification and ms-coco object classification demonstrate the effectiveness of logan. we show that besides 1 for example, @xcite and @xcite evaluate the bias in coreference resolution systems by measuring the difference in f1 score between cases where a gender pronoun refers to an occupation stereotypical to the gender and the opposite situation. ## methodology in this section, we first provide formal definitions of local group bias and then the details of the detection method logan. performance disparity assume we have a trained model f and a test corpus @xmath0, y i )} @xmath1...n that is used to evaluate the model. let p f (d) represents the performance of the model f evaluated on the corpus d. based on the applications, the performance metric can be accuracy, auc, false positive rates, etc. for the sake of simplicity, we assume each input example x i is associated with one of demographic groups (e.g., male or female), i.e., x @xmath2. foot_0 as a running example, we take performance disparity as the bias metric. that is, if p f (a 1 ) -p f (a 2 ) > , then we consider that the model exhibits bias, where is a given threshold. definition of local group bias we define local group bias as the bias exhibits in certain local region of the test examples. formally, given a centroid c in the input space, let while this definition is based on performance disparity, it is straightforward to extend the notion of local group bias to other bias metrics. logan the goal of logan is to cluster instances in d such that (1) similar examples are grouped together, and (2) each cluster demonstrates local group bias contained in f . to achieve this goal, logan generates cluster @xmath3,j } @xmath4...n,@xmath5...k by optimizing the following objective: clustering objective the loss l c is derived from a standard clustering technique. in this paper, we consider the k-means clustering method @xcite . specifically, the loss l c of k-means is note that our framework is general and other clustering techniques, such as spectral clustering @xcite , dbscan @xcite , or gaussian mixture model can also be applied in generating the clusters. besides, the features used for creating the clusters can be different from the features used in the model f . local group bias objective for the local group bias loss l b , the goal is to obtain a clustering that maximizes the bias metric within each cluster. in the following descriptions, we take the performance gap between different attributes (see eq. ( 1 )) as an example to describe the bias metric. let ŷi = f (x i ) be the prediction of f on x i . the local group bias loss l b is defined as the negative summation of performance gaps over all the clusters. if accuracy is used as the performance evaluation metric, ## conclusion machine learning models risk inheriting the underlying societal biases from the data. in practice, many works use the global performance gap between different groups as a metric to detect the bias. in this work, we revisit the coarse-grained metric for group bias analysis and propose a new method, logan, to detect local group bias by clustering. our method can help detect model biases that previously are hidden from the global bias metrics and provide an explanation of such biases. we notice there are some limitations in logan. for example, the number of instances in clusters could be uneven (see appendix a.3).