Mitigating Gender Bias Amplification in Distribution by Posterior Regularization (2020)

Abstract: Advanced machine learning techniques have boosted the performance of natural language processing. Nevertheless, recent studies, e.g., (CITATION) show that these techniques inadvertently capture the societal bias hidden in the corpus and further amplify it. However, their analysis is conducted only on models’ top predictions. In this paper, we investigate the gender bias amplification issue from the distribution perspective and demonstrate that the bias is amplified in the view of predicted probability distribution over labels. We further propose a bias mitigation approach based on posterior regularization. With little performance loss, our method can almost remove the bias amplification in the distribution. Our study sheds the light on understanding the bias amplification.

## introduction data-driven machine learning models have achieved high performance in various applications. despite the impressive results, recent studies (e.g., @xcite ; @xcite ) demonstrate that these models may carry societal biases exhibited in the dataset they trained on. in particular, @xcite show that a model trained on a biased dataset may amplify the bias. for example, we can consider a task of labeling the activity and objects depicted in an image. the training set contains 30% more images with "woman cooking" than "man cooking". however, when evaluating the top predictions of a trained model, the disparity between males and females is amplified to around 70%. based on this observation, @xcite conduct a systematic study and propose to calibrate the top predictions of a learned model by injecting corpus-level constraints to ensure that the gender disparity is not amplified. however, when analyzing the top predictions, the models are forced to make one decision. therefore, even if the model assigns high scores to both labels of "woman cooking" and "man cooking", it has to pick one as the prediction. this process obviously has a risk to amplify the bias. however, to our surprise, we observe that gender bias is also amplified when analyzing the posterior distribution of the predictions. since the model is trained with regularized maximal likelihood objective, the bias in distribution is a more fundamental perspective of analyzing the bias amplification issue. in this paper, we conduct a systematic study to quantify the bias in the predicted distribution over labels. our analysis demonstrates that when evaluating the distribution, though not as significant as when evaluating top predictions, the bias amplification exists. about half of activities show significant bias amplification in the posterior distribution, and on average, they amplify the bias by 3.2%. we further propose a new bias mitigation technique based on posterior regularization because the approaches described in @xcite can not be straightforwardly extended to calibrate bias amplification in distribution. with the proposed technique, we successfully remove the bias amplification in the posterior distribution while maintain the performance of the model. besides, the bias amplification in the top predictions based on the calibrated distribution is also mitigated by around 30%. these results suggest that the bias amplification in top predictions comes from both the requirement of making hard predictions and the bias amplification in the posterior distribution of the model predictions. our study advances the understanding of the bias amplification issue in natural language processing models. the code and data are available at 2 related work algorithmic bias machine learning models are becoming more and more prevalent in the real world, and algorithmic bias will have a great societal impact @xcite . researchers have found societal bias in different applications such as coreference resolution @xcite , machine translation @xcite and online advertisement @xcite . without appropriate adjustments, the model can amplify the bias @xcite . different from the previous work, we aim at understanding the bias amplification from the posterior perspective instead of directly looking at the top predictions of the model. ## background we follow the settings in @xcite to focus on the imsitu vsrl dataset @xcite , in which we are supposed to predict the activities and roles in given images and this can be regraded as a structure prediction task (see fig. 1 ). we apply the conditional random field (crf) model for the structure prediction task. we denote y as a joint prediction result for all instances, and y i as a prediction result for instance i. we use y v to denote the predicted activity, and y r to denote the predicted role. an activity can have multiple roles and usually one of them conveys the gender information. for an instance i, the crf model predicts the scores for every activity and role, and the score for a prediction is the summation of all these scores. formally, ## bias amplification quantification and corpus-level constraints @xcite demonstrate bias amplification in the top prediction and present a bias mitigation technique by inference with corpus-level constraints. in the following, we extend their study to analyze the bias amplification in the posterior distribution by the crf model and define the corresponding corpus-level constraints. formally, the probability of prediction y i for instance i and the joint prediction y defined by crf model with parameters θ are given by we focus on the gender bias on activities in the vsrl task. to quantify the gender bias given a particular activity v * , @xcite uses the percentage that v * is predicted together with male agents among all prediction with genders. this evaluation focuses on the top prediction. in the contrast, we define bias function b(p, v * , d) w.r.t distribution p and activity v * , evaluating the bias toward male in dataset d based on the conditional probability p (x|y ), where event y : given an instance, its activity is predicted to be v * and its role is predicted to have a gender; event x : this instance is predicted to have gender male. formally, here we use b * (v * , male) to denote the "dataset bias" toward male in the training set, measured by the ratio of between male and female from the labels: ideally, the bias in the distribution given by crf model should be consistent with the bias in the training set, since crf model is trained by maximum likelihood. however, the amplification exists in practice. here we use the difference between the bias in the posterior distribution and in training set to quantify the bias amplification, and average it over all activities to quantify the amplification in the whole dataset: note that if we use the top prediction indicator function to replace p in a, ā, it is the same as the definition of the bias amplification in top prediction in @xcite . the corpus-level constraints aim at mitigating the bias amplification in test set d ts within a predefined margin γ, ## posterior regularization posterior regularization @xcite is an algorithm leveraging corpus-level constraints to regularize the posterior distribution for a structure model. specifically, given corpus-level constraints and a distribution predicted by a model, we 1) define a feasible set of the distributions with respect to the constraints; 2) find the closest distribution in the feasible set from given distribution; 3) do maximum a posteriori (map) inference on the optimal feasible distribution. the feasible distribution set q is defined by the corpus-level constraints defined in eq. ( 3 ): given the feasible set q and the model distribution p θ defined by eq. ( 1 ), we want to find the closest feasible distribution q * : this is an optimization problem and our variable is the joint distribution q with constraints, which is intractable in general. luckily, according to the results in @xcite , if the feasible set q is defined in terms of constraints feature functions φ and their expectations: eq. ( 5 ) will have a close form solution actually, we can derive the constraints into the form we want. we set @xmath0 we can choose a proper φ i (y i ) to make eq. ( 4 ) equal to eq. ( 6 ). the detailed derivation and the definition of φ i (y i ) are shown in appendix a. we can solve eq. ( 8 ) by gradient-based methods to get λ * , and further compute the close form solution in eq. ( 7 ). actually, considering the relation between y and y i in eq. ( 1 ) and ( 9 ), we can factorize the solution in eq. ( 7 ) on instance level: and the derivation details are in appendix b. with this, we can reuse original inference algorithm to conduct map inference based on the distribution q * for every instance seperately. ## experiments we conduct experiments on the vsrl task to analyze the bias amplification issue in the posterior distribution and demonstrate the effectiveness of the proposed bias mitigation technique. dataset our experiment settings follow @xcite . we evaluate on imsitu @xcite that activities are selected from verbs, roles are from framenet @xcite and nouns from wordnet @xcite . we filter out the non-human oriented verbs and images with labels that do not indicate the genders. model we analyze the model purposed together with the dataset. the score functions we describe in sec. 3 are modeled by vgg @xcite with a feedforward layer on the top of it. the scores are fed to crf for inference. ## conclusion we analyzed the bias amplification from the posterior distribution perspective, which provides a better view to understanding the bias amplification issue in natural language models as these models are trained with the maximum likelihood objective. we further proposed a bias mitigation technique based on posterior regularization and show that it effectively reduces the bias amplification in the distribution. due to the limitation of the data, we only analyze the bias over binary gender. however, our analysis and the mitigation framework is general and can be adopted to other applications and other types of bias. one remaining open question is why the gender bias in the posterior distribution is amplified. we posit that the regularization and the over-fitting nature of deep learning models might contribute to the bias amplification. however, a comprehensive study is required to prove the conjecture and we leave this as future work.