BLIND : Bias Removal With No Demographics (2023)

Abstract: Models trained on real-world data tend to imitate and amplify social biases. Common methods to mitigate biases require prior information on the types of biases that should be mitigated (e.g., gender or racial bias) and the social groups associated with each data sample. In this work, we introduce BLIND, a method for bias removal with no prior knowledge of the demographics in the dataset. While training a model on a downstream task, BLIND detects biased samples using an auxiliary model that predicts the main model’s success, and down-weights those samples during the training process. Experiments with racial and gender biases in sentiment classification and occupation classification tasks demonstrate that BLIND mitigates social biases without relying on a costly demographic annotation process. Our method is competitive with other methods that require demographic information and sometimes even surpasses them.

## introduction neural natural language processing (nlp) models are known to suffer from social biases, such as performance disparities between genders or races @xcite . numerous debiasing methods have been proposed in order to address this issue, with varying degrees of success. a disadvantage of these methods is that they require knowledge of the biases one wishes to mitigate (e.g., gender bias) @xcite @xcite . moreover, some methods require additional annotations for identifying the demographics for each sample in the data, such as the race of the writer @xcite @xcite . some annotations can be automatically collected from the data, while others require manual annotations or expert knowledge, which can be very costly. thus, existing methods are typically limited to a small number of datasets and tasks. in this paper, we propose a new debiasing method, blind -bias removal with no demographics. we see social bias as a special case of robustness issues resulting from shortcut learning @xcite . our goal is to down-weight samples that contain demographic features that may be used as shortcuts in downstream tasks. we first consider a case where we have demographic annotations for every sample in the training set, and train a demographics detector -an auxiliary classifier that takes the main model's representations and predicts the demographic attribute. then, we down-weight the importance of samples on which the classifier is confident (figure 1a ). to our knowledge, this is the first work to consider demographic information for re-weighting samples during training. when we do not have demographic annotations, we make the following observation: the main model has an easier job, or otherwise makes predictable mistakes, when demographic features are used as shortcut features. thus, we train a success detector -another auxiliary classifier, which takes the representations of the main model and predicts its success on the task. a correct prediction by the success detector means the main model made a shallow decision, since it is possible predict its success or failure without access to the main model's task labels. in such cases we expect that the main model relies on simple, shortcut features, and we use the success detector's confidence to down-weight such samples in the training data (figure 1b ). we call this method blind. in both cases, we adapt the debiased focal loss (dfl; karimi mahabadi et al., 2020), originally proposed for mitigating annotation artifacts, to down-weight samples that the detectors predicted correctly. in contrast to the original dfl work, which explicitly defined biases to mitigate, we design the bias detection mechanism in a general manner such that the model's biases are learned and mitigated dynamically during training. we perform experiments on two english nlp tasks and two types of social demographics: occupation classification with gender, and sentiment analysis with race. our methods successfully reduce bias, with blind sometimes succeeding in cases where other methods that use demographic information fail. our analysis shows that blind reduces demographic information in the model's internal representation, even though it does not have access to it. additionally, blind is particularly effective at mitigating bias due to its down-weighting of easy training samples, rather than relying on demographic information alone. this suggests that blind may be more robust in mitigating bias than other methods. ## results in the main body of the paper, we report accuracy and a representative subset of fairness metrics. the full set of fairness metrics is reported in appendix d. in ## prior work studies suggests a variety of ways for debiasing nlp models from social biases on downstream tasks, such as preprocessing the training data @xcite @xcite , modifying the training process @xcite , or applying post-hoc methods to neural representations of a trained model @xcite @xcite . all these methods, however, require that we define the bias we wish to operate upon, for example, gender bias. additionally, many of these methods require demographic annotations per data instance, such as the gender of the writer or the subject of the text. @xcite is an exception, since it performs gender-debiasing by modifying dropout parameters. another exception is jtt @xcite , which improved worstgroup errors by training a model twice: first a standard erm model, then a second model that upweights training samples misclassified by the first model. the authors of these studies chose hyperparameters based on fairness metrics they wanted to optimize, while we choose our hyper-parameters without explicitly measuring fairness metrics. to our knowledge, this is the first study to mitigate social biases in nlp without assuming any prior knowledge. other studies have focused on improving nlp models robustness without prior knowledge of bias issues, but without considering social bias. utama et al. ( 2020 ) and @xcite tackled dataset biases (a.k.a annotation artifacts) in natural language understanding tasks, by training a weak learner to identify biased samples and downweighting their importance. weak learners are either trained on a random subset of the dataset or have a smaller capacity. regarding social fairness without demographics, @xcite proposed adversarially reweighted learning, where an adversarial model is trained to increase the total loss by re-weighting the training samples. they used tabular, nontextual data in their experiments. we consider nonadversarial methods since adversarial training is known to be difficult to stabilize. @xcite proposed a method for minimizing the worst case risk over all distributions close to the empirical distribution, without knowledge of the identity of the group. @xcite considered fairness in unsupervised domain adaptation, where the source or target domain do not have demographic attributes, by proposing weighting methods. @xcite proposed debiasing via adversarial training with only a small volume of protected labels. focal loss @xcite was proposed as a method to address class imbalances by downweighting loss associated with well-classified samples. @xcite proposed using the original focal loss @xcite to improve robustness in natural language inference, leading to improved out-of-distribution accuracy. debiased focal loss (dfl) @xcite is a variant of focal loss proposed to improve natural language understanding models on out-of-distribution data. ## discussion and conclusion even though blind led to bias reduction, it was less effective than our method that used demographic annotations. analysis showed that the success detector is less accurate at classifying samples that fail the main model. additionally, the success detector might be less focused than demographicbased methods, but it might mitigate biases we have not identified and cannot measure without annotations. thus, it would be interesting to see how blind works on intersectional biases. in sentiment analysis, blind reduced bias less than jtt, which also does not use demographics for training, but does for hyper-parameter search. however, jtt was ineffective on the occupation classification task, while blind was effective for both tasks. the two tasks differ significantly, as well as their data. for bert and deberta, pretraining data is closer to biographies than tweets, so perhaps training for longer is beneficial for the tweets data used in the sentiment classification task, and repeating samples in the training set (as in jtt) is similar to training for more steps. in any case, blind proved more reliable and generalizable in reducing bias. our method has the potential for broader applications beyond demographic biases. while our primary focus was on mitigating demographic biases, the approach can be adapted to address other types of biases by identifying relevant proxy indicators via the success detector. to summarize, we demonstrated the reduction of racial and gender biases of nlp classification models, without any prior knowledge of those biases in the data. these results suggest that we can potentially apply blind to any dataset, which makes bias reduction a much more feasible endeavor and applicable to real-world scenarios.