Demographics Should Not Be the Reason of Toxicity: Mitigating Discrimination in Text Classifications with Instance Weighting (2020)

Abstract: With the recent proliferation of the use of text classifications, researchers have found that there are certain unintended biases in text classification datasets. For example, texts containing some demographic identity-terms (e.g., “gay”, “black”) are more likely to be abusive in existing abusive language detection datasets. As a result, models trained with these datasets may consider sentences like “She makes me happy to be gay” as abusive simply because of the word “gay.” In this paper, we formalize the unintended biases in text classification datasets as a kind of selection bias from the non-discrimination distribution to the discrimination distribution. Based on this formalization, we further propose a model-agnostic debiasing training framework by recovering the non-discrimination distribution using instance weighting, which does not require any extra resources or annotations apart from a pre-defined set of demographic identity-terms. Experiments demonstrate that our method can effectively alleviate the impacts of the unintended biases without significantly hurting models’ generalization ability.

## introduction with the development of natural language processing (nlp) techniques, machine learning (ml) models are being applied in continuously expanding areas (e.g., to detect spam emails, to filter resumes, to detect abusive comments), and they are affecting everybody's life from many aspects. however, human-generated datasets may introduce some human social prejudices to the models @xcite . recent works have found that ml models can capture, utilize, and even amplify the unintended biases @xcite , which has raised lots of concerns about the however, recent research points out that there widely exist some unintended biases in text classification datasets. for example, in a toxic comment identification dataset released by @xcite , it is found that texts containing some specific identity-terms are more likely to be toxic. more specifically, 57.4% of comments containing "gay" are toxic, while only 9.6% of all samples are toxic, as shown in because of such a phenomenon, models trained with the dataset may capture the unintended biases and perform differently for texts containing various identity-terms. as a result, predictions of models may discriminate against some demographic minority groups. for instance, sentences like "she makes me happy to be gay" is judged as abusive by models trained on biased datasets in our experiment, which may hinder those minority groups who want to express their feelings on the web freely. recent model-agnostic research mitigating the unintended biases in text classifications can be summarized as data manipulation methods @xcite . for example, @xcite propose to apply data supplementation with additional labeled sentences to make toxic/non-toxic balanced across different demographic groups. @xcite proposes to use data augmentation by applying gender-swapping to sentences with identityterms to mitigate gender bias. the core of these works is to transform the training sets to an identitybalanced one. however, data manipulation is not always practical. data supplementation often requires careful selection of the additional sentences w.r.t. the identity-terms, the labels, and even the lengths of sentences @xcite , bringing a high cost for extra data collection and annotation. data augmentation may result in meaningless sentences (e.g., "he gives birth."), and is impractical to perform when there are many demographic groups (e.g., for racial bias cases). in this paper, we propose a model-agnostic debiasing training framework that does not require any extra resources or annotations, apart from a pre-defined set of demographic identity-terms. we tackle this problem from another perspective, in which we treat the unintended bias as a kind of selection bias @xcite . we assume that there are two distributions, the non-discrimination distribution, and the discrimination distribution observed in the biased datasets, and every sample of the latter one is drawn independently from the former one following a discrimination rule, i.e., the social prejudice. with such a formalization, mitigating the unintended biases is equivalent to recovering the non-discrimination distribution from the selection bias. with a few reasonable assumptions, we prove that we can obtain the unbiased loss of the non-discrimination distribution with only the samples from the observed discrimination distribution with instance weights. based on this, we propose a non-discrimination learning framework. experiments on three datasets show that, despite requiring no extra data, our method is comparable to the data manipulation methods in terms of mitigating the discrimination of models. the rest of the paper is organized as follows. we summarize the related works in section 2. then we give our perspective of the problem and examine the assumptions of commonly-used methods in section 3. section 4 introduces our non-discrimination learning framework. taking three datasets as ex-amples, we report the experimental results of our methods in section 5. finally, we conclude and present the future works in section 6. in the nlp area, fairness and discrimination problems have also gained tremendous attention. @xcite show that semantics derived automatically from language corpora contain human biases. @xcite show that pre-trained word embeddings trained on large-scale corpus can exhibit gender prejudices and provide a methodology for removing prejudices in embeddings by learning a gender subspace. @xcite introduce the gender bias problem in coreference resolution and propose a general-purpose method for debiasing. as for text classification tasks, @xcite first points out the unintended bias in datasets and proposes to alleviate the bias by supplementing external labeled data. @xcite examines gender and race bias in 219 automatic sentiment analysis systems and finds that several models show significant bias. @xcite focus on the gender bias in abusive language detection task and propose to debias by augmenting the datasets with gender-swapping operation. in this paper, we propose to make models fit a non-discrimination distribution with calculated instance weights. instance weighting instance weighting has been broadly adopted for reducing bias. for example, the inverse propensity score (ips) @xcite ) method has been successfully applied for causal effect analyses (austin and stuart, 2015), selection bias @xcite , position bias @xcite and so on. @xcite proposed a methodology for learning and evaluating classifiers under "missing at random" (mar) @xcite selection bias. @xcite study the selection bias in natural language sentences matching datasets, and propose to fit a leakage-neutral distribution with instance weighting. @xcite propose an instance weighting framework for domain adaptation in nlp, which requires the data of the target domain. in our work, we formalize the discrimination problem as a kind of "not missing at random" (nmar) @xcite selection bias from the non-discrimination distribution to the discrimination distribution, and propose to mitigate the unintended bias with instance weighting. ## perspective in this section, we present our perspective regarding the discrimination problem in text classifications. firstly, we define what the nondiscrimination distribution is. then, we discuss what requirements non-discrimination models should meet and examine some commonly used criteria for non-discrimination. after that, we analyze some commonly used methods for assessing discrimination quantitatively. finally, we show that the existing debiasing methods can also be seen as trying to recover the non-discrimination distribution and examine their assumptions. ## our instance weighting method in this section, we introduce the proposed method for mitigating discrimination in text classifications. we first make a few assumptions about how the discrimination distribution d in the datasets are generated from the non-discrimination distribution d. then we demonstrate that we can obtain the unbiased loss on d only with the samples from d, which makes models able to fit the non-discrimination distribution d without extra resources or annotations. ## experiments in this section, we present the experimental results for non-discrimination learning. we demonstrate that our method can effectively mitigate the impacts of unintended discriminatory biases in datasets. ## conclusion in this paper, we focus on the unintended discrimination bias in existing text classification datasets. we formalize the problem as a kind of selection bias from the non-discrimination distribution to the discrimination distribution and propose a debiasing training framework that does not require any extra resources or annotations. experiments show that our method can effectively alleviate discrimination. it's worth mentioning that our method is general enough to be applied to other tasks, as the key idea is to obtain the loss on the non-discrimination distribution, and we leave this to future works. released by @xcite . the testing set is created by several templates slotted by a broad range of identity-terms, which consists of 77000 examples, 50% of which are toxic.