Conditional Supervised Contrastive Learning for Fair Text Classification (2022)

Abstract: Contrastive representation learning has gained much attention due to its superior performance in learning representations from both image and sequential data. However, the learned representations could potentially lead to performance disparities in downstream tasks, such as increased silencing of underrepresented groups in toxicity comment classification. In light of this challenge, in this work, we study learning fair representations that satisfy a notion of fairness known as equalized odds for text classification via contrastive learning. Specifically, we first theoretically analyze the connections between learning representations with a fairness constraint and conditional supervised contrastive objectives, and then propose to use conditional supervised contrastive objectives to learn fair representations for text classification. We conduct experiments on two text datasets to demonstrate the effectiveness of our approaches in balancing the trade-offs between task performance and bias mitigation among existing baselines for text classification. Furthermore, we also show that the proposed methods are stable in different hyperparameter settings.

## introduction recent progress in natural language processing (nlp) has led to its increasing use in various domains, such as machine translation, virtual assistants, and social media monitoring. however, studies have demonstrated societal bias in existing nlp models @xcite @xcite @xcite @xcite . in one major nlp application, text classification, bias is referred as the performance disparity of the trained classifiers over different demographic groups such as gender and ethnicity @xcite . such bias poses potential risks: for example, toxicity classification models in online social media platforms show disparate performance in different social groups, leading to increased silencing of underserved groups @xcite . meanwhile, an increasing line of work in contrastive learning (cl) has led to significant advances in representation learning @xcite @xcite @xcite @xcite . the general idea of contrastive learning in these works is to learn representations such that similar examples stay close to each other while dissimilar ones are far apart. inspired by those works, recent works @xcite @xcite also propose to leverage contrastive learning to learn fair representations in classification. however, these works either lack theoretical justifications for the proposed approaches or adopt demographic parity @xcite as the fairness criterion, which eliminates the perfect classifier in the common scenario when the base rates differ among demographic groups @xcite . in this work, we aim to mitigate bias in text classification models via contrastive learning. in particular, we adopt the fairness notion, equalized odds (eo) @xcite , which asks for equal true positive rates (tprs) and false positive rates (fprs) across different demographic groups @xcite . based on information-theoretic concepts, we bridge the problem of learning fair representations with equalized odds constraint with contrastive learning objectives. we then propose an algorithm, called conditional supervised contrastive learning, to learn fair text classifiers. empirically, we conduct experiments on two text classification datasets (e.g., toxic comment classification and biography classification) to show the proposed methods (1) can flexibly tune the trade-offs between main task performance and the fairness constraint; (2) achieve the best tradeoffs between main task performance and equalized odds compared to the existing bias mitigation approaches in text classification; (3) are stable to different hyperparameter settings, such as data augmentations, temperatures, and batch sizes. to the best of our knowledge, our work is the first to both theoretically and empirically study how to ensure the eo constraint via contrastive learning in text classification. ## background we use @xmath0, respectively. furthermore, @xmath1.g., the gender information in the occupation classification task). the corresponding lowercase letters denote the instantiation of the random variables. given a text encoder f : @xmath2.g., bert @xcite ) and a classifier g : @xmath3, we first transform the input text x into latent representation z via f , and z is used to give a prediction ŷ via g in the context of contrastive learning, data augmentation strategies have been widely adopted. let t be a set of data augmentations and x ′ be the augmented input given the data augmentation t(•): x ′ = t(x), t ∼ t , where we assume that the augmentation t is sampled uniformly at random from t . similarly, we have let h denote the entropy and i denote the mutual information, e.g., h(z | z ′ , y ) is the conditional entropy of z given z ′ and y , and i(z ′ ; z | y ) is the conditional mutual information of z ′ and z given y . due to the space limit, we refer readers to cover (1999) for more background knowledge of the related notions (entropy and mutual information) in information theory. we assume there is a joint distribution over x, y , and a from which the data are sampled. figure 1 shows the graphical model of the dependencies between input variables and outputs. we also assume that the sensitive attribute a is available only during model training, but it is not available during the testing phase. as a result, any post-processing methods that leverage sensitive attributes for bias mitigation during the testing phase are not feasible in our setting. in this work, we use figure 1: graphical model of the dependencies between input variables and outputs. note that we only assume there is a joint distribution over x, y , and a from which the data are sampled, so the figure only shows one case of the dependencies over x, y , and a. definition 2.1 (equalized odds @xcite ). a model satisfies equalized odds if at a high level, eo asks the model prediction to be independent of the sensitive attribute conditioned on the task label. if a model perfectly satisfies equalized odds, the differences of true positive rates and false positive rates across demographic groups will be 0. equivalently, it also implies i( ŷ ; a | y ) = 0. consider online comment toxicity classification as a real-world example to motivate the use of eo as a notion of fairness. in this case, false positive cases (benign text comments marked as toxic) can be seen as unintentional censoring, and false negative cases (toxic text comments marked as benign) might result in debates and discomforts @xcite . in contrast to another well-known group fairness definition, i.e., demographic parity, eo does not require positive prediction rates to be the same across different demographic groups, which could possibly severely downgrade the model performance when the sensitive attribute is correlated to the task label @xcite . ## our method in this section, we first theoretically connect learning fair representations with contrastive learning (sec. 3.1). in particular, we first show that learning fair representations for equalized odds requires the minimization of i(z ′ ; z | y ) and the simultane-ous maximization of i(z ′ ; z | a, y ). to this end, we provide an upper bound of i(z ′ ; z | y ) and a lower bound of i(z ′ ; z | a, y ) to relax the original objective and then establish a relationship between the bounds and the (conditional) supervised contrastive learning objectives. finally, inspired by our theoretical analysis, we design two practical methods for learning fair representations (sec. 3.2). due to the space limit, we defer all detailed proofs to appendix a. ## experiments in this section, we conduct experiments to investigate the following research questions: rq 1. how can we control the trade-offs between model classification performance and fairness via conditional supervised contrastive learning? rq 2. how do conditional supervised contrastive learning methods perform in terms of tradeoffs between model performance and fairness compared to other in-processing bias mitigation methods in text classification? rq 3. is conditional supervised contrastive learning sensitive to hyperparameter changes? ## related work unintended social biases in nlp models have been identified in word/sentence embedding @xcite @xcite and applications such as coreference resolution @xcite @xcite , language modeling @xcite , machine translation @xcite , and text classification @xcite . in the literature, there are some recent works that aim to learn fair representations via contrastive learning @xcite @xcite . among these works, @xcite propose contrastive objectives to learn debiased sentence embeddings that minimize the correlation between embedded sentences and biased words. in classification tasks, @xcite proposed contrastive objectives to remove sensitive information; @xcite proposed a similar contrastive objective to achieve a similar goal. according to @xcite , all those proposed contrastive objectives target demographic parity in principle. our theoretical results involve key notions (e.g., entropy and mutual information) in information theory @xcite . information-theoretic-based methods have been used for representation learning for nlp applications. for example, @xcite proposed a variational upper bound of mutual information to learn disentangled textual representations for fair classification and style transfer. compared to the previous work, our work uses equalized odds as the fairness criterion. to the best of our knowledge, our work is the first to connect the problem of learning fair representations with contrastive learning to ensure the eo constraint and explore its effectiveness for bias mitigation in text classification in large language models (e.g., bert). ## conclusion in this paper, we theoretically and empirically study how to leverage contrastive learning for fair text classification. inspired by our theoretical results, we propose conditional supervised contrastive objectives to learn aligned and uniform representations while mixing the representation of different examples that share the same sensitive attribute for every task label. we conduct experiments to demonstrate the effectiveness of our algorithms in learning fair representations for text classification and show that our methods are stable in different hyperparameter settings. in the future, we plan to extend our algorithms to the settings of intersectional bias @xcite .