Fair NLP Models with Differentially Private Text Encoders (2022)

Abstract: Encoded text representations often capture sensitive attributes about individuals (e.g., race or gender), which raise privacy concerns and can make downstream models unfair to certain groups. In this work, we propose FEDERATE, an approach that combines ideas from differential privacy and adversarial training to learn private text representations which also induces fairer models. We empirically evaluate the trade-off between the privacy of the representations and the fairness and accuracy of the downstream model on four NLP datasets. Our results show that FEDERATE consistently improves upon previous methods, and thus suggest that privacy and fairness can positively reinforce each other.

## introduction algorithmically-driven decision-making systems raise fairness concerns @xcite as they can be discriminative against specific groups of people. these systems have also been shown to leak sensitive information about the data of individuals used for training or inference, and thus pose privacy risks @xcite . societal pressure as well as recent regulations push for enforcing both privacy and fairness in real-world deployments, which is challenging as these notions are multi-faceted concepts that need to be tailored to the context. moreover, privacy and fairness can be at odds with one another: recent studies have shown that preventing a model from leaking information about its training data negatively impacts the fairness of the model and vice versa @xcite @xcite . this paper studies fairness and privacy and their interplay in the nlp context, where these two notions have often been considered independently from one another. modern nlp heavily relies on learning or fine-tuning encoded representations of text. unfortunately, such representations often leak sensitive attributes (e.g., gender, race, or age) present explicitly or implicitly in the input text, even when such attributes are known to be irrelevant to the task @xcite . moreover, the presence of such information in the representations may lead to unfair downstream models, as has been shown on various nlp tasks such as occupation prediction from text bios @xcite , coreference resolution @xcite , or sentiment analysis @xcite . privatizing encoded representations is thus an important, yet challenging problem for which existing approaches based on subspace projection @xcite @xcite or adversarial learning @xcite @xcite do not provide a satisfactory solution. in particular, these methods lack any formal privacy guarantee, and it has been shown that an adversary can still recover sensitive attributes from the resulting representations with high accuracy @xcite . instead of relying on adversarial learning to prevent attribute leakage, @xcite ; @xcite recently propose to add random noise to text representations so as to satisfy differential privacy (dp), a mathematical definition which comes with rigorous guarantees @xcite . however, we uncover a critical error in their privacy analysis which drastically weakens their privacy claims. moreover, their approach harms accuracy and fairness compared to adversarial learning. in this work, we propose a novel approach (called federate) to learn private text representations and fair models by combining ideas from dp with an adversarial training mechanism. more specifically, we propose a flexible end-to-end architecture in which (i) the output of an arbitrary text encoder is normalized and perturbed using random noise to make the resulting encoder differentially private, and (ii) on top of the encoder, we combine a classifier branch with an adversarial branch to actively induce fairness, improve accuracy and further hide specific sensitive attributes. we empirically evaluate the privacy-fairnessaccuracy trade-offs achieved by federate over four datasets and find that it simultaneously leads to more private representations and fairer models than state-of-the-art methods while maintaining comparable accuracy. beyond the superiority of our approach, our results bring valuable insights on the complementarity of dp and adversarial learning and the compatibility of privacy and fairness. on the one hand, dp drastically reduces undesired leakage from adversarially trained representations, and has a stabilizing effect on the training dynamics of adversarial learning. on the other hand, adversarial learning improves the accuracy and fairness of models trained over dp text representations. our main contributions are as follows: the paper is organized as follows. section 2 provides background on differential privacy. section 3 presents our approach. section 4 reviews related work. experimental results and conclusions are given in sections 5 and 7. ## background: differential privacy differential privacy (dp) @xcite provides a rigorous mathematical definition of the privacy leakage associated with an algorithm. it does not depend on assumptions about the attacker's capabilities and comes with a powerful algorithmic framework. for these reasons, it has become a de-facto standard in privacy currently used by the us census bureau (abowd, 2018) and several big tech companies @xcite @xcite . this section gives a brief overview of dp, focusing on the aspects needed to understand our approach (see @xcite for an in-depth review of dp). over the last few years, two main models for dp have emerged: (i) central dp (cdp) @xcite , where raw user data is collected and processed by a trusted curator, which then releases the result of the computation to a third party or the public, and (ii) local dp (ldp) @xcite which removes the need for a trusted curator by having each user locally perturb their data before sharing it. our work aims to create an encoder that leads to a private embedding of an input text, which can then be shared with an untrusted curator for learning or inference. we thus consider ldp, defined as follows. definition 1 (local differential privacy). a randomized algorithm m : @xmath0, @xmath1: ldp ensures that the probability of observing a particular output o of m should not depend too much on whether the input is x or x . the strength of privacy is controlled by , which bounds the log-ratio of these probabilities for any x, x . @xmath2, @xmath3. in our approach described in section 3, x will be an input text and m will be an encoding function which transforms x into a private vector representation that can be safely shared with untrusted parties. in our work, we propose an architecture in which the laplace mechanism is applied on top of a trainable encoder to get private representations of input texts, and is further combined with adversarial training to learn fair models. ## approach we consider a scenario similar to @xcite , where a user locally encodes its input data (text) x into an intermediate representation e priv (x) which is then shared with an untrusted curator to predict the label y associated with x using a classifier c. additionally, an attacker (which may be the untrusted curator or an eavesdropper) may observe the intermediate representation e priv (x) and try to infer some sensitive (discrete) attribute z about x (e.g., gender, race etc.). our goal is to learn an encoder e priv and classifier c such that (i) the attacker performs poorly at inferring z from e priv (x), (ii) the classifier c(e priv (x)) is fair with respect to z according to some fairness metric, and (iii) c accurately predicts the label y. to achieve the above goals we introduce federate (for fair models with differentially private text encoders), which combines two components: a differentially private encoder and an adversarial branch. figure 1 shows an overview of our proposed architecture. ## related work adversarial learning. in order to improve model fairness or to prevent leaking sensitive attributes, several approaches employ adversarialbased training. for instance, @xcite propose to use a different adversary for each protected attribute, while @xcite consider additional loss components to improve the privacyaccuracy trade-off of the learned representation. @xcite introduce multiple adversaries focusing on different aspects of the representation by encouraging orthogonality between pairs of adversaries. recently, @xcite propose an adversarial scrubbing mechanism. however, they purely focus on information leakage, and not on fairness. moreover, unlike our approach, these methods do not offer formal privacy guarantees. in fact, it has been observed that one can recover the sensitive attributes from the representations by training a post-hoc non linear classifier @xcite . this is confirmed by our empirical results in section 5. several works have also explored the use of adversarial learning in inducing fairness. for instance, @xcite explore the effect of data distribution during fair adversarial training, while @xcite propose various adversarial objective and connects them with different group fairness measure. however, unlike our work, they do not consider fairness and privacy at the same time. ## experiments recall that we are interested in approaches that are not only accurate but also fair and private at the same time. however, these three dimensions are not independent and are not straightforwardly amenable to a single evaluation metric. thus, we present experiments aiming at (i) showcasing the privacy-fairness-accuracy tradeoffs of different approaches and then (ii) analyzing privacy-accuracy and fairness-accuracy tradeoffs separately. we begin by describing the datasets and the metrics. datasets. we consider 4 different datasets: (i) twitter sentiment (blodgett et al., 2016) consists of 200k tweets annotated with a binary sentiment label and a binary "race" attribute corresponding to african american english (aae) vs. standard american english (sae) speakers; (ii) bias in bios (de-arteaga et al., 2019) consists of 393,423 textual biographies annotated with an occupation label (28 classes) and a binary gender attribute; (iii) celeba @xcite is a binary classification dataset with a binary sensitive attribute (gender); (iv) adult income @xcite consists of 48,842 instances with binary sensitive attribute (gender). our setup for the first two dataset is similar to @xcite and @xcite . appendix d.2 provides detailed description of these datasets, including sizes, pre-processing, and the challenges they pose to privacy and fairness tasks. due to lack of space, results and analyses for adult income and celeba dataset are given in appendix d.5, but note that they exhibit similar trends. the preprocessed versions of the datasets can be downloaded from this anonymized url. 2 fairness metrics. for twitter sentiment we report the true positive rate gap (tpr-gap), which measures the true positive rate difference between the two sensitive groups (gender/race) and is closely related to the notion of equal opportunity. formally, denoting by @xmath4, 1} the ground truth binary label, ŷ the predicted label and @xmath5, ¬g} the sensitive attribute, tpr-gap is defined as: tpr-@xmath6. for bias in bios, which has 28 classes, we follow @xcite and report the root mean square of tpr-gaps (grms) over all occupations @xmath7: note that for grms essentially boils down to tpr-gap in binary setting. privacy metrics. we report two metrics for privacy: (i) leakage: the accuracy of a two-layer classifier which predicts the sensitive attribute from the encoded representation, and (ii) minimum description length (mdl) (voita and titov, 2020), which quantifies the amount of "effort" required by such a classifier to achieve a certain accuracy. a higher mdl means that it is more difficult to retrieve the sensitive attribute from the representation. the metric depends on the dataset and the representation dimension, and thus cannot be compared across different datasets. we provide more details about these metrics in sec. d.1. methods and model architectures. we compare federate to the following methods: (i) adversarial implements standard adversarial learning @xcite , which is equivalent to our approach without the priv layer, (ii) adversarial + multiple @xcite implements multiple adversaries, (iii) inlp @xcite ) is a subspace projection approach, and (iv) noise learns dp text representations as proposed by @xcite but with corrected privacy analysis: this corresponds to our approach without the adversarial component. these methods have been described in details in section 4 and their hyperparametrs in appendix d.4. we also report the performance of two simple baselines: random simply predicts a random label, and unconstrained optimizes the classification performance without special consideration for privacy or fairness. to provide a fair comparison, all methods use the same architecture for the encoder, the classifier and (when applicable) the adversarial branches. in order to evaluate across varying model complexities, we employ different architectures for the different datasets. for twitter sentiment, we follow the architecture employed by @xcite , while for bias in bios we use a deeper architecture. the exact architecture, hyperparameters, and their tuning details are provided in appendix d.3-d.4. we implement federate in pytorch @xcite . our implementation, training, and evalua-tion scripts are available here. foot_3 ## limitations a current limitation of this work in the context of fairness is that it is not designed to work with a specific definition of fairness, such as equal odds. instead, it enforces fairness by removing certain protected information, which can correlate with specific fairness notions. similarly, we do not provide formal fairness guarantees for our method as we do for privacy. we also do not provide privacy of training data, i.e., protection against reconstruction attacks. it is also necessary for the practitioner to monitor the fairness levels of the model over time, as due to data drift and other changes, the model's fairness level might change. ## conclusion and perspectives we proposed a dp-driven adversarial learning approach for nlp. through our experiments, we showed that our method simultaneously induces private representations and fair models, with a mutually reinforcing effect between privacy and fairness. we also find that our approach improves upon competitors on each dimension separately. while we focused on privatizing sensitive attributes like race or gender, our approach can be used to remove other types of unwanted information from text representations, such as tenses or pos tag information, which might not be relevant for certain nlp tasks. in the future, we aim to investigate fairness methods that explicitly optimize for a specific fairness definition and explore other privacy threats (e.g., reconstruction attacks).