Trade-Offs Between Fairness and Privacy in Language Modeling (2023)

Abstract: Protecting privacy in contemporary NLP models is gaining in importance. So does the need to mitigate social biases of such models. But can we have both at the same time? Existing research suggests that privacy preservation comes at the price of worsening biases in classification tasks. In this paper, we explore the extent to which this tradeoff really holds when we incorporate both privacy preservation and de-biasing techniques into training text generation models. How does improving the model along one dimension affect the other dimension as well as the utility of the model? We conduct an extensive set of experiments that include bias detection, privacy attacks, language modeling, and performance on downstream tasks.

## introduction fairness and privacy are two important concepts in contemporary nlp. unfairness caused by demographic biases can lead to unequal performance for different user groups @xcite , misidentification of speakers and their needs @xcite , or propagation of hurtful stereotypes @xcite . in addition, when nlp models leak data, it can lead to the disclosure of sensitive personal data which can hurt individuals @xcite . in an attempt to provide both privacy and fairness in nlp classifiers, existing research suggests an inherent trade-off between the two dimensions @xcite @xcite . introducing privacy may amplify bias in some social groups more than others, more specifically those groups that were already underrepresented and therefore a minority in the data. for example, @xcite find that classifiers across 1 our code is publicly available: https://github. com/cleolotta/fair-and-private-lm four diverse classification tasks perform worse for underrepresented groups due to the effects of gradient clipping implemented in differential privacy @xcite . however, current research on trade-offs between privacy and fairness in large language models remains inconclusive. in this work, we aim to fill this research gap by investigating language modeling under privacy and de-biasing paradigms. our research deals with scenarios in which there is arguably no quantitative minority group (our focus is on gender bias), as opposed to labeled data in fine-tuning used in previous works. we ask how fairness and privacy affect each other in this context, exploring differential privacy and two different debiasing objectives during fine-tuning stages. we examine how each objective in isolation and jointly affects (1) privacy, measured in terms of data leakage, and (2) biases, evaluated across three popular recent bias evaluation benchmarks. specifically, our paper aims to answer the following research questions: rq1: does training with a differential privacy objective lead to fairer lms? rq2: does training with debiasing objective lead to less leakage? rq3: how does training with debiasing as well as dp objective affect fairness and privacy? rq4: how does training with debiasing and/or dp objective affect the language ability in the resulting model? rq5: how does training with debiasing and/or dp objective affect downstream nlu performance? to our best knowledge, ours is the first study exploring such effects on language modeling. ## related work bias detection a test for detecting biases in word embeddings is the word embedding association test (weat; @xcite ) which computes the association between two target word sets with words from two attribute sets in vector space. an extension of this to sentence-level representations was created by @xcite . bias evaluation corpus with professions (bec-pro; @xcite and discovery of correlations (disco; @xcite are datasets that use predefined templates to determine gender bias with regard to different professions and other characteristics. @xcite further introduced the winobias benchmark in which a corpusbased on the winograd challenge @xcite -follows a certain scheme, each containing a person, a pronoun and an occupation. a model would pass the winobias test if the two binary genders were hit with the same accuracy. stereoset @xcite represents a crowd-sourced dataset through which it can be determined with what proportion a model meets a stereotypical association in terms of gender, occupation, race, and religion instead of the anti-stereotypical one. biasin-bios @xcite uses a dataset created from biographies found on the web containing a person's profession and asks a model to read the biographies and recognise the profession without making gender-based assumptions. bias-mitigation methods several methods have been proposed for mitigating a bias. @xcite proposed dropout as debiasing technique and aimed at reducing gender correlations through increasing dropout regularization. counterfactual data augmentation (cda; @xcite ) is a commonly used approach @xcite @xcite in which a dataset is practically rebalanced by exchanging bias attribute words (e.g. pronouns) in an automated process. @xcite proposed another method to mitigate biases in word embeddings, namely iterative nullspace projection (inlp). inlp aims to find a linear guardian function that removes the linear dependency between word embeddings and their associated protected attributes, which should not be considered in the decision of a fair classifier. self-debias @xcite poses a post-hoc text generation debiasing technique that does not change the model's internal representations. in this approach, the model is asked to make a biased statement, instead of an unbiased statement. the resulting probability distribution is then used to change the model's initial output distribution. ## methods, metrics, and datasets in the following, we introduce (1) datasets and methods to measure bias, (2) techniques to measure privacy, and (3) datasets to model the language modeling ability of our language models used in our work. ## conclusion existing literature has found a negative trade-off between differential privacy and fairness in nlp classifiers that results in minorities being classified worse, thus with lower accuracy @xcite @xcite . in our work, we explored this trade-off in language modeling with transformers. in particular, we applied debiasing methods and differential privacy to the pre-trained gpt-2 model in six different experimental setups and investigated their mutual effects measured by several complementary performance metrics. we found positive results when combining these two paradigms. first, the debiasing method cda combined with dp protects against membership inference attacks more than dp by itself. second, unlike previously found in classification models, we did not observe a negative effect of dp on fairness in language models. finally, it is worth highlighting that in training with both debiasing and privacy objective, cda mitigated the negative impact of dp on language modeling ability.