Characterizing Stereotypical Bias from Privacy-preserving Pre-Training (2024)

Abstract: Differential Privacy (DP) can be applied to raw text by exploiting the spatial arrangement of words in an embedding space. We investigate the implications of such text privatization on Language Models (LMs) and their tendency towards stereotypical associations. Since previous studies documented that linguistic proficiency correlates with stereotypical bias, one could assume that techniques for text privatization, which are known to degrade language modeling capabilities, would cancel out undesirable biases. By testing BERT models trained on texts containing biased statements primed with varying degrees of privacy, our study reveals that while stereotypical bias generally diminishes when privacy is tightened, text privatization does not uniformly equate to diminishing bias across all social domains. This highlights the need for careful diagnosis of bias in LMs that undergo text privatization.

## introduction language models (lms) @xcite are trained on large corpora of text that may contain confidential information. since such information can be recovered from word embeddings @xcite and language models @xcite , privacy emerged as an active concern for building trust and complying with stringent regulations on privacy protection. to protect against unintended disclosure of information, differential privacy (dp) @xcite has been integrated into machine learning @xcite and language models @xcite @xcite . dp formalizes privacy through a notion of indistinguishability so that the model outputs are not affected by the addition or removal of an entry in the training corpus. this is accomplished by injecting additive noise on gradients during model training. due to scaling issues associated with dp on lms during perturbation of per-sample gradient updates @xcite , there is a trend towards perturbing the raw text @xcite @xcite . by exploiting the geometric proximity of words in word embeddings @xcite , @xcite proposed a probabilistic mechanism grounded in metric dp @xcite to perturb all words in a text while ensuring plausible deniability @xcite of the text regarding its provenance and content. however, several studies documented that mechanisms for embedding words in a high-dimensional space harbor @xcite @xcite and transfer @xcite unwanted stereotypes and prejudices present in a text corpus. contribution. building on the rich body of research exploring privacy-fairness trade-offs @xcite @xcite , this study addresses the implications of text privatization on biased associations in lms. specifically, we pre-train bert @xcite models with masked language modeling and next sentence prediction on webscraped text modified under varying levels of privacy. we then score the stereotypical bias following the context association test of @xcite and stereotype pairs benchmark of @xcite . our findings reveal a nuanced landscape where stereotypical bias generally diminishes as privacy guarantees are tightened. this is in line with prior research indicating that lms with impaired language modeling capabilities tend to exhibit less stereotypical associations @xcite . however, this diminution is not uniform across all social categories as biases associated with certain attributes show varying trends of stability, amplification, and attenuation. we thus advocate for careful bias measurement when deploying privacy-preserving lms. ## background to ensure a consistent understanding of privacy and fairness in machine learning, we provide the foundations of differential privacy and a brief definition of stereotypical bias along with related work. ## methodology to test our hypothesis on amplification of stereotypical bias through text privatization, we need to define (1) a language model, (2) the mechanism for text privatization, and (3) a bias measurement. ## experiments prior to initiating our bias measurement, we conducted a preliminary sanity check by examining the pseudo-perplexity scores of bert models trained under varying degrees of privacy. pseudoperplexity serves an indicator of a lm's ability to accurately model the probability distribution of words within a text corpus, thereby reflecting the model's proficiency to comprehend the linguistic structures encountered during its training. we use a 10% subset of wikitext for computing the pseudo-perplexities. evaluated at privacy levels specified by the privacy parameter ε, the pseudo-perplexity scores were 93.51 with no privacy interventions, 502.67 with moderate privacy settings, and 2056.43 under conditions of high privacy. consistent with previous evidence that introducing noise at word-level compromises the linguistic proficiency of lms @xcite , these results demonstrate a substantial degradation as the level of privacy augmentation increases. the observed degradation raises an interesting question of whether private lms harbor stereotypical biases despite diminished language modeling capabilities. this question forms the basis for our subsequent analysis of the undesirable biases in lms stemming from text privatization. ## conclusion the interaction dynamics that govern the manifestation of bias in lms are equivocal @xcite . prior research indicates that stereotypical bias is related to language proficiency in lms @xcite . since text privatization is known to impair language modeling capabilities @xcite , one would expect a general diminution of stereotypical bias. however, the word embeddings used for text privatization are documented to harbor @xcite and transfer (papakyriakopoulos et al., 2020) stereotypical biases. this duality raises questions about whether text privatization leads to an amplification or an attenuation of stereotypical biases. by probing a lms tendency to default to stereotypical or anti-stereotypical associations, we aimed to elucidate the relationship between text privatization and the amplification or attenuation of biases. we find that different social domains react differently to privacy settings and recommend to carefully assess stereotypical bias after training a lm on a privatized corpus of text.