U pstream M itigation I s A ll Y ou N eed: T esting the B ias T ransfer H ypothesis in P re- T rained L anguage M odels (2022)

Abstract: A few large, homogenous, pre-trained models undergird many machine learning systems — and often, these models contain harmful stereotypes learned from the internet. We investigate the bias transfer hypothesis : the theory that social biases (such as stereotypes) internalized by large language models during pre-training transfer into harmful task-specific behavior after fine-tuning. For two classification tasks, we find that reducing intrinsic bias with controlled interventions before fine-tuning does little to mitigate the classifier’s discriminatory behavior after fine-tuning. Regression analysis suggests that downstream disparities are better explained by biases in the fine-tuning dataset. Still, pre-training plays a role: simple alterations to co-occurrence rates in the fine-tuning dataset are ineffective when the model has been pre-trained. Our results encourage practitioners to focus more on dataset quality and context-specific harms.

## introduction large language models (llms) and other massively pre-trained "foundation" models are powerful tools for task-specific machine learning @xcite . models pre-trained by wellresourced organizations can easily adapt to a wide variety of downstream tasks in a process called finetuning. but massive pre-training datasets and increasingly homogeneous model design come with well-known, immediate social risks beyond the financial and environmental costs @xcite . transformer-based llms like bert and gpt-3 contain quantifiable intrinsic social biases encoded in their embedding spaces @xcite . these intrinsic biases are typically associated with representational harms, including stereotyping and denigration @xcite @xcite . separately, many studies document the extrinsic harms of the downstream (fine-tuned & task-specific) ap-figure 1 : full pre-training to fine-tuning pipeline, with experimental interventions (green hexagons). despite these risks, no research has investigated the extent to which downstream systems inherit social biases from pre-trained models. 1 many stud-ies warn that increasing intrinsic bias upstream may lead to an increased risk of downstream harms @xcite . this hypothesis, which we call the bias transfer hypothesis, holds that stereotypes and other biased associations in a pre-trained model are transferred to post-fine-tuning downstream tasks, where they can cause further, task-specific harms. a weaker version of this hypothesis holds that downstream harms are at least mostly determined by the pretrained model @xcite . in the pre-training paradigm, the extent to which the bias transfer hypothesis holds will determine the most effective strategies for responsible design. in the cases we study, reducing upstream bias does little to change downstream behavior. still, there is hope: instead, developers can carefully curate the fine-tuning dataset, checking for harms in context. we test the bias transfer hypothesis on two classification tasks with previously demonstrated performance disparities: occupation classification of online biographies @xcite and toxicity classification of wikipedia talks comments @xcite . we investigate whether reducing or exacerbating intrinsic biases encoded by roberta @xcite decreases or increases the severity of downstream, extrinsic harms (figure 1 ). we find that the bias transfer hypothesis describes only part of the interplay between pretraining biases and harms after fine-tuning: without absolving llms or their owners of representational harms intrinsic to pre-trained models, our results encourage practitioners and application stakeholders to focus more on dataset quality and context-specific harm identification and reduction. ## related work little prior work directly tests the bias transfer hypothesis. the closest example of this phenomena is the "blood diamond" effect @xcite , in which stereotyping and denigration in the pre-training corpora pervade subsequently generated images and language even before the fine-tuning stage @xcite . still, it is unclear to what extent undesirable values encoded in pre-training datasets or benchmarks-such as wikipedia or imagenetinduce task-specific harms after fine-tuning @xcite . some work explores the consistency of intrinsic and extrinsic bias metrics: @xcite find that intrinsic and extrinsic metrics are not reliably correlated for static embeddings like word2vec. we focus instead on stateof-the-art transformer-based llms-the subject of intense ethical debate @xcite -which construct contextual, rather than static, embeddings. contextual embeddings-token encodings that are conditional on other, nearby tokens-pose an ongoing challenge for intrinsic bias measurement @xcite @xcite and bias mitigation @xcite . we find that intrinsic and extrinsic metrics are correlated for the typical llm-but that the correlation is mostly explained by biases in the fine-tuning dataset. other research tests the possibility that upstream mitigation could universally prevent downstream harm. @xcite show that an intermediate, bias-mitigating fine-tuning step can help reduce bias in later tasks. likewise, @xcite propose fine-tuning on carefully curated "values-targeted" datasets to reduce toxic gpt-3 behavior. our results tend to corroborate these methods: we find that the fine-tuning process can to some extent overwrite the biases present in the original pre-trained model. a recent post-hoc mitigation technique, on the other hand, debiases contextual embeddings before fine-tuning @xcite . our results imply that while this type of debiasing may help with representational harms upstream, it is less successful for reducing harms downstream. ## methods to empirically evaluate the bias transfer hypothesis, we examine the relationship between upstream bias and downstream bias for two tasks. we track how this relationship changes under various controlled interventions on the model weights or the fine-tuning dataset. ## experiments we measure changes in upstream and downstream bias subject to the following interventions (fig. 1 ): ## limitations our approach comes with several limitations. first, our results may not generalize to all tasksespecially non-classification tasks-or all kinds of bias (e.g., bias against aave or non-english speakers). also, while similar studies of bias have been successfully applied to vision transformers @xcite , our results may vary for substrates other than english language. second, @xcite conclude that the lack of correlation between intrinsic bias indicators and downstream bias is because some embedding bias metrics are unsuitable for measuring model bias. to ensure our intrinsic and extrinsic metrics measure the same construct, we chose upstream indicators that correlate with realworld occupation statistics @xcite . pronoun ranking in particular may be more reliable for transformer models than a combination of scrubbing and not pre-training (orange) results in a zero-mean, noticeably re-ordered bias distribution. scrubbing but still pretraining (purple) results in a bias distribution that is still correlated with the original bias distribution (pearson's ⇢ = 0.99, 0.93 for toxic and all respectively, @xmath0.01). third, there may be other explanations for the relationship between upstream and downstream bias: for example, decreasing the magnitude of upstream bias often requires a reduction in model accuracy, though we attempt to control for between-model variation with fixed effects and other controls. alternate regression specifications included in appendix c.1 show how our results change with the inclusion of controls. ## conclusion our results offer several points of guidance to organizations training and distributing llms and the practitioners applying them: but, our results also corroborate a nascent, somewhat optimistic view of pre-training bias. llms' intrinsic biases are harmful even before downstream applications, and correcting those biases is not guaranteed to prevent downstream harms. increased emphasis on the role of fine-tuning dataset bias offers an opportunity for practitioners to shift to more careful, quality-focused and context-aware approach to nlp applications @xcite .