Sustainable Modular Debiasing of Language Models (2021)

Abstract: Unfair stereotypical biases (e.g., gender, racial, or religious biases) encoded in modern pretrained language models (PLMs) have negative ethical implications for widespread adoption of state-of-the-art language technology. To remedy for this, a wide range of debiasing techniques have recently been introduced to remove such stereotypical biases from PLMs. Existing debiasing methods, however, directly modify all of the PLMs parameters, which – besides being computationally expensive – comes with the inherent risk of (catastrophic) forgetting of useful language knowledge acquired in pretraining. In this work, we propose a more sustainable modular debiasing approach based on dedicated debiasing adapters, dubbed ADELE. Concretely, we (1) inject adapter modules into the original PLM layers and (2) update only the adapters (i.e., we keep the original PLM parameters frozen) via language modeling training on a counterfactually augmented corpus. We showcase ADELE, in gender debiasing of BERT: our extensive evaluation, encompassing three intrinsic and two extrinsic bias measures, renders ADELE, very effective in bias mitigation. We further show that – due to its modular nature – ADELE, coupled with task adapters, retains fairness even after large-scale downstream training. Finally, by means of multilingual BERT, we successfully transfer ADELE, to six target languages.

## introduction recent work has shown that pretrained language models such as elmo @xcite , bert @xcite , or gpt-2 @xcite tend to exhibit a range of stereotypical societal biases, such as racism and sexism (e.g., @xcite @xcite @xcite . the reason for this lies in the distributional nature of these models: human-produced corpora on which these models are trained are abundant with stereotypically biased concept cooccurrences (for instance, male terms like man or son appear more often together with certain career terms like doctor or programmer than female terms like women or daughter) and the plms models, being trained with language modeling objectives, consequently encode these biased associations in their parameters. while this effect can lend itself to diachronic analysis of societal biases (e.g., @xcite , it represents stereotyping, one of the main types of representational harm @xcite and, if unmitigated, may cause severe ethical issues in various sociotechnical deployment scenarios. to alleviate this problem and ensure fair language technology, previous work introduced a wide range of bias mitigation methods (e.g., @xcite @xcite . all existing debiasing approaches, however, modify all parameters of the plms which has two prominent shortcomings: (1) it comes with a high computational cost foot_0 and (2) can lead to (catastrophic) forgetting @xcite of the useful distributional knowledge obtained during pretraining. for example, @xcite incorporate counterfactual debiasing already into bert's pretraining: this implies a debiasing framework in which a separate "debiased bert" instance needs to be trained from scratch for each individual bias type and specification. in sum, current debiasing procedures designed for pretraining or full fine-tuning of plms have a large carbon footprint @xcite and consequently jeopardize the sustainability @xcite of fair representation learning in nlp. in this work, we move towards more sustainable removal of stereotypical societal biases from pretrained language models. to this end, we propose adele (adapter-based debiasing of language models), a debiasing approach based on the the recently proposed modular adapter framework @xcite . in adele, we inject additional parameters, the socalled adapter layers into the layers of the plm and incorporate the "debiasing" knowledge only in those parameters, without changing the pretrained knowledge in the plm. we show that, while being substantially more efficient (i.e., sustainable) than existing state-of-the-art debiasing approaches, adele is just as effective in bias attenuation. contributions. the contributions of this work are three-fold: (i) we first present adele, our novel adapter-based framework for parameterefficient and knowledge-preserving debiasing of plms. we combine adele with one of the most effective debiasing strategies, counterfactual data augmentation (cda; @xcite , and demonstrate its effectiveness in gender-debiasing of bert @xcite , the most widely used plm. (ii) we benchmark adele in what is arguably the most comprehensive set of bias measures and data sets for both intrinsic and extrinsic evaluation of biases in representation spaces spanned by plms. additionally, we study a previously neglected effect of fairness forgetting present when debiased plms are subjected to large-scale downstream training for specific tasks (e.g., natural language inference, nli); we show that adele's modular nature allows to counter this undesirable effect by stacking a dedicated task adapter on top of the debiasing adapter. (iii) finally, we successfully transfer adele's debiasing effects to six other languages in a zero-shot manner, i.e., without relying on any debiasing data in the target languages. we achieve this by training the debiasing adapter stacked on top of the multilingual bert on the english counterfactually augmented dataset. ## adele: adapter-based debiasing in this work, we seek to fulfill the following three desiderata: (1) we want to achieve effective debiasing, comparable to that of existing state-ofthe-art debiasing methods while (2) keeping the training costs of debiasing significantly lower; and the debiasing adapters could, in principle, be trained using any of the debiasing strategies and training objectives from the literature, e.g., via additional debiasing loss objectives @xcite ; @xcite ; @xcite or data-driven approaches such as counterfactual data augmentation @xcite . for simplicity, we opt for the data-driven cda approach: it has been shown to offer reliable debiasing performance @xcite and, unlike other approaches, it does not require any modifications of the model architecture nor training procedure. ## experiments we showcase adele for arguably the most explored societal bias -gender bias -and the most widely used plm, bert. we profile its debiasing effects with a comprehensive set of intrinsic and downstream (i.e., extrinsic) evaluations. ## results and discussion monolingual evaluation. our main monolingual english debiasing results on three intrinsic and two extrinsic benchmarks are summarized in the results from two extrinsic evaluations -sts and nli -demonstrate that adele successfully attenuates the bias, while retaining the high task performance. zari variants yield slightly better task performance for both sts-b and mnli: this is expected, as they are instances of the bert large transformer with 336m parameters; in comparison, adele has only 110m parameters of bert base and approx. 885k adapter parameters. 7 according to weat evaluation on static embeddings extracted from bert ( §3.1), the original bert transformer is only slightly and insignificantly biased. consequently, adele inverts the bias in the opposite direction. in figure 1 , we further analyze the weat bias effects w.r.t. the subset of bert layers from which we aggregate the word embeddings. for the original bert (figure 1a ), we obtain the gender unbiased embeddings if we aggregate representations from higher layers (e.g., [5:12], [6:9], or by taking final layer vectors, [12:12]). for adele, we get the most genderneutral embeddings by aggregating representations from lower layers (e.g., [0:3] or [1:3]); representations from higher layers (e.g., [6:12]) flip the bias into the opposite direction (blue color). both zari models produce embeddings which are relatively unbiased, but zari cda still exhibits slight gender bias in higher layer representations. the dropoutbased debiasing of zari do results in an interesting per-layer-region oscillating gender bias. zero-shot cross-lingual transfer. we show the results of zero-shot transfer of gender debiasing with adele (on top of mbert) on german bec-pro in in like bec-pro, weat confirms that adele also attenuates the bias in en representations coming from mbert. the results across the six target languages are somewhat mixed, but overall encouraging: for all significantly biased combinations of languages and layer aggregations from original mbert ([0:12] -it, ru; [1:12] -hr, ru), adele successfully reduces the bias. e.g., for it embeddings extracted from all layers ([0:12]), the bias effect size drops from significant 1.02 to insignificant -0.25. in case of already insignificant biases in original mbert, adele often further reduces the bias effect size (de, tr) and if not, the bias effects remain insignificant. we additionally visualize all xweat bias effect sizes in the produced embeddings via heatmaps in figure 2 . the intuition we can get from the plots supports our conclusion: for all languages, especially for the source language en and the tar- fairness forgetting. finally, we investigate whether the debiasing effects persist even after the large-scale fine-tuning in downstream tasks. @xcite report the presence of debiasing effects after sts-b training. with merely 5,749 training instances, however, sts-b is two orders of magnitude smaller than mnli (392,702 training instances). here we conduct a study on mnli, testing for the presence of the gender bias in bias-nli after adele's exposure to varying amount of mnli training data. we fully fine-tune bert base and bert adele (i.e., bert augmented with debiasing adapters) on mnli datasets of varying sizes (10k, 25k, 75k, 100k, 150k, and 200k) and measure, for each model, the bias-nli net neutral (nn) score as well as the nli accuracy on the mnli (matched) development set. for each model and each training set size, we carry out five training runs and report the average scores. ## related work we provide a brief overview of work in two areas which we bridge in this work: debiasing methods and parameter efficient fine-tuning with adapters. adapter layers in nlp. adapters @xcite have been introduced to nlp by @xcite , who demonstrated their ef-fectiveness and efficiency for general language understanding (nlu). since then, they have been employed for various purposes: apart from nlu, task adapters have been explored for natural language generation @xcite and machine translation quality estimation @xcite . other works use language adapters encoding language-specific knowledge, e.g., for machine translation @xcite or multilingual parsing @xcite . further, adapters have been shown useful in domain adaptation @xcite and for injection of external knowlege @xcite . @xcite use adapters to learn both language and task representations. building on top of this, @xcite prevent adapters from learning redundant information by introducing orthogonality constraints. ## conclusion we presented adele, a novel sustainable and modular approach to debiasing plms based on the adapter modules. in contrast to existing computationally demanding debiasing approaches, which debias the entire plm via full fine-tuning, adele performs parameter-efficient debiasing by training dedicated debiasing adapters. we extensively evaluated adele on gender debiasing of bert, demonstrating its effectiveness on three intrinsic and two extrinsic debiasing benchmarks. further, applying adele on top of mbert, we successfully transfered its debiasing effects to six target languages. finally, we showed that by combining adele's debiasing adapters with task-adapters, we can preserve the representational fairness even after large-scale downstream training. we hope that adele catalyzes more research efforts towards making fair nlp fairer, i.e., more sustainable and more inclusive (i.e., more multilingual).