Parameter-efficient Modularised Bias Mitigation via A dapter F usion (2023)

Abstract: Large pre-trained language models contain societal biases and carry along these biases to downstream tasks. Current in-processing bias mitigation approaches (like adversarial training) impose debiasing by updating a model’s parameters, effectively transferring the model to a new, irreversible debiased state. In this work, we propose a novel approach to develop stand-alone debiasing functionalities separate from the model, which can be integrated into the model on-demand, while keeping the core model untouched. Drawing from the concept of AdapterFusion in multi-task learning, we introduce DAM (Debiasing with Adapter Modules) – a debiasing approach to first encapsulate arbitrary bias mitigation functionalities into separate adapters, and then add them to the model on-demand in order to deliver fairness qualities. We conduct a large set of experiments on three classification tasks with gender, race, and age as protected attributes. Our results show that DAM improves or maintains the effectiveness of bias mitigation, avoids catastrophic forgetting in a multi-attribute scenario, and maintains on-par task performance, while granting parameter-efficiency and easy switching between the original and debiased models.

## introduction large pre-trained language models (plm) and their variations fine-tuned on downstream tasks encode societal biases and stereotypes. a large body of research shows the existence of these biases @xcite @xcite , and discuss their potential harms @xcite in various tasks such as occupation prediction from biographies @xcite , information retrieval (ir) @xcite , and machine translation @xcite . a common approach to mitigating these biases is to adjust model parameters according to specific fairness criteria, achieved using optimization methods such as adversarial training @xcite @xcite . the resulting debiased models significantly diminish the effect of protected attributes (e. g. gender, race, etc.) on model's predictions. however, depending on the usage context, characteristics of the input or fairness-utility trade-off considerations, system designers or end-users might in practice still prefer to instead use the original (potentially biased) model for processing some requests. as an example, while a fairness-aware document retrieval model should tease out gender attributes when processing bias-sensitive queries @xcite (like how to become ceo?), other common queries (like earliest pregnancy symptoms) may specifically require gender information to obtain satisfactory retrieval results. previous studies also show that one may need to occasionally accept slight performance degradation in exchange for realizing higher fairness @xcite @xcite . we argue that ondemand control over the strength of bias mitigation is crucial for the broad adoption of bias mitigation models. using existing approaches, this would require maintaining and deploying multiple large parallel models for every protected attribute, in practice resulting in overly complex and resourceheavy pipelines and increased latency. to reduce this untenable burden, we introduce debiasing with adapter modules (dam). in our approach, bias mitigation functionalities can be trained in fully decoupled modules from the building blocks solving the main task (see figure 1 ). these debiasing modules are learned independently, and can be selectively inserted into the model to deliver debiasing qualities. our method draws from the recent success of adapter @xcite and adapter-fusion @xcite networks, originally introduced in the context of multi-task learning to enable parameter-efficient training and to avoid catastrophic forgetting. specifically in multi-task learning with adapterfusion, each task is encoded in specific parameter-efficient adapter networks attached to the transformer blocks @xcite of a plm, and the knowledge across the tasks is shared through a fusionadapter layer with an attention mechanism. dam extends this principle idea to bias mitigation by viewing the objective of "removing each protected attribute" as a standalone task, learned by an independent debiasing adapter module. these debiasing adapters are then integrated into the core model via a fusion layer to mitigate biases. this modular architecture of dam enables on-demand debiasing, where by simply removing the debiasing adapters the model would return to its original state. the modularity and parameter efficiency of dam also enables creating and sharing stand-alone debiasing solutions (in form of debiasing adapters), which can be applied to the underlying models on-demand. so, if one uses task adapter's output in dam, they get biased embedding else they can take output of fusion layer for debiased embedding. we evaluate dam in extensive experiments covering various bias mitigation applications in three classification tasks: we study occupation prediction from biographies @xcite , mention prediction @xcite , and hate speech detection @xcite , involving the protected attributes of gender, age, and dialect-based race. among these, the mention prediction task provides two protected attributes (gender, age), enabling the study of adapter-based bias mitigation on multiple factors. our evaluation results show that dam (1) consistently improves bias mitigation in terms of lower leakage of protected attribute in comparison with fully fine-tuned models, and (2) better avoids catastrophic forgetting @xcite in a multi-attribute debiasing scenario, while (3) maintaining on-par task performance. our results show the benefits of dam in supporting fairness, and reveal the challenges in the scenarios inherently bounded by trade-offs. our contribution is three-fold: the remainder of the paper is organized as follows: section 2 reviews the relevant literature in adapter networks and bias mitigation domains. in section 3, we explain the architecture and training strategy of dam. section 4 describes the setup of the experiments, whose results are reported and discussed in section 5. finally, sections 6 and 7, respectively, summarize and discuss limitations of our work. our code and trained resources are available in https://github.com/cpjku/ modulaizeddebiasing . ## bias mitigation with dam we approach the scenario in which the aim is to maximize a task's core objective (such as occupation prediction or hate speech prediction), while reducing the effects of the biases caused by k protected attributes. the crux of the proposed dam approach is to first learn a task adapter and k bias removal adapters all independently, and then combine them through a fusion module. once a model has been trained with dam, one can easily switch between the original model by only using the task adapter, or alternatively impose debiasing by "plugging in" the learned debiasing adapters as well as the fusion module. in what follows, we explain the architecture of dam followed by the optimization procedure. ## experiment setup we evaluate dam on three classification tasks and compare the results with a set of strong baselines. with biases towards any societal group can be extremely delicate (e. g. in automatic social media moderation). @xcite show the existence of a strong correlation between dialect-based racial bias and hate speech detection. in our experiments, we use fdcl18 @xcite , a dataset of tweets annotated with four hate speech labels: normal, spam, abusive, and hateful. following previous works @xcite @xcite , we assign artificial race labels of african american, and white american to fdcl18 (more details in appendix a.1). in this setting, the main task is hate speech classification, and the protected attribute is the dialect-based racial identity of the tweet author. our final dataset contains approximately 62k data points. while the first two tasks contain one protected attribute, in our third classification task, we aim to explore the effect of more than one biases on our models. we therefore opt for a subset of the pan16 dataset @xcite . this dataset has been designed for the task of mention detection, namely to predict whether a tweet includes the mention of other users, and the protected attributes are gender and age of the tweet's author. the dataset contains approximately 200k data points with binary task classes (mention, no mention), as well as two gender labels and five age groups. further details on the three datasets are provided in appendix a.1. ## results and discussion in this section, we first discuss the results on the datasets with one protected attribute (bios and fdcl18), followed by investigating the effect of dam on pan16 with two protected attributes. single-attribute bias mitigation the results on bios and fdcl18 datasets are reported in comparing the results between the core plms, we see that the above observations for bert-mini also hold true for bert-base on the bios dataset. on the fdcl18 dataset with bert-mini, dam is in fact the only bias mitigation approach that can effectively prevent leakage while also preserving performance. it is the best in terms of leakage reduction, while its task performance is better than adp. the situation differs for results of bert-base on fdcl18, as leakage balanced accuracy of ft-debias is surprisingly low. multi/two-attribute bias mitigation particularly with respect to multi-attribute bias mitigation, the results show the benefits of dam a+g in separately learning multiple debiasing adapters and combining them, when compared with ft-debias g+a , which simultaneously learns both debiasing factors. in particular, looking at the attacker's bacc, while ft-debias g+a is not able to debias the age attribute (in comparison to ft), our dam g+a provides strongly improved bias mitigation for age (and also gender), with similar or better results to only optimizing for gender or age (dam g and dam a , respectively). this highlights the effectiveness of dam in preventing catastrophic forgetting of debiasing functionalities in multi-attribute scenarios. 3 we further provide an investigation of the attention distributions of dam's fusion module in appendix b. ## conclusion we propose a novel bias mitigation approach which enables flexible switching between the original and debiased state of a model. our proposed dam method extends the idea of multi-task learning using adapterfusion to bias mitigation, by first learning the main task and the debiasing objectives as separate adapters, and then leveraging the attentionbased fusion approach to merge these adapters and deliver debiased results. our experiments on three classification tasks show that, beside flexible switching, dam improves bias mitigation, provides on par classification performance to vanilla adversarial debiasing, and addresses the issue of catastrophic forgetting in multi-attribute bias mitigation.