MAFIA : Multi-Adapter Fused Inclusive Language Models (2024)

Abstract: Pretrained Language Models (PLMs) are widely used in NLP for various tasks. Recent studies have identified various biases that such models exhibit and have proposed methods to correct these biases. However, most of the works address a limited set of bias dimensions independently such as gender, race, or religion. Moreover, the methods typically involve finetuning the full model in order to maintain the performance on the downstream task. In this work, we aim to modularly debias a pre-trained language model across multiple dimensions. Previous works extensively explored debiasing PLMs by using limited US-centric counterfactual data augmentation (CDA). We use structured knowledge and a large generative model to build a diverse CDA across multiple bias dimensions in a semi-automated way. We highlight how existing debiasing methods do not consider interactions between multiple societal biases and propose a debiasing model that exploits the synergy amongst various societal biases and enables multi-bias debiasing simultaneously. An extensive evaluation on multiple tasks and languages demonstrates the efficacy of the approach.

## introduction pretrained language models (plms) are growing in power and prominence across numerous nlp tasks @xcite . their reach has expanded beyond academia, reaching general users through services like code assistance and chatbots @xcite . despite the extraordinary performance of these models on their respective tasks, several works have identified the harmful social biases picked up by these models as an artifact of their pretraining on † equal contribution ‡ work done when the author was at microsoft web-scale corpus consisting of unmoderated usergenerated content @xcite @xcite . while most previous works focus on (binary) gender biases, other societal biases, such as race and religion, are less studied in the context of plms. moreover, these biases are often intertwined with each other, creating complex and nuanced forms of discrimination. we define intersectional biases as the biases that arise from the combination of different attributes, such as gender, race, and religion. in this work, we focus on building debiasing techniques that can model and mitigate gender (including non-binary), race, religion, profession, and intersectional biases, which are often ignored in previous works. the community has developed a gamut of methods to measure and mitigate biases in llms @xcite @xcite @xcite @xcite . the majority of these methods finetune all the parameters of a language model to debias it towards a particular bias dimension such as gender or race, and the escalating size of plms can pose computational challenges, particularly for smaller academic labs or enterprises. while some methods @xcite do not alter a model's internal representations or its parameters. thus, they cannot be used as a bias mitigation strategy for downstream nlu tasks. to this end, we aim to use adapters @xcite , which are small neural network layers inserted in transformer blocks @xcite of an llm as a way to effectively debias it towards a certain dimension. we further show that a soft combination of multiple such adapters can be used to exploit the synergy between various bias dimensions and can lead to a fairer and more accurate model on a downstream task. to train each of the individual debiasing adapters, we make use of the counterfactual data augmentation (cda) technique. while cda has been shown to be effective on gender debiasing @xcite @xcite @xcite , previous works @xcite have relied on a small set of handbuilt (mostly us-centric) counterfactual pairs. as llm bias is a complex and multifaceted issue, comprehensively addressing it requires considering diverse identities. hence, we propose a semi-automated method, general purpose to build a comprehensive cda pair list using wikidata @xcite and generative models. our results indicate that such a general method can be used to train strong debiasing adapters (ideb) for multiple dimensions. in particular, we perform experiments on gender, race, religion, and profession. we list our contributions and key findings below: ## methodology in our study, we examine four primary bias dimensions: gender, race (ethnicity), religion, and profession. first, we discuss the method for generating counterfactual (cf) pairs in section 2.1. subsequently, we outline the approach to train debiasing adapters (dbas) for each dimension in section 2.2. lastly, in section 2.3, we introduce our strategy for integrating individual dbas for application on a downstream task. ## results and analysis we also find that ideb baselines perform poorly in terms of the actual sts-b task. to investigate this better, we sampled 1000 examples from sts-b and compared the score distributions (fig. 3 ) from various models. in particular, ideb models become overly conservative (they output very similar scores for almost any pair) after debiasing as evidenced by a significant reduction in their score variance. the original bert model shows a reasonable spread of scores but is biased whereas mafia is both fairer and more accurate. we further present a qualitative analysis of the behavior of the model on 5 handcrafted pairs in ## conclusion we proposed a method called mafia that uses adapterfusion to leverage the interaction of multiple bias dimensions to debias a plm. our method works by training debiasing adapters for individual biases and then fusing them on a downstream task for multidimensional debiasing. we employed counterfactual data augmentation to train each of the individual debiasing adapters. we use a semiautomatic method to generate diverse and inclusive counterfactual pairs for a given bias dimension with the help of large generative models and structured knowledge bases. our evaluation indicates that mafia leads to a fairer and more accurate model on downstream tasks across multiple languages and various bias dimensions, including potentially unseen ones during training.