MABEL : Attenuating Gender Bias using Textual Entailment Data (2022)

Abstract: Pre-trained language models encode undesirable social biases, which are further exacerbated in downstream use. To this end, we propose MABEL (a Method for Attenuating Gender Bias using Entailment Labels), an intermediate pre-training approach for mitigating gender bias in contextualized representations. Key to our approach is the use of a contrastive learning objective on counterfactually augmented, gender-balanced entailment pairs from natural language inference (NLI) datasets. We also introduce an alignment regularizer that pulls identical entailment pairs along opposite gender directions closer. We extensively evaluate our approach on intrinsic and extrinsic metrics, and show that MABEL outperforms previous task-agnostic debiasing approaches in terms of fairness. It also preserves task performance after fine-tuning on downstream tasks. Together, these findings demonstrate the suitability of NLI data as an effective means of bias mitigation, as opposed to only using unlabeled sentences in the literature. Finally, we identify that existing approaches often use evaluation settings that are insufficient or inconsistent. We make an effort to reproduce and compare previous methods, and call for unifying the evaluation settings across gender debiasing methods for better future comparison.

## introduction pre-trained language models have reshaped the landscape of modern natural language processing @xcite @xcite . as these powerful networks are optimized to learn statistical properties from large training corpora imbued with significant social biases (e.g., gender, racial), they produce encoded representations that inherit undesirable associations as a byproduct @xcite @xcite . more concerningly, models trained on these representations can not only propagate but also amplify discriminatory judgments in downstream applications @xcite . a multitude of recent efforts have focused on alleviating biases in language models. these can be classed into two categories ( in this work, we present mabel, a novel and lightweight method for attenuating gender bias. mabel is task-agnostic and can be framed as an intermediate pre-training approach with a contrastive learning framework. our approach hinges on the use of entailment pairs from supervised natural language inference datasets @xcite . we augment the training data by swapping gender words in both premise and hypothesis sentences and model them using a contrastive objective. we also propose an alignment regularizer, which minimizes the distance between the entailment pair and its augmented one. mabel optionally incorporates a masked language modeling objective, so that it can be used for token-level downstream tasks. to the best of our knowledge, mabel is the first to exploit supervised sentence pairs for learning fairer contextualized representations. supervised contrastive learning via entailment pairs is known to learn a more uniformly distributed rep-resentation space, wherein similarity measures between sentences better correspond to their semantic meanings @xcite . meanwhile, our proposed alignment loss, which pulls identical sentences along contrasting gender directions closer, is well-suited to learning a fairer semantic space. we systematically evaluate mabel on a comprehensive suite of intrinsic and extrinsic measures spanning language modeling, text classification, nli, and coreference resolution. mabel performs well against existing gender debiasing efforts in terms of both fairness and downstream task performance, and it also preserves language understanding on the glue benchmark @xcite . altogether, these results demonstrate the effectiveness of harnessing nli data for bias attenuation, and underscore mabel's potential as a general-purpose fairer encoder. lastly, we identify two major issues in existing gender bias mitigation literature. first, many previous approaches solely quantify bias through the sentence encoding association test (seat) @xcite , a metric that compares the geometric relations between sentence representations. despite scoring well on seat, many debiasing methods do not show the same fairness gains across other evaluation settings. second, previous approaches evaluate on extrinsic benchmarks in an inconsistent manner. for a fairer comparison, we either reproduce or summarize the performance of many recent methodologies on major evaluation tasks. we believe that unifying the evaluation settings lays the groundwork for more meaningful methodological comparisons in future research. ## conclusion in this work, we propose mabel, a simple bias mitigation technique that harnesses supervised signals from entailment pairs in nli data to create informative and fair contextualized representations. we systematically compare mabel and other recent task-agnostic debiasing baselines across a range of intrinsic and extrinsic bias metrics, wherein mabel demonstrates a markedly better performance-fairness tradeoff. its capacity for language understanding is also minimally impacted, rendering it suitable for general-purpose use. additionally, systematic ablations show that both the choice of data and individual objectives are integral to mabel's good performance. our contribution and findings are complementary to the bias transfer hypothesis @xcite , which suggests that upstream bias mitigation effects may be transferable to downstream settings. we hope that mabel can add a new perspective toward creating fairer language models.