Can a Prediction’s Rank Offer a More Accurate Quantification of Bias? A Case Study Measuring Sexism in Debiased Language Models (2023)

Abstract: Pre-trained language models are known to inherit a plethora of contextual biases from their training data. These biases have proven to be projected onto a variety of downstream applications, making their detection and mitigation imminent. Limited research has been conducted to quantify specific bias types, such as benevolent sexism, which may be subtly present within the inferred connotations of a sentence. To this extent, our work aims to: (1) provide a benchmark of sexism sentences; (2) adapt two bias metrics: mean probability score and mean normalized rank; (3) conduct a case study to quantify and analyze sexism in base and de-biased masked language models. We find that debiasing, even in its most effective form (Auto-Debias), solely nullifies the probability score of biasing tokens, while retaining them in high ranks. Auto-Debias illustrates a 90%-96% reduction in mean probability scores from base to debiased models, while only a 3%-16% reduction in mean normalized ranks. Similar to the application of non-parametric statistical tests for data that does not follow a normal distribution, operating on the ranks of predictions rather than their probability scores offers a more representative bias measure.

## introduction masked language models (mlms) have proven to be an effective tool in a variety of natural language processing (nlp) tasks, notably, cloze-style prompt prediction @xcite @xcite . however, language models have also proven to project and inherit both structural (e.g. generic pronouns, explicit marking of sex) and contextual biases (e.g. sexism, stereotyping) from their training corpora, making their detection and mitigation imminent @xcite . previous attempts at measuring biases in language models were employed using benchmarks adapted from general attributetarget word pairs (seat adapted from weat) or figure 1 : overview of sexistly, a benchmark to quantify sexism in masked language models by incorporating both the probability score and topk index (rank) of masked predictions. the theory of ambivalent sexism posits that there is a distinction between two forms of sexism: hostile and benevolent sexism @xcite . hostile sexism is characterized by negative attitudes and beliefs, including dominative paternalism as well as derogatory principles @xcite . benevolent sexism is a form of sexism that appears positive towards women but is actually based on traditional gender roles and the belief in women's inferiority @xcite . it often involves protective paternalism and the idealization of women @xcite . the underlying positive connotation behind benevolently sexist statements impairs its opposition because it portrays advantageous aspects of being a woman @xcite . as a result, these bias types typically go unobserved as they are rooted within the inferred meaning of a sentence through sarcastic nuances rather than being structured within the typical attributetarget-adapted sentences. figure 1 illustrates our evaluation pipeline which forms the basis of the following contributions: the main finding of this work is that debiasing, even it in its most effective form (auto-debias), solely nulls out the probability score of biasing tokens while retaining them in high ranks. this has been made evident through the lens of mnt, which normalizes the ranks into a 0-1 range and computes their average across all biasing masked predictions in our benchmark. auto-debias illustrates a 90%-96% reduction in mean probability scores from base to debiased models, while only a 3%-16% reduction in mean normalized ranks. ## related work this section aims to describe three predominant intrinsic evaluation benchmarks geared towards measuring bias in masked language models. ## ambivalent sexism theory the theory of ambivalent sexism recognizes that sexism entails a mixture of antipathy and subjective benevolence @xcite . it argues that hostile and benevolent sexism are, in fact, not conflicting but complementary ideologies that present a resolution to the gender relationship paradox. ## benchmark construction the benchmark construction methodology is comprised of four main stages: (1) dataset curation; (2) dataset filtering; (3) sentence masking; (4) pronoun neutralization. ## bias metrics we quantify bias in masked language models using the following metrics: mean probability score (mps) and mean normalized top-k (mnt). ## conclusion in summary, this work presents sexistly, a benchmark to quantify sexism in masked language models. to do so, we curated positive samples of benevolent and hostile sexism from labelled datasets and processed them by masking the biasing tokens before passing them into the mask-filling pipeline. we propose two bias metrics: mean probability score (mps) and mean normalized top-k (mnt) to adequately measure sexism in language models. as a case study, we quantify and analyze sexism in base masked language models as well as their debiased variants using four prominent debiasing techniques: context-debias, auto-debias, cda, and dropout. our primary finding underscores that debiasing, even it in its most effective form (auto-debias), solely nulls out the probability score of biasing tokens while retaining them in high ranks. this has been made evident through the lens of mnt, which normalizes the ranks into a 0-1 range and computes their average across all biasing masked predictions in our benchmark. auto-debias illustrates a 90%-96% reduction in mean probability scores from base to debiased models, while only a 3%-16% reduction in mean normalized ranks. using the ranks of predictions, rather than their probability scores, offers a more robust bias measure in a manner analogous to applying non-parametric statistical tests to data not adhering to a normal distribution.