Counter- GAP : Counterfactual Bias Evaluation through Gendered Ambiguous Pronouns (2023)

Abstract: Bias-measuring datasets play a critical role in detecting biased behavior of language models and in evaluating progress of bias mitigation methods. In this work, we focus on evaluating gender bias through coreference resolution, where previous datasets are either hand-crafted or fail to reliably measure an explicitly defined bias. To overcome these shortcomings, we propose a novel method to collect diverse, natural, and minimally distant text pairs via counterfactual generation, and construct Counter-GAP, an annotated dataset consisting of 4008 instances grouped into 1002 quadruples. We further identify a bias cancellation problem in previous group-level metrics on Counter-GAP, and propose to use the difference between inconsistency across genders and within genders to measure bias at a quadruple level. Our results show that four pre-trained language models are significantly more inconsistent across different gender groups than within each group, and that a name-based counterfactual data augmentation method is more effective to mitigate such bias than an anonymization-based method.

## introduction it is a common practice to train state-of-the-art natural language processing (nlp) models by unsupervised pre-training and supervised fine-tuning (e.g., @xcite , both of which rely heavily on large corpora of real-world text. however, these corpora often reflect societal stereotypes and may lead to models exhibiting biased behaviors @xcite . hence, much research effort has been put to reveal and mitigate unintended biases @xcite . while early work focuses on detecting and mitigating gender bias in the space of word embeddings (e.g., @xcite , recent approaches turn to design bias-measuring datasets on specific nlp tasks @xcite @xcite . in this work, we focus on gender bias in coreference resolution and adopt a kind of representational harm @xcite to define gender fairness: a genderneutral model should rely on the semantic information, rather than on the gender information contained in the texts, to make predictions. otherwise, a model should be considered gender-biased. in line with this definition, winobias @xcite and winogender @xcite leverage pairs of minimally distant sentences, i.e., two sentences that contain the same semantic information but different gender information, to measure models' performance difference in resolving pronouns of different genders under the same context. this minimally distant setting enables us to isolate the influence of gender information on model predictions. a limitation of winobias and winogender is that they are made up of hand-crafted sentences, which prevents us from measuring gender bias in the more diverse real-world scenarios. an alternative to overcome this shortcoming is the gap dataset @xcite , which exploits linguistic patterns to automatically extract instances from a real-world corpus. however, since gap's masculine and feminine instances cannot be grouped into minimally distant pairs, we are not sure whether a difference in model performance is due to different gender information or to different semantic information. for example, compared to masculine instances, gap's feminine instances have more candidate entities serving as distractors, and longer distance between the correct name and the pronoun @xcite . so, it is not equally hard to resolve the masculine and feminine instances in gap. hence, the performance difference between masculine and feminine instances on gap is not a reliable measure of gender bias according to the above definition of gender fairness. 1 given these observations, we propose a novel method to construct coreference-resolution-based bias-measuring datasets consisting of minimally distant text pairs that originate from real-world corpora. specifically, we leverage the method from gap @xcite to extract original instances containing gendered ambiguous pronouns, and generate minimally distant instances by asking the counterfactual question "how would the prediction change if we swapped the roles of masculine and feminine people in this context?" @xcite . the resulting instances are grouped into quadruples, each of which consists of an original, a gender-controlled, and two gender-swapped instances. an example is shown in furthermore, we find that bias in different directions may be canceled out if we aggregate the results by performance difference across groups of instances, and we call this problem bias cancellation. to alleviate it, we propose a new metric, inconsistency across genders, to measure bias at the quadruple level. we also leverage the gendercontrolled instances to disentangle inconsistency within genders from inconsistency across genders, so that we can eliminate the impact of name perturbations. our contributions are as follows: (i) we propose a novel method to construct coreference resolution datasets consisting of diverse, natural, and minimally distant instances to reliably detect gender bias. (ii) we apply our method to online books and collect counter-gap, an annotated dataset with 4008 instances grouped into 1002 quadruples. (iii) we propose a new metric, the difference (∆i) between inconsistency across genders and within genders, to alleviate the bias cancellation problem of previous metrics. (iv) we use counter-gap to empirically evaluate four pre-trained language models and two debiasing methods based on counterfactual data augmentation (cda, @xcite . our results show that ∆i can detect significant gender bias hidden by group-level performance difference, and that namebased cda is more effective than vanilla cda in mitigating such bias. foot_2 ## dataset construction the counter-gap dataset is derived from 1575 fictional books in project gutenberg foot_3 and book-corpus @xcite . it is constructed through a generic multi-stage process, as described below. here, we follow the gap dataset @xcite and focus only on the english language, as well as adopting a notion of binary gender. ## evaluation metrics on counter-gap we use @xmath0, x c , x o , x c ) ∈ x to denote a quadruple, and lowercased x to denote an arbitrary instance, which could be each of x o , x c , x o , x c from a quadruple. given a model f (•), assume that f (x) ∈ {0, 1} indicates whether f 's prediction on instance x is correct (1) or not (0). ## bias evaluation on counter-gap for evaluation, we adopt the coreference resolution system c2f-coref foot_8 @xcite based on four pre-trained language models: bertbase/large and spanbert-base/large @xcite . all four models are fine-tuned on ontonotes (pradhan et al., 2012), 10 and training details are shown in appendix a. in our evaluation, no candidate names are provided as input to the models, and models are responsible to detect candidate names in the text by themselves. a model's prediction on an instance is considered correct if the candidate name with gold label true and none of those with gold label false are in the target pronoun's coreferent cluster. ## bias mitigation we evaluate two debiasing methods based on counterfactual data augmentation (cda) @xcite : (i) anonymizationbased cda (a-cda), where the training set (ontonotes) is augmented by substituting all gendered words with their opposite gendered words, while the gold coreference labels are kept unchanged. personal names in the training set are anonymized using place holders such as "e1, e2, . . . "; (ii) name-based cda (n-cda), where, in addition to the substitution between gendered words, masculine and feminine names also substitute each other according to their frequencies @xcite . see appendix a for more details. performance on bias mitigation is measured by acc dif f and ∆i, while performance on coreference resolution is measured by overall accuracy on counter-gap and f 1 score on ontonotes' dev set. results are shown in ## qualitative analysis in in example 2, spanbert-large correctly finds "alice" in the gender-swapped-2 instance, but confuses "alice" and "dora" in the gender-swapped-1 instance. this illustrates the inconsistency brought by name perturbations within the same gender, and we take this into account by subtracting inconsis-tency within genders from inconsistency across genders (∆@xmath1. measuring bias in nlp models. human-like biases are first detected and measured in word embeddings @xcite @xcite @xcite . for pre-trained language models, @xcite adopt bleached sentence templates to contextualize target words, while most recent works leverage crowd-sourced benchmark datasets on nlp tasks such as language modeling @xcite , sentiment analysis @xcite , dialog generation @xcite , natural language inference @xcite , and machine translation @xcite . our work follows gap @xcite , winobias @xcite , and winogender @xcite to measure gender bias in coreference resolution, with a specific focus on collecting diverse, natural, and minimally distant instances. counterfactual bias evaluation. @xcite propose the notion of counterfactual fairness, which requires similar predictions before and after counterfactual interventions in casual graphs. @xcite apply this notion to text classifi-example 1 original (correct) "in fact, roxanne told me that she had scheduled an interview with a source tonight." denise sipped at her lemonade through her straw until she found the bottom of her glass at last. scotty told chris that denise didn't give him any particulars about why she needed to hire a private detective when she sought his advice. gendercontrolled (correct) "in fact, denise told me that she had scheduled an interview with a source tonight." roxanne sipped at her lemonade through her straw until she found the bottom of her glass at last. chris told scotty that roxanne didn't give him any particulars about why she needed to hire a private detective when she sought his advice. genderswapped-1 (incorrect) "in fact, chris told me that he had scheduled an interview with a source tonight." scotty sipped at his lemonade through his straw until he found the bottom of his glass at last. denise told roxanne that scotty didn't give her any particulars about why he needed to hire a private detective when he sought her advice. genderswapped-2 (incorrect) "in fact, scotty told me that he had scheduled an interview with a source tonight. denny said, ""you ought not to bet, especially on sunday,"" and oswald altered it to ""you may be sure."" ""well, but what then?"" alice asked dora. ""out with it,"" for she saw that her youthful friend had got an idea and couldn't get it out. name artifacts in nlp models. since neural language models do not treat personal names as interchangeable, there are various biases in the learned representations of personal names @xcite @xcite . counter-gap considers name biases as the source of gender bias, and exhibits these name biases through the task of coreference resolution. ## summary and outlook in this work, we proposed a method to construct minimally distant bias-measuring datasets for coreference resolution, and exemplified it in the collection of counter-gap. we proposed the inconsistency metric ∆i to overcome the bias cancellation problem and noise from name perturbations. we showed that four pre-trained language models exhibit significant gender bias, and name-based cda is most effective in mitigating the detected bias. limitations of counter-gap include that around half of the instances are from historical fictions in project gutenburg, making the dataset less representative of contemporary bias; the rules in our method for constructing counter-gap are specific for english, and might not be easily adapted to languages with more complex morphology; while we recognize that gender is non-binary, we adopt the simplifying setup of binary gender construct, which prevents us from detecting gender bias against minority groups with non-binary genders. in future work, we will apply our method to different domains and more contemporary corpora such as news articles. leveraging the data augmentation method for languages with grammatical genders @xcite , as well as linguistic resources for non-binary genders @xcite is also an important future direction to construct more gender-and language-inclusive datasets.