Collecting a Large-Scale Gender Bias Dataset for Coreference Resolution and Machine Translation (2021)

Abstract: Recent works have found evidence of gender bias in models of machine translation and coreference resolution using mostly synthetic diagnostic datasets. While these quantify bias in a controlled experiment, they often do so on a small scale and consist mostly of artificial, out-of-distribution sentences. In this work, we find grammatical patterns indicating stereotypical and non-stereotypical gender-role assignments (e.g., female nurses versus male dancers) in corpora from three domains, resulting in a first large-scale gender bias dataset of 108K diverse real-world English sentences. We manually verify the quality of our corpus and use it to evaluate gender bias in various coreference resolution and machine translation models. We find that all tested models tend to over-rely on gender stereotypes when presented with natural inputs, which may be especially harmful when deployed in commercial systems. Finally, we show that our dataset lends itself to finetuning a coreference resolution model, finding it mitigates bias on a held out set. Our dataset and models are publicly available at github.com/SLAB-NLP/BUG. We hope they will spur future research into gender bias evaluation mitigation techniques in realistic settings.

## introduction gender bias in machine learning occurs when supervised models predict based on spurious societal correlations in their training data. this may result in harmful behaviour when it occurs in models deployed in real-world applications @xcite @xcite . 1 recent work has quantified bias mostly using carefully designed templates, following the winograd schema @xcite . @xcite figure 1 : we propose a semi-automatic method to vastly extend synthetic, small diagnostic datasets. we start with the texts of winogender @xcite and winobias @xcite , specifically designed to to be challenging for coreference and machine translation (top), extract syntactic patterns focusing on the salient entities in the artificial sentences (middle), and query real-world datasets for matching texts, using spike @xcite . the result is a large collection of diverse real-world texts exhibiting similar challenging properties which lends itself to both finetuning and testing (bottom). while these experiments are useful for quanti-fying gender bias in a controlled environment, we identify two shortcomings with this approach. first, the artificially-constructed texts diverge from natural language training distribution, which may inadvertently cause models to use prior distributions on such unseen constructions. second, the small-scale templated data does not lend itself to training or finetuning to mitigate gender bias, limiting these datasets to diagnostic purposes. in this work, outlined in figure 1 , we address both of these limitations by creating bug, a largescale dataset of 108k sentences, sampled semiautomatically from large corpora using lexicalsyntactic pattern matching (see figure 2 for examples). to construct bug, we devise 14 diverse syntactic patterns, matching a wide range of sentences, ensuring that each mentions a human entity and a pronoun referring to it. following, we use the spike engine @xcite foot_1 to retrieve matching sentences over three diverse domains, including wikipedia, covid19 research, and pubmed abstracts. finally, we filter the resulting sentences and mark each as either stereotypical or anti-stereotypical with respect to gender role assignments. the result is large corpus which is diverse, challenging, and accurate. we use bug to conduct a first large-scale evaluation of gender bias on real-world texts. we find that popular machine translation and coreference models struggle with feminine entities and antistereotypical assignments. furthermore, bug enables us to identify novel insights. for example, that machine translation models tend to be more biased when there are many pronouns in the input sentence. finally, we show that bug can also help in mitigating gender bias. we finetune a state-ofthe-art coreference resolution model on the antistereotypical portion of bug and achieve a 50% error reduction on a held out test set, at the cost of only a modest drop in overall accuracy. to conclude, our main contributions are: in this section, present bug, a semi-automatic collection of natural, "in the wild" english sentences which are challenging with respect to societal gender-role assignments. similarly to some of the synthetic gender bias datasets @xcite , we are looking for sentences with a human entity, identified by their profession (e.g., "cop", "dancer") and a gendered pronoun (e.g., "he", "she"). for example, see the first sentence in figure 2 , where the cop co-refers with a feminine pronoun ("she"), while the judge in the last sentence in figure 2 co-refers with a masculine pronoun ("his"). as opposed to previous work, we are interested in naturally occurring sentences, rather than generating artificial sentences from fixed lexical templates. the process for achieving this is outlined in figure 1 and elaborated below. first, we perform syntactic search for sentences with challenging syntactic properties over corpora from three domains (section 2.1). we then filter the sentences to verify they contain at least one entity, and a corresponding pronoun (section 2.2). finally, we manually assess bug, finding it to be 85% accurate (section 2.3). ## bug analysis the collection described in the previous section resulted in 108k sentences and 1700 human annotations. following, we analyze key characteristics of bug, finding it to be lexically diverse, and an order of magnitude larger than previous gender bias corpora. ## evaluating gender bias in the wild we evaluate the performance of machine translation and coreference resolution models on bug, using the metrics and tools established in previous work @xcite @xcite . to the best of our knowledge, this is the first quantitative evaluation of gender bias in such systems on a large scale using naturally occurring sentences. such inputs better resemble real-world use where biases can affect many users. ## debiasing with bug finally, we show that bug's size and diverse instances make it amenable for finetuning, which results in more robust models, less prone to rely on gender stereotypes. in the second row in we note however, that this comes at the cost of an absolute 1% drop in overall performance accuracy, which may be an expected side-effect due to the shift in training set distribution. future work can explore ways to find better trade-offs between accuracy and reliance of gender bias with the help of bug. ## related work several works created synthetic datasets to evaluate gender bias @xcite @xcite , e.g., in the context of coreference @xcite and machine translation @xcite @xcite , and some works used synthetic datasets to debias models @xcite . @xcite and @xcite , collected natural medium-scale (4.4k sentences) datasets from wikipedia and reddit, re-spectively, and use them to evaluate gender bias in models of coreference resolution and machine translation. however, their datasets focused on the difference in performance between masculine and feminine entities (population bias), while in this work we also measure historical bias as the difference in performance between stereotypical and anti-stereotypical gender role assignment. in section 3, we compare bug to these datasets, finding it is more diverse and challenging in various respects. ## conclusion and future work we presented bug, a large-scale corpus of 108k diverse real-world english sentences, collected via semi-automatic grammatical pattern matching. we use bug to evaluate gender bias in various coreference resolution and machine translation models, finding that models tend to make predictions in accordance with gender stereotypes, even when in conflict with opposite gendered pronouns in the sentence. finally, we finetuned a coreference resolution model on bug, finding it reduces its gender bias on a held out set. our data and code are publicly available at github.com/slab-nlp/ bug. future work can extend bug by including more patterns and by extracting sentences from corpora with gold annotations for machine translation and coreference resolution. this will allow exploration of the effect that exposure to anti-stereotypical examples during finetuning has on gender bias reduction.