Second Order W ino B ias ( S o W ino B ias) Test Set for Latent Gender Bias Detection in Coreference Resolution (2021)

Abstract: We observe an instance of gender-induced bias in a downstream application, despite the absence of explicit gender words in the test cases. We provide a test set, SoWinoBias, for the purpose of measuring such latent gender bias in coreference resolution systems. We evaluate the performance of current debiasing methods on the SoWinoBias test set, especially in reference to the method’s design and altered embedding space properties. See https://github.com/hillary-dawkins/SoWinoBias .

## introduction explicit (or first-order) gender bias was observed in coreference resolution systems by @xcite , by considering contrasting cases: here we make a new observation of genderinduced (or second-order) bias in coreference resolution systems, and provide the corresponding test set sowinobias. consider cases: indirect bias in the embedding space was viewed as an undesirable property a priori, but we do not yet have a good understanding of the effect on downstream applications. here we test debiasing methods from both camps on sowinobias, and make a series of observations on sufficient and necessary conditions for mitigating the latent genderbiased coreference resolution. additionally, we consider the case that our coreference resolution model employs both static and contextual word embeddings, but debiasing methods are applied to the static word embeddings only. post-processing debiasing techniques applied to static word embeddings are computationally inexpensive, easy to concatenate, and have a longer development history. however contemporary models for downstream applications are likely to use some form of contextual embeddings as well. therefore we might wonder whether previous work in debiasing static word embeddings remains relevant in this setting. the winobias test set for instance was developed and tested using the "end-to-end" coreference resolution model @xcite , a state-of-the-art model at that time using only static word embeddings. subsequent debiasing schemes reported results on winobias using the same model, just plugging in different debiased embeddings, for the sake of fair comparison. however this is becoming increasingly outdated given the progress in coreference resolution systems. a contribution of this work is to report winobias results for previous debiasing techniques using a more updated model, one that makes use of unaltered contextual embeddings in addition to the debiased static embeddings. the remainder of the paper is organized as follows: in section 2, we further define the type of bias being measured by the sowinobias test set and discuss some limitations. in section 3, we review the 4 word embedding debiasing methods that we will analyze, in the context of how each method aims to alter the word embedding space. in section 4, we provide details of the experimental setup and report results on both coreference resolution test sets, the original winobias and the newly constructed sowinobias. in section 5, we discuss the results with respect to the geometric properties of the altered embedding spaces. in particular, we review whether mitigation of intrinsic measures of bias on the embedding space, quantified as direct bias and indirect bias by various definitions, are related to mitigation of the latent bias in a downstream application. ## bias statement within the scope of this paper, bias is defined and quantified as the difference in performance of a coreference resolution system on test cases aligning with a socially-held stereotype vs. test cases opposing a socially-held stereotype. we observe that gender-biased systems perform significantly better in pro-stereotypical situations. such difference in performance creates representational harm by implying (for example) that occupations typically associated with one gender cannot have attributes typically associated with another. throughout this paper, the term "second-order" is used interchangeably with "latent". characterizing the observed bias as "second-order" follows from the observation of a gender-induced bias in the absence of gender-definitional vocabulary, resting on the definition of "they" as a gender-neutral pronoun. therefore, a limitation in the test set construction is the possible semantic overloading of "they". as discussed, the intention throughout this paper is to use the singular "they" as a pronoun that does not carry any gender information (and could refer to someone of any gender). however, different contexts may choose to treat "they" exclusively as a non-binary gender pronoun. the gender stereotypes used throughout this paper are sourced from peer-reviewed academic journals written in english, which draw from the us labor force statistics, as well as us-based crowd workers. therefore a limitation may be that stereotypes used here are not common to all languages or cultures. ## detection of gender bias in coreference resolution: experimental setup all systems were built using the "higher-order coreference resolution with coarse-to-fine inference" model @xcite foot_4 . it is important to keep in mind that this model uses both static word embeddings and contextual word embeddings (specifically elmo embeddings @xcite ). our experimental debiasing methods were applied to static word embeddings only, and contextual embeddings are left unaltered in all cases. all systems were trained using the ontonotes 5.0 foot_5 train and development sets, using the default hyperparameters 8 , for approximately 350,000 steps until convergence. baseline performance was tested using the ontonotes 5.0 test set (results shown in ## conclusion in this paper, we demonstrate the existence of observable latent gender bias in a downstream application, coreference resolution. we provide the first gender bias assessment test set not containing any explicit gender-definitional vocabulary. although the present study is limited to binary gender, this construction should allow us to assess gender bias (or other demographic biases) in cases where explicit defining vocabulary is limited or unavailable. however, the construction does depend on knowl-edge of expected relationships or stereotypes (here occupations and adjectives). therefore interdisciplinary work drawing from social sciences is encouraged as a future direction. our observations indicate that mitigation of indirect bias in the embedding space, according to our current understanding of such a notion, does not reduce the latent associations in the embedding space (as measured by weat), nor does it mitigate the downstream latent bias (as measured by sowino-bias). future work could seek bias assessment tasks in downstream applications that do depend on the reduction of gender-based proximity bias or non-linear recoverability. currently the motivation for such reduction is unknown, despite being an active direction of debiasing research. finally, we do observe that an early debiasing method, gn-glove, combined with simple data augmentation, can mitigate the latent gender biased coreference resolution, even when contextual embeddings in the system remain unaltered. future work could extend the idea of the sowinobias test set to more complicated sentences representative of real "in the wild" cases, in order to determine if this result holds. the sowinobias test set, all trained models presented in this paper, and code for reproducing the results are available at https://github.com/hillarydawkins/sowinobias.