Toward Gender-Inclusive Coreference Resolution (2020)

Abstract: Correctly resolving textual mentions of people fundamentally entails making inferences about those people. Such inferences raise the risk of systemic biases in coreference resolution systems, including biases that can harm binary and non-binary trans and cis stakeholders. To better understand such biases, we foreground nuanced conceptualizations of gender from sociology and sociolinguistics, and develop two new datasets for interrogating bias in crowd annotations and in existing coreference resolution systems. Through these studies, conducted on English text, we confirm that without acknowledging and building systems that recognize the complexity of gender, we build systems that lead to many potential harms.

## introduction coreference resolution-the task of determining which textual references resolve to the same realworld entity-requires making inferences about those entities. especially when those entities are people, coreference resolution systems run the risk of making unlicensed inferences, possibly resulting in harms either to individuals or groups of people. embedded in coreference inferences are varied aspects of gender, both because gender can show up explicitly (e.g., pronouns in english, morphology in arabic) and because societal expectations and stereotypes around gender roles may be explicitly or implicitly assumed by speakers or listeners. this can lead to significant biases in coreference resolution systems: cases where systems "systematically and unfairly discriminate against certain individuals or groups of individuals in favor of others" @xcite . gender bias in coreference resolution can manifest in many ways; work by @xcite , @xcite , and @xcite focused largely on the case of binary gender dis-crimination in trained coreference systems, showing that current systems over-rely on social stereotypes when resolving he and she pronouns 1 (see §2). contemporaneously, critical work in human-computer interaction has complicated discussions around gender in other fields, such as computer vision @xcite . building on both lines of work, and inspired by @xcite study of vision-based automatic gender recognition systems, we consider gender bias from a broader conceptual frame than the binary "folk" model. we investigate ways in which folk notions of gender-namely that there are two genders, assigned at birth, immutable, and in perfect correspondence to gendered linguistic formslead to the development of technology that is exclusionary and harmful of binary and non-binary trans and cis people. 2 addressing such issues is critical not just to improve the quality of our systems, but more pointedly to minimize the harms caused by our systems by reinforcing existing unjust social hierarchies @xcite . there are several stakeholder groups who may easily face harms when coreference systems is used @xcite . those harms includes several possible harms, both allocational and representation harms @xcite , including quality of service, erasure, and stereotyping harms. following bender's (2019) taxonomy of stakehold-1 throughout, we avoid mapping pronouns to a "gender" label, preferring to use the pronoun directly, include (in english) she, he, the non-binary use of singular they, and neopronouns (e.g., ze/hir, xey/xem), which have been in usage since at least the 1970s @xcite @xcite @xcite . 2 following @xcite , transgender individuals are those whose gender differs from the sex they were assigned at birth. this is in opposition to cisgender individuals, whose assigned sex at birth happens to correspond to their gender. transgender individuals can either be binary (those whose gender falls in the "male/female" dichotomy) or non-binary (those for which the relationship is more complex). @xcite taxonomy of harms, there are several ways in which trans exclusionary coreference resolution systems can cause harm: indirect: subject of query. if a person is the subject of a web query, pages about xem may be missed if "multiple mentions of query" is a ranking feature, and the system cannot resolve xyr pronouns ⇒ quality of service, erasure. direct: by choice. if a grammar checker uses coreference, it may insist that an author writing hir third-person autobiography is repeatedly making errors when referring to hirself ⇒ quality of service, stereotyping, denigration. direct: not by choice. if an information extraction system run on résumés relies on cisnormative assumptions, job experiences by a candidate who has transitioned and changed his pronouns may be missed ⇒ allocative, erasure. many stakeholders. if a machine translation system uses discourse context to generate pronouns, then errors can results in directly misgendering subjects of the document being translated ⇒ quality of service, denigration, erasure. to address such harms as well as understand where and how they arise, we need to complicate (a) what "gender" means and (b) how harms can enter into natural language processing (nlp) systems. toward (a), we begin with a unifying analysis ( § 3) of how gender is socially constructed, and how social conditions in the world impose expectations around people's gender. of particular interest is how gender is reflected in language, and how that both matches and potentially mismatches the way people experience their gender in the world. then, in order to understand social biases around gender, we find it necessary to consider the different ways in which gender can be realized linguistically, breaking down what previously have been considered "gendered words" in nlp papers into finer-grained categories that have been identified in the sociolinguistics literature of lexical, referential, grammatical, and social gender. toward (b), we focus on how bias can enter into two stages of machine learning systems: data annotation ( § 4) and model definition ( § 5). we construct two new datasets: (1) map (a similar dataset to gap @xcite but without binary gender constraints) on which we can perform counterfactual manipulations and (2) gicoref (a fully annotated coreference resolution dataset written by and about trans people). 3 in all cases, we focus largely on harms due to over-and underrepresentation @xcite , replicating stereotypes @xcite (particular those that are cisnormative and/or heteronormative), and quality of service differentials @xcite . the primary contributions of this paper are: ## related work there are four recent papers that consider gender bias in coreference resolution systems. @xcite evaluates coreference systems for evidence of occupational stereotyping, by constructing winograd-esque @xcite test examples. they find that humans can reliably resolve these examples, but systems largely fail at them, typically in a gender-stereotypical way. in contemporaneous work, @xcite proposed a very similar, also winograd-esque scheme, also for measuring gender-based occupational stereotypes. in addition to reaching similar conclusions to @xcite , this work also used a similar "counterfactual" data process as we use in § 4.1 in order to provide additional training data to a coreference resolution system. @xcite produced the gap dataset for evaluating coreference systems, by specifically seeking examples where "gender" (left underspecified) could not be used to help coreference. they found that coreference systems struggle in these cases, also pointing to the fact that some success of current coreference systems is due to reliance on (binary) gender stereotypes. finally, ackerman (2019) presents an alternative breakdown of gender than we use ( § 3), and proposes matching criteria for model-ing coreference resolution linguistically, taking a trans-inclusive perspective on gender. gender bias in nlp has been considered more broadly than just in coreference resolution, including, natural language inference @xcite , word embeddings (e.g., @xcite @xcite , sentiment analysis @xcite , machine translation @xcite @xcite @xcite @xcite , among many others @xcite . gender is also an object of study in gender recognition systems @xcite . much of this work has focused on gender bias with a (usually implicit) binary lens, an issue which was also called out recently by @xcite and @xcite . ## linguistic & social gender the concept of gender is complex and contested, covering (at least) aspects of a person's internal experience, how they express this to the world, how social conditions in the world impose expectations on them (including expectations around their sexuality), and how they are perceived and accepted (or not). when this complex concept is realized in language, the situation becomes even more complex: linguistic categories of gender do not even remotely map one-to-one to social categories. as observed by @xcite : furthermore, despite the impossibility of a perfect alignment with linguistic gender, it is generally clear that an incorrectly gendered reference to a person (whether through pronominalization or otherwise) can be highly problematic @xcite . this process of misgendering is problematic for both trans and cis individuals to the extent that transgender historian @xcite writes: "[o]ne's gender identity could perhaps best be described as how one feels about being referred to by a particular pronoun." ## bias in human annotation a possible source of bias in coreference systems comes from human annotations on the data used to train them. such biases can arise from a combination of (possibly) underspecified annotations guidelines and the positionality of annotators themselves. in this section, we study how different aspects of linguistic notions impact an annotator's judgments of anaphora. this parallels ackerman (2019) linguistic analysis, in which a broad matching criterion is proposed, which posits that "matching gender requires at least one level of the mental representation of gender to be identical to the candidate antecedent in order to match." our study can be seen as evaluating which conceptual properties of gender are most salient in human judgments. we start with natural text in which we can cast the coreference task as a binary classification problem ("which of these two names does this pronoun refer to?") inspired by @xcite . we then generate "counterfactual augmentations" of this dataset by ablating the various notions of linguistic gender described in §3.2, similar to @xcite . we finally evaluate the impact of these ablations on human annotation behavior to answer the question: which forms of linguistic knowledge are most essential for human annotators to make consistent judgments. see appendix a for examples of how linguistic gender may be used to infer social gender. ## bias in model specifications in addition to biases that can arise from the data that a system is trained on, as studied in the previ-ous section, bias can also come from how models are structured. for instance, a system may fail to recognize anything other than a dictionary of fixed pronouns as possible referents to entities. here, we analyze prior work in models for coreference resolution in three ways. first, we do a literature study to quantify how nlp papers discuss gender. second, similar to @xcite and @xcite , we evaluate five freely available systems on the ablated data from §4. third, we evaluate these systems on the dataset we created: gender inclusive coreference (gicoref).