Uncertainty and Inclusivity in Gender Bias Annotation: An Annotation Taxonomy and Annotated Datasets of B ritish E nglish Text (2022)

Abstract: Mitigating harms from gender biased language in Natural Language Processing (NLP) systems remains a challenge, and the situated nature of language means bias is inescapable in NLP data. Though efforts to mitigate gender bias in NLP are numerous, they often vaguely define gender and bias, only consider two genders, and do not incorporate uncertainty into models. To address these limitations, in this paper we present a taxonomy of gender biased language and apply it to create annotated datasets. We created the taxonomy and annotated data with the aim of making gender bias in language transparent. If biases are communicated clearly, varieties of biased language can be better identified and measured. Our taxonomy contains eleven types of gender biases inclusive of people whose gender expressions do not fit into the binary conceptions of woman and man, and whose gender differs from that they were assigned at birth, while also allowing annotators to document unknown gender information. The taxonomy and annotated data will, in future work, underpin analysis and more equitable language model development.

## background and introduction the need to mitigate bias in data has become urgent as evidence of harms from such data grows @xcite @xcite @xcite . due to the complexities of bias often overlooked in machine learning (ml) bias research, including natural language processing (nlp) @xcite , @xcite , @xcite , and crawford (2017) call for greater interdisciplinary engagement and stakeholder collaboration. the gallery, library, archive, and museum (glam) sector has made similar calls for interdisciplinary engagement, looking to applications of data science and ml to better understand and mitigate bias in glam collections @xcite @xcite . supporting the nlp and glam communities' shared aim of mitigating the minoritization foot_0 of certain people that biased language causes, we provide a taxonomy of gender biased language and demonstrate its application in a case study with glam documentation. we use glam documentation to refer to the descriptions of heritage items written in glam catalogs. adapting our previously published definition, we use gender biased language to refer to "language that creates or reinforces inequitable power relations among people, harming certain people through simplified, dehumanizing, or judgmental words or phrases that restrict their @xcite identity; and privileging other people through words or phrases that favor their @xcite identity" @xcite . we focus on gender bias due to the contextual nature of gender and bias (they vary across time, location, culture, and people), as well as the existing efforts of our partner institution, the archives of the centre for research collections at the university of edinburgh, to mitigate gender bias in its documentation. glam documentation provides a unique benefit compared to many text sources: it contains historical and contemporary language. glam continually acquire and describe heritage items to enable the items' discoverability. in archives, heritage items include photographs, handwritten documents, instruments, and tweets, among other materials. heritage items and the language that describes them influence society's understanding of the past, the present, and the direction society is moving into the future @xcite @xcite @xcite . through research with glam documentation, variations in biased language could be better understood. should diachronic patterns emerge, the nlp community could train models to identify newly-emerging, previously unseen types of bias. this paper presents an annotation taxonomy ( §5) to label gender biased language inclusive of trans and gender diverse identities, foot_1 as well as a dataset of historical and contemporary language from british english archival documentation annotated according to the taxonomy. linguistics, gender studies, information sciences, and nlp literature inform the taxonomy's categorization of gender biased language. as a result, the taxonomy holds relevance beyond the glam sector in which we situate our work. the taxonomy may be applied when creating nlp datasets or models, or when measuring varieties of gender bias in language, because the taxonomy's definitions of types of gender biases are rooted in the language of text, rather than an abstracted representation of text. uniquely, our taxonomy includes labels that record uncertainty about a person's gender. as we situate our work in the glam sector, this paper provides a case study ( §6) demonstrating how the annotation taxonomy was applied to create an annotated dataset of archival documentation. for future nlp work, the resulting dataset of historical and contemporary language annotated for gender biases provides a corpus to analyze gender biased language for diachronic patterns, to analyze correlations between types of gender biases, and to develop gender bias classification models. specific to the glam sector, gender bias classification models could enhance reparative description practices. a model's ability to automatically identify descriptions of heritage items that contain gender biases would enable efficient prioritization of the additions and revisions needed on outdated, harmful descriptions in glam documentation. ## bias statement this paper adopts our previously published definition of biased language @xcite , narrowing the focus to gender bias as written in §1. gender biased language may cause representational or allocative harms to a person of any gender @xcite . the taxonomy created in this paper considers a person's gender to be self-described and changeable, rather than being limited to the binary and static conceptualization of gender as either a man or woman since birth @xcite . recognizing that a person's gender may be impossible to determine from the information available about them, the taxonomy also allows annotators to record uncertainty @xcite . furthermore, the paper acknowledges that characteristics other than gender, such as racialized ethnicity and economic class, influence experiences of power and oppression @xcite . drawing on archival science and feminist theories, the paper considers knowledge derived from language as situated in a particular perspective and, as a result, incomplete @xcite @xcite . to communicate this paper's perspective, we as authors report our identification as three women and one man; and our nationalities, as american, german, and scots. annotators identify as women (one specifying queer woman and two, cis women); they are of american, british, hungarian, and scots nationalities. though annotators do not represent great gender diversity, foot_2 the annotation process still contributes to the advancement of gender equity. as women, the annotators identify as a minoritized gender. the evolution of british english demonstrates the historical dominance of the perspective of the heteronormative man, and the pejoration of terms for women @xcite @xcite . 4 creating a women-produced dataset challenges the dominant gender perspective by explicitly labeling where minoritized genders' perspectives are missing (d' @xcite @xcite . ## related work evidence of bias in ml data and models abound regarding gender @xcite , disability @xcite , racial-ized ethnicities @xcite , politics and economics @xcite , and, for an intersectional approach @xcite , a combination of characteristics @xcite @xcite . harms from such biases are also well documented @xcite @xcite @xcite . despite numerous bias mitigation approaches put forth (cao and @xcite @xcite @xcite , many have limited efficacy, failing to address the complexity of biased language @xcite @xcite . methods of removing bias tend to be mathematically focused, such as @xcite and @xcite . as mccradden et al. (2020) state, typical ml bias mitigation approaches assume biases' harms can be mathematically represented, though no evidence of the relevance of proposed bias metrics to the real world exists. on the contrary, @xcite found no correlation between a commonly used intrinsic bias metric, word embedding association test, and extrinsic metrics in the downstream tasks of coreference resolution and hate speech detection. due to the misalignment between abstract representations of bias and the presence and impact of bias, this paper presents a taxonomy to measure biased language at its foundation: words. limitations to bias mitigation efforts also result from overly simplistic conceptualizations of bias @xcite @xcite . nlp gender bias work, for example, often uses a binary gender framework either in its conceptualization (such as @xcite ) or application (such as @xcite ), and tends to focus on one variety of gender bias, stereotypes @xcite @xcite . nlp bias work more generally often asserts a single ground truth @xcite @xcite . despite evidence that bias varies across domains @xcite , approaches to mitigating bias have yet to address the contextual nature of biased language, such as how it varies across time, location, and culture @xcite @xcite . this paper adopts a data feminist (d' @xcite and perspectivist ap-proach (basile, 2022) to situate identification and measurement of bias in a particular context. data feminism views data as situated and partial, drawing on feminist theories' view of knowledge as particular to a time, place, and people @xcite @xcite . similarly, the perspectivist data manifesto encourages disaggregated publication of annotated data, recognizing that conflicting annotations may all be valid @xcite . indigenous epistemologies, such as the lakota's concept of wa ȟkàn, further the notion of the impossibility of a universal truth. translated as "that which cannot be understood," wa ȟkàn communicates that knowledge may come from a place beyond what we can imagine @xcite . our taxonomy thus permits annotations to overlap and record uncertainty, and our aggregated dataset incorporates all annotators' perspectives. encouraging greater transparency in dataset creation, @xcite and jo and gebru (2020) caution against creating datasets too large to be adequately interrogated. @xcite , @xcite , and @xcite propose new documentation methods to facilitate critical interrogation of data and the models trained on them. our appendices include a data statement documenting the creation of the annotated data presented in this paper ( §b). to maximize the transparency of our data documentation, we will publish the data only after further interrogation of its gender bias annotations, including collaborative analysis with the centre for research collections. ## methodology to practically apply theories and approaches from nlp, data feminism, and indigenous epistemologies, we apply the case study method, common to social science and design research. case studies use a combination of data and information gathering approaches to study particular phenomena in context @xcite , suitable for annotating gender biased language because gender and bias vary across time, location, and culture. furthermore, case studies report and reflect upon outliers discovered in the research process (ibid.), supporting our effort to create space for the perspectives of people minoritized due to their gender identity. after first developing the annotation taxonomy through an interdisciplinary literature review and participatory action research with archivists ( §5), we applied the taxonomy in a case study to create datasets annotated for gender bias ( §6). adopting our previously published bias-aware methodology @xcite , we employed participatory action research @xcite , collaborating with the institution that manages our data source: the centre for research collections. due to validity @xcite and ethical concerns @xcite with crowdsourcing, we hired annotators with expertise in archives (the domain area of the case study's data) and gender studies (the focus area of this paper's bias mitigation) to apply the taxonomy in a case study. hiring a small number of annotators will enable us to publish disaggregated versions of the annotated data, implementing data perspectivism @xcite . following the approach of @xcite to heritage, we consider heritage to be a process of engaging with the past, present, and future. annotators in this paper's case study visited, interpreted, and negotiated with heritage @xcite in the form of archival documentation. annotating archival documentation with labels that mark specific text spans as gender biased transforms the documentation, challenging the "authorized heritage discourse" (ibid., 29) of the heteronormative man. we aim such explicit labeling to recontextualize the archival documentation, transforming its language by placing it in a new social context @xcite : the 21 st century united kingdom, with gender conceptualized as a self-defined, changeable identity characteristic. we aim this negotiationthrough-annotation to guide the nlp models we will create with the data in the future towards more equitable representations of gender. ## annotation taxonomy our annotation taxonomy organizes labels (lettered) into three categories (numbered). category and label names are bolded. each label's listing includes a definition and example. examples are italicized; labeled text in each example is underlined. for every label, annotators could label a single word or multiple words. examples come from the archival documentation summarized in §6 except for 1(a), non-binary, and 3(d), empowering, because annotators did not find text relevant to their definitions (the "fonds id," or collection identifier, indicates where in the documentation example descriptions may be found). §7 further explains the rationale for the taxonomy's labels, and how they facilitate analysis and measurement of gender biased language. 1. person name: the name of a person, including any pre-nominal titles (i.e., professor, mrs., sir, queen), when the person is the primary entity being described (rather than a location named after a person, for example) we chose to build on the gender bias taxonomy of @xcite because the authors grounded their definitions of types of gender bias in gender studies and linguistics, and focused on identifying gender bias at the word level, aligning with our approach. though @xcite also provide a framework for defining types of gender bias, their framework focuses on relationships between people in a conversation, identifying "bias when speaking about someone, bias when speaking to someone, and bias from speaking as someone" (316). the nature of our corpus makes these gender bias dimensions irrelevant to our work: glam documentation contains descriptions that only contain text written about a person or people (or other topics); it does not contain text that provides gender information about who is speaking or who is being spoken to. additionally, despite writing of four gender values (unknown, neutral, feminine, and masculine), the dataset and classifiers of @xcite are limited to "masculine and feminine classes" (317). the authors also do not explain how they define "bias," limiting our ability to draw on their research. @xcite provide another gender bias taxonomy that builds on that of @xcite , resulting in overlaps between our taxonomies. however, @xcite focus on gender stereotypes, while our taxonomy considers other types of gender biases. though less explicit in the names of our taxonomy's labels, we also looked to the descriptions of gender and gender bias from cao and daumé @xcite , who point out the limited gender information available in language. the aim of our dataset creation differs from cao and daumé @xcite , though. they created data that represents trans and gender diverse identities in order to evaluate models' gender biases, specifically looking at where coreference resolution fails on trans and non-binary referents. by contrast, we aim to create a dataset that documents biased representations of gender, with the future aim of creating models that are able to identify types of gender bias in language. ## case study to demonstrate the application of the taxonomy, we present a case study situated in the united kingdom in the 21 st century, annotating archival documentation written in british english from the centre for research collections at the university of edinburgh (crc archives). this paper thus takes the first step in building a collection of case studies that situate nlp bias research in a specific context. @xcite . a collection of case studies would enable the nlp community to determine which aspects of bias mitigation approaches generalize across time, location, culture, people, and identity characteristics. the crc's archives' documentation served as a suitable data source because the documentation adheres to an international standard for organizing archival metadata (isad(g) (ica, 2011)), the archivists at the institution had found gender bias in the documentation's language, and the archivists were already engaged in efforts to mitigate gender bias in the archival documentation. the documentation describes a variety of heritage collections and items, such as letters, journals, photographs, degree certificates, and drawings; on a variety of topics, such as religion, research, teaching, architecture, and town planning. employees at the partner institution describe themselves as activists changing archival practices to more accurately represent the diverse groups of people that the archival collections are intended to serve. the annotation corpus consists of 24,474 sentences and 399,957 words, selected from the first 20% of the entire corpus of archival documentation from the partner institution's catalog (see §b.9 for more on this corpus). the categories of labels in the annotation taxonomy were divided among annotators according to the textual relations the labels record. hired annotators 1 and 2 (a1 and a2) labeled internal relations of the text with person name and linguistic categories, hired annotators 3 and 4 (a3 and a4) labeled external relations of the text with the contextual category, and the lead annotator (a0) labeled both relations with all categories. a1 and a3 labeled the same subset of archival documentation, and a2 and a4 labeled the same subset of archival documentation, ensuring every description had labels from all categories. the lead annotator labeled the same descriptions as a1 and a3, and a subset of the descriptions that a2 and a4 labeled (due to time constraints, a0 could not label all the same descriptions). prior to beginning annotation, gendered pronoun, gendered role, and occupation labels were automatically applied. the annotators corrected mistakes from this automated process during their manual annotation. we produced three instances of the annotation corpus: one for a0, one for each pair of hired annotators (a1 and a3, and a2 and a4), and one aggregated dataset. the aggregated dataset combines annotations from all five annotators, totaling 76,543 annotations with duplicates and 55,260 annotations after deduplication. manual reviews of each annotator's dataset informed the aggregation approach, which involved a combination of programmatic and manual steps. the data statement in §b details the aggregation approach. figure 1 displays the number of annotations in the aggregated dataset by label ( §a contains additional annotation figures). in line with perspectivist nlp (basile, 2022), the individual annotator's datasets will be published alongside the aggregated dataset, enabling researchers to interrogate patterns of agreement and disagreement, and enabling future work to compare the performance of classifiers trained on disaggregated and aggregated datasets. ## discussion and limitations the paper's annotation taxonomy builds on biased language research from nlp, information sciences, gender studies, and linguistics literature. the gender bias taxonomy of @xcite , which categorizes gender biases based on whether the bias comes from the sentence structure or the context (i.e. people, relationships, time period, location) of the language, served as a foundation. we adopted four labels from that taxonomy: gendered pronoun, gendered role, generalization, and stereotype (merging hitti et al.'s societal stereotype and behavioral stereotype categories). drawing on archival science and critical discourse analysis, and guided by participatory action research with archivists (e.g., interviews, workshops), we added to and restructured hitti et al.'s taxonomy. the person name labels were added so that the representation of people of different genders in the archival documentation could be estimated. annotators chose which label to apply to a person's name based on gendered pronouns or roles that refer to that person in the description in which their name appears. for example, "they" as singular for non-binary, "his" for masculine, and "she" for feminine; or "mx." for non-binary, "lady" for feminine, or "son" for masculine. the unknown, feminine, and masculine labels distinguish our approach from previous nlp gender bias work that has not allowed for uncertainty. guessing a person's gender risks misgendering (scheuerman et al., 2020), a representational harm @xcite , and fails to acknowledge that sufficient information often is not available to determine a person's gender with certainty @xcite . this led us to replace the initial labels of woman and man with feminine and masculine, recognizing that pronouns and roles are insufficient for determining how people define their gender. each person name label encompasses multiple genders. for instance, a person who identifies as a transwoman, as genderfluid, or as a cis woman may use feminine pronouns, such as "she," or feminine roles, such as "wife." though we aimed to create a taxonomy inclusive of all genders, we acknowledge this may not have been achieved, and welcome feedback on how to represent any genders inadvertently excluded. we also added three labels to the contextual category: occupation, omission, and empowering. occupation was added because, when combined with historical employment statistics, occupationlabeled text spans could inform estimates of the representation of particular genders within the collaborating archive's collections. furthermore, person name annotations combined with their occupations could guide researchers to material beyond the archive that may provide information about those people's gender identity. omission was added because, during group interviews, representatives from the collaborating archive described finding gender bias through the lack of information provided about women relative to the detail provided about men. empowering was added to account for how communities reclaim certain derogatory terms, such as "queer," in a positive, self-affirming manner @xcite . figure 1 displays how prevalent omission was in the annotated data: this label is the most commonly applied label from the contextual category. such prevalence demonstrates the value of interdisciplinary collaboration and stakeholder engagement, carried out in our participatory action research with domain experts. had archivists at the partner institution not been consulted, we would not have known how relevant omitted information regarding gender identities would be to identifying and measuring gender bias in archival documentation. the final annotation taxonomy includes labels for gendered language (specifically, gendered role, gendered pronoun, and all labels in the person name category), rather than only explicitly gender biased language (specifically, generalization, stereotype, and omission), because measuring the use of gendered words across an entire archives' collection provides information about gender bias at the overall collections' level. for example, using a gendered pronoun such as "he" is not inherently biased, but if the use of this masculine gendered pronoun far outnumbers the use of other gendered pronouns in our dataset, we can observe that the masculine is over-represented, indicating a masculine bias in the archives' collections overall. labeling gender-biased language focuses on the individual description level. for example, the stereotype of a wife playing a supporting role to her husband comes through in this description: jewel took an active interest in her husband's work, accompanying him when he travelled, sitting on charitable committees, looking after missionary furlough houses and much more. instructions for applying the taxonomy permitted labels to overlap as each annotator saw fit, and asked annotators to annotate from their contemporary perspective. approaching the archival metadata descriptions as discourse (meaning language as representations of the material, mental, and social worlds @xcite ), the taxonomy of labels represents the "internal relations" and "external relations" of the descriptions (ibid., 37). the person name and linguistic categories annotate in-ternal relations, meaning the "vocabulary (or 'lexical') relations" (ibid., 37) of the descriptions. to apply their labels, annotators looked for the presence of particular words and phrases (i.e., gendered pronouns, gendered titles, familial roles). the contextual category annotates external relations: relations with "social events ... social practices and social structures" @xcite . to apply contextual labels, annotators reflected on the production and reception of the language in the archival documentation. for instance, to apply the stereotype label, annotators considered the relationship between a description's language with social hierarchies in 21 st century british society, determining whether the term or phase adequately represented the possible gender diversity of people being described. ## conclusion and future work this paper has presented a taxonomy of gender biased language with a case study to support clarity and alignment in nlp gender bias research. recognizing the value of clearly defined metrics for advancing bias mitigation, the taxonomy provides a structure for identifying types of gender biased language at the level they originate (words and phrases), rather than at a level of abstraction (i.e., vector spaces). still, the case study presented in this paper demonstrates the difficulty of determining people's gender with certainty. while recognizing the value of nlp systems for mitigating harms from gender biased language at large scale, we contend that conceptualizations of gender must extend to trans and gender diverse gender expressions if nlp systems are to empower minoritized gender communities. future work will include the publication of the case study's datasets, analysis of the datasets, and document classification models trained on the datasets. the datasets will include each individual annotator's dataset and two aggregated datasets, one with duplicates across different annotators, and one deduplicated to exclude matching and overlapping annotations from different annotators. the analysis of the datasets and creation of models trained on them will be informed by participatory action research, incorporating perspectives from archivists, and from people of trans and gender diverse identities not represented in the research team. the dataset will be published in the same location as the code written to create the corpus of archival documentation and the annotated datasets. 5 the taxonomy and forthcoming datasets aim to guide nlp systems towards measurable and inclusive conceptualizations of gender. elizabeth yale. 2015. the history of archives: the state of the discipline. book history, 18(1):332-359. jieyu zhao, tianlu wang, mark yatskar, ryan cotterell, vicente ordonez, and kai-wei chang. 2019. gender bias in contextualized word embeddings. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 629-634, minneapolis, minnesota. association for computational linguistics. jieyu zhao, tianlu wang, mark yatskar, vicente ordonez, and kai-wei chang. 2018. gender bias in coreference resolution: evaluation and debiasing methods. in proceedings of the 2018 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 2 (short papers), pages 15-20, new orleans, usa. association for computational linguistics. a additional tables and figures figure 6 : confusion matrices normalized with a weighted average on the aggregated data's labels, so that class imbalances are taken into account. the top left confusion matrix displays intersections between the aggregated datasets labels, illustrating where the same text spans have more than one label. the remaining confusion matrices to display the agreement between an annotator's labels (y axis) and the aggregated data's labels (x axis). the y axis scale is the same for all matrices, ranging from zero to one. figure 7 : disagreeing and agreeing label counts across all annotators' datasets. the bar chart displays counts of the occurrence of disagreements and agreements across annotators' labels. annotations by two annotators with the same or overlapping text span but different labels are considered to be in disagreement. annotations by two annotators with the same or overlapping text span and the same labels are considered to be in agreement. agreements with the same text span are considered to be exact matches. agreements with different but overlapping text spans are considered to be overlaps. combined, the annotated datasets contain 198,520 annotations.