Inferring Gender: A Scalable Methodology for Gender Detection with Online Lexical Databases (2022)

Abstract: This paper presents a new method for automatic detection of gendered terms in large-scale language datasets. Currently, the evaluation of gender bias in natural language processing relies on the use of manually compiled lexicons of gendered expressions, such as pronouns and words that imply gender. However, manual compilation of lists with lexical gender can lead to static information if lists are not periodically updated and often involve value judgements by individual annotators and researchers. Moreover, terms not included in the lexicons fall out of the range of analysis. To address these issues, we devised a scalable dictionary-based method to automatically detect lexical gender that can provide a dynamic, up-to-date analysis with high coverage. Our approach reaches over 80% accuracy in determining the lexical gender of words retrieved randomly from a Wikipedia sample and when testing on a list of gendered words used in previous research.

## introduction there is a growing body of research on gender bias embedded in trained language models as well as on allocational and representational harms caused by the deployment of these models. there have moreover been increasing calls for early and thorough data description and curation in order to gain insights into how, for instance, gender stereotyping or quality of service bias is propagated from data into a language model. what both of these strands of research have in common is their reliance on pre-defined lexicons of terms related to gender. in english, gendered words most commonly include pronouns (he, she, they, etc.), and also words that carry lexical gender, such as boyfriend, policewoman, or prince. previous works on gender bias in language technologies often use manually compiled lists of words carrying lexical gender to, for example, mitigate gender stereotyping through data augmentation @xcite , assess trans-exclusionary bias in co-reference annotations (cao and @xcite or evaluate gender inequalities in wikipedia article titles @xcite . however, curated lists are limited in their coverage of terms that contain lexical gender and can become outdated if not maintained. to address this issue, we present a scalable algorithmic method to determine lexical gender by querying a word's dictionary definitions for a small subset of definitively gendered words. our method allows for high-coverage, instantaneous detection of words carrying lexical gender, which eliminates the need to manually compile and maintain static lists of gendered words. this not only facilitates the extension of previous work on gender bias in nlp, but can also be used for a more detailed analysis on the representation of gender in large-scale language datasets used to train large language models like bert @xcite or gpt-2 @xcite . by combining the gender labels obtained from @xcite , wordnet ® (princeton university, 2010) and dictionary.com @xcite , our method reaches an accuracy of 84% in determining the lexical gender of words in a random sample of 1,000 wikipedia articles and 87% accuracy on a list of words carrying lexical gender adapted from previous research. the code for the algorithm, evaluation methods and datasets are available foot_0 . in the following section we first outline the conceptions of linguistic gender used in this research and secondly present an overview of research on gender in language technology that relies on curated lists of gendered words. thirdly, we discuss prior approaches to algorithmic gender inference. section 3 gives a detailed overview of the algorithm and section 4 introduces the datasets used to assess our gender detection algorithm. we present quantitative and qualitative results in section 5 and discuss limitations as well as avenues for future development. ## background when dealing with the category of gender in language technology, it is important to make a distinction between the social category of gender and gender in a linguistic sense. while social gender relates to the complex property, performance and experience of one's own and others' gender within society @xcite , linguistic gender describes the expression of gender within grammar and language. in english, linguistic gender mainly encompasses ways to express gender as female, male or gender-indefinite @xcite . social gender, as an extra-linguistic category, includes a more fluid view of gender aside from male and female categories. this includes transgender, genderqueer and other non-binary experiences and expressions of gender @xcite . as @xcite and cao and daumé iii (2020) point out, there is no "one-to-one" mapping between social and linguistic gender. however, the two are influenced by each other: on one hand, expressions of gender in language are subject to changing norms in society @xcite , on the other hand, the way gender is represented in language influences the conception of gender within society @xcite . thus, being able to evaluate gendered expressions in language provides insights into societal conceptualisations of gender. since this research explicitly focuses on lexical gender in english, which is a linguistic category, we give an overview of linguistic gender in english in section 2.1. section 2.2 explores the role lexical gender information plays in different areas of research on gender bias in nlp, which simultaneously present possible areas of application for our method of lexical gender inference. section 2.3 discusses two prior algorithmic systems for lexical gender inference in english. ## data we used two test datasets to evaluate and run the algorithm. the first dataset, which we call gold standard hereafter, contains nouns that have a clear lexical gender and were mainly sourced from previous research on gender bias. the second dataset contains 1,000 randomly sampled wikipedia articles, which we used to extract gendered nouns. the following describes both datasets in detail. ## conclusion we have presented a method to automatically determine the lexical gender of a given word by querying its dictionary definitions. the performance of the algorithm on a gold standard dataset of gendered nouns based on related literature, as well as a set of nouns sampled from 1,000 randomly selected wikipedia articles, reached up to 87% accuracy. previous research on gender bias in nlp used manually compiled lists of gendered words for data evaluation, retrieval, manipulation, and the synthetic creation of data. in contrast, our method is scalable and has a high, dynamic coverage, which gives it a variety of applications within past and future research on gender bias in nlp. these include e.g. the assessment of gender representations in large-scale corpora, the retrieval of gendered words for which gender-neutral replacements need to be found, as well as determining whether male-centric language such as epicene he is used in coreference resolution clusters.