MISGENDERED : Limits of Large Language Models in Understanding Pronouns (2023)

Abstract: Content Warning: This paper contains examples of misgendering and erasure that could be offensive and potentially triggering. Gender bias in language technologies has been widely studied, but research has mostly been restricted to a binary paradigm of gender. It is essential also to consider non-binary gender identities, as excluding them can cause further harm to an already marginalized group. In this paper, we comprehensively evaluate popular language models for their ability to correctly use English gender-neutral pronouns (e.g., singular they, them) and neo-pronouns (e.g., ze, xe, thon) that are used by individuals whose gender identity is not represented by binary pronouns. We introduce Misgendered, a framework for evaluating large language models’ ability to correctly use preferred pronouns, consisting of (i) instances declaring an individual’s pronoun, followed by a sentence with a missing pronoun, and (ii) an experimental setup for evaluating masked and auto-regressive language models using a unified method. When prompted out-of-the-box, language models perform poorly at correctly predicting neo-pronouns (averaging 7.6% accuracy) and gender-neutral pronouns (averaging 31.0% accuracy). This inability to generalize results from a lack of representation of non-binary pronouns in training data and memorized associations. Few-shot adaptation with explicit examples in the prompt improves the performance but plateaus at only 45.4% for neo-pronouns. We release the full dataset, code, and demo at https://tamannahossainkay.github.io/misgendered/ .

## introduction from document retrieval to virtual assistants, large language models (llms) @xcite @xcite have become indispensable for various automated language processing tasks. given their proliferation, it is vital figure 1 : evaluation examples. each instance begins with a declaration of an individual's preferred pronouns, followed by text where a [pronoun] is missing. language models are evaluated for their ability to predict the pronoun accurately. the correct answer along with predictions from gpt-j are shown. efforts to address gender bias in natural language processing primarily focus on binary gender categories, female and male. they are aimed at either upstream bias, e.g., gendered associations in language models @xcite @xcite , or downstream bias, e.g., gendered information used for decision-making in tasks such as coreference resolution @xcite , machine translation @xcite etc. however, this is restrictive as it does not account for non-binary gender identities as they become more commonplace to openly discuss. this can perpetuate harm against non-binary individuals through exclusion and marginalization @xcite . this paper comprehensively evaluates popular language models' ability to use declared thirdperson personal pronouns using a framework, mis-gendered. it consists of two parts: (i) instances declaring an individual's pronoun, followed by a sentence with a missing pronoun ( § 3.1), and (ii) an experimental setup for evaluating masked and auto-regressive language models using a unified method ( § 3.2). we create a template-based evaluation dataset, for gendering individuals correctly given a set of their preferred pronouns. each evaluation instance begins with an individual's name and an explicit declaration of their pronouns, followed by a sentence in which the model has to predict a missing @xcite . for instance (fig. 1 ), 'aamari's pronouns are xe/xem/xyr/xyrs/xemself. aamari is undergoing a surgery. please pray for [pronoun] quick recovery.' we evaluate language models on their ability to fill in @xcite correctly, here with the possessive-dependent pronoun, xyr. sentences in our evaluation cover 5 different pronoun forms: nominative, accusative, possessivedependent, possessive-independent, and reflexive (e.g., they, them, their, theirs, and themself, respectively) for 11 sets of pronouns from 3 pronoun types: binary (e.g., he, she) foot_0 , gender-neutral (e.g., they, them), and neo-pronouns (e.g., xe, thon) foot_1 . we create 10 variations for each pronoun form and populate them with popular unisex, female, and male names, resulting in a total of 3.8 million instances. our evaluation shows that current language models are far from being able to handle gender-neutral and neo-pronouns. for direct prompting, we use models of varying sizes from six families comprising both auto-regressive and masked language models ( § 4.1). while most models are able to correctly use binary pronouns (average accuracy of 75.3%), all models struggle with neo-pronouns (average accuracy of 7.6%), and most with gender-neutral pronouns as well (average accuracy of 31.0%). this poor zero-shot performance could be due to the scarcity of representation of neo-pronouns and gender-neutral pronouns in pre-training corpora ( § 4.2). for example, there are 220× more occurrences of masculine pronoun tokens in c4 @xcite , the pre-training corpus for t5 @xcite ) models, than for the xe neo-pronouns. additionally, we also notice some memorized associations between pronouns and the gender of names. language models identify the non-binary pronouns most accurately for unisex names, whereas the bottom-performing names are either masculine or feminine. similarly, for binary pronouns, language models correctly predict masculine pronouns for masculine names with almost 3× more accuracy than feminine names. although language models do not perform well on predicting neo-pronouns in a zero-shot setting, models with few-shot learning abilities are able to adapt slightly with a few examples (in-context learning achieves an accuracy of up to 45.4% for neo-pronouns). however, performance plateaus with more shots, and it is not clear how this method of prompting with examples can be used to mitigate bias in downstream applications. future work should focus on further evaluation of language technologies on their understanding of nonbinary pronouns and mitigating biases. while we have made progress towards recognizing pronouns as an open class in nlp rather than a closed one, there is still much work to be done in this regard. overarching limitations of our work are its adherence to a western conceptualization of gender, as well as being confined to english. to facilitate further research, we release foot_2 the full dataset, code base, and demo of our work at https://tamannahossainkay.github. io/misgendered/ . ## background in this section, we present the social context in which our work is situated. the contemporary western discourse regarding gender differentiates between biological sex and gender identity. an individual's biological sex is assigned at birth and is associated with physical characteristics, such as chromosomes, reproductive organs, etc. (who, 2021; nih; @xcite . biological sex can be binary (female or male) or non-binary, eg. intersex with x, xxy genotypes (nih, 2021) etc. on the other hand, gender identity is an individual's names unisex, female, male pronouns exclusively binary female-male third-person personal pronouns are insufficient in such a diverse and dynamic landscape of gender. rather, expanding pronouns to include neo pronouns, such as, singular they, thon, ze, etc. is essential @xcite . spaces inclusive of lgbtqia+ persons encourage everyone to declare what pronouns to use to refer to them @xcite . pronoun declarations often include at least two pronoun forms, such as nominative and accusative (e.g., they/them, she/her), but can consist of all five pronoun forms (e.g., they/them/their/theirs/themself ). misgendering, i.e., addressing individuals using gendered terms that are not aligned with their gender identity are associated with a variety of harms @xcite . note that while an expanding view of gender identity creates a corresponding need for a wider range of pronouns, we cannot infer an individual's gender-identity from their preferred pronouns. for instance, the use of binary pronouns, such as she or he, does not necessarily indicate a binary gender identity, and similarly, the use of neo-pronouns, such as xe, does not imply an identity outside of the female-male binary. in this paper, we aim to establish a paradigm of evaluation of gender bias in nlp which takes into account the growing use of non-binary pronouns. we evaluate language models for one type of misgendering, which is @xcite we use in this paper for evaluating the ability of language models to correctly gender individuals. each row of this table consists of a pronoun group, with each column specifying the pronoun for each of the form for that group. ## misgendered framework the misgendered framework for evaluating the pronoun usage abilities of language models consists of (i) instances specifying an individual's pronoun, succeeded by a sentence missing a pronoun, and (ii) a unified method for evaluating masked and auto-regressive language models. ## results we test popular language models on their ability to correctly use declared pronouns when directly promoted using our evaluation dataset ( § 3.1). we conduct a thorough analysis of the variations in performance varies based on how pronouns were declared, the size of the models used, the form of the pronouns, and individual pronoun sets. we also illustrate the effect of using in-context learning, i.e., by providing models with examples of correct declared pronoun usage within the input prompts. ## related work there has been extensive work to understand and mitigate gender bias in language technologies @xcite @xcite . however, this has mostly been restricted to a binary view of gender. recently some work has been done to explore gender bias in a non-binary paradigm. for instance, @xcite discuss ways in which genderexclusivity in nlp can harm non-binary individuals. @xcite design open language generation (olg) evaluation focused on the experiences of transgender and non-binary individuals and the everyday sources of stress and marginalization they face. @xcite show that gender-neutral pronouns in danish, english, and swedish are associated with higher perplexities in language models. cao and daumé iii (2020) create specialized datasets for coreference resolutions with neo-pronouns, while @xcite provide desiderata for modelling pronouns in language technologies. however, these studies only focus on a few neo-pronouns (eg. xe and ze), and only @xcite and @xcite evaluate misgendering but only on a few language models and in zero-shot settings. we are the first to comprehensively evaluate large language models on a wide range of pronouns and pronoun forms. ## conclusion in this work, we show that current language models heavily misgender individuals who do not use feminine or masculine personal pronouns (e.g. he, she). despite being provided with explicitly declared pronouns, these models do not use the correct neopronouns and struggle even with gender-neutral pronouns like they. our analysis suggests the poor performance may be due to the scarcity of neo pronouns in the pre-training corpora and memorized associations between pronouns and names. when prompted with a few explicit examples of pronoun use, the language models do improve, suggesting some ability to adapt to new word use. nevertheless, it is unclear how few-shot prompting of pronoun use can mitigate bias and exclusion harms in practice in real-world downstream applications of language models. we hope researchers will expand upon our work to evaluate language techwe see improvement from gpt-j and op-6.7b as the number of shots is increased up to @xmath0. however, this performance increase does not continue to larger k. bold numbers represent the highest accuracy for a model and pronoun type, whereas underlined values represent the highest accuracy for a pronoun type.