Robust Pronoun Fidelity with E nglish LLM s: Are they Reasoning, Repeating, or Just Biased? (2024)

Abstract: Robust, faithful, and harm-free pronoun use for individuals is an important goal for language model development as their use increases, but prior work tends to study only one or two of these characteristics at a time. To measure progress towards the combined goal, we introduce the task of pronoun fidelity: Given a context introducing a co-referring entity and pronoun, the task is to reuse the correct pronoun later. We present RUFF, a carefully designed dataset of over 5 million instances to measure robust pronoun fidelity in English, and we evaluate 37 model variants from nine popular families, across architectures (encoder-only, decoder-only, and encoder-decoder) and scales (11M-70B parameters). When an individual is introduced with a pronoun, models can mostly faithfully reuse this pronoun in the next sentence, but they are significantly worse with she/her/her, singular they, and neopronouns. Moreover, models are easily distracted by non-adversarial sentences discussing other people; even one sentence with a distractor pronoun causes accuracy to drop on average by 34 percentage points. Our results show that pronoun fidelity is not robust, in a simple, naturalistic setting where humans achieve nearly 100% accuracy. We encourage researchers to bridge the gaps we find and to carefully evaluate reasoning in settings where superficial repetition might inflate perceptions of model performance.

## introduction third-person pronouns (he, she, they, etc.) are words that construct individuals' identities in conversations @xcite . in english, these pronouns mark referential gender for the entity they are referring to, which can also index an individual's social gender, e.g., man, woman, nonbinary (cao and @xcite . correctly using the pronouns an individual identifies with is im-portant, as misgendering (including through incorrect pronoun use) can in the best case be a social faux pas @xcite and in the worst case, cause psychological distress, particularly to transgender individuals @xcite . accordingly, it is important for large language models (llms) to use pronouns faithfully and without causing harm. to this end, many studies have explored how llms handle pronouns, showing that they stereotypically associate pronouns and occupations @xcite , reason about co-referring pronouns and entities better when they conform to stereotypes @xcite , fail when exposed to novel pronoun phenomena such as neopronouns @xcite , and cannot consistently reuse neopronouns during generation @xcite . these shortcomings create differences in quality of service and cause representational harm, amplifying discrimination against certain pronoun users @xcite . in work on llm pronoun use, a question that has gone unexamined thus far is: how robust is model faithfulness to pronouns when discussing more than one person? to answer this question, we propose pronoun fidelity ( §2), a new task to investigate realistic model reasoning about pronouns, and we introduce ruff ( §3), a novel, large-scale dataset of over 5 million instances, to evaluate this task. with this dataset, we present an analysis of pronoun fidelity across 37 variants from nine popular language model families covering architectures and scales, to investigate whether models are reasoning, repeating, or just biased. first, we collect model pronoun predictions for occupations in the absence of context, to establish a ''bias baseline'' ( §5). next, we evaluate whether models can overcome their biased pronoun predictions when explicitly shown what pronoun to use in context ( §6). all models are good at this task, but there are significant disparities across pronoun sets. we then test the robustness of this result by inserting naturalistic distractor sentences using a different pronoun to talk about another person ( §7). even one non-adversarial distractor sentence vastly deteriorates model performance, as shown in figure 1 . finally, in a detailed error analysis ( §8), we disentangle whether model errors can be attributed to distraction or falling back to bias, finding that encoder-only and decoder-only models behave in fundamentally different ways. overall, our results show that models struggle to reason about pronouns in a simple, naturalistic setting and highlight the need for careful task design to ensure that superficial repetition does not lead to inflated claims about model reasoning. we release all code and data to encourage researchers to bridge the gaps we find: https://github .com/uds-lsv/robust-pronoun-fidelity . introduction: the accountant had just eaten a big meal so her stomach was full. task sentence: the accountant was asked about charges for preparing tax returns. more formally, an introduction sentence i(e a , p a ) establishes a coreference between an entity e a and a pronoun p a . a distractor sentence the probability p of reconstructing the correct pronoun p a in the sentence t(e a , p), given the context. ## ruff dataset to evaluate robust pronoun fidelity at scale, we create ruff, an evaluation dataset of narratives. each dataset instance describes a simple narrative with 1-2 people, but rather than narrative data that focuses on commonsense event reasoning @xcite , we focus on pronominal reasoning, as in @xcite . specifically, we examine four third-person pronouns in three grammatical cases (nominative, accusative, and possessive dependent); in addition to the english masculine (he/him/his) and feminine (she/her/her) pronouns, we heed @xcite call for more inclusive nlp research by examining two more pronoun sets that are less wellstudied in nlp: singular they (they/them/their), the pronoun of choice of over 75% of respondents to the gender census @xcite , and xe/xem/ xyr, the most popular neopronoun according to the same census. our narratives cover 60 occupations and corresponding participants (see appendix a), following winogender schemas @xcite , as their bias characteristics are well-studied in nlp. in total, ruff contains over 5 million data instances. each instance is designed to have an unambiguous answer, and is constructed with a 3-step pipeline: template creation ( §3.1), template assembly ( §3.2), and data validation ( §3.3). ## experimental setup we list our models, evaluation methods, and metrics. further details are provided in appendix e. ## model predictions with no context we begin by creating a ''bias baseline,'' i.e., obtaining pronoun predictions from models on our task sentences in the absence of any context. in section 6, we will examine whether models can overcome this bias with reasoning when provided with context establishing a single correct answer. example: the accountant was asked about charges for preparing tax returns. no single answer (among his, her, their, xyr) as we cannot evaluate accuracy on a task with no single correct answer, we show the counts of model predictions of different pronoun declensions in figure 4 , averaged over all models. model-specific counts are shown in appendix g. even though our task sentences are designed such that any pronoun set can be used grammatically, all models tend to assign higher probability to he/him/his than other pronoun sets. obtaining pronoun predictions without context is a popular method to measure model bias, with numerous papers @xcite showing that associations between occupations and pronouns are based on social gender stereotypes, e.g., doctor-he and nurse-she. however, model pronoun predictions might reflect dataset artifacts such as the choice of occupations, or be a statistical accident of the chosen templates @xcite . in addition, intrinsic biases may not correlate with actual pronoun use with context @xcite . in order to test for such extrinsic behaviors, the rest of this paper examines whether models can override their intrinsic statistical biases on these same templates when provided with the right pronoun to use. ## injecting an introductory context when models are provided with an introductory sentence explicitly establishing the pronoun to use for an entity, can they use that pronoun to refer to the same entity in the immediate next sentence? example: the accountant had just eaten a big meal so her stomach was full. the accountant was asked about charges for preparing tax returns. correct answer: her as figure 5 shows, all models perform better than chance at pronoun fidelity with a simple introduction (up to 0.95 with mosaicbert), but not as well as humans, who achieve perfect performance. we also see improvements with increasing model scale, with the exception of albert-v2, as in tay et al. @xcite . which pronouns are harder? even in the simplest case of the pronoun fidelity task, patterns emerge when split by pronoun, as shown in figure 6 . overall model accuracy on he/him/his is significantly higher than she/her/her, which in turn is significantly higher than both they/ them/their and xe/xem/xyr, in line with previous findings that language technology has gaps when it comes to neopronouns @xcite . models show intriguing patterns with these last two pronoun sets. most encoder-only models appear to handle the neopronoun better than singular they (e.g., bert-large has an accuracy of 0.78 on xe/xem/xyr compared to 0.60 on they/ them/their), which warrants further investigation. decoder-only models smaller than 6.7b parameters struggle with the neopronoun, with every opt and pythia model smaller than 2.7b parameters performing below chance, and in some cases (e.g., pythia-14m, pythia-70m, and pythia-60m) even performing close to 0.0. beyond this scale, however, models perform better on xe/xem/xyr than on singular they, with llama-13b achieving 0.96 accuracy on the neopronoun. these differences are statistically significant. as the training data for individual model families is the same, this might suggest that decoder-only models generalize to novel pronouns starting at the scale of 6.7b parameters, but in light of @xcite , this result could just as well be a mirage resulting from our use of accuracy, a discontinuous metric. in either case, our observations could also explain the poor performance that some previous studies of neopronouns find, as the largest model that @xcite experiment with, for instance, is opt-6.7b. the lower performance of bigger models with singular they could also be a reflection of human processing difficulties with definite, specific singular they, as has been observed in linguistics @xcite . ## adding distractors to further probe whether models actually ''reason'' when provided with context, we system-atically inject sentences containing distractor pronouns between the introduction and the task, reflecting a natural usage scenario where multiple people are discussed with definite references and pronouns. example: the accountant had just eaten a big meal so her stomach was full. the taxpayer needed coffee because their day had started very early. their sleep had been fitful. the accountant was asked about charges for preparing tax returns. correct answer: her figure 7 shows that distractors degrade performance for all models. encoder-only and decoderonly models show different performance curves as more distractors are added: all decoder-only models get steadily worse, whereas encoder-only models perform the worst with one distractor and then seem to slowly recover, never quite reaching figure 7 : with more distractors, decoder-only models (above) get steadily worse; encoder-only models (below) get worse with one distractor and then recover, plateauing below their no-distractor accuracy. using vanilla and chain-of-thought prompting (appendix h) show the same patterns of degradation, reinforcing that model pronoun fidelity is not robust, and good performance with no distractors ( §6) is likely not due to ''reasoning'' at all. ## distractibility versus bias in adding distractor sentences, we add distance from the introduction via additional tokens that might make the model forget the original occupation-pronoun association, and the distractor pronoun also acts as a competing token that the model might accidentally repeat. in this section, we focus on the error cases to disentangle whether models are ''forgetting'' and reverting to biased predictions from section 5, or if they are actually being distracted. when a model gets the answer wrong, it is for one of three reasons: (1) distractibility, i.e., repeating the distractor pronoun, (2) bias, i.e., reverting to the model's context-free prediction, or (3) picking a completely different pronoun. our example illustrates all three possibilities, and we hypothesize that the first two possibilities are much more frequent than the third. in cases where the distractor pronoun is the same as the model's context-free prediction, it is impossible to disentangle distractibility and bias just from the model's prediction. hence, we exclude these and focus on the unambiguous error cases. as expected, we find that 74-93% of unambiguous model errors can be attributed to either model distractibility or bias. ## discussion and future work our results show that even the biggest models of today are not up to the task of pronoun fidelity once it includes a single sentence discussing another person. all models are easily distracted, but encoder-only models and decoder-only models show very different patterns both in performance degradation with more distractors and their reasons for errors. performance on this type of reasoning task should be evaluated carefully, with attention to how the overall patterns break down by different pronouns, and accounting for the possibility of repetition. below we expand on some questions raised by our findings. ## conclusion we introduce the task of pronoun fidelity to evaluate robust, faithful and harm-free pronoun use in language models, and we present ruff, a dataset we designed to evaluate it. we find evidence of faithful pronoun use only in a very simple setting, i.e., when only one person is discussed. even here, models show significant performance disparities with neopronouns, singular they and she/her/her, compared to he/him/his. even adding a single sentence about a second individual with a different pronoun causes accuracy to drop dramatically, showing that pronoun fidelity is neither robust to non-adversarial distractors nor due to ''reasoning.'' as more distractor sentences are added, encoder-only models perform better overall, but increasingly revert to biased predictions, while decoder-only models get increasingly distracted. our results show that in a setting that is very simple for humans, widely used large language models are unable to robustly and faithfully reason about pronouns, and continue to amplify discrimination against users of certain pronouns. we encourage researchers to bridge the performance gaps we report and to more carefully evaluate ''reasoning,'' especially when simple repetition could inflate perceptions of model performance. whose bias? our task as it is defined in section 2 is much broader than the scope of our dataset. we focus on occupations due to the wide attention they have received in prior literature, but we continue a long tradition of ignoring biases relating to the participants, e.g., child, taxpayer, etc. in addition, pronoun fidelity is only one dimension of inclusive language model behavior, and indeed only one way in which misgendering occurs in language, even in morphologically poor languages such as english.