“You are grounded!”: Latent Name Artifacts in Pre-trained Language Models (2020)

Abstract: Pre-trained language models (LMs) may perpetuate biases originating in their training corpus to downstream models. We focus on artifacts associated with the representation of given names (e.g., Donald), which, depending on the corpus, may be associated with specific entities, as indicated by next token prediction (e.g., Trump). While helpful in some contexts, grounding happens also in under-specified or inappropriate contexts. For example, endings generated for ‘Donald is a’ substantially differ from those of other names, and often have more-than-average negative sentiment. We demonstrate the potential effect on downstream tasks with reading comprehension probes where name perturbation changes the model answers. As a silver lining, our experiments suggest that additional pre-training on different corpora may mitigate this bias.

## introduction pre-trained language models (lms) have transformed the nlp landscape. state-of-the-art performance across tasks is achieved by fine-tuning the latest lm on task-specific data. lms provide an effective way to represent contextual information, including lexical and syntactic knowledge as well as world knowledge @xcite . lms conflate generic facts (e.g."the us has a president") with grounded knowledge regarding specific entities and events (e.g."the (current) president is a male"), occasionally leading to gender and racial biases (e.g."women can't be presidents") @xcite . in this work we focus on the representations of given names in pre-trained lms ( in a series of experiments we show that, depending on the corpus, some names tend to be grounded to specific entities, even in generic contexts. the most striking effect is of politicians in gpt2. for example, the name donald: 1) predicts trump as the next token with high probability; 2) generated endings of "donald is a" are easily distinguishable from any other given name; 3) their sentiment is substantially more negative; and 4) this bias can potentially perpetuate to downstream tasks. although these results are expected, their extent is surprising. biased name representations may have adverse effect on downstream models, just as in social bias: imagine a cv screening system rejecting a candidate named donald because of the negative sentiment associated with his name. our experiments may be used to evaluate the extent of name artifacts in future lms. 1 ## last name prediction as an initial demonstration of the tendency of pretrained lms to ground given names to prominent named entities in the media, we examine the nextword probabilities assigned by the lm. if high probability is placed on a named entity's last name conditioned on observing their given name (e.g., p (trump|donald) = 0.99), we take this as evidence that the lm is, in effect, interpreting the first-name mention as a reference to the named entity. we note that this is a lower bound on evidence for grounding: while it is reasonable to assume that nearly all mentions of, e.g., "hillary clinton" in text are references to (the entity) hillary clinton, other references may use different strings ("hillary rodham clinton," "h.r.c.," or just "hillary"). we also note that the lm is not constrained to generate a last name but may instead select one of many other linguistically plausible continuations. we examine greedy decoding of named entity last names systematically for each generative lm. to this end, we compile two sets of prominent named entities from the media and from history. 2 we construct four prompt templates ending with a given name to feed to each lm: (1) minimal: "[name]", (2) news: "a new report from cnn says that [name]", (3) history: "a newly published biography of [name]", and (4) informal: "i want to introduce you to my best friend, [name]". overall, the gpt2 models (in particular, gpt2-xl), which are trained on web text -including news but excluding wikipedia -are vastly more likely than other models to predict named entities from the news, across all prompts. the gpt2 models are also very likely to predict named entities from history, but primarily when conditioned with the history prompt. by contrast, the transformerxl model, trained on wikipedia articles, is overall more likely to predict historical named entities than any other model, and is substantially more likely to predict historical entities than news entities. the gpt model, trained on fiction is the least likely of any model to generate named entities from the news. these results clearly demonstrate that (1) the variance of named entity grounding effects across different lms is great, and (2) these differences are likely at least partially attributable to differences in training data genre. ## given name recovery given a text discussing a certain person, can we recover their (masked) given name? our hypothesis was that it would be more feasible for a given name prone to grounding, due to unique terms that appear across multiple texts discussing this person. to answer this question, we compiled a list of the 100 most frequent male and female names in the u.s., foot_3 to which we added the first names of the most discussed people in the media (section 2). using the template "[name] is a" we generated 50 endings of 150 tokens for each name, with each of the generator lms ( figure 1 illustrates the ease of distinguishing texts discussing hillary from others (gpt2-large). we masked the name ("[mask] is a..."), computed the bert vectors, and projected them to 2d using t-sne @xcite . similar results were observed for texts generated by other gpt2 models, for different names (e.g., donald, bernie), and with other input representations (tf-idf). ## sentiment analysis following @xcite , we can expect endings ( §3) discussing specific named entities to be associated with sentiment more consistently than those discussing hypothetical people. we predict sentiment using the allennlp sentiment analyzer @xcite trained on the stanford sentiment treebank @xcite . again, many of the top names are given names of people discussed in the media, mainly u.s. politicians, and more so in the gpt2 models. 6 we found the variation among the most positive scores to be low. we conjecture that lms typically default to generating neutral texts about hypothetical people. pre-trained lms are now used as a starting point for a vast array of downstream tasks @xcite , raising concerns about unintended consequences in such models. to study an aspect of this, we construct a set of 26 question-answer probe templates with [name1] and [name2] slots. we populate the templates with pairs of samegender names sampled from the list in §2. we evaluate the expanded templates on a set of lms fine-tuned for either squad (exemplified in figure 2 ; @xcite , or (slightly tweaked) winogrande @xcite , with optional pre-fine-tuning on race @xcite . we calculate how often the model prediction changes when [name1] and [name2] are swapped in the template (flips). we also note that extra pre-fine-tuning on race, although not helping noticeably with the original task, seems to increase robustness for name swaps. 6 see section c for examples. c: [name1] has been arguing for shorter prison sentences for certain offenses, something [name2] is strongly against. q: who is more likely to be considered tough on crime? ## related work social bias. there is multiple evidence that word embeddings encode gender and racial bias @xcite @xcite , in particular in the representations of given names @xcite . bias can perpetuate to downstream tasks such as coreference resolution @xcite @xcite , natural language inference @xcite , machine translation @xcite , and sentiment analysis @xcite . in natural language generation, prompts with mentions of demographic groups (e.g., "the gay person was") may generate stereotypical texts @xcite . named entities. @xcite used pre-trained lms to analyze power, sentiment, and agency aspects of entities, and found the representations were biased towards the lm training corpus. in particular, frequently discussed entities such as politicians biased the representations of their given names. @xcite showed that bias reflected in the language describing named entities is encoded into their representations, in particular reporting bias. people rarely state the obvious @xcite , thus uncommon events are reported disproportionally, and their frequency in corpora does not directly reflect real-world frequency @xcite . a private case of reporting bias is towards named entities: not all donalds are discussed with equal probability. web corpora specifically likely suffer from media bias, making some entities more visible than others (coverage bias; d'alessio and @xcite , sometimes due to "newsworthiness" (structural bias; van dalen, 2012). ## ethical considerations and conclusion we explored biases in pre-trained lms with respect to given names and the named entities that share them. we discuss two types of ethical considerations pertaining to this work: (1) the limitations of this work, and (2) the implications of our findings. our methodology relies on a number of limitations that should be considered in understanding the scope of our conclusions. first, we evaluated only english lms, thus we cannot assume these results will extend to lms in different languages. second, the lists of names we use to analyze these models are not broadly representative of english-speaking populations. the list of most common given names in the u.s. are over-representative of stereotypically white and western names. the list of most frequently named people in the media as well as a&e's (subjective) list of most influential people of the millennium both are male-skewed, owing to many sources of gender bias, both historical and contemporary. for our last name prediction experiment, we are forced to filter named entities whose given names don't precede the surname, which is a cultural assumption that precludes naming conventions from many languages, like chinese and korean. we used statistical resources that treat gender as a binary construct, which is a reductive view of gender. we hope future work may better address this limitation, as in the work of cao and daumé @xcite . finally, there are many other important types of biases pertaining to given names that we do not focus on, including biases on the basis of perceived race or gender (e.g. @xcite . while our experiments shed light on artifacts of certain common u.s. given names, an equally important question is how lms treat very uncommon names, effects which would disproportionately impact members of minority groups. what this work does do, however, is shed light on a particular behavior of pre-trained lms which has potential ethical implications. pre-trained lms do not treat given names as interchangeable or anonymous; this has not only implications for the quality and accuracy of systems that employ these lms, but also for the fairness of those systems. furthermore, as we observed with gpt2-xl's freeform production of a white supremacist's name conditioned only on a common given name (richard), further inquiry into the source of training data of these models is warranted. xlnet: generalized autoregressive pretraining for language understanding. corr, abs/1906.08237. jieyu zhao, tianlu wang, mark yatskar, vicente ordonez, and kai-wei chang. 2018. gender bias in coreference resolution: evaluation and debiasing methods. in proceedings of the 2018 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 2 (short papers), pages 15-20, new orleans, louisiana. association for computational linguistics.