Low Frequency Names Exhibit Bias and Overfitting in Contextualizing Language Models (2021)

Abstract: We use a dataset of U.S. first names with labels based on predominant gender and racial group to examine the effect of training corpus frequency on tokenization, contextualization, similarity to initial representation, and bias in BERT, GPT-2, T5, and XLNet. We show that predominantly female and non-white names are less frequent in the training corpora of these four language models. We find that infrequent names are more self-similar across contexts, with Spearman’s rho between frequency and self-similarity as low as -.763. Infrequent names are also less similar to initial representation, with Spearman’s rho between frequency and linear centered kernel alignment (CKA) similarity to initial representation as high as .702. Moreover, we find Spearman’s rho between racial bias and name frequency in BERT of .492, indicating that lower-frequency minority group names are more associated with unpleasantness. Representations of infrequent names undergo more processing, but are more self-similar, indicating that models rely on less context-informed representations of uncommon and minority names which are overfit to a lower number of observed contexts.

## introduction human social perception is linked to frequency of observation. @xcite show using functional magnetic resonance imaging (fmri) scans that humans are more aware of variation in the faces of members of their own race, and perceive members of other races as repeated instances of a social class, rather than as individuals. most people interact more with individuals of their own race, and develop better cognitive skills for differentiating members of the race they see most frequently. recent research indicates that state-of-the-art ai systems mirror such biased and unequal human perceptions. for example, @xcite show that under-representation in the training data of computer vision models causes poor performance on classification tasks for women and minority racial groups. first names are used in both social psychology and natural language processing (nlp) as a proxy to ground truth data for studying racial and gender biases. the implicit association test (iat) of @xcite finds that study participants perceive european-american names as more pleasant than african-american names, and @xcite demonstrate with the word embedding association test (weat) that human biases observed in the iat exist in static word embeddings, learned vector representations of words widely used in nlp. we use a list of first names labeled by gender and racial group based on u.s. social security administration data and the names dataset of @xcite to analyze how name frequency affects minority social groups in four neural language models: bert, gpt-2, xlnet, and t5. neural language models have advanced the state of the art in nlp, and are found in consequential nlp contexts such as google search @xcite . these models produce contextualized word embeddings, which incorporate information from the context in which the word occurs over a series of neural network layers. @xcite and @xcite show that racial, gender, and intersectional biases exist in neural language models. we examine whether under-representation in the training corpora of such models causes them to overfit nonwhite and female names, reducing model generalization for underrepresented minorities. we list our research questions and contributions: we report the percentage of names singly tokenized for eight demographic groups. minority and female group names are singly tokenized less than white and male names. single tokenization correlates with frequency, with spearman's ρ up to .835. we use spearman's ρ rather than pearson's ρ to capture monotonic with name frequency, as we observe effects primarily on a log scale. the null hypothesis is that frequency does not affect tokenization, social bias, and contextualization, and we disprove it by obtaining a two-tailed p-value. ## related work we survey prior work related to frequency's influence on static and contextualized word embeddings. static word embeddings are vector representations of words based on co-occurrence statistics of words in a training corpus. contextualized word embeddings are vector representations of words from a neural language model which incorporates information from context to minimize loss on a training objective, such as next-word prediction (language modeling). static word embeddings have one vector per word, while contextualized embeddings vary with context, allowing them to capture, for example, the sense of a polysemous word. the weat of @xcite shows that the strength of association of a set of target static word embeddings (e.g., two social groups) to two sets of polar attribute static word embeddings (e.g., pleasant/unpleasant) encodes widely shared non-social biases, stereotypes, and factual information about the world. @xcite extend the weat to neural language models with the sentence encoder association test (seat), which inserts weat target words into semantically bleached sentence templates such as "this @xmath0. @xcite mask target and attribute words in template sentences, and directly query bert's masked language modeling objective to compute an association of target words to attributes. @xcite evaluate the overall magnitude of bias in language models by extending the weat to the contextualized embedding association test (ceat). @xcite perturb the sparse word cooccurrence matrix of the glove static word embedding algorithm of @xcite by omitting co-occurrence information from a particular subsection of the corpus, and show that embeddings of rare words are the most biased and the most sensitive to corpus perturbations. @xcite find that the frequency of a word in a training corpus can twist gender direction and create features in static word embeddings which vary based on frequency. our work analyzes neural language models, which do not form representations directly based on co-occurrence statistics. @xcite examine static word2vec and transformer word embeddings used in machine translation and find that embeddings of high and low-frequency words lie in separate subregions of the embedding space, and that an embedding of a rare word and a common word can be distant even if they are semantically similar. the authors train an adversarial model to produce embeddings not separable based on frequency. @xcite propose byte-pair encoding dropout, a subword regularization algorithm which allows multi-ple byte-pair encoding segmentations of the same word and corrects the problem of rare subtokens existing in a separate subregion of embedding space. @xcite show that the top two directions using principal component analysis (pca) in skip-gram, glove, and continuous bag of words (cbow) static word embeddings encode frequency-related features, and that removing frequency-related features improves performance on tasks related to word similarity. @xcite find that beam search in neural machine translation prefers common tokens, including common subword tokens, to uncommon tokens. @xcite measure the stability of static word embeddings, defined as consistency in the percent overlap of nearest neighbors, and find that frequency contributes to semantic stability in word embeddings. @xcite demonstrate that contextualized word embeddings in bert, gpt-2, and elmo occupy an increasingly anisotropic vector space as they incorporate context. the most contextspecific, least self-similar tokens are the most frequent in the training corpora, and in bert, a token embedding becomes more similar to the embeddings in the context around it as it is contextualized. @xcite find that frequency distorts the geometry of contextualized embeddings from bert, which causes the model to over or underestimate semantic similarity of words based on frequency in the training corpus. ## background: neural language models our work considers bert, gpt-2, t5, and xlnet. we choose these neural language models both because they are commonly used and studied, and because they use three different subword tokenizers, allowing us to conduct our tokenization analysis across three different algorithms. we use the 12-layer cased implementation of each transformer in python with tensorflow and the hugging face transformers library of @xcite . appendix a provides details on these models. ## datasets we use a dataset of first names from tzioumis (2018) cross-referenced with u.s. social security administration (ssa) data to analyze frequency in language models. to obtain contextualized representations of the names from language models, we gather contexts from the reddit corpus of @xcite . for name frequency statistics, we process the training corpus of each model. ## approach and experiments we measure the effects of frequency on tokenization, social bias, and contextualization to understand disparities between majority social groups and the minority racial and gender social groups identifiable in our names dataset. ## results we find that common names are singly tokenized more than less common names; that name frequency correlates with social bias in bert, and to a lesser extent in gpt-2 and t5; and that infrequent and multiply tokenized names exhibit higher intralayer self-similarity and lower inter-layer similarity to initial representation, suggesting overfitting. ## discussion white male names occur more than any other social group in all training corpora we process. black and hispanic female names are least frequent, showing that frequency-based disparities in language models have notable consequences for non-white females. each language model represents white male names with a single token more than any other group. black and hispanic female names are the least represented with a single subtoken, with less than 10% singly tokenized in most models, reflecting disparities based on gender and race. in bert, we observe spearman's ρ of name frequency and racial bias of .492, indicating that less frequent minority names are more associated with unpleasantness. gender bias in bert exhibits negative correlation with frequency, indicating that it is reinforced by frequently observing a female name. while increased representation may address disparities in contextualization, fair and diverse context occurrence is also important to ensure that language models do not learn biased embeddings. infrequent and multiply tokenized names exhibit higher intra-layer self-similarity across language models, but lower inter-layer similarity to initial representation. such names incorporate less information from context, and change more from initial representation, suggesting that disparities in contextualization are the result of overfitting. our work focuses on first names, but underrepresentation of minority names suggests that words and ideas specific to minority groups are also likely under-represented in the training corpora of neural language models, and will have the same issues with regard to tokenization, bias, and contextualization. increased representation of minority groups in training corpora would mitigate many frequency-related problems, and correct biases in the corpora themselves. however, some words will always be more common than others. to reduce frequency-based disparities, language models trained on multiple objectives, such as t5, might incorporate an adversarial training objective such as the method described by @xcite . limitations of our work relate to using a list of names, which cannot capture many characteristics which may be biased in language models, such as sexual orientation. please see our ethical considerations section for a detailed discussion of ethical concerns and limitations related to our work. ## conclusion we show that names predominantly belonging to members of minority social groups occur less frequently than majority group names in training corpora of neural language models, and that low frequency results in lower rates of single tokenization, less contextualization, and overfit representations in neural language models.