Nichelle and Nancy: The Influence of Demographic Attributes and Tokenization Length on First Name Biases (2023)

Abstract: Through the use of first name substitution experiments, prior research has demonstrated the tendency of social commonsense reasoning models to systematically exhibit social biases along the dimensions of race, ethnicity, and gender (An et al., 2023). Demographic attributes of first names, however, are strongly correlated with corpus frequency and tokenization length, which may influence model behavior independent of or in addition to demographic factors. In this paper, we conduct a new series of first name substitution experiments that measures the influence of these factors while controlling for the others. We find that demographic attributes of a name (race, ethnicity, and gender) and name tokenization length are both factors that systematically affect the behavior of social commonsense reasoning models.

## introduction social science studies have shown that individuals may face race or gender discrimination based on demographic attributes inferred from names @xcite @xcite . similarly, large language models exhibit disparate behaviors towards first names, both on the basis of demographic attributes @xcite and prominent named entities @xcite . such model behavior may cause representational harms @xcite if names associated with socially disadvantaged groups are in turn associated with negative or stereotyped attributes, or allocational harms @xcite if models are deployed in real-world systems, like resume screeners @xcite . the task of social commonsense reasoning @xcite , in which models must reason about social norms and basic human psychology to answer questions about interpersonal situations, provides a particularly fruitful setting figure 1 : a social commonsense reasoning multiplechoice question example identified by sodapop @xcite where the model differentially associates "nichelle" with "violent" and "nancy" with "quiet". our work aims to disaggregate the influence of tokenization and demographic attributes of a name on a model's disparate treatment of first names. we obtain the race statistics from @xcite . for studying the phenomenon of name biases in nlp models. questions in the social iqa dataset @xcite , for example, describe hypothetical social situations with named, but completely generic and interchangeable, participants (e.g. "alice and bob"). social iqa questions require models to make inferences about these participants, yet they maintain the convenient property that correct (or best) answers should be invariant to name substitutions in most or all cases. leveraging this invariance property, prior work @xcite has demonstrated that social commonsense reasoning models acquire unwarranted implicit associations between names and personal attributes based on demographic factors (fig. 1 ). building upon this finding, we investigate a natural follow-up question: why? we identify two possible factors that cause a model's disparate treatment towards names: demographic attributes and tokenization length. we hypothesize that names associated with different demographic attributes, in particular race, ethnicity, and gender may cause a model to represent and treat them differently. these demographic attributes are also strongly correlated with corpus frequency and tokenization length (wolfe and caliskan, 2021). tokenization (or segmentation) breaks down an input sentence into a series of subword tokens from a predefined vocabulary, each of which is then, typically, mapped to a word embedding as the input to a contemporary language model. a name's tokenization length refers to the number of subwords in the name following tokenization. in this work, we refer to singly tokenized and multiply tokenized names as those consisting of one or multiple tokens after tokenization, respectively. as a result, singly tokenized names are represented with a single embedding vector, while multiply tokenized names are represented by two or more. with these potential confounds, we attempt to address the research question: in social commonsense reasoning, to what extent do demographic attributes of names (race, ethnicity, and gender) and name tokenization length each have an impact on a model's treatment towards names? we first conduct an empirical analysis to understand the distribution of tokenization lengths in names given demographic attributes, and viceversa. adopting the open-ended bias-discovery framework, sodapop (an et al., 2023), we then analyze the impact of demographic attributes and tokenization length on model behavior. we find that both factors have a significant impact, even when controlling for the other. we conclude that due to correlations between demographics and tokenization length, systems will not behave fairly unless both contributing factors are addressed. finally, we show that a naïve counterfactual data augmentation approach to mitigating name biases in this task is ineffective (as measured by sodapop), concluding that name biases are primarily introduced during pre-training and that more sophisticated mitigation techniques may be required. ## demographic attributes and tokenization length are correlated previously, wolfe and caliskan (2021) have shown that white male names occur most often in pretraining corpora, and consequently, white male names are more likely to be singly tokenized. we replicate this finding by collecting 5,748 first names for 4 races/ethnicities (white, black, hispanic, and asian) and 2 genders (female and male) from a u.s. voter files dataset compiled by @xcite (specific data processing and name inclusion criteria are in appendix b.1). we compute and plot the conditional probabilities of tokenization length given demographic attributes (race/ethnicity and gender) and vice-versa in fig. 2 using the bert tokenizer @xcite . let st be the event that a name is singly tokenized. we see in fig. 2 that p (white|st ), p (st |white), p (male|st ), and p (st |male) are substantially higher than other conditional probabilities involving st foot_0 , confirming @xcite . these observations suggest that a model tends to represent white names and male names differently from others in terms of the tokenization length. given these substantial differences in tokenization lengths across demographic groups, we are motivated to investigate whether tokenization is a primary cause of disparate treatment of names across demographic groups. it is important to note here that, even if tokenization were the primary cause of disparate treatment of names across demographic groups, this discovery would not in itself resolve the fairness concerns of representational and allocational harms based on race, ethnicity and gender, but it might point to possible technical solutions. however, as we will show in the next section, disparate treatment of names across demographic attributes persists strongly even when controlling for tokenization length (and vice-versa). ## analyzing the influences via sodapop we follow sodapop @xcite to investigate how the two factors in § 2 influence a social iqa model's behavior towards names. ## counter-factual data augmentation we apply counter-factual data augmentation (cda) to the social iqa training set as we attempt to finetune a model that is indifferent to both tokenization length and the demographic attributes of names. we choose to experiment with cda because it would shed light on the source of name biases. if biases mostly arise from finetuning, we expect finetuning on social iqa with cda would largely address the problem; otherwise, biases mostly originate from pre-training and are not easily overridden during finetuning. for each social iqa sample, we identify the original names using stanford ner @xcite . we find that more than 99% of samples contain one or two names. we create copies of the mcq samples and replace the identified names with random names from our sampled sub-groups such that the overall name frequency is evenly distributed over tokenization lengths and demographic attributes, resulting in an augmented set whose size increases by 16×. we finetune a bert model with the augmented set (details in appendix b.2). however, this naïve solution is rather ineffective (fig. 4b ). this negative result is not surprising as it aligns with the observations that sodapop could detect biases even in models debiased with state-of-the-art algorithms @xcite . it also indicates that pre-training contributes to the biased model behavior. hence, a more sophisticated solution is needed to tackle this problem. name artifacts previous research indicates that language models exhibit disparate treatments towards names, partially due to their tokenization or demographic attributes @xcite @xcite . however, thorough analyses of the factors influencing first name biases are lacking in these works. while wolfe and caliskan (2021) study the systematic different internal representations of name embeddings in language models due to the two factors, we systematically study how the two factors both connect with the disparate treatment of names by a model in a downstream task. ## conclusion we have demonstrated that demographic attributes and tokenization length are both factors of first names that influence social commonsense model behavior. each of the two factors has some independent influence on model behavior because when controlling one and varying the other, we observe disparate treatment of names. when controlling for tokenization length (e.g. black male singlytokenized names vs white male singly-tokenized names) we still find disparate treatment. conversely, when we control for demographics (e.g. black female singly-tokenized vs black female triply-tokenized names), the model also treats those names differently. because demographic attributes (race, ethnicity, and gender) are correlated with tokenization length, we conclude that systems will continue to behave unfairly towards socially disadvantaged groups unless both contributing factors are addressed. we demonstrate the bias mitigation is challenging in this setting, with the simple method of counterfactual data augmentation unable to undo name biases acquired during pre-training.