Examining the Causal Impact of First Names on Language Models: The Case of Social Commonsense Reasoning (2023)

Abstract: As language models continue to be integrated into applications of personal and societal relevance, ensuring these models’ trustworthiness is crucial, particularly with respect to producing consistent outputs regardless of sensitive attributes. Given that first names may serve as proxies for (intersectional) socio-demographic representations, it is imperative to examine the impact of first names on commonsense reasoning capabilities. In this paper, we study whether a model’s reasoning given a specific input differs based on the first names provided. Our underlying assumption is that the reasoning about Alice should not differ from the reasoning about James. We propose and implement a controlled experimental framework to measure the causal effect of first names on commonsense reasoning, enabling us to distinguish between model predictions due to chance and caused by actual factors of interest. Our results indicate that the frequency of first names has a direct effect on model prediction, with less frequent names yielding divergent predictions compared to more frequent names. To gain insights into the internal mechanisms of models that are contributing to these behaviors, we also conduct an in-depth explainable analysis. Overall, our findings suggest that to ensure model robustness, it is essential to augment datasets with more diverse first names during the configuration stage.

## introduction recent language models (lms) @xcite have shown remarkable improvements when used in nlp tasks and are increasingly used across various application domains to engage with users and address their personal and social needs, such as ai-assisted autocomplete and counseling @xcite . as these lms models are adopted, their social intelligence and commonsense reasoning have become more important, especially as ai models are deployed in situations requiring social skills @xcite . in this paper, we examine how first names are handled in commonsense reasoning (fig 1 ). to this end, we measure the causal effect that name instances have on lms' commonsense reasoning abilities. a key aspect of commonsense reasoning of lms should be that they provide consistent responses regardless of the subject's name or identity @xcite . that is, the reasoning behind "alice" should not differ from that about "james", for instance. given that first names can be a proxy for representation of gender and/ or race, this consistency is essential not only for the robustness but also for the fairness and utility of a lm. previous studies have revealed that pre-trained language models are susceptible to biases related to peoples' first names. for instance, in the context of sentiment analysis, certain names have been consistently associated with negative sentiments by language models @xcite . additionally, during text generation, names have been found to be linked to well-known public figures, indicating biased representations of names @xcite . furthermore, @xcite demonstrated that less common names are more likely to be 'subtokenized' and associated with negative sentiments compared to frequent names. these studies shed light on how pretrained language models disproportionately process name representations, potentially leading to biased outputs. while examining pre-trained language models is valuable to understand their capabilities and limitations, in many cases the models are fine-tuned, or adapted and optimized, to guarantee improved performance on specific downstream tasks, such as text classification, machine translation, and question answering, among others @xcite @xcite . given that fine-tuning pre-trained language models can lead to major performance gains @xcite , in this paper, we ask if performance disparities based on names still exist even when the models are fine-tuned. if so, we ask which components of the models contribute to performance disparities and to what extent. we design a controlled experimental setting to determine whether performance differences arise by chance or are caused by names. our contributions are three-fold foot_0 : ## task formulation we consider a dataset of commonsense reasoning examples @xmath0, where each item consists of a question @xmath1, three possible answer candidates @xmath2, c 2 , c 3 }, and a label @xmath3, which is the correct answer among the candidates. q and c serve as a template t, containing placeholders for names [n] and pronouns referring to the names, to ensure grammatical correctness, a pronoun placeholder np is set in variants of subject pronoun np 1 , object pronoun np 2 , and dependent possessive pronouns np 3 . an example of the data template is as follows: question q: typically every four months, [n] went to the doctor for a routine checkup and was told [np 1 ] needs rest. what will [n] want to do next? candidates c:{ (a) call the doctor, (b) finish all [np 2 ] projects and postpone the rest, (c) take time off from work} label y: (c) take time off from work ## causal graph a language model can be denoted as a function f , taking inputs as follows: we are interested in how first names (@xmath4. we hypothesize that there is a causal graph g that encodes possible causal paths relating first names to the model's prediction (fig 1 , right). 2 we identify both the direct effect and indirect effect on model prediction @xcite : we hypothesize that pronouns are an intermediate variable (@xmath5. intuitively, pronouns that refer to names can influence how models make their predictions. for example, this indirect effect indicates changes in model prediction when the pronouns differ (e.g. he vs. she) but the names remain the same or fixed (e.g. pat). pronouns inherently associate with the names they refer to, and this association may cue models to consider those names more strongly when generating a response. thus, we posit the effect of pronouns as an indirect effect. below, we formalize the causal mechanisms, intervention lists, and the effect size that measures the change in model prediction. ## explanations of causal effects the causal analysis shows the surface-level comparison of model outputs but fails to capture the nuanced processes underlying each model's reasoning. by probing the internal workings of the models, we seek to gain insights into how the models derive their conclusions and also where their approaches diverge. we use two approaches to gain a deeper understanding of the models' predictions. first, we analyze the models' internal representations to discern how they encode various names. specifically, we focus on the distinction in contextualization between the embeddings of frequent names and less frequent names. second, we apply a diagnostic technique based on neuron activation to pinpoint how the models process names. ## experimental setup dataset we use the socialiqa dataset from @xcite . the selection of this dataset is motivated by its suitability for investigating model behavior in a social context, as the dataset consists of questions for probing emotional and social intelligence in everyday situations. by analyzing the model's responses to questions pertaining to social and emotional intelligence, valuable insights can be gleaned regarding the models' handling of some nuances of human behavior. since the dataset is based on a social setting, it would be misleading if the models yielded different predictions based on different names. to construct the template t, we used the allennlp coreference resolution model @xcite , which has high performance 3 . this model is used to detect 3 f1 score 80.2 on conll benchmark dataset named entities and resolve their corresponding pronouns, facilitating the construction of templates for our experiments. names list we use u.s. census names dataset foot_2 , following @xcite to intervene the name placeholders. it contains 139 years of u.s. census baby names, their corresponding gender, and respective frequencies. to form intervention name lists based on frequency, we filtered out the most frequent k names over all years for n most , and the least frequent k names over all years for n least . we set @xmath6. model we use three widely used models, gpt2 @xcite , bert @xcite , and roberta @xcite . we customized each model with a linear layer not-finetuned (epoch 0) finetuned (epoch10) gpt2 bert roberta gpt2 bert roberta indirect effect most 0.055 0.107 0.074 0.052 0.047 0.037 least 0.043 0.091 0.171 0.053 0.039 0.031 female 0.072 0.145 0.185 0.079 0.063 0.051 male 0.030 0.059 0.034 0.0260 0.025 0.018 1 . comparing the first three columns (not-finetuned) with the subsequent three columns (fine-tuned), we observe that the causal effect of accuracy is not statistically significant when the models are fine-tuned. this trend holds consistently true across all three models examined in this study. this suggests that the direct effect of name characteristics on accuracy is not significant when fine-tuned. the effect sizes of the not-finetuned models are reported in accordance with previous literature that predominantly focuses on these models @xcite . however, it is crucial to emphasize the efficacy of fine-tuning, as it reflects a more realistic scenario for model deployment @xcite . we compared the effect sizes of the not-finetuned models with those of the fine-tuned models, thereby examining the impact of fine-tuning on model behavior. we also provide an analysis of the correlation between the model's accuracy and effect sizes in appendix d. agreement the analysis of the direct causal effect of agreement (d agr ) shows that a significant difference in name lists based on frequency persists even after fine-tuning all three models ( figure 2 illustrates the disentangled values for d agr across different epochs during the training phase. for both gpt2 and bert, a consistent gap between most and least is observed throughout the training epochs. in contrast, for roberta, although the gap is not consistent across all epochs, the agreement measures for most remain consistently higher than those for least. this discrepancy in the gap between roberta and the other models could potentially be attributed to the robust optimization design of roberta, which complements that of bert @xcite . also, these findings are consistent with the conclusion drawn by @xcite , who also observed that roberta generates the most robust results. overall, the findings indicate that the agreement ratio of least consistently remains lower than that of most throughout the training phase, suggesting that the predictions for least are more divergent. ## broader impact the data used in our analysis contains no private user information. as for ethical impact, the systematic experimental design we used provides an approach for conducting controlled experiments in the context of natural language processing research, particularly with a focus on the influence of first names on language models. ## limitation our investigation focuses on one aspect of commonsense reasoning restricted to one dataset. there may be numerous other factors in real-world applications. therefore, our findings may not comprehensively capture the entirety of commonsense reasoning phenomena. another limitation is that for the sake of simplicity and feasibility, we assumed a fixed threshold of @xmath8. however, this threshold may not be universally applicable to all contexts or datasets, and different thresholds could lead to different results. effect size: model d acc d agr gpt2 bert roberta gpt2 bert roberta @xmath9.473 (.142) -.427 (0.19) -.109 (.75) .536 (.089) -.500 (.117) .500 (.17) @xmath10.045 (.894) -.264 (.433) .055 (.873) -.555 (.077) -.55 (.077) -.591 (.056) most @xmath11.264 (.433) -.609 * (.047) -.073 (.832) .473 (.142) -.645 * (.032) -.6 (.051) most @xmath12.618 * (.043) -.264 (.433) .191 (.574) .391 (.235) -.418 (.201) -.145 (.67)