A Study of Nationality Bias in Names and Perplexity using Off-the-Shelf Affect-related Tweet Classifiers (2024)

Abstract: In this paper, we apply a method to quantify biases associated with named entities from various countries. We create counterfactual examples with small perturbations on target-domain data instead of relying on templates or specific datasets for bias detection. On widely used classifiers for subjectivity analysis, including sentiment, emotion, hate speech, and offensive text using Twitter data, our results demonstrate positive biases related to the language spoken in a country across all classifiers studied. Notably, the presence of certain country names in a sentence can strongly influence predictions, up to a 23% change in hate speech detection and up to a 60% change in the prediction of negative emotions such as anger. We hypothesize that these biases stem from the training data of pre-trained language models (PLMs) and find correlations between affect predictions and PLMs likelihood in English and unknown languages like Basque and Maori, revealing distinct patterns with exacerbate correlations. Further, we followed these correlations in-between counterfactual examples from a same sentence to remove the syntactical component, uncovering interesting results suggesting the impact of the pre-training data was more important for English-speaking-country names.

## introduction recent trend in natural language processing research, like in works published at conference such as acl @xcite , is to provide opensource data and models @xcite . this practice not only enhances its value for general research purposes but also facilitates the deployment of these models in diverse operational settings by companies or stakeholders. applications such as customer experience, cv screening, social media analyses and moderation are example of applications that will directly impact the users in different ways. for this reason, the models applied at large scale should be scrutinized in order to understand their behavior and should tend to be fair by passing successfully a series of test to reduce their biases toward various target groups. past study @xcite showed that plms are impacted by names, and @xcite proposed a method to quantify this to detect biases of the model toward specific countries, using the country most common names as a proxy. we are showing in this paper that this bias is systematic in several widely-used off-the-shelf classifiers on english data, and propose a method to directly link the bias level with the perplexity of the plm contributions we propose an investigation into biases related to country-specific names in widely used off-the-shelf models @xcite , commonly deployed in production environments for twitter data. 1 our analysis reveals distinct biases in sentiment, emotion, and hate speech classifiers, showing a propensity to favor names from certain countries while markedly disfavoring those from less westernized nations, often by a large margin. furthermore, we establish a global-level correlation between the perplexity of associated plms and model predictions across both known and unknown (i.e., out-of-distribution; ood) languages, demonstrated through examples in english, basque, and maori. at a local level, we mitigate the influence of syntax on perplexity by examining the correlation among counterfactual examples generated through minor perturbations. notably, our findings suggest that the frequency of a name's occurrence during the training phase directly impacts the sentiment model's tendency to produce positive outputs, which highly disadvantage the non-english (i.e., ood) persons in a world we show examples with sentiment and hate speech for variation of the name "alexander" and two sentences s 1 and s n . s 1 : "i do not like you [per] you fucking bitch". the ner is applied to the production data to create templates, which are then filled randomly with most common names from gazeeters of different countries to create a pool of counterfactuals. the discrepancies in probabilities is quantified using metrics such as ∆. our method is unsupervised, moreover it can be applied to any classifier and any dataset. ## related work as it is known that models still learn bias when fine-tuned on downstream tasks and that the correlation is low between the intrinsic bias scores of the initial model and its extrinsic bias scores after fine-tuning @xcite , we use a method to evaluate an already trained classifier and not the pre-trained language model. some works propose such thing as general "unit-test" for nlp models @xcite or even applying a battery of fairness tests @xcite . however, extrinsic methods mainly relies on template or datasets @xcite @xcite , which have been proven to influence considerably the bias estimation and conclusion across template modification @xcite . a potential solution is to apply perturbation on the test data. perturbations can be used for attribution methods @xcite , but also for testing a model's robustness @xcite . they allow getting rid of the aforementioned template issue and data collection methodology: directly used on the target domain data, it prevents for not properly evaluating the intended notion of bias @xcite . the origin of the bias generally comes from the training data @xcite , as a lot of information can be stored in the network @xcite @xcite due to repetitions of the same sentences or concepts. this type of over-representation in the training data involve a representation bias, such as the one demonstrated by @xcite regarding the gen-der as masculine was over-represented. this was found out to be correlated with the likelihood of the model. for example, @xcite propose a perplexity-based bias measure meant to quantify the amount of bias in generative language models along several bias dimensions. for this reason, @xcite propose to use the likelihood as a proxy to estimate the bias on gender. in our case, we validate that the bias is already present in the plm, by calculating the correlation between the likelihood and different classes for country-name. this technique is even more efficient with generative models @xcite as one can apply it directly on production model. although names are not inherently linked to a specific nationality, research has revealed the presence of nationality biases within them. delving into this underexplored domain, @xcite shed light on the influence of demographic on biases associated with countries in language models. an and rudinger (2023) offer insights into the intricate relationship between demographic attributes and tokenization length, particularly focusing on biases related to first names. @xcite propose to mitigate name bias by disentangling it from its semantic context in machine reading comprehension tasks. @xcite investigate the propagation of name-nationality bias, demonstrating through intrinsic evaluation with templates how names and nationalities are intrinsically linked and how biases manifest as hallucinations. lastly, barriere and cifuentes (2024) showed that using names as proxy works to detect country-related biases depends on the sentence's language, in multilingual sentiment and stance recognition models @xcite @xcite . ## method we first rely on named entity recognition (ner) to create counterfactual examples from the targetdomain, specific of target groups, following the methodology of @xcite . the bias is assessed by quantifying the differences in the model outputs. second, we ran a series of experiences studying the correlation between the output variations and the perplexity. figure 1 shows an overview of the bias detection. ## conclusion bias at the nationality level can also occur with the most common entities of the country such as names. we show its occurrence in this paper for a set of tasks that are related with affect and subjectivity classification, using several transformer models widely used on twitter data. motivated by prior research, we studied the link between this bias and the perplexity of the plm showing (i) exacerbate correlations in unknown languages, and (ii) verify that correlation can be related to names using counterfactual sentences. we found out interesting patterns using the pearson correlations between the classes and perplexity, revealing higher correlations for english-speaking country names, meaning that the exposition bias on names impacts the predictions also in-between a country.