Language-Agnostic Bias Detection in Language Models with Bias Probing (2023)

Abstract: Pretrained language models (PLMs) are key components in NLP, but they contain strong social biases. Quantifying these biases is challenging because current methods focusing on fill-the-mask objectives are sensitive to slight changes in input. To address this, we propose a bias probing technique called LABDet, for evaluating social bias in PLMs with a robust and language-agnostic method. For nationality as a case study, we show that LABDet “surfaces” nationality bias by training a classifier on top of a frozen PLM on non-nationality sentiment detection. We find consistent patterns of nationality bias across monolingual PLMs in six languages that align with historical and political context. We also show for English BERT that bias surfaced by LABDet correlates well with bias in the pretraining data; thus, our work is one of the few studies that directly links pretraining data to PLM behavior. Finally, we verify LABDet’s reliability and applicability to different templates and languages through an extensive set of robustness checks. We publicly share our code and dataset in https://github.com/akoksal/LABDet.

## introduction pretrained language models (plms) have gained widespread popularity due to their ability to achieve high performance on a wide range of tasks @xcite . smaller plms, in particular, have become increasingly popular for their ease of deployment and finetuning for various applications, such as text classification @xcite , extractive text summarization @xcite , and even non-autoregressive text generation @xcite . despite their success, it is established that these models exhibit strong biases, such as those related to gender, occupation, and nationality @xcite . however, quantifying intrinsic biases of plms remains a challenging task @xcite . dutch plm (bertje) turkish plm berturk sentiment classifier sentiment classifier this [nationality] person is neutral. template nationality sentiment surfacing language turkish turkish dutch +0.16 (positive) -0.17 (negative) -0.40 (negative) +0.40 (positive) greek turkish greek deze turk is neutraal. bu türk adam tarafsız biri. bu yunan adam tarafsız biri. deze griek is neutraal. figure 1: our bias probing method surfaces nationality bias by computing relative sentiment change: subtracting absolute positive sentiment of an example with a nationality from a neutral example without nationality (i.e., with the [mask] token). therefore, the turkish plm exhibits a positive interpretation for turks and a negative interpretation for greeks in the same context. conversely, the dutch plm demonstrates the opposite trend, with a positive sentiment towards greeks and a negative sentiment towards turks. recent work on social bias detection in plms has mainly focused on english. most of those approaches have limited capabilities in terms of stability and data quality @xcite . therefore, we propose a robust 'language-agnostic bias detection' method called labdet for nationality as a case study and analyze intrinsic bias in monolingual plms in arabic, dutch, english, french, german, and turkish with bias probing. labdet addresses the limitations of prior work by training a sentiment classifier on top of plms, using templates containing positive/negative adjectives, without any nationality information. this lets labdet learn sentiment analysis, but without the bias in existing sentiment datasets @xcite . the second key idea of bias probing is to surface bias by using templates and corpus examples with a nationality slot for which we compare substitutions, e.g., "turkish" vs "greek" in turkish and dutch plms as illustrated in figure 1 . when analyzing the template "this @xcite person is neutral." in turkish and dutch, we found that the turkish plm, berturk @xcite , gives a relative sentiment score of +0.16 for the turkish nationality and -0.17 for the greek nationality, while the dutch plm, bertje @xcite , gives a relative sentiment score of -0.40 for the turkish nationality and +0.40 for the greek nationality. the relative sentiment score surfaces the effect of nationality on the sentiment by subtracting absolute sentiment scores from the sentiment score of a neutral example without nationality, e.g., "this [mask] person is neutral.". this difference in relative sentiment scores between the two models aligns with historical and political context: turkish-greek conflicts, the turkish minority in the netherlands etc. these patterns are examined across various templates and corpus examples to identify consistent preferences exhibited by plms. we then show the links between biases extracted with our method and bias present in the pretraining data of examined plms. we provide a comprehensive analysis of bias in the pretraining data of bert. we examine the context positivity rate of sentences containing nationality information and also investigate the nationalities of authors in the wikipedia part of the pretraining data. furthermore, we present connections between biases present in the real world and biases extracted via labdet, particularly in relation to minority groups, geopolitics, and historical relations. finally, the consistency of these patterns across different templates enhances the robustness and validity of our findings, which have been rigorously confirmed through an extensive testing process in six languages. our paper makes the following contributions: (i) pretraining data bias: we quantify the nationality bias in bert's pretraining data and show that labdet detects bert's bias with a significant correlation, thus strongly suggesting a causal relationship between pretraining data and plm bias. (ii) linkage to real-world biases: we apply labdet to six languages and demonstrate the relationship between real-world bias about minorities, geopolitics, and historical relations and intrinsic bias of monolingual plms identified by labdet, finding support in the relevant political science literature. ## related work measuring social bias: one approach identifies associations between stereotypical attributes and target groups @xcite by analyzing their embeddings. this approach utilizes single tokens and "semantically bleached" (i.e., "sentiment-less") templates, which limits its applicability @xcite . another approach (crows-pairs (nangia et al., 2020), stereoset @xcite ) compares the mask probability in datasets of stereotypes. this is prone to instability and data quality issues @xcite and difficult to adapt across different languages. additionally, plms are sensitive to templates, which can result in large changes in the masked token prediction @xcite . bias detection in non-english languages: many studies on bias detection for non-english plms, primarily focus on developing language-specific data and methods. for instance, @xcite adapts the crows-pairs dataset for the french language, while another recent approach (kurpicz-briki, 2020) extends the bias detection method in word embeddings, weat @xcite , to include the french and german languages. chávez mulsa and spanakis (2020) also expand weat to dutch and analyze bias at the word embedding level. however, these languagespecific methods face similar challenges regarding robustness and reliability. pretraining data analysis: recent work examines the relationship between plm predictions and their pretraining data, particularly in the context of fact generation @xcite and prompting for sentiment analysis and textual entailment @xcite . other work focuses on determining the amount of pretraining data necessary for plms to perform specific tasks, such as syntax @xcite or natural language understanding (nlu) @xcite . to the best of our knowledge, our work is the first to establish a connection between pretraining data and plm behavior for intrinsic social bias. ## dataset our bias probing method, labdet, includes two steps for detecting and quantifying social bias in pretrained language models (plms). the first concerns sentiment training: we train a classifier on top of a frozen plm with generic sentiment data without nationality information. this step aims to map contextual embeddings to positive/negative sentiments without changing any underlying information about nationality. in the second step, we create minimal pairs for bias quantification via sentiment surfacing. we provide minimal pairs with different nationalities (e.g., "turk" vs "greek") and see how nationalities surfaces different sentiments. our dataset covers six languages: arabic, dutch, english, french, german, and turkish. ## bias probing we propose a robust and language-agnostic bias probing method to quantify intrinsic bias in plms. to extend and improve prior work that mainly focuses on the english language or large language models with prompting, we propose bias probing with sentiment surfacing. first, we train a classifier such as svm or mlp on top of the frozen plms to find a mapping between contextual embeddings and sentiments. for this, we utilize our sentiment training dataset created via templates in order to prevent possible leakage of nationality information to the classifier. this helps to extract positive and negative sentiment information present in the pretrained language models. in the second step, we propose the sentiment surfacing method by computing the relative sentiment change. absolute sentiment values vary across models, languages, and contexts such as templates' sentiment. in the relative approach, the placeholders in two sentences with the same context are filled with a nationality term and a neutral word, @xcite . as illustrated in figure 1 , we compare the relative sentiment change of the "this [nationality] person is neutral." template in turkish with the turkish nationality and the [mask] token. this change shows that the "turkish" nationality surfaces positive sentiment with +0.16 score while the "greek" nationality surfaces negative sentiment with -0.17 score. then, we compare these changes between across different nationalities and templates and evaluate if there is a consistent negative bias towards specific nationalities. to surface sentiment change, we utilize the three different sources of minimal pairs presented in §3.2: one coming from template pairs we curated and two coming from examples in mc4 @xcite and wikipedia corpora for six languages. additionally, we also modify and use the previously proposed eec templates @xcite for english to show robustness of our approach to different template sources. ## conclusion our bias probing method, labdet, allows for the analysis of intrinsic bias in monolingual plms and is easily adaptable to various languages. through rigorous testing and qualitative analysis, we have demonstrated the effectiveness of labdet, such as finding a strong correlation between bias in the pretraining data of english bert and our results. we also identified consistent patterns of bias towards minority groups or nationalities associated with conflict and tension across different languages. additionally, we found that large minority groups who produce content in the target language tend to have more positive sentiment, such as turks in german plms and senegalese/moroccans in french plms. finally, we show that our findings are statistically consistent across template and corpus examples, different sources, and languages.