Probing Toxic Content in Large Pre-Trained Language Models (2021)

Abstract: Large pre-trained language models (PTLMs) have been shown to carry biases towards different social groups which leads to the reproduction of stereotypical and toxic content by major NLP systems. We propose a method based on logistic regression classifiers to probe English, French, and Arabic PTLMs and quantify the potentially harmful content that they convey with respect to a set of templates. The templates are prompted by a name of a social group followed by a cause-effect relation. We use PTLMs to predict masked tokens at the end of a sentence in order to examine how likely they enable toxicity towards specific communities. We shed the light on how such negative content can be triggered within unrelated and benign contexts based on evidence from a large-scale study, then we explain how to take advantage of our methodology to assess and mitigate the toxicity transmitted by PTLMs.

## introduction the recent gain in size of pre-trained language models (ptlms) has had a large impact on state-of-theart nlp models. although their efficiency and usefulness in different nlp tasks is incontestable, their shortcomings such as their learning and reproduction of harmful biases cannot be overlooked and ought to be addressed. present work on evaluating the sensitivity of language models towards stereotypical content involves the construction of assessment benchmarks @xcite @xcite in addition to the study of the potential risks associated with the use and deployment of ptlms @xcite . previous work on probing ptlms focuses on their syntactic and semantic limitations @xcite , lack of domainspecific knowledge @xcite , and absence of commonsense @xcite . however, except for a recent evaluation process of hurtful sentence completion @xcite , we notice a lack of large-scale probing experiments for quantifying toxic content in ptlms or systemic methodologies to measure the extent to which they generate harmful content about different social groups. in this paper, we present an extensive study which examines the generation of harmful content by ptlms. first, we create cloze statements which are prompted by explicit names of social groups followed by benign and simple actions from the atomic cause-effect knowledge graph patterns @xcite . then, we use a ptlm to predict possible reasons for these actions. we look into how bert @xcite , roberta @xcite , and gpt-2 @xcite associate unrelated and detrimental causes to basic everyday actions and examine how frequently the predicted words relate to specific social groups. moreover, we study the same phenomenon in two other languages by translating more than 700 atomic commonsense actions to arabic and french, along with names of social groups, then run the same experiments using the french ptlm camembert @xcite , and the arabic arabert @xcite . we find that, overall, the predicted content can also be irrelevant and offensive especially when the subject of the sentence is part of a marginalized community in the predominant culture of the language. in order to gauge the generated toxicity by different language models, we train simple toxicity classifiers based on logistic regression using available hate speech and offensive language datasets. we reduce the classification bias using a two-step approach to first, filter out examples with identity words which typically lead classifiers to predict a toxic label, then perform a second classification step on the remaining examples. we further con- masked lm top 10 predictions jewish, muslim, hungry, poor, gay, rich, afraid, arab, blind, sick an arab calls everyone because he is [mask] input sentence x jewish sick p([mask]|x) 0.26 0.06 0.02 0.01 muslim blind figure 1: an example of generated content using bert. intuitively, one would think that adjectives would be prioritized over than ethnic/religious affiliations in a cause-effect cloze statement which appears not to be the case. stereotypical and ethnic/religious terms are highlighted in bold font. our main contributions can be summarized in the following. ## methodology we adopt a rule-based methodology based on masked language modeling (mlm) in order to probe the toxicity of the content generated by different ptlms. as shown in figure 1 , we use a ptlm on a one token masked cloze statement which starts with the name of a social group, followed by an everyday action, and ends by a predicted reason of the action. our goal is to provide a set of tests and a process to assess toxicity in ptlms with regard to various social groups. ## experiments we use the huggingface @xcite to implement our pipeline which, given a ptlm, outputs a list of candidate words and their probabilities. the ptlms we use are bert, roberta, gpt-2, camembert, and arabert. ## related work the large and incontestable success of bert @xcite revolutionized the design and performance of nlp applications. however, we are still investigating the reasons behind this success with the experimental setup side @xcite . classification models are typically fine-tuned using ptlms to boost their performance including hate speech and offensive language classifiers @xcite . ptlms have even been used as label generation components in tasks such as entity type prediction @xcite . this work aims to assess toxic content in large ptlms in order to help with the examination of elements which ought to be taken into account when adapting the formerly stated strategies during the fine-tuning process. similarly to how long existing stereotypes are deep-rooted in word embeddings @xcite , ptlms have also been shown to recreate stereotypical content due to the nature of their training data @xcite among other reasons. nadeem et al. (2020); tay et al. (2020); forbes et al. (2020); sheng et al. (2019) have introduced datasets to evaluate the stereotypes they incorporate. on the other hand, ettinger (2020) introduced a series of psycholinguistic diagnosis tests to evaluate what ptlms are not designed for, and @xcite thoroughly surveyed their impact in the short and long terms. different probing experiments have been proposed to study the drawbacks of ptlms in areas such as the biomedical domain @xcite , syntax @xcite , semantic and syntactic sentence structures @xcite , prenomial anaphora @xcite , common-sense @xcite , gender bias @xcite , and typicality in judgement @xcite . except for hutchinson et al. (2020) who examine what words bert generate in some fill-in-the-blank experiments with regard to people with disabilities, and more recently nozza et al. (2019) who assess hurtful auto-completion by multilingual ptlms, we are not aware of other strategies designed to estimate toxic content in ptlms with regard to several social groups. in this work, we are interested in assessing how ptlms encode bias towards different communities. bias in social data is a broad concept which involves several issues and formalism @xcite @xcite . for instance, @xcite present a framework to predict the origin of different types of bias including label bias @xcite , selection bias @xcite , model overamplification @xcite , and semantic bias @xcite . other work investigate the effect of data splits @xcite and mitigation strategies @xcite . bias in toxic language classification has been addressed through mitigation methods which focus on false positives caused by identity words and lack of context @xcite @xcite . we take this issue into account in our experiments by looking at different parts of the generated statements. consequently, there has been an increasing amount of work on explainability for toxic language classifiers @xcite . for instance, aluru et al. (2020) use lime @xcite to extract explanations when detecting hateful content. akin to @xcite , a more recent work on explainability by @xcite provide a methodology for testing nlp models based on a matrix of general linguistic capabilities named checklist. similarly, we present a set of steps in order to probe for toxicity in large ptlms. ## conclusion in this paper, we present a methodology to probe toxic content in pre-trained language models using commonsense patterns. our large scale study presents evidence that ptlms tend to generate harmful biases towards minorities due to their spread within the pre-trained models. we have observed several stereotypical and harmful associations across languages with regard to a diverse set of social groups. we believe that the patterns we generated along with the predicted content can be adopted to build toxic language lexicons that have been noticed within ptlms, and use the observed associations to mitigate implicit biases in order to build more robust systems. furthermore, our methodology and predictions can help us define toxicity anchors that can be utilized to improve toxic language classification. the generated words can also be used to study socio-linguistic variations across languages by comparing stereotypical content with respect to professions, genders, religious groups, marginalized communities, and various demographics. in the future, we plan to revise our data by adding actions, more fluent and complex patterns, and longer generated statements which involve human interactions between people within the same social group, and people who belong to different ones.