HONEST : Measuring Hurtful Sentence Completion in Language Models (2021)

Abstract: Language models have revolutionized the field of NLP. However, language models capture and proliferate hurtful stereotypes, especially in text generation. Our results show that 4.3% of the time, language models complete a sentence with a hurtful word. These cases are not random, but follow language and gender-specific patterns. We propose a score to measure hurtful sentence completions in language models (HONEST). It uses a systematic template- and lexicon-based bias evaluation methodology for six languages. Our findings suggest that these models replicate and amplify deep-seated societal stereotypes about gender roles. Sentence completions refer to sexual promiscuity when the target is female in 9% of the time, and in 4% to homosexuality when the target is male. The results raise questions about the use of these models in production settings.

## method template-closed forms are an effective way of evaluating language models. @xcite use cloze-based forms to evaluate the amount of relational knowledge included in bert, and ettinger (2020) use them as a set of psycholinguistic diagnostic tools. cloze-based forms have a long history in psycholinguistics to understand human sentence processing @xcite . here, we use a similar methodology to test hurtful language in different language models. for example, our templates look as follows: "x are good at ___", 3 grammatical gender is not the same as biological sex or societal gender, but gender-inflected languages do usually assign different grammatical gender to male and female subjects. lang term template it donna la donna sogna di essere una ___ en woman the woman dreams of being a ___ fr femme la femme rêve d'être une ___ pt mulher a mulher sonha ser ___ es mujer la mujer sueña con ser una ___ ro femeia femeia viseazȃ sȃ fie ___ language models bert is natively trained on a cloze-style masked language modeling task: random tokens in the input sentence are replaced with a [mask] token, for bert to predict the correct token. after training, given a new sentence with a [mask] token (the "___" in our templates), bert returns the most probable tokens that could replace @xcite . gpt-2 does not use masked tokens. instead, we give the model the unfinished sentence up to the blank and let it complete it. this potentially returns several words. lexicon we use hurtlex @xcite , a multilingual lexicon of hurtful language, to evaluate the completions. we want to quantify how many times a sentence is completed with a hurtful word and if there is a difference between genders (represented by the identity terms). hurtlex maps each word to a class of hurtful expressions (e.g., derogatory words, ani-mals). for each bert completion, we check if the word is present in the lexicon. for gpt-2, we honest finally, we define a global score to get a birds-eye view of the models' performance, which is complementary to the category analysis. the honest score is the average of hurtful completions of any class. more formally, for a language model lm , assuming we have a set t templates filled with the identity terms, we define honest as follows: ## experimental evaluation we use the huggingface @xcite pretrained language-specific bert and gpt-2 models (see appendix b for more details). tables 3 and 4 show the percentage of hurtful words bert and gpt-2 have generated to complete the templates. 4 the tables distinguish the languages, gender target, and the different hurtlex categories. higher percentages are reported in red, lower ones in green. the average higher percentages of gpt-2 with respect to the bert ones (6.2% vs. 3.1% respectively) are due to the different evaluation settings (see section 2). because gpt-2 potentially produces more words, this difference suggests that the more text we generate, the more likely one is to find an appearance of hurtful stereotypes. the category with the highest percentages across all languages is, unsurprisingly, derogatory words. they are defined in @xcite as words with a clearly offensive and negative value, which are not negative stereotypes or slurs. these terms are associated with both men and women with relatively high percentages (10.7% on average). nevertheless, there is a slight imbalance between male and female subjects (12.3% vs. 9.2%, respectively). this is due to some limitations of the lexicon. while it includes numerous genderspecific insults, nouns and adjectives with negative connotations that can be used for both genders are predominantly reported in their male form. for example, the italian lexicon contains male insults such as buffone, cretino, zotico (en. clown, moron, lout), but not their female versions buffona, cretina, zotica. this does not affect the insights presented in this paper. on the contrary, it suggests that our results are an underestimation of the real problem, making this only the tip of the iceberg. some categories have more pronounced differences between men and women. our findings cor-roborate the study by @xcite . thirty years later, insults against women still continue to be based on sexual promiscuity, and slurs against men refer to homosexuality or sexual inadequacy @xcite . words containing references to sexual promiscuity (the categories prostitution and female genitalia) are associated predominantly with women (9% each for women, vs. 1.4% and 3% for men). this is not surprising, considering the disproportionately larger number of terms for a sexually promiscuous woman in all our languages @xcite . examples of sentence completions for this category are: the woman is a whore, the girl likes to have sex, the woman is known as a prostitute looking for money. on the other hand, the homosexuality category is more strongly associated with men (4% vs. 1.2% for women). a psychological study @xcite demonstrated that gay-related namecalling is associated with the endorsement of hypermasculine norms. similar to the previous case, this is also motivated by the fact that epithets associated with male homosexuality are more frequent than ones associated with female homosexuality in all our languages. examples of sentence completions for this category are: the husband is known as a homosexual, he is gay. while these sentences are not necessarily hurtful, sexual orientation terms (a protected category) should not be the first completion given by a language model for general purpose templates. we also investigated the possible impact of model design and training data but did not identify any systematic differences. honest. in ## related work the analysis of bias in natural language processing has gained a lot of attention in recent years @xcite , specifically on gender bias @xcite @xcite . this interest is also reflected in the organization of dedicated workshops @xcite . more generally, language models generating taboo words and insults is the result of nlp systems not incorporating social norms @xcite . the pioneering work of @xcite ) demonstrated that word embeddings (even when trained on formal corpora) exhibit gender stereotypes to a disturbing extent. on top of that, several studies have been proposed to measure and mitigate bias in word embeddings @xcite @xcite and more recently on pre-trained contextualized embeddings models @xcite @xcite @xcite @xcite . however, most studies focus on english. despite a plethora of available language-specific models @xcite , there currently exist few studies on biases in other languages. this is a severe limitation, as english findings do not automatically extend to other languages, especially if those exhibit morphological gender agreement. only @xcite ; @xcite examine the bias in word embeddings of gender-inflected languages, demonstrating the need for an adequate framework different from the ones proposed for english. to the best of our knowledge, we are the first to investigate stereotype bias in various language model completions beyond english. ## conclusion we present the first analysis of stereotyped sentence completions generated by contextual models in gender-inflected languages. we introduce the honest score to quantify the amount of hurtful completions in a language model. we release a novel benchmark data set of manually created templates, validated by native speakers in five gender-inflected languages, i.e., italian, french, portuguese, romanian, and spanish. our results show that bert and gpt-2, nowadays ubiquitous in research and industrial nlp applications, demonstrate a disturbing tendency to generate hurtful text. in particular, template sentences with a female subject are completed in 10% of the time with stereotypes about sexual promiscuity. sentences with male subjects are completed 5% of the times with stereotypes about homosexuality. this finding raises questions about the role of these widespread models in perpetuating hurtful stereotypes. in future work, we will investigate sentence completions with "benevolent sexism" categories @xcite , e.g., stereotypes like women are good at cooking or men are good at ruling. moreover, we plan to study the handling of protected category terms in natural language generation systems with data augmentation @xcite and regularization techniques @xcite .