Measuring Harmful Sentence Completion in Language Models for LGBTQIA + Individuals (2022)

Abstract: Current language technology is ubiquitous and directly influences individuals’ lives worldwide. Given the recent trend in AI on training and constantly releasing new and powerful large language models (LLMs), there is a need to assess their biases and potential concrete consequences. While some studies have highlighted the shortcomings of these models, there is only little on the negative impact of LLMs on LGBTQIA+ individuals. In this paper, we investigated a state-of-the-art template-based approach for measuring the harmfulness of English LLMs sentence completion when the subjects belong to the LGBTQIA+ community. Our findings show that, on average, the most likely LLM-generated completion is an identity attack 13% of the time. Our results raise serious concerns about the applicability of these models in production environments.

## introduction several works in nlp have investigated issues arising from unfair language technology @xcite @xcite , such as stereotypically biased and polarized embeddings (e.g., @xcite @xcite , and unfairly biased task-specific models, e.g., for natural language inference (e.g., @xcite , natural language generation (e.g., @xcite and generally text transformations @xcite . in this context, some dimensions of stereotyping, such as (binary) sexism (e.g., @xcite , and racism (e.g., @xcite , are relatively well explored, while others still remain under-researched. even though the united nations have repeatedly called attention to lgbtqia+ ## method we follow the two-step methodology proposed in @xcite of template creation and harmfulness computation. we compute two scores to measure harm: honest (completion-level), and harmfulness (sentence-level). ## experimental evaluation we test several llms using the huggingface library @xcite . the selected llms differs from training data domain: we choose two popular llms in their base and large form (bert @xcite and roberta @xcite ), and one trained on tweets (bertweet (nguyen et al., 2020)). ## limitations we are aware that the two methods we used have some limitations that impact the shown values. honest is strongly dependent on the hurtlex lexicon @xcite . as a lexicon, it has the advantage of being an efficient and interpretable solution that can be easily adapted to different use-cases, if needed. the limitations regard its independence from the context and the presence of some words that may be not harmful per se. for example, the hurtlex lexicon comprises as hurtful word the term "homosexual". while we disagree on this word perceived as hurtful, we believe that of bertweet due to processed data unavailability. most sentences completed by llms with this term should still be flagged (e.g., "the lgbt person is a homosexual"). the harmfulness score is regulated by the sentence classifier used for detecting hate speech. in this work, we used perplexity api. however, this tool came with its own limitations. first, we cannot intervene on the model and we can just decide the threshold to control the precision of the api. second, it has been demonstrated that it has a high false alarm rate in scoring high toxicity to benign phrases @xcite and that it is very susceptible to profanity presence foot_3 . nevertheless, @xcite demonstrated that the detection of identity attacks by the perplexity api is robust to several functional tests, showing the highest performance across all the tested models. in our analysis, we observe that perplexity api is able to recognize subtle forms of harm correctly, but at the same time, it seems sensible to the presence of some identity terms. in order to have a glimpse of the problem, we manually evaluated the classification of the top-1 completion by bertlarge with "demisexual" as subject. out of the 13 templates classified as harmful, we found that 4 were positive or neutral sentences. we believe that, despite these limitations, the findings of our work still hold. moreover, the two experimented methodologies provide two different and complementary views of the problem. ## related work while there is a plethora of work relating to binary gender bias in nlp (e.g., @xcite @xcite the research landscape analyzing harms against individuals of the lgbtqia+ community is extremely scarce. @xcite were the first to study gender inclusion. they focused on biases in co-reference resolution and provided a test set, which includes pronouns referring to non-binary individuals. later, @xcite presented redditbias, a data set created from reddit comments based on a first bias specification reflecting individuals of the lgbtqia+ community. recent work has proposed the crowdsourcing collection of stereotypes also related to gender identity and sexual orientation @xcite . however, we found their set of identities limited to gender-conforming male and female indicators and a few others (gay, heterosexual, homosexual, straight, trans, transgender). most recently, @xcite surveyed harms arising from gender-exclusivity in language technology. they also conducted preliminary studies showing the (mis)representation of terms relating to nonbinary gender in data sets and embeddings, e.g., glove @xcite and bert @xcite . however, they neither focused on sexual or romantic orientation nor quantified harmfulness. research in hate speech detection considering gender and sexuality have mostly focus on sexism @xcite @xcite @xcite @xcite @xcite . few recent works covered hate speech on the basis of sexual orientation @xcite @xcite @xcite . closest to us, @xcite and ousidhoum et al. (2021) present easily extendable template-based approaches for measuring harmful llm completions, which we extend in our work for providing a more extensive perspective and fueling more research on lgbtqia+-inclusive nlp. ## conclusion this paper introduces a systematic evaluation of harmful sentence completion by llms when the subjects belong to the lgbtqia+ community. we exploit two state-of-the-art approaches to evaluate the harmfulness at completion and sentence levels. the analysis shows alarming results: the mostlikely word that llms uses for filling lgbtqia+focused templates is harmful 7% of the time, while the resulting sentence is harmful 13% of the time. we believe that these results can inform future research on fair and inclusive nlp and that the created identity term list will serve as a useful starting point for future studies. in the future, we will test the misgendering pitfalls of llms exploiting the generated completions. a identity terms queer they, xe, ze, e, fae, nonbinary, enby, gender @xcite nonqueer cisgender, gender conforming, nonqueer, straight, heterosexual, heteroromantic