W ino Q ueer: A Community-in-the-Loop Benchmark for Anti- LGBTQ + Bias in Large Language Models (2023)

Abstract: We present WinoQueer: a benchmark specifically designed to measure whether large language models (LLMs) encode biases that are harmful to the LGBTQ+ community. The benchmark is community-sourced, via application of a novel method that generates a bias benchmark from a community survey. We apply our benchmark to several popular LLMs and find that off-the-shelf models generally do exhibit considerable anti-queer bias. Finally, we show that LLM bias against a marginalized community can be somewhat mitigated by finetuning on data written about or by members of that community, and that social media text written by community members is more effective than news text written about the community by non-members. Our method for community-in-the-loop benchmark development provides a blueprint for future researchers to develop community-driven, harms-grounded LLM benchmarks for other marginalized communities.

## introduction recently, there has been increased attention to fairness issues in natural language processing, especially concerning latent biases in large language models (llms). however, most of this work focuses on directly observable characteristics like race and (binary) gender. additionally, these identities are often treated as discrete, mutually exclu- * work done at usc information sciences institute. this work addresses the lack of suitable benchmarks for measuring anti-lgbtq+ bias in large language models. we present a communitysourced benchmark dataset, winoqueer, which is designed to detect the presence of stereotypes that have caused harm to specific subgroups of the lgbtq+ community. this work represents a significant improvement over winoqueer-v0, introduced in @xcite . our dataset was developed using a novel community-in-the-loop method for benchmark development. it is therefore grounded in real-world harms and informed by the expressed needs of the lgbtq+ community. we present baseline winoqueer results for a variety of popular llms, as well as demonstrating that antiqueer bias in all studied models can be partially mitigated by finetuning on a relevant corpus, as suggested by @xcite . the key contributions of this paper are: lgbtq+ bias in llms. foot_0 ## related work although the issue of gender biases in nlp has received increased attention recently @xcite , there is still a dearth of studies that scrutinize biases that negatively impact the lgbtq+ community @xcite . @xcite surveyed 176 papers regarding gender bias in nlp and found that most of these studies do not explicitly theorize gender and that almost none consider intersectionality or inclusivity (e.g., nonbinary genders) in their model of gender. they also observed that many studies conflate "social" and "linguistic" gender, thereby excluding transgender, nonbinary, and intersex people from the discourse. as @xcite observed, there is a growing body of literature that examines antiqueer biases in large language models, but most of this work fails to consider the full complexity of lgbtq+ identity and associated biases. some works (e.g. @xcite treat queerness as a single binary attribute, while others (e.g. @xcite assume that all subgroups of the lgbtq+ community are harmed by the same stereotypes. these benchmarks are unable to measure biases affecting specific lgbtq+ identity groups, such as transmisogyny, biphobia, and lesbophobia. despite such efforts, scholars have pointed out the lack of grounding in real-world harms in the majority of bias literature. for instance, blodgett et al. (2020) conducted a critical review of 146 papers that analyze biases in nlp systems and found that many of those studies lacked normative reasoning on "why" and "in what ways" the biases they describe (i.e., system behaviors) are harmful "to whom." the same authors argued that, in order to better address biases in nlp systems, research should incorporate the lived experiences of community members that are actually affected by them. there have been a few attempts to incorporate crowd-sourcing approaches to evaluate stereotypical biases in language models such as stereoset @xcite , crows-pairs @xcite , or gender lexicon dataset @xcite . @xcite used a recruited volunteers on a citizen science platform rather than using paid crowdworkers. however, these studies lack the perspective from specific communities, as both crowdworkers and volunteers were recruited from the general public. while not directly related to lgbtq+ issues, bird (2020) discussed the importance of decolonial and participatory methodology in research on nlp and marginalized communities. recently, smith et al. (2022) proposed a bias measurement dataset (holisticbias), which incorporates a participatory process by inviting experts or contributors who self-identify with particular demographic groups such as the disability community, racial groups, and the lgbtq+ community. this dataset is not specifically focused on scrutinizing gender biases but rather takes a holistic approach, covering 13 different demographic axes (i.e., ability, age, body type, characteristics, cultural, gender/sex, sexual orientation, nationality, race/ethnicity, political, religion, socioeconomic). nearly two dozen contributors were invovled in creating holisticbias, but it is uncertain how many of them actually represent each demographic axis, including the queer community. this study fills the gap in the existing literature by introducing a benchmark dataset for homophobic and transphobic bias in llms that was developed via a largescale community survey and is therefore grounded in real-world harms against actual queer and trans people. ## conclusions this paper presented winoqueer, a new bias benchmark for measuring anti-queer and anti-trans bias in large language models. winoqueer was developed via a large survey of lgbtq+ individuals, meaning it is grounded in real-world harms and based on the experiences of actual queer people. we detail our method for participatory benchmark development, and we hope that this method will be extensible to developing community-in-the-loop benchmarks for llm bias against other marginalized communities. we report baseline wq results for 20 popular off-the-shelf llms, including bert, roberta, albert, bart, gpt-2, opt, and bloom. in general, we find that off-the-shelf models demonstrate substantial evidence of anti-lgbtq+ bias, autoregressive models show more of this bias than masked language models, and there is no significant correlation between number of model parameters and wq bias score. we also demonstrate that wq bias scores can be improved by finetuning llms on either news data about queer issues or tweets written by queer people. finetuning on queertwitter is generally more effective at reducing wq bias score than finetuning on queernews, demonstrating that direct input from the affected community is a valuable resource for debiasing large models. the prevalence of high wq bias scores across model architectures and sizes makes it clear that homophobia and transphobia are serious problems in llms, and that models and datasets should be audited for anti-queer biases as part of a comprehensive fairness audit. additionally, the large variance in bias against specific subgroups of the lgbtq+ community across tested models is a strong reminder that llms must be audited for potential biases using both intrinsic, model-level metrics like wq and extrinsic, tasklevel metrics to ensure that their outputs are fair in the context where the model is deployed. our results show that llms encode many biases and stereotypes that have caused irreparable harm to queer individuals. models are liable to reproduce and even exacerbate these biases without careful human supervision at every step of the training pipeline, from pretraining data collection to downstream deployment. as queer people and allies, the authors know that homophobia and transphobia are ubiquitous in our lives, and we are keenly aware of the harms these biases cause. we hope that the winoqueer benchmark will encourage allyship and solidarity among nlp researchers, allowing the nlp community to make our models less harmful and more beneficial to queer and trans individuals.