“Fifty Shades of Bias”: Normative Ratings of Gender Bias in GPT Generated E nglish Text (2023)

Abstract: Language serves as a powerful tool for the manifestation of societal belief systems. In doing so, it also perpetuates the prevalent biases in our society. Gender bias is one of the most pervasive biases in our society and is seen in online and offline discourses. With LLMs increasingly gaining human-like fluency in text generation, gaining a nuanced understanding of the biases these systems can generate is imperative. Prior work often treats gender bias as a binary classification task. However, acknowledging that bias must be perceived at a relative scale; we investigate the generation and consequent receptivity of manual annotators to bias of varying degrees. Specifically, we create the first dataset of GPT-generated English text with normative ratings of gender bias. Ratings were obtained using Best–Worst Scaling – an efficient comparative annotation framework. Next, we systematically analyze the variation of themes of gender biases in the observed ranking and show that identity-attack is most closely related to gender bias. Finally, we show the performance of existing automated models trained on related concepts on our dataset.

## introduction harms perpetuated due to human biases are innumerable, and gender bias is one of the most prevalent biases in our society. past work shows the harm due to gender-based bias ranges from underrepresentation in media stories @xcite to mis-or no representation in computational models @xcite . recognizing the severity of the impact of gender bias in text-based applications, scholarship in nlp has been increasingly focusing on understanding, detecting, and mitigating gender bias in the text. past * equal contribution + work done while at microsoft research india work on detecting gender bias has mostly focused on a lexica-based approach or used templatized sentences to create datasets that inform models for downstream tasks @xcite @xcite . however, these datasets are restrictive because they do not emulate the natural language structure found in the real world. this problem is further aggravated by real-world data with bias being sparse and difficult to mine @xcite . further, annotation of gender bias is a challenging task. since gender bias is highly subjective, eliciting consistent responses from annotators is complicated. annotators' perception of bias is heavily influenced by factors such as lived experience, education, community, personal sensitivity, and more @xcite @xcite . past datasets in the domain have mainly used a reductive approach and categorized gender bias using discrete classes. however, @xcite shows that there is much to gain from a more fine-grained categorization of the concept for offensive language detection. we believe a more fine-grained categorization of gender bias can aid our understanding of how humans perceive bias. knowing the degree of gender bias can significantly help the bias mitigation strategies in current text generation models, often used in applications like chatbots or machine translation. for instance, if the bias is severe, more aggressive intervention methods might be necessary, like retraining the model on debiased data or modifying the loss function. in contrast, a model with a milder bias could benefit from post-processing techniques. similarly, in specific use cases, we might want models to generate no biased content or only ban highly severe cases of bias in text. since data collection and annotation is an expensive procedure, more so for tasks such as gender bias identification, which deal with data sparsity issues, there's a growing interest in the community to leverage the fluent text generation and zero-shot learning capabilities of llms like gpt-3.5 and gpt-4 @xcite . these models are also being increasingly used in everyday applications. therefore, it is imperative to understand the biases they can propagate. in our work, we prompt gpt-3.5-turbo to generate graded gender-biased text. to ground the generation, we use a list of carefully curated seeds. this serves two purposes: (1) we can navigate data sparsity issues while still grounding gpt generations to real-world data, and (2) we can understand the biases gpt-3.5-turbo can propagate via its generations. studying (2) becomes increasingly relevant given that models have been shown to represent opinionation as a by-product of being trained on poorly representative data @xcite . this paper introduces a novel dataset consisting of 1000 gpt-generated english text annotated for its degree of gender bias. the dataset includes finegrained, real-valued scores ranging from 0 (least negatively biased) to 1 (most negatively biased)normative gender bias ratings for the statements. notably, this is the first time that comparative annotations have been utilized to identify gender bias. in its simplest form, comparative annotations involve presenting two instances to annotators simultaneously and asking them to determine which instance exhibits a greater extent of the targeted characteristic. this approach mitigates several biases commonly found in standard rating scales, such as scale-region bias @xcite and enhances the consistency of the annotations @xcite . however, this method necessitates the annotation of n 2 (where @xmath0, which can be prohibitive. therefore, for the annotation of our dataset, we employ an efficient form of comparative annotation called best-worst scaling (bws) @xcite @xcite . our annotation study provides valuable insights into how humans perceive bias and a nuanced understanding of the biases gpt models can generate. we conduct in-depth qualitative and quantitative analyses of our dataset to obtain insights into how different prompting strategies and source selection can affect the generation of statements. we analyze the different themes that humans consider to be more or less biased. we assess the performance of a few neural models used for related downstream tasks on our new dataset. finally, we also investigate gpt-4's reasoning capabilities to provide an appropriate reason for a given gender bias score. foot_0 2 related work gender bias datasets existing studies on gender bias have relied on datasets that are either (a) templatized sentences or (b) sentences mined using lexical terms and rule-based patterns from web sources. the templatized sentences are structured as [noun/pronoun] is a/has a [occupation/adjectives] @xcite @xcite . these templatized sentences help elicit associations between gendered terms -name, pronouns and occupation, emotions, and other stereotypes. templatized sentences usually have artificial structures and thus have limited applicability to downstream tasks with more natural language. some prior works mine data from web sources like wikipedia and common crawl @xcite to create a dataset for coreference resolution. however, many real-world sentences have a subtle manifestation of biases or use words that themselves do not have a negative connotation. hence, rule-based sentence mining may not be able to capture the more implicit biases that humans have @xcite . a detailed analysis of the datasets used for gender bias was conducted by @xcite . two more recently created datasets not introduced in the survey are the bug dataset @xcite and the corgi-pm dataset @xcite . @xcite uses lexical, syntactic pattern matching to create bug, a dataset of sentences annotated for stereotypes. corgi-pm is a chinese corpus of 32.9k sentences that were human-annotated for gender bias. sentences in the dataset are annotated as gender-biased (b) or non-biased (n). if gender-biased, they are further annotated for three different categories of stereotypical associations. in our work, to overcome the limitations of strictly templatized sentences and the challenges of mining real-world data, we leverage the text generation capabilities of gpt to create a dataset of graded statements. annotation tasks to identify gender bias can be categorized under the descriptive paradigm, where capturing disagreements in annotation due to annotator identity like gender, race, age, etc., and their lived experiences is important @xcite . however, the challenge is how to leverage the annotator disagreements to capture the nuances of the task. studies on annotator disagreements have used different multi-annotator models @xcite . past work has shown the efficacy of the best-worst scaling framework in subjective tasks @xcite @xcite . hence, in this study, we adopt this framework. bws x nlp bws or maximum difference scaling (maxdiff) proposed by @xcite has been long used in many psychological studies. bws is an efficient comparative annotation framework. studies @xcite have shown that bws can produce highly reliable real-valued ratings. in the bws annotation setup, annotators are given a set of n items (where @xmath1, often @xmath2. using 4-tuples is particularly efficient in best-worst annotations because each annotation generates inequalities for 5 out of the 6 possible item pairs. for instance, in a 4-tuple comprising items a, b, c, and d, where a is deemed the best and d is deemed the worst, the resulting inequalities would be: @xmath3, @xmath4, @xmath5, @xmath6, and @xmath7. by analyzing the best-worst annotations for a set of 4tuples, real-valued scores representing the associations between the items and the property of interest can be calculated @xcite . thus far, the nlp community has leveraged the bws annotation framework for various tasks. more recently, bws has been used for the task of harshness modeling by @xcite , determining degrees of offensiveness by @xcite , and quantifying intimacy in language by @xcite . in the past, bws has been used for tasks such as relational similarity @xcite , word-sense disambiguation @xcite , word-sentiment intensity @xcite , phrase sentiment composition @xcite , and tweet-emotion intensity @xcite . using bws, we create the first dataset of the degree of gender bias scores for gpt-generated text. broadly, the efficacy of using preference-based evaluation methods has also been established in studies by @xcite ; @xcite . ## data generation pipeline in this section, we discuss how we prompt gpt-3.5-turbo to generate statements included in our dataset. ## annotation akin to offensive language and harshness, 'perceived gender bias' is an inherently subjective concept based on lived experiences, community, education, etc., @xcite @xcite . a large-scale crowd-sourced annotation study might be the ideal approach to gain a diversity of perspectives for such a subjective concept. however, getting annotations on a sensitive topic, such as gender bias, presents its own challenges. quality control is one major issue in crowdsourcing annotations @xcite . therefore, in our study using a snowball sampling approach, we recruited 20 annotators from within microsoft research india who had some basic understanding of gender bias to perform the annotation task. all annotators were from india and had native proficiency in english. the annotators had at least an undergraduate degree as their minimum educational qualification. out of the 20 annotators, 12 were male, and 8 were female. ## qualitative analysis of reasoning while the results in section 6.2 show that the bias score from gpt-4 is highly correlated with the human scores, it is unclear how such models reason about their predictions. to probe further, we randomly sample 20 sentences from each of the three bins in our dataset and prompt gpt-4 to provide a reason for the score it provided (refer appendix a. 4 ). an analysis of some interesting examples is discussed in this section. no. statement bws score gpt-4 score gpt-4 explanation 1 i always share my deepest secrets with my sister, because she is my best friend and i know she will never judge me. 0.02 0.1 ... because it has a slight gender bias, as it assumes that the speaker's sister is their best friend and confidant. however, the bias is not negative or harmful, as it portrays the sister in a positive light, being supportive and non-judgmental.... 2 the trip took a long time because she insisted on stopping at every single rest area and taking forever to get ready, causing unnecessary delays and frustration for everyone else. 0.61 0.25 ...because it does have a slight negative gender bias. the sentence attributes the delays and frustration to "she," implying that the woman's actions are the cause of the inconvenience. however, the bias is not too strong... 3 his room looked boring, lacking any sense of style or personality as if he had no interest in creating a space that reflected his unique tastes and preference 0.1 0.38 ...because it has a very minimal gender bias. the sentence is describing a room and the person's lack of interest in personalizing it. the only gender-specific aspect is the use of the pronoun "he," which indicates that the person being described is male. however, the sentence does not make any negative assumptions or stereotypes about men in general, so the bias is very low... 4 the grandfather was a kind man, but he always told his grandsons that boys don't cry and that they should toughen up and be strong. to these sentences, we notice that as the stereotype becomes more explicit in statement 4, both the human annotators and gpt-4 score align well, and the reasoning is well articulated. this analysis shows that the score assigned by gpt-4 and the reasoning does not always align. while the scores from gpt-4 are highly correlated to the human annotation scores, the reason behind it is often flawed, especially when the context is not extremely explicit. this indicates that effort akin to work like @xcite that contextualize the generation of offensive statements should be accelerated to improve the contextual reasoning capabilities of llms. ## conclusion we created a dataset with fine-grained human ratings of the degree of gender bias scores for gptgenerated english text. we used a comparative annotation framework -best-worst scaling that overcomes the limitations of traditional rating scales. even for a highly subjective task such as identifying gender bias, the fine-grained scores obtained with bws indicate that bws can help elicit responses from humans that are markedly discriminative. the high correlations (r ≈ 0.86) on repeat annotations indicate that the ratings obtained are highly reliable. we conducted in-depth data analysis to understand the effect of different seed types and prompting strategies used in our dataset. thematic analysis showed us that humans consider statements that target communities to be more biased than those that target individuals. we presented benchmarking experiments on our dataset and showed that identity attack most highly corre-lates with gender bias. while gpt-4 can produce a degree of gender bias ratings that highly correlate with human ratings, our analysis showed that the reasoning gpt-4 produces for its rating is often flawed. we make our dataset freely available to the research community.