F air P rism: Evaluating Fairness-Related Harms in Text Generation (2023)

Abstract: It is critical to measure and mitigate fairness-related harms caused by AI text generation systems, including stereotyping and demeaning harms. To that end, we introduce FairPrism, a dataset of 5,000 examples of AI-generated English text with detailed human annotations covering a diverse set of harms relating to gender and sexuality. FairPrism aims to address several limitations of existing datasets for measuring and mitigating fairness-related harms, including improved transparency, clearer specification of dataset coverage, and accounting for annotator disagreement and harms that are context-dependent. FairPrism’s annotations include the extent of stereotyping and demeaning harms, the demographic groups targeted, and appropriateness for different applications. The annotations also include specific harms that occur in interactive contexts and harms that raise normative concerns when the “speaker” is an AI system. Due to its precision and granularity, FairPrism can be used to diagnose (1) the types of fairness-related harms that AI text generation systems cause, and (2) the potential limitations of mitigation methods, both of which we illustrate through case studies. Finally, the process we followed to develop FairPrism offers a recipe for building improved datasets for measuring and mitigating harms caused by AI systems.

## introduction both text written by humans and text generated by ai systems can cause harms, yet creating datasets that capture the complex ways that human-authored and ai-generated text cause harms remains a challenge. @xcite and @xcite underscore the need to understand the complexities of harms caused by human-authored text. other recent studies indicate that ai text generation systems also produce text that stereotypes, demeans, and otherwise harms marginalized demographic groups @xcite ; @xcite @xcite . however, exactly how ai text generation systems cause harms that are less prevalent in human-authored text remains understudied. to address these issues, it is crucial to systematically measure the harms caused by ai text generation systems. enabling better measurement and mitigation methods for fairness-related harms in ai-generated text requires a mapping of the problem space and the subsequent delineation of the types of harms that ai text generation systems can cause. data collection supporting this work must therefore be informed by the needs of the whole pipeline of ai system usage, including downstream harm measurement and mitigation methods. we introduce fairprism, foot_0 a dataset of 5,000 examples of ai-generated english text with detailed human annotations covering a diverse set of harms relating to gender and sexuality. to better capture the varied contexts in which ai text generation systems are used, fairprism contains examples of text generated in both reply scenarios (e.g., autoreplies or chatbots) and continuation scenarios (e.g., writing emails or generating stories from a prompt). fairprism is designed to help diagnose (1) the extent to which ai text generation systems exhibit different types of fairness-related harms, and (2) the potential limitations of mitigation methods used to prevent the generation of harmful text. our development process was informed by the following needs: -improved transparency regarding the types of fairness-related harms that ai systems can cause; -clearer specification of the dataset's coverage of types of harms, including the groups targeted; -accounting for annotator disagreement about whether harms are present; and -accounting for context-dependent harms, including specific harms that occur in interactive contexts and harms that raise normative concerns when the "speaker" is an ai system. we include case studies on using fairprism, as well as cautionary guidance about unintended uses. finally, we provide recommendations for developing improved datasets for measuring and mitigating harms caused by ai text generation systems. ## related work most commonly used datasets for hate speech classification (e.g., @xcite consist of text written by humans. although older datasets often consist of human-authored text and accompanying binary labels resulting from aggregated annotator judgments, recent work has incorporated more detailed information. for example, annotators for the social bias frames dataset @xcite ) were asked to report the demographic groups targeted and stereotypes implied by harmful text, and to distinguish between lewd and offensive text; the dataset also included disaggregated annotator judgments on a yes/maybe/no scale. other datasets have instead used classifiers to automatically label harms in ai-generated text. real-toxicityprompts (gehman et al., 2020) consists of ai-generated text labeled automatically for toxicity and other issues using the perspective api @xcite . bold @xcite ) contains examples of ai-generated text labeled automatically for toxicity, sentiment, "regard" toward targeted demographic groups, psycholinguistic norms, and gender polarity. toxigen @xcite consists of text generated by gpt-3 in response to either toxic or benign inputs, then labeled automatically as toxic or benign using the preexisting hatexplain classifier (a 792-example subset was also labeled by annotators for characteristics such as harmfulness, the demographic groups targeted, and group framing). @xcite asked annotators to label 360 template-generated examples for "regard," which measures language polarity toward and social perceptions of demographic groups (i.e., whether an ai system causes "group a to be more highly thought of than group b"). as a result, there is a gap when it comes to developing human-labeled datasets of ai-generated text at reasonable scale, particularly with an eye to distinguishing between types of fairness-related harms and providing features that allow for harm measurement and mitigation. identifying harms that are unique to or particularly prevalent in text generated by ai systems, accounting for context-dependent harms, and distinguishing between different types of harms that are often clustered under "toxicity" or "hate speech" are also overlooked concerns. ## dataset rationale when developing fairprism, we focused on broader harm measurement and mitigation needs: improved transparency, clearer specification of dataset coverage, and accounting for annotator disagreement and harms that are context-dependent. as part of this, we considered two broad classes of applications in which ai text generation systems are used: reply scenarios, such as autoreplies or conversations with ai assistants or chatbots; and continuation scenarios, such as composing emails, writing text messages, or generating stories from a prompt. in continuation scenarios, the ai-generated text is often meant to reflect the viewpoint of the human who provided the input to the system; in reply scenarios, the ai-generated text can be seen as reflecting the viewpoint of a different speaker, such as the ai system itself. this distinction affects how harms are perceived and what kinds of text are considered coherent or effective (e.g., disagreeing with human inputs is difficult in continuation scenarios). as a result, we considered both types of scenarios and instructed annotators appropriately (see section 4). to understand what kinds of human inputs cause ai systems to generate harmful text and how ai systems handle harms in human inputs, we asked annotators separate questions about human inputs and ai-generated outputs. we began by asking whether each human input contains stereotypes or demeaning content. we then explicitly asked whether the corresponding ai-generated output disagrees, agrees with, or ignores the harms in the human input: whether an ai system exacerbates the harms in the human input, counters the harms, or attempts to avoid the topic affects the extent to which harms are present in the interaction as a whole, even if the ai-generated output seems innocuous in isolation @xcite . we also asked annotators to rate the extent of stereotyping and demeaning harms in the human input and the ai-generated output on a three-point not at all/somewhat/very scale. in the rest of this section, we discuss considerations relating to transparency, dataset coverage, annotator disagreement, and context-dependent harms; appendix d contains additional examples and the set of questions that annotators were asked. ## approach fairprism consists of human inputs and text generated by ai systems in response to those inputs. to develop fairprism, we used toxigen @xcite and the social bias frames dataset @xcite . toxigen contains both human inputs and corresponding ai-generated outputs, which we used directly. the social bias frames dataset contains human inputs only, which we used to prompt instructgpt (the text-davinci-002 model) @xcite , gpt-3 (the davinci and curie models) @xcite , and xlnet @xcite . to obtain examples of text generated in reply scenarios, we prompted the models with the human inputs as though in a conversation with a chatbot (see appendix d); to obtain examples of text generated in continuation scenarios, we prompted the models with the human inputs directly. the resulting dataset contains equal numbers of examples for reply scenarios and continuation scenarios. we used only data from toxigen and the social bias frames dataset labeled as targeting demographic groups based on gender or sexuality (including intersectional groups based on multiple factors). this enabled us to prioritize deeper coverage of a smaller set of demographic groups over shallower coverage of a larger set of demographic groups. to ensure some diversity in the severity and explicitness of harms, we used the hatexplain classifier @xcite to rate the perceived toxicity of the ai-generated outputs for each of our data sources: toxigen, social bias frames + instructgpt, social bias frames + gpt-3, and social bias frames + xlnet. we then split the examples into 5 buckets based on the difference between the predicted "toxic" and "not toxic" labels according to hatexplain, where the top bucket contained examples predicted as toxic and the others contained examples with increasing differences between the "nontoxic" and "toxic" label probabilities. ## dataset composition figure 1 shows the distribution of stereotyping and demeaning harms in the human inputs and the aigenerated outputs. on average, the human inputs were labeled as having slightly higher levels of demeaning (1.20 on a 0-2 scale) and stereotyping (1.09) harms than the ai-generated outputs (0.83 for demeaning harms and 0.77 for stereotyping harms). this likely reflects the fact that the human 5 since our focus was on harms relating to gender and sexuality, we included a basic check to ensure workers were able to annotate transphobic content. this check asked whether gender is different from sex assigned at birth. we then restricted our sample to those workers who said they were different. among the other issues that annotators were asked to identify, ignoring characteristics of demographic groups and responding with illogical text were most commonly selected, followed by attributing harmful views to named real-world sources (see figure 2 ). recurring trends in annotator comments from the "other issues" free text field illustrate additional issues (see appendix e for examples): -insulting the human who provided the input to the system, sometimes in response to harms in the human input, and sometimes using stereotypes or demeaning content in response. -not recognizing that the human input was stereotyping or demeaning, or "missing the point." -introducing stereotypes or demeaning content about demographic groups other than the demographic groups targeted by the human input. -reusing derogatory terms from the human input even if the ai-generated output did not otherwise contain stereotypes or demeaning content. ## dataset usage and broader impacts fairprism is intended to be used by researchers and practitioners who wish to diagnose (1) the types of fairness-related harms that ai text generation systems cause, and (2) the potential limitations of mitigation methods. in this section, we suggest possible analyses, along with illustrative case studies. ## recommendations our experiences developing fairprism suggest several recommendations for others who wish to develop improved datasets for measuring and mitigating harms caused by ai text generation systems. improve transparency. instead of clustering different types of harms under "toxicity" or "hate speech," providing clarity about the ways that ai-generated text can cause harms helps annotators provide high-quality labels and makes it easier to use the resulting dataset to measure or mitigate a more diverse set of harms @xcite . our distinctions between stereotyping and demeaning harms, and between simply discussing sensitive topics versus perpetuating stereotyping or demeaning harms, are intended to provide a clearer normative framing for what constitutes harmful text. ## conclusion we introduced fairprism, a dataset of 5,000 examples of ai-generated english text with detailed human annotations covering a diverse set of harms relating to gender and sexuality. by grounding our approach in broader harm measurement and mitigation needs, including transparency, clearer specification of dataset coverage, and accounting for annotator disagreement and harms that are context-dependent, fairprism aims to address several limitations of existing datasets. in turn, fairprism provides a richer lens for diagnosing (1) the types of fairness-related harms that ai text generation systems cause, and (2) the potential limitations of mitigation methods. the process we followed to develop fairprism offers a recipe for building improved datasets for measuring and mitigating harms caused by ai systems. in addition, since we limited the scope of fairprism to stereotyping and demeaning harms relating to gender and sexuality, future work could create similar datasets for other demographic groups, such as those based on race, ethnicity, religion, age, national origin, or disability status.