ROBBIE : Robust Bias Evaluation of Large Generative Language Models (2023)

Abstract: As generative large language models (LLMs) grow more performant and prevalent, we must develop comprehensive enough tools to measure and improve their fairness. Different prompt-based datasets can be used to measure social bias across multiple text domains and demographic axes, meaning that testing LLMs on more datasets can potentially help us characterize their biases more fully, and better ensure equal and equitable treatment of marginalized demographic groups. In this work, our focus is two-fold: (1) Benchmarking: a comparison of 6 different prompt-based bias and toxicity metrics across 12 demographic axes and 5 families of generative LLMs. Out of those 6 metrics, AdvPromptSet and HolisticBiasR are novel datasets proposed in the paper. The comparison of those benchmarks gives us insights about the bias and toxicity of the compared models. Therefore, we explore the frequency of demographic terms in common LLM pre-training corpora and how this may relate to model biases. (2) Mitigation: we conduct a comprehensive study of how well 3 bias/toxicity mitigation techniques perform across our suite of measurements. ROBBIE aims to provide insights for practitioners while deploying a model, emphasizing the need to not only measure potential harms, but also understand how they arise by characterizing the data, mitigate harms once found, and balance any trade-offs. We open-source our analysis code in hopes of encouraging broader measurements of bias in future LLMs.

## introduction the recent explosion of large generative language models has brought with it an increased focus on the potential risks posed by these models. previously released base llms have displayed strong social biases as a function of gender, race, and other demographic axes @xcite @xcite , and many recent works have found that biases tend to increase as models grow in size @xcite @xcite @xcite . although some post hoc techniques relying on human feedback for mitigating bias have shown promise @xcite , the extent to which such approaches actually remove problematic biases, as opposed to simply hiding them (c.f. gonen and goldberg 2019), is not fully known. therefore, in this work, we focus on base (i.e. foundational) llms, prior to the application of finetuning techniques such as reinforcement learning from human feedback (rlhf), to better understand their core social biases, so that we can target mitigations at their source. to distinguish bias from related societal harms such as offensiveness, we define "bias" in this work as the proportion of subgroups for which the frequency of toxicity and negative regard generations falls outside an acceptable threshold. this definition is rooted in the principle of demographic parity, serving as a benchmark for equality and fairness, as previously applied in the context of fairness assessment within natural language processing @xcite @xcite @xcite @xcite -the field is still in a very preliminary stage, with coverage often restricted to measuring bias for only one demographic axis, most commonly binary gender ( dataset age body type class culture disability gender/sex nationality occupation political ideologies race/ ethnicity religion sexual orientation advpromptset in this work, we enable direct model comparison by evaluating llms from several model families on an expanded suite of bias and toxicity metrics across an expanded set of demographic axes. to further foreground often-overlooked demographic axes, we augment the community standard regard dataset @xcite with 700+ demographic identity terms from the holisticbias dataset @xcite . we also perform stratified sampling from two jigsaw toxicity datasets in order to create advpromptset, a novel dataset that allows for expanded testing of bias across intersections of identities. we are open-sourcing our model suite so that others can easily utilize our tooling. a crucial reason to expand our analysis of bias in llms to more demographic axes and metrics is to potentiate the development of bias and toxicity mitigation techniques: most recent mitigation work reports information about only a single metric, demographic axis, or model, raising serious open questions as whether they can be applied to new settings. as we expand our ability to uncover biases along more axes and for more metrics, determining which mitigations will be most effective at addressing them becomes increasingly important. we take initial steps to investigate this by comparing 3 bias/toxicity mitigation techniques across our suite of metrics. our results suggest that some mitigations are better suited to some settings than others: for example, biases exposed by the bold evaluations can generally be lessened using selfdebiasing, but the mitigation is more effective for gpt-2 than for bb3. we hope that our results will provide useful insights that can guide practitioners in selecting mitigation techniques appropriate for their setting. to summarize, we analyze different measurements and mitigations for bias and toxicity in generative llms. our main contributions are (1) a comparison of 6 different prompt-based bias and toxicity metrics across 12 demographic axes and 5 families of generative llms; (2) an extension of prompt-based metrics to more intersections of demographic groups via a new dataset, advprompt-set, and the demographic terms of holisticbias; (3) a comparison of how well 3 bias and toxicity mitigation techniques compare across our suite of measurements; (4) an exploration of the frequency of demographic terms in several llm pretraining corpora and how this may relate to model biases; and ## conclusions and future directions in our analysis, we find that each prompt dataset causes the llm models to output generations with different rates of toxicity and negative regard. notably, even when the baseline toxicity rate is minimal, certain demographic biases manifest prominently across specific prompt datasets. moreover, the prompt datasets studied in this paper, when used in combination with each other, are able to surface a more diverse set of risks posed by llms, providing a holistic view into which subgroups may be at higher risk of marginalization by llms. we hope that our measurement results show how multimetric measurement can enable us to better understand the possible risks llms can pose, and can better expose at-risk groups that may be affected. we accentuate the significance of assessing toxicity and bias concerning intersectional demographics, underscoring instances where the toxic content frequency surges for these groups in contrast to individual demographics. moreover, we explored several mitigation techniques, gauging their efficacy via both automated metrics and human evaluation. we observed that the self-debiasing technique is mostly effective in smaller llms, while prompting is more effective in larger llms. we hypothesize that the much larger capacity of larger llms may make them much more capable of adjusting their output via prompting. moreover, these techniques exhibit promising impact in mitigating biases, a finding that encourages further research into their enhancement and expansion for pre-trained llms, in addition to instruction-tuning and rlhf, which apply at later stages of model training. analyzing the demographic distribution in common training corpora, we unveiled an underrepresentation of gender and sex minority terms. this potentially enhances biases against lgbtq+ groups in llms. we aspire for llms to effortlessly generate respectful and insightful content about all demographics. using diverse datasets together helps us analyze bias in a more inclusive way. while the list of demographic and subgroup labels in each prompt dataset is not fully comprehensive, ongoing expansion will boost the inclusiveness of bias analysis. this list of relevant subgroups should evolve constantly to reflect societal and cultural changes. in light of our findings, we recognize the tendency for toxicity and negative regard to escalate with model size. given the rapid development of larger llms and the widespread use of rlhf models, future endeavors could concentrate on establishing benchmarks to assess bias and toxicity within instruction-tuned models. moving forward, we envision the field's progression towards improved and widespread utilization of multi-metric bias measurements similar to our exemplified approach, enabling a more comprehensive evaluation of models across a broad spectrum of potential biases.