GPT is Not an Annotator: The Necessity of Human Annotation in Fairness Benchmark Construction (2024)

Abstract: Social biases in LLMs are usually measured via bias benchmark datasets. Current benchmarks have limitations in scope, grounding, quality, and human effort required. Previous work has shown success with a community-sourced, rather than crowd-sourced, approach to benchmark development. However, this work still required considerable effort from annotators with relevant lived experience. This paper explores whether an LLM (specifically, GPT-3.5-Turbo) can assist with the task of developing a bias benchmark dataset from responses to an open-ended community survey. We also extend the previous work to a new community and set of biases: the Jewish community and antisemitism. Our analysis shows that GPT-3.5-Turbo has poor performance on this annotation task and produces unacceptable quality issues in its output. Thus, we conclude that GPT-3.5-Turbo is not an appropriate substitute for human annotation in sensitive tasks related to social biases, and that its use actually negates many of the benefits of community-sourcing bias benchmarks.

## introduction though seemingly ubiquitous, large language models (llms) still treat users unequally and exhibit harmful social biases @xcite . 1 quantitative llm bias measurement is a necessary first step to understanding and mitigating bias-related harms of ai systems. measurement is essential because it allows model creators to understand potential fairness issues with 1 clear and explicit definitions of the terms bias and harm are essential for productive discussion of ai fairness @xcite . for purposes of this paper, we define bias as "substantially differing treatment of a marginalized group relative to a dominant group that replicates existing social stereotypes about the marginalized group" and harm as "physical, psychological, financial, or professional events that affect a person in perceived negative way." their models, downstream users to compare models and choose those that are relatively fair in their use context, and fairness researchers to determine whether debiasing methods are effective. the current standard for bias measurement in llms is paired sentence bias benchmarks, which consist of pairs of similar sentences and generally rely on comparing a model's probability of predicting the stereotypical sentence to the probability of predicting a contrasting sentence. there are significant quality and grounding issues with most current benchmarks, especially those developed via crowd-sourcing. current methods for communitysourced benchmark development, which mitigate some of the problems with crowd-sourcing, require significant human effort for annotation of survey responses. this work is time-consuming; in an unfunded, community-led benchmark development effort, this is either cost-prohibitive or requires asking annotators to work for free. this annotation also places a significant psychological burden on annotators. in order to maintain the usefulness and participatory nature of community-sourced bias benchmarks while reducing the financial and psychological costs, we tested model-assisted harm extraction. the main contributions of our work are as follows: foot_0 the current standard for bias measurement in llms is paired sentence bias benchmarks, such as stereoset @xcite and crows-pairs @xcite . these benchmark datasets consist of pairs of stereotypical (e.g. "women are bad at math.") and counter-stereotypical (e.g. "men are bad at math.") sentences. metrics vary slightly between benchmarks, but generally rely on comparing a model's probability of predicting the stereotypical sentence to its probability of predicting the other sentence. while paired sentence benchmarks are important tools for understanding and mitigating bias, the current benchmarks have several important weaknesses. first, most benchmarks attempt to be general-purpose, covering a variety of bias axes (e.g. race, gender, nationality, religion, etc.) with the intention that downstream users need only test their models on one benchmark. however, even the most extensive benchmarks leave out many biases that exist and are harmful in the real world. even when they cover several bias axes, benchmarks are often oversimplified and lacking nuance along each axis. for example, @xcite include homophobia in their benchmark, but treat lgbtq+ identity as a binary attribute rather than a complex set of related communities. similarly, @xcite include religion as one of their bias axes, but only consider christianity, islam, and hinduism, ignoring antisemitism and other religious biases. llm benchmarking work on antisemitism is limited; however, there is closely related work on antisemitism in the context of hate speech detection @xcite . second, many existing benchmarks have serious quality control issues, including ungrammatical or nonsensical sentences, mismatching of stereotypes to target groups, and vague or nonspecific stereotypes @xcite . finally, most existing benchmarks are insufficiently grounded, i.e. the stereotypes and biases described in the benchmark may not be well-aligned with the affected communities' opinions on what stereotypes are harmful and what constitutes unacceptable or undesirable llm behavior @xcite . grounding issues often stem from the method of collecting stereotypes to include in bias benchmarks. many benchmark creators use crowdsourced stereotypes, usually from crowdworkers on amazon mechanical turk, who have varying degrees of knowledge and personal experience with social biases. in contrast, recent work @xcite has had success with communitysourced bias benchmarks in which stereotypes were derived from a large-scale online survey of members of the affected community. this yielded a large, specific, well-grounded, and high-quality bias benchmark dataset for harms affecting a specific community. community-sourcing, while a significant improvement over crowd-sourcing in many respects, has its own problems. current methods require survey responses to be annotated by hand, ideally by researchers who are themselves members of the communities surveyed. this annotation relies heavily on the lived experience and subjective opinions of annotators to extract attested harm predicates from survey responses, which are then inserted into template sentences to create the stereotypical and counter-stereotypical sentences in the benchmark. human annotation by researchers with relevant lived experience is time-consuming, and therefore expensive and often cost-prohibitive for grassroots, community-based efforts. additionally, the annotation is psychologically taxing on annotators. annotators spend hours reading detailed descriptions of violence and hatred toward their own communities. this is often triggering for annotators who have first-hand experiences with such harm, and it is exhausting and depressing even for those who have somewhat less painful experience. the situation is somewhat analogous to content moderation, in which repeated exposure to disturbing content causes secondary traumatization @xcite for moderators, often leading to post-traumatic stress disorder (ptsd) @xcite . this annotation is similar to volunteer moderation of identityspecific online communities, such as those studied by @xcite , who found that moderation constitutes significant emotional labor and often leads to burnout. ## conclusions this paper introduced the winosemitism dataset, a community-sourced benchmark dataset for measuring bias against jewish people in large language models (llms). we also investigated the feasibility of using a large language model to assist humans with the difficult, emotionally taxing work of developing llm fairness benchmarks. in particular, we used gpt-3.5-turbo to extract attested harm predicates from responses to surveys of communities affected by specific biases. even on this extremely limited subtask with additional information that would be unavailable to a model in a realistic scenario, we find that gpt-3.5-turbo has unacceptably poor performance on attested harm extraction. we present quantitative and qualitative evidence of serious quality issues with model-extracted predicates, including high rates of both misrepresenting survey responses and hallucinating harms not present in input text. these quality issues replicate many of the problems of crowd-sourced (i.e. not community-sourced) fairness benchmarks, including lack of specificity, nonsensical stereotypes, and ungrammatical sentences, all of which would require considerable human effort to correct. we also show that model-created benchmarks are poorly correlated with human-created benchmarks from the same underlying community surveys, indicating that gpt-created benchmarks are not measuring the same model behavior as humancreated benchmarks and are poorly grounded in the actual experiences of marginalized communities. finally, the impacts of poor extraction quality disparately affect already-marginalized subgroups of affected communities, meaning that model-created benchmarks are likely to replicate and exacerbate existing intracommunity inequality. thus, we conclude that using llms to process human survey responses into fairness benchmark datasets yields unnacceptably low-quality benchmarks and largely negates the positive impacts of conducting a community survey in the first place. annotations from expert humans with lived experience as members of the relevant community are absolutely essential to the construction of high-quality llm fairness benchmarks. given these findings, we strongly caution against the use of llms in the creation of fairness benchmarks intended to evaluate llms and stress the important of human annotators, especially in sensitive and highly context-dependent tasks.