Annotation alignment: Comparing LLM and human annotations of conversational safety (2024)

Abstract: Do LLMs align with human perceptions of safety? We study this question via *annotation alignment*, the extent to which LLMs and humans agree when annotating the safety of user-chatbot conversations. We leverage the recent DICES dataset (Aroyo et al. 2023), in which 350 conversations are each rated for safety by 112 annotators spanning 10 race-gender groups. GPT-4 achieves a Pearson correlation of r=0.59 with the average annotator rating, higher than the median annotator’s correlation with the average ( r=0.51 ). We show that larger datasets are needed to resolve whether GPT-4 exhibits disparities in how well it correlates with different demographic groups. Also, there is substantial idiosyncratic variation in correlation within groups, suggesting that race & gender do not fully capture differences in alignment. Finally, we find that GPT-4 cannot predict when one demographic group finds a conversation more unsafe than another.

## introduction as large language models have gained broad use as conversational agents, there has been increased focus on aligning models with human perceptions of safety. however, it remains unclear how well llm assessments of conversational safety agree with human assessments, and how this agreement varies with annotator background (e.g., along demographic or cultural lines). probing how an llm evaluates safety relative to humans can inform when the model may deviate from desired behavior in certain contexts or for certain users. additionally, understanding how llms annotate safety carries direct importance for multiple stages of model development, as llm annotations are increasingly used during training @xcite , evaluation (lin and chen, 2023), and deployment (e.g., to evaluate safety of another chatbot's outputs; de @xcite ). here, we compare llms as annotators of chatbot safety to a diverse group of human annotators. we use the dices dataset @xcite , in which 350 user-chatbot conversations are each annotated for safety by 112 crowdworkers spanning 10 race-gender groups. we re-annotate each conversation for safety using five leading models and study three questions: annotator demographic information includes race (one of five possibilities: asian, black, latinx, multiracial, or white), gender (either female or male; there were no nonbinary annotators), and age group (gen z, millenial, or gen x). all demographics are self-reported. we only focus on the first two demographic categories in this work. appendix a further describes the dataset, and table s1 provides example conversations and ratings. to generate corresponding llm annotations, we zero-shot prompt gpt-{3.5, 4, 4o}, gemini 1.5 pro, and llama 3.1 405b instruct (version ids in appendix b). these five models lead recent leaderboards @xcite , and openai's models in particular are the most widely used for annotation @xcite . our prompt -provided in full in figure s1 -mostly reflects the instructions given to annotators, except that it requests a single likert score from 1 (completely safe) to 5 (definitely unsafe), incorporating all five safety criteria simultaneously. this decision and other prompt optimizations are based on heldout validation results on the companion dices-990 dataset, with further details in appendix c. following prior findings that explanations may improve llm annotations @xcite , we compare two prompt variants: a base rating-only prompt, and a chain-of-thoughtstyle prompt, analyze-rate, in which the model is asked to analyze the conversation according to the safety criteria prior to providing a rating. ## results rq1: gpt-4 and llama 3.1 surpass the median annotator in terms of correlation with the average annotator rating. to evaluate alignment with the collective annotator pool, we compute pearson correlation between llm ratings and µ all , the fraction of annotators who rated a conversation unsafe -or, equivalently, the average annotator rating (from 0 to 1). we compute correlations both using the likert ratings, and, for a more apples-toapples comparison with individual annotators (who provide binary ratings), we also binarize the likert ratings @xmath0. to test whether correlation r 1 is significantly larger than r 2 , we check if r 1 > r 2 in at least 95% of 1000 bootstrap resamples. 1 reports results. as a reference, we compare to individual human annotators, for whom the median correlation with the annotator average is @xmath1.51, and the middle 25th-75th interval across the 112 annotators is [0.44, 0.58]. overall, gpt-4 with the chain-of-thought analyze-rate prompt produces the best ratings: its ratings achieve @xmath2.61 as likert and @xmath3.59 as binary, the latter of which falls at the 81st percentile relative to individual human annotators. that is, gpt-4 outperforms the median annotator at providing ratings that match the average of 112 humans. llama 3.1 rating-only performs nearly as well as gpt-4, and the binarized ratings from both gpt-4 and llama 3.1 achieve higher correlations than the other three models in at least 95% of bootstraps. although llms perform well relative to individual annotators, there remains substantial room for improvement on this task: the average of 3 random humans achieves correlation @xmath4.72 with the human consensus, surpassing gpt-4. further, when annotators are randomly split into two halves, their averages are strongly correlated: @xmath5.95 (95% ci: [0.93, 0.96]). given that the average over multiple raters exhibits high construct reliability @xcite , instances where the llm disagrees with the consensus are not attributable to human noise. analyzing examples where gpt-4 ratings disagree with the annotators. we perform a qualitative analysis of examples where annotator ratings and gpt-4 ratings disagree. there are 27 conversations which gpt-4 rates safe (@xmath6, but more than half the annotators rate as unsafe; representative examples are provided in table s2 . the most common pattern (17 of 27 conversations) is that the user asks for advice -medical, legal, financial, or interpersonal -and the chatbot provides an innocuous response (e.g., "i'd always advocate open and honest communication" to a question about relationships). these disagreements indicate a normative difference between the annotator ratings and gpt-4 ratings on the acceptability of a chatbot providing sensitive advice, with annotators being less permissive of even seemingly benign responses. on the other hand, there are 21 examples that gpt-4 rates unsafe (≥ 3) but the majority of annotators rate as safe (table s3 ). the most common pattern (15 of 21 conversations) is that the chatbot produces a biased statement which the annotators did not view as harmful (e.g., positive stereotyping), or the chatbot responds neutrally to a hateful user statement rather than denouncing it. here, the value difference is that gpt-4 tends to be more stringent than the annotators regarding hate speech and stereotypes. these examples reveal how llms and humans may annotate safety in internally consistent yet systematically different ways. rq2: the dataset is underpowered to detect demographic differences in annotator-llm alignment. next, we ask who an llm's safety ratings align with: do they correlate more or less with certain demographic groups? such disparities could suggest quality-of-service harms for groups which the llm does not align with @xcite . this question is especially pertinent in our setting, as, using dices, prabhakaran et al. (2023) report significant differences in how annotators from different race-gender groups rate safety (e.g., white male annotators are more likely to rate conversations safe). to measure group alignment, we compute cor-relations between llm ratings and µ g , the mean rating for annotators in group g, where g ranges over the 10 race-gender subgroups in dices foot_2 . we test whether each group's correlation differs significantly from its null distribution, constructed by re-computing correlations after randomly permuting demographic labels @xcite . the null distribution allows us to test how often a random, size-matched subset of annotators would have achieved a correlation as extreme as a group's true correlation. we construct null distributions over 5000 permutations. in figure 2 , we plot true group-model correlations in green and null distributions in grey, for gpt-4. none of the true correlations fall outside of their respective permuted 99% cis foot_3 . these cis are wide (e.g., @xmath7.44, 0.64] for the latinx female group), suggesting not enough statistical power to detect potentially meaningful disparities. we verify that we are similarly underpowered to detect differences in group alignment with llama and gemini. relatedly, we observe that human-llm alignment varies substantially for individuals within demographic subgroups. there is often more variation within a group than across the entire annotator pool: across all annotators, the standard deviation of an individual annotator's correlation with gpt-4 is 0.107; for the raters within a group, the sd in correlation with gpt-4 ranges from 0.075 (latinx female) to 0.132 (asian male). if group membership were predictive of the level of alignment with gpt-4, we would expect the standard deviation to be lower when conditioning on group, but this is not the case in general. overall, our findings reveal that (1) the dataset is too small to resolve whether there are significant demographic differences in alignment or not, since confidence intervals are wide; and (2) there is substantial idiosyncratic variation in alignment with gpt-4 within demographic groups. the latter point suggests that context and characteristics beyond race & gender may be necessary to explain why annotators align with gpt-4 to differing extents. rq3: gpt-4 cannot predict demographic disagreements. our previous analyses assess how an llm's default safety ratings align with the average annotator, or with certain demographic groups. here, we ask whether gpt-4 can tailor its predictions to different groups' perceptions of safety. due to the high per-example replication rate, dices is a rich source of group disagreements -conversations that one group finds more unsafe than another. to test whether an llm can capture when groups differ in their perceptions of safety, we use a paired design, where the model is asked to identify which of two groups rated a conversation more unsafe than the other (if either) foot_4 . we use a zero-shot prompt (figure s2 ) in which we detail the annotation process and then ask the model to predict if group a or b rated it more unsafe on a likert scale from 1 (group a rated more unsafe) to 5 (group b rated more unsafe), where 3 is neutral. we use gpt-4 for this task, testing the following group pairs, (g a , g b ): (white, black), (white, asian), and (white, latinx) foot_5 . we compute pearson correlation between the likert ratings and the true group differences in safety ratings µ g bµ g a across the 350 conversations. all correlations are near zero and not significant, with additional corroborating checks in appendix e. we conclude that there is no evidence that gpt-4 can identify conversations which annotators of marginalized races rate differently than white annotators. ## discussion and related work our work builds on the emerging literature of benchmarking human-llm alignment, with a focus on safety. the dominant approach to evaluating safety is to generate llm completions over a dataset of harmful or adversarial prompts, and verify whether the outputs are benign @xcite . our approach is distinct and complementary: we study annotation alignment, i.e., whether an llm annotates a chatbot's safety and harms similarly to humans. in rq1, we find that annotations from gpt-4 and llama 3.1 405b are better-aligned with the human consensus than most individual annotators in dices. these two models improve significantly over gemini 1.5 pro, gpt-4o, and gpt-3.5. though a strong proof-of-concept, llms still often disagree with the human consensus. our qualitative analysis reveals systematic patterns of annotator-gpt disagreements, suggesting revisions to the llm prompt which may further increase alignment, if desired. in rq2, we ask whether gpt-4 annotations align more with certain groups than others. prior work has found that llms align with certain demographic groups' perceptions of social norms (santy et al., 2023), politics @xcite , or offensive text @xcite . to the best of our knowledge, our work is first to look at the group-specific alignment of llm safety anno-tations. surprisingly, we find that all group-llm correlations lie within their null confidence intervals, though our results do not disprove meaningful demographic differences in alignment. yet it is notable that even though dices is powered to identify differences in groups' underlying annotations @xcite find), we show that it is not powered to detect differences in alignment with gpt-4. our experiments illustrate the importance of confidence intervals when studying group differences, substantiating prior calls for careful statistics in nlp @xcite . our second finding in rq2, that alignment often varies as much within groups as across them, implies a large amount of idiosyncratic annotator variation which is not explained by race or gender. these findings extend @xcite 's observation that demographics are insufficient to capture annotator behavior: similarly, we show that identifying why annotators do or do not align with an llm requires looking beyond demographics. alignment benchmarks should continue to prioritize demographic diversity, but other context-specific annotator information @xcite , like prior experience with online harm @xcite , should also be considered. our analysis in rq3 relates to conversations on language model pluralism: the extent to which an llm can reflect different viewpoints observed in a diverse population @xcite . a pluralistic model should capture different groups' perceptions of safety, and therefore be able to identify cases where groups disagree in their annotations. prior work has focused on supervised disagreement prediction for tasks like hate speech detection @xcite , informing decisions like when to abstain from prediction @xcite or defer to targeted groups @xcite . in our case, we are trying to evaluate whether gpt-4 already captures multiple groups' perspectives, so we focus on unsupervised disagreement prediction. we find that gpt-4 cannot predict true group disagreements. we conclude that current models may not be calibrated to how groups differ in their perceptions of harm, echoing findings that llms struggle to portray identity groups @xcite . ## conclusion our experiments use the recently released dices dataset to assess how language models annotate chatbot safety relative to a diverse set of annotators. going forward, larger human-annotated datasets with characteristics beyond race and gender will help clarify who llms are aligning to, and we encourage further work on disagreement prediction as a tool to probe whether llms are capturing a pluralistic conception of safety.