GRASP : A Disagreement Analysis Framework to Assess Group Associations in Perspectives (2024)

Abstract: Human annotation plays a core role in machine learning — annotations for supervised models, safety guardrails for generative models, and human feedback for reinforcement learning, to cite a few avenues. However, the fact that many of these human annotations are inherently subjective is often overlooked. Recent work has demonstrated that ignoring rater subjectivity (typically resulting in rater disagreement) is problematic within specific tasks and for specific subgroups. Generalizable methods to harness rater disagreement and thus understand the socio-cultural leanings of subjective tasks remain elusive. In this paper, we propose GRASP, a comprehensive disagreement analysis framework to measure group association in perspectives among different rater subgroups, and demonstrate its utility in assessing the extent of systematic disagreements in two datasets: (1) safety annotations of human-chatbot conversations, and (2) offensiveness annotations of social media posts, both annotated by diverse rater pools across different socio-demographic axes. Our framework (based on disagreement metrics) reveals specific rater groups that have significantly different perspectives than others on certain tasks, and helps identify demographic axes that are crucial to consider in specific task contexts.

## introduction automatic detection of unsafe, offensive or toxic text has long been an active area of research in natural language processing (nlp). originally aimed at online content moderation @xcite , and recently, triggered by academic and governmental calls for action @xcite , these efforts are also addressing the urgent need to equip generative technologies with safety guardrails that prevent inadvertent generation of offensive or harmful content @xcite . much of this work relies on human annotation for evaluating and training offensiveness or safety classifiers, or fine-tuning generative models. current approaches largely overlook cultural and individual factors that shape raters' perspectives on what is safe or offensive @xcite @xcite . systematic rater disagreements are instead circumvented by enforcing a single ground truth or using majority vote, which inadvertently marginalizes minority perspectives and further amplifies societal biases in data @xcite . recent work points to the need for greater diversity in rater pools @xcite and proposes ways to incorporate disagreements in the learning pipeline (e.g., @xcite ). however, incorporating rater diversity at scale is still a challenge, as there are numerous diversity axes to consider, and it is unclear which ones are relevant for particular tasks. for instance, in sentiment analysis, @xcite found that, while there were systematic disagreements between raters from different racial groups, there were no significant differences across gender groups. in contrast, @xcite found that safety annotations did not differ substantially across race/ethnicity or gender groups individually, but they did differ across intersectional race/ethnicity-gender groups. the lack of effective metrics that can capture such inter-group and intra-group cohesion at scale to determine group-level associations, is a critical issue. in this paper, we propose grasp (group associations in perspctives), a framework to measure the magnitude and strength of systematic diversity of perspectives among rater subgroups. grasp combines a suite of metrics that measure group associations in annotations with a permutation tests based significance testing approach that assesses the reliability of these associations without any independence assumptions. we apply grasp to two datasets: dices-350 @xcite )-350 chatbot conversations annotated for safety by 104 raters from a diverse pool across age, gender, and race; and d3 @xcite )-social media comments annotated for offensiveness by 4000 raters balanced across cultural regions, gender, and age. grasp reveals systematic disagreements in annotations along demographic lines, and shows that it picks up task-dependent group associations in an efficient and effective manner, furthering the objective of identifying meaningful diversity in annotator perspectives in subjective tasks. ## related work prior work on detecting harmful language, such as toxicity @xcite , offensiveness @xcite , and hate speech @xcite , has led to curating datasets and developing models for social media content moderation @xcite @xcite . recent advancements in conversational ai also increased attention to ensure safety and mitigate potential harms (e.g., @xcite @xcite @xcite @xcite . the latest generation of ai-driven language technologies @xcite @xcite is based on large language models @xcite using reinforcement learning from human feedback (rlhf) @xcite . studies show that on human alignment tasks, rater disagreement can be as high as @xcite . however, not much work has gone into developing scalable methods to meaningfully measure and tackle these high levels of rater disagreement. rater disagreement has a long history in nlp research as a challenge for crowd-sourced annotations and as a potential indication of human biases @xcite @xcite . though traditionally viewed as a mark of poor quality data, disagreement is increasingly seen as an important qualitative signal in its own right, one that is present in most tasks that requires human judgement @xcite @xcite @xcite . empirical analyses of inter-rater disagreements put forth raters' backgrounds and experiences as crucial to their annotations in such tasks, leading to systematic disagreements (e.g., @xcite @xcite @xcite @xcite . for instance, raters' demographics, including first language, age, and education, can significantly impact the performance of hate speech and abusive language detectors trained on that rater's behavior @xcite , and raters' stereotypes about different social groups and attitudes toward racism impact their annotations of hate speech and racist language targeting those groups @xcite . @xcite show that annotations of offensiveness vary across geo-cultural contexts. consequently, a large body of work has emerged to quantify, model, and measure rater disagreement (e.g., @xcite @xcite @xcite @xcite @xcite . in early work, @xcite introduce mace, an unsupervised item-response model to capture raters' relative trustworthiness to more accurately aggregate annotations into a final label. recently, weerasooriya et al. (2020) proposed predictive models for rater disagreement that take into account sampling error, a common problem in datasets with very few annotations per item. novel modeling efforts have further incorporated raters' demographics and other background attributes to improve the predictions @xcite @xcite . using multi-task modeling frameworks, @xcite add an auxiliary task to predict the soft label distribution over rater labels, davani et al. (2022) model individual raters using a shared network to preserve their systematic disagreements until prediction, and @xcite incorporates a group-specific layer to assess the benefits of sociodemographic attributes in modeling annotations. @xcite demonstrating the performance improvement when predicting raters' age and gender is coupled with language modeling objectives. our work provides a framework that anchors on intra-group and inter-group cohesion to qualify the strength of disagreements within and across groups, and provide statistical tests to assess the reliability of these observed group-level patterns. ## group associations in annotations as outline above, recent studies have established the need to account for systematic rater disagreement in subjective tasks by demonstrating sociodemographic differences in rater perceptions. however, systematic approaches to reliably assess whether and how much diversity axes impact disagreement for different tasks are still missing. to address this gap, we introduce a comprehensive analysis framework to measure statistically significant group associations within human annotations. ## discussion we propose the grasp framework that provides a means to assess the cohesion and strength of group associations along different axes of diversity that matter for a given task, identifying different groups, including intersectional groups, that are relevant for specific tasks. grasp is generic and versatile, however different task and data settings can lead to different results in group associations, depending on the metrics we use. our intent in presenting this breadth of results using different metrics is both to show the versatility of the framework w.r.t. underlying metrics, and also to demonstrate the differences that different underlying metrics yield. in practice, one should choose the metrics suitable to the specific task and data characteristics (e.g., number of raters, replication factor, and data skew). task specific insights: our analysis provides insights about specific rater groups for each task. for instance, in the conversational safety task (dices-350), white men having the highest and asian women the lowest in-group cohesion. interestingly, white women and asian men had opposite cohesion trends from their alter-genders. this suggests that men are driving the high cohesion observed in white raters, and that women and men counteract each other in the weak effects observed in asian raters overall. high coherence among white men is due to their strong tendency to prefer safe to unsafe annotations by a nearly 3 : 1 ratio. on the other hand, for the offense annotation task (d3), most regional groups show significant group associations. notably, indian cultural sphere and sinosphere shows no significant in-group cohesion (or gai), although 50+ groups within sinosphere show high in-group cohesion. age is a notable factor across board, both individually and within intersectional groups, suggesting the need for diversification of rater pools around age groups. flexibility of group granularity: our analysis is generic enough that it can be applied groups defined by any subset of demographic characteristics, enabling it to easily reveal intersectional group as-sociations. for instance, although age and gender groups revealed no association for safety, intersectional analysis revealed that gender plays a substantial role in driving race-level group tendencies. flexibility of metrics: our framework is extensible to any (comparable) underlying in-group cohesion and cross-group divergence metrics. we observe that the values across our metrics vary (see versatility across dataset characteristics: the two datasets we applied our framework to differ not only on the underlying tasks, but also on other dataset characteristics/structure. dices-350 contains fully parallel annotations (i.e., all 104 annotators annotated all 350 items), whereas d3 contains batches of annotations where sets of 35 items contain fully parallel annotations from 24+ raters. these differences did not hinder the applicability of the analysis framework. in fact, the d3 analysis provides a potential pathway where such highly parallel annotations by broadly diverse rater pools could be performed in early phases, that can then inform more streamlined data collection through curated rater pools representing selected diversity axes based on this analysis, essentially saving cost while ensuring diversity in data. exploratory analysis: our approach also illustrate the usefulness of significance testing to exploratory analysis. we see the role of significance testing in exploratory research as a compass that provides perspective in light of conflicting results that lack inherent scales for interpretation. while they impose a hefty computational burden, the permutation tests control for joint dependencies in the data between raters and conversations that simpler tests do not. however, we believe the extra computational effort is well worth the trouble, especially in informing rater recruitment decisions. ## conclusion we introduced grasp, an analytical framework to measure systematic diversity in annotations among rater subgroups, to better understand the sociocultural leanings of subjective tasks. we proposed a group association index that combines in-group and cross-group cohesion, along with statistical significance using permutation tests. applying this framework to two datasets of subjective annotations, we demonstrated how it reveals systematic disagreements across various intersectional subgroups. our work contributes to the efforts on bringing in diverse perspectives in data in an efficient and effective manner, furthering the goal of robust socio-technical evaluations of ai models. future work could explore adopting our framework to measure systematic disagreements between rater groups in other subjective tasks (e.g., kumar 2022) to further validate its utility. furthermore, our analysis framework provides actionable insights for practitioners to help prioritize demographic axes when diversifying rater pools. future work could investigate how the framework may enable dynamic data collection that can adapt to emergent group associations among raters across different types of content and tasks. while our framework identifies systematic disagreements between groups, further investigation is needed to understand the underlying reasons that cause these disagreements, for instance, individual moral values @xcite .