Rater Cohesion and Quality from a Vicarious Perspective (2024)

Abstract: Human feedback is essential for building human-centered AI systems across domains where disagreement is prevalent, such as AI safety, content moderation, or sentiment analysis. Many disagreements, particularly in politically charged settings, arise because raters have opposing values or beliefs. Vicarious annotation is a method for breaking down disagreement by asking raters how they think others would annotate the data. In this paper, we explore the use of vicarious annotation with analytical methods for moderating rater disagreement. We employ rater cohesion metrics to study the potential influence of political affiliations and demographic backgrounds on raters’ perceptions of offense. Additionally, we utilize CrowdTruth’s rater quality metrics, which consider the demographics of the raters, to score the raters and their annotations. We study how the rater quality metrics influence the in-group and cross-group rater cohesion across the personal and vicarious levels.

## introduction a crucial part of many ai systems is the humans who provide feedback for learning or evaluation @xcite . as ai systems grow more powerful, aligning models with human values becomes even more critical. recent work in reinforcement learning with human feedback (rlhf) @xcite @xcite highlights the gains in model performance from aligning them to human values. the research into rlhf also notes the technical challenges associated with doing so. a major challenge to eliciting human feedback is that raters frequently disagree with each other @xcite . annotating political discourse is particularly challenging because disagreements are tied to human raters' values @xcite , making disagreement in political domains more explicit than in other annotation tasks @xcite @xcite . @xcite introduced the concept of vicarious offense, where human raters are asked to annotate data according to their own opinions, and also vicariously, e.g., on behalf of specific groups to which they do not belong. such vicarious annotations can reveal whether a group can be trusted to represent the opinions of other groups. if the group can be trusted, then we can recruit fewer raters from the other groups and still have enough annotations to represent the population from which the raters are drawn. if the group cannot be trusted, we need to find another group that can be trusted; otherwise, the only way to obtain a representative set of annotations is to recruit from all groups. this paper explores group coherence in vicarious annotation tasks investigating the following research questions. rq1 are some groups more cohesive than others when disclosing their own perceptions of offense? rq2 how much variance is there among the cohesion levels observed when different groups predict vicarious offense for other groups? rq3 what is the impact of removing raters deemed low-quality by crowdtruth on group cohesion? we address questions using the metrics introduced in the grasp framework @xcite for understanding rater cohesion and crowdtruth @xcite , two approaches for measuring the impacts of rater disagreement. rater cohesion metrics measure the extent to which rater disagreement is based on group membership. crowdtruth teases disagreement apart due to differences of opinion from poor rater quality. the major takeaways are that, of the political groups, independents are the most cohesive, both with themselves and with others. democrats are the least cohesive with others. republicans are the least internally cohesive. ## related work prior work has highlighted the prevalence of disagreement in aggregated labels for subjective nlp tasks such as toxic language detection @xcite @xcite @xcite . disagreement is often due to rater identity (race, gender, age, education, and first language) and their beliefs (political leaning) @xcite @xcite @xcite @xcite @xcite . studies have also highlighted the impact of rater bias on nlp datasets @xcite . to uncover and analyze these differences, previous work has relied on regression models and training classifiers using demographic information and comparing their predictions @xcite @xcite @xcite . recent work has advocated the use of nonaggregated (rater-level) labels @xcite @xcite to enable an extensive treatment of this variation. to this end, @xcite used bayesian multilevel models to discover intersectional effects between rater demographics and their ratings. @xcite proposed grasp (group associations in perspectives), a framework to analyze (dis)agreement among rater subgroups. crowdtruth @xcite is another framework that benefits from rater-level labels for evaluating the quality of a dataset through three dimensions: individual raters, input data items, and the overall dataset. our approach has one similarity to bayesian truth serum, (bts), where "impersonally informative" questions garner more honest answers @xcite when these answers are gathered in groups. in practice, this means using pairs of questions where one question asks for the individual's opinion and the second asks them to estimate the group distribution for this question. "bts relies on the bayesian assumption that people maintain a mental model of the world that is biased by their personal experiences, which leads to a belief that personally held opinions are disproportionately present amongst peers" @xcite . by asking raters from one political group to consider what they believe another political group thinks, we replicate the first part of bts methodology by getting distanced, and thus more honest perceptions of how the original rater perceives a topic. another difference is bts works less optimally for judging subjective social posts because it requires experts to agree on a single truth, not multiple valid truths. bts (and its variants) are used in various crowd-sourcing projects to effectively gather more honest self-reported data on non-subjective topics @xcite @xcite . ## results tables 1, 2, 3, 5, 6, and 7 show results for in-group (indicated by ∩'s) and cross-group (indicated by ⊗'s) cohesion for the personal and vicarious offense. significant results (see appendix a.2 for details) are indicated in bold at the @xmath0.05 significance level, ↓ indicates the result is less than expected under the null hypothesis and ↑ indicates the result is greater than expected. ## discussion regarding rq1, the major takeaways are that, of the political groups, independents are the most cohesive, both with themselves and with others. democrats are the least cohesive with others. republicans are the least internally cohesive. green indicates the result is significant before and after applying ct. ∆ is the mean absolute difference of metric scores before and after applying ct. because independents split their votes between democrats and republicans in most elections, we were not surprised by their relatively high level of cohesion with other groups. however, their internal cohesion was not as commonsensical. perhaps it was due to the democrats and republicans containing both extremist and more moderate members, who tend to agree on inoffensive content but whose extreme members are more readily "triggered" by moderately offensive content. and perhaps independents contain fewer extreme members. regarding intersections between gender and political leaning, considering women raters mostly amplifies the results seen by political leaning. independent women have the highest cohesion among themselves as well as with other groups. democrat women have lower cohesion among themselves and with other groups. our results suggest that women are driving disagreement among democrats and agreement among independents. regarding rq2, independents have higher cohesion with democrats and republicans while predicting vicariously for them. republicans have a higher cohesion with independents while predicting vicarious offense. democrats, again, appear to be the most isolated, this time in terms of their ability to predict what other groups find offensive. for rq3, it is not surprising that using crowdtruth to remove raters leads to higher cohesion scores; the crowdtruth quality metrics depend on within-group agreement levels. one notable exception is independent women. in terms of p-value, xrr is the most stable metric. crowdtruth filtering seems to particularly benefit irr and xrr, and hurt the negentropy metrics the most. ## conclusion our investigation into the dynamics of rater cohesion in politically charged content moderation settings, through the lens of self and vicarious an-notation, gender, and political affiliations, reveals valuable insights into the challenges of building inclusive and human-centered ai systems. our findings reveal notable disparities in cohesion levels, highlighting the influence of gender and political affiliation. for instance, independent women and democrat women show significantly different patterns of cohesion both within their groups and with other groups. we also note that independents show higher vicarious cohesion with other groups. this finding opens up a strategic avenue for more efficient rater recruitment, implying that independents can effectively approximate the viewpoints of democrats and republicans. consequently, vicarious annotation emerges as a valuable tool for optimizing rater recruitment, ensuring diverse representation under resource constraints.