When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks (2023)

Abstract: Though majority vote among annotators is typically used for ground truth labels in machine learning, annotator disagreement in tasks such as hate speech detection may reflect systematic differences in opinion across groups, not noise. Thus, a crucial problem in hate speech detection is determining if a statement is offensive to the demographic group that it targets, when that group may be a small fraction of the annotator pool. We construct a model that predicts individual annotator ratings on potentially offensive text and combines this information with the predicted target group of the text to predict the ratings of target group members. We show gains across a range of metrics, including raising performance over the baseline by 22% at predicting individual annotators’ ratings and by 33% at predicting variance among annotators, which provides a metric for model uncertainty downstream. We find that annotators’ ratings can be predicted using their demographic information as well as opinions on online content, and that non-invasive questions on annotators’ online experiences minimize the need to collect demographic information when predicting annotators’ opinions.

## introduction for many machine learning tasks in which the ground truth is clear, having multiple people label examples, then averaging their judgments, is an effective strategy: if nearly all annotators agree on a label, the rest were likely inattentive. however, the assumption that disagreement is noise may no longer hold if the task is more subjective, in the sense that the ground truth label varies by person. for example, the same text may truly be offensive to some people and not to others (figure 1 ). recent work has questioned several assumptions of majority-vote aggregated labels: that there is a single, fixed ground truth; that the aggregated judgments of a group of annotators will accurately capture this ground truth; that disagreement among annotators is noise, not signal; that such disagreement is not systematic; and that aggregation is equally suited to very straightforward tasks and more nuanced ones @xcite @xcite @xcite . in tasks such as hate speech detection, annotator disagreement often results not from noise, but from differences among populations, e.g., demographic groups or political parties @xcite . thus, training with aggregated annotations can cause the opinions of minoritized groups with greater expertise or personal experience to be overlooked @xcite . in particular, annotators who are members of the group being targeted by a possibly offensive statement often differ in their opinions from the majority of annotators who rate the text. because these annotators may have relevant lived experi-ence or greater familiarity with the context of the statement, understanding when and why their opinions differ from the majority can help to capture cases where the majority is wrong or opinions are truly divided over whether the example is offensive. we construct a model that predicts individual annotators' ratings on the offensiveness of text, as well as the potential target group(s) 1 of the text, and combines this information to predict the ratings of target group members on the offensiveness of the example. compared to a baseline that does not model individual annotators, we raise performance by 22% at predicting individual annotators' ratings, 33% at predicting variance among annotators, and 22% at predicting target group members' aggregate ratings. we find that annotator survey responses and demographic features allow ratings to be modeled effectively, and that survey questions on annotators' online experiences allow annotators' ratings to be predicted while minimizing intrusive collection of demographic information. ## motivation and related work our work draws on recent studies that investigate the causes of disagreement among annotators. @xcite and @xcite note that disagreement among human annotations is not necessarily noise because there is a plausible range of human judgments for some tasks, rather than a single ground truth. @xcite finds that less experienced annotators are more likely to label items as hate speech than more experienced ones. when evaluating gang-related tweets, domain experts' familiarity with language use in the community resulted in more informed judgments compared to those of annotators without relevant background @xcite . annotator perception of racism varies with annotator race and political views @xcite and awareness of speaker race and dialect also affects annotation, such that annotators were less likely to find text in african-american english (aae) offensive if they were told that the text was in aae or likely written by a black author @xcite . in addition, aggregation obscures crucial differences in opinion among annotators. for example, aggregated labels align more with the opinions of 1 throughout the paper, we use "target group" or "target groups" to mean the demographic groups to which a text may cause harm, i.e., the groups that are the subject of the text's stereotyping, demeaning, or otherwise hateful content. white annotators than those of black annotators because black annotators are underrepresented in typical annotation pools @xcite . this motivates our first objective, identifying crucial examples where target group members disagree. disagreement among annotators may stem from a variety of sources, including level of experience, political background, and background experience with the topic on which they are annotating. that is, only some disagreement stems from the annotators who disagree being better-informed annotators or key stakeholders. we argue that if a statement may stereotype, demean, or otherwise harm a demographic group, members of that demographic group are typically well-suited to determining whether the statement is harmful: both from a normative standpoint, as they would be the group most affected, and because they likely have the most relevant lived experience regarding the harms that the group faces. however, majority vote aggregation often obscures the opinion of the demographic being targeted by a statement, and so explicitly modeling the opinion of the target group provides a useful source of information for downstream decisions. thus, finding cases where members of the group targeted by a harmful statement disagree with the majority opinion is crucial to determining whether the majority opinion is actually a useful label on a particular example, and provides a vital source of information for downstream decisions on whether content should be filtered. we address this by constructing a model that predicts the target group(s) of a statement, then predicts the individual ratings of annotators who are target group members (section 3.1). drawing on earlier approaches such as dawid and @xcite , newer work has used disagreement between annotators as a source of information. @xcite use probability distributions over annotator labels as an auxiliary task to reduce model overfitting. @xcite use multitask models that predict each annotator's rating of an example, then aggregate these separate predictions to produce a final majority-vote decision as well as a measure of uncertainty based on the variance among predicted individual annotators' ratings of the example. @xcite directly predict the degree of disagreement among annotators on an example using annotator demographics, and simulate potential annotators to predict variance in the degree of disagreement. @xcite predict individual annotator judgments based on ratings linked to individual annotators and annotators' demographic information. for each example, their system returns a distribution and overall verdict from a "jury" of annotators with demographic characteristics chosen by an evaluator in an interactive system. these approaches motivate our second objective, using disagreement as an independent source of information. previous work, such as @xcite , uses human-in-the-loop supervision to find alternatives to aggregated majority vote. since human judgments depend on the person's background and level of experience with the language usage and content of a text, not all evaluators may be able to correctly identify the target group to select an appropriate jury; their own background may also influence their jury selection (e.g., through confirmation bias) and thus the final system judgment. @xcite investigate the benefits of the interactive approach; we extend on this work to examine the question of how to account for annotator disagreement without human-in-the-loop supervision. our model directly identifies the target group(s) of a statement and predicts the ratings of target group members, providing an independent source of information about possible errors by the majority of annotators. our model's predictions can be used as a source of information by human evaluators in conjunction with humanin-the-loop approaches, or function independently (section 3.2). our third objective considers how to minimize collection of annotators' demographic or identifiable information. using individual annotator ids that map annotators to their ratings permits fine-grained prediction of individual annotator ratings. however, this information is often unavailable (e.g., in the data used by @xcite and its collection may raise privacy concerns for sensitive tasks. our method uses demographic information and survey responses regarding online preferences to predict annotator ratings without the need to individually link responses to annotators (section 4.1). in addition, approaches to predicting annotator opinions that rely on extensive collection of demographic information also raise privacy concerns, particularly when asking about characteristics that are the basis for discrimination in some environments (e.g., sexual orientation). thus, an open question is whether annotator ratings may be predicted while using less invasive questions and minimizing unnecessary demographic data collection. we find that survey information about online preferences provides an extremely useful alternate source of information for predicting annotator ratings, while remaining less invasive and thus easier to obtain than extensive demographic questions (section 4.4). ## approach our approach consists of two connected modules that work in parallel. given information about the annotator and the text itself, the rating prediction module predicts the rating given by each annotator who labeled a piece of text (figure 2 , in red). for this prediction task, we used a roberta-based module @xcite initially fine-tuned on the jigsaw toxicity dataset, which we then fine-tuned on the hate speech detection dataset from @xcite . the target group prediction module predicts the demographic group(s) harmed by the input text (figure 2 , in blue). for this prediction task, we fine-tuned a gpt-2 based module on the text examples and annotated target groups from the social bias frames dataset @xcite . at test time, our model predicts the target group for the input text, then predicts the rating that the members of the target group would give to that text (figure 2 , in purple). ## results the key hypotheses we aimed to test were: can we predict the ratings of individual annotators, using only their demographic information and information about their opinions on online content, without a specific mapping from each annotator to the statements they annotated? to measure this, we examine how well the individual rating prediction module predicts individual annotators' information given demographic information and/or survey responses, compared to a baseline that does not use information about the annotators (section 4.1). can we minimize unnecessary collection of in-formation about annotators, while still accurately predicting their ratings? to examine this, we train models that omit some survey and demographic information and measure their performance to investigate what information contributes most to the accuracy of model predictions (section 4.4). most importantly, given a statement that may target a specific demographic group, can we predict the ratings of target group members on whether the statement is harmful? we evaluate the full model's performance at predicting the ratings of target group members on a statement (predicting the target group + predicting the ratings of that group's members). to measure performance on this task, we examine the model error at predicting the ratings given by annotators who labeled examples where they themselves were members of the group targeted by the example (section 4.3). ## conclusion we presented a model that predicts individual annotator ratings on the offensiveness of text, which we applied to automatically flagging examples where the opinion of the predicted target group differs from that of the majority. we found that our model raises performance over the baseline by 22% at predicting individual annotators' ratings, 33% at predicting variance among annotators, and 22% at predicting target group members' aggregate ratings. annotator survey responses and demographic features allow ratings to be modeled without the need for identifying annotator ids to be tracked, and we found that a model given only annotators' race, opinion on online toxic content, and social media usage performs comparably to one trained with annotators' full demographic information. this suggests that questions about online preferences provide a future avenue for minimizing collection of demographic information when modeling annotator ratings. our findings provide a method of modeling individual annotator ratings that can support research on the downstream use of information about annotator disagreement. predicting the ratings of target group members allows hate speech detection systems to flag examples where the target group disagrees with the majority as potentially mislabeled or as examples that should be routed to human content moderation. understanding the variance among annotators provides an estimate of model confidence for content moderation or filtering, facilitating a rapid method of spotting cases where a model is uncertain. in addition, since opinions within a demographic group are not monolithic, accurately predicting the variance among target group members is especially useful: text that generates disagreement among people who are typically experienced stakeholders may require particularly attentive care. potential applications of this work that have yet to be explored include improving resource allocation for annotation by identifying the labelers whose judgments it would be most useful to have for a given piece of text, and routing content moderation decisions to experts in particular areas.