Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations (2022)

Abstract: Majority voting and averaging are common approaches used to resolve annotator disagreements and derive single ground truth labels from multiple annotations. However, annotators may systematically disagree with one another, often reflecting their individual biases and values, especially in the case of subjective tasks such as detecting affect, aggression, and hate speech. Annotator disagreements may capture important nuances in such tasks that are often ignored while aggregating annotations to a single ground truth. In order to address this, we investigate the efficacy of multi-annotator models. In particular, our multi-task based approach treats predicting each annotators’ judgements as separate subtasks, while sharing a common learned representation of the task. We show that this approach yields same or better performance than aggregating labels in the data prior to training across seven different binary classification tasks. Our approach also provides a way to estimate uncertainty in predictions, which we demonstrate better correlate with annotation disagreements than traditional methods. Being able to model uncertainty is especially useful in deployment scenarios where knowing when not to make a prediction is important.

## introduction obtaining multiple annotator judgements on the same data instances is a common practice in nlp in order to improve the quality of final labels @xcite . in case of disagreements between annotations, they are often aggregated by majority voting, averaging @xcite , or adjudicating by an ''expert'' @xcite , to derive a singular ground truth or gold label that is later used for training supervised machine learning models. however, in many subjective tasks, there often exists no such single ''right @xcite and enforcing a single ground truth sacrifices the valuable nuances embedded in annotator's assessments of the stimuli and their disagreements @xcite . annotators' sociodemographic factors, moral values, and lived experiences often influence their interpretations, especially in subjective tasks such as identifying political stances @xcite , sentiment @xcite , and online abuse and hate speech @xcite @xcite . for instance, @xcite found that feminist and anti-racist activists systematically disagree with crowd workers on their hate speech annotations. similarly, annotators' political affiliation affects how they annotate the neutrality of political stances @xcite ). an adverse effect of majority vote in such cases is limiting representation of minority perspectives in data @xcite , potentially reinforcing societal disparities and harms. another consequence of majority voting, when applied to subjective annotations, is that the resulting labels may not be internally consistent. for example, consider a scenario where a sentence in a hate-speech dataset is annotated by a set of annotators, the majority of whom consider a phrase in it to be offensive, yet another sentence with the same phrase is annotated by a different set of annotators, the majority of whom do not find the phrase to be offensive. upon majority vote, the first sentence would be labeled as hate speech and the second sentence would not, despite containing similar content. such inconsistencies in the majority label will add noise to the learning step, while the systematicity in the individual annotations is lost. finally, majority vote and similar aggregation approaches assume that an annotator's judgements about different instances are independent from one another. however, as outlined above, annotators' decisions are often correlated, reflecting their subjective biases. prior work has investigated bayesian methods to account for such systematic differences between annotators @xcite , however, they approach this as an alternate means to derive a single ground truth label, thereby masking the degree to which annotators disagreed. our proposed solution is simple: we introduce multi-annotator architectures to preserve and model the internal consistency in each annotators' labels as well as their systematic disagreements with other annotators. we show that the multi-task framework @xcite provides an efficient way to implement a multi-annotator architecture that captures the differences between individual annotators' perspectives using the subset of data instances they labeled, while also benefiting from the shared underlying layers fine-tuned for the task using the entire dataset. preserving different annotators' perspectives until the prediction step provides better flexibility for downstream applications. in particular, we demonstrate that it provides better estimates for uncertainty in predictions. this will improve decision making in practice-for instance, to determine when not to make a prediction or when to recommend a manual review. our contributions in this paper are three-fold: (1) we develop an efficient multi-annotator strategy that matches or outperforms baseline models on seven different subjective tasks by preserving annotators' individual and collective perspectives throughout the training process. (2) we obtain an interpretable way to estimate model uncertainty that better correlates with annotator disagreements than traditional uncertainty estimates across all seven tasks. (3) we demonstrate that model uncertainty correlates with certain types of error, providing a useful signal to avoid erroneous predictions in real-world deployments. ## literature review learning to recognize and interpret subjective language has a long history in nlp @xcite . while all human judgements embed some degree of subjectivity, it is commonly agreed that certain nlp tasks tend to be more subjective in nature. examples of such relatively subjective tasks include sentiment analysis @xcite , affect modeling @xcite , emotion detection @xcite , and hate speech detection @xcite . @xcite argues that achieving a single real ''ground truth'' is not possible, nor essential, in subjective tasks, and calls for finding ways to model subjective interpretations of annotators, rather than seeking to reduce the variability in annotations. while which nlp tasks count as subjective may be contested, we focus on two tasks that are markedly subjective in nature. ## methodology we define the classification task on an annotated dataset @xmath0, a, y ), in which x is a set of text instances, a is the set of annotators and y is the annotation matrix, in which each entry y @xmath1, 1} represents the label assigned to x @xmath2. in most annotated datasets y includes many missing values, because each annotator only labels a subset of all instances. we use ȳi, to refer to the annotations present for item x i . similarly, we use ȳ,j to refer to the annotations made by annotator a j . the classification task aims to predict maj( ȳi, ) ∈ {0, 1}, which is the label assigned to x i based on the majority vote over ȳi, . we use majority vote, the most commonly used aggregation method; however, our proposed approach leaves open the choice of the aggregation method depending on deployment contexts. we consider three different multi-annotator architectures: ensemble, multi-label, and multi-task. figure 1 shows the schematic differences between these three variations. all variations use bidirectional encoder representations from transformers (bert-base; @xcite . for each instance x i , a generic representation h @xmath3, and then fine-tuned along with other components of the classifier during training. the size of the representation vector, d, is defined by the bert configuration and is set to 768 for the pre-trained bert-base. while our experiments are all performed with bert-base, our methods are not restricted to bert in their nature, and can be implemented with other pre-trained language models, for example, roberta @xcite . ## analysis in this section, we further analyze the multi-task model and its outputs, as it posted the overall best performance among the three approaches, considering the predictive performance, uncertainty modeling correlation, and time efficiency. we focus on the ghc model for this analysis. ## discussion we presented multi-annotator approaches that predict individual labels corresponding with each annotator of a subjective task, as an alternative to the more common practice of deriving (and predicting) a single ''ground-truth'' label, such as the majority vote or average of multiple annotations. we demonstrate that our method based on multi-task architecture obtains better performance for modeling each annotator (63.2 f 1 -score, micro-averaged across annotators in ghc), and even when aggregating annotators' predictions, our approach matches or outperforms the baseline across seven tasks. our study focuses on majority vote as the baseline aggregation approach to demonstrate how this commonly used approach loses meaningful information. other aggregation strategies such as mace @xcite and bayesian methods @xcite ) could be explored in future work as complementary approaches that can work with the multi-annotator framework. ## conclusion we present a multi-annotator approach that employs a different classifier head for each annotator of a dataset as an alternate method to the practice of predicting the aggregated majority vote. we demonstrate that our method can efficiently obtain better performance in modeling each annotator as well as match the majority vote prediction performance. we present experiments across different subjective classification tasks, including hate speech detection and six different emotion detection tasks. the model uncertainty estimated based on our multi-annotator model(s)' predictions obtains a higher correlation to the annotation disagreement than more traditional methods. we expect future work to investigate our multi-annotator approach as a means to detect and mitigate model biases. moreover, monitoring the performance of annotator heads and model uncertainty in an active learning setting has the potential to capture a more diverse and comprehensive set of perspectives in data.