On Releasing Annotator-Level Labels and Information in Datasets (2021)

Abstract: A common practice in building NLP datasets, especially using crowd-sourced annotations, involves obtaining multiple annotator judgements on the same data instances, which are then flattened to produce a single “ground truth” label or score, through majority voting, averaging, or adjudication. While these approaches may be appropriate in certain annotation tasks, such aggregations overlook the socially constructed nature of human perceptions that annotations for relatively more subjective tasks are meant to capture. In particular, systematic disagreements between annotators owing to their socio-cultural backgrounds and/or lived experiences are often obfuscated through such aggregations. In this paper, we empirically demonstrate that label aggregation may introduce representational biases of individual and group perspectives. Based on this finding, we propose a set of recommendations for increased utility and transparency of datasets for downstream use cases.

## introduction obtaining multiple annotator judgements on the same data instances is a common practice in nlp in order to improve the quality of final labels @xcite . cases of disagreement between annotations are often resolved through majority voting, averaging, or adjudication in order to derive a single "ground truth", often with the aim of training supervised machine learning models. however, in relatively subjective tasks such as sentiment analysis or offensiveness detection, there often exists no single "right" answer @xcite . enforcing such a single ground truth in such tasks will sacrifice valuable nuances about the task that are embedded in annotators' assessments of the stimuli, especially their disagreements @xcite . annotators' socio-demographic factors, moral values, and lived experiences often influence their interpretations of language, especially in subjective tasks such as identifying political stances @xcite , sentiment @xcite , and online abuse @xcite . for instance, feminist and anti-racist activists systematically disagree with crowd workers in their hate speech annotations @xcite . similarly, annotators' political affiliation is shown to correlate with how they annotate the neutrality of political stances @xcite . a potential adverse effect of majority voting in such cases is that it may sideline minority perspectives in data. in this paper, we analyze annotated data for eight different tasks across three different datasets to study the impact of majority voting as an aggregation approach. we answer two questions: our analysis demonstrates that in the annotations for many tasks, the aggregated majority vote does not uniformly reflect the perspectives of all annotators in the annotator pool. for many tasks in our analysis, a significant proportion of the annotators had very low agreement scores (0 to 0.4) with the majority vote label. while certain individual annotator's labels may have low agreement with the majority label due to valid/expected reasons (e.g., if they produced noisy labels), we further show that these agreement scores may vary significantly across different socio-demographic groups that annotators identify with. this finding has important fairness implications, as it demonstrates how the aggregation step can sometimes cause the final dataset to under-represent certain groups' perspectives. meaningfully addressing such issues in multiplyannotated datasets requires understanding and accounting for systematic disagreements between annotators. however, most annotated datasets often only release the aggregated labels, without any annotator-level information. we argue that dataset developers should consider including annotator-level labels as well as annotators' sociodemographic information (when viable to do so responsibly) when releasing datasets, especially those capturing relatively subjective tasks. inclusion of this information will enable more research on how to account for systematic disagreements between annotators in training tasks. however, the current practice in the nlp community continues to be applying different aggregation strategies to arrive at a single score or label that makes it amenable to train and evaluate supervised machine learning models. oftentimes, datasets are released with only the final scores/labels, essentially obfuscating important nuances in the task. the information released about the annotations can be at one of the following four levels of information-richness. firstly, the most common approach is one in which multiple annotations obtained for a data instance are aggregated to derive a single "ground truth" label, and these labels are the only annotations included in the released dataset (e.g., founta et al. ( 2018 )). the aggregation strategy most commonly used, especially in large datasets, is majority voting, although smaller datasets sometimes use adjudication by an 'expert' (often one of the study authors themselves) to arrive at a single label (e.g., in @xcite ) when there are sub-stantial disagreements between annotators. these aggregation approaches rely on the assumption that there always exist a single correct label, and that either the majority label or the 'expert' label is more likely to be that correct label. what it fails to account for is the fact that in many subjective tasks, e.g., detecting hate speech, the perceptions of individual annotators may be as valuable as an 'expert' perspective. secondly, some datasets (e.g., jigsaw (2018); davidson et al. (2017) ) release the distribution across labels rather than a single aggregated label. in binary classification tasks, this corresponds to the percentage of annotators who chose one of the labels. in multi-class classification, this may be the distribution across labels obtained for an instance. while this provides more information instances, annotators and individual annotations present in the datasets. for hate-speech and emotion datasets, we use the binary label in the raw annotations, whereas for the sentiment dataset, we map the 5-point ordinal labels (-2, -1, 0, +1, +2) in the raw data to a binary distinction denoting whether the text was deemed positive or negative. foot_1 while the emotion dataset contains annotations for 28 different emotions, in this work, for brevity, we focused on the annotations for only the six standard ekman emotions @xcite ) -anger, disgust, fear, joy, sadness, and surprise. in particular, we use the raw annotations for these six emotions, rather than the mapping of all 28 emotions onto these six emotions that @xcite use in some of their experiments. ## utility of annotator-level labels another argument in favor of retaining annotatorlevel labels is their utility in modeling disagreement during training and evaluation. @xcite and @xcite incorporated annotator disagreement in the loss functions used during training to improve predictive performance. @xcite and @xcite use a multi-task approach to incorporate annotator disagreements to improve machine translation and part-of-speech tagging performance, respectively. chou and lee (2019) and guan et al. (2018) developed learning architectures that model individual annotators as a way to improved performance. @xcite show the utility of detecting clusters of annotators in hate-speech detection based on how often they agree with each other. finally, davani et al. (2021) introduce a multi-annotator architecture that models each annotators' perspectives separately using a multi-task approach. they demonstrate that this architecture helps to model tators, and non-binary gender identity with one annotator). ## discussion and conclusion building models to predict or measure subjective phenomena based on human annotations should involve explicit consideration for the unique perspectives each annotator brings forth in their annotations. annotators are not interchangeable-that is, they draw from their socially-embedded experiences and knowledge when making annotation judgments. as a result, retaining their perspectives separately in the datasets will enable dataset users to account for these differences according to their needs. we demonstrated that annotation aggregation may unfairly disregard perspectives of certain annotators, and sometimes certain sociodemographic groups. based on our analysis, we propose three recommendations aimed to avoid these issues: annotator-level labels: we urge dataset developers to release the annotator-level labels, preferably in an anonymous fashion, and leave open the choice of whether and how to utilize or aggregate these labels for the dataset users. socio-demographic information: sociodemographic identity of the annotators is crucial to ascertain that the datasets (and the models trained on them) equitably represent perspectives of various social groups. we urge dataset developers to include socio-demographic information of annotators, when viable to do so responsibly. documentation about recruitment, selection, and assignment of annotators: finally, we urge dataset developers to document how the annotators were recruited, the criteria used to select them and assign data to them, and any efforts to ensure representational diversity, through transparency artefacts such datasheets @xcite or data statements @xcite .