Capturing Perspectives of Crowdsourced Annotators in Subjective Learning Tasks (2024)

Abstract: Supervised classification heavily depends on datasets annotated by humans. However, in subjective tasks such as toxicity classification, these annotations often exhibit low agreement among raters. Annotations have commonly been aggregated by employing methods like majority voting to determine a single ground truth label. In subjective tasks, aggregating labels will result in biased labeling and, consequently, biased models that can overlook minority opinions. Previous studies have shed light on the pitfalls of label aggregation and have introduced a handful of practical approaches to tackle this issue. Recently proposed multi-annotator models, which predict labels individually per annotator, are vulnerable to under-determination for annotators with few samples. This problem is exacerbated in crowdsourced datasets. In this work, we propose Annotator Aware Representations for Texts (AART) for subjective classification tasks. Our approach involves learning representations of annotators, allowing for exploration of annotation behaviors. We show the improvement of our method on metrics that assess the performance on capturing individual annotators’ perspectives. Additionally, we demonstrate fairness metrics to evaluate our model’s equability of performance for marginalized annotators compared to others.

## introduction supervised machine learning tasks often train on data provided by human annotators. to improve the quality of ground truth data, the data samples are labeled by multiple annotators, whose independent decisions are then aggregated, often by taking the majority label or average value. this approach works well in some tasks, like image recognition, where "objective" ground truth exists. objective judgments should not be affected by individual differences among annotators and in practice, most people would perceive and classify them similarly, except for obvious errors or oversight @xcite . however, some annotation tasks are subjective by nature, and traditional approaches to aggregating annotations that ignore context and subjectivity may lose important information @xcite . for instance, individuals may differ about what they consider to be toxic or hate speech on social media, depending on their background and attitudes @xcite . as a result, not all annotators will agree when a particular message constitutes hate speech. disagreement among annotators in subjective tasks is expected because, for each item, the annotation outcome depends on (1) the annotator's perspective and lived experience, (2) the item being evaluated, (3) the time and circumstances surrounding the annotation task, and (4) the clarity of the designed prompts or instructions given to annotators @xcite . in addition, annotators vary in how much effort they put into the task, with prolific annotators labeling many samples and sparse annotators labeling few samples. a classifier that does not take variety in human labels into account risks overlooking minority perspectives by ignoring potentially informative differences between annotators and integrating the biases of a particular annotator pool. historically, most public datasets did not provide labels at the individual annotator level. this limited the ability of machine learning researchers to develop methods that take advantage of differences in annotator perspectives in subjective learning tasks. recently, researchers have acknowledged the importance of incorporating this information and have released raw annotations, often accompanied by demographic information about the annotators. it has now become commonplace to include multiple annotations per item in published datasets @xcite pretrained language model text i embedding annotator j embedding text i + annotator aware representation for text annotator embeddings p ij aart pretrained language model text i text i embedding p ij p ij p ij multi-task et al. @xcite @xcite . recent studies have revealed that the aggregation of annotations exhibits detrimental biases toward specific groups of annotators @xcite . instead of adhering to the conventional notion that labels should be aggregated across all stages of ai pipelines, recent research advocates for leveraging the inherent diversity in human-assigned labels by pioneering innovative approaches to accommodate multiple votes per item at different stages of ai pipelines. this can be applied across domains such as model architecture design @xcite @xcite @xcite , performance evaluation metrics @xcite @xcite , or label aggregation, surpassing simplistic methods like averaging or majority voting @xcite . despite some progress, few researchers have systematically tackled the problem of learning from multiple annotations. we address this gap by proposing a new approach for modeling annotator perspectives in subjective classification tasks via annotator aware representations for texts (aart). our approach learns annotator embeddings that represent their perspectives in downstream classification tasks (see §3 for more details). we demonstrate empirically that aart affords a more accurate and context-sensitive interpretation of subjective data at the individual annotator level. we compare our approach to a widely recognized method @xcite that captures differences in annotator judgments using a multi-task architecture consisting of a separate trainable classification-head per annotator. we show that the multi-task learning suffers from having under-trained classification heads for sparse annotators that contribute few annotations. although this work marks an advance in the study of annotations of subjective tasks, its difficulties in accounting for differences in perspectives of infrequent, sparse annotators limits its utility in practical applications. in contrast, our method captures the semantics inherent in subjective classification tasks, accounting for the diverse perspectives of annotators, including those who contribute few annotations. the aart architecture combines text embedding with annotator embeddings and provides this joint information to the classification layer. we couple this design choice with using the l2 regularizer in the loss, to prevent annotator embeddings from overfitting. additionally, we use the contrastive loss to aid in learning from the similarities and dissimilarities of label choices among annotators, a capability lacking in multi-task. furthermore, in section 4, we examine the learned annotation behaviors. by leveraging metrics from algorithmic fairness, we demonstrate that the viewpoints of sparse annotators-who either contribute few samples or select less popular labels-are accurately represented (see §4.5). this inclusivity is crucial for applications where the diversity of perspectives is essential for understanding patterns in subjective annotations. practitioners can utilize this approach to anticipate how individuals and, by extension, subgroups would react to a particular input. it is important to note that many public datasets lack information regarding the demographics of the annotators alongside their annotations. moreover, even in cases where datasets do report annotator demographics, mostly there exists an imbalance in the representation of demographics among annotators @xcite . for instance, while females constitute the majority of crowd workers in the us, they are significantly underrepresented in most other countries. additionally, online platforms tend to attract a predominantly young demographic of annotators @xcite . however, it is essential to recognize that relevant lived experience and mindset among annotators do not always perfectly align with demographic characteristics @xcite . @xcite found that amateur annotators were more prone to mislabel content as racist or sexist compared to experts with backgrounds in feminism and anti-racism activism. this finding emphasizes the need for researchers to identify annotator behaviors beyond relying solely on demographic characteristics. hence, some previous work has explored methods to identify groups of annotators with similar annotation behaviors. @xcite al. (2020) employed graph methods, and (lo and basile, 2023) conducted hierarchical clustering of annotators, while @xcite and @xcite utilized exhaustive-search algorithms to automatically group human annotators in order to maximize the inter-group differences in annotation patterns. ## methodology this section presents the details of our proposed approach and baselines used for evaluation. our proposed method aart ( §3.2) learns separate representations for each annotator in an end-to-end fashion during the training phase of the main classification task. in the inference phase, we are able to predict personalized labels for each annotator based on the annotation behaviors observed during training. a commonly used method for learning subjective perspectives is multi-task model ( §3.3) which trains a separate classification head for each of the annotators. other than comparing our method to the multi-task model we also compare it to a single-task ( §3.4) model which is fed the aggregated labels during training and is not aware of the variety of annotations for each item. the overall framework of these approaches is shown in fig. 1 . ## discussion predicting a single label remains the most popular problem in classification applications. previous studies have demonstrated that multiannotator models can perform comparably to single-task models in predicting the aggregated labels @xcite @xcite . in the single-task model, the aggregated labels are computed directly, while in multi-annotator models, the aggregated label is derived by consolidating predictions across annotators. we assess the performance of predicting the aggregated label and present the results in an additional advantage of training multiannotator models compared to single-task models is the ability to learn the entire label distribution rather than solely focusing on aggregated labels. this flexibility proves valuable in various real-world scenarios. as models are increasingly tailored for more personal, subjective use cases, a direct application of this work could be to predict personalized preferences for each annotator. furthermore, @xcite introduces an inference approach for trained multi-annotator models, advocating for the selection of a taskspecific subset of annotators akin to a "jury." for instance, for the task of toxicity detection in online posts, prioritizing votes from women and black individuals-often targeted by online harassmentcould prove advantageous. their goal with jury learning is to assist practitioners in acknowledging and incorporating annotator disagreement within the classifier pipeline. to achieve this, they encourage practitioners to form a jury that specifies the individuals or groups their classifiers should emulate. rather than completely disregarding the diversity present in human labels, our paper proposes an effective approach to predict individual perspectives while maintaining competence in predicting the aggregated label. instead of eliminating disagreement at the initial stages of machine learning, our aim is to learn from the various perspectives of annotators. by embracing the diversity of opinions through data-driven pipelines, which enhances data transparency, we are able to present it to end-users and human decision-makers. this empowerment allows them to take corrective actions and interventions by leveraging common sense and contextual information. ## conclusion this paper tackles the challenges posed by annotator disagreements in subjective tasks, where establishing a single ground truth is not straightforward. we propose an approach named annotator aware representations for texts (aart) for modeling annotators' perspectives. our methodology combines representations of individual annotators with those of text in the classification task. by learning the embeddings of annotators through cross entropy loss, contrastive loss, and regularization, we capture their individual labeling patterns. we compare our approach to a popular multi-task architecture, which is constrained by under-trained classification heads for annotators with limited annotations. through several experiments, we demonstrate that aart, despite its simplicity, effectively captures the semantics of textual data for classification tasks while accommodating different annotators' perspectives. moreover, we show that aart is fair and equitable when it comes to marginalized annotators, who may have few labeled samples or provide votes that often differ from the majority votes for samples.