Unveiling the Multi-Annotation Process: Examining the Influence of Annotation Quantity and Instance Difficulty on Model Performance (2023)

Abstract: The NLP community has long advocated for the construction of multi-annotator datasets to better capture the nuances of language interpretation, subjectivity, and ambiguity. This paper conducts a retrospective study to show how performance scores can vary when a dataset expands from a single annotation per instance to multiple annotations. We propose a novel multi-annotator simulation process to generate datasets with varying annotation budgets. We show that similar datasets with the same annotation budget can lead to varying performance gains. Our findings challenge the popular belief that models trained on multi-annotation examples always lead to better performance than models trained on single or few-annotation examples.

## introduction the process of creating datasets often involves practical constraints such as time, resources, and budget that limit the number of annotators or experts available for collecting annotations @xcite . as a result, there is a prevalence of single or few labels per instance (depending on the limited number of annotators) in the collected data. however, training models on these datasets pose challenges to their generalization abilities, primarily because the data lacks diversity. with a scarcity of different perspectives and variations in the training data @xcite , models may struggle to learn robust representations and fail to generalize effectively @xcite . to address these challenges, the nlp community has highlighted the advantages of utilizing multi-annotator datasets @xcite and also emphasized the importance of releasing multiannotator datasets and associated information (cultural and demographic, etc.) @xcite . however, this approach introduces its own set of challenges. collecting data with multiple annotators requires significant time, annotation budget, and annotator expertise to ensure the creation of high-quality datasets with diverse perspectives. moreover, with a limited annotation budget, it becomes crucial to determine the optimal number of annotators within the given constraints. this not only helps save annotation time and budget but also ensures efficient utilization of available resources. while some research @xcite has provided insights and suggestions on finding the optimal number of annotators, a definitive solution to this problem has yet to be achieved. another challenge is the restricted number of annotations available per instance, typically not exceeding 6 -10, even with a large number of recruited annotators @xcite . this limitation arises from the considerable annotation efforts required for a large volume of instances. as a result, when models are trained on such datasets, they only capture the opinions and information of a small subset of the annotator pool. additionally, certain datasets have not released annotator-specific labels or established mappings to individual annotators @xcite @xcite . however, the trend is gradually shifting, and there is a growing recognition that annotatorlevel labels should be made available @xcite @xcite . this study aims to tackle the challenge of lacking annotator-specific labels by simulating a multiannotation process. through this study, we provide insights into how the inclusion of more annotators can introduce variations in model performance and identify the factors that influence this variation. considering that previous research @xcite has highlighted the influence of individual instance difficulty on model performance, we examine how the addition of more annotations alters the difficulty level of instances and conse-quently affects model performance. in summary, our main contributions are: ## the multi-annotated dataset in practical scenarios, the annotation process begins by hiring one or more annotators who annotate each instance in the dataset. to enhance the representation of the true label distribution, we have the option to extend this process by recruiting additional annotators. we continue this iterative process until either the annotation budget is exceeded or we observe saturation in the model's performance in predicting the true label distribution. as a result, we obtain multiple annotations assigned to each instance in this multi-annotated dataset. a multi-annotator dataset d is formally characterized as a triplet @xmath0, a, y ) in this research paper. the set x represents n text instances, denoted as x 1 , x 2 , . . . , x n . the set a corresponds to m annotators, represented as a 1 , a 2 , . . . , a m . the annotation matrix y captures the annotations, with rows indexed by x and columns indexed by a. specifically, @xmath1; a] = y [x 1 , x 2 , . . . , x n ; a 1 , a 2 , . . . , a m ]. in simpler terms, the entry y [x i ; a j ] stores the label y i,j assigned to instance x i by annotator a j . furthermore, an annotator-set a k , which comprises k annotators where 1 ≤ @xmath2, is defined. consequently, the subset of d restricted to a k is denoted as this paper refers to d k as the dataset subset with k annotations per instance. figure 1 illustrates a toy multi-annotator dataset, showcasing m annotators, and n instances along with its subsets comprising 2 and k annotators. ## simulating the multi-annotation process based on our current knowledge, it is worth noting that existing multi-annotator datasets typically do not include annotator-specific labels. instead, the available information is limited to the label distribution for each instance @xcite @xcite . for instance, in cases with m annotations per instance and three possible labels, the label distribution is commonly represented by a list [p, q, r], where p, q, and r are positive integers that sum up to m . to address this constraint, we introduce a simulation process for multi-annotator scenarios that leverages the instance-level label distribution. our proposed approach (see algorithm 1), encompasses the following steps: ## related works human disagreements in annotations. traditional approaches like majority voting or averaging can overlook important nuances in subjective nlp tasks, where human disagreements are prevalent. to address this issue, multi-annotator models treat annotators' judgments as separate subtasks, capturing the distribution of human opinions, which challenges the validity of models relying on a majority label with the high agreement as ground truth @xcite . human variation in labeling, which is often considered noise @xcite , should be acknowledged to optimize and maximize machine learning metrics, as it impacts all stages of the ml pipeline @xcite . incorporating annotation instructions that consider instruction bias @xcite , which leads to the over-representation of similar examples, is crucial. this bias can limit model generalizability and performance. future data collection efforts should focus on evaluating model outputs against the distribution of collective human opinions to address this issue. all of the above works study annotator disagreements and how they affect the performance of models on downstream tasks. however, in our work, considering disagreements' effect on model performance, we try to find out how the model performance varies as we increase the number of annotations per instance, i.e., varying the annotator disagreement, overall, we try to answer, does more annotation per instance leads to better performance or is the other way around? annotation under restricted annotation budget. also, prior studies have investigated how to achieve optimal performance in natural language processing (nlp) models under restricted annotation budgets. one such study by @xcite examined the impact of repeated labeling on the quality of data and model performance when labeling is imperfect and/or costly. another study by @xcite framed domain adaptation with a constrained budget as a consumer choice problem and evaluated the utility of different combinations of pretraining and data annotation under varying budget constraints. another study by @xcite explored new annotation distribution schemes, assigning multiple labels per example for a small subset of training examples, and proposed a learning algorithm that efficiently combines signals from uneven training data. finally, a study by @xcite proposed an approach that reserves a fraction of annotations to explicitly clean up highly probable error samples to optimize the annotation process. all these studies contribute to the understanding of how to maximize the performance of nlp models under restricted annotation budgets. our study aimed to address a specific question within this context: assuming a fixed annotation budget, which dataset would yield the highest performance? previous studies have demonstrated that annotation disagreements affect model performance. however, our study aims to explore how performance varies as we change the level of disagreement. we consider ideas from @xcite who proposed a learning algorithm that can learn from training examples with different amounts of annotation (5-way, 10-way, 20-way) in a multilabel setting, but we expand the number of annotations from 1-way till 100-way and train our model in a label distribution setting rather than in a multi-label setting. to investigate the reasons for performance variation as we increase the number of annotations, we incorporate @xcite )'s ideas and @xcite 's concepts of dataset difficulty. while previous studies focused on building datasets and models and their impact on performance when the annotation budget is restricted, our work answers whether increasing the annotation budget necessarily leads to improved model performance. overall, our study aims to demonstrate that, even with less annotation budget than its upper bound, it is possible to achieve optimal performance compared to the performance at the upper bound thereby saving annotation budget and time. our findings provide insights into optimizing annotation budgets. ## conclusion in this paper, we introduced a novel approach to handle the absence of annotator-specific labels in the dataset through a multi-annotator simulation process. additionally, we investigated the impact of varying the number of annotations per instance on the difficulty of instances and its effect on model performance. our results highlighted that increasing the number of annotations does not always lead to improved performance, emphasizing the need to determine an optimal number of annotators. this has important implications for optimizing annota-tion budgets and saving time. our findings provide valuable insights for optimizing annotation strategies and open up new possibilities for future research in this direction.