Selective Annotation via Data Allocation: These Data Should Be Triaged to Experts for Annotation Rather Than the Model (2024)

Abstract: To obtain high-quality annotations under limited budget, semi-automatic annotation methods are commonly used, where a portion of the data is annotated by experts and a model is then trained to complete the annotations for the remaining data. However, these methods mainly focus on selecting informative data for expert annotations to improve the model predictive ability (i.e., triage-to-human data), while the rest of the data is indiscriminately assigned to model annotation (i.e., triage-to-model data). This may lead to inefficiencies in budget allocation for annotations, as easy data that the model could accurately annotate may be unnecessarily assigned to the expert, and hard data may be misclassified by the model. As a result, the overall annotation quality may be compromised. To address this issue, we propose a selective annotation framework called SANT. It effectively takes advantage of both the triage-to-human and triage-to-model data through the proposed error-aware triage and bi-weighting mechanisms. As such, informative or hard data is assigned to the expert for annotation, while easy data is handled by the model. Experimental results show that SANT consistently outperforms other baselines, leading to higher-quality annotation through its proper allocation of data to both expert and model workers. We provide pioneering work on data annotation within budget constraints, establishing a landmark for future triage-based annotation studies.

## introduction creating a high-quality and fully annotated dataset by human experts is criticized as being expensive. it is impossible to complete the annotation if we are operating on a limited budget @xcite , especially when dealing with large-scale data corpora on the web @xcite . in this paper, we study the data annotation problem within the constraints of limited budgets, where the goal is to achieve highquality annotations despite having insufficient budgets to hire experts for annotating the entire dataset. to tackle this challenge, a typical solution is to conduct semi-automatic annotation @xcite (cf. fig. 1 (middle)). it typically allocates the budget to hire experts for annotating a portion of the data, and then utilizes the annotated data to train a prediction model for completing the annotations for the remaining data. the choice of the model can vary, ranging from sophisticated models with large parameters @xcite @xcite to lightweight models @xcite depending on the available hardware budgets. however, such methods often overlook the data allocation problem, i.e., the process of deciding which data should be annotated by human experts and which should be left for the prediction models to annotate. this may lead to inefficiencies in budget allocation for annotations, as certain data that the model could accurately annotate is unnecessarily assigned to experts. consequently, the overall annotation quality within the same budget may be compromised. in the light of this challenge, we take the first step to re-formulate the semi-automatic annotation under limited budgets as a data triage problem, which aims to optimize the annotation quality under limited budgets by determining the allocation of data to human annotator (namely triageto-human data) and data to be assigned for model prediction (namely triage-to-model data). existing semi-automatic annotation methods @xcite @xcite mainly focus on selecting unlabeled data for expert annotations to improve the model predictive ability (i.e., triage-to-human data) through random strategies or active learning (al), while the rest of data is indiscriminately assigned to model annotation (i.e., to this end, we propose a novel selective annotation framework (sant), which optimizes the utilization of limited annotation budgets by balancing the importance of triage-to-human and triage-to-model data. firstly, an al-based mechanism is deployed to assign higher weights to data that help improve the model predictive ability. secondly, an error-aware triage (eat) mechanism is proposed to assess the "hardness" weight of data, with the aim of redirecting the focus toward triageto-model data. in specific, eat dynamically estimates the probability of model error using triageto-human data from previous rounds. finally, a bi-weighting mechanism is adopted to aggregate different advantages of prioritizing triage-to-human and triage-to-model data. such mechanism adaptively adjusts the importance of data so that informative or hard-to-predict data is assigned to humans for annotation. as such, the annotation budget are efficiently utilized to secure the model predictive ability and to leverage the strengths of the expert on hard data annotation simultaneously. our experiment results show that sant consistently improves annotation quality across various annotation budgets, surpassing the best semiautomatic annotation method by an average of +0.50%, +4.86%, and +4.54% on the three annotation tasks, respectively. recently, as large language models (llms) possess strong zero-shot and few-shot capabilities for data annotation @xcite @xcite , we further adopt chatgpt as a strong automatic annotation baseline. notably, sant exhibits a better annotation quality over chatgpt, even with the aid of few-shot in-context learning and chain-of-thought (cot) prompting @xcite . these results highlight that proper allocation of data to both human and model workers results in higher-quality annotation. further analysis highlights the importance of the triage-tomodel data, which are more beneficial to obtain higher-quality annotations compared to the triageto-human ones. in conclusion, we pioneer work on data annotation with limited budgets by formulating a data triage problem. we set a landmark for future triage-based annotation methods. our contributions are as follows: we verify the importance of triage-to-model data, which is largely overlooked by existing studies. the remaining data is allocated for model annotation (triage-to-model data). to further enhance the model's predictive capabilities, existing methods are often formulated as a "model training and prediction" paradigm, where the model iteratively learns from the annotated data and then makes predictions. @xcite @xcite @xcite . considering the hardware budgets and time efficiency requirements, light-weighted models @xcite , such as label propagation @xcite and mlp are preferred. recently, uncertainty-based methods have been used to select a data subset for human annotations @xcite @xcite , improving the overall annotation quality by refining the model using more informative data. however, current methods concentrate solely on selecting triage-to-human data, while neglecting the importance of triage-to-model data. this fails to optimize data allocation effectively. in contrast, sant considers both types of data. active learning. current data allocation practices in the field of data annotation draw inspiration from the active learning @xcite , where data with high uncertainty are prioritized for human annotation. this aligns with active learning's preference for informative or representative data @xcite . to our knowledge, no existing research has explored the application of other active learning techniques for data allocation in annotation. our work, sant, extends this approach by introducing a novel perspective. it prioritizes triage-to-model data, essentially introducing the anti-current learning @xcite into the task of data allocation (i.e., we assign hard-to-predict data for human annotation). sant seamlessly integrates active learning algorithms with the anti-current learning approach through our proposed bi-weighting mechanism. our experiments demonstrate the effectiveness of this novel framework in enhancing the quality of data annotation. ## sant: selective annotation we introduce the proposed selective annotation (sant) framework. in section 3.1, we provide the problem definition of the data annotation process, while section 3.2 introduces the proposed framework sant and two novel mechanisms. in appendix f, we introduce the learning details of our framework. following previous works on semiautomatic annotation @xcite @xcite and humanmodel interaction @xcite @xcite , we assume the cost required for human annotation is constant for each data point, and estimate the annotation budgets using the number of data points. ## evaluation in this section, we conduct extensive experiments to assess the effectiveness of sant. given limited annotation budgets, we evaluate if sant is more desirable to obtain high-quality annotations, compared to existing annotation methods (cf. section 4.2). furthermore, we comprehensively analyze the advantages of sant and uncover the characteristics of triage-to-human and triage-to-model data in data annotation task (cf. section 4.3). ## conclusion we present the first work to study the roles of both triage-to-human and triage-to-model data. we formulate semi-automatic data annotation as a data triage problem via sant, achieved by optimizing two novel mechanisms (i.e., eat and bi-weighting mechanisms) in utilizing limited annotation budgets. empirical experiments show that sant sig-nificantly outperforms the existing semi-automatic annotation methods. although the effectiveness of llm-based annotator, we find that experts are still indispensable to achieving high-quality annotations, especially on challenging data annotation tasks. coordinately delegating the human and the model as a team can be a feasible solution. we also observe that relying on triage-to-human data to create a better model may not be the most effective approach for high-quality annotations. it may be more beneficial to identify triage-to-model data.