Clean or Annotate: How to Spend a Limited Data Collection Budget (2022)

Abstract: Crowdsourcing platforms are often used to collect datasets for training machine learning models, despite higher levels of inaccurate labeling compared to expert labeling. There are two common strategies to manage the impact of such noise: The first involves aggregating redundant annotations, but comes at the expense of labeling substantially fewer examples. Secondly, prior works have also considered using the entire annotation budget to label as many examples as possible and subsequently apply denoising algorithms to implicitly clean the dataset. We find a middle ground and propose an approach which reserves a fraction of annotations to explicitly clean up highly probable error samples to optimize the annotation process. In particular, we allocate a large portion of the labeling budget to form an initial dataset used to train a model. This model is then used to identify specific examples that appear most likely to be incorrect, which we spend the remaining budget to relabel. Experiments across three model variations and four natural language processing tasks show our approach outperforms or matches both label aggregation and advanced denoising methods designed to handle noisy labels when allocated the same finite annotation budget.

## introduction modern machine learning often depends on heavy data annotation efforts. to keep costs in check while maintaining speed and scalability, many people turn to non-specialist crowd-workers through platforms like mechanical turk. although crowdsourcing reduces costs to a reasonable level, it also tends to produce substantially higher error rates compared with expert labeling. the classic approach for improving reliability in classification tasks is to perform redundant annotations which are later aggregated using a majority vote to form a single gold label @xcite @xcite . this ## related work the standard method for learning in the presence of unreliable annotation is to perform redundant annotation, where each example is annotated multiple times and a simple majority vote determines the final label @xcite @xcite . while effective, this can be costly since it severely reduces the amount of data collected. to tackle this problem, researchers have developed several alternative methods for dealing with noisy data that can be broken down into three categories. ## methods under study we study how to maximize model performance given a static data annotation budget. concretely, we are given some model m for a target task, along with a budget as measured by b number of annotations, where each annotation allows us to apply a possibly noisy labeling function f r (x), where r is the number of redundant annotations applied to a single example. annotating some set of unlabeled instances produces noisy examples (x, f r (x)) = (x, ỹ ). our goal is to achieve the best score possible for some primary evaluation metric s on a given task by cleaning the noisy labels ỹ clean ---→ y . afterwards, we train a model with the cleaned data and then test it on a separate test set. for all our experiments, we set @xmath0, 000 as the total annotation budget. as a default setting, we start with a control baseline which uses the entire budget to annotate 12k examples, once each (@xmath1, 000; @xmath2. to simulate a single annotation, we randomly sample a label from the set of labels offered for each example by the dataset. to obtain more accurate labels, people often perform multiple annotations on each example and use majority vote to aggregate the annotations. accordingly, as a second baseline we annotate 4k examples three times each (@xmath3, 000; @xmath4, matching the same total budget as before. in the event of a tie, we randomly select one of the candidate labels. finally, we also include an oracle baseline which uses the gold label for 12k examples (@xmath5, 000; @xmath6. the gold label is either given by the dataset or generated by majority vote, where the label might result from aggregating five annotations rather than just three annotations. ## major results breakdown by task ## discussion and analysis to better understand how the proposed clean methods operate, we conduct additional analysis with the sentiment analysis task. ## conclusion noisy data is a common problem when annotating data under low resource settings. performing redundant annotation on the same examples to mitigate noise leads to having even less data to work with, so we propose data cleaning instead through targeted relabeling. we apply our methods on multiple model sizes and nlp tasks of varying difficulty, which show that saving a portion of a labeling budget for re-annotation matches or outperforms other baselines despite requiring no extra parameters to train or hyper-parameters to tune. intuitively, our best method can be summarized as double-checking the examples that the model gets wrong to see if it is actually an incorrect label causing problems. thus, to make the most out of the scarce labeled data available, we believe a best practice should include targeting examples for cleaning rather than spending the entire annotation budget upfront. future work includes exploring more sophisticated techniques for identifying examples to relabel and understanding how such cleaning models perform on additional nlp tasks such as machine translation or dialogue state tracking, which have distinct output formats.