On Efficient and Statistical Quality Estimation for Data Annotation (2024)

Abstract: Annotated datasets are an essential ingredient to train, evaluate, compare and productionalize supervised machine learning models. It is therefore imperative that annotations are of high quality. For their creation, good quality management and thereby reliable quality estimates are needed. Then, if quality is insufficient during the annotation process, rectifying measures can be taken to improve it. Quality estimation is often performed by having experts manually label instances as correct or incorrect. But checking all annotated instances tends to be expensive. Therefore, in practice, usually only subsets are inspected; sizes are chosen mostly without justification or regard to statistical power and more often than not, are relatively small. Basing estimates on small sample sizes, however, can lead to imprecise values for the error rate. Using unnecessarily large sample sizes costs money that could be better spent, for instance on more annotations. Therefore, we first describe in detail how to use confidence intervals for finding the minimal sample size needed to estimate the annotation error rate. Then, we propose applying acceptance sampling as an alternative to error rate estimation We show that acceptance sampling can reduce the required sample sizes up to 50% while providing the same statistical guarantees.

## introduction having large, high-quality annotated datasets available is crucial for successfully training machine learning models, as well as for evaluating and bringing them into production @xcite @xcite . therefore, during the creation of new datasets, it is essential to gauge dataset quality efficiently and statistically soundly. in case the quality is too low, then rectifying measures can be taken to improve it. these can be, among others, to figure 1 : overview of agile data corpus creation, the recommended workflow to annotate high-quality datasets. this work explores how to efficiently estimate annotation quality using statistics. estimating annotation quality, however, is expensive, time consuming and tedious, as it typically requires skilled annotators or experts that manually inspect the annotated instances @xcite . therefore, in practice, often only small subsamples are inspected and used to estimate data quality or no estimate at all is performed @xcite . small sample sizes have low statistical power and large error margins, which bears the risk of having inaccurate and too optimistic error rate estimates @xcite . having no error estimate at all makes it difficult to track data quality throughout the annotation process. these issues can then result in datasets that contain a non-negligible percentage of incorrect annotations @xcite . subsequently, using datasets with label errors then can lead to inference errors in production, or wrong conclusions when comparing model architectures and training regimes @xcite @xcite . in this work, we hence emphasize the importance of proper quality estimation during, as well as after, the annotation process. we then describe how to calculate the sample size needed to estimate the proportion of annotation errors using confidence intervals given certain desired statistical guarantees. our analysis shows that estimating the annotation error rate, which is expected to be small, requires relatively large sample sizes to achieve tight bounds and high confidence in the estimate. as this can be prohibitively expensive, we suggest using acceptance sampling @xcite , a statistical quality management technique stemming from industrial manufacturing, instead. acceptance sampling does not estimate the error proportion directly. it judges batches of annotated instances on a accept/reject basis whether they reach a desired quality level or not. this is done by inspecting subsamples whose size is based on the desired statistical guarantees. we show that it can be a viable approach for annotation quality estimation: it is able to provide similar statistical guarantees as confidence intervals while requiring up to 50% less samples. our contributions are as follows: to the best of our knowledge, we are the first to investigate statistical error rate estimation and acceptance sampling in the context of data annotation. ## background performing annotation quality estimation is crucial during and after the annotation process. accurate quality estimates are needed, because if it is known that quality is insufficient, countermeasures can be taken to improve the quality. this is closely intertwined with how the annotation process is structured, which we discuss first. then, we present the most common ways to estimate annotation quality. annotation process how the annotation process is structured has a large impact on the resulting data quality. the traditional dataset creation process is similar to the waterfall model, in which phases of data collecting, annotation scheme creation, annotation and validation follow another @xcite . a more modern approach is agile corpus creation @xcite @xcite where annotations are given out in batches. after each step, the quality is estimated and annotations are validated. this allows evaluating the quality throughout the annotation process to incorporate feedback, develop or adjust the annotation guidelines, retrain annotators, or re-annotate batches with too many errors @xcite . in both cases, it is recommended that annotations are not just published, but the quality is validated via manual inspection during the creation and before release. agreement a common approach to determine annotation reliability is to collect multiple annotations per instance. these can then be used to compute inter-annotator agreement @xcite . however, agreement does not automatically mean correctness, as recent work has shown that state-of-the-art datasets still can contain a non-negligible percentage of errors despite high agreement @xcite . using agreement can also substantially increase the cost. for many datasets, only one annotation per instance is collected for saving save costs, or because qualified annotators are difficult to recruit. then, only a subset of instances are annotated multiple times, based on which agreement is calculated. oftentimes, the resulting sample size is too low for a confident estimate with low margin of error @xcite . the computation and interpretation of agreement is often also not straightforward @xcite . hence, agreement should not be the only metric of quality @xcite . error rate estimation due to the aforementioned issues, it is recommended that annotation projects use (expert) annotators that inspect a (sufficiently large) subset of annotations to estimate the percentage of errors, which we call here the error rate @xcite ). during agile corpus annotation, the error rate can then be used to determine whether batches are of sufficient quality or require further improvement. what is missing in many annotation projects, however, is statistical grounding and analysis of their quality estimation step. for instance, sample sizes to estimate the error rate are often picked ad-hoc. that is, they are not grounded in statistics and are often too small @xcite . the error rate then is often only a point estimate, confidences in the estimates are only rarely taken into account. this can lead to wrong estimates of the error rate and negatively impact the underlying data quality and downstream usage of the final dataset. to alleviate the aforementioned issues, we propose using error rate estimation with a sound, statistical footing. for this, we first define best practices for estimating the annotation error rate using confidence intervals to determine the minimal sample size needed given desired statistical guarantees. in addition, we introduce acceptance sampling to the annotation process, which we describe and analyze in the following. to the best of our knowledge, we are the first to discuss statistical quality control and especially acceptance sampling, coming from manufacturing, to the annotation process. ## annotation quality estimation the goal of this work is to ground annotation error estimation in statistics and to make it more econom-ical. therefore, in the following, we first model the process of manual inspection as sampling without replacement. we then describe two statistical quality estimation methods for annotation, confidence intervals and acceptance sampling. the setting we consider here is annotation in batches, the modus operandi of the widely recommended, agile annotation process @xcite @xcite . working with batches instead of all data at once allows stakeholders to incorporate feedback during an annotation project and improve the data quality during annotation if necessary. but to apply countermeasures in case quality is not sufficient, it is first necessary to know whether the quality is sufficient or not. as our goal is to estimate annotation quality with a minimum of manual inspection for a given level of confidence and precision, we also discuss how to compute the sample size needed for each method. we compare the presented methods in section 4. ## sample size analysis to make quality management more feasible, our goal is to estimate the quality in annotated batches given a desired set of statistical guarantees while inspecting as few samples as possible. to emphasize the importance of choosing a proper sample size, we first briefly simulate the relation between sample size and the reliability of an estimate. if it is too small, then the margin of error is large compared to the proportion estimated and the quality estimate cannot be trusted. then, we compare confidence intervals and acceptance sampling with regard to their required average sample number (asn). the following results are derived analytically and are agnostic to the underlying datasets, as long as the statistical assumptions hold; only dataset size and error counts are used. we show the application of acceptance sampling on existing datasets with known error rate section 5. the asn for a method is the expected number of instances that need to be inspected until a decision can be made to accept or reject a batch. in case of confidence intervals and single sampling, the asn is a constant; the whole sample is always inspected as there is no form of early stopping. for double and sequential sampling, the asn depends on the true error rate in the current batch. for our analysis, we assume a batch size of @xmath0. if it was smaller, then it is often feasible to inspect whole batches. if it was larger, then this causes only minor changes to the required sample sizes (see fig. 9 ). we choose two sets of values simulating two different quality control regimes, which we call strict and relaxed. they are listed section 4. computing the minimum sample size to achieve a certain confidence interval width given a confidence level requires assuming a value for the a-priori unknown true error rate p e . we choose to calculate the sample size both for p @xmath1. the closer p e is to 0.5, the larger the required sample size would be. note that using a sample size estimate for a p e leads to underpowered estimates if @xmath2, which makes using confidence intervals more difficult to use correctly compared to acceptance sampling, where this is not needed. therefore, the rates for which we estimate confidence interval sample sizes can be seen as an optimistic estimate. ## acceptance sampling in practice in the following, we show the application of acceptance sampling on actual datasets with known error rates and observe their behaviour. for this, we use two datasets that have been found to contain non-negligible amounts of annotation errors: conll 2003 is a dataset for named entity recongition, introduced by @xcite ; errors analysis was conducted by @xcite . imdb is a dataset for sentence-level sentiment classification, introduced by @xcite , error analysis was performed by @xcite . we use the dataset and error counts as described in @xcite . these are listed in for conll 2003, we consider annotations and errors on sentence-level in order to satisfy the independence assumption needed for the statistical methods used; a sentence is considered erroneous if at least one span is annotated incorrectly. we use the same configurations as described in section 4. fig. 5 shows the results. it can be seen that for conll-2003, almost all samples are rejected. this is expected, because the error rate is above 3% and 5% for the strict and relaxed configurations, respectively. for imdb, the error rate is in the zone of indifference for the strict configuration. thus, the results are inconclusive. in a real setting, these batches would be either accepted or inspected further to determine the underlying error rate as well as to correct instances (institute of @xcite . in its relaxed configuration, most samples are accepted even though the error rate is slightly above p a . most methods still accept there, except for sequential sampling without curtailment. average sample sizes for these plans are listed in ## conclusion in this work, we proposed two methods statistical concerning quality control for data annotation. first, we showed how to estimate the error rate statistically sound using confidence intervals. given