Don’t waste a single annotation: improving single-label classifiers through soft labels (2023)

Abstract: In this paper, we address the limitations of the common data annotation and training methods for objective single-label classification tasks. Typically, when annotating such tasks annotators are only asked to provide a single label for each sample and annotator disagreement is discarded when a final hard label is decided through majority voting. We challenge this traditional approach, acknowledging that determining the appropriate label can be difficult due to the ambiguity and lack of context in the data samples. Rather than discarding the information from such ambiguous annotations, our soft label method makes use of them for training. Our findings indicate that additional annotator information, such as confidence, secondary label and disagreement, can be used to effectively generate soft labels. Training classifiers with these soft labels then leads to improved performance and calibration on the hard label test set.

## introduction reliable, human-annotated data is crucial for training and evaluation of classification models, with the quality of annotations directly impacting the models' classification performance. traditionally, in order to ensure high quality annotated data, multiple annotators are asked to judge each individual data instance, and the final 'gold standard' hard label is determined by majority vote. however, this hard label approach tends to ignore valuable information from the annotation process, failing to capture the uncertainties and intricacies in real-world data @xcite ). an emerging alternative approach that addresses these limitations is the use of soft labels through techniques such as knowledge distillation @xcite , label smoothing @xcite , confidence-based labeling @xcite , and annotation aggregation @xcite . these soft label approaches demonstrate potential for improved robustness @xcite , superior calibration, enhanced performance @xcite and even enable less than one-shot learning @xcite . this paper's primary focus is on exploring effective ways for improving classification performance using soft labels. our experimental findings indicate that confidence-based labelling significantly enhances model performance. nevertheless, the interpretation of confidence scores can also profoundly influence model capability. given the variability in confidence levels among different annotators @xcite , aligning these disparate confidence levels emerges as the central research question of this paper. to address this challenge, we propose a novel method for generating enhanced soft labels by leveraging annotator agreement to align confidence levels. our contributions include: ## related work current research typically interprets annotator disagreement in two primary ways, either by capturing diverse beliefs among annotators, or by assuming a single ground truth label exists despite disagreement @xcite . this paper focuses on situations where the latter viewpoint is more applicable. thus, we focus on traditional "hard" evaluation metrics such as f1score which rely on a gold-label, despite the emergence of alternative, non-aggregated evaluation approaches @xcite @xcite . this is made possible because we evaluate on high-agreement test sets, where the 'true' label is fairly certain. aggregation of annotator disagreement generally falls into two categories: aggregating labels into a one-hot hard label @xcite @xcite , or modeling disagreement as a probability distribution with soft labels @xcite @xcite @xcite . similar to @xcite , our study explores how soft labels can be generated from a small pool of annotators, using additional information such as their self-reported confidence. this has benefits over traditional hard/soft label aggregation, which requires extensive annotator resources and/or reliance on potentially unreliable crowd-sourced annotators @xcite @xcite . ## methodology in order to generate soft labels, our methodology requires annotators to provide confidence scores. figure 1 shows how each annotator provides both a primary class label and a confidence rating that represents their certainty. this ranges from 0 to 1, with 1 representing 100% confidence. 2 in addition, annotators can also provide an optional 'secondary' class label. this represents their selection of the second most probable class. thus, formally, the annotation of the text x i by an annotator a m consists of a primary label l im , its confidence rating c im , and an optional secondary label l 2 im . we use y i to denote the text's true label. overall, there are three steps to generating soft labels as shown in figure 1: 1. annotator confidences are calibrated using our bayesian method (section 3.1) 2. annotations are converted to soft labels (section 3.2) 3. annotator soft labels are merged into a final 2 in practice, annotators can provide this directly as a percentage or choose from a likert-style numerical rating (e.g. 1-5) that is then converted to 0-1 scale. ## results soft labels improve classification performance across all of our scenarios. combining confidence scores and secondary labels generates better soft labels. using annotators' self-reported confidence scores helps to improve soft labels, as shown by the f1 and calibration improvements between soft labels in the 'with' and 'without confidence' settings ( furthermore, for the vaxxhesitancy dataset, once secondary label information is included, classification performance is further improved from 73.7 to 74.5. this suggests that more consideration should be taken during the data annotation stage to collect such information, as it can be greatly beneficial for the creation of effective soft labels. ## conclusion we demonstrate the benefits of using soft labels over traditional hard labels in classification tasks. we propose a novel bayesian method for annotation confidence calibration, and efficiently utilising all available annotation information, outperforming other methods for the vaxxhesitancy dataset. the performance improvements offered by soft labels suggests the importance of collecting additional information from annotators during the annotation process, with annotator confidence being particularly important. land). 5 . ben wu is supported by an epsrc doctoral training partnership grant and yue li is supported by a sheffield-china scholarships council phd scholarship.