Which Examples Should be Multiply Annotated? Active Learning When Annotators May Disagree (2023)

Abstract: Linguistic annotations, especially for controversial topics like hate speech detection, are frequently contested due to annotator backgrounds and positionalities. In such situations, preserving this disagreement through the machine learning pipeline can be important for downstream use cases. However, capturing disagreement can increase annotation time and expense. Fortunately, for many tasks, not all examples are equally controversial; we develop an active learning approach, Disagreement Aware Active Learning (DAAL) that concentrates annotations on examples where model entropy and annotator entropy are the most different. Because we cannot know the true entropy of annotations on unlabeled examples, we estimate a model that predicts annotator entropy trained using very few multiply-labeled examples. We find that traditional uncertainty-based active learning underperforms simple passive learning on tasks with high levels of disagreement, but that our active learning approach is able to successfully improve on passive and active baselines, reducing the number of annotations required by at least 24% on average across several datasets.

## introduction disagreement in annotations is natural for humans, often depending on one's background, identity, and positionality. this is especially salient when building classifiers for hate speech, toxicity, stereotypes, and offensiveness, where recent work has shown the importance of modeling annotator diversity and accounting for the full distribution of annotations rather than just a "majority vote" label @xcite @xcite . however, collecting annotations in high-disagreement scenarios is expensive in time, effort, and money, because modeling annotator uncertainty may require collecting many labels for each example. figure 1 : utility of annotations when annotators disagree/agree (rows) and when the model is unconfident/confident (columns). when model uncertainty is well-calibrated with annotator uncertainty, no more annotations are needed. however, additional annotation(s) can be advantageous when the model is underconfident (e.g., uncertain on high agreement examples early in training) or overconfident (i.e., overly certain on high disagreement examples). examples are edited to remove swears and slurs, and the high annotator uncertainty example is lightly paraphrased for anonymity. to decrease labeling costs, we turn to active learning, a machine learning framework that selectively elicits annotations on examples that are most likely to improve a model's performance while minimizing annotation costs @xcite . many active learning approaches select examples to label based on some measure of model uncertainty, with the aim of driving down model uncertainty as quickly as possible. however, in the case of potential annotator disagreement, uncertainty-based sampling is not obviously a good strategy. intuitively, an algorithm should collect annotations on examples for which the model uncertainty is significantly different from the annotator uncertainty, so these new annotations are able to help calibrate the model. similarly, an active learning algorithm might plausibly request new labels on already labeled samples to better model the full distribution of possible annotations. this raises a "goldilocks problem": on examples with complete annotator agreement, we do not need more than one annotation, while on examples with complete disagreement, no annotations are needed; it is precisely those examples in the middle-some, but not perfect agreement-on which multiple annotations are potentially useful. in this paper, we develop daal (disagreement aware active learning), 1 an active learning algorithm for training classifiers to predict full label distributions on tasks with likely disagreement. daal first builds an entropy predictor that estimates, for a given example, how much annotator disagreement there is likely to be on that example. then, using this entropy predictor, daal trains a task predictor that queries examples for which the current task predictor's current entropy is most different from its estimated human entropy (figure 1 ). we evaluate daal on several text classification problems related to english hate speech and toxicity detection, finding that: one of the most commonly deployed approaches to minimize the number of collected annotations to train a model is active learning, where the main idea is to collect only those annotations that might be helpful for improving model performance. active learning algorithms operate iteratively, where in each round a small number (often one) of examples are requested to be annotated. these annotated examples are added to a training set, a model is trained on that dataset, and then the process repeats. one popular strategy for selecting which examples to have annotated in each round is uncertainty sampling, where the model queries on examples on which it is the least certain @xcite @xcite , with uncertainty often measured by the current entropy of the label distribution produced by the model at the current round. ## experimental setup in this section, we introduce the datasets we use and experimental details. ## results and analysis in this section, we present results for baseline methods ( §5.1) and daal ( §5.2). we also investigate how the budget size and the number of annotations per example affect the entropy predictor's performance ( §5.3). in addition, we discuss in which sitfor these experiments, we define n ≈ 3, which means that there are approximately 3 annotations per example available in the data pool. 6 both baselines have two variations when querying: "batched" receives all 3 annotations per example while "single" receives only one. ## conclusion in this paper, we emphasize the importance of accounting for disagreement present in data. we propose daal, an active learning approach, which incorporates both annotator and model uncertainties, aiming to reduce the cost of annotation. this cost includes both time and money, but also an often overlooked cost related to the repeated exposure of annotators to toxic and harmful content. when the annotation is performed on crowdsourcing platforms, where workers are often from vulnerable populations who may require more flexible employment options-such as those with disabilities or who have caregiver roles @xcite -this mental health cost compounds existing marginalization. in our experiments on training classifiers for hate speech and toxicity detection, we show that daal achieves comparable jensen-shannon di-10359 vergence with the classic baselines' performance but requires an average of 1.235× fewer annotations in the worst case. it is also equally effective when there is little annotator disagreement, making it a strong general solution candidate even when one does not know ahead of time how much annotator disagreement is likely for a given task.