ACTOR : Active Learning with Annotator-specific Classification Heads to Embrace Human Label Variation (2023)

Abstract: Label aggregation such as majority voting is commonly used to resolve annotator disagreement in dataset creation. However, this may disregard minority values and opinions. Recent studies indicate that learning from individual annotations outperforms learning from aggregated labels, though they require a considerable amount of annotation. Active learning, as an annotation cost-saving strategy, has not been fully explored in the context of learning from disagreement. We show that in the active learning setting, a multi-head model performs significantly better than a single-head model in terms of uncertainty estimation. By designing and evaluating acquisition functions with annotator-specific heads on two datasets, we show that group-level entropy works generally well on both datasets. Importantly, it achieves performance in terms of both prediction and uncertainty estimation comparable to full-scale training from disagreement, while saving 70% of the annotation budget.

## introduction an important aspect of creating a dataset is asking for multiple annotations and aggregating them in order to derive a single ground truth label. aggregating annotations, however, implies a single golden ground truth, which is not applicable to many subjective tasks such as hate speech detection @xcite . a human's judgement on subjective tasks can be influenced by their perspective and beliefs or cultural background @xcite . when addressing disagreement in annotation, aggregating them by majority vote could result in the viewpoints of the minority being overlooked @xcite . in order to address this issue, many works have been proposed to directly learn from the annotation disagreements in subjective tasks. there are two major approaches to achieving that: learning from the soft label @xcite , ## method multi-head model we use a multi-head model where each head corresponds to one unique annota-tor, following davani et al., 2022. in the fine-tuning stage, annotations are fed to the corresponding annotator heads, adding their losses to the overall loss. during testing, the f1 score is calculated by comparing the majority votes of the annotator-specific heads with the majority votes of the annotations. ## result single-head vs multi-head model figure 2 shows the comparison of the multi-head model and the single-head during the active learning process in the upper row, we compare the multi-head approach with single-majority approach on majority f1 score and uncertainty estimation. in terms of predicting the majority vote, the multi-head model performs on par with the best-performing singlehead method on both datasets, such as bald. for uncertainty estimation measured against annotator disagreement, the multi-head model outperforms the single-head model by a large margin. we have the same observation when comparing with single-annotation model, shown in the bottom row. therefore, we recommend using a multi-head model in a subjective task where humans may disagree and uncertainty estimation is important. ## conclusion we presented an active learning framework that embraces human label variation by modelling the annotator with annotator-specific classification heads, which are used to estimate the uncertainty at the individual annotator level and the group level. we first showed that a multi-head model is a better choice over a single-head model in the active learning setting, especially for uncertainty estimation. we then designed and tested five acquisition functions for the annotator-heads model on two datasets. we found that group-level entropy works generally well on both datasets and is recommended. depending on the dataset properties, the individual-level entropy method performs differently.