Annotator-Centric Active Learning for Subjective NLP Tasks (2024)

Abstract: Active Learning (AL) addresses the high costs of collecting human annotations by strategically annotating the most informative samples. However, for subjective NLP tasks, incorporating a wide range of perspectives in the annotation process is crucial to capture the variability in human judgments. We introduce Annotator-Centric Active Learning (ACAL), which incorporates an annotator selection strategy following data sampling. Our objective is two-fold: (1) to efficiently approximate the full diversity of human judgments, and (2) to assess model performance using annotator-centric metrics, which value minority and majority perspectives equally. We experiment with multiple annotator selection strategies across seven subjective NLP tasks, employing both traditional and novel, human-centered evaluation metrics. Our findings indicate that ACAL improves data efficiency and excels in annotator-centric performance evaluations. However, its success depends on the availability of a sufficiently large and diverse pool of annotators to sample from.

## introduction a challenging aspect of natural language understanding (nlu) is the variability of human judgment and interpretation in subjective tasks (e.g., hate speech detection) @xcite . in a subjective task, a data sample is typically labeled by a set of annotators, and differences in annotation are reconciled via majority voting, resulting in a single (supposedly, true) "gold label" @xcite . however, this approach has been criticized for treating label variation exclusively as noise, which is especially problematic in sensitive subjective tasks @xcite since it can lead to the exclusion of minority voices @xcite . subjectivity can be addressed by modeling the full distribution of annotations for each data sample instead of employing gold labels @xcite . however, resources for such approaches are scarce, as most datasets do not (yet) make fine-grained annotation details available @xcite , and representing a full range of perspectives is contingent on obtaining costly annotations from a diverse set of annotators @xcite . one way to handle a limited annotation budget is to use active learning @xcite . given a pool of unannotated data samples, al employs a sample selection strategy to obtain maximally informative samples, retrieving the corresponding annotations from a ground truth oracle (e.g., a single human expert). however, in subjective tasks, there is no such oracle. instead, we rely on a set of available annotators. demanding all available annotators to annotate all samples would provide a truthful representation of the annotation distribution, but is often unfeasible, especially if the pool of annotators is large. thus, deciding which annotator(s) should annotate is as critical as deciding which samples to annotate. in most practical applications, annotators are randomly selected. this results in an annotation distribution insensitive to outlier annotators-most annotations reflect the majority voices and fewer reflect the minority voices. this may not be desirable in applications such as hate speech, where the opinions of the majority and minority should be valued equally. in such cases, a more deliberate annotator selection is required. to ensure a balanced representation of majority and minority voices, we leverage strategies inspired by rawls' principle of fairness @xcite , which advocates that a fair society is achieved when the well-being of the worst-off members of society (the minority annotators, in this case) is maximized. we introduce annotator-centric active learning (acal) to emphasize and control who annotates which sample. in acal (figure 1 ), the sample selection strategy of traditional al is followed by an annotator selection strategy, indicating which of the available annotators should annotate each selected data sample. contributions (1) we present acal as an extension of the al approach and introduce three annotator selection strategies aimed at collecting a balanced distribution of minority and majority annotations. (2) we introduce a suite of annotatorcentric evaluation metrics to measure how individual and minority annotators are modeled. (3) we demonstrate acal's effectiveness in three datasets with subjective tasks-hate speech detection, moral value classification, and safety judgments. our experiments show that the proposed acal methods can approximate the distribution of human judgments similar to al while requiring a lower annotation budget and modeling individual and minority voices more accurately. however, our evaluation shows how the task's annotator agreement and the number of available annotations impact acal's effectiveness-acal is most effective when a large pool of diverse annotators is available. importantly, our experiments show how the acal framework controls how models learn to represent majority and minority annotations. this is crucial for subjective and sensitive applications such as detecting human values and morality @xcite , argument mining @xcite , and hate speech @xcite . ## method first, we define the soft-label prediction task we use to train a supervised model. then, we introduce the traditional al and the novel acal approaches. ## experimental setup we describe the experimental setup for the comparisons between acal strategies. in all our experiments, we employ a tinybert model @xcite to reduce the number of trainable parameters. appendix a includes a detailed overview of the computational setup and hyperparameters. we make the code for the acal strategies and evaluation metrics available via github. foot_0 ## results we start by highlighting the benefits of acal over al and pl (section 5.1). next, we closely examine acal on efficiency and fairness (section 5.2). then, we select a few cases of interest and dive deeper into the strategies' behavior during training (section 5.3). finally, we investigate acal across varying levels of subjectivity (section 5.4). ## conclusion we present acal as an extension of al to emphasize the selection of diverse annotators. we introduce three novel annotator selection strategies and four annotator-centric metrics and experiment with tasks across three different datasets. we find that the acal approach is especially effective in reducing the annotation budget when the pool of available annotators is large. however, its effectiveness is contingent on data characteristics such as the number of annotations per sample, the number of annotations per annotator, and the nature of disagreement in the task annotations. furthermore, our novel evaluation metrics display the trade-off between modeling overall distributions of annotations and adequately accounting for minority voices, showing that different strategies can be tailored to meet different goals. especially early in the training process, strategies that are aggressive in obtaining diverse labels have a beneficial impact in accounting for minority voices. however, we recognize that gathering a distribution that uniformly contains all possible (minority and majority) labels can be overly sensitive to small minorities or noise. future work should integrate methods that account for noisy annotations @xcite . striking a balance between utilitarian and egalitarian approaches, such as between modeling aggregated distributions and accounting for minority voices @xcite is crucial for inferring context-dependent values @xcite .