Cost-Efficient Subjective Task Annotation and Modeling through Few-Shot Annotator Adaptation (2024)

Abstract: In subjective NLP tasks, where a single ground truth does not exist, the inclusion of diverse annotators becomes crucial as their unique perspectives significantly influence the annotations. In realistic scenarios, the annotation budget often becomes the main determinant of the number of perspectives (i.e., annotators) included in the data and subsequent modeling. We introduce a novel framework for annotation collection and modeling in subjective tasks that aims to minimize the annotation budget while maximizing the predictive performance for each annotator. Our framework has a two-stage design: first, we rely on a small set of annotators to build a multitask model, and second, we augment the model for a new perspective by strategically annotating a few samples per annotator. To test our framework at scale, we introduce and release a unique dataset, Moral Foundations Subjective Corpus, of 2000 Reddit posts annotated by 24 annotators for moral sentiment. We demonstrate that our framework surpasses the previous SOTA in capturing the annotators’ individual perspectives with as little as 25% of the original annotation budget on two datasets. Furthermore, our framework results in more equitable models, reducing the performance disparity among annotators.

## introduction the common pipeline for supervised learning in natural language processing (nlp) starts by collecting annotations from multiple annotators. these annotations are often aggregated through majority voting @xcite to construct a ground truth or gold standard on which the subsequent modeling is performed. in recent years, researchers have advocated for a transition from single ground-truth labels to annotator-level modeling, aiming to capture diverse perspectives, enhance contextual understanding, and incorporate cultural nuances @xcite , and have proposed different frameworks that take into account unique perspectives of the annotators by modeling them as separate subtasks @xcite . the impact of individual annotators' backgrounds and life experiences on annotations in subjective tasks signifies the importance of incorporating a diverse set of annotators. nevertheless, the primary constraint on achieving this diversity is often the annotation budget, limiting the number and, consequently, the diversity of perspectives considered. in this paper, we introduce a novel framework for annotation collection and modeling in subjective tasks. our framework is designed to minimize the annotation budget required to model a fixed number of annotators, while maximizing the predictive performance for each annotator. our framework operates in two stages. in the first stage, data is collected from a small pool of annotators. this data serves as a foundation for building a multitask model that captures the general patterns for the task and provides a signal of differences among individual annotators. informed by the first stage annotations, the second stage involves collecting a few samples from each new annotator that best capture their differences from the general patterns. we use this data to augment the model from the first stage to learn the new annotators' perspective from a few examples (figure 1 ). we introduce a unique dataset that enables the study of detecting moral content, an understudied subjective task, at a scale that was not possible before foot_0 . the moral foundations subjective corpus (mfsc) is a collection of 2000 reddit posts, each annotated by 24 annotators for moral content along with annotators' responses to a range of psychological questionnaires ( §4.1).we use the mfsc dataset in conjunction with the brexit hate dataset @xcite to extensively study each component of our proposed framework. we evaluate our framework on three models: roberta-base, roberta-large, and llama-3. in section 5.1, we demonstrate the efficacy of our framework in capturing diverse annotator perspectives under budget constraints. our framework achieves a 4% increase in f 1 score with access to just 50% of the annotation budget in hate speech detection, and a 2% gain in moral sentiment detection with as little as 25% of the original annotation budget. additionally, we evaluate the scalability of our framework by incorporating new annotators into an existing annotated dataset and model using our second-stage fewshot adaptation process. our results show an f 1 score gain of 9% and 4% in the brexit and mfsc datasets, respectively, demonstrating the scalability of our framework. next, in section 5.2, we show that our proposed framework yields a more equitable model by minimizing performance disparity across annotators. specifically, in the lowest budget scenarios, our approach reduces the standard deviation of the performance across annotators by 7% in hate speech detection and by 1% in moral sentiment classification. finally, in section 5.3, we extend our analysis to investigate whether the selection of the initial set of annotators in the first stage of our framework affects the model's performance. our experiments on two subjective datasets revealed that our framework consistently surpasses previous state-of-the-art models with access to as little as 25% of the original annotation budget. in addition, our framework produced more equitable models with reduced performance disparities among the annotators. by minimizing data requirements, our cost-efficient framework for subjective tasks enables us to scale the number of included annotators and, hence, improve the diversity of captured perspectives. furthermore, the two-stage design of our framework facilitates the integration of new annotators into pre-existing datasets. ## conclusion we introduced a framework for annotation collection and annotator modeling in subjective tasks. our framework aims to minimize the annotation budget required to model a fixed number of annotators while maximizing the predictive performance for each annotator. our approach involves collecting annotations from an initial set of annotators and building a multitask model that captures general task patterns while signaling differences among individual annotators. subsequently, we utilize the annotations from the first stage to select a small set of samples from new annotators that best highlight their deviations from the general patterns. finally, we use these samples to augment the initial model in a few-shot setting to learn the new annotator's perspective. we evaluated our framework using three base models, and explored four distinct methods for few-shot sample selection and found that the most effective approach involves balanced and random sample selections. we introduced a new subjective task dataset moral foundations subjective corpus (mfsc), of 2000 reddit posts annotated by 24 annotators for moral sentiment which enabled us to test our framework in scale. our experiments on mfsc and a hate speech dataset revealed that our framework consistently surpasses previous sota with access to as little as 25% of the original annotation budget. in addition, we showed that our framework yields more equitable models that reduce performance disparities among annotators. our cost-efficient framework for subjective tasks allows enhancing the diversity of the captured perspectives, and facilitates the integration of new annotators into pre-existing datasets and models.