Corpus Considerations for Annotator Modeling and Scaling (2024)

Abstract: Recent trends in natural language processing research and annotation tasks affirm a paradigm shift from the traditional reliance on a single ground truth to a focus on individual perspectives, particularly in subjective tasks. In scenarios where annotation tasks are meant to encompass diversity, models that solely rely on the majority class labels may inadvertently disregard valuable minority perspectives. This oversight could result in the omission of crucial information and, in a broader context, risk disrupting the balance within larger ecosystems. As the landscape of annotator modeling unfolds with diverse representation techniques, it becomes imperative to investigate their effectiveness with the fine-grained features of the datasets in view. This study systematically explores various annotator modeling techniques and compares their performance across seven corpora. From our findings, we show that the commonly used user token model consistently outperforms more complex models. We introduce a composite embedding approach and show distinct differences in which model performs best as a function of the agreement with a given dataset. Our findings shed light on the relationship between corpus statistics and annotator modeling performance, which informs future work on corpus construction and perspectivist NLP.

## introduction an integral aspect of dataset creation is obtaining multiple annotations from annotators on the same instances of data @xcite . more often than not, these are aggregated through a majority vote to arrive at a single ground truth label @xcite . however, an annotator's background influences the label they assign. this divergence is often * denotes equal contribution evident in the judgments and perceptions of annotators of subjective tasks. in the social media domain, where people's reactions are influenced by their personal experiences and vested interests, relying solely on a majority label to determine or reach a consensus proves challenging @xcite . hence, it becomes crucial to improve annotator modeling frameworks for robust user representations that capture the diverse views inherent in our datasets, while preserving individual perspectives. interest in data perspectivism has been growing and with it, approaches for annotator modeling @xcite @xcite . the approaches for annotator modeling are built from corpora with unaggregated labels widely ranging in the number of annotators, data type and volume available per annotator, the type of task, and the magnitude of disagreement @xcite @xcite @xcite . though a few recent works have mentioned the impact of the number of annotators or the level of agreement on annotator modeling methods @xcite @xcite , they have not been systematically explored. in this paper, we perform the first systematic study of the scalability of annotator modeling methods and the relationship between annotator modeling methods and corpus statistics. we implement annotator modeling and personalization techniques used in recent work @xcite @xcite and implement our own novel composite embedding approach. this work sheds light on the effectiveness of annotator modeling methods under various realworld scenarios for subjective tasks. we provide recommendations for which methods to use based on the available data. we find that when agreement is high, our composite embedding performs best, while when agree-ment is lower, the user token approach common in previous work performs best. we find that the user token approach often outperforms other more complex methods developed in previous work, including the averaged sbert, authorship attribution embeddings @xcite , and multi-tasking @xcite . we investigate the scalability of these methods with respect to the amount of overall data, the number of annotations per annotator, and the number of annotators in the dataset across seven different datasets and approximately 3k subsamples representing artificial datasets with controllable properties. we find that the number of annotations per annotator is the most important factor for annotator modeling, though the number of instances and annotators in the corpus both had weak but significant correlations with performance. our code and the statistics of all trials in our experiments are publicly available foot_0 to support future work that examines the relationship between corpus statistics and annotator modeling performance. ## related work in the first part of this section, we provide an overview of annotator modeling approaches, from modeling annotators by numeric identifiers over multitask models to representations of both, annotator and annotation information. in the second part, we review scaling analyses, identifying the lack of a systematic comparison between annotator modeling methods and corpus statistics. ## datasets we use seven datasets from recent work on annotator modeling. all datasets use binary labels for classification. we include four datasets from the recent semeval-2023 task on learning with disagreements @xcite , two datasets used by @xcite , and the social chemistry dataset @xcite that was adapted for personalization and annotator modeling by @xcite . dataset statistics are presented in gab hate speech corpus the gab hate corpus (ghc) @xcite comprises 27,665 social media posts from gab.com annotated by a minimum of three annotators. the ghc features an extensive coding framework that encompasses hierarchical labels denoting dehumanizing and violent speech, markers indicating targeted groups, and rhetorical framing. as in previous annotator modeling work @xcite , we use labels indicating the presence or absence of hate speech. ## methodology in our study, we investigated five distinct methods for capturing personalized attributes within a collective context, encompassing varying levels of complexity. our models take the annotator id and text that was annotated as input and are designed to distinguish between unique perspectives. in addition to representing the text in the input data, the model also incorporates a representation of the annotator who provided the annotations for the text. these representations are encoded as real-valued, low-dimensional vectors. the choice of how to represent the annotator embedding depends on the specific method being used and the information available about the annotator. with the exception of social chemistry, the annotator tokens are the sole explicit distinguishing attribute of the annotators within the dataset. for several of our methods, annotator embeddings were obtained using special tokens and concatenated with the input strings for each text instance to link each text instance to its corresponding annotator. for the multi-tasking model, the annotator id instead determines which layer of a multi-task model is responsible for predicting the label. additionally, we experimented with personalization techniques for the social chemistry dataset. this dataset contains additional writing for each annotator that can be used to derive annotator representations. we implemented the authorship attribution approach @xcite and averaged sbert embeddings @xcite . note that the averaged sbert embeddings are averaging the additional text provided by annotators (which is only available in soc) rather than the embeddings of the texts that were assigned an annotation. we also introduce a composite embedding approach, where all instances of annotations for each annotator were aggregated for each class. we calculated the average embeddings of the positive class and negative class and concatenated them to derive a unified representation for each annotator. ## experimental setup first, following @xcite , we implemented a text-only baseline that also serves as the base model, which we extend by different methods for annotator modeling. specifically, we used sbert @xcite , a consistently highperforming bert-based model trained to encode sentences for a variety of downstream tasks. we used a pre-trained foot_2 sbert model with the dis-tilroberta @xcite backbone, which features a 768-dimensional representation and a maximum sequence length of 512 tokens. on top of the text encoding provided by sbert, we implemented a classification head and fine-tuned the model for binary classification. for the multi-task model, we used the same setup as @xcite who used a bert model with separate output layers for each annotator @xcite . lastly, since all of our datasets are in english except armis, which is in arabic, we used the arabic bert model from @xcite , which was trained on a combination of data from wikipedia and common crawl. to evaluate, we used the macro f1 score across individual annotator's labels. we trained our models for 10 epochs, employing early stopping based on the validation set performance. the adam optimizer was used, with an initial learning rate set at 2e -5 . we split the data into 80% train and 10% for each of validation and test with the same annotators in all splits. our experiments were conducted on a single nvidia a100 40gb gpu. the average running time for both training and inference phases combined was around 15 minutes per model. ## scaling up subsequently, to further test the impact of dataset statistics reported in when scaling the number of annotators, the three smallest datasets were scaled in increments of one annotator. for the other four, we scaled in increments of two annotators for the range of 6 to 18 annotators; the upper limit for ghc. then for goemotions and social chemistry we scaled from 18 to 82 annotators in increments of 4. lastly, we scaled social chemistry from 100 to 2.5k in increments of 100. this is repeated for each method and ekman emotion across five runs. we tested each method on each subset, which yields a total of 1,670 trials using artificially constructed datasets. figure 1 shows each method for the goemotions dataset when scaling the number of annotators from 6 to 82. the baseline sbert method is marked by a dashed line. user token performed best overall and even performs strongly when the number of annotators and amount of data is low. the composite embedding and multi-tasking methods perform poorly in this setting but approach the performance method ge ghc soc md hsb cva arrmis sbert 67.0 67.5 51.6 79.8 72.7 87.6 ## discussion when examining the dataset statistics and performance, we notice that the user token method performs well when the krippendorff alpha is relatively low (k-α < 0.4). for the social chemistry, convabuse, and armis datasets, the agreement is higher (k-α > 0.5) and we find that in these cases the composite embedding boosts performance, becoming the best model. this may be because when agreement is higher, the composite embedding is more informative as an initialization of the annotator representation. when annotators agree more with each other, the aggregate of their positive and negative labels is more generalizable, as opposed to the random initialization of the user token. the largest boost provided by the composite embedding occurred in the social chemistry dataset, where we have the lowest number of annotations per annotator. this led us to check the correlation between the performance of the user token and composite embedding methods with the number of annotations per annotator when examining the scaling experiments. we found a slightly stronger correlation for the user token (@xmath0.40) than for the composite embedding (@xmath1.35) suggesting that the low number of annotations per annotator is more detrimental to the user token method, but further work is needed to understand this relationship. interestingly, while the multi-task model performed the best in previous work on goemotions and ghc @xcite , we found that it performed worse in our experiments and was more expensive to train. this was the case even when using the full datasets, while our reimplementation of the approach outperforms results reported in their paper. it tends to perform worse than the baseline for the datasets that had higher agreement. our hypothesis when we began our study was that multi-tasking would outperform other methods, as it allocates a larger number of dedicated parameters to learning representations of each annotator. we expected to see a trade-off in the model complexity and annotator modeling performance, with multi-tasking being the highest on both. lastly, it is important to note that our analysis is focused on the correlations between performance and surface-level features. this is to provide initial exploratory insight into an area that deserves much further analysis and experimentation. corpora have idiosyncrasies and future work can explore how to measure such qualities of dataset construction to support a more human-centric and personalized approach to annotator modeling, rather than one that abstracts away from these qualities. ## conclusion as research on subjective tasks in natural language processing has grown, the importance of modeling annotators and minority opinions has become more apparent. with the recent growth in work on annotator modeling and data perspectivism, it is becoming important to understand how the properties of our datasets and methods impact the effectiveness of annotator modeling methods. we examined seven corpora to better understand recent methods and introduced our method, the composite embeddings. we found that when annotator agreement on a dataset is low (k-α < 0.4), the user token embedding was most effective. when the annotator agreement was higher (k-α > 0.5), our new composite embedding gave the best performance. surprisingly, the user token and composite embeddings, which are simple and efficient to implement, outperformed the multi-tasking model that was the highest performing model in prior work. importantly, we also note that the number of annotations per annotator is correlated much more strongly with performance improvements than other corpus statistics, suggesting that this should be an area of focus for those constructing new datasets and collecting annotations. our code and collection of over 3k trial experiments statistics are publicly available to support further work in this area.