University at Buffalo at S em E val-2023 Task 11: MASDA –Modelling Annotator Sensibilities through D is A ggregation (2023)

Abstract: Modeling the most likely label when an annotation task is perspective-dependent discards relevant sources of variation that come from the annotators themselves. We present three approaches to modeling the controversiality of a particular text. First, we explicitly represented annotators using annotator embeddings to predict the training signals of each annotator’s selections in addition to a majority class label. This method leads to reduction in error relative to models without these features, allowing the overall result to influence the weights of each annotator on the final prediction. In a second set of experiments, annotators were not modeled individually but instead annotator judgments were combined in a pairwise fashion that allowed us to implicitly combine annotators. Overall, we found that aggregating and explicitly comparing annotators’ responses to a static document representation produced high-quality predictions in all datasets, though some systems struggle to account for large or variable numbers of annotators.

## introduction the primary challenge of accommodating disagreement in natural language processing is in aggregating (or disaggregating) the decisions of individual annotators. as part of this submission to lewidi23 @xcite , we present several possible approaches, deployed to varying degrees of success, for modeling annotators individually or in parallel. building on the general idea that individual annotators largely process language in similar ways, minus differences of political opinion or life experience, we create features that allow us to compare and contrast annotators to each other directly. our first approach creates annotator layers that represent individual annotators using smalldimensionality embeddings. the decisions of the annotators (their personal label) then impact the final (gold) label. these separate networks enable the joint prediction of each annotator while considering the sentence embedding separately. we present two ablation studies showing that integrating the features in this way is critical for performance of the system. ## approach 1: annotator layers and document embeddings the model consists of a pretrained sentence transformer coupled with a classification head (see figure 1 ). we use this template for the hs-brexit @xcite and armis @xcite datasets, each modified to account for the main differences in the number of annotators and the language -we refer interested readers to the appendix for dataset-specific modifications to this architecture. the model predicts a soft label for a given input, which is defined as the average of the predicted labels for each individual annotator (i.e. ensemble averaging; @xcite . aach annotator for the dataset is represented by a unique linear layer (annnet), which produces an embedding from the sentence transformer that reflects that annotator's likely decision on the input text as a proxy of their viewpoint on the topic. the final (sigmoid) classification layer (decnet) produces a predicted label from each annotator embedding, which is shared by all of the annotator layers. to predict the hard label, an additional layer (goldnet) produces a weighted sum of the annotator embeddings in the form of attention weights. these annotator attention weights are learned over the sentence transformer output. this weighted sum of the goldnet is then passed to the decnet to produce the predicted hard label. the goldnet attention mechanism therefore identifies which annotators' labels align with the gold/hard labels on different kinds of sentence inputs. note that the collection of annotator networks is similar to the crowd layer introduced by @xcite ; the unique contribution of our architecture lies in the goldnet aggregation layer and the parameter-sharing implemented by the dec-net layer. ## approach 2: topic infusion this method is used primarily in the md-agreement @xcite dataset. we introduce the polarizability metric which reflects the ability of a tweet to divide annotator opinion (see equation 3 ). our implementation of the polarizability metric is based on the polarization measure discussed in @xcite . however, the original metric relies on a sophisticated model of group dynamics, which can be challenging to obtain without significant data and expertise. to mitigate this issue, we have designed our polarizability metric to be a plug-and-play feature, which can be easily replaced with other implementations. while we acknowledge the potential benefits of incorporating a more accurate model of group dynamics, we leave this as an area for future exploration. the prop(pos) and prop(neg) indicate the fraction of annotators that assigned a positive and negative label respectively, resulting in a value that ranged from 0 (unanimity) to 1 (an even split). we found that different tweets topics varied in polarizability (see ## approach 3: annotator interaction modeling the conversations in the convabuse @xcite dataset consisted of two parts, the history and the current turn. somewhat mirroring our approach in topic infusion, we separate these two segments using the "[sep]" token (refer figure 2 ). notably, we retained the markers for the conversation agent and user as we believe that these markers will enhance the expressiveness of the conversation embeddings. we had a limited pool of 8 annotators, with each conversation having at least 3 annotators giving us confidence in our ability to reasonably model their sensitivities when it comes to detecting offensive content. we modeled how each annotator would respond to the conversation using annotator embeddings built upon a paraphrase model @xcite . further, to maintain continuity we optimize the kl-divergence objective in this approach as well. we detail the modeling below. ## discussion as part of this shared task, we leveraged the ability for transformer-based neural networks to encode annotators' perspectives somewhat independently from the "gold" label associated with a controversial text. we find that different methods of annotator representation show significant variability in their effectiveness. ultimately, finding an approach that is as universal as possible should be the main focus of future work.