Accurate and Data-Efficient Toxicity Prediction when Annotators Disagree (2024)

Abstract: When annotators disagree, predicting the labels given by individual annotators can capture nuances overlooked by traditional label aggregation. We introduce three approaches to predict individual annotator ratings on the toxicity of text by incorporating individual annotator-specific information: a neural collaborative filtering (NCF) approach, an in-context learning (ICL) approach, and an intermediate embedding-based architecture. We also study the utility of demographic information for rating prediction. NCF showed limited utility; however, integrating annotator history, demographics, and survey information permits both the embedding-based architecture and ICL to substantially improve prediction accuracy, with the embedding-based architecture outperforming the other methods. We also find that, if demographics are predicted from survey information, using these imputed demographics as features performs comparably to using true demographic data. This suggests that demographics may not provide substantial information for modeling ratings beyond what is captured in survey responses. Our findings raise considerations about the relative utility of different types of annotator information and provide new approaches for modeling annotators in subjective NLP tasks.

## introduction disagreement among data annotators can reveal nuances in nlp tasks that lack a simple ground truth, such as hate speech detection. for instance, what one group of annotators deems acceptable might be considered offensive by another. the current standard for resolving such disagreement, aggregation via majority voting, casts aside variance in annotator labels as noise, when in subjective tasks this variance is key to understanding the perspectives that arise from the annotators' individuality and backgrounds. to address this problem, recent research has explored alternatives to majority voting. most notably, studies have taken the approach of predicting the ratings of individual annotators @xcite @xcite . we aim to improve the prediction of rating behavior, guided by the following questions: • does incorporating annotator information via collaborative filtering, embedding-based architecture, or in-context learning improve downstream rating predictions? • what annotator information best informs toxicity rating predictions? do demographics provide useful information beyond what survey information can provide? we proposed and tested a neural collaborative filtering (ncf) module, an embedding-based architecture, and an in-context learning (icl) module for individual rating prediction. first, we incorporated ncf to the classification head of a robertabased model @xcite . embedded annotator information foot_0 was combined with a separate embedding of annotators' rating history to predict individual annotator toxicity ratings. secondly, we used embedding models to encode annotator information, then performed regression to predict toxicity ratings. lastly, we prompted llms such as mistral @xcite and gpt-3.5 @xcite to study different ways of integrating annotator information. our findings indicate that while ncf does not outperform baseline models, icl and our embedding-based architecture improve performance, with the embedding-based architecture significantly outperforming all other approaches tested. in addition, our research on the effectiveness of demographic information as a feature indicates that imputing demographics from survey data performs similarly to using direct demographic inputs, suggesting that survey responses already capture the relevant demographic information for rating prediction. this suggests that, on this task, demographics have little predictive power beyond what survey information provides. ## motivation and related work our work is fundamentally motivated by the need for alternatives to majority-vote label aggregation in nlp tasks. @xcite find that disagreement among annotators is partially attributed to differences in human judgment. @xcite underscore the importance of the consideration of a system's output over instances where annotators disagree. newer work in this field aims to directly model individual annotator rating behavior. davani et al. (2022) employ a multi-task based approach, where predicting each annotators' judgment is a subtask to their larger architecture. @xcite use a roberta-based model to predict an individual annotators' ratings. @xcite put together a jury of annotators, predicting individual judgments. for the individual annotator rating prediction task, @xcite create individual annotator embeddings and annotation embeddings. this idea of learning embeddings based on user-specific data has been applied in various domains successfully, e.g., imitation learning @xcite or recommendation systems @xcite . collaborative filtering (cf) learns user embeddings based on their past behaviors @xcite . @xcite show that neural collaborative filtering (ncf) offers better performance than more naive cf implementations. this motivates our ncf approach to learning annotator embeddings. intuitively, this approach would be effective in learning deeply rooted preferences and behaviors of annotators. thus, we hypothesized that this method would more accurately predict individual annotator ratings. several recent approaches use sociodemographic traits of individual annotators to learn for the rating prediction task @xcite , but @xcite warn that legal and organizational constraints, such as privacy laws and concerns around self-reporting, often make collecting demographic data challenging. @xcite suggest using semantically related fea-tures in the absence of sensitive demographic data. for instance, in the absence of gender information, @xcite use other demographic features -age, relation, and marital status -for their prediction task. this work motivates our objective of incorporating auxiliary annotator information (survey information and annotator history) in the prediction task. lastly, @xcite challenge the utility of demographic information, since they do not find strong evidence that explicitly modeling demographics helps to predict annotation behavior. in concurrent work, @xcite argue that there is an inherent limit to how much predictive power can be provided by demographics. their findings indicate that while incorporating demographic variables can provide modest improvements in prediction accuracy, these gains are often constrained by the relatively low variance explained by these variables. this motivates our final objective, studying the efficacy of demographics as a useful mediating variable for rating prediction. ## approach our approach includes creating three separate modules based on neural collaborative filtering (ncf), an embedding-based architecture, and in-context learning (icl). we evaluate each approach's efficacy in predicting annotator rating behavior. the latter two modules are used to investigate our second research question; we integrate different ablations of annotator information as input to the rating prediction models to study their effect on toxicity rating prediction. we used @xcite 's dataset to evaluate the performance of our rating prediction modules. this dataset consists of sentences rated for toxicity (0 = least toxic, 4 = most toxic). each sentence has been labeled by 5 annotators and each annotator has labeled 20 distinct sentences. for each annotator, the dataset contains their rating behavior; demographic information (race, gender, importance of religion, lgbt status, education, parental status, and political stance); and survey information, e.g., their preferred forums, social media, whether they have seen toxic content, if they think toxic content is a problem, and their opinion on whether technology impacts peoples' lives. for ablations, we took distinct combinations of annotator information (rating history, demographics, survey information) along with the text to be rated, assessing the impact of each on the model's performance. to study whether demographics are a necessary feature for predicting annotator ratings, we also used a separate model to predict annotator demographics using rating history and survey information and applied these predicted demographics as input for our ablations. for all three methods, we used mean absolute error (mae) of predicting individual annotators' ratings as the evaluation metric, allowing us to quantify the performance of different model configurations. ## results our three approaches predicted annotators' toxicity ratings on a scale from 0 to 4, based on both textual data and various combinations of annotator-specific information (demographics, survey responses, rating history). we also examine how well these models handle predicted demographic data rather than using the ground truth demographic values for each annotator. this helps to assess the data efficiency and effect of demographics as an input to the rating prediction task. for our ablations that studied the improvement on rating predictions, we compared our results to previous baselines that predicted ratings of annotators using the same dataset. q1: does incorporating annotator information via collaborative filtering, the embeddingbased architecture, or in-context learning improve downstream rating predictions? our embedding-based architecture outperformed all other experiments with an mae of 0.61; the best icl approach (with mistral) reached an mae of 0.69. both the icl approach and embedding-based architecture outperform the most recent baseline for the dataset @xcite and the embedding-based architecture matches the best previous mae on this dataset @xcite . the best-performing models use all available annotator-specific information as input (annotator demographics, survey information, and historical rating data). at its best, our icl configuration with mistral had an mae of 0.69 (using annotator demographics, survey information, and historical rating data). the ncf approach had consistently poorer results, with a best mae of 0.79 when including all annotator-specific information. when creating the ncf architecture, we tested several variations. we first created a baseline from which we compared different outputs of our ncf module. evaluating the finetuned roberta model with all annotator-specific information as input along with the text to be rated yielded a baseline mae of 0.81. we experimented with integrating embeddings through dot product vs. concatenation, freezing roberta during the training process, and placing the collaborative filtering task in different parts of the roberta architecture. our best performing model froze the pretrained roberta model, used concatenation, and placed the collaborative filtering piece in the classification head. however, it was only able to achieve an mae of 0.80, not significantly improving on our baseline. our embedding-based architecture consistently outperformed other approaches on every ablation, suggesting that a feature-extraction and regression hybrid approach most effectively uses annotatorspecific information in rating predictions. ## conclusion leveraging the embedding-based architecture and icl methods substantially improved toxicity rating predictions. ncf, by contrast, was not a competitive method for predicting ratings. incorporating annotator information significantly enhances model performance. the best-performing embeddingbased architecture achieved the lowest mae of 0.61 by integrating demographics, annotator history, and survey data. this suggests that personalized predictions based on individual annotator preferences can lead to more accurate outcomes. meanwhile, the ability to predict some demographics from survey information, and the fact that these imputed demographics nearly match performance with the true demographics, suggest that although demographics are helpful, individual annotator ratings can be predicted effectively without demographic data. this finding suggests that some differences in annotator opinions may be best captured by modeling individual preferences rather than demographic trends. in addition, the effectiveness of our embedding-based architecture suggests that it could help to inform future frameworks for annotator rating prediction.