M c M aster at L e W i D i-2025: Demographic-Aware R o BERT a (2025)

Abstract: We present our submission to the Learning With Disagreements (LeWiDi) 2025 shared task. Our team implemented a variety of BERT-based models that encode annotator meta-data in combination with text to predict soft-label distributions and individual annotator labels. We show across four tasks that a combination of demographic factors leads to improved performance, however through ablations across all demographic variables we find that in some cases, a single variable performs best. Our approach placed 4th in the overall competition.

## introduction the shift in natural language processing toward more perspectivist approaches has been positive, in that it allows us to incorporate a variety of viewpoints for subjective tasks and construct models that are more aligned with, and useful for individuals. the number of available disaggregated corpora is small but growing, allowing us to test more techniques in annotator modeling. while the number of available corpora has increased, the amount and type of meta-data about annotators has not significantly changed. sociodemographic variables are sometimes collected with annotations for analysis or modeling of the annotators. without this information, we are often left with only the set of annotations themselves from which to learn patterns. while these demographic variables are not sufficient to represent people or populations and their diverse viewpoints, they give us a starting point to building annotator models that can be expanded in future work as more relevant information becomes available. with this in mind, we developed a demographicaware roberta model for the shared task competition. we chose roberta as the transformer of choice as it is well established, and well finetuned † denotes equal contribution. ## background the learning with disagreements shared task is motivated by recent efforts in annotator modeling, pluralistic alignment, and data perspectivism. we first describe work along these directions and follow this with an in-depth description of the shared task. ## system description we fined tuned roberta-large @xcite to develop a general model and apply this to all datasets through finetuning. the model architecture consisted of many different layers. this model encodes a variable-length text sequence as input and produces embeddings of each token, and a sequence embedding, represented with the [cls] token. for all datasets we used all available demographics. we embedded these demographics as follows. to simplify age, we binned the ranges into groups of: 18 -24, 25 -35, 35 -44, 45 -55, 55+, for the datasets that provided age. other demographic variables had predefined sets of categorical values from their original work. these are further listed in the demographic breakdowns for each dataset in the appendix. for each field, a learnable embedding matrix is created, and the text embedding and the demographic embedding are concatenated into a single feature vector. this vector is then normalized using layernorm, regularized with dropout and also passed through a linear classifier to produce the logits for classification. the mp and csc corpora had many annotators with no instances annotated by all annotators, whereas the par and varerr nli datasets each had only four annotators who annotated all instances. this allows for a slightly different approach. for the first two corpora, we predict each annotators label individually and aggregate them afterward to compute evaluation metrics. in this case, there are no parameters that are specifically designated to any individual annotator. for the latter two corpora, we take a different approach, predicting all anno-tator labels at the same time. this is similar to the multi-label model described in mostafazadeh @xcite . this approach is tractable due to the very small number of annotators in these corpora. the training time substantially increases with the number of annotators. we also compared our approach to the mistral-7b model @xcite . this is a large language model shown to outperform similar sized models across reasoning, mathematics, and code generation tasks using several recent optimization techniques. this is an instruction-tuned model that is more receptive to prompting. ## discussion we noted that the llm performance was substantially worse than the roberta-based models. it is possible that the llm could perform better with more effort put into prompt-tuning, though this remains to be shown. the added computational overhead and tuning efforts pose barriers to their practical use, over much more readily high-performing, and smaller bert-based models. the much smaller roberta models were successful in this task, placing high on the leaderboard and showing greater improvement in our subsequent ablation experiments. where a person is from, which is partially covered by four different demographic variables, appeared to have the strongest effect. as participants in the studies which collected the four datasets come from many different countries (see appendix for details), it makes sense that this would be a variable that correlates strongly with differences in viewpoints or opinion. a pca plot of the embeddings learned by our best roberta model is shown in figure 1 , showing some regional clusters. @xcite found that of the datasets for this shared task, varierr nli had the lowest annotator agreement measured by krippendorff's alpha, α = 0.06, while par agreement was α = 0.09, mp α = 0.26 and csc α = 0.34. the low agreement for varierr nli, coupled with the low number of annotators may contribute to our lower performance on this task. as noted in previous work, it is important to emphasize that demographics do not and cannot tell the full story @xcite . given the historical context in which data as been collected and annotated for building nlp models, it is often the case that no meta-data is available for annotators, and when data is available it is often in the form of a handful of demographic variables. this provides us a rough starting point for beginning to explore annotator modeling, but future work must find ways to gather or infer more individual annotation patterns or those that do not directly align with sociodemographic factors. ## conclusion we developed a demographic-aware roberta model for annotator modeling on four tasks, includ-ing irony detection, sarcasm detection, paraphrase detection, and nli. we found that our model could outperform baselines including a large language model; mistral-7b. in an ablation of demographic factors, we found that nationality and ethnicity led to the biggest performance increases. we note that although demographics provide a starting point to exploring annotator modeling approaches, more individualized approaches will be needed to fully capture differences in annotation patterns.