NLP - R es T eam at L e W i D i-2025:Performance Shifts in Perspective Aware Models based on Evaluation Metrics (2025)

Abstract: Recent works in Natural Language Processing have focused on developing methods to model annotator perspectives within subjective datasets, aiming to capture opinion diversity. This has led to the development of various approaches that learn from disaggregated labels, leading to the question of what factors most influence the performance of these models. While dataset characteristics are a critical factor, the choice of evaluation metric is equally crucial, especially given the fluid and evolving concept of perspectivism. A model considered state-of-the-art under one evaluation scheme may not maintain its top-tier status when assessed with a different set of metrics, highlighting a potential challenge between model performance and the evaluation framework. This paper presents a performance analysis of annotator modeling approaches using the evaluation metrics of the 2025 Learning With Disagreement (LeWiDi) shared task and additional metrics. We evaluate five annotator-aware models under the same configurations. Our findings demonstrate a significant metric-induced shift in model rankings. Across four datasets, no single annotator modeling approach consistently outperformed others using a single metric, revealing that the “best” model is highly dependent on the chosen evaluation metric. This study systematically shows that evaluation metrics are not agnostic in the context of perspectivist model assessment.

## introduction the primary aim of perspectivism in (nlp) is to preserve and leverage the diverse, subjective decisions of individual annotators, both in the modeling process and in the subsequent evaluation of those models @xcite . given the variety of annotator representation methods, a key challenge lies in how to effectively incorporate annotator-specific information during model training to capture these unique perspectives @xcite . the efficacy of such annotator modeling techniques is influenced by several critical factors. a foundational element is the annotation paradigm used to create the dataset @xcite . furthermore, the performance is heavily dependent on the dataset's statistical properties, including the number of training instances required to reliably model an annotator, the volume of annotations per annotator, the degree of inter-annotator agreement (iaa), and the number of annotations per instance. @xcite showed that the number of contributions from an annotator and the iaa are particularly crucial statistics to consider. while existing approaches capture annotator diversity to varying extents, their evaluation has predominantly relied on conventional metrics like the f1-score @xcite @xcite @xcite and in some cases cross entropy, especially for soft label prediction @xcite . it has been argued, however, that such metrics are insufficient as they often collapse multiple valid perspectives into a single ground truth, failing to truly reflect the goals of a perspectivist evaluation @xcite . as part of our submission to lewidi 2025, we present a comparative study of different annotator modeling approaches. we analyze how their performance shifts when assessed using a range of evaluation metrics, including those provided by the organizers. our aim is to advance a more nuanced view within the perspectivist framework. we hypothesize that the performance of a given modeling approach is not absolute but is contingent upon the evaluation metric used. a model that is best performing under one metric may not perform as well under another, especially when applied to datasets with different underlying statistical properties and task natures. to investigate this, we implemented five distinct modeling approaches and evaluated them on the perspectivist subtask (b) using additional evaluation metrics. ## background and summary one of the primary challenges of the 2025 edition of the lewidi shared task is the two concurrent tasks designed to model and evaluate variations in annotations @xcite . task a, the soft label approach, focused on predicting the probability distribution of labels for each instance and task b, the perspectivist approach, focused on predicting the individual label assigned by each annotator. the organizers introduced four new datasets and adopted a tailored evaluation framework for each, rather than relying on a single unifying metric. the conversational sarcasm corpus (csc) @xcite , consists of context-response pairs rated for sarcasm on a likert scale from 1 to 6, with soft label evaluation based on wasserstein distance and perspectivist evaluation based on mean absolute distance (mad). the multipico (mp) dataset @xcite is a crowdsourced multilingual irony detection resource containing post-reply pairs from twitter and reddit, annotated with binary labels across 11 languages. the datasets also contained annotator metadata such as gender, age, nationality, and student or employment status. evaluation for the soft label task used manhattan distance, while the perspectivist task used the error rate. the paraphrase detection (par) dataset (mainlp lab, 2025) contains question pairs collected from quora and annotated on a likert scale from -5 to +5, with each annotator providing a brief explanation for their score, as in csc, evaluation for the soft task used wasserstein distance and for the perspectivist task used mad. finally, the varierrnli dataset (weber-genzel et al., 2024) was designed for error detection by distinguishing between annotation mistakes and legitimate human label variation in natural language inference; it includes both labels and annotator explanations and was evaluated using the same metrics as the mp dataset. in this study, we used the official training and validation splits provided by the organizers, and our final models performed inference on the unlabeled test sets. the dataset statistics are presented in