L e W i D i-2025 at NLP erspectives: The Third Edition of the Learning with Disagreements Shared Task (2025)

Abstract: Many researchers have reached the conclusion that ai models should be trained to be aware of the possibility of variation and disagreement in human judgments, and evaluated as per their ability to recognize such variation. The LeWiDi series of shared tasks on Learning With Disagreements was established to promote this approach to training and evaluating ai models, by making suitable datasets more accessible and by developing evaluation methods. The third edition of the task builds on this goal by extending the LeWiDi benchmark to four datasets spanning paraphrase identification, irony detection, sarcasm detection, and natural language inference, with labeling schemes that include not only categorical judgments as in previous editions, but ordinal judgments as well. Another novelty is that we adopt two complementary paradigms to evaluate disagreement-aware systems: the soft-label approach, in which models predict population-level distributions of judgments, and the perspectivist approach, in which models predict the interpretations of individual annotators. Crucially, we moved beyond standard metrics such as cross-entropy, and tested new evaluation metrics for the two paradigms. The task attracted diverse participation, and the results provide insights into the strengths and limitations of methods to modeling variation. Together, these contributions strengthen LeWiDi as a framework and provide new resources, benchmarks, and findings to support the development of disagreement-aware technologies.

## introduction the assumption that natural language (nl) expressions have a unique and clearly identifiable interpretation has been recognized in ai as just a convenient idealization for over twenty years @xcite @xcite @xcite @xcite @xcite . more recently, the increasing focus in nlp on tasks depending on subjective judgments @xcite @xcite @xcite @xcite led to the realization that in many nlp tasks the traditional approach to dealing with disagreement of 'reconciling' different subjective interpretations is not tenable. many ai researchers concluded therefore that rather than eliminating disagreements from annotated corpora, we should preserve them (e.g. @xcite @xcite @xcite @xcite . as a result, a number of corpora with these characteristics now exist, and more are created every year @xcite @xcite @xcite @xcite @xcite @xcite @xcite . much recent research has therefore investigated whether corpora of this type are also useful resources for training nlp models, and if so, what is the best way for exploiting disagreements @xcite @xcite @xcite @xcite . this research in turn led to questions about how such models can be evaluated @xcite @xcite @xcite . a succinct overview of the literature on how the problem affects data, modeling and evaluation in nlp is given in @xcite , and an extensive survey can be found in @xcite . such research also led to the establishment of the learning with disagreements (lewidi) shared tasks. the first edition, organized at semeval 2021 task 12 @xcite , introduced the idea of providing a unified testing framework for modeling disagreement and evaluating systems on such data. the benchmark combined six widely used corpora spanning semantic and inference tasks as well as image classification tasks. while the resource attracted considerable attention (the benchmark was downloaded by more than 100 teams worldwide) participation in the evaluation was limited, possibly due to the difficulty of the provided baselines or the need for expertise in both nlp and computer vision. in addition, the benchmark only covered a single subjective task, i.e., humour detection, @xcite , and a single language (english). a second edition followed at semeval 2023 @xcite , designed to address these limitations and to better reflect the growing interest in subjective nlp tasks. in contrast to the first edition, all datasets were textual and the focus shifted entirely to inherently subjective phenomena such as misogyny, hate-speech and offensiveness detection, where training with aggregated labels makes much less sense. moreover, arabic was added as a second language. finally, evaluation combined the soft-label approach also used in the first edition, based on cross-entropy, with the more traditional f1 metric. the reformulated task attracted broad interest in the community: more than 130 groups registered, with 30 submitting predictions and 13 contributing system papers. the third edition of the lewidi shared task, described in this manuscript and co-located with the nlperspectives workshop at emnlp 2025, builds on these experiences while further broadening the scope of the task. like the earlier editions, its central goal is to provide a common evaluation framework for systems trained on disagreement-rich data. however, lewidi-2025 introduces several innovations. new tasks include natural language inference (nli), irony detection, conversational sarcasm detection, and paraphrase detection. on the evaluation side, we move entirely to soft metrics, which are organized into two complementary tasks: (i) soft-label evaluation, refining methods from lewidi 2 with several distance-based metrics (e.g., manhattan distance, @xcite ; and (ii) perspectivist evaluation, where systems must model the labeling behavior of individual annotators, again with newly developed metrics tailored to this setting. in addition, two of the datasets adopt likert-scale annotation, posing further challenges for evaluation. lewidi 3 engaged a smaller but dedicated group of participants relative to the previous edition. a total of 53 individuals registered on the competition platform, with 15 teams providing submissions, which resulted in 9 system papers. ## the lewidi 3 benchmark the four selected datasets are summarized in all datasets were released in a harmonized json format, identical to that of the previous lewidi edition, to ensure consistent access across datasets and shared tasks editions. each item contains the same fields, foot_0 while the field other info is dataset-specific and includes additional subfields particular to each dataset. annotator age and gender is available for all four datasets, with some datasets providing further attributes. this metadata was distributed separately in an additional json file. all datasets are publicly available. foot_1 ## task definition the main goal of the shared task is to provide a unified testing framework for learning from disagreements and evaluating models on such datasets. given the heterogeneous nature of the datasets, participants were free to design dataset-specific approaches; however, they were encouraged to adopt a unified crowd learning methodology or framework across all datasets, rather than optimizing a separate best-performing model for each dataset. ## evaluation metrics two complementary paradigms for disagreement evaluation were employed in lewidi-2025: softlabel and perspectivist evaluation. ## participating systems the third edition of the lewidi shared task attracted a smaller but more focused community compared to the previous edition. in total, 53 people subscribed to the competition codabench, and 15 teams submitted predictions. among them, 6 teams participated across all datasets and both tasks; 2 teams submitted for three datasets and both tasks (excluding ven); and 5 teams focused on a single dataset with submissions only for task a. in terms of system papers, 9 were submitted: 6 from teams who participated in multiple tasks and datasets, and 2 from teams who worked on a single dataset and task a. task a was overall more popular, as the majority of teams who submitted exclusively for one dataset contributed only to task a, while 11 teams engaged also with task b. ## results and discussion this section presents the official results of the shared task and discusses key trends across systems and datasets. we also examine the role of evaluation metrics and summarize insights from ablation studies conducted by participating teams. ## conclusions we are delighted that the third edition of the lewidi shared task continued to attract the attention of the community researching disagreement and variation in nlp. again, we found that the participating teams engaged actively with the tasks, tackling interesting issues such as how best to use annotator information and the relation between soft-label modelling and perspectivist modelling. our hope is that the shared task and the datasets we released will stimulate further research in this area, by the participant groups and others. we believe that further thinking is still needed on issues such as the most appropriate form of evaluation for tasks in which human subjects express ordinal judgments, or the usefulness of modelling individual annotators or groups of annotators. to promote this, the codabench page will remain open to submissions after the deadline so that researchers can continue test their models on the datasets.