S em E val-2023 Task 11: Learning with Disagreements ( L e W i D i) (2023)

Abstract: NLP datasets annotated with human judgments are rife with disagreements between the judges. This is especially true for tasks depending on subjective judgments such as sentiment analysis or offensive language detection. Particularly in these latter cases, the NLP community has come to realize that the common approach of reconciling’ these different subjective interpretations risks misrepresenting the evidence. Many NLP researchers have therefore concluded that rather than eliminating disagreements from annotated corpora, we should preserve themindeed, some argue that corpora should aim to preserve all interpretations produced by annotators. But this approach to corpus creation for NLP has not yet been widely accepted. The objective of the Le-Wi-Di series of shared tasks is to promote this approach to developing NLP models by providing a unified framework for training and evaluating with such datasets. We report on the second such shared task, which differs from the first edition in three crucial respects: (i) it focuses entirely on NLP, instead of both NLP and computer vision tasks in its first edition; (ii) it focuses on subjective tasks, instead of covering different types of disagreements as training with aggregated labels for subjective NLP tasks is in effect a misrepresentation of the data; and (iii) for the evaluation, we concentrated on soft approaches to evaluation. This second edition of Le-Wi-Di attracted a wide array of partici- pants resulting in 13 shared task submission papers.

## introduction in recent years, the assumption that natural language (nl) expressions have a single and clearly identifiable interpretation in a given context is increasingly recognized as just a convenient idealization. virtually every large-scale annotation project in the last twenty years has found widespread evi-dence of disagreement @xcite @xcite @xcite @xcite . plenty of evidence has also been found of disagreements over the inferences that underlie much of semantic interpretation @xcite . more recently, the increasing focus in nlp on tasks depending on subjective judgments such as sentiment analysis @xcite , humour and sarcasm identification @xcite and offensive language detection (cercas @xcite @xcite led to the realization that in such tasks 'reconciling' different subjective interpretations does not make much sense @xcite . many ai researchers concluded therefore that rather than eliminating disagreements from annotated corpora, we should preserve them-indeed, some researchers have argued that corpora should aim to preserve all interpretations produced by annotators (e.g. @xcite @xcite @xcite @xcite . as a result, a number of corpora with these characteristics now exist, and more are created every year @xcite @xcite @xcite @xcite . yet this approach to corpus creation is still not widely adopted. one reason for this is that the usefulness of such resources has been questioned. in response, much recent research has investigated whether corpora of this type, besides being more accurate characterizations of the linguistic reality of language interpretation, are also useful resources for training nlp models, and if so, what is the best way for exploiting disagreements. for example, @xcite used information about disagreements to exclude items on which judgments are unclear @xcite . in crowdtruth @xcite , information about disagreement is used to weigh the items used for training. @xcite proposed to use the information about disagreement to supplement gold labels during training. finally, a number of methods for training directly from data with disagreements without first obtaining an aggregated label, have been proposed @xcite @xcite @xcite . this research in turn led to questions about how such models can be evaluated @xcite @xcite . a succinct overview of the literature on how the problem affects data, modeling and evaluation in nlp is given in @xcite , and an extensive survey can be found in @xcite . the considerations above led to the proposal of the first semeval shared task on learning with disagreement (lewidi) @xcite . the objective was to provide a unified testing framework for learning from disagreements, and evaluating with such datasets. lewidi 2021 created a benchmark consisting of 6 existing and widely used corpora containing information about semantic and inference disagreements and classifying images which raised a great deal of interest in the community-the benchmark was downloaded by more than 100 research groups worldwide-but in the end very few groups submitted runs for the evaluation phase, perhaps because the state of the art systems developed for @xcite proved too hard a baseline, or maybe due to the need for expertise in both nlp and computer vision. moreover, the prior benchmark only covered one subjective task, humour detection @xcite , and only one language, english. since then, there has been an increased interest in learning from disagreements with several new models and datasets published over the past year (e.g. @xcite @xcite . we therefore proposed a second shared task on learning with disagreement, which differs from the first edition in two crucial respects: (i) it focuses entirely on subjective tasks, where training with aggregated labels is obviously inappropriate. the datasets are entirely new, and many not yet publicly available; and we include arabic as a non-english language. (ii) for evaluation, we concentrate on soft approaches to evaluation @xcite . we believe that a shared task thus reformulated is extremely timely, given the current high degree of interest in subjective tasks such as offensive language detection, and in particular given the growing interest in research on the issue of disagreements in such data @xcite @xcite @xcite @xcite . this belief was supported by the much greater participation in this second edition, with over 130 groups registered, 29 submitting a run and 13 system description papers. ## task organization in order to provide a thorough benchmark, we employed four textual datasets of subjective tasks, all characterized by providing a multiplicity of labels for each instance and by having a size sufficient to train state-of-the-art models. both hard and soft evaluation metrics were employed @xcite , but with the priority given to soft metrics. in this section, we briefly introduce the datasets and our evaluation criteria. we also elaborate on the setup of the shared task. ## participating systems the lewidi 2 edition of the shared task was much more successful than the first one @xcite . more then 130 different teams subscribed to our competition page and 30 teams participated in the evaluation phase of our task submitting their predictions for at least a dataset. the majority of the teams (21) submitted predictions for all datasets, while one team submitted for three datasets and two teams for two datasets. a few teams (6) adopted a different strategy submitting their predictions for only one dataset. among the participants in the evaluation phase, 13 teams submitted a system paper @xcite @xcite @xcite @xcite @xcite @xcite . moreover, after the competition ended, the participants were asked to fill a short survey to provide key information about their strategies and systems. thus, in total (between surveys and system papers) we have retrieved information about the work of 18 teams. ## conclusions in this shared task we proposed a framework for learning from disagreements between multiple annotators across a variety of datasets and subjective tasks. we evaluated the ability to predict the distribution of different interpretations of the annotators so to include different point of views. we are delighted that this second lewidi shared task attracted the attention of a substantial number of research groups, and that the participants engaged so actively with the issues raised by the datasets, coming up with such creative solutions. we believe that proposing the same task on a multiplicity of datasets challenged participants and promoted constructive and fruitful thinking as it forced visualisation of the problem in its entirety rather then focusing on the solution to a specific dataset. one hope is that the shared task and the datasets we released will stimulate further research in this area, by the participant groups and others. to promote this, the codalab page will remain open to submissions after the deadline so that researchers can continue test their models on the datasets. we believe for instance that further thinking is still needed on issues such as the most appropriate form of evaluation for tasks in which human subjects frequently disagree, or the usefulness of modelling individual annotators or groups of annotators.