S em E val-2021 Task 12: Learning with Disagreements (2021)

Abstract: Disagreement between coders is ubiquitous in virtually all datasets annotated with human judgements in both natural language processing and computer vision. However, most supervised machine learning methods assume that a single preferred interpretation exists for each item, which is at best an idealization. The aim of the SemEval-2021 shared task on learning with disagreements (Le-Wi-Di) was to provide a unified testing framework for methods for learning from data containing multiple and possibly contradictory annotations covering the best-known datasets containing information about disagreements for interpreting language and classifying images. In this paper we describe the shared task and its results.

## introduction the assumption that natural language (nl) expressions have a single and clearly identifiable interpretation in a given context, or that images have a preferred labels, still underlies most work in nlp and computer vision. however, there is now plenty of evidence that this assumption is just a convenient idealization; virtually every project devoted to large-scale annotation has found that genuine disagreements are widespread. in nlp, that annotator/coder disagreement can be genuine-i.e., resulting from debatable, difficult, or linguistic ambiguity-has long been known for anaphora and coreference @xcite @xcite . but in recent years, we have also seen evidence that disagreements among subjects/coders are common with virtually every aspect of language interpretation, from apparently simple aspects such as partof-speech tagging @xcite , to more see also the analysis of disagreements in ontonotes and word senses in @xcite , @xcite , and martínez alonso et al. @xcite . complex ones like semantic role assignment @xcite , to subjective tasks such as sentiment analysis @xcite , and to the inferences that can be drawn from sentences @xcite . in computer vision, as well, the assumption that gold labels may be specified for items has proven an idealization @xcite -in fact, possibly even more than for nlp. in many widely used crowdsourced datasets for computer vision, different coders assign equally plausible labels to the same items. the problem of disagreement among coders, including experts, on the classification of noisy image data has arisen in many cv applications. this includes classification of astronomical images @xcite , medical image classification @xcite , and numerous others @xcite @xcite . many ai researchers have concluded that rather than attempting to eliminate disagreements from annotated corpora, we should preserve them-indeed, some researchers have argued that corpora should aim to collect all distinct interpretations of an expression @xcite @xcite @xcite @xcite . @xcite and @xcite suggest that the best way to create resources capturing disagreements is by preserving implicit ambiguity-i.e., having multiple annotators label the items, and then keeping all these annotations, not just an aggregated 'gold standard'. a number of corpora with these characteristics now exist @xcite @xcite @xcite much recent research has explored the question of whether corpora of this type, besides being more accurate characterizations of the linguistic reality of language interpretation and image categorization, are also better resources for training nlp and computer vision models, and if so, what is the best way for exploiting disagreements in modeling. @xcite used information about disagreements to exclude items on which judgements are unclear ('hard' items). in the crowdtruth project @xcite information about disagreement is used to weigh the items used for training. @xcite proposed to use the information about disagreement to supplement the gold label during training. finally, methods were proposed for training directly from the data with disagreements, without first obtaining an aggregated label @xcite @xcite . only limited comparisons of these methods have been carried out @xcite , and the sparse research landscape remains fragmented; in particular, methods applied in cv have not yet been tested in nlp, and vice versa. the objective of semeval-2021 task 12, learning with disagreements (le-wi-di), was to provide a unified testing framework for learning from disagreements in nlp and cv using datasets containing information about disagreements for interpreting language and classifying images. the expectation being that unifying research on disagreement from different fields may lead to novel insights and impact ai widely. ## task organization in order to provide a thorough benchmark for methods for learning from disagreements, we identified five well-known datasets for very different nlp and cv tasks, all characterized by providing a multiplicity of labels for each instance, by having a size sufficient to train state-of-the-art models, and by evincing different characteristics in terms of the crowd annotators and data collection procedure. we found or developed near-state-of-the-art models for the tasks represented by these datasets. both 'hard' and 'soft' evaluation metrics were employed @xcite . the shared task was set up on the codalab competitions platform, which enables training and uniform evaluation on these datasets, such that the crowd learning adaptations of the base models proposed by participants to the task would be directly comparable. in this section, we briefly introduce the five datasets included in the benchmark and our evaluation criteria. we also elaborate on the setup of the shared task. ## base models and baselines in order to encourage the participants to focus on the development of methods for learning from disagreement, as opposed to achieving higher performance by developing better models, we provided 'base' models for each of the tasks represented by the aforementioned corpora. in this section, we briefly discuss the baseline models for each task that we provided. in section 5, we report the results using these base models and two crowd learning approaches: majority voting and the soft loss method @xcite . this proved unnecessary as the inherent difficulty of the shared task was enough of a deterrent. the pos tagging model. the pos tagger is a bilstm @xcite with additional use of attention over the input word and character embeddings, as used in @xcite . the pdis classification model. the model for this task was developed by comparing architectures from two models: a state-of-the-art coreference model and a state-of-the-art is classification model. we combined the mention representation component of @xcite coreference resolution system with the mention sorting and non-syntactic feature extraction components of the is classification model proposed by @xcite to create a novel is classification model that outperforms @xcite on the pdis corpus. the training parameters were set following @xcite . the humour preference learning model. we use as base model for this task gaussian process preference learning (gppl) with stochastic variational inference, as described and implemented by @xcite . as an input vector to gppl, we first take the mean word embedding of a text, using 300-dimensional word2vec embeddings trained on the google news corpus @xcite . then, we compute the frequency of each unigram in the text in a 2017 wikipedia dump, and each bigram in the text in a google books ngram dataset. finally, we concatenate the mean unigram and bigram frequencies with the mean word embedding vector to obtain the input vector representation for each short text. the gppl model is trained on pairwise labels from the training set to obtain a ranking function that can be used to score test instances or output pairwise label probabilities. as a bayesian model, it takes into account sparsity and noise in the crowdsourced training labels, and moderates its confidence accordingly. hence, it is a strong baseline for accounting for disagreement among annotators. this same gppl approach set the previous state of the art on the humour dataset @xcite . the labelme image classification model. for this task, we replicated the model from @xcite . the images were encoded using pretrained cnn layers of the vgg-16 deep neural network @xcite . this encoding is passed into a feed-forward neural network layer this model was developed for fine-grained information status classification on the isnotes corpus @xcite . with a relu activated hidden layer with 128 units. a 0.2 dropout is applied to this learned representation which is then passed through a final layer with softmax activation to produce the model's predictions. the cifar-10 image classification model. the trained model provided for this task is the resnet-34a model @xcite , a deep residual framework which is one of the best performing systems for the cifar-10 image classification. we made available to participants the publicly available pytorch implementation of this resnet model. ## participating systems unfortunately, we observed a dramatic difference in the number of participants that signed up to the competition (over 100 groups), the number of groups that participated in the trial phase, and the number of groups that submitted a run for official evaluation. only one group, uor, submitted in the evaluation phase @xcite . however, they did submit models for each of the tasks, and did adopt a learning from disagreements approach. pos tagging. for pos tagging, uor developed a novel pos tagging model by fine-tuning the bert language model @xcite . the (tweet, token) pairs were encoded in the form where the '[cls]' token was added for classification and the '[sep]' token separated the tweet from the token under consideration. to learn the class for the token, the learned classification token was passed through a single feed-forward neural network layer with softmax activation. the output of this layer represented the probabilities of the token belonging to each of the 12 classes. to extend this model for crowd learning, uor added an adaptation of the crowd layer from @xcite . rather than compute a single loss from the crowd layer as @xcite do, uor compute a joint loss from both the crowd layer and the base model (without the crowd layer bottleneck). https://github.com/kellerjordan/resnet-pytorchcifar10 two participating groups cited an inability to come up with a novel crowd learning paradigm as the reason they did not submit for official evaluation. ## results and discussion with the pos and pdis datasets, the model proposed by uor, adding a crowd layer on top of bert, achieved substantially worse results than training from a label aggregated using majority voting or training using a soft-loss function, both according to the hard evaluation metric (f 1 ) and the soft metric (ce). the ranking between soft-loss method, aggregation, and crowd layer with pos is consistent with that obtained by @xcite , but the results obtained by uor are much worse for reasons that will require further investigation. (with pdis, @xcite obtain comparable results with soft-loss functions and with the crowd layer.) more generally, the results show that although the hard label (the majority voting aggregate of the annotator distribution) and the soft label (a probability distribution encoding of the annotator distribution) were drawn from the same annotator distribution with this dataset, given the same base model, training by targeting the soft label (base model + soft loss) outperforms training using majority voting aggregates (base model + majority voting) regardless of which evaluation metric is used to compare the models. for the humour preference learning task, again, the base model outperforms uor's submission on both metrics, but in this case the difference in performance between gppl and uor is much less substantial with the hard metric, although it remains large according to the soft metric. this large difference may be due to a technical issue that requires further investigation, since uor's submission was also supposed to have been produced by the same base system. a possible reason for poor crossentropy error is the use of discrete labels, which are heavily penalized for overconfidence by crossentropy error. on this soft metric, the bayesian probabilistic approach of gppl may have advantages over approaches with poorer calibration, which remains to be explored in future work. the gppl approach therefore remains the state of the art with this dataset. for ic-labelme, again, soft-loss training achieved better hard and soft scores than both aggregation training with majority voting labels and the uor extension of the base model using a crowd layer adapted from @xcite . the finding that the uor group's adaption of the rodrigues and pereira (2018) crowd layer yielded lower f 1 than training using majority voting is unexpected, given that in @xcite ; @xcite and @xcite , the crowd layer, particularly the dl-mw variant, was shown to be a competitive approach to learning from crowds and always outperforms majority voting. however, uor's crowd layer does achieve better soft evaluation (cross-entropy) scores than majority voting. there is one dataset, however, on which uor outperformed the two baselines: ic-cifar10h. for this dataset, uor used zagoruyko and komodakis's (2016) wideresnet image classifier trained using majority voting aggregated labels and task model hard score (f 1 ) soft score (cross-entropy) pos base model + majority voting 0.753 2.263 pos base model + soft loss 0.767 1.084 pos uor (bert + crowd layer) 0.125 2.331 pdis base model + majority voting 0.906 0.397 pdis base model + soft loss 0.928 0.273 pdis uor (bert + crowd layer) 0.474 0.830 humour base model (gppl) 0.557 0.728 humour uor 0.513 3.697 ic-labelme base model + majority voting 0.806 2.833 ic-labelme base model + soft loss 0.833 1.691 ic-labelme uor (base model + crowd layer) 0.784 1.769 ic-cifar10h base model + majority voting 0.646 2.610 ic-cifar10h base model + soft loss 0.698 1.052 ic-cifar10h uor (wideresnet + sam) 0.769 0.827 ## conclusion this shared task presented the first unified testing framework for learning with disagreements. the datasets include sequence labelling, three classification tasks, and preference learning, hence provide a testbed for a wide range of challenges when learning from multiple annotators. we proposed to evaluate not just the 'hard' performance against a gold standard, but also the ability to predict the distribution of different interpretations of the data-that is, the alternative labellings provided by different annotators. the results show the benefit of soft loss functions that account for the distribution of labels in the training data. however, modelling alternative as a postscript, we should note that after the end of the official competition we did carry out an investigation of the reasons for the poor performance of uor's models on the tasks other than ic-cifar10h. some points emerging from the discussion are presented in the participants' paper for the shared task.