CICL _ DMS at S em E val-2023 Task 11: Learning With Disagreements (Le-Wi-Di) (2023)

Abstract: In this system paper, we describe our submission for the 11th task of SemEval2023: Learning with Disagreements, or Le-Wi-Di for short. In the task, the assumption that there is a single gold label in NLP tasks such as hate speech or misogyny detection is challenged, and instead the opinions of multiple annotators are considered. The goal is instead to capture the agreements/disagreements of the annotators. For our system, we utilize the capabilities of modern large-language models as our backbone and investigate various techniques built on top, such as ensemble learning, multi-task learning, or Gaussian processes. Our final submission shows promising results and we achieve an upper-half finish.

## introduction this paper is a description of the methods we used for our entry foot_0 on this year's semeval task 11, learning with disagreements (le-wi-di) competition @xcite . semeval is a yearly workshop which comprises various natural language processing shared tasks. each team that participates in one of these tasks will try to come up with systems that deepen the understanding or improve results on one kind of semantic evaluation challenge. the task we participated in was task 11, learning with disagreements (le-wi-di). the goal of the task was to investigate how learning can be best achieved when there is no single gold label available, and instead the opinions of multiple annotators are given. for instance, consider the case of trying to classify if a sentence x contains hate speech or not. instead of pretending that there is a single ground-truth, binary label to it that every human would agree on, in the task we were provided with a multitude of labels y for x (i.e. @xmath0,1,0,0]), where each label reflects the opinion of one annotator. thus, there were both more uncontroversial instances where most annotators agreed and also more controversial instances where annotators disagreed a lot. scoring was done in two ways: first, standard f1 scores were calculated ("hard evaluation"), based upon a majority voting of the annotator's opinion as labels. however, as this was exactly the assumption that the task tries to overcome, hard evaluation was only secondary and mainly used to provide a link to other research. instead, the main evaluation criterion was "soft evaluation". here, the cross-entropy between the predictions and the disaggregated crowd-annotations label is measured. the disaggregated crowd-annotations label is describing how many annotators labeled with 1 relative to 0. for instance, the disaggregated crowdannotations label for @xmath1,0,0,0] would be [1,0] and for @xmath2,0,0,1] it would be [0.5, 0.5]. four datasets were included in the task. each of them had the individual annotator labels given and also the disaggregated crowd-annotations label. in that sense, they could be used as a single harmonized dataset. however, there still were some significant differences between them. please refer to the system description paper @xcite or appendix a for further description. we have structured our system description in three main sections. first, we describe the backbone and common techniques that we use in all our approaches. this includes the use of large language models, soft loss and the general training scheme. second, we describe our main approach that we submitted as our final result. this approach is a multi-task learning regime where we use soft and hard labels to fine-tune multiple output layers of our backbone large language model, dependent on the different tasks. finally, we describe the alternative approaches that we tried which include the use of different loss functions, ensemble techniques, dataset balancing, gaussian processes, an additional dataset and statistical features. in addition to the system description, we also report our experimental setup in section 4 and our results in section 5. we conclude with a summary of our approach and a discussion about possible future work. ## related work the realization that it is not sufficient for complex language tasks to only consider one ground truth and instead consider the opinion of a wide range of people with different backgrounds has become apparent at the latest with the advent of chatgpt and reinforcement learning from human feedback @xcite . further approaches to learn with ambiguous data include @xcite who train a reinforcement learning model to predict the preferences of individuals and thus are able to generate text that is more in line with a broad consensus. moreover, work from @xcite uses a multi-task learning setting to leverage information between hard labels and soft labels as uncertainty measures on ambiguous nlp tasks which they combine via kl divergence. and for the case that soft labels are not available, @xcite augment hard labels with disagreement. this approach takes a gold standard but includes the possibility of uncertainty by learning from hard labels as well as uncertainty measures. ## experimental setup we used the transformer library of hugging face @xcite to download our pre-trained models. furthermore, we used pytorch @xcite for all of our code. for experiment tracking, we used weights & biases @xcite . finally, for the gaussian process implementation, we used @xcite . for the common ground model of our submitted approach, we calculated the mean cross-entropy score across the whole dev dataset. for the submitted subjective task model, the cross entropy for a specific dataset was used to evaluate the corresponding heads. the models were trained on the whole train dataset and evaluated on the dev dataset. the training process was stopped if the epoch cross-entropy error on the dev dataset has risen two epochs in a row (early stopping). for submission and further use, the model weights for the epoch with the lowest cross-entropy on the unseen dev dataset were used. further information on hyperparameter tuning and training setup can be found in appendix b or our code on github 2 . ## results for our main approach, the common ground model achieved a mean cross-entropy on the dev datasets of 0.428, whereas the finally submitted model had a score on the dev datasets of 0.424. the second fine-tuning step of the multi-task learning setup did not seem to benefit the model a lot. all other approaches that we tried either had the same or were very slightly below the score of our main submission, though mostly in the range of 0.1 difference in terms of cross-entropy score. ## conclusion in this paper, we presented our submission for semeval-2023, task 11: learning with disagree-2 available at https://github.com/cicl-iscl/ lewidi_semeval2023 ments (le-wi-di). our main approach, which combines a pre-trained large language model with multi-task learning shows promising results for capturing disagreement as described in the task. furthermore, extensive investigation into other techniques, such as ensembles, gaussian processes and different loss functions provides a basis to more accurately determine which approaches are sensible or not to capture disagreement in the future. one way we think our approach can be improved in the future is to include more data, for instance by developing methods to collect annotations akin to the methods introduced in @xcite . furthermore, we think it could be worthwhile to investigate the use of different uncertainty measures, such as krippendorffs alpha which measures annotator disagreement for a dataset. finally, as all of our results are based on the embeddings of large language models, we certainly think that using bigger and more capable models will also improve results. dealing with uncertainty in the learning process with soft labels, it would be interesting to integrate an uncertainty measure in the training process. proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38-45, online. association for computational linguistics. zhi-hua zhou. 2008. semi-supervised learning by disagreement. in 2008 ieee international conference on granular computing, pages 93-93.