i L ab at S em E val-2023 Task 11 Le-Wi-Di: Modelling Disagreement or Modelling Perspectives? (2023)

Abstract: There are two competing approaches for modelling annotator disagreement: distributional soft-labelling approaches (which aim to capture the level of disagreement) or modelling perspectives of individual annotators or groups thereof. We adapt a multi-task architecture which has previously shown success in modelling perspectives to evaluate its performance on the SEMEVAL Task 11. We do so by combining both approaches, i.e. predicting individual annotator perspectives as an interim step towards predicting annotator disagreement. Despite its previous success, we found that a multi-task approach performed poorly on datasets which contained distinct annotator opinions, suggesting that this approach may not always be suitable when modelling perspectives. Furthermore, our results explain that while strongly perspectivist approaches might not achieve state-of-the-art performance according to evaluation metrics used by distributional approaches, our approach allows for a more nuanced understanding of individual perspectives present in the data. We argue that perspectivist approaches are preferable because they enable decision makers to amplify minority views, and that it is important to re-evaluate metrics to reflect this goal.

## introduction many natural language processing (nlp) tasks follow a supervised learning paradigm, i.e. classification of labelled data where multiple annotations are aggregated into a hard label using averaging or majority voting. hard labels are based on the assumption that each instance in a dataset has one singularly correct response-often referred to as 'ground truth'. however, this assumption is highly unrealistic for social computing tasks, such as such as toxicity and hate speech detection (e.g. @xcite , where lived experiences, biases, opinions and annotator experience all play a role in the subjective response an annotator might give. hard labels especially disadvantage minority groups @xcite . for example, in abusive language classification, where a minority is disproportionately affected (such as minoritised people who have faced online harassment), an aggregated majority label can obscure the perspective of the most vulnerable groups. thus, there is growing awareness that modelling multiple perspectives is necessary, particularly for inherently subjective tasks and those concerned with social issues @xcite @xcite . le-wi-di the semeval 2023 shared task 'learning with disagreements' @xcite aims to capture and model annotator disagreement -going beyond the assumption of one aggregated 'ground truth' label. participating teams are required to propose methods that consider disaggregated annotations, used to create a soft label, which represents the probabilistic distribution of the annotations. soft labels can then be used to predict the level of disagreement for each instance in a dataset. the task presents a benchmark of four datasets, including a hard label and soft labels for each instance. the datasets were chosen specifically to represent tasks that are highly subjective (e.g. hate speech detection) and show high annotator disagreement. participating teams are evaluated on how well their proposed model predicts both hard and soft labels, via f1 accuracy score and crossentropy loss, respectively. the shared task prioritised the soft evaluation, i.e how well the model's probabilities reflect annotator agreement, rather than simply proposing models that would outperform the current state-of-the-art for the hard labels. for further details of the datasets, see section 3. our approach proposes a modified version of the multi-task model introduced by @xcite , which aims to predict individual annota-tor judgments. by training our model to predict each annotator's judgement for each instance in a dataset, we can use the resulting predictions to infer the level of disagreement for that instance, without directly training for such a purpose. the main benefit of our approach is that opinions present in the dataset are preserved beyond the simplistic form of a polarised agreement/disagreement distribution. instead, we focus on representing individual opinions, also knows as 'perspectives' @xcite . this allows modelling of specific perspectives present in a dataset, potentially enabling the amplification of minority opinions. for the purposes of this paper, the former is relevant, as multi-task learning, enables the resulting model to optimise across different tasks through the use of both hard and soft labels, providing predictions for the hard label, as well as degrees of confidence for each @xcite . while these approaches make use of disagreement to enhance optimisation, they have not been used to preserve the different perspectives represented in the data. ## data the le-wi-di foot_0 shared task consists of the following four datasets, that have all been synthesised into a common json format. hs-brexit @xcite : a dataset of english tweets on the topic of brexit, labelled for hate speech by six annotators belonging to two different groups; three muslim immigrants and three other individuals. armis @xcite : arabic tweets annotated for misogyny and sexism. the three annotators for this dataset have different selfidentified demographics of 'moderate female', 'liberal female' and 'conservative male'. convabuse (cercas @xcite : a dataset of english dialogues between users and two conversational agents. the dialogues have been been annotated for abusiveness by eight gender studies experts. multidomain agreement (md): @xcite : english tweets on three topics: blm, election and covid-19, labelled for offensiveness by crowdsourced annotators and specifically selected to elicit disagreement. see ## results the task is evaluated using cross entropy loss (@xmath0. our results are shown in ## further analyses for deeper analysis, we used different methods depending on the specifics of the datasets. for convabuse and md, we followed @xcite to deal with missing annotator labels during the evaluation stage. although not all annotators annotated every instance, our model still predicts labels for all annotators in the dataset (e.g. predicting eight annotator labels in convabuse for all instances that only have four true annotator labels as shown in we also investigated reasons for our poorest cross entropy scores, on the armis dataset. we found that our model did not perform as well as expected in this scenario in which it should theoretically have performed well, i.e. with the annotators of the dataset self-identifying with a distinct ideological background (conservative, moderate, and liberal) and no missing annotator labels. we found that this may have been the result of the model's architecture. through testing with different batch sizes, we found that our model performed better when we added an extra hidden layer of 384 units to the existing 768-unit one. this would indicate that while the model was indeed learning, the original architecture with a single hidden layer was not sufficient to adequately disentangle enough information to make accurate predictions. ## conclusion we evaluated the performance of a multi-task model on predicting disagreement in four datasets, evaluated with both hard and soft labels, through f1 and cross-entropy loss respectively. our model learned and predicted individual annotator perspectives for each instance. our model's findings did not outperform our single-task bert baseline in terms of the shared tasks evaluation metric. this was due to model employing a strong perspectivist approach, which prioritised capturing individual perspectives present in the dataset, over high performance. we argue that a strong perspectivist approach is preferable to merely modelling disagreement as it allows to capture different opinions present in a dataset, and can be used to further amplify minority views. evaluation metrics for this edition of le-wi-di are geared towards measuring the overall levels of disagreement present in the datasets. however, if we wish to model stronger versions of perspectivism, we will need to develop new, more suitable metrics that can capture varying judgements in the kinds of different scenarios we have seen here.