MIND at S em E val-2023 Task 11: From Uncertain Predictions to Subjective Disagreement (2023)

Abstract: This paper describes the participation of the research laboratory MIND, at the University of Milano-Bicocca, in the SemEval 2023 task related to Learning With Disagreements (Le-Wi-Di). The main goal is to identify the level of agreement/disagreement from a collection of textual datasets with different characteristics in terms of style, language and task. The proposed approach is grounded on the hypothesis that the disagreement between annotators could be grasped by the uncertainty that a model, based on several linguistic characteristics, could have on the prediction of a given gold label.

## introduction nowadays, natural language processing models are consistently adopted to address several tasks, such as abusive and offensive language detection @xcite and hate speech detection @xcite . the state-of-the-art works address both general and specific targets (e.g., women, lgbtq, immigrants) from a unimodal and multimodal perspective, also considering several languages @xcite @xcite @xcite . however, although these tasks are subjective, the majority of the proposed works are based on the assumption that a unique perception and interpretation exists for each instance. however, this assumption does not reflect what happens in the real world, where multiple readers could have different points of view and understanding, resulting in a sort of disagreement. such disagreement, which typically reflects the subjectivity of the task, can be due to linguistic ambiguity or different beliefs originated from different socio-cultural aspects. to this purpose, many researchers have proposed corpora that aim at capturing all the distinctive interpretations of a given text by preserving the annotations disagreement instead of just aggregating them in a single gold label. the information brought by the annotators' disagreement has been mainly exploited in three different ways: (1) to improve the quality of the dataset by removing those instances characterized by a disagreement between annotators @xcite , (2) to weight the instances during the training phase @xcite or (3) to directly train a machine learning model from disagreement, without considering any aggregated label @xcite . an important contribution in this field has been given by the semeval-2021 task 12 -learning with disagreements (le-wi-di) @xcite , whose aim was to provide a unified testing framework from disagreement for interpreting language and classifying images. in this paper, we address the learning with disagreements (le-wi-di) task at semeval-2023 task 11 @xcite , where the main goal is to model the disagreement between annotators on different types of textual messages. in particular, we proposed a straightforward strategy that given the probability of the hard-label prediction, it creates an optimal soft-label mapping. the paper is organized as follows. detail about the shared task and the related datasets are reported in section 2. in section 3 an overview of the state of the art is provided. in section 4 the proposed approach is detailed focusing on the prediction model, linguistic characteristics, preprocessing and post-processing operations. in section 5 the results achieved by the proposed models are reported. finally, conclusions and future research directions are summarized in section 7. ## task description the main goal of the semeval-2023 task 11 -learning with disagreements (le-wi-di) @xcite , is to develop methods able to capture the agreement/disagreement among the annotators toward a specific sentence label. the organizers encourage the usage of shared features between the available datasets for the definition of a model that is able to generalize over potentially different domains. given the subjectivity of the proposed tasks, and therefore the absence of a golden label, the evaluation is based on two different strategies: (i) hard evaluation based on the f1 measure estimated on the prediction capabilities on the hard label and (ii) soft evaluation that considers how well the model's probabilities reflect the level of agreement among annotators using the cross-entropy as a performance metric. ## related work machine learning models require to be trained with a significant amount of data, which usually needs to be labeled by human annotators. in order to collect such data in an efficient and cost-effective way, many researchers have relayed on crowd-sourcing platforms. since the collected labels typically show disagreement among the annotators, in many cases, the gold label is obtained by a majority voting mechanism, that also involve the intervention of a domain expert or more complex approaches based on annotators commonalities or sample weights @xcite . moreover, disagreement can also derive from overlapping labels, subjectivity, or annotator error @xcite . nowadays, datasets with multiple annotations have become increasingly common and disagreement information have been used as additional information for training predictive models @xcite . in @xcite the authors model the annotators behavior in a graph structure in order to represent their frequency of disagreement with each other and take advantage of information collected through community detection algorithms to train classifiers. in @xcite the authors implement a multitask learning ## proposed approach the proposed approach is grounded on the hypothesis that the disagreement between annotators could be grasped by the uncertainty that a model, based on several linguistic characteristics, could have on the prediction of a given gold (hard) label. this hypothesis reflects the uncertainty that a human annotator might have when annotating a text with disagreement. to validate this hypothesis, the proposed approach maps the probability values returned by a classifier trained to predict the hard label into agreement/disagreement values. the main two steps of the proposed approach are synthesized in figure 1 and described as follows: the probability distributions over all the sam-ples in the training dataset are used to compute the optimal classification threshold according to the youden's j statistics @xcite . the youden's j statistics, which is a linear combination of sensitivity and specificity, is maximized evaluating several cut off of the decision threshold of the classifier to obtain the optimal value. the proposed strategy, which basically optimizes the area under the curve (auc), allowed us to select the best classification threshold for the hard label and partially overcome the imbalance of the dataset labels. the model designed to predict the hard label is used to determine representative probability values that can be associated to each soft label. in particular, focusing on the positive label, the sub-samples of the training data sharing the same soft label (e.g. @xmath0.8) are used to obtain the corresponding probability distributions from the hard label model. such probability distributions are then used to compute, for the positive class, the corresponding mean value subsequantly used as representatives for determining the corresponding sl labels. once the model for the hard label prediction has been trained and the soft label representative identification has been performed, we can move to the inference phase. in particular, when an unseen sample has to be classified, the hard label model is exploited. for the hard label prediction, the most probable class is selected according to the comparison of the probability distribution and the optimal cut-off obtained via the youden's statistics. for the soft label prediction, the probability of the positive class is compared to the representative mean values estimated in the previous step, and assigned to the closest one. this allows us to automatically map the probability of the hard label to the corresponding soft label. in order to accomplish the above-mentioned steps, we developed a system based on four main constituents subsequently described. ## results in order to provide the label predictions for the test set, several input configurations have been investigated related to the input layer of the hard label classifier. in particular, the following models have been considered during the training phase: the results achieved by the proposed approaches on development and test data are summarized in tables 3 and 4 respectively foot_0 . the achieved results highlight that the exploitation of the considered metadata, i.e., emotion, hashtag disagreement score, language pragmatics and stylometric features, allows a better representation of disagreement resulting in an improvement of the cross-entropy values in all the considered datasets. however, since this representation emphasizes the uncertainty in the model predictions it also results in a decreasing performance in terms of f1 measure on the hard label. it is important to note that while m4 brings an overall improvement (with respect to the other models) when considering the cross-entropy measure, we can not observe any difference in terms of f1-micro measure. ## considerations about the evaluation metrics 6.1 performance evaluation the state of the art about performance evaluation is characterized by two main types of evaluation metrics @xcite in order to evaluate how well the model performs when (i) all items are treated equally and (ii) the items are weighted depending on disagreement. the most frequent hard label measures in the first scenario are the percentage accuracy and the class-weighted f1. an alternative is based on the crowd-truth weighted f-measure (ct-f1). this approach is based on the intuition that items characterized by a large value of disagreement should be weighted less than items characterized by a large value of agreement. regarding the soft label predictions, many approaches have been discussed @xcite ## conclusions and future work the proposed approach represents a preliminary attempt to deepen the relation between model predictions and samples' disagreement. the achieved results highlight a correlation between metadata and disagreement predictions: considering the average performances, the inclusion of emotion and lexical-related metadata resulted in an improvement of the performance in terms of disagreement detection. as future research direction, the correlation between annotators' agreement and both emo-tion and lexical-based features will be deepened in order to achieve a representation more suitable for the model to address the task of disagreement prediction. despite the poor improvement introduced by the universal model, the data provided by the organizer resulted to be referred to tasks too different to be assimilated into a single classification model. as a future study, it would be interesting to integrate additional datasets collected for the same tasks in order to realize task-related models. finally, more complex architectures (e.g. transformer-based) will be implemented to improve prediction capabilities.