Confidence-based Ensembling of Perspective-aware Models (2023)

Abstract: Research in the field of NLP has recently focused on the variability that people show in selecting labels when performing an annotation task. Exploiting disagreements in annotations has been shown to offer advantages for accurate modelling and fair evaluation. In this paper, we propose a strongly perspectivist model for supervised classification of natural language utterances. Our approach combines the predictions of several perspective-aware models using key information of their individual confidence to capture the subjectivity encoded in the annotation of linguistic phenomena. We validate our method through experiments on two case studies, irony and hate speech detection, in in-domain and cross-domain settings. The results show that confidence-based ensembling of perspective-aware models seems beneficial for classification performance in all scenarios. In addition, we demonstrate the effectiveness of our method with automatically extracted perspectives from annotations when the annotators’ metadata are not available.

## introduction human label variability @xcite , traditionally considered a source of noise in annotated data, has recently become the subject of a series of research works in natural language processing. leveraging disagreement in annotation has been found beneficial towards more accurate modelling of natural language phenomena @xcite and fairer evaluation @xcite . these observations are particularly relevant when the focus of the study is on some latent linguistic phenomena at the level of pragmatics, such as irony or hateful speech, because of the high degree of subjectivity involved in the annotation of these phenomena @xcite . taking this research direction one step further, the perspectivist approach @xcite aims at creating models capable of encoding different human points of view on observed phenomena @xcite . this paper proposes a strongly perspectivist model for the supervised classification of natural language utterances. while our method is agnostic to the specific phenomenon that is the subject of classification, we expect it to work well when the focus is on pragmatic language phenomena. the reason for this expectation lies in the subjectivity of the annotation process for this level of analysis, which our model aims to model and exploit for a better classification. to prove it, we tested this methodology in two largely explored pragmatic phenomena in nlp: irony and hate speech detection. irony is a figurative language device that conveys the opposite or an extension of the literal meaning; hate speech produces the incitement of hate and violence against a specific target (individuals or entire communities) or the reinforcement of negative stereotypes @xcite . all experiments are on datasets in english. to test the robustness of the proposed perspectivist method, we performed in-domain and crossdomain experiments, i.e., testing models fine-tuned on a dataset on data from the same set and from different sets. we took inspiration from the observations in @xcite ) about the higher confidence exhibited by perspective-aware models trained on longitudinal subsets of the disaggregated data with respect to models trained on aggregated data. starting from this observation, we propose a method to ensemble perspective-aware models that take model confidence into account. furthermore, we prove that our methodology can also be applied when the annotators' metadata are unavailable and describe a methodology to mine perspectives from annotations directly. with this approach, on the one hand, we test whether our method can be applied when no explicit demo-graphic data of the annotators are available. on the other hand, we test whether the annotations can be used to identify interesting implicit discriminative dimensions that differ from those encoded in the metadata. this paper is organized as follows. in section 2, we present related previous works, with a specific focus on methods to build perspective-aware models; section 3 describes our perspectivist method. we perform in-domain and cross-domain experiments for irony (section 4) and hate speech detection (section 5). we detail our approach to mine perspectives and the results obtained with these implicit perspectives in section 6. the discussion and conclusive observations that emerged from our work are in section 7. ## related work the development of computational methodologies to account for the differing viewpoints of different annotators in detecting pragmatic phenomena is not new. however, most previous research focused on taking advantage of disagreement in the data to improve the classification performance @xcite @xcite . recently, attention to the disagreement among the annotators has been increasing. @xcite and @xcite organized two editions of the lewidi (learning with disagreement) shared tasks, respectively, in semeval 2021 (task 12) and semeval 2023 (task 11). the organizers of both editions of the workshop remark on the concept of soft label (vs. hard label) proposed by @xcite , a method for learning from a distribution over the label space to maximize classification performance and provide a robust evaluation metric. the paradigm of learning from disagreement takes into account the presence of disagreement in the annotated data, because different opinions could arise from different annotators. data perspectivism overcomes the idea of "ground truth" in constructing datasets and creating nlp models to give space to different perspectives that can be encoded in the annotated corpora, focusing on who the annotators are. following this new theoretical framework, two editions of the nlperspectives workshop foot_0 have been organized in 2022 and 2023. for modelling perspectives in hate speech detection, scholars have proposed various approaches, taking into account the individual annotation of each annotator @xcite , group-based annotations @xcite , specific aspects of annotators such as biases @xcite and beliefs @xcite , or using personalized methods @xcite . in particular, @xcite proposed to exploit annotators' systematic disagreement and investigated multiple approaches for hate speech and emotions detection: an ensemble model, where each classifier is trained on annotator-specific data only; a multi-label model, where each label corresponds to that assigned by an individual annotator; a multi-task model, where the prediction of labels from each annotator is modelled as a separate task. the latter is superior to a baseline model trained on the aggregated data. @xcite , instead, divided the annotators into two groups; they took into account common personal characteristics such as ethnicity, social background, and culture and modelled the group-based perspectives in recognition of hate speech. in contrast to perspectivist literature on hate speech detection, perspectivist approaches to irony detection are less explored. to our knowledge, only two disaggregated datasets for english exist on humour @xcite and irony @xcite . the first dataset was used as a benchmark in the first edition of lewidi @xcite . the second, the english perspectivist irony corpus (epic), was used to explore the agreement among annotators with similar socio-demographic traits. based on these traits, the authors also created perspectivist models showing the variation of the perception of irony among different population segments. for irony detection. while these models are trained on mostly similar texts, we expect differences to emerge in their predictions due to the different labels they were exposed to at training time. furthermore, the models will output predictions with different degrees of confidence, which is the critical information we exploit for the proposed ensemble model. formally, for each instance at inference time, we obtain from the trained models n predictions (l 1 , c 1 ), ..., (l n , c n ) where l @xmath0, 1} is a binary label and c @xmath1, 1] is a confidence score. we use the normalized difference between the two logits rescaled by softmax @xcite as a confidence score. specifically, we compute the confidence as: the predicted label of our confidence-based ensemble model is therefore computed as: in plain terms, the confidence scores of the models predicting the negative class are summed, as are the confidence scores of the models predicting the positive class, and the ensemble selects the label associated with the highest confidence sum. we also propose a simpler version of the ensemble computing a confidence-based vote: in this version, the ensemble selects the label predicted by the classifier with the highest confidence in its prediction. ## perspectivist classification of irony we present an irony detection experiment with the goal of proving the informative value of annotator perspectives in the annotated data. ## perspectivist hate speech detection considering the promising results achieved by our method on the irony detection task presented in the previous sections, we further validated our approach on a different task. hate speech (hs), similarly to irony, is a pragmatic phenomenon in language, with a high degree of subjectivity involved in its annotation @xcite . unsurprisingly, hs and related phenomena of undesirable language are objects of several works in perspectivist nlp (cercas @xcite @xcite . this section presents the results of experiments analogous to the one we presented on irony detection on similar-shaped data annotated with hs. ## mining perspectives the method we presented so far takes advantage of some division of the annotators known in advance. however, this is not always the case, for instance, when an annotation results from crowdsourcing with anonymous contributors. therefore, we devised a method to automatically group the annotators meaningfully to represent different perspectives. in this section, we present the result of a similar experiment to the main one presented in section 4, with the difference that the perspectives are mined automatically. in this setting, the data split to train the perspective-aware models depends on annotations rather than demographic traits. following @xcite , given n annotators and k instances, we constructed a label matrix v n×k . since each instance was labeled, on average, by 4.72 annotators only, the resulting matrix was highly sparse. we modelled the similarity among annotators in terms of inter-annotator agreement, using krippendorff's alpha (α) @xcite . we thus computed the pairwise agreement between annotators i, j, obtaining a similarity matrix a i,@xmath2: , v j: ), from which we calculated the distance matrix @xmath3, used as input for the clustering algorithm. we found 82 cases in which two annotators had no common annotated instances, and we decided to assign α = 0 for all of them. in 158 cases, there was perfect agreement between pairs of annotators on a small sample of data, having as a consequence that the α was not well-defined - @xcite describes it as a "paradox". we set α = 1 for these pairs. we used hierarchical agglomerative clustering 4 with ward's linkage criterion to group the annotators, computing the full dendrogram. then, we applied the calinski-harabasz @xcite and davies-bouldin indexes @xcite to assess the best number of clusters between 2 and 5, and 11 clusters as the perspectives. the two metrics re- looking at the annotators' distribution, there is no evident mapping between the demographic-based perspectives and the mined clusters, for any of the gender, nationality, and generation dimensions. this confirms our hypothesis that the metadata provided by epic do not necessarily induce the best possible split among the annotators, and therefore automatically mining perspectives could uncover latent dimension that better discriminates the annotators. as for gender, cluster 3 represents a small percentage of females, but male annotators are better distributed among the five clusters. unbalanced representations can be seen also when considering nationality; cluster 3 hosts 40% of the british annotators, and a small percentage of both indian and irish annotators (represented especially in the first three clusters), while cluster 1 has no british annotators at all. similarly, focusing on age, cluster 0 and 1 have no boomer annotators, and genz is more represented by clusters 0, 2 cluster 3 in respect to the remaining two. despite these differences, none of the demographic groups merge homogeneously into specific clusters. finally, we tested the validity of the clusters by reproducing the method described in section 3. we obtained 5 classification models, which we tested in an in-domain setting (section 4) and in a cross-domain one (section 4.5); in both cases, we averaged the results over 10 runs. ## conclusion we showed how computational models of irony and hate speech are more proficient when built on disaggregated data, i.e., explicitly modeling the disagreement between the annotators. one step further, our experiments validate the perspectivist approach of grouping annotators to better encode their shared background, either by using their metadata or by automatically grouping them with a clustering approach. the main novelty of the method described in this paper is the use of the model confidence as a computational tool to ensemble a set of predictions coming from different perspective-aware models. this intuition, originated by the work of @xcite , was proved central in building more accurate predictive models for irony and hate speech detection. the experimental results presented in this paper show that our method produces more accurate classifications of irony. in the case of hate speech, the results were mixed. however, the shape of the available data used for training the models in the two tasks varies greatly, in particular with respect to the number of annotators -epic (irony) has 74 annotators while mhs has thousands, resulting in a much greater variety of individual perspectives beyond those captured in the annotator metadata. more impressively, we have shown how the method is equally valid (and in many cases performing better) when the perspectives are automatically extracted from the disaggregated annotations rather than explicitly provided (section 6). our experiments focused on the two case studies of irony and hate speech detection, and it remains to be explored how well our approach generalizes to other pragmatic phenomena. we plan to apply our methods to other pragmatic phenomena, irony-adjacent and in the undesirable language area, where we expect the most impact, as well as other languages. future research should also include a broader range of demographic profiles to ensure greater inclusivity and improve the generalizability of perspectivist models across various linguistic contexts. finally, perspective-aware models and ensembles naturally have the potential to improve explainability: they provide a layer of information that is useful for explaining the output of the classifiers, i.e., in terms of the human perspectives more likely to be sensitive to specific labels.