EPIC : Multi-Perspective Annotation of a Corpus of Irony (2023)

Abstract: We present EPIC (English Perspectivist Irony Corpus), the first annotated corpus for irony analysis based on the principles of data perspectivism. The corpus contains short conversations from social media in five regional varieties of English, and it is annotated by contributors from five countries corresponding to those varieties. We analyse the resource along the perspectives induced by the diversity of the annotators, in terms of origin, age, and gender, and the relationship between these dimensions, irony, and the topics of conversation. We validate EPIC by creating perspective-aware models that encode the perspectives of annotators grouped according to their demographic characteristics. Firstly, the performance of perspectivist models confirms that different annotators induce very different models. Secondly, in the classification of ironic and non-ironic texts, perspectivist models prove to be generally more confident than the non-perspectivist ones. Furthermore, comparing the performance on a perspective-based test set with those achieved on a gold standard test set, we can observe how perspectivist models tend to detect more precisely the positive class, showing their ability to capture the different perceptions of irony. Thanks to these models, we are moreover able to show interesting insights about the variation in the perception of irony by the different groups of annotators, such as among different generations and nationalities.

## introduction a recent trend in natural language processing (nlp) postulates that the disagreement among annotators in a language resource is a valuable source of knowledge, rather than noise that ought to be minimized or discarded @xcite . going one step further, the perspectivist approach aims at leveraging the disagreement in annotated data in order to model different points of view on the same phenomenon @xcite . applied to the study of natural language, this approach is particularly effective when the focus phenomena belong to semantic and pragmatic areas @xcite such as undesirable language detection, or irony and sarcasm. although related, the interpretation of irony involves linguistic patterns, such as the reference to an opposite or secondary meaning, and pragmatic features @xcite which could make it possible to recognize the phenomenon for people with different social backgrounds. this differs from the perception of abusive language, proved to be highly affected by different subjectivities @xcite . thus, a fundamental peculiarity of irony is that it tends to be both strongly dependent on the cultural background of the recipients @xcite , and, thanks to certain linguistic patterns, it may be understandable regardless of their country of origin. in this paper, we present epic (english perspectivist irony corpus), a corpus of short social media conversations annotated by taking into account the perspective of the annotators. in our view, and according to the perspectivist view, multi-faceted annotation represents an instrument to explore how demographic aspects may influence annotators' opinions, rather than a source of risk of bias. we created epic by collecting english messages and their direct replies from public online platforms, and annotated them by crowdsourcing. crucially, the texts are written in five varieties of english from different countries (ireland, the united kingdom, the united states, india and australia). the annotators, from the same five countries and with different demographic characteristics, expressed their opinion on their perception of irony in texts from all varieties. we believe that a non-aggregated corpus of irony analysis is a useful resource to train perspectiveaware models for irony detection, similarly to the approach of @xcite for hate speech modelling. in this direction, we validate the quality of this resource by creating various perspectiveaware models for irony detection encoding the perspective of annotators grouped according to their demographic characteristics. these models prove to be more confident in the recognition of irony in comparison with a non-perspectivist model, showing also an interesting increase of the precision in the detection of ironic messages when the various perspectives are represented in the test set. moreover, the usefulness of epic as perspectivist resource is confirmed by the variation in the perception of irony captured through the created perspectivist models. to sum up, the contributions of this paper are the following: i) a non-aggregated resource for english irony foot_0 ; ii) an analysis of analogies and differences in the annotation on the basis of demographic information about annotators, and correlations between these dimensions and ironic topics; iii) experiments with supervised learning that validate both the quality of the resource and the need for multiple perspectives explicitly encoded in the corpus. ## related work recent improvements in state-of-the-art language models have shown that the quality of the annotated data required for training automated systems is significantly more important than the amount of data itself @xcite . for this reason, in nlp, it becomes particularly important to devote special attention to benchmark datasets created within shared tasks and freely available to the research community, as their quality is assessed and improved through multiple uses by researchers. within the last ~10 years, the amount of irony-annotated resources and the organization of shared tasks regarding figurative language processing (among which, irony and sarcasm) for an increasing amount of different languages has considerably grown. the most resourced language for irony detection is english @xcite @xcite , but benchmarks have been proposed for other languages, in-cluding spanish @xcite , italian @xcite , dutch @xcite , chinese @xcite , and arabic @xcite . until 2016, the nlp community has mostly investigated irony as a "general way for describing different kinds of humorous content", @xcite , as one of the most specific cases of figurative language @xcite , or as a "polarity reverser" @xcite . starting from 2017, more specific interest in the phenomenon was deepened, so the community began to study its relationship with sarcasm, hate speech @xcite @xcite , also in different geographical variants of the same language (i.e., castilian, mexican, and cuban variety of spanish in ortega-bueno et al., 2019), and its importance in spreading of stereotypes as well as in author profiling tasks @xcite . as for works on irony that take a perspective approach, we think that the literature on this is not very extensive nowadays; ours is one of the few attempts in this direction. indeed, after more than a decade of investigation on this subject, it clearly emerged how irony is a highly subjective phenomenon in natural language, for which humans show divergent understanding and interpretation. as with other subjective phenomena, there is therefore an urgent need for the release of datasets with annotator-level labels and socio-demographic information about the annotators @xcite . a disaggregated dataset about humour in english @xcite has been released on the occasion of se-meval 2021 -task 12 on learning with disagreement @xcite . however, the currently available lists of disaggregated datasets show that no such kind of dataset exists for irony analysis. 2,3 this paper addresses this issue, since the availability of disaggregated data is a precondition to the study of divergent perspectives on the perception of natural language phenomena @xcite . ## corpus the corpus we are releasing is called epic and is made of 3, 000 short social media text pairs (post-reply) collected from twitter (1, 500) and reddit (1, 500). each pair has been annotated by multiple annotators that were asked to provide a binary label (either irony or not-irony) for the reply text given the context provided by post. in the following sections, we describe in detail how we collected the corpus (3.1) and conducted the annotation (3.2). ## statistical analysis epic contains 3, 000 unique annotated instances (post, reply) collected and annotated as described in sections 3.1 and 3.2. in this section, we provide high-level statistics about annotators and annotations and explore annotations at a deeper level. similarly to @xcite , we prove that aggregation by majority voting would introduce representational biases of individual and group viewpoints. in addition, we show how annotators' perceptions differ depending on the topic for which irony is being labelled. ## perspective-aware modelling results in this section, we describe computational experiments to detect irony using the epic dataset. as described above, this dataset has been annotated by different annotators coming from five englishspeaking countries and with different demographic characteristics. using the available information, we designed several classifiers that take into account the subjectivity of various groups of annotators divided according to their demographic characteristics. indeed, the epic dataset offers the opportunity to explore perspectivist approaches for irony detection, exploiting the information available about annotators. in these experiments, we want to understand the importance of a perspectivist approach for irony detection compared to a standard nonperspectivist approach, whose training and testing are based on a gold standard dataset. in particular, we want to answer the following questions: (1) what is the difference, especially in terms of confidence, between perspectivist and non-perspectivist models? (2) along which dimension can we observe the highest variation in the perception of irony? the first step was the creation of specific datasets to train and test the perspective-aware models, grouping the annotated texts on the basis of age, gender, and provenance of annotators as shown in ## conclusion in this paper, we presented epic, a corpus of short social media conversations from five english varieties (australian, british, indian, irish, american) collected from twitter and reddit and annotated with a binary label, irony or not-irony, by speakers from the five countries. we performed statistical analyses resulting in two key takeaways. the first is that aggregating the dataset with a majority voting scheme would introduce biases, thus hiding the perspective of some groups of annotators (e.g., those identifying as asian). this confirms the hypothesis that the perception of irony is dependent on the cultural background of the recipient. the second is that polarization among annotators depends on the topic. this means that though it is true that cultural background influences the perception of ironic content, there exist topics (such as arts and health) on which the influence is less evident than on others (such as labour, lifestyle or politics). moreover, we performed predictive experiments creating perspective-aware models for irony detection, that show how different annotators induce very different models, and how these perspectivist models, trained on subsets of the annotation coming from identifiable perspectives, are more confident at prediction time. finally, looking at the detection of irony, we believe that the best approach is based on assembling perspective-aware models plus perspective-based explanations. this is beyond the scope of the current work, which wants to present a solid basis on which to build such models. we plan to continue our research in two main directions. firstly, we intend to expand the dataset beyond english (i.e., spanish, german, french, italian, arabic, and others) in order to create the first multilingual perspectivist dataset for irony detection. secondly, we will employ epic as the basis for more advanced perspective-aware models and as a perspectivist benchmark for irony detection.