M ulti PIC o: Multilingual Perspectivist Irony Corpus (2024)

Abstract: Recently, several scholars have contributed to the growth of a new theoretical framework in NLP called perspectivism. This approach aimsto leverage data annotated by different individuals to model diverse perspectives that affect their opinions on subjective phenomena such as irony. In this context, we propose MultiPICo, a multilingual perspectivist corpus of ironic short conversations in different languages andlinguistic varieties extracted from Twitter and Reddit. The corpus includes sociodemographic information about its annotators. Our analysis of the annotated corpus shows how different demographic cohorts may significantly disagree on their annotation of irony and how certain cultural factors influence the perception of the phenomenon and the agreement on the annotation. Moreover, we show how disaggregated annotations and rich annotator metadata can be exploited to benchmark the ability of large language models to recognize irony, their positionality with respect to sociodemographic groups, and the efficacy of perspective-taking prompting for irony detection in multiple languages.

## introduction the pervasiveness of ai-based technologies has renewed the interest in making artificial intelligence more inclusive and attentive to users' needs. aibased technology mirrors the quality and problems of data that feed it @xcite . therefore, the way corpora are created and the design biases that are -consciously or unconsciously -included in datasets and models are of central importance. perspectivist corpora allow turning bias from an undesirable criticality into a measurable trait by reporting the perception of multiple individuals with different social and cultural traits on pragmatic phenomena (e.g., hate speech or irony). s. casola, s. frenda, and s. m. lo contributed equally to this work. in this work, we present multipico (multilingual perspectivist irony corpus), a multilingual corpus of short conversations (post-reply) extracted from twitter and reddit and annotated as ironic or not ironic by crowdsourcing workers with different social backgrounds foot_0 . multipico covers 9 languages and 25 varieties, ranging from high to low resources. specifically, we include 5 varieties of english (australian, british, indian, irish and us english), 5 varieties of spanish (argentinean, castilian, colombian, mexican, and us spanish), 5 varieties of arabic (egyptian, iraqi, moroccan, saudi arabian, and yemen), 2 varieties of french (canadian and french), 3 varieties of german (austrian, german, and swiss), 2 varieties of portuguese (brazilian and portuguese), and italian, dutch, and hindi. selected annotators' nationality is related to these linguistic varieties, the annotators' gender is balanced, and a rich set of other sociodemographic information (age, ethnicity, student and employment status) is provided. starting from these metadata, we analyze the perception of irony in different sociodemographic groups and explore which demographic dimensions primarily explain the annotation. furthermore, we also show how to exploit this multifaceted corpus as a benchmark for ai-based technologies, including large language models (llms). these models are becoming pervasive, but the lack of information on their training data and methods makes their intrinsic bias hard to measure. in short, our contributions are the following: ## related work the nlp community is increasingly questioning how annotators' background can influence their choices and perceptions of pragmatic linguistic phenomena, such as irony or hate speech. @xcite proposed a set of best practices for nlp technicians to avoid ethical issues such as exclusion and misrepresentation of users. their recommendations include reporting annotators' demographic data, as they can impact how annotators use and interpret language. numerous works have highlighted human annotation is subjective @xcite , especially for semantic and pragmatic phenomena -as the point of view might differ in relation to one's social background, beliefs, and demographics. an increasing number of works emphasize abandoning a single ground truth and preserving the disagreement among annotators @xcite @xcite . dealing with human label variation has consequences on the whole machine learning pipeline @xcite , from data to modeling @xcite and evaluation @xcite . the perspectivist approach aims at leveraging disagreement and modelling annotators' perspectives @xcite . to do so, scholars have exploited individual annotators' decisions @xcite or grouped their annotations based on their beliefs @xcite or demographic traits @xcite . to work in this direction, disaggregated corpora with demographic information become of fundamental importance @xcite . in fact, aggregated data tend to reflect a minority of perspectives, under-representing others @xcite . disaggregated datasets have become more widespread in recent years, as listed in the per-spectivist data manifesto foot_1 and by @xcite 3 . some of these corpora account for annotators' demographic. for example, @xcite built a perspectivist corpus for hate speech detection, asking annotators to self-identify their race, gender, sexuality, religion, education, and income. @xcite collected a dataset about misogyny and sexism in arabic and had it annotated by three muslim annotators (2 female, 1 male), who self-identified their religious belief as liberal, moderate, and conservative. they also provided annotations by a larger cohort on a small set of their dataset. the work showed a correlation between annotators' beliefs and their perception of misogyny and sexism in the texts. similarly, @xcite developed a dataset for hate speech detection involving migrants as victims of offensive texts and demonstrating that members of the same group tend to agree more. moreover, they implemented perspectiveaware models by creating a gold standard for each group, training two separate models, and finally ensembled by majority vote. perspective-based ensembling methods have also been explored by @xcite for irony and hate speech detection in english. in their work, perspectives are extracted using sociodemographic information of annotators and mined from annotations. the use of demographic data when working on highly subjective language phenomena has shown improvement in nlp research. @xcite investigated how annotators with different identities and beliefs perceive toxic content considering the characteristics of texts along three dimensions: racial prejudices, african american english vernacular and dialects, and vulgar words. the authors demonstrated that identities and beliefs impact annotators' judgments. sociodemographic information has also been used to build a disagreement predictor by @xcite , who tried to quantify how controversial an utterance is. they show that adding demographic information to the input improves results and returns insights on which instances need a more diverse group of annotators. finally, @xcite developed a framework to identify datasets and model design biases, highlighting their positionality. results show available datasets strongly align with the young weird population (western, educated, industrialized, rich, democratic). few disaggregated datasets for irony detection exist. @xcite released a corpus about humor detection in english, used as a benchmark in the first edition of the learning with disagreement shared task @xcite . no annotators' metadata, however, are included. @xcite proposed a dataset for irony detection and investigated the influence of the annotators' characteristics on their perception @xcite ; the dataset, however, contains english text only. to the best of our knowledge, no multilingual disaggregated dataset exists. we fill this gap by proposing a perspectivist dataset for irony in 9 languages and a total of 25 high and low-resourced varieties. we also provide demographic information on all the annotators involved. ## multipico in this section, we describe multipico, a corpus of 18, 778 short conversations collected from reddit (8, 956) and twitter (9, 822) in 9 languages, and a total of 25 varieties. data has been collected reproducing the structure of short conversations. annotators were asked to read a set of post and reply pairs and answer whether the text of the reply was ironic or not, given the context. we collected reddit comments (first comments in the vast majority of cases and second-level comments in a few cases) with their direct replies and conversation-starting messages. for twitter, the post could be both a conversation starter or a direct reply to it. reddit data were retrieved using the pushshift repository foot_3 from january 2020 to june 2021. to collect data in several linguistic varieties, we picked 26 subreddits, as reported in the human annotation of the collected data was performed on the crowdsourcing platform prolific foot_5 , through an integrated custom-built annotation interface designed to collect a diverse and balanced set of annotators. the interface mimicked a message conversation, having the post as context and asking whether the reply was ironic or not ironic foot_6 . we prevented an instance from being annotated more than a predefined number of times, with a mean of 5.02 annotations per instance (see the quality of the annotation has been further assured using attention check questions in the form of "please answer x to this question". annotators had 1% probability of receiving these special questions. we set a threshold of 50% correct answers; only 13 annotators failed the test, and their annotations were excluded from the final corpus, leading to a total of 506 annotators. together with the annotators' gender and nationality, we collected other demographic information, specifically: age group (13 baby boomer, 66 gen-x, 260 gen-y, 162 gen-z, 5 null), ethnicity (315 white, 64 mixed, 44 asian, 13 black, 66 other, 4 null), student status (260 no, 165 yes, 81 null), employment status (178 full-time, 74 part-time, 74 unemployed and job seeking, 24 not in paid work, 11 due to start a new job within the next month, 36 other, 109 null) 10 . ## annotators' polarization in this section, we present an exploratory analysis of multipico. we employ primarily the polarization index (p-index) proposed by @xcite , which measures the instance-based polarization for k groups. for each instance i, the pindex is defined as: for instance, looking at we applied the p-index for two analyses to understand the relevance of sociodemographic information in the interpretation of irony and to investigate its cultural basis: to evaluate the significance of the obtained pindex scores, we computed random p-index values for each instance using random partitions with the same number of annotators for dimension and trait (see ## multipico as a benchmarking tool given its fine-grained metadata, both at the annotation and the annotator level, multipico allows for detailed performance evaluation in a perspectivist setting. in this section, we will first analyze llms zero-shot performance for irony detection (section 5.1) and then show some of the evaluation possibilities in exploring the llms' positionality (section b.3) and the effectiveness of socio-demographic prompting (section 5.3). ## conclusions and future work we have presented multipico, a multilingual perspectivist corpus for irony. the corpus consists of post-reply pairs from social media platforms, and it is released with disaggregated annotations and annotators' metadata. our analysis of the corpus shows a high variability in the perception of irony across sociodemographic groups, confirming the importance of considering the annotators' characteristics and background for irony analysis. some demographic dimensions like age and student status particularly induce a strong polarization of annotations. in some languages, such as german, english, and spanish, we notice a connection between annotators' nationality and their sensitivity to irony across their linguistic varieties. it suggests that annotators with a background that is in line with a specific language variety tend to have higher agreement. furthermore, we showcase the utility of multi-pico as a benchmark for perspectivist modeling of irony through experiments of perspective-taking prompting with llms. in particular, we compare prompts without sociodemographic information, with trait-based perspectives, and with data-driven perspective examples, showing that it is possible to evaluate models in a perspectivist setting. moreover, multipico allowed us to perform a positionality analysis of chatgpt, discovering that tends to be aligned with the youngest annotators. in the future, we plan to expand the analysis of linguistic patterns that characterize irony across languages and language varieties, looking also at the different types of irony (i.e., situational and verbal irony) involved in the post-reply pairs. furthermore, we plan to mine meaningful clusters of annotators based on their annotation rather than on their demographic traits and to analyze whether a meaningful relationship exists between these groups and those based on sociodemographic information.