S em E val-2022 Task 6: i S arcasm E val, Intended Sarcasm Detection in E nglish and A rabic (2022)

Abstract: iSarcasmEval is the first shared task to target intended sarcasm detection: the data for this task was provided and labelled by the authors of the texts themselves. Such an approach minimises the downfalls of other methods to collect sarcasm data, which rely on distant supervision or third-party annotations. The shared task contains two languages, English and Arabic, and three subtasks: sarcasm detection, sarcasm category classification, and pairwise sarcasm identification given a sarcastic sentence and its non-sarcastic rephrase. The task received submissions from 60 different teams, with the sarcasm detection task being the most popular. Most of the participating teams utilised pre-trained language models. In this paper, we provide an overview of the task, data, and participating teams.

## introduction sarcasm is a form of verbal irony that occurs when there is a discrepancy between the literal and intended meanings of an utterance. through this discrepancy, the speaker expresses their position towards a prior proposition, often in the form of surface contempt or derogation @xcite . sarcasm is present on the social web and, due to its nature, it can be disruptive of computational systems that harness this data to perform tasks such as sentiment analysis, opinion mining, author profiling, and harassment detection @xcite @xcite . in the context of semeval, in particular, @xcite show a significant drop in sentiment polarity classification performance when processing sarcastic tweets, compared to non-sarcastic ones. such computational systems are widely deployed in industry, driving marketing, administration, and investment decisions @xcite . in the context of arabic, @xcite show the effect of sarcasm on arabic sentiment analysis systems, where the performance dropped significantly for the sarcastic tweets. as such, it is imperative to devise models for sarcasm detection. such models are usually built in a supervised learning paradigm, relying on a dataset of texts labelled as either sarcastic or non-sarcastic. two methods have typically been used to label texts for sarcasm: distant supervision @xcite @xcite , where texts are considered sarcastic if they meet predefined criteria, such as including the tag #sarcasm; or manual labelling @xcite @xcite , where texts are presented to human annotators. however, as argued by @xcite , both methods could produce noisy labels, in terms of both false positives, and false negatives. for instance, when human annotators label texts, they are limited by their subjective perception of sarcasm, which might differ from the intention of the authors of those texts. in response, we suggest the current shared task, isarcasmeval 1 . we rely on a novel method of labelling texts for sarcasm, where the sarcastic nature of texts is self-reported by the authors of those texts. our shared task is also novel in that it includes two languages, english and arabic, and includes three subtasks. the first subtask, covering both languages, is sarcasm detection as commonly understood: given a text, determine whether or not it is sarcastic. next, as the sarcastic texts in our english dataset are also further labelled for the ironic speech category that they represent out of the categories specified by @xcite , the second subtask is: given an english text, determine which ironic speech category it represents, or whether it is non-sarcastic. finally, for both languages, we also ask authors to provide non-sarcastic rephrases of their sarcastic texts. as such, the third subtask, covering both languages, is: given a sarcastic text and its non-sarcastic rephrase, identify the sarcastic text. we discuss related work in dataset creation, and related semeval tasks, in section 2. we introduce the data labelling method, and present statistics on the resulted datasets, in section 3. we provide details on the shared task in section 4, and on the submissions in section 5. ## related work most previous textual sarcasm detection datasets have been annotated using a distant supervision method. in this approach, texts are considered sarcastic if they meet predefined criteria, such as including specific tags (e.g. #sarcasm, #irony) @xcite , or being generated by specific accounts @xcite . however, this can lead to noisy labels for several reasons. first, the tags may not mark sarcasm, but may constitute the subject or object of conversation, e.g. "there is so much #sarcasm around!". second, the assumption that certain tags always appear in conjunction with sarcasm, or that certain accounts always generate sarcasm @xcite , could lead to further false positives. third, considering those texts that do not meet the criteria as non-sarcastic is a strong assumption that can lead to false negatives. due to the issues outlined above, other work has relied on manual labelling, where sarcasm labels are provided by human annotators @xcite @xcite . as such, the labels represent annotator perception, which may actually differ from author intention. annotators might lack awareness of the contextual devices that, as linguistic studies suggest @xcite @xcite , could be essential for clarifying the sarcastic intention of the authors. previous shared tasks in sarcasm detection @xcite @xcite present datasets annotated via the two methods discussed above. the potential noisy labels that these methods can produce gives us reason to be concerned about the effectiveness of models that were trained on such datasets. recently, @xcite proposed a third method, reactive supervision, which aims to collect sarcastic examples based on the conversation dynamics, addressing some of these issues by using statements such as "i was being sarcastic" to automatically label texts. however, this method relies on specific cues of sarcasm which may lead to a sample that is biased toward more confusing examples that required clarification. further, the vast majority of sarcasm detection work @xcite @xcite @xcite @xcite @xcite has focused exclusively on the english language and, due to the sociocultural aspects of sarcastic communication @xcite , it is unclear if models trained on english could generalise to other languages. to our knowledge, the small amount of work on other languages such as arabic @xcite @xcite relies on either manual labelling or distant supervision. representative of distant supervision is the work of @xcite , who consider arabic equivalents of #sarcasm, such as # , # , and # , to collect sarcastic tweets. other work, @xcite @xcite , used either manual labelling, or a mix between manual labelling and distant supervision. when working with arabic data, these two labelling methods are even more problematic considering the large number of dialects of the language that vary both across and within countries. relying on predefined tags in modern standard arabic (msa), such as those specified above, can thus lead to a plethora of false negatives. similarly, the third-party annotators might be unfamiliar with the dialect of the texts they are annotating, resulting in erroneous manual labels. ## conclusion this paper provides an overview of semeval-2022 task 6, isarcasmeval, which targets intended sarcasm detection. we provide an overview of the current state of research on sarcasm detection focusing on data collection methods. we introduce two new datasets for sarcasm detection in english and arabic. the data was collected by asking people to provide and label their own words as sarcastic or not, hence intended sarcasm. isarcasmeval contains three subtasks: sarcasm detection, sarcasm category classification, and sarcasm identification given a pair of sentences. the task was quite popular with the participation of around 62 teams. in this paper, we provide a high-level overview of the approaches of top teams in each of the subtasks. transformer models were dominant in all subtasks. detecting sarcasm in texts remains challenging; detecting the ironic speech category even more so. we hope our shared task will draw the attention of the community towards these important tasks. we suggest two main directions that future work could consider. first, in this shared task, sarcasm detection was performed by solely mining lexical and pragmatic cues from the texts being classified. however, the sarcastic intention of the authors might be unclear without reference to their previous utterances, and their sociocultural background @xcite . we suggest future datasets are needed to provide access to such information, and future models that account for it effectively. second, the low performance achieved by the models on subtask b requires further investigation. first, alternative categorisations could be considered. second, the ironic speech category labels should either be provided by the authors themselves, to avoid any bias introduced by trained annotators, or more emphasis should be placed on annotator training and annotation guideline clarity, to mitigate labelling noise that might indeed account, at least in part, for the low performance presented here. finally, more effort is needed to develop more effective models, likely making use of information outside of the texts being classified, including prior assumptions about the nature of ironic speech, sociocultural information about the authors, if available, as well as commonsense facts.