Generalizable Sarcasm Detection is Just Around the Corner, of Course! (2024)

Abstract: We tested the robustness of sarcasm detection models by examining their behavior when fine-tuned on four sarcasm datasets containing varying characteristics of sarcasm: label source (authors vs. third-party), domain (social media/online vs. offline conversations/dialogues), style (aggressive vs. humorous mocking). We tested their prediction performance on the same dataset (intra-dataset) and across different datasets (cross-dataset). For intra-dataset predictions, models consistently performed better when fine-tuned with third-party labels rather than with author labels. For cross-dataset predictions, most models failed to generalize well to the other datasets, implying that one type of dataset cannot represent all sorts of sarcasm with different styles and domains. Compared to the existing datasets, models fine-tuned on the new dataset we release in this work showed the highest generalizability to other datasets. With a manual inspection of the datasets and post-hoc analysis, we attributed the difficulty in generalization to the fact that sarcasm actually comes in different domains and styles. We argue that future sarcasm research should take the broad scope of sarcasm into account.

## introduction sarcasm can be used to hurt, criticize, or deride @xcite @xcite ) but also to be mocking, humorous, or to bond @xcite @xcite . it comes in many different shapes such as understatement, hyperbole, rhetorical questions @xcite , deliberate falsehood @xcite , or selfdeprecation @xcite . given such complexity and diversity of sarcasm, one should ask whether current sarcasm detection models are robust enough to be easily applied to any kind of sarcasm. this question is particularly valuable in computational linguistics, as most prior work addressing sarcasm adopted a narrow working definition of sarcasm as 'saying the opposite of the true message (often with the intent to be hurtful)' @xcite @xcite @xcite . this definition is based on the traditional theory of sarcasm by @xcite , but is not at all comprehensive @xcite . prior work in sarcasm detection has used one dataset at a time that only includes a specific style and domain of sarcasm. this can lead to frail sarcasm detection models because different datasets come from different sources such as social media @xcite , online forums @xcite , tv series @xcite , product reviews @xcite , or conversations @xcite . the vast range of domains, styles, and topics that sarcasm can be associated with invites the question of whether sarcasm detection models fine-tuned on a specific dataset are generalizable to other datasets. also, existing datasets of sarcasm contain sarcasm labels from different sources: tags (e.g., #sarcasm or /s) @xcite , labels by annotators @xcite @xcite , or labels by the authors of the posts @xcite . collecting data with labels based on tags is easy and scalable, but the data tend to be noisy and it poses a high risk of including false positives in the dataset @xcite . most manually annotated data come with labels provided by crowd workers or experts (though it raises the question of "who is an expert of sarcasm?"). these labels are more reliable than tags, but they still pose a risk of wrongly reflecting the intention of the original utterance maker @xcite as sarcasm can be missed (referred to by some as sarchasm) even in multimodal settings @xcite . in text-only settings, which is what a lot of sarcasm detection models are based on, the chance of misinterpreted sarcasm would naturally be much higher as the available cues are more limited. recent work has reported that previous state-of-the-art models fine-tuned on datasets with third-party labels showed vastly different (poorer) performance when predicting sarcasm on data with author labels, going from an f-score of 0.75 to 0.33 @xcite . thus, in this work, we investigate the generalizability of sarcasm detection models fine-tuned on different datasets that contain sarcasm from different domains (social media/online vs. offline conversations/dialogues), styles (aggressive vs. humorous), and labels from different perspectives (author labels vs. third-party labels). specifically, we make the following contributions: c1. through cross-dataset comparisons, we test the generalizability of sarcasm detection models, which have been fine-tuned on various datasets, to other datasets. we identify the datasets that lead to the highest prediction accuracy scores across datasets. c2. we test the effect of label source (author vs. third-party) on model performance by using a new dataset that we release. ## related work if there is any consensus around the topic of sarcasm, it is arguably the fact that sarcasm is hard to define @xcite . but much is known about the communicative functions of sarcasm as they have been extensively studied in theoretical and experimental work. sarcasm can be used to mock @xcite , to criticize harsher @xcite , to be humorous @xcite @xcite , or to save one's own face @xcite . sarcasm is not only multifaceted in its functions but also manifests itself in diverse forms, including hyperbole, often accompanied by interjections and intensifiers @xcite , rhetorical questions @xcite , violation of maxim of relevance @xcite or maxim of quality @xcite . despite the complications surrounding this topic, a lot of work in computational linguistics came up with operational definitions of sarcasm for au-tomatic sarcasm detection (i.e., contextual incongruity @xcite , and discrepancy between positive sentiment and negative situation @xcite ). sarcasm detection has since had fluctuating success rates from an f-score of 0.51 to 0.97 @xcite . one reason behind varying results may come from the nature of the data. twitter is the source that most previous computational work on sarcasm has relied on as training data @xcite @xcite . other sources of sarcasm data include online forums @xcite , reddit @xcite , or tv series @xcite . data sources matter for building a more robust sarcasm detector; @xcite showed that the same models performed quite differently across different datasets, with variations in f-scores of around 0.20. more recently, the importance of addressing the sources of sarcasm labels has also been raised in multiple studies @xcite @xcite . they collectively showed that sarcasm detection models perform significantly worse when making predictions on data with labels provided by the authors of sarcastic utterances. this is a factor worth examining as most datasets were created by having human annotators rate the level of sarcasm from the observer's point-of-view @xcite @xcite . in our work we investigate the robustness of sarcasm detection models by taking into account the label source, data source, types, and styles of sarcasm. ## datasets we take four existing datasets of sarcasm and use them in our experiment. we select these datasets because they each have different elements that facilitate cross-data comparison. in section 3.1, we provide descriptions for each dataset, followed by the strengths and limitations of each that motivate the creation of a new dataset, which we describe in section 3.2. we provide examples from each dataset in ## results and discussion ## post-hoc qualitative analysis we probe for reasons behind the low generalizability of lms with a post-hoc analysis. we conducted a quantitative analysis in which we quantitatively analyze and demonstrate how lms get accustomed to different types of cues for detecting sarcasm when fine-tuned with different datasets (c3). we focus on bert only for this analysis. methods. we identified certain patterns of language found in each dataset that may have led to differing results. as a proxy for styles or registers, we chose 17 semantic and psycholinguistic categories provided by the linguistic analysis tool liwc @xcite , which quantitatively analyzes text in terms of psychological con-structs such as emotion, cognition, or perception, among others (see results. figure 2 reports the unique features that help bert detect sarcasm in each dataset. a major characteristic of sarcasm corpus v2 is that it contains many words related to negative emotions such as anger, and words related to social issues, swearwords, online-style words, and disfluencies. this matches our preliminary description of the dataset described in section 3.3 that sarcasm corpus v2 contains the most aggressive and critical types of sarcasm of out of all the datasets. mustard contains a lot of words related to family and drives (i.e., achievement, risk, reward, etc.). we suspect that this phenomenon may be influenced by the tv series the big bang theory, where the characters often converse on topics related to achievement. csc shows a lot of words related to agreement (i.e., ok, yes, etc.) possibly because the speakers were instructed to talk to a close friend or colleague. the analysis also reveals a significant presence of words associated with religion, predominantly due to the use of terms like 'god' or 'jesus' in expressions such as 'oh my god' or 'jesus christ'. this trend might be attributed to numerous situational prompts where the interlocutor's behavior is portrayed as silly, eliciting strong emotional responses. the results for isarcasmeval are not reported because there were no instances for which fine-tuning on isarcasmeval only and not on the other datasets produced correct predictions. the post-hoc analysis of different datasets and model performance shows that sarcasm does indeed come in different shapes and domains, some serious and genuinely meant to inflict verbal pain to strangers, and some humorous and friendly used among well-meaning acquaintances, which is also supported by prior psycholinguistic work @xcite @xcite @xcite . ## conclusion we conducted intra-and cross-dataset sarcasm detection experiments to examine the robustness of sarcasm detection models. we introduced a new dataset, which we compared with other existing datasets of varying characteristics. several characteristics regarding sarcasm in each dataset, such as label source (authors vs. third-party), domain (social media/online vs. offline conversations/dialogues), style (aggression vs. harmless mocking) were used as points of comparison. all lms gave better predictions on the same dataset they were fine-tuned on and showed much lower prediction performance on the other datasets (c1). still, lms fine-tuned on our new dataset csc generalized the best to the other datasets. this was the case even when the domain of sarcasm in the target dataset was different from that of csc. lms performed better when the ground-truth labels were from third-party annotators rather than authors themselves (c2). a post-hoc analysis and a closer look at each dataset supported our assumption that such low cross-dataset predictions may be attributed to sarcasm coming in various styles, intent, and shapes (c3). therefore, future research should take the vast scope of sarcasm into consideration rather than only focusing on a narrow definition of sarcasm as 'the utterance of the opposite of the intended meaning' or as 'a figure of speech intended to be hurtful, insensitive, and critical'.