The Difficult Case of Intended and Perceived Sarcasm: a Challenge for Humans and Large Language Models (2025)

Abstract: We examine the cases of failed communication in sarcasm, defined as ‘the discrepancy between what speakers and observers perceive as sarcasm’. We identify factors that are associated with such failures, and how those difficult instances affect the detection performance of encoder-only and decoder-only generative models. We find that speakers’ incongruity between their felt annoyance and sarcasm in their utterance is highly correlated with sarcasm that fails to be communicated to human observers. This factor also relates to the drop of classification performance of large language models (LLMs). Additionally, disagreement among multiple observers about sarcasm is correlated with poorer performance of LLMs. Finally, we find that generative models produce better results with ground-truth labels from speakers than from observers, in contrast to encoder-only models, which suggests a general tendency by generative models to identify with speakers’ perspective by default.

## introduction an utterance that is intended to be sarcastic by the speaker is sometimes not understood as such by the listener or external observers, or vice versa @xcite . consider the example below. scenarios, especially those requiring layers of inferences, which are common features of sarcasm @xcite . discussing the divergence between intended and perceived sarcasm is not new. prior work in psycholinguistics has widely discussed the differences in what speakers intend with sarcastic utterances and how listeners or observers interpret them @xcite . nlp tasks and datasets are also affected by such discrepancies. for instance, oprea and magdy (2020) demonstrated that there are many instances for which external annotators provide different sarcasm judgments from the producers of the utterances. sarcasm detection by language models, especially bert-like models, also show different classification performances according to ground-truth labels based on self-evaluation versus external evaluation @xcite @xcite . since the capacity of (large) language models has increased exponentially over the years with the advent of generative models, which are often placed in direct conversations with human users, it has become an important question to ask how language models navigate different perspectives in communication involving sarcasm. although it is evident that sarcasm judgment is contingent on the different perspectives of speakers and observers @xcite , there is a lack of systematic investigation on what factors contribute to the general difficulty of the task for llms as well as for human observers. one of the numerous keys to identifying the source of such discrepancy between speakers and observers is to think about why sarcasm is used in the first place. sarcasm is used to convey specific communicative intentions, such as to mock the addressee @xcite , which in turn is motivated by speaker's emotion in a given communicative situation @xcite . the strong link between emotion and sarcasm has long been identified and discussed in numerous previous studies (e.g., @xcite @xcite . as such, we focus on the close connection between sarcasm and emotion to examine the discrepancy between speakers and external observers in the use of sarcasm. specifically, we focus on annoyance, an emotion shown to strongly influence sarcasm production and identification @xcite . though annoyance is not the only reason why a speaker chooses to use sarcasm, we focus on annoyance in this work based on @xcite , who report that a strong connection is observed between speaker's annoyance and the level of sarcasm in their utterance, and that external observers are also able to capture this connection. the availability of such information in the dataset described in section 3.1 also motivates such research design. 2 we demonstrate which factors are associated with the divergence of sarcasm judgment between speakers and observers and how this affects (l)lm performance on sarcasm detection. ## experiment 1: sarcasm detection by human observers the first experiment inspected the factors related to sarcasm communication failure between human speakers and observers, by testing two hypotheses described in section 3.2. ## experiment 2: sarcasm detection by llms in section 4, we showed that speaker's annoyancesarcasm congruity is highly correlated with sarcasm being correctly transmitted to human observers. based on these results, we examined whether the same factor influences the sarcasm detection performance of llms. we conducted a classification experiment with encoder-only mod-els and decoder-only models. the encoder-only models are classical observer models suitable for the task of sarcasm detection, and the decoder-only models are generative models that have shown their impressive capabilities to handle numerous nlp tasks. we used the fine-tuning settings for the encoder-only models, because they tend to require task-specific tuning to ensure a reasonable level of performance @xcite . we used zeroshot settings for the generative models, without additional fine-tuning that requires substantial computational resources. we binary-coded the original sarcasm ratings by both sources -speakers and observers (averaged) -by using the midpoint of the scale (3.5) as the cut-off point. we downsampled csc to have an equal number of sarcastic and non-sarcastic instances (@xmath0,210 vs. 2,398). ## decoder-only vs. encoder-only: identification with speaker's perspective in sections 6 and 7, we conduct further experiments to examine the difference between encoderonly and decoder-only models. in section 5.2, the decoder-only generative models showed better performance with speaker ground-truth labels than observer ground-truth labels in the congruous condition. this pattern is in contrast to the pattern we observed with encoder-only models, which demonstrated better performance with labels judged by the observers (section 5.1). one possible explanation for such difference is that generative models are more sensitive in interpreting speakers' "point-of-view" than encoderonly models. in section 5.2, the prompt for the generative models asking "how sarcastic is someone's response" could have biased the models to take the speaker's perspective by default. we investigated whether prompting the llms with more explicit instructions to take the perspective of an observer figure 3 : macro f-scores (y-axis) for sarcasm detection by generative llm models when explicitly prompted to take observer's perspective. results shown in two conditions (congruous vs. incongruous). figure 3 shows the prediction results of llama3.2-3b and qwen2.5-3b explicitly prompted to take the perspective of an external observer. for both models, the prediction performance does not increase with the new prompting method. when we manually inspect the responses by the llms, the general tendency of these models is that they provide plenty of descriptions about the emotions that the speaker would be experiencing, or the message that the speaker is trying to express (see appendix c for sample responses by the llms). even when instructed to take the observers' perspective, the models still focus on the speaker's experience in the conversation, and use the conclusion about this as a basis to determine an external observer's sarcasm judgment. 6 given these results, we detect a tendency that generative models identify with the speaker's perspective by default rather than observer's perspective. ## decoder-only vs. encoder-only: sensitivity to disagreement among observers in investigating the reason why generative models perform better with speaker ground-truth labels, another possibility is that generative models are more sensitive to the disagreement among multiple observers and therefore may struggle to per-form at their best when the ground-truth labels are the result of simple averaging. if true, the sensitivity would have influenced the results in section 5.2, in which the sarcasm labels by observers were averaged and binary-coded, which discards information about potential disagreement among the observers. annotator-wide disagreement in general is considered as an important topic in nlp, as ground-truth labels cannot always come down to one single judgment @xcite @xcite @xcite . to test this, we examined whether the disagreement among the observers influences the performance of the generative llms more than that of the encoder-only models. we quantified the level of disagreement among different observers using the normalized mae described in section 4.1. for the purpose of visual inspection, we split the scores we obtained using this formula at the mean value into low versus high. we inspected f-scores of both encoder-only models and generative models in the two groups of disagreement (low vs. high). both encoder-only models and generative models show better performance when human annotators agree on the sarcasm judgment (figures 4 and 5 ). for encoder-only models, the difference in f-score between the two groups (low vs. high) is comparable for both ground-truth labels (e.g., 0.80 vs. 0.57 in the left panel ≈ 0.76 vs. 0.61 in the right panel of figure 4 ). in contrast, for all generative models, the difference in f-score is larger for observer ground-truth when the disagreement is low versus high (e.g., 0.71 vs. 0.53 on bottom left panel in figure 5 ), than it is for speaker ground-truth (0.70 vs. 0.58 on bottom right panel in figure 5 ). this difference is bigger for llms with a larger number of parameters, which suggests their higher sensitivity to disagreement among observers. these results suggest that both types of language models are influenced by the disagreement among the observers. but in the face of this challenge, the generative models, especially those with more parameters, show a somewhat higher sensitivity by reacting against observer ground-truth to a greater extent than speaker ground-truth, which is in principle an expected behavior (i.e., in an ideal scenario, the performance with speaker ground-truth should see no change). this contrasts with encoderonly models, which show an equal drop against both sources of ground-truth labels and therefore demonstrates fragility to challenges stemming from human disagreement. ## experiment 3: sarcasm detection by llms with additional information sections 4 and 5 showed that speaker's annoyancesarcasm congruity influences the judgment of sarcasm both by human observers and llms. here we tested whether adding information about the speaker's annoyance to llms would then improve the classification results. ## general discussion when speakers use sarcasm without any noticeable emotional cues, external observers lose an important source of information for judging the level of sarcasm in the provided utterance (h1). this tendency in human observers is also reflected in llms. for both humans and llms, sarcasm is difficult to detect when the speakers' annoyance seems unmatched with the output utterance. in contrast, in sarcastic utterances where proportional annoyance can be perceived as an underlying motivation, models are better at detecting sarcasm. some differences are observed between encoder-only models and generative models, in terms of which source of ground-truth labels (speaker vs. observer) they match better. encoder-only models show better performance with observer ground-truth, in line with prior work @xcite . however, generative models show better performance with speaker ground-truth. a further analysis suggests that generative models may impersonate speakers' perspective by default compared to encoder-only models. this aligns with the capabilities that these models are expected to have, exemplified by one of the evaluation suites for llama-3 models "inhabiting a character/persona". 7 it is important for the evaluation of llm capabilities to investigate the patterns by which llms navigate through varying linguistic judgments by humans, especially on heavily subjective topics such as sarcasm. the findings in this work also suggest that future research should address both perspectives of conversational partners (speaker vs. listener) when evaluating llm output. examining which perspective is reflected in the output of llms would help understand the competence of llms in more depth. ## conclusion we showed that speaker's incongruity between their utterance and the annoyance they felt is associated with their judgment of sarcasm diverging from the judgment by external observers. this factor, as well as disagreement among the observers, also presented challenges to language models (both generative and encoder-only). lastly, we discovered that the generative models are more likely to impersonate speakers' perspective more than observers' perspective, in contrast to encoder-only models.