Context vs. Human Disagreement in Sarcasm Detection (2024)

Abstract: Prior work has highlighted the importance of context in the identification of sarcasm by humans and language models. This work examines how much context is required for a better identification of sarcasm by both parties. We collect textual responses to dialogical prompts and sarcasm judgment to the responses placed after long contexts, short contexts, and no contexts. We find that both for humans and language models, the presence of context is generally important in identifying sarcasm in the response. But increasing the amount of context provides no added benefit to humans (long = short > none). This is the same for language models, but only on easily agreed-upon sentences; for sentences with disagreement among human evaluators, different models show different behavior. We also show how sarcasm detection patterns stay consistent as the amount of context is manipulated despite the low agreement in human evaluation.

## introduction and related work this work examines the role of the presence and amount of contextual information in detecting sarcasm. previous work in cognitive science has shown the importance of context in sarcasm comprehension @xcite and production @xcite for humans. in computational linguistics, similar observations were made: supplying context to the target utterance boosts sarcasm detection performance of language models, though with more conflicting results: some studies report that supplying context leads to a performance boost in sarcasm detection by neural models @xcite , whereas other studies report no such benefit @xcite or marginal benefit @xcite in using context for the same task. however, there has not been much effort in exploring the benefit of varying amounts of contextual information, or in addressing what counts as context. the term 'context' varies a lot work by work; it can mean any number of preceding strings such as previous posts on social media @xcite or previous utterances in a dialogue @xcite , or any additional information that can help detect sarcasm, such as eye-tracking data @xcite or images @xcite . in this work, we define context as the preceding textual utterances that can trigger sarcasm in people (section 2), and then examine what is a good amount of contextual information that facilitates sarcasm identification for humans (section 3) and language models (section 4). we further show how context interacts with the level of disagreement among human evaluators (section 4.3). ## data creation we created a new dataset based on the multimodal sarcasm detection dataset (mustard; @xcite . the mustard dataset contains written transcriptions of "contexts" (preceding utterances) and the following "response" foot_0 from multiple tv series, and binary labels of sarcasm for the responses (sarcastic or not sarcastic). we selected 24 contexts that are generalizable enough, all of which were from the tv series 'friends' and situations happening between two conversation partners. the names of all conversation partners were modified to detach the stimuli from the tv show as much as possible. for all the selected contexts, we collected new responses in an online data collection. here, we manipulated the amount of context. additional to the original contexts available in a short utterance form, we described each context in a narrative form by manually referring to the scenes and episodes of the tv show to restore the relevant information that would allow the following utterance to be correctly judged as sarcastic or not. a. stimuli generation b. sarcasm evaluation of collected responses read evaluate read type 12 long contexts (lc) 12 short contexts (sc) response (r) x18 x3 per stimulus lc sc nc r 8x 8x 8x lc examples sc r crofton and his friends are at a bar. ryan starts crying and tells the group that his marriage is officially over as of today. everybody encourages him to go to a strip club since he is now single. ryan, still crying, says "i don't want to be single. i just, i just want to be married again!", and right then, a woman in a wedding dress walks into the bar. ryan: i don't want to be single. i just, i just want to be married again. there you go ryan, a bride for you! 1 2 4 5 6 ev1 ev3 ev2 1 2 3 5 6 1 3 4 5 6 @xmath0, episode-level, or series-level information not reflected in the transcripts. therefore, each context was represented twice both as short context (sc) in its original utterance form and as long context (lc) in a descriptive/narrative form. the average number of words was 26 for sc and 66 for lc. for each lc and sc, we collected new responses to make the stimuli comparable, given that the original dataset had responses only to short contexts. this also allowed us to collect spontaneous responses from multiple lay people as opposed to responses generated by professional screenwriters. we recruited 32 native english-speaking participants based in the uk, usa, canada, australia, new zealand or ireland foot_1 . they read 24 contexts and freely responded to each (they were not instructed to be sarcastic). half of the contexts (@xmath1. at the end of the collection, participants reported their familiarity to the tv show friends and how many of the situations they recognized as being from the show. to control for the expectation of sarcasm arising from the familiarity to the tv show, we discarded data from the participants who were quite familiar, very familiar, or extremely familiar to the show or who recognized at least 3 scenes from the show. after removing data from 14 such participants, data 3 influence of context for sarcasm judgment by humans here we identify what amount of context affects human judgment of sarcasm on the following response. ## influence of context on sarcasm detection by large language models here we test if manipulating the amount of context directly affects the performance of three language models in the detection of sarcasm on the following response. as gold standard we use the humanevaluated scores described in section 3. ## conclusion this work systematically tested the amount of contextual information required for humans and language models to evaluate the following utterance in terms of sarcasm. we showed that in general, the presence of context leads to better detection of sarcasm both by humans and by three lms. but, providing a higher amount of information in the context did not present clear additional benefit for humans, which was also true for lms for sentences for which human evaluators agreed on a label. however, when humans disagreed, the performance of language models improved when a longer context was provided. we lastly showed that low inter-rater agreement did not affect the overall classification patterns, due to a high variability in the sentences that the models misclassify each time they are tested against labels from different human evaluators. this is a relevant finding for many nlp tasks prone to disagreement and susceptible to subjectivity, which must continue to be addressed in future research.