Sarcasm Detection is Way Too Easy! An Empirical Comparison of Human and Machine Sarcasm Detection (2022)

Abstract: Recently, author-annotated sarcasm datasets, which focus on intended, rather than perceived sarcasm, have been introduced. Although datasets collected using first-party annotation have important benefits, there is no comparison of human and machine performance on these new datasets. In this paper, we collect new annotations to provide human-level benchmarks for these first-party annotated sarcasm tasks in both English and Arabic, and compare the performance of human annotators to that of state-of-the-art sarcasm detection systems. Our analysis confirms that sarcasm detection is extremely challenging, with individual humans performing close to or slightly worse than the best trained models. With majority voting, however, humans are able to achieve the best results on all tasks. We also perform error analysis, finding that some of the most challenging examples are those that require additional context. We also highlight common features and patterns used to express sarcasm in English and Arabic such as idioms and proverbs. We suggest that to better capture sarcasm, future sarcasm detection datasets and models should focus on representing conversational and cultural context while leveraging world knowledge and common sense.

## introduction sarcasm is a form of verbal irony that is often used to express ridicule or contempt. sarcasm is usually correlated with expressing an opinion in an indirect way where there would be a discrepancy between the literal and intended meaning of an utterance @xcite . for example the sentence "i love being ignored" is an example of sarcasm, where there is a discrepancy between the positive surface meaning and the negative implied one. sarcasm is present on the social web and, due to its nature, it can be disruptive to computational systems that harness this data to perform tasks such as sentiment analysis, opinion mining, author profiling, and hate-speech detection @xcite @xcite . @xcite show a significant drop in sentiment polarity classification performance when processing sarcastic tweets, compared to non-sarcastic ones. such computational systems are widely deployed in industry, driving marketing, administration, and investment decisions @xcite . while much of the computational sarcasm detection work focuses on the english language, in the context of @xcite show the effect of sarcasm on arabic sentiment analysis systems, where the performance dropped significantly for the sarcastic tweets. as such, it is imperative to devise models for sarcasm detection in languages other than just english. doing this, however, requires reliable evaluation datasets, and further, understanding where current models (and even humans) fail on these datasets. most of the previous sarcasm datasets have been created using either distant supervision or manual labelling. those approaches produce unreliable labels since rule-based systems can suffer from sampling bias, and third-party annotators do not know whether the intention of the author was to truly be sarcastic. recently, new datasets that contain first-party labels have been released. one method to collect such data is reactive supervision, where conversational cues such as "i was being sarcastic" are used as labels of previous comments @xcite . this increases reliability by identifying texts in which the authors do claim to be sarcastic, but since texts are sampled according to predefined patterns, the data may be biased toward cases that required clarification. yet another method for collecting first-party labels even more directly is to ask authors to provide explicit annotations of their own texts @xcite . this approach eliminates annotation proxies, further reduces sampling and annotator bias, and allows for the collection of additional data about each sarcastic text, such as explanations and rephrases. although more difficult than earlier sarcasm detection tasks, classification models have shown promising performance on first-party annotated datasets, as evidenced by a recent shared task @xcite . however, there is currently no analysis of how humans would perform on these datasets. this kind of analysis can provide a range of benefits, including empirical insights into the difference in quality between third-party and first-party annotations. additionally, most of previous works focused on building resources and detection models, without as much attention being paid to error analysis. these analyses are necessary to give insights about the limitations of the current best models, and pave the way to mitigate these limitations in the future. in this paper, we aim to fill this gap and answer the following research questions: in this paper, we answer these questions by measuring both human and machine performance on isarcasmeval's datasets @xcite , which have first-party sarcasm labels. we make the following contributions: (1) we collect new human annotations for the isarcasmeval datasets; (2) we analyse both humans' and state-of-the-art (sota) models' performance on both english and arabic tasks, identifying cases where each succeeds and fails; (3) we analyse the error cases in order to determine the current limitations of sarcasm detection methods; and (4) we provide recommendations, based on our empirical evidence, for improving sarcasm detection models in the future. our analysis shows that sarcasm detection is challenging for humans, who perform nearly as well as state-of-the-art models on their own, and even better when their annotations are combined through majority voting. however, human performance using third-party labels is still imperfect, and casts doubt on its utility as a source of ground truth for this task. we find that context and world knowledge are necessary to understand sarcasm in many cases. thus, future works on sarcasm should focus on including this kind of information into datasets and leveraging it in detection models. ## related work previous work on sarcasm detection falls into one of two branches: creating datasets @xcite @xcite @xcite @xcite or creating detection models @xcite @xcite @xcite @xcite @xcite . most, but not all, of this work has focused on sarcasm detection in english. in this section, we cover the literature of sarcasm detection in both english and arabic, since these are the languages represented in our dataset. ## results and analysis in this section, we compare the performance of humans against state-of-the-art models. the task organisers agreed to share these teams' detailed submissions, including individual predictions for each item of the test set, allowing us to perform this comparison. we consider comparing the human predictions with the top-performing system for each subtask, as well as with a combination of the top five performing systems using majority voting. however, in all cases, we find that using the output from the single top team for the subtask outperformed the combination of the top five. therefore, we only compare the human predictions with the single 4 best performing model in each subtask. for subtask a (english), yuan et al. (2022) were ranked first with an f sarcastic roberta @xcite . for arabic, el @xcite were ranked first with an f sarcastic 1 of 0.563. they used an ensemble of models based on marbert @xcite . for subtask c, @xcite were ranked first on english with an accuracy of 0.870. they used an ensemble of ernie-m @xcite and deberta. for arabic, the top team was @xcite with an accuracy of 0.930. their model is based on arabic bert @xcite . ## discussion here we revisit and answer our research questions and provide some additional discussion. rq1: how do humans perform on authorannotated sarcasm detection tasks? human annotations from non-authors of the text are vastly different from the labels provided by the authors themselves. this suggests that there is an important difference between intended and perceived sarcasm, as suggested by @xcite , who also argue that first-party annotations are more reliable as being sarcastic is an intentional act. not only for sarcasm, but also for which intention and perception may not be consistent, the use of thirdparty annotations has serious implications for the reliability of our datasets' ground truth: as shown in our results, there are cases where these annotations do not align with the labels provided by the texts' authors themselves. rq2: how does human performance on these tasks compare with state-of-the-art text classifiers? we found that on their own, humans performed almost as well as the state-of-the-art sarcasm detection systems, but when working together using majority voting, humans achieve the best results. on the other hand, using majority voting for the systems led to worse performance. this suggests that humans provide complementary knowledge when it comes to the task of sarcasm detection, while the text classification models' predictions typically have high overlap with one another. rq3: what makes sarcasm challenging for both humans and classification models? for english, we observed that many of the less challenging sentences were those with clear sentiment incongruity, e.g., "i love failing exams", or sentences criticising governmental figures by sarcastically presenting them in overly positive contexts. for arabic, idioms and proverbs were quite common (around 23% of sarcastic sentences in the test set) and detected easily by humans and the models. it seems that arabic speakers rely heavily on proverbs and the dense meaning they contain, while having a shared cultural context. the other common arabic sarcasm pattern is to say " you look like/act animal/object", which is often used for derogatory remarks. however, the most challenging cases on both languages were the sentences that require additional context. this can be in the form of conversational context, information about the author of the text, world knowledge, and dialectal awareness in the case of arabic. for example, the model was better than humans in detecting sarcasm in sentences that used local words which some of the humans may be unfamiliar with. given that marbert(abdul-mageed et al., 2021) was trained on 1b tweets, it seems that it had better coverage of some specific dialects than the annotators. interestingly, this model was able to classify levantine examples better than the maghrebi ones, which could be attributed to a possible bias in the training data. the fact that missing context led to a large num-ber of error cases implies that it is necessary for detection systems to have representations for world knowledge and cultural background, and be aware of the language/dialect of the sarcastic utterance. progress in this direction is possible: for example, dialectal awareness can be addressed by using language models that are trained on a large dialectal variety. ## recommendations based on analyses and discussions in the previous sections, we make the following recommendations: avoid 3rd-party annotations. we should reevaluate third-party annotation as a method to create datasets of subjective content, particularly when author intention is important. the analysis of the performance of human annotators on two languages, english and arabic, shows that their performance is comparable to state-of-the-art models. however, the performance of both the models and humans still has much room for improvement. develop models that incorporate context. in order to improve the performance of detection models, we need to better incorporate contextual information such as cultural references, author tendencies, world knowledge, and dialect awareness. the need for this has been demonstrated through the fact that both humans and models failed to detect sarcasm that relied on such information. include contextual features in shared datasets. further, in order to train these models, sarcasm detection datasets that contain a wealth of contextual information should be created and released, especially conversational and author-level information which cannot be obtained from external knowledge bases. however, releasing this type of data brings new challenges in the space of privacy, as conversations contain texts written by other authors who may not have consented to sharing their content, and including more author-level information may lead to deanonymization and loss of privacy. build accurate representations of idioms and proverbs. more focus should be put into building accurate representations of idioms and proverbs, which are extensively used in sarcastic communication, especially by arabic speakers. ## conclusions in this paper, we analyse human performance on sarcasm detection and compare it to state-of-the-art detection models. we use semeval's 2022 task 6 datasets, which has first-party sarcasm labels for both english and arabic texts. our analyses show that sarcasm detection is challenging for humans on both languages with performance only slightly better than trained models, and only when using majority voting between the human predictions. the low human performance emphasizes the subjective nature of sarcasm and indicates that third-part labels for subjective tasks are noisy. consequently, we urge the community to re-evaluate third-party annotations for extremely subjective tasks, such as sarcasm, and use first-party labels. we conduct a thorough error analysis, revealing that the most challenging sarcastic sentences are those that require additional contextual information to accurately resolve, suggesting that future work focus on creating context-rich datasets and models with the ability to adequately leverage contextual information. our analyses show that idioms and proverbs common linguistic tools used to express sarcasm, especially in arabic, yet trained models often struggle with examples that contain them.