A Report on the 2020 Sarcasm Detection Shared Task (2020)

Abstract: Detecting sarcasm and verbal irony is critical for understanding people’s actual sentiments and beliefs. Thus, the field of sarcasm analysis has become a popular research problem in natural language processing. As the community working on computational approaches for sarcasm detection is growing, it is imperative to conduct benchmarking studies to analyze the current state-of-the-art, facilitating progress in this area. We report on the shared task on sarcasm detection we conducted as a part of the 2nd Workshop on Figurative Language Processing (FigLang 2020) at ACL 2020.

## introduction sarcasm and verbal irony are a type of figurative language where the speakers usually mean the opposite of what they say. recognizing whether a speaker is ironic or sarcastic is essential to downstream applications for correctly understanding speakers' intended sentiments and beliefs. consequently, in the last decade, the problem of irony and sarcasm detection has attracted a considerable interest from computational linguistics researchers. the task has been usually framed as a binary classification task (sarcastic vs. non-sarcastic) using either the utterance in isolation or adding contextual information such as conversation context, author context, visual context, or cognitive features @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite . in this paper, we report on the shared task on sarcasm detection that we conducted as part of the 2nd workshop on figurative language processing (figlang 2020) at acl 2020. the task aims to study the role of conversation context for sarcasm detection. two types of social media content are used as training data for the two tracks -microblogging platform such as twitter and online discussion forum such as reddit. section 2 discusses the current state of research on sarcasm detection with a focus on the role of context. section 3 provides a description of the shared task, datasets, and metrics. section 4 contains brief summaries of each of the participating systems whereas section 5 reports a comparative evaluation of the systems and our observations about trends in designs and performance of the systems that participated in the shared task. turns message context 1 this is the greatest video in the history of college football. context 2 hes gonna have a short career if he keeps smoking . not good for your health response awesome !!! everybody does it. that's the greatest reason to do something. ## related work a considerable amount of work on sarcasm detection has considered the utterance in isolation when predicting the sarcastic or non-sarcastic label. initial approaches used feature-based machine learning models that rely on different types of features from lexical (e.g., sarcasm markers, word embeddings) to pragmatic such as emoticons or learned patterns of contrast between positive sentiment and negative situations @xcite @xcite @xcite @xcite @xcite . recently, deep learning methods have been applied for this task @xcite . for excellent surveys on sarcasm and irony detection see @xcite . however, when recognizing sarcastic intent even humans have difficulties sometimes when considering an utterance in isolation @xcite . recently an increasing number of researchers have started to explore the role of contextual information for irony and sarcasm analysis. the term context loosely refers to any information that is available beyond the utterance itself @xcite . a few researchers have examined author context @xcite @xcite @xcite , multi-modal context @xcite @xcite , eye-tracking information @xcite , or conversation context @xcite @xcite @xcite . related to shared tasks on figurative language analysis, recently, van hee et al. (2018) have con-ducted a semeval task on irony detection in twitter focusing on utterances in isolation. besides the binary classification task of identifying the ironic tweet the authors also conducted a multi-class irony classification to identify the specific type of irony: whether it contains verbal irony, situational irony, or other types of irony. in our case, the current shared task aims to study the role of conversation context for sarcasm detection. in particular, we focus on benchmark the effectiveness of modeling the conversation context (e.g., all the prior dialogue turns or a subset of the prior dialogue turns) for sarcasm detection. ## task description the design of our shared task is guided by two specific issues. first, we plan to leverage a particular type of context -the entire prior conversation context -for sarcasm detection. second, we plan to investigate the systems' performance on conversations from two types of social media platforms: twitter and reddit. both of these platforms allow the writers to mark whether their messages are sarcastic (e.g., #sarcasm hashtag in twitter and "/s" marker in reddit). the competition is organized in two phases: training and evaluation. by making available common datasets and frameworks for evaluation, we hope to contribute to the consolidation and strengthening of the growing community of researchers working on computational approaches to sarcasm analysis. @xcite introduced the self-annotated reddit corpus which is a very large collection of sarcastic and non-sarcastic posts (over one million) curated from different subreddits such as politics, religion, sports, technology, etc. this corpus contains self-labeled sarcastic posts where users label their posts as sarcastic by marking "/s" to the end of sarcastic posts. for any such sarcastic post, the corpus also provides the full conversation context, i.e., all the prior turns that took place in the dialogue. ## systems the shared task started on january 19, 2020, when the training data was made available to all the registered participants. we released the evaluation data on february 25, 2020. submissions were accepted until march 16, 2020. overall, we received an overwhelming number of submissions: 655 for the reddit track and 1070 for the twitter track. the codalab leaderboard showcases results from 39 systems for the reddit track and 38 systems for the twitter track, respectively. out of all submissions, 14 shared task system papers were submitted. in the following section we summarize each system paper. we also put forward a comparative analysis based on their performance and the choice of features/models in section 5. interested readers can refer to the individual teams' papers for more details. but first, we discuss the baseline classification model that we used. ## results and discussions context usage: one of the prime motivating factors for conducting this shared task was to investigate the role of contextual information. we notice the most common approach for integrating context was simply concatenating it with the response text. novel approaches include : leveraging other datasets the large difference between the best model (miroblog) and other systems can be attributed to their dataset augmentation strategies. using just the context thread as a negative example when the context+response is a positive example, is a straight-forward approach for augmentation from labeled dialogues. their novel contribution lies in leveraging large-scaled unlabelled dialogue threads, showing another use of bert by using nsp confidence score for assigning pseudo-labels. ## conclusion this paper summarizes the results of the shared task on sarcasm detection using conversation from two social media platforms (reddit and twitter), organized as part of the 2nd workshop on the figurative language processing at acl 2020. this shared task aimed to investigate the role of conversation context for sarcasm detection. the goal was to understand how much conversation context is needed or helpful for sarcasm detection. for reddit, the training data was sampled from the standard corpus from @xcite whereas we curated a new evaluation dataset. for twitter, both the training and the test datasets are new and collected using standard hashtags. we received 655 submissions (from 39 unique participants) and 1070 submissions (from 38 unique participants) for reddit and twitter tracks, respectively. we provided brief descriptions of each of the participating systems who submitted a shared task paper (14 systems). we notice that almost every submitted system have used transformer-based architectures, such as bert and roberta and other variants, emphasizing the increasing popularity of using pre-trained language models for various classification tasks. the best systems, however, have employed a clever mix of ensemble techniques and/or data augmentation setups, which seem to be a promising direction for future work. we hope that some of the teams will make their implementations publicly available, which would facilitate further research on improving performance on the sarcasm detection task.