D artmouth at S em E val-2022 Task 6: Detection of Sarcasm (2022)

Abstract: This paper introduces the result of Team Dartmouth’s experiments on each of the five subtasks for the detection of sarcasm in English and Arabic tweets. This detection was framed as a classification problem, and our contributions are threefold: we developed an English binary classifier system with RoBERTa, an Arabic binary classifier with XLM-RoBERTa, and an English multilabel classifier with BERT. Preprocessing steps are taken with labeled input data prior to tokenization, such as extracting and appending verbs/adjectives or representative/significant keywords to the end of an input tweet to help the models better understand and generalize sarcasm detection. We also discuss the results of simple data augmentation techniques to improve the quality of the given training dataset as well as an alternative approach to the question of multilabel sequence classification. Ultimately, our systems place us in the top 14 participants for each of the five subtasks.

## introduction sarcasm is a form of irony that occurs when there is a discrepancy between the literal and intended meanings of a text or utterance. this discrepancy typically manifests itself in the form of dislike, contempt, or derogation. furthermore, sarcasm can be divided into multiple types: general sarcasm, irony, satire, understatement, overstatement, and rhetorical question. this task concerns itself with the detection of sarcasm in online tweets @xcite . this is an important issue to solve because the nature of sarcasm can interfere with the effectiveness of natural language processing models, particularly when conducting sentiment analysis, opinion mining, or other emotion-based tasks. machine learning models deployed for such business use cases can be negatively impacted when processing sarcastic texts and provide inaccurate results, thereby harming an organization's bottom line. therefore, it is critical that machine learning models be developed that can understand how to detect, ingest, and truly understand sarcasm. this paper discusses how to identify sarcastic tweets in binary and multi-label classification contexts for both english and arabic, as directed by @xcite . there are five subtasks: general sarcasm detection in standalone english and arabic tweets (task a), identification of sarcastic category in an english tweet (task b), and identification of the sarcastic tweet in an english/arabic pair of tweets (task c). our experiments show that pre-trained transformer models demonstrate a strong ability of solving most of the subtasks of this challenge. specifically, we fine-tune a roberta base model to detect the presence of sarcasm in tweets for the english components of tasks a and c. for the arabic version of tasks a and c, we apply an xlm-roberta base model. for task b, which is framed as a multilabel sequence classification problem, after experimenting with several roberta and bert models, we report the performance of a bert base model which is fine-tuned to detect the categories of sarcasm in english tweets, if any. while these models perform reasonably well in the evaluations, the imbalanced distributions of labels and poor annotation quality for some instances introduce unexpected noise to the fine-tuning process of these models and harm their performance in the evaluations. despite our effort to augment unrepresented classes in the training data, simple data augmentation approaches do not show clear positive effects on the models' performance. more advanced data augmentation methods could be tried to trigger notable performance improvements on the challenge test dataset. ## approaches as mentioned above, there are three systems designed for the five subtasks. we discuss the model architecture, input processing, and other key elements for each of the systems below. ## experimental results ## conclusion and future work in this paper, we have described three binary and multilabel sequence classification systems using the bert, roberta, and xlm-roberta architectures from huggingface for the detection of sarcasm in english and arabic tweets. we found that additional work to augment the training data with duplication of sentences and manually/automatically synthesizing new sarcastic sentences did not improve the results of the model. furthermore, challenges were observed with the multilabel classifier in learning to extract the key characteristics that categorize a tweet as a distinct example of satire, understatement, or overstatement -categories which were generally underresourced in the training dataset. further investigation could include implementing six binary classifiers instead of a single multilabel sequence classifier for task b. given enough training time, data, and resources, it could certainly be the case that aggregating results across specialized binary classifiers provide more concrete results than what has been produced. in particular, this may allow each of the binary classifiers to more deeply learn the unique characteristics, keywords, and structure of the sarcastic sentences it ingests. it would also be interesting to see how the results across all three systems change with sufficient training data, with perhaps tens of thousands of more valid examples that can allow the models to truly capture the essence of sarcasm across a wide and varied set of training examples.