UTNLP at S em E val-2022 Task 6: A Comparative Analysis of Sarcasm Detection Using Generative-based and Mutation-based Data Augmentation (2022)

Abstract: Sarcasm is a term that refers to the use of words to mock, irritate, or amuse someone. It is commonly used on social media. The metaphorical and creative nature of sarcasm presents a significant difficulty for sentiment analysis systems based on affective computing. The methodology and results of our team, UTNLP, in the SemEval-2022 shared task 6 on sarcasm detection are presented in this paper. We put different models, and data augmentation approaches to the test and report on which one works best. The tests begin with traditional machine learning models and progress to transformer-based and attention-based models. We employed data augmentation based on data mutation and data generation. Using RoBERTa and mutation-based data augmentation, our best approach achieved an F1-score of 0.38 in the competition’s evaluation phase. After the competition, we fixed our model’s flaws and achieved anF1-score of 0.414.

## introduction billions of internet users use social networks not only to stay in touch with friends, meet new people, and share user-generated content but also to express their opinions on a wide range of topics using a variety of methods such as posting comments, videos, photos, etc. with specific groups of people @xcite . in these platforms, users could submit information on whatever topic they wanted, with no restrictions on the sort of content they may share. the lack of constraints and individuals' anonymity on these networks led to humorous sarcastic texts. because sarcasm indicates sentiment, detecting sarcasm in a text is critical for anticipating the text's accurate sentiment, making sarcasm detection a valuable tool with multiple applications in domains such as security, health, services, product evaluations, and sales. sarcasm detection is an * equal contribution essential aspect of creative language comprehension @xcite and online opinion mining @xcite . even for humans, identifying sarcasm is difficult due to heavily contextualized expressions @xcite . there are few labeled data resources for sarcasm detection. any available texts that can be collected (for example, tweets) contain many issues, such as an evolving dictionary of slang words and abbreviations, requiring many hours of human annotation to prepare the data for any potential use. furthermore, the nature of sarcasm identification adds to the task's difficulty, as sarcasm may be considered relative and varies significantly across people, depending on a variety of criteria such as the context, area, time, and events surrounding the statement. in an attempt to solve this issue, we participated in semeval-2022 shared task 6 @xcite , which aims to recognize whether a tweet is sarcastic or not. our contributions are as follows: with roberta (a robustly optimized bert pretraining approach), no preprocessing, and mutation-based data augmentation, our top result gets an f1-sarcastic of 0.38. however, we obtain better outcomes, with a 0.414 f1-sarcastic after fixing the problems of our proposed method. the rest of this paper is organized as follows. in section 2, we discuss the related work, section 3 introduces the dataset. in sections 4 and 5, we present our methodology and results, respectively. finally section 6 concludes the paper. ## related work we give a quick review of previous works on sarcasm detection in this part, followed by works on data augmentation. ## dataset we mostly used the isarcasm @xcite dataset in this study. in specific experiments, we integrated the primary dataset with various secondary datasets, including the sarcasm headlines dataset @xcite and sentiment140 dataset @xcite to increase the quantity of data and compensate for the lack of sarcastic data. for each dataset, the details are further discussed. it is worthy to mention that all of the supplementary datasets we included had a negative impact on our model's performance. we believe this was the result of a different data gathering method. because to the differing labeling process and domain, the distribution diverged from that of isarcasm. as a result, the following sections are solely dependent on the isarcasm dataset, with no other datasets being used. ## methodology in this study we examined and analyzed various models and data augmentation strategies for sarcasm detection. first, we go through data augmentation methods; then, we discuss the structure and hyperparameters of these models in this section. the codes of all models are available on github foot_0 . ## results in this section we report the results of our models introduced in section 4. it's important to note that after the competition, we discovered that none of our preprocessing strategies improved the performance of our model. so we were able to get an f1-sarcastic of 0.414 without using any preprocessing methods, which was 0.034 higher than our performance in the competition, which was based on the best combination of preprocessing methods. ## conclusion in this study, we reviewed and contrasted a number of sarcasm detection methods. to improve the performance of our model, we experimented with two different types of augmentation. in the job of sarcasm detection, we observed that mutation-based data augmentation can assist us in achieving better results than generative-based data augmentation. additionally, we tested with other deep-learning techniques, including rnn and bert-based models. our best system, an ensemble model, has an f1-sarcastic of 0.414.