Plumeria at S em E val-2022 Task 6: Sarcasm Detection for E nglish and A rabic Using Transformers and Data Augmentation (2022)

Abstract: The paper describes our submission to SemEval-2022 Task 6 on sarcasm detection and its five subtasks for English and Arabic. Sarcasm conveys a meaning which contradicts the literal meaning, and it is mainly found on social networks. It has a significant role in understanding the intention of the user. For detecting sarcasm, we used deep learning techniques based on transformers due to its success in the field of Natural Language Processing (NLP) without the need for feature engineering. The datasets were taken from tweets. We created new datasets by augmenting with external data or by using word embeddings and repetition of instances. Experiments were done on the datasets with different types of preprocessing because it is crucial in this task. The rank of our team was consistent across four subtasks (fourth rank in three subtasks and sixth rank in one subtask); whereas other teams might be in the top ranks for some subtasks but rank drastically less in other subtasks. This implies the robustness and stability of the models and the techniques we used.

## introduction sarcasm is a figurative language where speakers or writers usually mean the contrary of what they say. recognizing whether a speaker or writer is sarcastic is essential to downstream applications to understand the sentiments, opinions, and beliefs correctly @xcite . sarcasm is ubiquitous on the social media text and, due to its nature, can be highly divisive of computational systems that perform tasks on that kind of data such as sentiment analysis, opinion mining, and harassment detection @xcite @xcite . our team plumeria participated in semeval 2022 task 6 @xcite in all its subtasks on english and arabic. previous shared tasks * these authors contributed equally to this work on sarcasm detection @xcite @xcite have only two subtasks; one is sarcasm detection and another is predicting the type of sarcasm. however, in semeval 2022 task 6 @xcite organizers formulate three subtasks for both languages following the methods described in @xcite . using the two datasets for english and arabic, organizers formulate three subtasks as follows: in this shared task, our submitted systems primarily focused on the transformer based approaches because of their success in the field of nlp. the multi-head attention mechanism in transformers captures the relations between the words in a sentence which helps in identifying sarcasm. moreover, to capture long-term dependencies between words, especially the contradicting ones, we used a hierarchical network by stacking a bilstm layer on top of a transformer. in order to emphasize the important tokens afterwards, we tried adding a dotproduct attention layer to give different weights to the tokens. for prediction, the final information are passed to a fully connected layer followed by a linear layer with a softmax activation function for classification (i.e. subtask a and c) or a sigmoid activation function for multi-label classification (i.e. subtask b). the major constraint of a neural network is that they need lots of data for training to give satisfactory results. in addition, if the dataset is imbalanced with few instances of a class, this can result in poor results for detecting instances that belong to the class. this motivates us to create biased datasets, towards the concerned class/label, from existing datasets by increasing the number of instances of such a class/label. a detailed explanation of dataset creation is in section 3. we also illustrate the composition of created datasets in each subtask and the performance of the models on them. the rank of our team was consistent across most subtasks (fourth rank in three subtasks, sixth rank in one subtask, and tenth rank in one subtask); unlike many teams which scored high in one or two subtasks but scored considerably less in other subtasks. this shows that the robustness and the consistency of our methods. the biased datasets were crucial for subtask a and subtask b, while augmenting a dataset plays a key role in subtask c. we released the codes and datasets for all subtasks via github foot_0 . the paper is organized as follows. section 2 lists some abbreviations used across the paper. section 3 shows the datasets we used for fine-tuning the models. in section 4 we list the preprocessing types applied on the datasets. the experiments and results are presented in section 5, and the analysis of the results is presented in section 6. this is followed by a conclusion in section 7. ## abbreviations the following abbreviations were used frequently, and are shown in ## datasets the organizers provided datasets for english and arabic. regardless of the dataset, these are the fields in each row of a dataset: in the following sections we will describe the datasets given by the organizers, other available datasets, and augmented datasets. ## analysis 6.1 subtask a english: there are 1400 instances in the test set out of which our model correctly classified 1060 instances (@xmath0. there are bigger number of misclassified negative (ns) instances (@xmath1.44) and the recall (77.5). this can be due to the fact that our model was trained on b4 dataset (59% s and 41% ns) taking f1-score for the sarcastic class as the metric for evaluation. this made the model focus more on identifying the positive instances sacrificing the considerable number of misclassified negative instances. we dived into the details to see when the model predicted well and when it could not predict properly. we found that the majority of tweets which have specific punctuation marks like the exclamation mark were classified correctly. short tweets were not classified properly which can be due to the insufficient information present in the tweets for classification. interestingly, the existence of emojis highly increased the recall but not the f1 score and this is because sarcastic tweets can be easily recognized from their emojis but this does not apply on non-sarcastic ones. also, tweets which include opposite emotions tend to be more sarcastic and to give better f1-score. moreover, we found that tweets which contain misspellings or that need human knowledge to interpret can cause misclassification. examples of all previous cases are presented in arabic: there are 1400 instances in the test set out of which our model correctly classified 903 instances (@xmath2. there are bigger number of misclassified negative (ns) instances (@xmath3.69) and the recall (85). the number of instances which contain exclamation marks are less compared to english, and short tweets also were not classified properly for the same reason. tweets that contain emojis were classified poorly and this is because we used type ii preprocessing which converted the emojis to text similar to the preprocessing of the model we used for english "bertweet-large" and unlike the preprocessing of the model we used for arabic "camelbert-mix". ## conclusion the paper describes our participation in semeval-2022 task 6. the models used for sarcasm detection were mainly stand-alone transformers. in addition to this, we ran other experiments by stacking a bilstm layer with or without attention mechanism on top of the transformers. we created new datasets for each subtask by augmenting with external datasets, word embedding, or repetition. our results shows that the augmented datasets enhanced the results for most subtasks. moreover, we found that the fine-tuned stand-alone transformers gave the best results especially with type ii preprocessing. we also showed the enhancement when using a weighted loss function and the effect of using different learning-rates, epochs, and preprocessing types. we gave analysis of the performance of the models for each subtask, and revealed the possible cases that might have enhanced or deteriorated the performance. finally, the rank of our team is consistent across most of the subtasks (the fourth rank) which shows the robustness of the used techniques. 2. irony: tweets that contradict the state of affairs but are not obviously critical towards an addressee.