aka BERT at S em E val-2022 Task 6: An Ensemble Transformer-based Model for A rabic Sarcasm Detection (2022)

Abstract: Due to the widespread usage of social media sites and the enormous number of users who utilize irony implicit words in most of their tweets and posts, it has become necessary to detect sarcasm, which strongly influences understanding and analyzing the crowd’s opinions. Detecting sarcasm is difficult due to the nature of sarcastic tweets, which vary based on the topic, region, the user’s attitude, culture, terminologies, and other criteria. In addition to these difficulties, detecting sarcasm in Arabic has its challenges due to its complexities, such as being morphologically rich, having many different dialects, and having low resources. In this research, we present our submission of (iSarcasmEval) sub-task A of the shared task on SemEval 2022. In Sub-task A; we determine whether the tweets are sarcastic or non-sarcastic. We implemented different approaches based on Transformers. First, we fine-tuned the AraBERT, MARABERT, and AraELECTRA. One of the challenges that faced us was that the data was not balanced. Non-sarcastic data is much more than sarcastic. We used data augmentation techniques to balance the two classes, significantly affecting the performance. The performance F1 score of the three models was 87%, 90%, and 91%, respectively. Then we boosted the three models by developing an ensemble model based on hard voting. The final performance F1 Score was 93%.

## introduction according to the cambridge dictionary definition of sarcasm, it is the use of remarks that mean the opposite of what they say, made to hurt someone's feelings or humorously criticize something, like when you say love this weather. (when the weather is horrible). sarcasm detection is determining whether or not a piece of text is sarcastic. sarcasm is a significant challenge for sentiment analysis systems. this is because a sarcastic sentence usually contains an implicit negative sentiment that is expressed with positive expressions. this discrepancy between the surface and intended sentiments creates a difficult challenge for sentiment analysis systems. sarcasm detection is a difficult task for a variety of reasons. first of all, there aren't many labeled data resources for sarcasm detection. moreover, any available texts that can be collected (for example, tweets) contain many issues, such as an evolving dictionary of slang words and abbreviations, so it usually takes many hours for human annotators to prepare the data for any potential use. furthermore, the nature of sarcasm detection adds to the task's difficulty, as sarcasm can be considered relative and varies significantly between people, and it depends on many factors such as the topic, region, time, the events surrounding the sentence, and the readers/writers mentality and the; in other words, a sentence that one person finds sarcastic may sound normal to another. @xcite in addition to these previous challenges, discovering irony in the arabic language has its own set of challenges due to the complexities of the language, such as being formally rich, different dialects, lack of resources, and rapid development due to the inclination of the arabic language. the arab citizen makes fun of all his affairs, especially politics, which uses many words and terms that are implicit in it. transformers greatly assisted in significant advancements in nlp tasks. they are a new neural network that does not employ convolution or recursion. they instead use their attention to find correlations between words in the text. transformers can process text in parallel, allowing them to learn much faster than sequential methods. they also outperform previous methods in terms of results. transformer-based language models have re-cently proven to be highly efficient at language understanding, giving promising results across various nlp tasks and benchmark datasets. the language modeling capability of these models aids in capturing the literal meaning of context-heavy texts. for arabic nlp in particular, the best results for sentiment analysis are currently achieved by arabert, a language model proposed by @xcite . despite recent advances, detecting sarcasm remains a difficult task because of implicit, indirect phrasing and the symbolic nature of language. when working with twitter data, the task becomes even more difficult because the social media posts are often short and contain noise sources, codeswitching, and the use of nontraditional dialectal variations. furthermore, bert-based models have struggled with rare words, which are more common in social media texts due to their informal nature and the prevalence of slang words. it is difficult for language models like arabert, which have been trained on structured corpora from wikipedia. @xcite in this study, we tackle (isarcasmeval) sub-task a of the shared task on semeval 2022.in subtask a; we determine whether the tweets are sarcastic or non-sarcastic. we have proposed models based on transformers to discover the sarcasm that transformers have proven to excel in other nlp tasks such as sentiment analysis. we summarize what has been accomplished in this research in the following: the following sections are organized as follows; section 2 presents the background, section 3 describes the task and dataset description, section 4 gives an overview of the proposed system, section 5 explores the results and discussion, and section 6 concludes and gives possible directions for the future. ## background there are few attempts to work on arabic sarcasm. the workshops in the field of natural language processing for the arabic language revived the interest in detecting sarcasm, as this workshop provided annotated datasets, which was considered a challenging obstacle in front of researchers. such as the previous shared tasks on irony detection @xcite along with the participants' submissions and dialectal sarcasm datasets by @xcite ; @xcite ; @xcite . this survey mainly focuses on the wanlp 2021 workshop and its shared task on sarcasm in arabic. this shared task seeks to promote and draw attention to arabic sarcasm detection, which is critical for improving performance in other tasks such as sentiment analysis. the dataset used in this collaborative task, arsarcasm-v2, comprises 15,548 tweets that have been labeled for sarcasm, sentiment, and dialect. subtask 1 on sarcasm detection received 27 submissions. the majority of approaches relied on using and fine-tuning pretrained language models like arabert and mar-bert (abdel-salam, 2021 the bhamnlp(alharbi and lee, 2021) team was ranked best in the sarcasm detection task, with an f1-score 0.6225. they deployed a multi-task learning architecture trained for sarcasm and sentiment classification in their approach. the model is based on both a marbert and a cnn-lstm model, with each model's output being concatenated and supplied to the final output layer. the cnn-lstm utilized both word and character embeddings. ## task and dataset description in semeval 2022 task 6: isarcasmeval (intended sarcasm detection in english and arabic) @xcite . the organizer offered two datasets, arabic and english. there are three subtasks. subtask a its aim is to determine whether the given text is sarcastic or non-sarcastic; subtask b is applied in (english only): it aims to determine which ironic speech category the given text belongs to, if an. it's a binary multi-label classification task. subtask c: it aims to determine which text is the sarcastic one; if it is given two texts that convey the same meaning, we have to clarify that our proposed system is a solution to subtask a on the offered arabic dataset. the arabic dataset consists of 3,102 tweets with 2357 non-sarcastic tweets and 745 sarcastic ones. the test set contains 1400 tweets. it was released without labels for evaluation purposes. the dataset is provided with dialects and sarcasm labels. ## system overview the proposed model comprises two main phases: data augmentation and the ensemble module, consisting of three pre-trained transformers(arabert, marabert, araelctra) and the voting module, as shown in this figure 2 . ## conclusion we presented our submission on the shared subtask a on arabic sarcasm detection isarcasmeval 2022 to tackle the problem of detecting sarcasm in arabic. we explored different pre-trained models based on bert. we noticed that the performance of araelectra, marabert, and the ensemble model is greatly improved after data augmentation and balancing the sarcastic and non-sarcastic tweets. the best submission model was the ensemble model; it applies hard voting on the arabert, marabert, and araelctra. in the future, we plan to improve the performance of the proposed model by understanding and identifying which features are essential that contribute to enhancing the model prediction and troubleshooting unexpected model outputs.