S arcasm D et at Sarcasm Detection Task 2021 in A rabic using A ra BERT Pretrained Model (2021)

Abstract: This paper presents one of the top five winning solutions for the Shared Task on Sarcasm and Sentiment Detection in Arabic (Subtask-1 Sarcasm Detection). The goal of the task is to identify whether a tweet is sarcastic or not. Our solution has been developed using ensemble technique with AraBERT pre-trained model. We describe the architecture of the submitted solution in the shared task. We also provide the experiments and the hyperparameter tuning that lead to this result. Besides, we discuss and analyze the results by comparing all the models that we trained or tested to achieve a better score in a table design. Our model is ranked fifth out of 27 teams with an F1 score of 0.5985. It is worth mentioning that our model achieved the highest accuracy score of 0.7830

## introduction sarcasm, bullying, and offense are critical topics that dramatically increase in social media, leading to countless problems, whether on the personal or community level. irony and sarcasm use positive words commonly applied in public life to make fun, ridicule, or annoy someone, either directly or indirectly @xcite . detecting sarcasm depends on the sentence context and understanding the current situation, which can be challenging to be discovered using traditional techniques. with the development of artificial intelligence (ai) methods, specifically natural language processing (nlp), predicting and reducing such words that denote sarcasm are investigated thoroughly using sentiment and emotion analysis methodologies @xcite @xcite . sentiment analysis (sa) refers to the use of natural language processing, text analysis, and linguistics to identify, extract, quantify, and systematically determine whether a sentence or a document is positive or negative @xcite . sa techniques have been applied to utilize the process of predicting whether the text contains sarcasm and offensive words within it @xcite . the lack of resources and annotated datasets for the arabic language is considered one of the challenges in this field @xcite . for detecting sarcasm, a group of researchers @xcite provided new datasets for the shared task on sarcasm and sentiment detection in arabic (wanlp 2021). the shared task includes two subtasks: the sarcasm detection sub-task for detecting if the tweet is sarcastic or not. the sentiment analysis sub-task, which is focused on analyzing tweets then identifying their sentiment. this paper presents the sarcasemdet team model at the wanlp 2021 shared task for subtask 1 (sarcasm detection). our solution system (sarcasmdet) is one of the top five teams among 27 participated teams. the proposed approach uses the pre-trained language model arabert @xcite . we also have experimented with several pre-trained language models using the simple transformers library. it is worth mentioning that using the hard-voting ensemble technique has increased the performance of the model, remarkably. the rest of the paper goes as follows. section 2 overviews and discusses the existed work related to this research. section 3 illustrates the shared task and describes the dataset. section 4 displays our solution architecture and preprocessing method to achieve the best accuracy score in this task. section 5 presents the experiments with hyperparameters tuning analysis. section 6 presents the results. finally, in section 7, the conclusion. ## related work there are numerous resources on sarcasm detection in english and european languages, but there is a lack of resources and datasets in the arabic language. the focus on detecting sarcasm in the arabic language emerged in recent years. in @xcite , the researchers were among the first people to provide a dataset to detect sarcasm in arabic. they collected arabic tweets of different arabic dialects, such as egyptian, syrian and saudi dialects. they cleaned the dataset, 5,479 tweets, including 1733 irony, and added four features for each tweet: surface, sentiment, shifter, and internal context features. the experiments showed that the random forest classifier achieved high accuracy with 72.76% to detect sarcasm in the arabic language tweets. another sarcasm dataset in the arabic language is provided by @xcite . this dataset contains sarcasm, sentiment, and dialect labels incorporating 10,547 rows of tweets where 16% are sarcastic tweets. the researchers applied the bidirectional lstm (bilstm) deep learning approach to achieve an f1-score of 0.46, which indicates the hardness of detecting sarcasm in arabic. various machine learning mechanisms have been used to detect sarcasm and offensive in arabic tweets. in @xcite , the authors used the supervised naïve bayes multinomial text algorithm to detect the sarcasm in arabic tweets. they collected 344 rows of tweets data manually, where 238 out of them are sarcastic tweets, and 106 are nonsarcastic tweets. to evaluate the nb model, they used recall, precision, f1-score metrics, and the resultant outputs are 0.659, 0.710, and 0.676, respectively. in @xcite , the researchers participated in the semeval 2020 task 12 arabic offensive language dataset for two subtasks: offensive language detection and hate speech detection. they were the winning solution system for discovering offensive language in the arabic language. the dataset they used contains 10,000 rows of tweets data manually labeled for offensiveness (off or not off). they applied several deep and machine learning algorithms, such as dnn, cnn, rnn, and svm. they achieved the best results with an f1-score of 90.51% by ensembling svm and dnn. ## task and dataset description in the shared task on sarcasm and sentiment detection in arabic from wanlp 2021 @xcite challenge, all tasks have different requirements. the shared task includes two subtasks: subtask 1, sarcasm detection, which aims to detect if the tweet is sarcastic or not. subtask 2, sentiment analysis, focused on analyzing tweets then identifying their sentiments. table1 shows a sample of the training dataset. it is worth mentioning that we have participated in subtask 1. in our solution system, we have used the arsarcasm-v2 dataset provided by @xcite . the data is separated into two parts: a training dataset containing 12,548 tweets and four features labeled as follows: tweet, sarcasm, sentiment, and dialect. the second is the test dataset containing 3,000 tweets and two features labeled as a tweet and dialect. table1 shows a description of the 15,548 rows of data. one of the challenges in this task is the imbalanced dataset that contains 2,168 sarcastic tweets and 10,380 nonsarcastic tweets. figure 1 shows the distribution of classes in the training set. data preprocessing used are from the nltk library @xcite , such as normalized duplicated characters and cleaning data from emojis, links, and html tags. ## sarcasmdet description in our approach, we have used the arabic pretrained model called @xcite . arabert is a pre-trained model that focuses directly on the arabic language, and it is based on the bert architecture @xcite . the arabert model's strength is being one of the best language models for nlp and sentiment analysis for the arabic language. there are two versions of @xcite . the first version, arabert-v01, was trained on 77m sentences, with a size of 23gb and 2.7b of words. the second version, arabert-v02, was trained on 200m sentences with a size of 77gb and 8.6b words. in our approach, we have used both versions that are located in huggingface library @xcite . we have experimented with several hyperparameters and the best result achieved from arabert-v02 is 0.5650 f1-score for the sarcastic class (f1sarcastic), with learning @xmath0, manual @xmath1, train batch @xmath2. to enhance our results, we have used the ensemble technique, hard-voting @xcite . the best-ensembled model, sarcasmdet, achieved an f1-sarcastic (f1-score for sarcastic class) with 0.5989 in the testing phase. sarcasmdet has ensembled the arabert-v01 and arabert-v02 models. figure 2 shows the architecture of sarcasmdet in detail. ## experiments we have experimented with several deep learning models to detect sarcastic arabic tweets through our training and testing phases. one of the experimented models is xlm-roberta(xlm-r) @xcite , which performed less than other models in this task. the multilingual bert (mbert) model obtained one of the best results in our experiments because of multilingual support. the arabert was the one that captured the highest score with the v02 model. all of the models have been implemented using the huggingface library and simpletransforner pre-trained package. some models, such as xlm-r, have default learning @xmath3, but we changed it to 1e-5, which increased the results. ## results and discussion our solution was separated into two phases. the first phase was the development phase. we have experienced several deep learning models xlm-r, arabert(v01,v02), mbert(cased/uncased) by implementing them into the dataset to make the solution that suits the next phase, which is the test phase. in the test phase, our model performed great by using the ensemble technique and was ranking in the top 5 models. we believe that the arabert-v02 model has an excellent understanding of the arabic language and sentence analysis. we have noticed that, with or without data preprocessing and hard-voting ensemble techniques, the model outperformed other models remarkably. we expect the exceeding of the arabert-v02 since it is already trained and tested on a massive amount of arabic data. in the test phase section, we showed our experiments and the results that we achieved. figure3 shows a plot chart for the best results. ## conclusion our paper showed and described our approach that achieved the best accuracy score model in the shared task on sarcasm and sentiment detection in arabic (sub-task 1 -sarcasm detection). we have implemented and experimented with several nlp-based language models, such as xlm-r, mbert, and arabert, with hard-voting en-semble techniques to detect and solve the sarcasm in arabic tweets. our last and best solution is to ensemble the four models scoring f1-sarcastic 0.5650, 0.5459, 0.5560, 0.5419 by applying hardvoting ensemble technique. the sarcasmdet model achieved 0.5989 f1-sarcastic score, which outperformed the baseline model (0.3993 f1-sarcastic). our proposed model has earned 5th place among 27 teams.