S arcasm D et at S em E val-2022 Task 6: Detecting Sarcasm using Pre-trained Transformers in E nglish and A rabic Languages (2022)

Abstract: This paper presents solution systems for task 6 at SemEval2022, iSarcasmEval: Intended Sarcasm Detection In English and Arabic. The shared task 6 consists of three sub-task. We participated in subtask A for both languages, Arabic and English. The goal of subtask A is to predict if a tweet would be considered sarcastic or not. The proposed solution SarcasmDet has been developed using the state-of-the-art Arabic and English pre-trained models AraBERT, MARBERT, BERT, and RoBERTa with ensemble techniques. The paper describes the SarcasmDet architecture with the fine-tuning of the best hyperparameter that led to this superior system. Our model ranked seventh out of 32 teams in subtask A- Arabic with an f1-sarcastic of 0.4305 and Seventeen out of 42 teams with f1-sarcastic 0.3561. However, we built another model to score f-1 sarcastic with 0.43 in English after the deadline. Both Models (Arabic and English scored 0.43 as f-1 sarcastic with ranking seventh).

## introduction in recent years, sarcasm has received remarkable research attention due to its importance and extraordinary impact on society, especially with the significant increase in its use in social media @xcite @xcite . however, sarcasm is saying something, and what is meant is the opposite. adding to the difficulty of detecting sarcasm is that all the words in the sentence are positive, but the intended meaning is just the opposite @xcite . therefore, the detection of sarcasm depends mainly on the general meaning of the sentence, which makes us move away from traditional methods and use artificial intelligence, especially natural language processing. natural language processing (nlp) is a branch of artificial intelligence that gives machines the ability to read and understand the meanings of hu-mans @xcite . as a result, machines became able to understand the underlying meanings of words and not just rely on keywords. moreover, it helps reveal forms of speech, such as emotion analysis, humor, ridicule, abuse, etc @xcite . task 6 at semeval-2022, "isarcasmeval: intended sarcasm detection in english and arabic" @xcite suggested three main subtasks in both arabic and english languages: sub-task a is a binary classification problem that determines whether a tweet is sarcastic or not, subtask b is a multi-label classification problem which determines the category of tweets sarcastic if it is found in english only, subtask c is two sentences, one sarcastic and the other paraphrased to be nonsarcastic, define which one is sarcastic. we only participated in subtask a. our solution combines four state-of-the-art pre-trained nlp models: arabert and marbert for arabic and bert and roberta for english. sarcasmdet placed seventh out of 32 teams in the subtask a-arabic and seventh (after the competition deadline) out of 42 teams. we have experimented with the pre-trained language models with different hyperparameters using the simple transformers library. it is worth mentioning that using the ensemble technique has increased our score remarkably. the paper is constructed as follows: section 2 provides the related works. section 3 describes the shared task and the provided dataset. section 4 describes our system solution. section 5 shows our experiments. section 6 provides the results, and finally, the conclusion is in section 7. ## related work sarcastic texts over social media are among the most researched issues of importance for sentiment analysis. authors of @xcite proposed sarcasm classification using a behavioral modeling approach. they investigate users' past tweets and use psychological studies to better sarcasm detection of tweets. they used three models to perform their evaluation (logistic regression (lr) decision tree (dt) and support vector machine (svm)). the evaluation is performed on a dataset obtained from twitter and provided on kaggle with 9104 sarcastic tweets. they used different class distributions (1:1, 10:90, and 20:80) and 10-fold cross-validation techniques. the best performance was obtained from the lr model and class distribution (1:1) with 83.46% accuracy. in 2016, researchers of @xcite provided pattern-based features approach to detect sarcasm expressions posted on twitter which extracted with the hashtag sarcasm. they applied four models (random forest (rf), svm, k-nearest neighbour (knn), and max-entropy (max.ent.)) on three different balanced datasets with different sizes (1000, 2256, and 6000 instances). rf model outperformed the rest with 81.3% in terms of f1score. in this paper @xcite , researchers almost did the same work; they used the pattern-based approach and the same classifiers (rf, svm, and knn). the used dataset consists of 9104 sarcastic tweets. rf outperforms the rest in accuracy and f1-score with 81% and 79%, respectively. the focus on detecting sarcasm in arabic has emerged in recent years, and researchers have become increasingly interested in this field. in @xcite , the authors were among the beginning people to supply a dataset to detect sarcasm in arabic. first, they collected the dataset from twitter of the arabic tweets with different arabic dialects, such as the egyptian, syrian, and saudi dialects. then, they cleaned the dataset, 5,479 tweets, including 1733 irony, and added four features for each tweet: surface, sentiment, shifter, and internal context features. finally, they applied several machine learning algorithms, and experiments showed that the random forest classifier achieved a high accuracy of 72.76% for detecting sarcasm in arabic tweets. on the other hand, another dataset is provided for sarcasm in arabic by @xcite . this dataset contains three topics: sarcasm, sentiment, and dialect labels include 10,547 rows of tweets where 16% are sarcastic tweets. the researchers applied a bidirectional lstm deep learning approach (bilstm) to achieve an f1-score of 0.46, which indicates the difficulty of detecting sarcasm in the arabic language. the researchers in (faraj and abdullah, 2021) participated in the wanlp 2021 shared task for subtask 1 (sarcasm detection) competition. first, they used the arasarcasm-v2 dataset and then cleaned data by using nltk library @xcite , such as normalized and removed emojis, links, and html tags. next, they implemented several pre-trained models such as arabert, multilingual bert cased and uncased, and xlm-roberta. their solution, called sarcasmdet, is based on finetuning of the large arabert and base arabert, and they got first place with an accuracy of 0.7830 and fourth place with f1-sarcastic 0.5989. ## task and data description in shared task on semeval 2022 -task 6 (isar-casmeval): intended sarcasm detection in english and arabic @xcite , has three subtasks in two languages: arabic and english, and each task solves different requirements. we participated in subtask a: a binary classification problem that determined whether a tweet contains sarcasm or not. the dataset consists of nine columns: the tweet, the rephrase, sarcastic, which defines whether the tweet is sarcastic or non-sarcastic, irony, stair, understatement, overstatement, and the rhetorical question. the last five columns are the type of sarcasm contained in the tweets. moreover, the dataset contains 3467 non-null rows. regarding our task, a binary sarcasm classification of the tweets, the tweet, and the sarcasm are the only required columns to solve the task. the arabic training dataset contains 3102 tweets and five columns: id, tweet, sarcastic, rephrase, and dialect. they have categorized the text into sarcastic or non-sarcastic based on the author himself. ## sarcasmdet description texts are sequential, so they must be trained by models supporting data in which the order of its features is an important factor. transformers are deep learning techniques that utilize the idea of self-attention mechanism @xcite . in this work, two transformer-based pre-trained models are fine-tuned to achieve the requirements of the sarcasm detection task for each language. the two models are the bert and the roberta for english, where we combined two pre-trained models, arabert and marbert, for arabic. subtask a -english figure 1 shows the architecture of the proposed model for english. the ensemble technique is adopted by performing a weighted sum for the predictions of bert and roberta. one bert machine with weight one and four roberta machines, each with weight one, is deployed except for one machine with a weight of 0.5. regarding the bert model, the dataset is tokenized by a pre-defined tokenizer of the model. the bert model originally consists of 12 layers, the first ten layers are chosen to be untrainable and the last two layers to be trainable. moreover, the model is trained using a batch size equal to 6 and the number of epochs equal to 3. as to the roberta model, the dataset is tokenized using its pre-defined tokenizer. the layers are fine-tuned. the first nine layers are untrainable, the last three layers are trainable, and the model is trained using a batch size equal to 12 and the number of epochs equal to 3. arabert is a pre-trained model that focuses directly on the arabic language, and it is based on the bert architecture @xcite . there are two versions of arabert(v01 and v02). the first version, arabert-v01, was trained on 77m sentences, with a size of 23gb and 2.7b of words. the second version, arabert-v02, was trained on 200m sentences with 77gb and 8.6b words. marbert @xcite is also based on the bert architecture without the next sentence prediction and focuses on dialectal arabic and msa. in sarcasmdet, tweets are fed to the arabert and marbert. next, we added the final layer, which is fine-tuning with the best hyperparameters as shown in ## conclusion sarcasm is an influential issue in human life, whether written or spoken. however, sarcasm detection in texts is a challenging task. this paper presented our model sarcasmdet for solving subtask a-english and arabic in task6 at semeval 2021 -isarcasmeval: intended sarcasm detection in english and arabic. sarcasmdet is based on the fine-tuning of two pre-trained nlp models for each language and then applied ensemble technique to improve the model. the models trained on a dataset obtained from a competition semeval 2022 subtask a of task 6. for english subtask a, the dataset consists of 3467 records, 866 of them are sarcastic, and the rest are non-sarcastic. the results showed that the roberta model outperformed the bert model, where the ensembling technique outperformed both in the f1-sarcastic score, which was 0.43. on the other hand, the arabic dataset consists of 8k tweets divided into training and testing sets. our solution sarcasmdet is ranked 7th out of 32 teams with an f1-sarcastic score of 0.4305.