S a‘7r: A Saudi Dialect Irony Dataset (2022)

Abstract: In sentiment analysis, detecting irony is considered a major challenge. The key problem with detecting irony is the difficulty to recognize the implicit and indirect phrases which signifies the opposite meaning. In this paper, we present Sa‘7r ساخرthe Saudi irony dataset, and describe our efforts in constructing it. The dataset was collected using Twitter API and it consists of 19,810 tweets, 8,089 of them are labeled as ironic tweets. We trained several models for irony detection task using machine learning models and deep learning models. The machine learning models include: K-Nearest Neighbor (KNN), Logistic Regression (LR), Support Vector Machine (SVM), and Naïve Bayes (NB). While the deep learning models include BiLSTM and AraBERT. The detection results show that among the tested machine learning models, the SVM outperformed other classifiers with an accuracy of 0.68. On the other hand, the deep learning models achieved an accuracy of 0.66 in the BiLSTM model and 0.71 in the AraBERT model. Thus, the AraBERT model achieved the most accurate result in detecting irony phrases in Saudi Dialect.

## introduction the content of social media networks such as twitter shows unlimited daily feeds of millions of users' interactions @xcite . the massive amount of data attracts researchers to conduct several types of data and textual analysis for different purposes, such as detecting opinions, sentiment and irony. one of the fundamental nlp tasks is detecting ironic expressions which is considered one of the complex language phenomena and was widely studied in linguistics, philosophy, and psychology @xcite @xcite . although several studies were conducted on irony from different perspectives, the definition of irony has not reached a consensus yet @xcite . one of the obstacles to defining irony is that irony has various vocabularies that undergo language changes @xcite . in addition, the irony definition is affected by the variation of regional languages and dialects @xcite . on the other hand, the literature shows a similar term to irony, which is sarcasm, and some studies used both terms interchangeably @xcite . on the contrary, many studies tackled the delusion problem of sarcasm and irony, such as @xcite . kreuz and his colleague defined sarcasm as a prominent victim and the target of ridicule, whereas, in irony, there is no individual or victim. in addition, ironic language is usually less cruel, harmful, and aggressive than sarcasm. however, due to the high similarity between sarcasm and irony definitions and the complexity of distinguishing between the two phonenma, we considered, in this paper, both terms as synonym to define any expression in which a person uses words that deliver the opposite of literal meaning. detecting ironic expressions is important and fundamental, especially in sentiment analysis @xcite . the automatic detection of irony can assist many essential domains, such as gaining business insights into public opinion to improve certain services. moreover, detecting ironic expressions can help identify threats and distinguish between fake and real threatening messages @xcite . although social media companies provide analytic tools to analyze the vast amount of data available, these tools do not provide the best accuracy when applied to some text that contains irony and sarcasm or hidden meaning @xcite . detecting this type of speech is considered difficult, especially in the arabic language, because of its complexity and variations of the arabic written styles. additionally, arabic language is also considered a challenging language in the field of nlp, due to its morphological richness, orthographic ambiguity and inconsistency, and dialectal variations @xcite . in this paper, we focus on collecting tweets for saudi dialect to build an irony dataset extracted from twitter. this work has two main contributions: the rest of the paper is organized as follows: section 2 gives an overview of related work in the area of irony and sarcasm detection for arabic language. section 3 presents the dataset generation stages, including data collection, dataset annotation, dataset statistics, and dataset evaluation. the experiment results and evaluation are described in section 4. we then discuss the challenges faced through the experiment in section 5. finally, in section 6, we concluded the paper with suggestions for future works. ## related work detecting sarcasm and irony in the textual contents has been extensively studied in different languages, especially the english language. the increasing popularity of shared tasks for irony detection and sentiment analysis has increased the interest in this field and attracted more researchers to develop robust irony detection tools. the first shared task for irony detection in english tweets was proposed in 2018 (van hee, lefever, and hoste 2018, 20), the organizers proposed fine-grained multiclassification task on different types of irony instead of binary classification. a more profound analysis of linguistic phenomena of the ironic expression has been proposed by @xcite ) that analyzes different linguistic categories of irony in different languages in the social media contents. this approach was established by implementing a multilingual corpus annotated based on a multilayered schema to measure the impact of different pragmatic phenomena used in the expression of irony in three indo-european languages, including english, french, and italian. the efficiency of neural networks has been investigated to detect sarcastic texts @xcite ) by implementing a model composed of convolutional neural networks (cnn), long short term memory (lstm), and deep neural network (dnn) to detect sarcasm over social media contents, the proposed model compared against svm-based models and showed an improvement for the neural networks. another work @xcite applying deep learning techniques to detect sarcasm in the twitter news dataset, the proposed model was implemented based on the convolutional-recurrent neural network (c-rnn) to discover sarcastic pattern detection and achieved an accuracy of 84.73%. for the arabic language, there have been few papers that tackled sarcasm and irony detection in the arabic language. twitter is the most widely used source for data collection for detecting irony due to the huge amount of textual and the large availability of ironic texts among different languages and cultures. one of the earliest studies was conducted by @xcite and @xcite where they both used supervised learning algorithm to develop a classifier model. @xcite used naïve bayes multinomial text algorithm for detecting tweets and the model evaluation achieved 0.659 in recall, 0.71 in precision, and 0.676 in f-score. while @xcite used random forest with gainratio algorithm to detect irony in arabic tweets and achieved an accuracy of 72.36%. similarly, @xcite proposed a system based on several language models: word-n-grams, topic models, sentiment models, statistical models, and embeddings of words. in addition, bi-lstm, random forest, and xgboost were some of the classifiers that were used to evaluate the system. based on the f1-score, the proposed system achieved 0.85. also another submission has achieved 81.7% and 79.4% for two different neural networks models for word embedding respectively. recently, there has been renewed interest in detecting sarcasm and irony with a dedication on constructing datasets for arabic ironic language. in a shred task conducted by @xcite , they released arsarcasm-v2 dataset, which consists of 15,548 tweets labelled for sarcasm, sentiment and dialect. the shared task received 27 submissions for the sarcasm detection subtask. among the techniques used in the shared task is the work @xcite . they used a deep multitask learning model to develop a model that allows knowledge to be accessed for sarcasm detection. their work incorporated bert model and multitask attention interaction module into a single model architecture which produced a better performance in detecting sarcasm. furthermore, @xcite proposed an approach which consists of two phases: the dataset preprocessing phase which involves inserting, deleting, and segmenting various fragments of the text. the second phase was experimenting with two transformer-based models araelectra and arabert. author found out that arabert has the highest weighted f1-score while araelectra has the worst weighted f1score and accuracy. in addition, abuzayed and al-khalifa (2021) employed seven bert-based models which are: marbert, arbert, qarib, arabertv02, gigabert, arabic bert and mbert, also to fix the problem of imbalanced data they combined the shared task dataset with additional information. ameur and aliane (2021) created a sarcasm and sentiment detection dataset for arabic tweets during the pandemic, the dataset is called "aracovid19-ssd". they collected 5,162 tweets that are annotated with two labels related to the two tasks: sarcasm detection (yes or no) and sentiment analysis (positive, negative, or neutral). they used three pre-trained transformer models for classification (arabert, mbertm and xlm-roberta) and other supervised models (svm, lr, and random forest). their experiments showed that the svm and arabert models performed better than other models by reaching an f1-score of more than 95%. another work proposed by @xcite , they collected arabic tweets for sarcasm detection. the prediction task was tackled as a regression problem instead of a classification problem by quantifying the level of sarcasm for a given tweet instead of deciding if a tweet is sarcastic or not. the experiment was evaluated using mean squared error (mse) as a loss function and it obtained a 0.011 loss value. ## experiments and results in this section we conducted different experiments to set a baseline system for the new dataset. we started with a set of machine learning algorithms, then a classifier built using word embeddings vectorization technique with bilstm, lastly, we tested a bert-based model. we have split the dataset into two parts, the first is the training set, which represents 80% of the data which equals 15843 entries, while the testing set represents the remaining 20% of the data which equals 3961 entries. for the evaluation, we used the f1-score to compare the results of all the models, since f1-score delivers a realistic score that does not get affected by the data imbalance @xcite . ## conclusion in the era of social media, irony detection is considered a challenging and important task to understand a person's intentions. in this paper, we presented a new arabic irony detection dataset for the saudi dialects called sa`7r foot_1 . we collected the corpus from twitter using specific hashtags, keywords, and phrases related to irony based on the saudi dialects. we plan to expand this dataset to include more linguistic content in the future. additionally, we would like to determine whether there are any similarities and differences between ironic arabic expressions used by speakers in other countries. in terms of modelling, we aim to solve the dataset imbalance in order to obtain more accurate results with a model trained with balanced dataset, we also aim to manipulate the hyperparameters of the bilstm model so that we can enhance the models learning abilities. for the transformer-based model, other options of pretrained arabic bert-