From A rabic Sentiment Analysis to Sarcasm Detection: The A r S arcasm Dataset (2020)

Abstract: Sarcasm is one of the main challenges for sentiment analysis systems. Its complexity comes from the expression of opinion using implicit indirect phrasing. In this paper, we present ArSarcasm, an Arabic sarcasm detection dataset, which was created through the reannotation of available Arabic sentiment analysis datasets. The dataset contains 10,547 tweets, 16% of which are sarcastic. In addition to sarcasm the data was annotated for sentiment and dialects. Our analysis shows the highly subjective nature of these tasks, which is demonstrated by the shift in sentiment labels based on annotators’ biases. Experiments show the degradation of state-of-the-art sentiment analysers when faced with sarcastic content. Finally, we train a deep learning model for sarcasm detection using BiLSTM. The model achieves an F1 score of 0.46, which shows the challenging nature of the task, and should act as a basic baseline for future research on our dataset.

## introduction work on subjective language analysis, has been prominent in the literature during the last two decades. a major theme that dominated the area is the work on sentiment analysis (sa). according to @xcite , sa is a process where we extract and analyse the emotional polarity in a given piece of text. large amount of work focused on classifying the text into its sentiment class, which varies based on the granularity. sa is one of the research areas within the larger natural language processing (nlp) field. the interest in sa research was embarked by the advent of user-driven platforms such as social media websites. research on sa started with the early work of @xcite , where they analysed the sentiment in movie reviews. since then, the work has developed and spanned different topics and fields such as social media analysis, computational social science and others. most of the work is focused on english, whereas arabic did not receive much attention until after 2010. the work on arabic sa was kicked off by @xcite , but it still lacks behind the progress in english. this can be attributed to the many challenges of arabic language; including the large variety in dialects @xcite and the complex morphology of the language @xcite . as the work on sa systems developed, researchers started analysing the intricacies of such systems in order understand their performance and where they fail. there are many challenges when doing sa, such as negation handling, domain dependence, lack of world knowledge and sarcasm @xcite . sarcasm can be defined as a form of verbal irony that is intended to express contempt or ridicule @xcite . sarcasm is correlated with expressing the opinion in an indirect way, where the intended meaning is different from the literal one @xcite . additionally, sarcasm is highly context-dependent, as it al-ways takes part between parties where shared knowledge exist. usually, a speaker will not use sarcasm unless he/she thinks that it will be understood as so @xcite . sarcasm detection is a crucial task for sa. the reason for this is that a sarcastic utterance usually carries a negative implicit sentiment, while it is expressed using positive expressions. this contradiction between the surface sentiment and the intended one creates a complex challenge for sa systems @xcite . there has been lots of work on english sarcasm detection, those include datasets such as the works of @xcite @xcite @xcite and detection systems such as @xcite @xcite . work on arabic sarcasm is yet to follow. up to our knowledge, work on arabic sarcasm is limited to the work of @xcite , a shared task on irony detection @xcite along with the participants' submissions and a dialectal sarcasm dataset by @xcite . currently, there is no publicly available dataset for arabic sarcasm detection. the data in @xcite is not publicly available and most of the tweets provided in @xcite were deleted. in this paper, we present arsarcasm dataset, a new arabic sarcasm detection dataset. the dataset was created using previously available arabic sa datasets and adds sarcasm and dialect labels to them. the dataset contains 10,547 tweets, 1,682 (16%) of which are sarcastic. in addition, we analyse annotators' subjectivity regarding sentiment annotation, hoping to promote finding better procedures for collecting and annotating new datasets. the analysis shows that annotators' biases could be reflected on the annotation. moreover, we provide an analysis of the performance of sa systems on sarcastic content. finally, our bilstm based model , which serves as a baseline for this dataset, achieves an f1-score of 0.46 on the sarcastic class, which indicates that sarcasm detection is a challenging task. arsarcasm is publicly available for research purposes, and it can be downloaded for free foot_0 . ## proposed dataset in this work, we present arsarcasm, a new dataset for arabic sarcasm detection. the dataset consists of a combination of arabic sa datasets, where we reannotated them for sarcasm. in addition to that, we also provide labelling for the dialect and sentiment. ## effect of sarcasm on sentiment analysis to better understand how sarcasm can be disruptive for sa systems, we conducted an experiment on the newly annotated data. this was done through comparing the performance of an available sa system on both sarcastic and non-sarcastic tweets. in this experiment, we used mazajak (abu farha and magdy, 2019), state-of-the-art arabic sentiment analyser. in order to have an informative comparison, we separated the dataset into two sets, sarcastic id tweet original label new label 1 (google is competing apple and samsung with a new phone) positive neutral 2 .. (congratulations on windows 10, god keeeeep it for you) positive negative 3 (shame, they are playing a justin bieber song) neutral negative 4 . (sir, we will respond to you soon) neutral positive 5 .. (apple's reputation is on the line ... a real problem in iphone 7) negative neutral 6 (deceitful weather, they say it will snow and it is warm) negative positive ## sarcasm detection baseline system in this section, we conduct an experiment to set a baseline system for the new dataset. we tested a deep learning model, which consists of a bidirectional long shortterm memory (bilstm) followed by a fully connected layer. we used the hyper-paremeters shown in ## conclusion and future work sarcasm is an important aspect of any language. it includes expressing ideas, opinions and emotions in an indirect implicit way. this nature of implicitness makes sarcasm problematic for sa systems which mostly rely on the surface meaning/features. in this work, we presented arsarcasm, a new arabic sarcasm dataset. the dataset was created through the reannotation of available arabic sentiment datasets. the new dataset contains sarcasm, sentiment and dialect labels. analysis shows that sarcasm is highly prominent in sentiment datasets with 16% of them being sarcastic. we also show the high subjective nature of such datasets, which was demonstrated by the change in sentiment labels in the new annotation. the experiments show the gap between sa systems' performance on non-sarcastic tweets compared to sarcastic tweets, which urges the need to study such phenomena. finally, our initial experiments on sarcasm detection show that it is a challenging task. we believe that this dataset is a starting point in the direction of full study of sarcasm and figurative language in arabic. however, due to the highly subjective nature of sarcasm, its reliance on world knowledge, cultural background and the perspectives of the communication parties, we believe that the data collection procedure should incorporate more signals about these information. in the future, we hope to prepare a new dataset that incorporates more textual information. we also hope to study and analyse the differences and similarities among sarcastic expressions used by arabic speakers in different countries.