Overview of the WANLP 2021 Shared Task on Sarcasm and Sentiment Detection in A rabic (2021)

Abstract: This paper provides an overview of the WANLP 2021 shared task on sarcasm and sentiment detection in Arabic. The shared task has two subtasks: sarcasm detection (subtask 1) and sentiment analysis (subtask 2). This shared task aims to promote and bring attention to Arabic sarcasm detection, which is crucial to improve the performance in other tasks such as sentiment analysis. The dataset used in this shared task, namely ArSarcasm-v2, consists of 15,548 tweets labelled for sarcasm, sentiment and dialect. We received 27 and 22 submissions for subtasks 1 and 2 respectively. Most of the approaches relied on using and fine-tuning pre-trained language models such as AraBERT and MARBERT. The top achieved results for the sarcasm detection and sentiment analysis tasks were 0.6225 F1-score and 0.748 F1-PN respectively.

## introduction work on opinion mining and subjective language analysis has been prominent in the natural language processing (nlp) field during the last two decades. one of the main tasks in this area is sentiment analysis (sa). one of the early works on sa is @xcite , where the authors analysed the sentiment in movie reviews. following that, and embarked with the popularity of social media, sa became one of the popular topics in nlp. most of the work on sa targeted english, while other languages, including arabic, lagged behind. in the last decade, researchers on arabic nlp started targeting sa such as the work of @xcite . since then, there have been numerous works on arabic sa such as the works of @xcite @xcite @xcite . work on arabic sa has been hindered by many challenges such as the large variation in dialects @xcite and the complex morphology of the language @xcite . with the advancement of work on sa, researchers started tackling the challenges affecting this task such as sarcasm @xcite . sarcasm can be defined as a form of verbal irony that is intended to express contempt or ridicule @xcite . sarcasm is considered one of the main challenges for sa systems since it implies expressing the opinion in an indirect way, where the intended meaning is different from the literal one @xcite . there have been several related works on english sarcasm detection including datasets such as the works reported in @xcite @xcite @xcite and detection systems such as @xcite @xcite . currently, there are few attempts to work on arabic sarcasm. those include the work by soukhria2017, a shared task on irony detection @xcite along with the participants' submissions and dialectal sarcasm datasets by @xcite . in this shared task, we offer our sarcasm and sentiment detection in arabic task that is co-organised with the wanlp 2021 workshop on arabic nlp. the goal of the shared task is to provide resources and encourage researchers to work on arabic sarcasm detection. the shared task has two subtasks, sarcasm detection (subtask 1) and sentiment analysis (subtask 2). we provided the participant with a new dataset (arsarcasm-v2), which is publicly available 1 . the dataset is annotated for sarcasm, sentiment and dialect. we received 27 submissions for subtask 1 and 22 submissions for subtask 2. this paper provides an overview of the shared task and the achieved results by the participants along with their approaches. most of the approaches used by participants were based on fine-tuning pre-trained language models. a small number of participants utilised other deep learning and conventional machine learning algorithms. the top team in the sarcasm detection task was bhamnlp @xcite , who achieved an f1-score of 0.6225 over the sarcastic class. while the top team in the sentiment analysis task was cs-um6p @xcite , who achieved f p n 1 of 0.748. ## related work our shared task offers two subtasks on arabic sarcasm and sentiment detection. in the following, we discuss the literature in both tasks within the arabic nlp community. ## dataset the shared task provides the arsarcasm-v2, which is a new dataset for arabic sarcasm detection. the dataset is an extension of the original arsarcasm dataset (abu farha and magdy, 2020). ## shared task this section provides an overview of the shared task, the description of the subtasks and the evaluation metrics. ## conclusion and future directions this paper provides an overview of the shared task on sarcasm detection and sentiment analysis in ara-bic. we provide an overview of the current state of research on arabic sarcasm. the paper provides an overview of the new arsarcasm-v2 dataset which was used for the shared task. we also provide a high-level description of the top participating teams in the shared task. the aim of this shared task is to encourage researchers to work on arabic sarcasm, which was reflected by the popularity of the task and having 27 run submissions and 17 system description papers discussing different approaches applied on this task. we hope that this task would not be the last on arabic sarcasm detection. more datasets on arabic sarcasm would further help the development of better detection models. in addition, much work is still required for this challenging task, since as it is noticed, the state-of-the-art performance is 0.62 f-score, which shows that there is large room of improvement to be achieved.