Deep Multi-Task Model for Sarcasm Detection and Sentiment Analysis in A rabic Language (2021)

Abstract: The prominence of figurative language devices, such as sarcasm and irony, poses serious challenges for Arabic Sentiment Analysis (SA). While previous research works tackle SA and sarcasm detection separately, this paper introduces an end-to-end deep Multi-Task Learning (MTL) model, allowing knowledge interaction between the two tasks. Our MTL model’s architecture consists of a Bidirectional Encoder Representation from Transformers (BERT) model, a multi-task attention interaction module, and two task classifiers. The overall obtained results show that our proposed model outperforms its single-task and MTL counterparts on both sarcasm and sentiment detection subtasks.

## introduction the popularity of the internet and the unprecedented reach of social media platforms allow users to express their opinions on a wide range of topics. thereby, sentiment analysis (sa) has become a cornerstone for many applications such as digital marketing, product review analysis, customer feedback, social media monitoring, etc. sa consists of determining the expressed sentiment (positive, negative, or neutral) conveyed by a text or a piece of text. over the past decade, significant research advances have been achieved for arabic sa @xcite @xcite . however, the mutual interaction and impact of figurative language devices, like sarcasm and irony, and arabic sa remain under-explored @xcite @xcite . these latter devices allow us to express ourselves intelligently beyond the literal meaning of words. although the literature uses the terms irony and sarcasm interchangeably, they have different meanings and there is no consensus on their definition @xcite @xcite . both sarcasm and irony devices pose a real challenge for sa as they can reverse the expressed sentiment polarity from positive to negative @xcite @xcite . therefore, there is an urgent need to develop sarcasm-aware sa tools. previous research works on arabic sa and sarcasm detection have dealt with both tasks separately @xcite @xcite . @xcite have built a corpus for irony and sarcasm detection in arabic language from twitter using a set of ironic hashtags. unlike the previous work of @xcite that have relied on ironic hashtags to label the tweets, the annotation is performed manually by two arabic language specialists. @xcite have presented an overview of existing arabic sa methods and approaches, and a benchmarking using three existing datasets. their results have shown that most of the evaluated models perform poorly on the se-meval and astd datasets. due to the label inconsistencies discovered, they have re-annotated the previously mentioned datasets for sa and sarcasm detection. in addition to the highly subjective nature of sa task, they have reported a large performance drop in the case of sarcastic tweets @xcite . following the recent breakthroughs in arabic natural language processing (nlp), achieved using arabert model ( antoun et al., 2020 ), abdul-mageed et al. (2020) have introduced two arabic transformer-based language models, namely ar-bert and marbert. arbert is trained on large textual corpora of modern standar arabic (msa), while marbert is trained on 1 billion da and msa tweets corpus. they have shown new cutting edge performances on wide range of da and msa nlp tasks (arabench datasets), including, among others, sa and sarcasm detection. in this paper, we present our end-to-end deep mtl model, submitted to sa and sarcasm detection in arabic language shared task @xcite . our approach is based on marbert (abdul-mageed et al., 2020), and a multi-task attention interaction module. the latter consists of two task-specific attention layers for extracting taskdiscriminative features, and of a sigmoid interaction layer @xcite for allowing interaction and knowledge sharing between sarcasm detection and sa. the task-interaction is performed using the task-specific attention outputs, a learnable shared matrix, and the sigmoid activation. the obtained results show that our mtl model surpasses the other evaluated single-task and mtl models. besides, the incorporation of an attention mechanism and the task-interaction boosts the performance of both sarcasm detection and sa. the rest of the paper is organized as follows. section 2 presents the shared task's dataset. section 3 introduces the proposed method. in section 4, we present the obtained results for both sarcasm detection and sa subtasks. section 5 discusses the obtained results. finally, section 6 concludes the paper. ## data the arsarcasm shared task consists of two subtasks for sarcasm detection and sa in arabic language @xcite . the shared task's dataset, arsarcasm-v2, is built from the previously introduced datasets for sarcasm and irony detection @xcite . the provided dataset consists of 12,548 and 3,000 tweets for the training set and test set, respec-tively. the task's dataset is annotated for sa and sarcasm detection as well as the regional dialect of the tweets. figure 1 presents the distribution of sarcastic tweets and their sentiment polarities (figures 1a and 1c ). the distribution of all sentiment polarities in the dataset is illustrated in 1c. the dataset is unbalanced for both subtasks. most sarcastic tweets are written in msa and egyptian dialect (figure 1a ), and are labeled with a negative sentiment (figure 1b ). furthermore, approximately half of the tweets convey a neutral sentiment (figure 1c ). ## method our multi-task model consists of three main components: bert encoder, a multi-task attention interaction module, and two task classifiers. ## results in this section, we present the experiment settings and the obtained results. ## discussion to investigate the strengths and weaknesses of our model, we have analyzed the confusion matrix of each subtask (figures 2a and 2b ) as well as the confusion matrices of sentiment analysis among sarcastic and non-sarcastic tweets respectively (figures 2d and 2c ). the analysis of these matrices shows that our mtl model leverages signals from both tasks and boosts the performances. this can be explained by the fact that most sarcastic tweets convey a negative sentiment. besides, negative tweets tend to have a large probability of being sarcastic than the positive ones. this could be also deduced from ## conclusion in this paper, we have proposed an end-to-end deep multi-task learning model for sa and sarcasm detection. our model leverages the marbert's contextualized word embedding with a multi-task attention interaction module. the aim is to allow task-interaction and knowledge sharing for both sa and sarcasm detection. our model shows very promising results on both subtasks. therefore, it proves the effectiveness of using task-specific attention layers as well as the task-interaction mechanism in multi-task learning. future research work will focus on developing task-interaction and class-interaction modules and mechanisms for sa and sarcasm detection.