WANLP 2021 Shared-Task: Towards Irony and Sentiment Detection in A rabic Tweets using Multi-headed- LSTM - CNN - GRU and M a RBERT (2021)

Abstract: Irony and Sentiment detection is important to understand people’s behavior and thoughts. Thus it has become a popular task in natural language processing (NLP). This paper presents results and main findings in WANLP 2021 shared tasks one and two. The task was based on the ArSarcasm-v2 dataset (Abu Farha et al., 2021). In this paper, we describe our system Multi-headed-LSTM-CNN-GRU and also MARBERT (Abdul-Mageed et al., 2021) submitted for the shared task, ranked 10 out of 27 in shared task one achieving 0.5662 F1-Sarcasm and ranked 3 out of 22 in shared task two achieving 0.7321 F1-PN under CodaLab username “rematchka”. We experimented with various models and the two best performing models are a Multi-headed CNN-LSTM-GRU in which we used prepossessed text and emoji presented from tweets and MARBERT.

## introduction text on social media as twitter and facebook involves the usage of expressive and figurative languages like irony and sarcasm. a text is considered ironic if it is intended to mean the opposite of what it literally expresses. while sentiment is the interpretation and classification of emotions (positive, negative, and neutral) within text. irony detection has recently become more relevant due to its importance in extracting information from texts. for example, in order to move beyond the literal match of user queries, irony detection will enhance information retrieval with new operators to allow nonliteral retrieval of creative expressions. sentiment analysis has been an important topic in the literature for a while. the interest in sentiment analysis arises due to the wide and increase in usage of social media platforms, and online shopping. companies can benefit from understanding the feedback and thoughts of their customers. the sixth workshop for arabic natural language processing co-located with eacl 2021, had two shared tasks: sarcasm and sentiment detection in arabic tweets. the shared task on sarcasm detection is a binary classification where it is required to identify whether a piece of text is sarcastic or not. while shared task in sentiment detection is multi-class classification, where it is required to identify sentiment of a piece of text as positive or negative or neutral. in this paper, we describe the work we performed on both shared tasks. marbert, and multi-headed-lstm-cnn-gru models foot_0 were used in this study. the rest of this paper is structured as follows: section 2 discusses the related work that has been performed on automatic sarcasm and irony detection, section 3 describes the dataset used in this shared task, section 4 discusses the approach we used in our study, and section 6 discusses the results we obtained. ## related work there have been large attempts in the literature to tackle irony and sentiment detection. most of the work is biased towards machine learning approaches, with few attempts to use deep learning approaches. a recent work @xcite proposed two powerful bidirectional based transformers models arbert and marbert. the models were trained and evaluated based on arbenchmark which consists of 41 datasets targeting 5 different tasks/task clusters including irony and sarcasm tasks. in @xcite , the authors used a random forest classifier based on some set of features, in which they believe are powerful, for irony detection. the set of features are surface features, sentiment features, shifter fea-tures, and contextual features. in another work @xcite , the authors used ensemble learning in irony detection in arabic tweets. the ensemble learning consists of three ensemble models. the first ensemble model uses different machine learning classifiers which uses tf-idf word n-gram features, topic modeling features, bag-of-words, and sentiment features. the second ensemble model uses 8 different word-level bi-lstm while the third ensemble model is a combination of both the first and second ensemble models. moreover alayba et al. (2018) the authors tested their model on two datasets, ar-twitter @xcite and arabic health services @xcite datasets, where they achieved accuracies of 88.1% and 94.3% respectively. @xcite proposed hilatsa for arabic semantic analysis. hilatsa combines lexicon-based and machine learning approaches. the authors experimented with several datasets in their work as astd @xcite and artwitter. the authors made use of different classifiers, which varied from using classical machine learning to deep learning models. in another study @xcite , the authors proposed two models: a character-level bidirectional lstm along with a conditional random field classifier (bi-lstm-crf) for aspect opinion target expressions extraction, and an aspect-based lstm for aspect sentiment polarity classification. they tested their model on arabic hotels' reviews @xcite dataset, they achieved an f-score of 70%. ## data this section shows the used dataset in the shared task. we used arsarcasm-v2 dataset @xcite , which was provided in the training phase by arsarcasm shared task. the dataset is a twitter text with labels "false" and "true" for the first shared task and "pos", "neg", and "neu" for the second shared task. the dataset provided in the training phase contains 12548 training examples, while the test set provided in the test phase consists of 3000 examples. ## methodology this section discusses the details required for reproducing the results. it mentions the preprocessing steps, the architecture of the classifiers used, and hyperparameter values. ## experimental setup experiments were conducted via python and tensorflow framework, running on google colab resources, which are nvidia tesla p100-pcie-16gb gpu, intel ® xeon ® cpu @ 2.20 ghz, and 12gb ram. we used 70%-10%-20% strategy for trainvalidation-test splits respectively for the training phase. eventually, we had 8783 examples for the training set, 753 examples for the validation set, and 3012 examples for the test set. all of the presented models were trained on the same dataset and splitting criteria. in the multi-headed cnn-lstm-gru model, weights were initialized with he initialization. adam optimizer with a 0.0001 learning rate was used, with cross-entropy as a loss function. the model was trained for 150 epochs which took around 2-3 hours of training, with 32 batch sizes. for the marbert model, it was finetuned for 5 epochs using the initial learning rate 2e-06, with a batch size of 32 with adam weight decay optimizer and cross-entropy loss function. precision, recall, f-score (f1-sarcastic for shared task one, and macro average f1-score for shared task two), and accuracy were used as evaluation metrics. ## conclusion in this paper, the results and the main findings of arsarcasm shared task one and two, on identifying irony and sentiment in arabic tweets were presented, in which different experiments were carried out with marbert and multi-headed-lstm-cnn-gru model. information presented in text as emojis were used to improve model performance. however, hyperparameter optimization and the usage of pre-trained word embeddings for the multi-headed-lstm-cnn-gru network showed its effectiveness and impressive results for a small model compared to the pre-trained marbert model based on bert and transformers. future work will include investigating text representations, improving the deep learning model, and exploring different architectures.