Combining Context-Free and Contextualized Representations for A rabic Sarcasm Detection and Sentiment Identification (2021)

Abstract: Since their inception, transformer-based language models have led to impressive performance gains across multiple natural language processing tasks. For Arabic, the current state-of-the-art results on most datasets are achieved by the AraBERT language model. Notwithstanding these recent advancements, sarcasm and sentiment detection persist to be challenging tasks in Arabic, given the language’s rich morphology, linguistic disparity and dialectal variations. This paper proffers team SPPU-AASM’s submission for the WANLP ArSarcasm shared-task 2021, which centers around the sarcasm and sentiment polarity detection of Arabic tweets. The study proposes a hybrid model, combining sentence representations from AraBERT with static word vectors trained on Arabic social media corpora. The proposed system achieves a F1-sarcastic score of 0.62 and a F-PN score of 0.715 for the sarcasm and sentiment detection tasks, respectively. Simulation results show that the proposed system outperforms multiple existing approaches for both the tasks, suggesting that the amalgamation of context-free and context-dependent text representations can help capture complementary facets of word meaning in Arabic. The system ranked second and tenth in the respective sub-tasks of sarcasm detection and sentiment identification.

## introduction with the advent of social media platforms as a valuable source of opinion-rich information, work on subjectivity language analysis has continued to receive increasing interest from the arabic nlp community. sentiment analysis (sa) has been the dominant theme in this area, with notable works ranging from the creation of lexical resources and sentiment datasets @xcite @xcite to the contrivance of neural network-based classification models @xcite @xcite . in comparison, the literature in arabic sarcasm detection is still in its nascent stage, limited to a few notable works @xcite @xcite . recently, transformer-based language models have proved highly efficient at language understanding, achieving promising results across multiple nlp tasks and benchmark datasets. the language modeling capability of these models is aiding in capturing the literal meaning of context-heavy texts. for arabic nlp in particular, the best results for sentiment analysis are currently achieved by arabert, a language model proposed by @xcite . despite this recent progress, sarcasm detection remains a challenging task, primarily due to the use of implicit, indirect phrasing and the figurative nature of language @xcite . the task becomes even more challenging when working with twitter data, as the social media posts tend to be short and often contain sources of noise, codeswitching, and the use of nonstandard dialectal variations @xcite . furthermore, bertbased models are found to struggle with rare words @xcite , which can be encountered more in social media texts due to their informal nature and the prevalent use of slang words. for language models like arabert, this can pose a challenge, primarily since it has been trained on structured corpora from wikipedia. building on the capabilities of language models, some recent studies have shown that incorporating entity vectors can benefit the bert-based language models, especially for domain-specific tasks or datasets @xcite @xcite ). an interesting approach followed by @xcite suggests that the performance of language models can be boosted by incorporating static word embeddings trained on specific social media corpora. in this study, we posit a solution for subtask-1 and subtask-2 of the wanlp arsarcasm shared task 2021 @xcite . while subtask-1 focuses on identifying sarcasm, subtask-2 deals with classifying the sentiment polarity in arabic tweets. inspired by the works of @xcite and @xcite , we propose a hybrid model, combining the sentence representations learned from arabert with pre-trained arabic word vectors proposed by @xcite . results attest that the proposed methodology can provide a competent way of subsuming the advantages of both the contextualized and context-free word representations, outperforming all the baseline models for both tasks. the rest of the paper is organized as follows: in section 2, we provide a concise literature review of previous works in arabic sentiment and sarcasm detection. section 3 provides a descriptive analysis of the dataset at hand. in section 4, we present the proposed system, and in section 5, we describe the experimental setup details. section 6 interprets the results. finally, section 7 concludes the study and points to possible directions for future work. ## related work early works in arabic subjectivity analysis focused on using conventional machine learning approaches and lexical methods @xcite . with the emergence of deep learning techniques, research in arabic nlp has shifted from the traditional statistical standpoint to designing complex neural network models and learning word representations. al sallab et al. (2015) the success of the english word2vec @xcite and fast-text @xcite motivated other works to achieve the same feat by creating language-specific word embeddings. for arabic nlp, some early attempts include word2vec-based aravec @xcite , followed by fast-text @xcite . @xcite proposed the mazajak embeddings, trained exclusively on a large arabic twitter corpus for handling the varied arabic dialects. multiple studies leveraged this advancement in word representations for both the sarcasm and sentiment detection tasks. for instance, @xcite developed a cnn and lstm ensemble model for arabic sa. the authors employed pretrained aravec word embeddings for the text representation. @xcite proposed a hybrid model for arabic sa, concatenating pre-trained ar-avec and fast-text vectors. abu farha and magdy (2019) used the pre-trained mazajak vectors on a cnn-bilstm ensemble model, achieving stateof-the-art results on three benchmark datasets. for sarcasm detection, a similar approach was followed by @xcite , where the authors used the mazajak vectors in combination with a bil-stm model. the best results for multiple datasets is currently achieved by fine-tuning the arabert model @xcite , eliminating the need to use the static word vectors in standard settings. despite this fact, we believe that arabert and mazajak have complementary strengths and can lead to improved results if used in coalescence. in this study, we investigate the effectiveness of combining the word representations obtained from these two models on the sarcasm detection and sentiment identification tasks. ## dataset the wanlp arsarcasm shared-task 2021 follows the arsarcasm v2 dataset @xcite . the dataset contains sarcasm, sentiment and dialect labels of 12,549 arabic tweets. the tweets span across five arabic dialects including msa, gulf, egyptian, levantine, and maghrebi, with msa and egyptian dialects having the highest percentage of tweets. for development, we follow a standard 80-20 stratified split on the arsarcasm v2 dataset. this leaves us with a validation set of 2510 tweets, which are used for the primary evaluation of the proposed system along with the baseline models. the organizers provide a separate dataset for testing, consisting of 3,000 tweets. ## proposed system unlike the static word models such as word2vec or fast-text, language models like arabert follow a different tokenization strategy; wherein each word is split into one or more wordpiece tokens (antoun et al., 2020). thus, we cannot simply concatenate the pre-trained mazajak embeddings with the contextualized representations predicted by arabert at the word level. we instead combine these word representations at the sentence level, following an approach similar to the one used by @xcite . the final sentence representation is passed to the dropout layer, followed by a dense layer with softmax activation for classification. fig 1 gives an overview of the proposed system's architecture. the following section describes each system component in detail. ## results and discussion the official evaluation metric for the sarcasm detection subtask is the f-score of the sarcastic class, while that for the sentiment identification subtask is the f-pn score (macro average of the f-score of the positive and negative classes). for each model, the validation set results are averaged over five runs to ensure a fair comparison. overall, the proposed method shows improved results across all the metrics for both the sarcasm and sentiment detection tasks. the performance improvements can be attributed to the fact that unlike the mazajak word embeddings, which are exclusively trained on a twitter corpus, the arabert model is trained on the arabic wikipedia and news corpora, preventing it from witnessing the varied dialects in which social media posts are written. it is reasonable to postulate that while language models like arabert capture rich contextual information, the mazajak word vectors can provide valuable complementary information for rare words, which can be found abundantly in social media texts. hence, using them in combination can only help a system capture complementary facets of word meaning, thereby enhancing its performance on the downstream sentiment and sarcasm detection tasks. ## conclusion in this study, we proposed a hybrid model to combine the contextualized sentence representations obtained from arabert with pre-trained mazajak word vectors. we show that the proposed model outperforms the standalone arabert model for both the sarcasm and sentiment detection tasks. our findings suggest that incorporating static word vectors might help language models like arabert to deal with rare words and the constantly updating language of social media. an alternative strategy would be to pre-train arabert on specific social media corpora like twitter. however this can prove to be extremely expensive and is not feasible in practice. it is also important to note that, while the proposed model leads in performance, it is more complex, and has a greater number of trainable parameters. hence, it would be essential to test its feasibility on datasets larger than the arsarcasm v2 dataset.