A Contextual Word Embedding for A rabic Sarcasm Detection with Random Forests (2021)

Abstract: Sarcasm detection is of great importance in understanding people’s true sentiments and opinions. Many online feedbacks, reviews, social media comments, etc. are sarcastic. Several researches have already been done in this field, but most researchers studied the English sarcasm analysis compared to the researches are done in Arabic sarcasm analysis because of the Arabic language challenges. In this paper, we propose a new approach for improving Arabic sarcasm detection. Our approach is using data augmentation, contextual word embedding and random forests model to get the best results. Our accuracy in the shared task on sarcasm and sentiment detection in Arabic was 0.5189 for F1-sarcastic as the official metric using the shared dataset ArSarcasmV2 (Abu Farha, et al., 2021).

## introduction sarcasm detection is the process of classifying whether the text (i.e., a full document, a single sentence, a word, etc.) is sarcastic or not. sarcastic writing is a common way for people to share their sentiment in many online texts (i.e., social media, reviews, posts, comments, etc.). this linguistic phenomenon has much significance for tasks such as sentiment analysis and opinion mining. sarcasm detection has received much research attention in other languages especially in the english language, but the arabic language still lags behind. a few efforts have done in arabic sarcasm detection such as the works of @xcite and the shared task held by @xcite . there have been some recent efforts to build standard datasets for this task such as @xcite but still not enough to get good results. a lot of challenges we face when dealing with arabic sarcasm analysis such as 1) lack of datasets, 2) imbalanced classes, 3) lack of prepreprocessing tools (i.e., autocorrect, stemmer, etc.) for arabic language, 4) diversity of dialects which affects the results. most recent methods and approaches (i.e., machine-learning or deep learning) use one of the following two intermediate representations foot_0 for language modeling. firstly, and the very common intermediate representation which proved its efficiency in many experiments is tf-idf (term frequency-inverse document frequency). tf-idf is a very good way to start your initial representation because it has a lot of advantages such as 1) ease of computing, 2) extracting the most descriptive terms in a document and 3) you can easily compute the similarity using it. but with all of this, there is a very big disadvantage such as it cannot capture semantic in the given text which affects the algorithms effectiveness. secondly, is another very good intermediate representation that proved its effectiveness called word embedding. fixed word embeddings solve the problem of capturing the semantic of a word that we faced in tf-idf intermediate representation by providing a fixed embedding to every word which was too useful (examples: word2vec, fasttext). the limitations of the fixed word embeddings are that they cannot provide us with the information about words meaning in different a contextual word embedding for arabic sarcasm detection with random forests hazem e. elagbry, shimaa attia, ahmed abdel-rahman, ahmed abdel-ate, sandra girgis faculty of computer science and artificial intelligence helwan university, egypt {hazememad, shimaa_20170268, ahmed_20170044, ahmed_20170056, totajo1999}@fci.helwan.edu.eg contexts when setting a fixed embedding to every word. finally, to solve this problem there is a very good language modeling technique which is called contextual word embedding. contextual word embeddings set a different embedding to the same word according to its meaning in the context which handles the problem of diversity of a single word meaning. in literature, several researches focus on enhancing the arabic natural language understanding such as @xcite . in this paper, we attempt to extract contextual word embedding using ara-bert @xcite then feed it to an ensemble learning method called random forests classifier. we provide the git hub link of the code used to extract the contextual embeddings foot_1 . ## data the data we have used to get our results in this shared task are: ar-sarcasm-v2 @xcite , emojis dataset, stop words dataset, and unseen dataset from the shared task on sarcasm and sentiment detection in arabic as a test set to test our results. ## system in our approach, we have used the pre-trained algorithm ara-bert for extracting the contextual word embeddings that we have talked about in section 1. bert is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers @xcite . according to bert's state-of-the-art results, it is a big deal to include it in our classification task, but there is a problem which is bert alone is not enough and there are limits to it for getting state-of-the-art results. so, and because of ara-bert powerful in natural language understanding and its huge corpus which is trained with. we have used ara-bert for language modeling (i.e., extracting contextual embeddings) with its base-configuration. bertbase consists of (@xmath0, @xmath1, @xmath2, total @xmath3. in this notation, we denote the number of layers (i.e., transformer blocks) as l, the hidden size as h, and the number of selfattention heads as a. till now we did not remove stop words or performing any stemming for our data. we did this removes and stemming after extracting the word embeddings because contextual word embeddings rely on extracting different embeddings for the same word in different contexts, and even ‫ال‬ in arabic language can affect our understanding of the context. according to this assumption, we have implemented a second text cleaning/preprocessing after extracting our contextual word embeddings. we have made another interesting mechanism for doing something very close to the stemming which we are going to discuss here. ara-bert's input is not a feature but is a sequence of words that is tokenized with a pre-trained ara-bert tokenizer. the ara-bert tokenizer is created with a word piece model. in this tokenization every word has one of the following forms: 1) the word still the same which means that this word is in bert's vocabulary, 2) the word will come with a # sign which means that this word is part of a bigger word (i.e., the original word is not in its vocabulary so, bert split the word into a group of subwords with #), 3) the word with a + sign means that this word is a prefix or suffix to another word (i.e., ‫الكتاب‬ the ‫ال‬ here is a prefix so it will appear like ‫كتاب‬ + ‫.)ال‬ according to this form, we have used it to perform stemming especially with dialects which are very hard to stem by removing every word that has the + sign. after this second cleaning, we get the average of word embeddings to get our sentence embeddings with vector size of 768 feature. after second cleaning and to avoid empty tweets in train or test set, we have removed every empty tweet after saving its index and assign to this tweet a false label which means it is not sarcastic by default. finally, we feed this sentence embedding to random forests (which is an ensemble learning method for classification, regression, and other tasks) with its default hyperparameters configuration using scikit-learn package in python see table @xcite . we provide the code foot_3 of our approach for publicly use to push the arabic classification tasks to another state-of-the-art accuracy. ## results in this section, we are going to discuss the key results of our approach by sharing with you our evaluation metrics with its key results in section 4.1, then the official metrics of the shared task on sarcasm and sentiment detection in arabic with its key results in section 4.2. ## discussion our observations and experiments show that in arabic sarcasm detection 2 main challenges affect the accuracy, validation, and our improvement such as 1) the dataset was too small, 2) dealing with dialects was making this harder (ex. stemming was a very challenging problem when dealing with dialects) to our knowledge, a very small number of researches are studying the effectiveness of contextual word embeddings in arabic which are very helpful in dealing with different dialects and unseen data, and the most interesting example for such effectiveness is ara-bert which reaches a very good accuracy in dialects while he was trained in msa corpus. dealing with incorrectly entered data especially in social media was incredible and very challenging (i.e., if a user wrote ' ‫ﺣﻘﻚ‬ ‫ﻻﻣﻦ‬ ' the tokenization phase will treat this 3 words as a 2 words ' ‫ﻻﻣﻦ‬ ' and ' ‫ﺣﻘﻚ‬ ' but if he wrote ' ‫ﺣﻘﻚ‬ ‫ﻣﻦ‬ ‫ﻻ‬ ' with a space between every word the tokenization phase will treat this as 3 words not two ' ‫,'ﻻ‬ ' ‫'ﻣﻦ‬ and ' ‫)'ﺣﻘﻚ‬ there always a problem when dealing with every step in pre-processing with dialects. for the above reasons, our next steps are like the following: 1) focusing more on preprocessing problems (i.e., produce an algorithm for stemming dialects, autocorrect arabic words), 2) looking for more data to get better accuracy and validation, 3) we have tried a machine learning approach which is not very good in capturing the amount of information that the contextual word embeddings transfer (i.e., it is like a bottleneck). so, one of our important next steps is dealing with the problem using deep learning models. ## conclusion our task allowed us to explore a lot of challenges with natural language processing in arabic. we believe that arabic natural language processing applications such as sarcasm detection, sentiment analysis, pos, etc. has a lot of areas to be investigated and developed. in our paper, we tried to overcome the problem of dialects and semantic diversity for every word in different contexts by generating a contextual word embedding using ara-bert which achieved very good results just using random forests ensemble learning with its initial scikit-learn configuration. we hope that problems such as pos, autocorrect, and such important problems take a more interest to solve it because we believe that such problems are a big deal when we talk about enhancing the accuracy of other tasks like: sentiment or sarcasm analysis.