The IDC System for Sentiment Classification and Sarcasm Detection in A rabic (2021)

Abstract: Sentiment classification and sarcasm detection attract a lot of attention by the NLP research community. However, solving these two problems in Arabic and on the basis of social network data (i.e., Twitter) is still of lower interest. In this paper we present designated solutions for sentiment classification and sarcasm detection tasks that were introduced as part of a shared task by Abu Farha et al. (2021). We adjust the existing state-of-the-art transformer pretrained models for our needs. In addition, we use a variety of machine-learning techniques such as down-sampling, augmentation, bagging, and usage of meta-features to improve the models performance. We achieve an F1-score of 0.75 over the sentiment classification problem where the F1-score is calculated over the positive and negative classes (the neutral class is not taken into account). We achieve an F1-score of 0.66 over the sarcasm detection problem where the F1-score is calculated over the sarcastic class only. In both cases, the above reported results are evaluated over the ArSarcasm-v2–an extended dataset of the ArSarcasm (Farha and Magdy, 2020) that was introduced as part of the shared task. This reflects an improvement to the state-of-the-art results in both tasks.

## introduction sarcasm and sentiment detection are two central tasks in the research field of natural language processing (nlp). sentiment classification can be defined as the process of estimating emotion polarity of a given piece of text @xcite . sarcasm detection is usually treated as the process of determining whether a piece of text was sarcastically written or not. the free dictionary foot_0 defines sarcasm as a form of verbal irony that is intended to express contempt or ridicule. according to @xcite , sarcasm is correlated with expressing the author's opinion in an indirect form, where the intended meaning is different from the literal one. although these two tasks are highly related to each other and are usually solved by using a classification model, they differ in significant nuances. sarcasm heavily dependent on context -in most of the cases, a given sentence can be interpreted as sarcastic only if it is written (or said) in a specific context. moreover, a speaker will use sarcastic language in cases where she thinks that her words will be interpreted as so @xcite . in many cases, sarcasm detection is treated as a sub-task of the broader problem of sentiment anlaysis. it does not mean that sarcasm detection becomes redundant if the sentiment classification problem is solved accurately enough, since sarcastic language remains the biggest challenge for sentiment-analysis models. the reason behind this is the fact that a sarcastic language usually carries a negative implicit sentiment, while it is expressed using positive expressions @xcite . the nuanced differences between the two tasks motivated us to take the approach of building two distinct solutions. there is an active research area on sarcasm and sentiment analysis, but it is mostly done on english @xcite @xcite . the research efforts that focus on sarcasm and sentiment classification in arabic is limited. however, with the recent growing interest in non-english nlp, processing arabic becomes one of the most active research topics @xcite @xcite . for the last two decades, social networks (e.g., twitter,reddit) became a central communication tool for humans. these data sources enable the usage of textual data for sarcasm detection and sentiment classification models building. in this paper, we present our solution to the shared task of sarcasm and sentiment detection in arabic, organized as part of the the sixth workshop for arabic nlp @xcite . we implemented two distinct algorithms for each task (sentiment and sarcasm). both algorithms use some of the recent advanced transfer-learning models followed by traditional machine-learning techniques (e.g., usage of meta-feature, down-sampling). ## data as part of the shared task, the arsarcasm-v2 dataset is introduced @xcite . the dataset consists of two separate datasets-train and test. we note that the arsarcasm (farha and magdy, 2020) dataset (10587 tweets) is a subset of the train dataset. we denote the two parts of the train dataset as train arsar and train arsar throughout the paper. each tweet in the train dataset is additionally tagged with a dialect value (five arabic dialects), a sentiment value (three classes), and whether the tweet is sarcastic or not (two classes). the test dataset contains 3k tweets for which only the dialect value is available. dataset statistics are provided in ## system description for both tasks, we use a transformer architecture as a major part of the system. for sentiment classification we use @xcite while for the sarcasm detection we use marbert @xcite . both are recent pretrained autoregressive models in arabic. in both systems, we preprocess the data prior to training. the preprocessing steps include tokenization, removal of redundant text and punctuation marks (e.g., 'rt'), and replacement of links and mentions with placeholders. figure 1 provides a high-level overview of both systems. ## results the results of the sentiment classification and the sarcasm detection tasks are reported in and ## discussion one of the challenges we were facing in both tasks is overfitting. we assume that the root cause of the problem is the relative small corpus size and label imbalance. to handle the challenge, we control the learning process in different ways (e.g., small number of training epochs, relative low learning rate). another challenge we identified and dealt with is the significant difference between train arsar and train arsar . the difference is reflected through the dialect distribution as well as the label distribution (see ## conclusion as observed in previous works as well as in this one, sarcasm detection is a "harder" task to solve than sentiment classification. in this work we introduced two different approaches for solving each of the tasks. both approaches rely on transformer architectures. however, in both algorithms we adjust the transformer model by using machinelearning techniques (e.g., instance weighting, a more suitable loss function). in addition, in both solutions we add machine-learning components to the pipeline prior or subsequent to the transformer model (e.g., down-sampling, data augmentation). we showed that these techniques improve the existing baselines in both tasks when we evaluate results over the dataset provided as part of the shared task.