A r S arcasm Shared Task: An Ensemble BERT Model for S arcasm D etection in A rabic Tweets (2021)

Abstract: Detecting Sarcasm has never been easy for machines to process. In this work, we present our submission of the sub-task1 of the shared task on sarcasm and sentiment detection in Arabic organized by the 6th Workshop for Arabic Natural Language Processing. In this work, we explored different approaches based on BERT models. First, we fine-tuned the AraBERTv02 model for the sarcasm detection task. Then, we used the Sentence-BERT model trained with contrastive learning to extract representative tweet embeddings. Finally, inspired by how the human brain comprehends the surface and the implicit meanings of sarcastic tweets, we combined the sentence embedding with the fine-tuned AraBERTv02 to further boost the performance of the model. Through the ensemble of the two models, our team ranked 5th out of 27 teams on the shared task of sarcasm detection in Arabic, with an F1-score of %59.89 on the official test data. The obtained result is %2.36 lower than the 1st place which confirms the capabilities of the employed combined model in detecting sarcasm.

## introduction what is sarcasm? according to cambridge dictionary, it is "the use of remarks that clearly mean the opposite of what they say, made in order to hurt someone's feelings or to criticize something in a humorous way" @xcite . former research in psychology and neurology reports that the mind requires the use of two distinct regions to process sarcasm as it implies a contradicting meaning to the literal one @xcite . according to capelli and her colleagues, the ability to analyze sarcasm develops from childhood. in their early stages, children are not able to perceive sarcasm by the text alone and it has to be accompanied by intonation for clearer understanding. unlike adults who identify sarcasm effortlessly with intonation or from the text alone @xcite . megan dress and her colleagues noticed that there are regional differences in the use of sarcasm when other demographic factors were controlled @xcite . moreover, yoritaka akimoto and his colleague concluded in their paper that there are individual differences in the intended use of sarcasm that affects the processing and interpretation of verbal irony @xcite . from this perspective, we can understand the challenging aspect of processing the different meanings of sarcasm with machines. automatic sarcasm detection is the process of identifying if the text has a sarcastic meaning or not. this process is an essential step in the sentiment analysis task which requires a clear understanding of the intended meaning of the text. over the years, researchers have proposed several methods to detect sarcastic features in texts. applying several algorithms like gradient boosting @xcite , using linguistic features @xcite , using incongruity phenomenon for creating a sarcasm detection system @xcite , a2text-net deep learning model @xcite , and using a model based on embeddings from a language model @xcite . in gonzalez's paper and joshi's paper, they both utilized linguistic features to assess sarcasm detection in written texts. furthermore, liyuan liu and suzana ili separately investigated two different deep learning methods for the sake of better sarcastic detection. liu's and his colleagues' model used a hypothesis layer and a feature processing layer to test the hypothesis and to select the supported features before training a neural network for detection. moving to suzana ili s paper, she and her colleagues found that the word-level representations model lacks the richness that embeddings from language models or elmo have. accordingly, they used the elmo model to investigate the sarcastic features in textual data. even more, arabic sarcasm detection is lagging behind in the field of sarcastic detection in nlp. some efforts have been done such as jihen karoui and her colleagues who worked to detect irony in arabic tweets @xcite . our goal in this task is to classify arabic tweets into sarcastic or not sarcastic by taking into account the direct and the overall meaning of the tweet. to this end, we explored the use of the bert model in three different ways. first, we performed a knowledge-transfer from arabertv02 to a sarcasm detection task. secondly, we employed sentence-bert trained model with the contrastive loss for extracting representative sentence-level embedding for sarcasm. finally, mimicking the way the human brain processes the surface and intended meaning of a sentence in two different regions, we processed the data into two different models and the outputs of the two models were then concatenated into a one-tuned ensemble model by stacking a classifier on top of them. ## experimental results the results of the five experiments of the method are shown in ## conclusion we presented our submission on the shared task on sarcasm detection and sentiment analysis in arabic to tackle the problem of detecting sarcasm in arabic. we explored different models based on bert. we utilized a fine-tuning bert and a sentence-bert to generate sentence embeddings that can be effective in detecting sarcasm. the final submission model combined the arabertv02 model and a representation obtained from a sentence-bert model to determine the sarcastic meaning of tweets. in the future, we will work on improving the performance of the model by exploring more sentence representation techniques.