CS - UM 6 P at S em E val-2022 Task 6: Transformer-based Models for Intended Sarcasm Detection in E nglish and A rabic (2022)

Abstract: Sarcasm is a form of figurative language where the intended meaning of a sentence differs from its literal meaning. This poses a serious challenge to several Natural Language Processing (NLP) applications such as Sentiment Analysis, Opinion Mining, and Author Profiling. In this paper, we present our participating system to the intended sarcasm detection task in English and Arabic languages. Our system consists of three deep learning-based models leveraging two existing pre-trained language models for Arabic and English. We have participated in all sub-tasks. Our official submissions achieve the best performance on sub-task A for Arabic language and rank second in sub-task B. For sub-task C, our system is ranked 7th and 11th on Arabic and English datasets, respectively.

## introduction sarcasm is an important aspect of human natural language. it is characterized by the occurrence of a discrepancy between the intended and the literal meaning of utterance @xcite . the prevalence of this phenomenon may jeopardize the performance of many nlp applications, such as sentiment analysis, opinion mining, and emotion detection, among others @xcite @xcite . indeed, sarcasm detection has been the subject of many systematic investigation, where several shared tasks have been organized and a number of datasets have been introduced @xcite @xcite . the existing datasets are either labeled by a human annotator or using distant supervision signals such as the presence of a set of predefined hashtags @xcite . however, these labeling methods might be sub-optimal for intended sarcasm detection as the author's sarcastic intent may differ from an annotator's perceived meaning @xcite . besides, most existing research works have focused on english language @xcite , while few studies have been introduced for other languages such as arabic @xcite . to overcome the aforementioned limitations, abu @xcite have organized the isar-casmeval shared task for intended sarcasm detection in english and arabic languages. in contrast to previous research work, the introduced dataset, for intended sarcasm detection, is labeled by the authors themselves. the authors are then asked to provide non-sarcastic rephrases that covey the same intended meaning of their sarcastic texts. further, the isarcasmeval's organizers have relied on linguistic experts to categorize sarcastic texts into sarcasm, irony, satire, understatement, overstatement, and rhetorical questions @xcite . in this paper, we present our participating system to isarcasmeval shared task @xcite . our system rely on three transformer-based deep learning models. in all our models, we use existing pre-trained language model (plm) to encode the input text and apply a single attention layer to the contextualized word embedding of plm @xcite . for all our models, we use the same classifier architecture composed of one hidden layer and one classification layer. the classifier is fed with the concatenation of the pooled output of the plm as well as the attention layer output. we model the sub-task a as a binary classification (first model) and as a multi-class classification (second model). motivated by the small size of the datasets and similarly to gan-bert architecture @xcite , the third model uses a conditional generator that tries to generate fake samples that are similar to the plm's embedding of the real input data. the discriminator of the third model is trained to discriminate between fake and real samples as well as classify the real ones correctly. for sub-task b, we only employ the gan based model (third model), while for sub-task c, we utilize all models trained on task a and we compare the probabilities of the sarcastic class of the left and the right text. we train our models using several loss functions, including the focal loss @xcite . besides, for subtask a, we train our models with and without the non-sarcastic rephrase texts. for the official submissions, we employ hard voting ensemble of our trained models. our system achieve promising results as we rank 1st, 15th, 2nd, 7th, and 11th on sub-task a ar, sub-task a en, sub-task b, sub-task c ar, and sub-task c en, respectively. ## system overview the submitted system to the isarcasmeval shared task relies on three transformer-based models for intended sarcasm detection. in order to encode the input text, we utilize the twitter-xlm-robertabase @xcite and marbert (abdul-mageed et al., 2021) for english and arabic texts, respectively. the former is a variant of xlm-roberta plm that is adapted to twitter data using mlm objective, while the latter is a variant of bert plm that is pre-trained from scratch on arabic tweet corpora. in the following subsection, we describe the components of our system. ## results in this section, we present the obtained results of our models as well as our official submissions. it is worth mentioning that for sub-task c, we employ the trained models on sub-task a and we use their output probabilities to discriminate between the sarcastic text and the non-sarcastic rephrase. besides, for our official submissions, we use the hard vote ensemble of our trained models. for each loss function, the best performance obtained is highlighted in bold font, while the overall best performance for each task is highlighted in bold-italic font. ## conclusion in this paper, we present our participating system in the isarcasmeval shared task for intended sarcasm detection in english and arabic. our system relies on three deep learning-based models that leverage two existing pre-trained language models for arabic and english. we participate in all sub-tasks, investigate several training objectives, and we study the impact of including non-sarcastic rephrase in the training data. the results show that ensembling models that are trained with and without rephrases have a negative impact on the official results. our official submissions achieve the best performance on sub-task a for the arabic language and rank in the second position on sub-task b.