R 2 D 2 at S em E val-2022 Task 6: Are language models sarcastic enough? Finetuning pre-trained language models to identify sarcasm (2022)

Abstract: This paper describes our system used for SemEval 2022 Task 6: iSarcasmEval: Intended Sarcasm Detection in English and Arabic. We participated in all subtasks based on only English datasets. Pre-trained Language Models (PLMs) have become a de-facto approach for most natural language processing tasks. In our work, we evaluate the performance of these models for identifying sarcasm. For Subtask A and Subtask B, we used simple finetuning on PLMs. For Subtask C, we propose a Siamese network architecture trained using a combination of cross-entropy and distance-maximisation loss. Our model was ranked 7^{th} in Subtask B, 8^{th} in Subtask C (English), and performed well in Subtask A (English). In our work, we also present the comparative performance of different PLMs for each Subtask.

## introduction sarcasm, a form of figurative speech, allows people to express contempt or mock using irony. the irony is used to communicate the opposite of the intended meaning to express humour or mock something. sarcasm plays an essential role in people's daily conversation and finds its use across social media to express thoughts. recently, social media has drawn in millions of users around the world. owing to its figurative nature, sarcasm poses a significant challenge to systems performing sentiment analysis on these social media platforms. therefore, it is essential to design and develop systems that efficiently identify sarcasm. the most challenging aspect of sarcasm is the different socio-cultural backgrounds of people who drive it. therefore, sarcasm intended by the author may not always be perceived by the audience from different backgrounds. most datasets on sarcasm detection are collected using predefined criteria or human annotators @xcite . this is a suboptimal approach that may not always capture the sarcasm intended by the author leading to noise in the models trained on it. semeval 2022 task 6: isarcasmeval: intended sarcasm detection in english and arabic @xcite draws attention to the problem of identifying sarcasm using a dataset of intended sarcasm. our proposed approach for subtask a and subtask b approach uses a classification objective to finetune plms. plms learn semantic and syntactic features via training on large amounts of a text corpus. this information is used in downstream tasks by simply finetuning task-specific datasets. plms have shown remarkable performance on such downstream tasks using the simple finetuning approach @xcite . we extend the same idea to identify sarcasm (subtask a) and identify the type of irony (subtask b). subtask c aimed at identifying sarcastic text from its non-sarcastic counterpart. we propose a siamese network-based architecture using plms trained on a combination of cross-entropy and distance-maximisation loss. the classification objective(cross-entropy) identifies the sarcastic/non-sarcastic text, while the distance-maximisation loss maximises the distance between sarcastic/non-sarcastic features learnt by the model during training. we experimented with different plms, namely bert, roberta, mpnet, deberta, to present a comparative study of their performance for the task of sarcasm detection. our final submissions for subtask a and subtask b used finetuning on mpnet, while we used an electra-based model for subtask c. our proposed system performed well in subtask b and subtask c(english), attaining a 7 th and 8 th rank, respectively on the official leaderboard and performed well in subtask a. our experiments show that all plms had almost similar performance with slight variation in results. the overall scores were low for subtask a and subtask b, indicating that sarcasm detection and identifying the type of irony is a difficult task for plms. however, subtask c results show that the models can efficiently differentiate between a sarcastic text and its non-sarcastic rephrase. our code available at github foot_0 for method replicability. ## background identifying sarcasm is an essential task in natural language processing (nlp). owing to its figurative nature, it affects the performance of sentiment analysis systems whose performance have significantly improved over the years @xcite . in social media, sarcasm is mainly used for humour but can hide hateful content, making the identification of sarcasm a vital topic. methods to deal with sarcasm detection can be separated into two categories, i.e. contentbased models and context-based models @xcite . text-based models model the problem as a classification task using pragmatic and lexical features to identify sarcasm. @xcite shows sarcasm is expressed as a combination of positive sentiment words and negative situations. work done in @xcite uses the concept of context incongruity for sarcasm detection. contextual methods use information about the text and the context in which the text is used. @xcite uses the sentiment of the tweet as well as the history of the author's previous tweets on similar topics to identify sarcasm. work done in @xcite uses nouns and sentiments presented in a forum towards irony/sarcasm detection. @xcite worked on using both content as well as contextual information for identifying sarcasm. @xcite presents work done to identify sarcasm from tv shows in a multimodal setting. @xcite used a multimodal feature fusion model using attention @xcite for identifying sarcasm in internet memes. @xcite use models trained on emoji using distant supervision to identify sentiment, sarcasm, and emotion. another challenge in sarcasm detection is the availability of data. due to the highly subjective nature of sarcasm, it is challenging to collect high-quality data. @xcite tries to solve this problem by introducing a dataset of intended sarcasm where the sarcastic data is labelled by the authors removing any noise or ambiguity in labels. the subjective and figurative nature of sarcasm makes it a formidable task, and it poses a challenge for affective systems performing sentiment analysis @xcite . therefore, the task of sarcasm detection is essential to advance state-of-the-art sentiment analysis systems. moreover, most datasets for sarcasm detection contain much noise and are sub-optimal in capturing the sarcasm intended by the author of the text. semeval 2022 task 6: isarcasmeval: intended sarcasm detection in english and arabic @xcite aims to use a dataset of intended sarcasm for identifying sarcasm in text. the task has three subtasks which we define as: subtask a (english and arabic): given a labelled dataset d of texts, the task aims to learn a classification function that can identify sarcastic/non-sarcastic texts. subtask b (english only): for a given labelled dataset d of texts, the objective of the task is to learn a multilabel classification function that can predict the type of irony i where @xmath0, irony, satire, understatement, overstatement, rhetorical }. subtask c (english and arabic): given a dataset d of sarcastic texts and their non-sarcastic rephrase, i.e. both texts convey the same meaning, the objective of the task is to learn a classification function that can identify the sarcastic text from its nonsarcastic rephrase. our team participated in subtask a(english), subtask b, and subtask c(english). dataset statistics: a challenging task. transfer learning using glove @xcite and fasttext @xcite is one of the popular choices for solving this problem. most recently, researchers came up with a method called pre-training @xcite , which involves training general-purpose models from unannotated text data. this allows models to learn syntactic and semantic features in the text in an unsupervised setting. transformer architecture proposed in @xcite is the most common choice for training plms. these models can be finetuned on various downstream tasks using task-specific datasets. finetuning allows models to adapt to small task-specific datasets easily and shows promising results @xcite . next, we provide a summary of plms used in our approach. ## experimental setup text pre-processing: the text was first passed through a pre-processing pipeline to remove noise and normalize into standard features. we removed any website names in the text as they add noise to the data. we also found certain chat words like lol (laugh out loud) present in the data and converted them into their respective full forms. emojis are converted to their actual meanings. we used the emoji foot_1 library for emoji conversion. lastly, we used ekphrasis @xcite to normalize date, numbers to a standard format and perform spelling correction. plms require the text to be tokenized as part of pre-processing step. we use finetuning: our approach for all subtasks involves finetuning plms and using their features. we used features of @xcite token for bert, elec-tra and start token (<@xmath3. these features are then passed to further layers of models as per the architecture we defined above. hyperparameters and training: we developed our models on keras foot_3 @xcite and used hugging face's foot_4 implementation of transformer foot_5 @xcite models. finetuning was performed on colab using tpus. for finetuning we used adam (kingma and ba, 2015) optimiser. we experimented with learning rates ranging from 2e-5 to 5e-5. for subtask a and subtask b, we used a binary cross-entropy loss. for subtask c, we used a binary cross-entropy loss on the classification branch and distance maximization objective on the similarity branch. we finetuned the models for ten epochs and used the weights with the best performance on the development set to make predictions on the test set. evaluation metric: subtask a uses the f1 score of the sarcastic class as an evaluation metric. for subtask b, the macro averaged f1 score is the official metric, while for subtask c, accuracy is used as the evaluation metric. ## results we performed experiments using bert, roberta, mpnet, and electra during the evaluation phase. we evaluated these models using the validation set, and the model with the best performance was used to make a submission on the test data. ## conclusion this paper describes our proposed model used for semeval 2022 task 6: isarcasmeval: intended sarcasm detection in english and arabic. different plms were used to do a comparative analysis of their performance for the sarcasm detection task. our fine-tuning approach worked well for subtask b, with the best score for satire and second-best performance for the sarcasm and understatement category. for subtask c, we proposed a novel siamese network architecture to identify sarcastic content from it's non-sarcastic rephrase. it performed well, attaining 8 th rank on the leaderboard. our comparative analysis shows that sarcasm detection is a difficult task for the plms, and there is scope for further improvements, which we will take up in future works.