niksss at S em E val-2022 Task 6: Are Traditionally Pre-Trained Contextual Embeddings Enough for Detecting Intended Sarcasm ? (2022)

Abstract: This paper presents the 10th and 11th place system for Subtask A -English and Subtask A Arabic respectively of the SemEval 2022 -Task 6. The purpose of the Subtask A was to classify a given text sequence into sarcastic and nonsarcastic. We also breifly cover our method for Subtask B which performed subpar when compared with most of the submissions on the official leaderboard . All of the developed solutions used a transformers based language model for encoding the text sequences with necessary changes of the pretrained weights and classifier according to the language and subtask at hand .

## introduction according to @xcite , "sarcasm detection is the task of identifying irony containing utterances in sentiment-bearing texts". even though sarcastic humor is present throughout social media, it is hard for even humans to comprehend it certainly as the reader doesn't always perceive it the same way the speaker intended with intricate socio-psychological and cultural references. with the way, current models for text representation are trained like word2vec @xcite and glove @xcite , which have the same representation for a particular word irrespective of the context they fail to incorporate the meaning of sarcastic texts. even though the recent state-of-the-art language models provide contextual embeddings for words, it still fails to tell the sarcastic humor from normal as we explain later in the paper. there have been a lot of attempts at computationally automating sarcasm detection in the literature. discernibly, the task of sarcasm detection can be classified into content and context-based methods. with features like structural, morphosyntactic and semantic ambiguity features @xcite ,user mentions (replies), emoticons, n-grams, dictionary-and, sentiment-lexicon-based features @xcite ) and features based on word embedding similarity @xcite coming inside content based method. with the surge of seq2seq based model such as bert (bidirectional encoder representations from transformers @xcite , roberta @xcite , and xlnet @xcite , etc. have been heavily employed for sarcasm detection in the literature. @xcite proposed an r-cnn-roberta which is a hybrid model leveraging roberta's contextual embedding into a recurrent convolutional neural network. @xcite ) created an ensemble of roberta and albert @xcite . seeing the success of pre-trained language model, we decided to leverage the pre-trained contextual embeddings and transformers toward sarcasm detection. while some of the previous textual sarcasm detection datasets involved annotation via finding some predefined criteria, such as including specific tags (e.g. sarcasm, irony) @xcite associated with the text. other datasets involved manual labeling @xcite . however, the mentioned labeling techniques produced noisy or uncertain labels which would further compromise the effectiveness of the models trained on them. further, most of the sarcasm detection work has been done in the english language and it is highly unlikely that the models trained on one language would generalize well on other languages. the purpose of semeval-2022 task 6 -intended sarcasm detection in english and arabic @xcite is to advance the development of automatic textual sarcasm detection by providing a dataset which overcomes the problem of noisy and uncertain labels by having the authors provide the labels themselves. the shared task is divided into three subtasks: ## system overview here, we describe the data pre-processing steps along with the models for each subtask, and the experimental setup for the system. we provide an overview of the system in figure 1 . ## error analysis after examining the predictions from the submitted model, we saw that the model struggled significantly in classifying the text sequences to sarcastic type. we inferred that, even though we put less weight on the non-sarcastic class during the loss computation, the model overfitted to the abundant class of non-sarcastic text sequences with a ratio of roughly around 3:1 between the two classes for both engligh and arabic tasks. we also noted that individual hyper-parameters had significant roles in the performance of the model. training different models with different hyper-parameters and ensembling them together showed a significant increase in performance in the post evaluation period. for subtask b, the main reason for the poor performance of our model and most submitted models on the leaderboard, is the inter-class difference between individual sarcasm types is very low.which in-turn confuses the model and the output probability is roughly close to each other. ## conclusion we developed a system to classify sarcastic text from non-sarcastic text using contextualized embeddings from a language model which didn't have any prior information about what the fundamental concepts of sarcasm. it inculcates language understanding through self-supervised training techniques namely, masked words prediction and next sentence prediction. however, sarcasm doesn't work in the same way as declarative, exclamatory, imperative, and interrogatory sentences. these were the major type of sentences used for pretraining the language models. in future work, we plan to use a seq2seq based encoder-decoder model for instilling knowledge of sarcasm in the already available seq2seq models like t5 @xcite . wherein we'll use data similar to what was provided in subtask c and train a seq2seq model with input as the sarcastic text and the model will learn to paraphrase that input sentence into a non-sarcastic sentence as output.