Neural Sarcasm Detection using Conversation Context (2020)

Abstract: Social media platforms and discussion forums such as Reddit, Twitter, etc. are filled with figurative languages. Sarcasm is one such category of figurative language whose presence in a conversation makes language understanding a challenging task. In this paper, we present a deep neural architecture for sarcasm detection. We investigate various pre-trained language representation models (PLRMs) like BERT, RoBERTa, etc. and fine-tune it on the Twitter dataset. We experiment with a variety of PLRMs either on the twitter utterance in isolation or utilizing the contextual information along with the utterance. Our findings indicate that by taking into consideration the previous three most recent utterances, the model is more accurately able to classify a conversation as being sarcastic or not. Our best performing ensemble model achieves an overall F1 score of 0.790, which ranks us second on the leaderboard of the Sarcasm Shared Task 2020.

## introduction sarcasm can be defined as a communicative act of intentionally using words or phrases which tend to transform the polarity of a positive utterance into its negative counterpart and vice versa. the significant increase in the usage of social media channels has generated content that is sarcastic and ironic in nature. the apparent reason for this is that social media users tend to use various figurative language forms to convey their message. the detection of sarcasm is thus vital for several nlp applications such as opinion minings, sentiment analysis, etc @xcite . this leads to a considerable amount of research in the sarcasm detection domain among the nlp community in recent years. the shared task on sarcasm detection 2020 aims to explore various approaches for sarcasm detection in a given textual utterance. specifically, the task is to understand how much conversation context is needed or helpful for sarcasm detection. our approach for this task focuses on utilizing various state-of-the-art plrms and fine-tuning it to detect whether a given conversation is sarcastic. we apply an ensembling strategy consisting of models trained on different length conversational contexts to make more accurate predictions. our best performing model (team name -nclabj) achieves an f1 score of 0.790 on the test data in the coadalab evaluation platform. ## problem description the dataset assigned for this task is collected from the popular social media platform, twitter. each training data contains the following fields: "label" (i.e., "sarcasm" or "notsarcasm"), "response" (the tweet utterance), "context" (i.e., the conversation context of the "response"). our objective here is to take as input a response along with its optional conversational context and predict whether the response is sarcastic or not. this problem can be modeled as a binary classification task. the predicted response on the test set is evaluated against the true label. three performance metrics, namely, precision, recall, and f1 score are used for final evaluation. ## related work various attempts have been made for sarcasm detection in recent years @xcite . researchers have approached this task through different methodologies, such as framing it as a sense disambigua-tion problem @xcite or considering sarcasm as a contrast between a positive sentiment and negative situation @xcite @xcite @xcite . recently, few works have taken into account the additional context information along with the utterance. @xcite demonstrate how additional contextual information beyond the utterance is often necessary for humans as well as computers to identify sarcasm. @xcite propose a multi-modal approach to combine textual and visual features for sarcasm detection. @xcite ) model sarcasm detection as a sequence labeling task instead of a classification task. @xcite investigated that the conditional lstm network @xcite and lstm networks with sentence-level attention on context and response achieved significant improvement over the lstm model that reads only the response. therefore, the new trend in the field of sarcasm detection is to take into account the additional context information along with the utterance. the objective of this shared task is to investigate how much of the context information is necessary to classify an utterance as being sarcastic or not. ## system description we describe our proposed system for sarcasm detection in this section. we frame this problem as a binary classification task and apply a transfer learning approach to classify the tweet as either sarcastic or not. we experiment with several state of the art plrms like bert @xcite , roberta @xcite , as well as pre-trained embeddings representations models such as elmo @xcite , use @xcite , etc. and fine-tune it on the assigned twitter dataset. we briefly review these models in subsections 4. for fine-tuning, we add additional dense layers and train the entire model in an end to end manner. figure 1 illustrates one such approach for fine-tuning a roberta model. we sequentially unfreeze the layers with each ongoing epoch. we apply a model ensembling strategy called "majority voting", as shown in figure 2 to come out with our final predictions on the test data. in this ensemble technique, we take the prediction of several models and choose the label predicted by the maximum number of models. figure 2: the majority voting ensemble methodology consisting of three sample models ## conclusion & future work in this work, we have presented an effective methodology to tackle the sarcasm detection task on the twitter dataset by framing it as a binary classification problem. we showed that by finetuning plrms on a given utterance along with its specific length context history, we could successfully classify the utterance as being sarcastic or not. we experimented with different length context history and concluded that by taking into account the most recent three conversation turns, the model was able to obtain the best results. the fine-tuned roberta large model outperformed every other experimented models in terms of precision, recall, and f1 score. we also demonstrated that we could obtain a significant gain in the performance by using a simple ensembling technique called majority voting. in the future, we would like to explore with these plrms on other publicly available datasets. we also aim to dive deep into the context history information and derive insights about the contextual part, which helps the model in improvising the classification result. we also wish to investigate more complex ensemble techniques to observe the performance gain.