D etecting S arcasm in C onversation C ontext U sing T ransformer- B ased M odels (2020)

Abstract: Sarcasm detection, regarded as one of the sub-problems of sentiment analysis, is a very typical task because the introduction of sarcastic words can flip the sentiment of the sentence itself. To date, many research works revolve around detecting sarcasm in one single sentence and there is very limited research to detect sarcasm resulting from multiple sentences. Current models used Long Short Term Memory (LSTM) variants with or without attention to detect sarcasm in conversations. We showed that the models using state-of-the-art Bidirectional Encoder Representations from Transformers (BERT), to capture syntactic and semantic information across conversation sentences, performed better than the current models. Based on the data analysis, we estimated that the number of sentences in the conversation that can contribute to the sarcasm and the results agrees to this estimation. We also perform a comparative study of our different versions of BERT-based model with other variants of LSTM model and XLNet (both using the estimated number of conversation sentences) and find out that BERT-based models outperformed them.

## introduction for many nlp researchers from both academia and industry, sarcasm detection has been one of the most focused areas of research among many research problems like code-mixed sentiment analysis @xcite , detection of offensive or hate speeches @xcite , questionanswering(soares and parreiras, 2018), etc. one of the main reasons why sarcasm finds a significant portion of research work is because of its nature that the addition of a sarcastic clause or a word can alter the sentiment of the sentence. sarcasm is used to criticize people, to provide political or apolitical views, to make fun of ideas, etc., and the most common form of sarcasm usage is through text. some major sources of the sarcastic text are social media platforms like twitter, instagram, facebook, quora, whatsapp etc. out of these, twitter forms the major source of sarcastic content drawing attention from researchers across the globe @xcite @xcite . due to its inherent nature of flipping the context of the sentence, sarcasm in a sentence is difficult to detect even for humans @xcite . here, the context is considered only in one sentence. how do we deal with situations where the sarcastic sentence depends on a conversation context and the context spans over multiple sentences preceding the response sarcastic sentence? addressing this problem may help in identifying the root cause of sarcasm in a larger context, which is even tougher because conversation sentences differ in number, some conversation sentences themselves may be sarcastic and response text may depend on more than one conversation sentences. this is the research problem that we are trying to address and are largely successful in building better models which outperformed the baseline f-measures of 0.6 for reddit and 0.67 for twitter datasets @xcite . we have achieved fmeasures of 0.752 for twitter and 0.621 for reddit datasets. ## related work sarcasm is a form of figurative language where the meaning of a sentence does not hold and the interpretation is quite contrary. a quick survey about sarcasm detection and some of the earlier approaches is compiled by @xcite . the problem of sarcasm detection is targeted in field field description label sarcasm or not sarcasm response tweet or a reddit post context ordered list of dialogue ## dataset description the data 1 we used for model building is taken from sarcasm detection shared task of the sec-1 https://competitions.codalab.org/competitions/22247 ond workshop on figurative language processing (figlang2020). there are two types of data provided by the organizers: 1. twitter dataset and 2. reddit dataset. training data contains the fields -"label", "response" and "context" and are described as shown in the if the "context" contains three elements, "c1", "c2", "c3", in that order, then "c2" is a reply to "c1" and "c3" is a reply to "c2". further, if the sarcastic "response" is "r", then "r" is a reply to "c3". consider the example provided by the organizers: label: "sarcasm" response: "did kelly just call someone else messy? baaaahaaahahahaha" context: ["x is looking a first lady should . #classact, "didn't think it was tailored enough it looked messy"] this example can be understood as "did kelly..." is a reply to its immediate context "didn't think it was tailored..." which is a reply to "x is looking...". and the label of the response is "sarcasm". testing data contains the fields -"id", "response" and "context" and are described as shown in the the data of both twitter tweets and reddit posts were organized into train and test sets. the number of samples in each of these datasets is shown in ## system description there are several nlp models at our disposal to work with, some are pre-trained while others need to be trained from scratch. we have done experiments with lstm, bilstm, stacked lstm and cnn-lstm (convolution neural network + lstm) models which can be trained to capture sequence information. to avoid over-fitting, we have introduced dropout layers and taken early stopping measures while training. we split the training data into train data (to train the model) and validation data (10 percent of actual training data to validate the model and employ early stopping). we also have worked with pre-trained transformer based bert (bert-base-uncased) model and xlnet. the following steps are used to fine-tune the pre-trained bert model: in our model, training stops when f1-score on validation data goes below the earlier epoch's f1-score and the prediction is done on the earlier model for which validation f1-score is highest. similar steps are performed to fine-tune xlnet model. ## results the lstm model variants -lstm, bilstm, stacked-lstm and conv-lstm models are applied to twitter dataset and the f1-scores on test data are 0.67, 0.66, 0.66 and 0.67 respectively. the f1-scores of variants of bert models considering different lengths of conversation sentences and xl-net are depicted in ## conclusion sarcasm detection in conversational context is an important research area which infuses more enthusiasm and encourages the researchers across the globe. we build models that outperformed the baseline results. though the results in the shared task leaderboard shows that the top model achieved fmeasure of 0.93 for the twitter dataset and 0.83 for the reddit dataset, there is a lot to work on the problem and find ways to improve the performance with a larger dataset. use of a larger dataset might help in adding more context and help in improving accuracy. currently, the models that are built are not generalised across datasets. further research can focus on building a generalized model for multiple datasets.