Transformers on Sarcasm Detection with Context (2020)

Abstract: Sarcasm Detection with Context, a shared task of Second Workshop on Figurative Language Processing (co-located with ACL 2020), is study of effect of context on Sarcasm detection in conversations of Social media. We present different techniques and models, mostly based on transformer for Sarcasm Detection with Context. We extended latest pre-trained transformers like BERT, RoBERTa, spanBERT on different task objectives like single sentence classification, sentence pair classification, etc. to understand role of conversation context for sarcasm detection on Twitter conversations and conversation threads from Reddit. We also present our own architecture consisting of LSTM and Transformers to achieve the objective.

## introduction with advent of internet and social media platforms, it is important to know actual sentiments and beliefs of its users, and recognizing sarcasm is very important for this. we can't always decide if a sentence is sarcastic or not without knowing its context. for example, consider below two sentences s1 and s2. s1 : "what you love on weekends?" s2 : "i love going to the doctor." just by looking at the 's2' sentence we can tag the sentence 's2' as "not sarcastic", but imagine this sentence as a reply to the sentence 's1' , now we would like to tag the sentence 's2' as "sarcastic". hence it is necessary to know the context of a sentence to know sarcasm. we were provided with conversation threads from two of popular social media, reddit and twitter. for this objective we used different pre-trained language model and famous transformer architecture like bert @xcite , roberta @xcite and spanbert @xcite . we also propose our own architecture made of transformers @xcite and lstm @xcite . ## datasets two types of datasets were used, corpus from twitter conversations and conversation threads from reddit. twitter corpus @xcite introduced a self label twitter conversations corpus. the sarcastic tweets were collected by relying upon hashtags, like sarcasm, sarcastic, etc., that users assign to their sarcastic tweets. for non-sarcastic they adopted a methodology, according to which nonsarcastic tweet doesn't contain sarcasm hashtag instead they were having sentiments hashtag like happy, positive, sad, etc. reddit corpus @xcite collected 1.5 million sarcastic statement and many of nonsarcastic statement from reddit. they self annotated all of these reddit corpus manually. for both datasets, the training and testing data was provided in json format where each utterance contains the following fields: 1) "label" : sar-casm or not sarcasm. for test data, label was not provided. 2) "response" : the sarcastic response, whether a sarcastic tweet or a reddit post. 3) "context" : the conversation context of the "response". 4) "id" : unique id to identify and label each data point in test dataset. twitter data set is of 5,000 english tweets balanced between the "sarcasm" and "not sarcasm" classes and reddit dataset is of 4,400 reddit posts balanced between the "sarcasm" and "not sarcasm" classes. ## pre-process we used different text pre-processing technique to remove noise from text provided to us. we removed unwanted punctuation, multiple spaces, url tags, etc. we changed different abbreviations to their proper format, for example: "i'm" was changed to "i am", "idk" to "i don't know", etc. ## experiments we experimented with different transformers and pretrained models like bert , roberta, span-bert and our own architecture built over these transformers. for both datasets, each training and testing utterance contains two major fields: "response" (i.e, the sarcastic response, whether a sarcastic tweet or a reddit post), "context" (i.e., the conversation context of the "response"). the "context" is an ordered list of dialogue, i.e., if the "context" contains three elements, "c1", "c2", "c3", in that order, then "c2" is a reply to "c1" and "c3" is a reply to "c2". further, if the sarcastic "response" is "r", then "r" is a reply to "c3". for instance, for the following example, "label": "sarcasm", "response": "did kelly just call someone else messy? baaaahaaahahahaha", "context": ["x is looking a first lady should", "didn't think it was tailored enough it looked messy"]. the response tweet, "did kelly..." is a reply to its immediate context "didn't think it was tailored..." which is a reply to "x is looking...". for each utterance in datasets, we defined 'response' as response string and concatenation of all the 'context' in reverse order as context string. response @xmath0 we approached this classification task in two ways, first as single sentence classification task and second as sentence pair classification tasks. we also experimented single sentence classification only with response string. throughout the experiment we used 'transformers' library by hugging face @xcite for experimenting with bert and roberta models and for span-bert we used their official released code, and incorporated new methods to suit our task. ## related work most of the existing works are on detecting sarcasm without considering context. @xcite , @xcite , @xcite have considered context and utterances separately for sarcasm detection and showed how context is helpful in sarcasm detection. ## conclusion to conclude, we showed effective method for sarcasm detection and how much context is necessary for it. we didn't use any dataset (reddit and twitter) specific pre-processing or hyperparameter tuning in order to evaluate effectiveness of models across various types of data. in future, we would like to experiment with supplementing external data or merging different types of data on this task.