A Transformer Approach to Contextual Sarcasm Detection in T witter (2020)

Abstract: Understanding tone in Twitter posts will be increasingly important as more and more communication moves online. One of the most difficult, yet important tones to detect is sarcasm. In the past, LSTM and transformer architecture models have been used to tackle this problem. We attempt to expand upon this research, implementing LSTM, GRU, and transformer models, and exploring new methods to classify sarcasm in Twitter posts. Among these, the most successful were transformer models, most notably BERT. While we attempted a few other models described in this paper, our most successful model was an ensemble of transformer models including BERT, RoBERTa, XLNet, RoBERTa-large, and ALBERT. This research was performed in conjunction with the sarcasm detection shared task section in the Second Workshop on Figurative Language Processing, co-located with ACL 2020.

## introduction sarcasm detection is an important step towards complete natural language comprehension since a sarcastic phrase typically expresses a sentiment contradictory to its literal meaning. humans usually detect sarcasm with contextual clues, especially intonation, which is not available in textbased social media data. one challenge of sarcasm is the frequent necessity of prior knowledge. consider an example: context: driver tailgating a cyclist gets instant justice response: maybe he tried to save gas like you do when you tailgate a transport truck as humans, we can discern this response is sarcastic since we know a large car behind a small bike would not improve aerodynamics and hence gas mileage. this is a herculean inference for an algorithm. in other situations, sarcastic inference relies on knowledge about science, sports, politics, or movies. a broader challenge in classifying sarcasm is that a model trained on one dataset does not necessarily generalize to another one. using a twitter dataset from the second workshop on figurative language preprocessing, we tackle this difficult challenge of sarcasm detection and its specific issues as discussed above. ## datasets the workshop provided two balanced sarcasm datasets from both twitter and reddit with 5,000 and 4,400 observations respectively. the workshop collected and labeled the twitter data using the hashtags #sarcastic and #sarcasm, and preprocessed the data by replacing urls and user mentions with placeholders. the reddit dataset is a subset of that from @xcite . both datasets consist of a sarcasm label, response, and conversation context. both the test datasets contained 1800 observations. unfortunately, there were notable limitations in the twitter data. we removed 300 observations that were duplicates (further discussed in preprocessing). nuances in twitter data such as acronyms, hashtags, and emojis needed to be processed. furthermore, the twitter dataset had missing hashtags, affecting the meaning of the response. some tweets also contained images in the response, but there were no images in the data. for example, one tweet contained "they gave me the most (#beautiful, removed in the dataset) eggs for breakfast" with an image of chickens, but without the image it would be hard to determine its label. there was also a considerable domain shift between training and test data for the workshop. for all models, training and validation scores were significantly higher than test scores. ## related works and how we use them a wide variety of models have been created for sarcasm detection. our baseline model stems from the work of @xcite about sarcasm analysis using conversation context. this model involves one lstm reading the context and another reading the response. while ghosh and veale (2017) proposed a similar architecture based on bidirectional lstms to detect sarcasm in twitter, we found that bidirectional lstms performed similarly to unidirectional lstms. we used two different word embedding architectures, one from a previous paper trained on a separate twitter data set and one using cbow with position-weights, with character n-grams of length 5, a window of size 5 and 10 negatives @xcite . sentence embeddings were obtained from averaging the word embeddings. we experimented with both word-level and sentence-level attention models, but we found that attention-based models performed similarly to those without @xcite . the gru architecture seemed to be a promising alternative option to the lstm. exploring tasks on audio, handwriting, and musical classification, @xcite discovered that grus performed the same as lstms. their finding across multiple domains suggested that grus would perform similarly to lstms in the twitter domain. given the small size of our dataset and the reduced parameter size and complexity in grus, we believed this architecture could generalize better than lstms. we also experimented with transformer models, which have been very successful for other applications such as sentiment analysis, question answering, and recently even for sarcasm detection @xcite @xcite . we experimented with using pre-trained representations from bert as well as roberta, obtained from a bidirectional approach of a masked language model and elmo which uses a concatenation of representations obtained from a left-to-right and a rightto-left language model @xcite . we applied ensemble learning using various pretrained transformer models (bert, roberta, xlnet, roberta-large, albert with no fine-tuning on the transformer weights), where each model has a learned weighted sum for the hidden layers that is concatenated with hand-made features, that is then fed into a dense layer for classification similar to the work of @xcite . ## error analysis these results, however, are far below the f1-macro scores that were achieved in validation, which led us to believe that our training data may have come from a different distribution than our test data. in order to investigate this further, we calculated distributions of our features for both the train and test set. we found a few differences between the distributions indicating that there may be a covariate shift between the training and test sets. we provide a few results from this analysis in as mentioned above, we also saw a large difference between our validation and test scores. for example, we achieved a validation f1-macro score of 0.767 for our lstm model with attention. however, this model achieved an f1-macro score of 0.669 on the test set. additionally, our ensemble model went from a validation f1-macro score of 0.779 to a test f1-macro score of 0.686. in conjunction with the covariate shift, we believe that our training and test set come from different distributions which would greatly increase our error. ## discussion and conclusions this research can hopefully guide future work on the topic of detecting sarcasm in social media. on that note, we would like to provide a few suggestions that others may find helpful in tackling the problem and a few of the findings that resulted in the largest improvements for our results. first, utilizing multiple transformers and weighting them by performance seems to perform far better than a single transformer approach. presumably, this allows the model to contain the information provided in each of the embeddings as opposed to a single form of embeddings. next, we believe that this task requires a large amount of training data. we believe the reason the transformers performed so well was that they were pre-trained on large datasets. the models that we trained from scratch did not have as much training data, and we believe that they would have performed better with more training data. we believe that the weighted binary cross-entropy loss function to incorporate both datasets is a potential approach to help with this, and a future step would be to incorporate this into our final transformer models. additionally, as emojis are an incredibly prevalent form of communication on social media, as a future step, we would like to incorporate emojis into the embedding space of our models. finally, sarcasm detection is a difficult task. even for humans, it is difficult to determine whether individuals are being sarcastic online. there are many complex variables that are difficult to quantify when determining if a short post is sarcastic. therefore, capturing this notion of sarcasm within a model is difficult as well. we hope our techniques may be improved and expanded upon to solve other challenging natural language tasks.