Explainable Detection of Sarcasm in Social Media (2021)

Abstract: Sarcasm is a linguistic expression often used to communicate the opposite of what is said, usually something that is very unpleasant with an intention to insult or ridicule. Inherent ambiguity in sarcastic expressions makes sarcasm detection very difficult. In this work, we focus on detecting sarcasm in textual conversations, written in English, from various social networking platforms and online media. To this end, we develop an interpretable deep learning model using multi-head self-attention and gated recurrent units. We show the effectiveness and interpretability of our approach by achieving state-of-the-art results on datasets from social networking platforms, online discussion forums, and political dialogues.

## introduction sarcasm is a rhetorical way of expressing dislike or negative emotions using different language constructs, such as exaggeration or ridicule. it is an assortment of mockery and false politeness to intensify hostility without explicitly doing so. in face-to-face conversation, facial expressions, gestures, and tone of the speaker provide cues that help in identifying sarcasm. however, recognizing sarcasm in textual communication is not a trivial task as none of these cues are readily available. with the explosion of internet usage, sarcasm detection in online communications from social networking platforms, discussion forums, and e-commerce websites has become crucial for opinion mining, sentiment analysis, and identifying cyberbullies, online trolls. thus, developing computational models for automatic detection of sarcasm gathered pace in recent times with multiple studies and collection of new datasets @xcite @xcite . earlier works on sarcasm detection on texts use lexical (content) and pragmatic (context) cues @xcite such as interjections, punctuation, and sentimental shifts, which are major indicators of sarcasm @xcite . in these works, the features are hand-crafted which cannot generalize in the presence of informal language and figurative slang widely used in online conversations. with the advent of deeplearning, recent works @xcite @xcite @xcite , leverage neural networks to learn both lexical and contextual features, eliminating the need for hand-crafted features. in these works, word embeddings are incorporated to train deep convolutional, recurrent, or attention-based neural networks to achieve state-of-the-art results. while deep learning-based approaches achieve impressive performance, they lack interpretability. in this work, we also focus on the interpretability of the model along with its high performance. the main contributions of our work are: a) propose an interpretable model for sarcasm detection using selfattention. b) achieve state-of-the-art results on diverse datasets and exhibit the effectiveness of our model with extensive experimentation and ablation studies. c) exhibit the interpretability of our model by analyzing the learned attention maps. ## proposed approach our proposed approach consists of five components: data pre-processing, multi-head self-attention, gated recurrent units(gru), classification, and model interpretability. the architecture of our sarcasm detection model is shown in figure 1 . data pre-processing involves converting input text to word embeddings, required for training a deep learning model. we employ the pre-trained language model, bert @xcite , to extract word embeddings. we use these word embeddings which capture global context as we believe context is essential for detecting sarcasm. these embeddings form the input to our multi-head self-attention module which identifies words in the input text that provide crucial cues for sarcasm. in the next step, the gru layer aids in learning long-distance relationships among these highlighted words and output a single feature vector encoding the entire sequence. finally, a fully-connected layer with sigmoid activation is used to get the final classification score. multi-head self-attention given a sentence s, we apply a standard tokenizer and use pre-trained models to obtain d dimensional embeddings for individual words in the sentence. these embeddings @xmath0, e 2 , ..., e n }, @xmath1. to detect sarcasm in sentence s, it is crucial to identify specific words that provide essential cues such as sarcastic connotations and negative emotions. the importance of these cue-words is dependent on multiple factors based on different contexts. in our proposed model we leverage multi-head self-attention to identify these cue-words from the input text. attention is a mechanism to discover patterns in the input that are crucial for solving the given task. in deep learning, self-attention @xcite is an attention mechanism for sequences, which helps in learning the task-specific relationship between different elements of a given sequence to produce a better sequence representation. in the self-attention module, three linear projections: key (k), value (v ), and query (q) of the given input sequence are generated, where k, q,@xmath2. attention-map is computed based on the similarity between k, q, and the output of this module @xmath3. in multihead self-attention, multiple copies of the self-attention module are used in parallel. each head captures different relationships between the words in the input text and identify those keywords that aid in classification. in our model, we use a series of multi-head self-attention layers (#l) with multiple heads (#h) in each layer. gated recurrent units self-attention finds the words in the text which are important in detecting sarcasm. these words can be close to each other or farther apart in the input text. to learn long-distance relationships between these words, we use grus. these units are an improvement over standard recurrent neural networks and are designed to dynamically remember and forget the information flow using reset (r t ) and update (z t ) gates to solve the vanishing gradient problem. classification a single fully-connected feed-forward layer is used with sigmoid activation to compute the final output. input to this layer is the feature vector h n from the gru module and the output is a probability score @xmath4, 1], where ŷ ∈ {0, 1} is the binary label i.e., 1:sarcasm and 0:no-sarcasm. ## experiments we implement our model in pytorch @xcite , a deep-learning framework in python. to tokenize and extract word embeddings for the input text, we use publicly available resources @xcite . specifically, we use tokenizer and pre-trained weights from the "bert-base-uncased" model to convert words to tokens and then convert tokens to word embeddings. the embeddings for the words in the input text are passed through a series of multi-head self-attention layers #l , with multiple heads #h in each of the layers. the output from the self-attention layer is passed through a single bi-directional gru layer with it's hidden dimension @xmath5. the 512-dimensional output feature vector from the gru layer is passed through the fully connected layer to get a 1-dimensional output. a sigmoid activation is applied to the final output and bce loss is used to compute the loss between the ground truth and the predicted probability score. we use adam optimizer to train our model with approximately 13 million parameters, using a learning rate of 1e-4, batch size of 64, and dropout set 0.2. we use one nvidia pascal titan-x with 16gb memory for all our experiments. we set #@xmath6. details of these datasets, including the sample counts in train/test splits and the data source, are presented in twitter, 2013 @xcite , dialogues, 2016 @xcite , and reddit, 2018 @xcite . these are sourced from varied online platforms including social networks and discussion forums. and accuracy as evaluation metrics to test the performance of the trained models. apart from these standard metrics we also compute area under the roc curve (auc score) which is threshold independent. ## results we present the results of our experiments on multiple publicly available datasets in this section. results on the twitter dataset are presented in apart from twitter and reddit data, we also experimented with data from one other data source, i.e., political dialogues. in ## model interpretability attention maps from the individual heads of the selfattention layers provide the learned attention weights for each time-step in the input. in our case, each time-step is a word and we visualize the per-word attention weights for sample sentences with and without sarcasm from the sarc 2.0 main dataset. the model we used for this analysis has 5 attention layers with 8 heads per attention. figures 2 shows attention analysis @xcite for sample sentences with and without sarcasm respectively. each column in these figures corresponds to a single attention layer and attention weights between words in each head are represented using colored edges. the darkness of an edge indicates the strength of the attention weight. cls and sep are classifications and separator tokens from bert. attention analysis for a sentence with sarcasm, figure 2 shows that certain words receive more attention than others. for instance, words such as "just", "again", "totally", "!", have darker edges connecting them with every other word in a sentence. these are the words in the sentence which hint at sarcasm and as expected these receive higher attention than others. also, note that each cue word is attended by a different head in the first three layers of self-attention. in the final two layers, we observe that the attention is spread out to every word in the sentence indicating redundancy of these layers in the model. attention weight for a word is computed by first considering the maximum attention it receives across layers and then averaging the weights across multiple-heads in the layer. finally, the weights for a word are averaged over all the words in the sentence. the stronger the highlight for a word, the higher is the attention weight placed on it by the model while classifying the sentence. words from the sarcastic sentences with higher weights show that the model can detect sarcastic cues from the sentence. for example, the words "totally", "first", "ever" from the first sentence and "even", "until", "already" from the third sentence. these are the words that exhibit sarcasm in the sentences, which the model can successfully identify. in all the samples which are classified as nonsarcasm, the weights for the individual words are very low in comparison to cue-words from the sarcastic sentences. our model can predict a high score for sarcastic sentences and low scores for non-sarcastic sentences. ## conclusion in this work, we propose a novel multi-head selfattention-based neural network architecture to detect sarcasm in a given sentence. our proposed approach has 5 components: data pre-processing, multi-head selfattention module, gated recurrent unit module, classification, and model interpretability. multi-head selfattention is used to highlight the parts of the sentence which provide crucial cues for sarcasm detection. grus aid in learning long-distance relationships among these highlighted words in the sentence. the output from this layer is passed through a fully-connected classification layer to get the final classification score. exper-models precision recall f1 auc fracking sarcasm @xcite 88.3 87.9 88.1 -grnn @xcite 66.3 64.7 65.4 -elmo-bilstm @xcite 75.9 75.0 75.9 -elmo-bilstm full @xcite 77.8 73.5 75.3 -elmo-bilstm aug @xcite 68.4 70.8 69.4 -a2text-net @xcite 91.7 91.0 90.0 97.0