Does Commonsense help in detecting Sarcasm? (2021)

Abstract: Sarcasm detection is important for several NLP tasks such as sentiment identification in product reviews, user feedback, and online forums. It is a challenging task requiring a deep understanding of language, context, and world knowledge. In this paper, we investigate whether incorporating commonsense knowledge helps in sarcasm detection. For this, we incorporate commonsense knowledge into the prediction process using a graph convolution network with pre-trained language model embeddings as input. Our experiments with three sarcasm detection datasets indicate that the approach does not outperform the baseline model. We perform an exhaustive set of experiments to analyze where commonsense support adds value and where it hurts classification. Our implementation is publicly available at: https://github.com/brcsomnath/commonsense-sarcasm .

## introduction & related work the topic of sarcasm has received attention in various research fields like linguistics @xcite , psychology @xcite and the cognitive sciences @xcite . identifying sarcasm is essential to understanding the opinion and intent of a user in downstream tasks like opinion mining, sentiment classification, etc. initial approaches for this task @xcite ) mostly relied on handcrafted features to capture the lexical and contextual information. on similar lines, the efficacy of special characters, emojis and n-gram features in the discrimination task have also been studied @xcite . in recent years, this task has gained traction in the machine learning and computational linguistic community @xcite @xcite @xcite "i loved the movie so much that i left during the interval" to watch the movie to watch it to go to the theatre to be fun to go home to be alone to watch something else ## model we use a graph convolution-based model to enable incorporation of comet sequences given an input sentence. the sentence representations are retrieved from the pre-trained encoder of our baseline model. our baseline model consists of a transformer @xcite based distil-bert @xcite is a light-weight encoder, which enables faster training, while achieving similar performance as other transformer based encoders. the model is shown in figure 2 . for every input instance, a graph is formed with edges between the input sentence and comet sequences. no edges are present between individual comet sequences. sentence embeddings retrieved from the baseline distilbert form the initial graph embeddings. the intuition behind leveraging a graph-based architecture was to enable information flow between the representations of the input sentence and comet sequences, thereby reducing the domain discrepancy between them. the graph is then fed into a graphsage @xcite network which produces the node embedding vector @xmath0, where m is the number of comet sequences and n is the output dimension of the gcn. the node embedding vector v is then forwarded to a fully connected neural network layer to produce the final output. in section 4, we experiment with different edge configurations and observe how each edge configuration affects the downstream performance. we experimented with another model that incorporated comet sequences with an attentionmechanism. in that model, the representation of the input sentence was concatenated with an aggregate representation of the comet sequences, formed in an attentive fashion. its performance was not better than the gcn-based model, so we do not describe it here. ## experimental setup we evaluate the models on three datasets (a) irony detection semeval task: van hee et al. (2018) conducted a semeval task for irony detection considering an utterance in isolation. they also released a secondary task where the sarcastic samples were classified into three broad categories: verbal irony with polarity contrast, situational irony, and others. (b) news headlines dataset (misra and we experiment on the reddit dataset of the shared task introduced by @xcite . the statistics of the datasets are specified in all the aforementioned datasets are balanced. we report our results by randomly splitting into training and test set, and averaging the accuracy over 5 iterations. in our experiments, we incorporate a subset of comet predicates (xwant and xeffect) related to the subject in a sentence. ## results we report the classification accuracy of the models for all datasets in we conduct an ablation study with three configurations of the graph edges input: "they should have put $125 million termination payout in each of their contract" xwant : to save money we wish to have more complete comet sentences like: they wanted to save money. in order to improve the setup, we replace comet sequences with complete sentences of the form: [subject] we also measure the share of instances in the test set having the same predicted label from the baseline and the model. we observe a significant overlap (>90%) between the predictions of the baseline and the proposed model across all datasets in ## saliency test we perform saliency tests to investigate whether the model is reliant on commonsense sequences while taking decisions. (a) gradient-based saliency @xcite ) measure for a feature x i given an output class c is computed as ∇ x i l(y, f (x)) • x i , where l(•, •) is the loss function. the saliency map is shown in figure 3 . the saliency map vector has a dimension of 3 × 768, where the first row showcases the saliency values of the features corresponding to the input sentence while the remaining two rows correspond to the saliency of comet sequences. for better visualization, all values are normalized between 0-1 and average pooling is performed on adjacent blocks of 8 to form a vector of dimension 3 × 192. from figure 3 , it is evident that the model learns to identify important input features but assigns similar saliency values to all comet features. (b) occlusion-based saliency test involves occlud-dataset c n s gcn news headline 83.8% semeval irony 64.6% figlang 2020 reddit 96.8% ## efficacy of commonsense in this section, we try to uncover why comet sequences don't help in the sarcasm detection task. in order to identify instances where commonsense incorporation hurts the performance, we focus on samples where the model's prediction is wrong but the baseline is correct. among these samples, we measure how many were non-sarcastic by defining a new measure non-sarcastic class coverage, ground truth input sentence comet support (xwant and xeffect) explanations non-sarcastic @usertag i wonder if they have that in an audio book • he gets to learns something new • he wants to be entertained comet sequences don't add value for classifying the non-sarcastic sample. sarcastic going to watch a movie about murder. merry christmas ;) • the person wants to have fun • the person gets tired as a result comet sequences fail to explain the satire. sarcastic final at 7am, i'm ready • the person wants to go to bed • the person has to go to work comet captures the contrast between the intentions and results. sarcastic as a girl my reason not to put on makeup is i'm satisfied with my face • she wanted to look pretty • as a result she got complements comet doesn't provide relevant commonsense for capturing polarity contrast. there are cases where the prediction failed either due to noisy input (prevalent in the twitter based semeval dataset) or subtle play of words which comet sequences fail to explain. in order to investigate the utility of commonsense for specific type of sarcasm, we form a subset of the semeval irony dataset with samples only from irony with polarity contrast and non-sarcastic class by leveraging labels from the secondary se-meval task (mentioned in section 3). c n s gcn for the new dataset is 57.1%, a significant reduction from the 64.6% in semeval dataset in ## qualitative analysis in this section, we analyze a few examples shown in ## conclusion and future work in this paper, we proposed the idea of integrating commonsense knowledge in the task of sarcasm detection. we observe that comet infused model performs at par with the baseline. through saliency tests, we observe that the model is less reliant on the commonsense representations in many cases. from our analysis, we infer that commonsense is most effective in identifying sarcasm with polarity contrast but fails to explain non-sarcastic samples or other types of sarcasm effectively, which hurts the overall performance. in the future, we will explore the utility of other forms of external knowledge such as factual world knowledge for sarcasm detection. we will also try to leverage commonsense to explain why a certain remark is sarcastic.