Affective and Contextual Embedding for Sarcasm Detection (2020)

Abstract: Automatic sarcasm detection from text is an important classification task that can help identify the actual sentiment in user-generated data, such as reviews or tweets. Despite its usefulness, sarcasm detection remains a challenging task, due to a lack of any vocal intonation or facial gestures in textual data. To date, most of the approaches to addressing the problem have relied on hand-crafted affect features, or pre-trained models of non-contextual word embeddings, such as Word2vec. However, these models inherit limitations that render them inadequate for the task of sarcasm detection. In this paper, we propose two novel deep neural network models for sarcasm detection, namely ACE 1 and ACE 2. Given as input a text passage, the models predict whether it is sarcastic (or not). Our models extend the architecture of BERT by incorporating both affective and contextual features. To the best of our knowledge, this is the first attempt to directly alter BERT’s architecture and train it from scratch to build a sarcasm classifier. Extensive experiments on different datasets demonstrate that the proposed models outperform state-of-the-art models for sarcasm detection with significant margins.

## introduction sarcasm is the use of language in which one conveys implicit information/intention with the opposite meaning of what is said or written. due to this deliberate ambiguity, sarcasm detection is a challenging task, especially in written expressions where body gestures, tone of voice, and facial expression are not known @xcite . sarcasm detection has attracted growing interest over the past decade as it draws a more accurate picture of users' intention on social media @xcite , and facilitates accurate sentiment analysis in online comments and reviews @xcite . it also has useful applications in areas such as healthcare @xcite , hate speech detection @xcite , disaster management @xcite . early attempts of sarcasm detection from text mainly relied on extracting a set of positive verbs and negative/undesirable situations (e.g. "i love [positive verb] the pain of breakup [negative situation]") @xcite . alternatively, one may use lexical features (e.g., capital letters, and excessive usage of exclamatory marks) @xcite in sarcasm detection. recently, psychological studies have showed a strong relationship between affect/sentiment features (e.g., sadness, happiness) and sarcasm @xcite . however, relying only on affect/sentiment features for sarcasm detection may not be effective, especially when there are no sentiment words in a sentence @xcite . for instance, in the sentence "is it time for your medication or mine?", the speaker's intention is to mock the person addressed, but there aren't any sentiment words used. later attempts for sarcasm detection mostly relied on language models that are based on continuous representation or embeddings of words, such as word2vec @xcite and glove @xcite . use of these general models can eliminate the need for feature engineering or dependence on large emotion labeled datasets. however, due to the mechanism with which word vectors are learned and embedded to a space, these models have been shown to be inadequate for affective tasks @xcite . for instance, two dissimilar words like "good" and "bad", which often occur in a similar context, will be embedded closer to each other than the words "good" and "happy" that express similar emotions. more recent advances in neural representation models, such as elmo @xcite and bert @xcite can overcome this limitation by taking into account both the context a word appears in and its importance in that context, using a self-attention mechanism @xcite . while these models have been applied successfully to a wide range of natural language processing (nlp) tasks, such as sentiment analysis @xcite and word similarity computation @xcite , they have not been fully exploited for the more challenging problem of sarcasm detection. a few studies that proposed to use these models @xcite , utilize the already pre-trained embeddings, which are not optimized for sarcasm detection, and their performance can be further improved @xcite . in this paper, we propose two novel models for sarcasm detection based on affective and contextual embeddings, namely ace 1 and ace 2. given as input a text passage (i.e., a sequence of sentences, which we call a document in this paper for brevity), the models predict whether it is sarcastic. the architecture of each model builds upon two components: a) affective feature embedding, and b) contextual feature embedding. the former utilizes a bi-lstm with multi-head attention neural architecture @xcite to obtain representations of affective features of a document. the latter is achieved by a bert model. in ace 1, the two components are combined by training a new bert model from scratch by adding affective feature embeddings into the input sequence of bert so that task-specific embeddings can be obtained. in ace 2, the two components are combined in a fully connected layer with a softmax to form a classifier that is trained with labeled sarcasm detection data. the main contributions of this work are as follows: each model extends the architecture of bert by incorporating both affective and contextual features of text to build a classifier that can determine whether a document is sarcastic or not. to the best of our knowledge, this is the first attempt to directly alter bert's architecture and train it from the ground-up (rather than using the already pre-trained bert embeddings) for sarcasm detection. the paper is organized as follows. section 2 presents the related work and section 3 presents the proposed models. section 4 describes the evaluation and results. we conclude the paper in section 5. ## related work in this section, we present an overview of the related work on sarcasm detection including models that use affective features, contextual information, or a combination of the two. we also explain how our work aims to bridge the gap among existing efforts. due to space limitations, we only provide a short literature review here. for a more comprehensive coverage, we refer interested readers to the supplementary material (appendix a.1). other important related work is also cited in context, throughout the manuscript. identifying sarcasm in text has evolved from simple lexical-based and syntactic pattern models @xcite @xcite to complex models that consider refined linguistic features, such as positive predicates, interjections, gestural cues (emoticons, quotation marks, etc.) @xcite or behavior modelling @xcite . with the advent of deep learning, there has been a shift in how prediction models are designed for sarcasm detection. @xcite proposed a conditional lstm network @xcite for detecting sarcasm in twitter using sentence-level attention mechanisms on hashtags and @xcite utilized a knowledge-based model of affective features based on a wide range of lexical resources. while these models mostly relied on manually engineered affective features, @xcite proposed two bi-lstm models to learn to identify sarcasm in tweets. a drawback of many of these models is that they rely on self-contained content (e.g. twitter hashtags #) and thus do not generalize well, i.e., when such content is not available, the model fails to detect sarcasm. to address issues of lack of generalization and manual feature engineering, general word representation learning models have been proposed, such as word2vec @xcite and glove @xcite . these models learn embeddings of words that locate them close to one another in the embedded space, if they share common contexts in the corpus. however, this means that even words opposite in meaning to another (antonyms), such as happy and sad that often occur in similar contexts, would be embedded close to each other. as a result, the use of general models can be inadequate for affective tasks @xcite . more recently, transformer-based models such as bert @xcite , roberta @xcite and xlnet @xcite have been proposed to advance the state of the art in many nlp tasks by combining word embeddings with context embedding by using attention mechanisms in a bidirectional manner. @xcite proposed rcnn-roberta for sarcasm detection in social media by leveraging the pre-trained embeddings from roberta combined with a recurrent convolutional neural network. @xcite investigated the impact of data pre-processing on word representation learning for affective tasks. the main drawback of using advanced pre-trained models is that they do not incorporate any affect-specific features during the training phase of the model -therefore the accuracy of the model on affective tasks can be lessened. more sophisticated approaches for sarcasm detection have tried to combine the best of the two worlds by incorporating affective features with general embedding models during training. for example, @xcite proposed deepmoji by training a bi-lstm model with emojis to learn representations of emotional context, @xcite developed pre-trained sentiment, emotion and personality models for identifying sarcastic text, and @xcite utilized a multi-dimension intra-attention mechanism to overcome limitations of sequential models and capture words' incongruities for sarcasm detection. @xcite incorporated affective information into word representations by training a bi-lstm using corpora with weak affect labels and used such representations for sarcasm detection. another important observation for the problem of sarcasm detection is that domain dataset is critical @xcite . for instance, previous studies @xcite have employed datasets related to comedy to improve on the sarcasm detection task, a literary genre and a type of dramatic work that is often satirical in its tone. our research is mostly related to this line of work. in particular, we advance the state of the art in sarcasm detection by combining recently proposed transformer-based language models, such as bert and sbert, with affective-specific features that leverage domain data of sarcasm-rich corpora. ## proposed models for sarcasm detection we propose two models, ace 1 and ace 2, for sarcasm detection, where each model takes a document (i.e., a sequence of sentences) as input and predicts whether the document is sarcastic or not. figure 1 and 2 depict the architecture of each model that builds upon two components: a) affective feature embedding (afe) (on the right), and b) contextual feature embedding (cfe) (on the left). the two models are different in (1) the way the two components are combined and (2) the input to the affective feature ## conclusions we proposed two novel models (ace 1 and ace 2) that incorporate contextual and affective features in a deep neural network architecture for sarcasm detection. each model extends bert's architecture by incorporating into it affective features. ace 1 uses them to adjust the contextual embeddings and also fine-tune the model, while ace 2 uses them along with sbert for performing the final classification. our evaluation results showed that combining the two types of features greatly improves the sarcasm detection accuracy. in particular, deeply incorporating the affective features in the embedding training process (as in ace 1) is more beneficial than simply concatenating the two types of features (as in ace 2). we also observed that training embeddings with corpora containing rich sarcastic or emotional utterances greatly benefits the sarcasm detection tasks. our findings suggest that transformer-based models like bert can be trained to incorporate task-specific features to improve downstream task performance. as future work, we plan to investigate whether affective and contextual embeddings, e.g., wikisarca-bert trained in ace 1, can improve the performance of other tasks, such as emotion detection.