A Novel Hierarchical BERT Architecture for Sarcasm Detection (2020)

Abstract: Online discussion platforms are often flooded with opinions from users across the world on a variety of topics. Many such posts, comments, or utterances are often sarcastic in nature, i.e., the actual intent is hidden in the sentence and is different from its literal meaning, making the detection of such utterances challenging without additional context. In this paper, we propose a novel deep learning-based approach to detect whether an utterance is sarcastic or non-sarcastic by utilizing the given contexts ina hierarchical manner. We have used datasets from two online discussion platforms - Twitter and Reddit1for our experiments. Experimental and error analysis shows that the hierarchical models can make full use of history to obtain a better representation of contexts and thus, in turn, can outperform their sequential counterparts.

## introduction in the current scenario, social media serves as the biggest platform for people to express their opinion and share information. many organizations use this data to understand the choices of people and amend their policies accordingly. on these platforms, people often express their opinion sarcastically which is inherently difficult even for humans to analyze. for example "it is a wonderful feeling to carry an expensive phone with short battery life." is a sarcastic sentence that complains about the battery life of the phone but with the positive set of words like "wonderful". therefore it is essential to identify sarcastic responses to comprehend the users' demands and complaints. however, detecting sarcasm from a text is a difficult task as such sentences have positive surface 1 both the dataset are provided by the organizers of shared task of sarcasm detection in figlang-2020 sentiment but negative implied sentiment. for example, in the sentence "yeah right! i bought that nice expensive phone for this only!", the phrase "nice expensive" may imply positive sentiment from the user, but, the phrase "yeah right!" may render the whole sentence as a negative statement given enough background or context. sarcasm detection is not an independent area of study and is closely related to sentiment analysis. in order to detect sarcasm, many works like, @xcite , @xcite have proposed the use of hand-crafted features to identify a sarcastic response. however, with the advent of deep learning, it became possible to automatically learn and extract these features, thereby reducing both time and effort. many of these approaches have also been applied in complex nlp problems, for example, @xcite proposed a convolution neural network (cnn) to extract n-gram features automatically from the text. @xcite and @xcite showed the efficacy of recurrent neural networks (rnns) like lstms @xcite and grus @xcite in handling the long term dependencies. attention mechanisms @xcite have further improved the performance of complex nlp tasks like machine translation and reading comprehension by attending or focusing on the important words/ phrases from the inputs before making a decision. recently, transformers @xcite have outperformed many traditional and recent approaches in nlp by allowing one to learn from the huge amount of data. in this paper, we present a hierarchical bert @xcite based model for sarcasm detection for a given response and its context. our model, first, extracts the local features from the words in a sentence, and then uses a convolution module to summarize all the sentences in a context. the summarized context is then passed through a recurrent layer to extract the temporal features from the input. these temporal features are then convoluted with the input response to detect whether the response is sarcastic or not. ## related work the task of sarcasm detection can be formulated as a binary classification task i.e. given a sentence, the task is to predict whether it is sarcastic or not. another area of study involves labeling utterances in a dialogue as sarcastic or non-sarcastic using sequence labeling. these approaches usually fall into three different categories namely, rule-based, statistical, and deep learning-based. rule based approaches in @xcite , @xcite , @xcite and @xcite authors proposed the use of hand-crafted features and rulebased approaches to perform classification. in order to learn a decision boundary, one has to model all the hand-crafted features beforehand, which is a big disadvantage with such approaches. ## dataset for sarcasm detection, we have used 2 datasets namely: (1) twitter dataset and (2) reddit dataset provided for the fig-lang shared task foot_2 . both the datasets have a 'context' provided in the form of a conversation between the users, and the final 'response' that has to be classified as sarcastic or non sarcastic response, using context. the twitter and reddit dataset contains 5000 and 4400 train instances respectively and 1800 test instances each. the train set was further divided into an 80:20 ratio in a stratified fashion to obtain our final dev (evaluation) and train sets. ## approach in our proposed approach, we hypothesize that the context must have a significant role in deciding the sarcastic orientation of the response. hence, in order to capture the temporal features from the context, we processed the contexts in a hierarchical manner. in this section, we describe all the components of our proposed architecture. sentence encoding layer: to obtain the initial representation of the input, we used 2 separate encoder layers for the context and its response. the utterances in a context are passed through the first bert layer to extract sentence level features of a context. for instance, if our context contains 'm' different utterances then, the output of this layer would be s @xmath0,dsen,d bert ) where, d sen is the maximum sentence length and d bert is the length of word vector obtained from bert. in our experiments, we have used all the context provided in the input to obtain initial context representation. similarly, the second bert layer is used to encode the response in a fixed length vector s @xmath1,d bert . this representation is further passed through a bilstm layer to capture the semantic relationship between the words of a response. the final response output is denoted by o @xmath2,d lstm where, d lstm is the number of bilstm units. context summarization layer: the size of initial context vector s con after the sentence encoder layer becomes too large to process. for instance, if d bert is 768, d sen is 100 and m is 10 then our initial representation will be of size 10 × 100 × 768. thus, in order to obtain a summarized context, we projected the utterances to a lower dimension space using a convolution operation. to achieve this, we passed all the utterances through a 2d convolution layer with kernel size of (k row , k col ) and a stride of 1. we obtain d sum such feature maps to output our summary, sum @xmath3,dsen-k col ,dsum) . context encoder layer: since the utterances are sequential in nature, i.e., one utterance is uttered in response to the previous one, it is essential to capture the contextual information between the utterances to obtain a better context representation. we have used bilstms to output a sequence of hidden state vectors [h 1 , h 2 , ..., h m ] corresponding to each of the m input vectors, where m is m -k row . the vector h m can be seen as a short summary for whole the context just like a short summary of a paragraph or a book. the final output of this layer is o @xmath4,d lstm ) cnn layer: in @xcite author proposed a hybrid multi-channel cnn to capture the n-grams features in a text by varying the kernel size. as our final output o, we again used a 2d convolution layer to extract relations between response o res and context o con . to obtain different n-grams features as mentioned in @xcite , we used different shared matrices of sizes (2, 2), (2, 3), and (2, 5). finally, we applied a max-pool layer to extract the most relevant features from each of these ngram features. our final architecture is described in figure 1 . ## experiments and results for our experiments, we compare proposed approach with two baselines defined next: baseline 1: hierarchical attention network proposed in @xcite , the authors applied attention mechanism to classify large documents. we used this model to visualize the attention given to a particular context and words in response while making a decision. baseline 2: memory networks this experiment is done based on the implementation of @xcite . the intuition was to use (context, response) as a (key, value) pair, and given this information we predicted whether our value is sarcastic or not. results have been shown in the experimental settings all the parameters in our architecture were tuned on val set. we have used small version of bert with d @xmath5, the number of bi-lstms, d lstm were varied between the range 200, 300. throughout our experiments on test and val sets, we found that the optimum value for (k row , k col ) were (2,2) and were kept same throughout all other experiments. the maximum sentence length, d sen was fixed to 100 and the dropout values were adjusted as described in @xcite . we used cross-entropy as the loss function and adam as optimizer (with default values) for all the models. the f1 scores were used as an evaluation metric for validation set. all these parameters were tuned on val f1 scores to determine the final optimal values. ## results and analysis we have reported the results of all the experiments in this paper, we proposed a novel hierarchical bert based neural network architecture to handle context and response. from analysis and results, we supported the facts that the hierarchical model can effectively model the context and can produce a better representation of input before making a decision. applying bert on large documents and in hierarchical setting is still an open problem and we would like to explore this aspect in depth in our future works. further, we would like to obtain a better representation of context by compressing the bert representation of context in a much more efficient way (the context summarization layer).