Context-Aware Sarcasm Detection Using BERT (2020)

Abstract: In this paper, we present the results obtained by BERT, BiLSTM and SVM classifiers on the shared task on Sarcasm Detection held as part of The Second Workshop on Figurative Language Processing. The shared task required the use of conversational context to detect sarcasm. We experimented by varying the amount of context used along with the response (response is the text to be classified). The amount of context used includes (i) zero context, (ii) last one, two or three utterances, and (iii) all utterances. It was found that including the last utterance in the dialogue along with the response improved the performance of the classifier for the Twitter data set. On the other hand, the best performance for the Reddit data set was obtained when using only the response without any contextual information. The BERT classifier obtained F-score of 0.743 and 0.658 for the Twitter and Reddit data set respectively.

## introduction figurative language refers to texts where the intended meaning does not coincide with the literal meanings of the words and sentences that are used @xcite . an example of such a sentence is "the economic impact of covid-19 that we have seen so far is just the tip of the iceberg". the principle of compositionality which states the meaning of a sentence can be obtained by combining the meaning of the constituent words do not apply in such sentences. some of the types of figurative language are metaphor, idioms, similie, personification, hyperbole, understatement, analogy, irony, and sarcasm. the second workshop on figurative language processing, co-located with acl 2020, had two shared tasks: methaphor detection and sarcasm detection. the shared task on sarcasm detection is a binary classification task where it is required to determine if the final response given in a conversation dialogue is sarcastic or not. so, the task was sarcasm detection given the context in which the response was made. to capture the context the full dialogue thread was provided. the task was held for two different data sets: the twitter dataset and the reddit dataset. in this paper, we describe the work we performed for context aware sarcasm detection for both the data sets. we used the bidirectional encoder representations from transformers (bert), bidirectional long short-term memory (bilstm) and support vector machine (svm) classifiers in our study. the rest of this paper is structured as follows: section 2 discusses the related work that has been performed on automatic sarcasm detection, section 3 describes the data set used in this shared task, section 4 discusses the approach we used in our study, and section 6 discusses the results we obtained. ## data set the shared task on sarcasm detection required detecting sarcasm for two different data sets: the twitter dataset and the reddit dataset. tables 1 and 2 show the statistics of the two data sets. as can be seen from the tables, both the train data sets are balanced with 50% of the instances labelled as sarcasm and 50% labelled as not sarcasm. the tables also list the minimum and maximum number of utterances included from the conversational dialogue apart from the response. as can be seen, the twitter train set included from 2 to 20 utterances, the twitter test set included from 2 to 18 utterances, the reddit train set included from 2 to 8 utterances, and the reddit test set included from 2 to 13 utterances. it was observed that for 48%, 52%, and 63% of the total instances in the twitter train, twitter test, and reddit train data set only two utterances were present in the dialogue. in the case of the reddit test data set, 24% of the instances had only two utterances. the rest of the instances had more than two utterances in the dialogue. the two tables also show the minimum and the maximum length (in terms of number of tokens) of the string obtained by concatenating the response and all the utterances in the conversational dialogue. this length varied from 27 to 1213 for twitter train data set, from 24 to 786 for twitter test data set, from 12 to 422 for reddit data set, and from 19 to 646 for reddit data set. although the maximum length is high, it was seen that 73% and 75% of the instances in the twitter train and test had length less than 150, and 99% and 75% of the instances in reddit train and test data set had length less than 100 respectively. ## methodology this section discusses the details required for reproducing the results. it mentions the preprocessing steps, the architecture of the classifiers used, and hyperparameter values. ## experiments in our experiments, we concatenated the response with a varied number of utterances from the dialogue. it this way we varied the amount of context that was used to detect sarcasm. while concatenating, the utterances were concatenated in the reverse order so that the last utterances appeared at the start of the string. the individual utterances were separated from each other using a special token. we performed the following types of experiments: ## results and discussion tables 4 and 5 shows the results that our classifiers obtained on the test data sets. the scores mentioned in the tables were obtained from the submission page of the shared task in codalab foot_0 . we cross validated our models by retaining 20% of the train set as our development set. this technique has a disadvantage that the model we submit does not see 20% of the instances from the train set. so, the other form of cross validation we performed is the 5-fold cross validation. the development set results were used to filter out some of the models and hyperparameter values (as we did not use grid search). the implementation details of our models and the hyperparameter values used are discussed in section 4. as can be seen from twitter data set when only the response and the last utterance in the conversational dialogue was used. the maximum sequence length of 140 was used for this run. this result, however, is close to the f-score of 0.741 obtained using only the response (without any conversational context). on including the last two and the last three utterances from the dialogue, the performance of the bert classifier degraded considerably with f-score of 0.500 and 0.334 respectively. the reason for this could be that the sequence length was increased for these runs to accommodate the extra contextual information. however, the majority of the instances were of shorter length. thus, the zero-padding performed on the shorter instances might have degraded the performance. however, it was also found that the performance of the classifier improved considerably (compared to including last two and last three utterances as mentioned above) to an f-score of 0.734 when all the utterances in the context were used and the maximum sequence length was set to 300. the bilstm and svm classifier obtained fscores of 0.672 and 0.676 respectively on the twit-ter data set. all the utterances were used for these runs. overall, our best performing runs performed better than the base line scores that were obtained using a bilstm with attention based classifier @xcite . the classifiers obtained the ranks 14/36 and 21/37 in the leaderboard. however, as our best performing runs were submitted beyond the last date of the competition they have been removed from the leaderboard. ## conclusion in our work, we found that including context in the form of the last utterance in a dialogue chain slightly improved the performance of the bert classifier for the twitter data set compared to just using the response alone. for the reddit data set, including the context did not improve the performance. the best performance for the reddit data set was obtained when using only the response. approaches other that only concatenating the utterances to make use of the context needs to be investigated as future work.