Transformer-based Context-aware Sarcasm Detection in Conversation Threads from Social Media (2020)

Abstract: We present a transformer-based sarcasm detection model that accounts for the context from the entire conversation thread for more robust predictions. Our model uses deep transformer layers to perform multi-head attentions among the target utterance and the relevant context in the thread. The context-aware models are evaluated on two datasets from social media, Twitter and Reddit, and show 3.1% and 7.0% improvements over their baselines. Our best models give the F1-scores of 79.0% and 75.0% for the Twitter and Reddit datasets respectively, becoming one of the highest performing systems among 36 participants in this shared task.

## introduction sarcasm is a form of figurative language that implies a negative sentiment while displaying a positive sentiment on the surface @xcite . because of its conflicting nature and subtlety in language, sarcasm detection has been considered one of the most challenging tasks in natural language processing. furthermore, when sarcasm is used in social media platforms such as twitter or reddit to express users' nuanced intents, the language is often full of spelling errors, acronyms, slangs, emojis, and special characters, which adds another level of difficulty in this task. despite of its challenges, sarcasm detection has recently gained substantial attention because it can bring the last gist to deep contextual understanding for various applications such as author profiling, harassment detection, and irony detection @xcite . many computational approaches have been proposed to detect sarcasm in conversations @xcite @xcite . however, most of the previous studies use the utterances in isolation, which makes it hard even for human to detect sarcasm without the contexts. thus, it's essential to interpret the target utterances along with contextual information comprising textual features from the conversation thread, metadata about the conversation from external sources, or visual context @xcite @xcite . this paper presents a transformer-based sarcasm detection model that takes both the target utterance and its context and predicts if the target utterance involves sarcasm. our model uses a transformer encoder to coherently generate the embedding representation for the target utterance and the context by performing multi-head attentions (section 4). this approach is evaluated on two types of datasets collected from twitter and reddit (section 3), and depicts significant improvement over the baseline using only the target utterance as input (section 5). our error analysis illustrates that the context-aware model can catch subtle nuance that cannot be captured by the target-oriented model (section 6). ## related work just as most other types of figurative languages are, sarcasm is not necessarily complicated to express but requires comprehensive understanding in context as well as commonsense knowledge rather than its literal sense @xcite . various approaches have been presented for this task. most earlier works had taken the target utterance without context as input. both explicit and implicit incongruity features were explored in these works @xcite . to detect whether certain words in the target utterance involve sarcasm, several approaches based on distributional semantics were proposed @xcite . additionally, word embedding-based features like distance-weighted similarities were also adapted to capture the subtle forms of context incongruity @xcite . nonetheless, it is difficult to detect sarcasm by considering only the target utterances in isolation. non-textual features such as the properties of the author, audience and environment were also taken into account @xcite . both the linguistic and context features were used to distinguish between information-seeking and rhetorical questions in forums and tweets @xcite . traditional machine learning methods such as support vector machines were used to model sarcasm detection as a sequential classification task over the target utterance and its surrounding utterances @xcite . recently, deep learning methods using lstm were introduced, considering the prior turns @xcite as well as the succeeding turns @xcite . ## data description given a conversation thread, either from twitter or reddit, a target utterance is the turn to be predicted, whether or not it involves sarcasm, and the context is an ordered list of other utterances in the thread. the twitter data is collected by using the hashtags #sarcasm and #sarcastic. the reddit data is a subset of the self-annotated reddit corpus that consists of 1.3 million sarcastic and non-sarcastic posts @xcite . every target utterance is annotated with one of the two labels, sarcasm and not_sarcasm. notice the huge variances in the utterance lengths for both the twitter and the reddit datasets. for the reddit dataset, the average lengths of conversations as well as utterances are significantly larger in the test set than the training set that potentially makes the model development more challenging. ## approach two types of transformer-based sarcasm detection models are used for our experiments: these two models are coupled with the latest transformer encoders e.g., bert @xcite , roberta @xcite , and albert @xcite , and compared to evaluate how much impact the context makes to predict whether or not the target utterance involves sarcasm. ## analysis for a better understanding in our final model, errors from the following three situations are analyzed (to: target-oriented, ca: context-aware): utterance c 1 who has ever cared about y * utube r * wind . c 2 @user back when youtube was beginning it was a cool giveback to the community to do a super polished high production value video with yt talent . not the same now . the better move for them would be to do like 5-6 of them in several categories to give that shine . t @user @user i look forward to the eventual annual tubies awards livestream . (a) example when to is wrong and ca is correct. utterance c 1 i am asking the chairs of the house and senate committees to investigate top secret intelligence shared with nbc prior to me seeing it. c 2 @user good for you, sweetie! but using the legislative branch of the us government to fix your media grudges seems a bit much. t @user @user @user you look triggered after someone criticizes me, are conservatives skeptic of ppl in power? (b) example when to is correct and ca is wrong. utterance c 1 if i could start my #brand over, this is what i would emulate my #site to look like .. and i might, once my anual contract with #wordpress is up . even tho i don't think is very; i can't help but to find ... <@xmath0; wannabe fast due to caused by, and being just another and. is the light, bringing color back to this sad world of and. (c) example when both to and ca are wrong. ## conclusion this paper explores the benefit of considering relevant contexts for the task of sarcasm detection. three types of state-of-the-art transformer encoders are adapted to establish the strong baseline for the target-oriented models, which are compared to the context-aware models that show significant improvements for both twitter and reddit datasets and become one of the highest performing models in this shared task.