Sarcasm Detection Using an Ensemble Approach (2020)

Abstract: We present an ensemble approach for the detection of sarcasm in Reddit and Twitter responses in the context of The Second Workshop on Figurative Language Processing held in conjunction with ACL 2020. The ensemble is trained on the predicted sarcasm probabilities of four component models and on additional features, such as the sentiment of the comment, its length, and source (Reddit or Twitter) in order to learn which of the component models is the most reliable for which input. The component models consist of an LSTM with hashtag and emoji representations; a CNN-LSTM with casing, stop word, punctuation, and sentiment representations; an MLP based on Infersent embeddings; and an SVM trained on stylometric and emotion-based features. All component models use the two conversational turns preceding the response as context, except for the SVM, which only uses features extracted from the response. The ensemble itself consists of an adaboost classifier with the decision tree algorithm as base estimator and yields F1-scores of 67% and 74% on the Reddit and Twitter test data, respectively.

## introduction in this paper, an ensemble approach for the detection of sarcasm in social media data is described. the ensemble was designed in the context of the second workshop on figurative language processing held in conjunction with acl 2020 1 . it was the goal of the shared task to create a robust sarcasm detection model for tweets and reddit comments and investigate the role of conversational context in automatic sarcasm detection models. detecting sarcasm can be a challenging task, not only for machines, but also for humans, because sarcasm is subjective and culturally dependent, and because an utterance on its own can be both sarcastic and non-sarcastic (ghosh et al., 2018; joshi 1 https://sites.google.com/view/figlang2020/ @xcite . context is therefore vital for a correct interpretation of a comment on social media @xcite . for example, "well done, guys!" generally has a positive meaning, whereas it is used sarcastically in the context of a social media post about the governments mismanagement. therefore, conversational context is used in our approach described below to identify sarcasm. ## related research in this section, recent advances and papers related to sarcasm detection are described. the first advance is related to automatic annotation methods where the annotators use computational methods to obtain the labels @xcite , for instance by searching for "#sarcasm" in tweets (e.g. @xcite ). automatic labelling is often preferred to manual labelling, because it is faster, cheaper, allows for the creation of larger data sets, and because the author of an utterance knows best whether it was meant sarcastically or not. note that automatically annotated data can contain more false positives and/or false negatives than manually labeled data if the labeling method is not robust enough. an automatic method was used to label the data used in the present study and a more detailed description of that data can be found in section 3. a second advance in the field of sarcasm detection are pattern-based features @xcite . this term refers to using linguistic patterns as features. for example, @xcite use the pattern "presence of a positive verb and a negative situation" (e.g., "i love queueing for hours") as a feature. they hypothesized that this pattern is highly indicative of sarcasm. their approach achieved an f1-score of 51%. similarly, @xcite hypothesized that sentiment incongruity within an utterance signifies sarcasm. they did not only consider explicit expressions of sentiment, but also attempted to deal with sentiment implicitly embedded in world knowledge. to achieve this, the annotators gathered all real-world concepts that carried an implicit sentiment and labeled them with either a "positive" or "negative" sentiment label (e.g., "going to the dentist", which is usually associated with a negative sentiment). three approaches were then proposed that implemented these implicit sentiment labels. none of these approaches, however, outperformed the baseline (70% f1-score). thus, although sarcasm can be seen as an expression of sentiment, this study showed that successfully implementing sentiment in a classifier is not trivial. a third advance in the sarcasm detection field is using context as feature @xcite . three types of context can be distinguished: author context, e.g., @xcite , conversational context, e.g., @xcite , and topical context, e.g., @xcite . author context refers to the name of the author of the comment. the intuition behind using this type of context is that one individual uses sarcasm more regularly than another individual and can therefore improve the performance of sarcasm detection models. conversational context, on the other hand, refers to the conversational turns preceding (or following) the relevant utterance. as mentioned in the introduction, this type of context can clarify whether an utterance is sarcastic or not if that utterance can be perceived as both. finally, topical context is used, because it is hypothesized that certain topics (e.g., religion or politics) trigger more sarcastic responses than other topics. further, previous research has shown that for statistical models, svm performs best @xcite @xcite . conversely, the most successful deep learning algorithms are longshort term memory (lstm) networks and convolutional neural networks (cnn) @xcite . more recently, @xcite presented an lstm network with sentence-level attention heads that achieved the state-of-the-art performance: they reported f1-scores of 84% for sarcastic comments and 83% for non-sarcastic comments. the goal of their study was to investigate the role of conversational context in sarcasm detection and their experiments suggested that conversational context significantly improves the performance of their model. @xcite provide a survey of previous sarcasm detection studies and can be consulted for a more extensive overview of related research. ## data the training data comprises 5,000 tweets and 4,400 reddit comments, and was annotated automatically @xcite . each comment or "response" is accompanied by its context, i.e., an ordered list of all previous comments in the conversation, and a binary label indicating whether the response is sarcastic. both the twitter and reddit data are balanced. the test data contains 1,800 tweets and the same number of reddit comments. similar to the training data, the test responses are accompanied by their conversational context, and are balanced. in all instances, user mentions and urls are replaced with placeholders: "@user" and "<@xmath0, respectively. all data that was used in the present study was provided by the organizers of the workshop. however, the participating teams were allowed to collect extra data if desired. ## methodology four component models were used to construct the ensemble classifier. all of these models use conversational context as feature, with the exception of the svm model described in section 4.1.3, which focuses only on stylometric and emotion-based properties of the response. all other models use the two conversational turns preceding the response as context, since this was the minimum amount of context that was provided for each response. ## results in this section, the performance of the component models and of the ensemble model are described. the models were evaluated on the reddit test set and twitter test set separately, and f1-score was used as the official evaluation metric. in from these results, it can be concluded that the ensemble model has higher precision, recall, and f1-score than the models in isolation. this suggests that the ensemble does not simply predict the same label as the overall best performing component model, but learns which model performs best and when. what component model is globally the most robust, depends on the type of data (reddit or twitter) and on the setting (training or test data). nevertheless, each component model contributes to the results and therefore seems to capture different sarcasm characteristics, as evidenced by the increase in performance when all the models are combined through the ensemble and by an ablation study we conducted: removing any of the component models results in a decrease in performance. further, the results show that both official test sets contain comments that are, on average, more challenging to classify than the training data, since the 10-fold cross-validation scores ( ## conclusion we described an ensemble approach for sarcasm detection in reddit and twitter comments. the model consists of an adaboost classifier with the decision tree algorithm as base estimator and learns the sarcasm probabilities predicted by four different component models: an lstm model that uses word, emoji and hashtag representations; a model that uses cnns and lstm networks to learn word embeddings, and dense networks to learn punctuation, casing, sentiment and stop word features; an mlp based on infersent embeddings; and an svm approach that captures the stylometric and emotional characteristics of sarcastic content. all component models (except svm) use conversational context to make predictions, namely the two turns preceding the response. the sarcasm probabilities used to train the ensemble were obtained by training the component models using 10-fold cross-validation and saving the labels predicted for the validation set in each fold. in order to learn which model performs best and for what input, the ensemble also uses the lengths, the source and sentiment scores of the response and context as features. the ensemble yields f1-scores of 67% and 74% on the reddit and twitter test data, respectively. the imbalance between the reddit and twitter scores is consistent in all component models, suggesting that the reddit data is inherently more challenging to classify. however, more research on why this is the case is needed. future work may also include experimenting with other component models to improve the overall performance of the ensemble.