Irony Detection for D utch: a Venture into the Implicit (2022)

Abstract: This paper presents the results of a replication experiment for automatic irony detection in Dutch social media text, investigating both a feature-based SVM classifier, as was done by Van Hee et al. (2017) and and a transformer-based approach. In addition to building a baseline model, an important goal of this research is to explore the implementation of common-sense knowledge in the form of implicit sentiment, as we strongly believe that common-sense and connotative knowledge are essential to the identification of irony and implicit meaning in tweets. We show promising results and the presented approach can provide a solid baseline and serve as a staging ground to build on in future experiments for irony detection in Dutch.

## introduction irony is traditionally defined as a rhetorical device where an evaluative utterance expresses the opposite of what is actually intended @xcite @xcite . in order to understand the intended implicit meaning of such an ironic utterance, the message often requires presupposed common-sense knowledge. we expect most people to know that 'walking in the rain' is not pleasant or that 'visiting the dentist' can result in a painful experience. in addition to such presupposed knowledge, a sarcastic or ironic utterance is often enriched with an explicit mention of the opposite sentiment foot_0 . when using sarcasm foot_1 , we usually are not just 'fine' with walking in the rain but we say we 'love it'. people are able to recognize such figurative language because we possess both the common-sense knowledge and can catch explicit semantic or lexical cues. previous research has proven the value of lexical and semantic features for irony detection and has shown that they already allow us to recognize some cases of irony @xcite . gathering and modeling the common-sense knowledge required for irony recognition is more problematic. how can we expect an automatic system to know that an implicit negative connotation is attached to an event expressed in a given utterance, if the exact opposite information is provided in a text like 'oh god, i love it when i have to walk home in the rain!'. in our research, we aimed to identify the general implicit sentiment behind a concept or event by looking at the tweets other people have posted containing that very concept or event. if 9 out of 10 people complain about 'walking in the rain', people who say they 'love it' might very well say that ironically. the combination of lexical features and this kind of data-driven common sense are the foundation of our machine learning approach for irony detection, which has already been applied successfully to english data @xcite . yet, transferring the methodology from a high-resource language (english) to a lesser-resourced language (dutch) is not as straight-forward as it might seem. english language models and lexicons can generally rely on larger amounts of data, and not every previously-used resource for the task (e.g. sentic-net @xcite ) includes dutch data or has a dutch counterpart. in addition, the concept of irony might be language-universal, but the realization of irony might employ different languagespecific tools and structures. in the next section, we will discuss related research and the most recent approaches applied to irony detection. next, we give a short description of the experimental corpus. section 4 follows an elaborate description of the proposed systems and an explanation of the features we developed for our classifiers. finally, we present the results of our experiments and we wrap up the paper with a conclusion and identify some avenues for future research. ## related research the detection of sarcasm and irony remains one of the primary hurdles for sentiment analysis and other natural language processing (nlp) tasks like detection of cyberbullying, humor and toxicity. when it comes to methodology, the techniques applied to english irony detection range from feature-based classifiers to neural networks and transformers, including many combinations of (transformer-generated) embeddings with neural or traditional classifiers. feature-based classifiers like support vector machines or classifiers (svm or svc) are flexible and can easily be equipped with new features, but they generally require a lot of manual work. @xcite provides a successful example of an approach combining lexical, semantic and syntactic features for a strong baseline system. this kind of supervised classifier is advantageous when determining the informative value a specific linguistic feature (like part-of-speech tags) contributes to the decision-making process. transformer language models @xcite and bidirectional transformers @xcite currently occupy the throne of the state-of-the-art in nlp. while these language models are remarkably adaptable and perform well for a variety of tasks (providing they have been fine-tuned for classification), they usually do not suffice on their own. often the language models are used to generate word embeddings as a (partial) input for a traditional or neural classifier. @xcite combine the embeddings from roberta, a robust bi-directional transformer approach, with a recurrent convolutional neural network. cignarella et al. (2020) use a long short-term memory (lstm) neural network to exploit both transformer (bert) word embeddings and syntactic dependency-based n-gram features @xcite . in the semeval 2018 shared task for irony detection @xcite , the best-performing model @xcite exploited word embeddings, syntactic and sentiment features using a long short-term memory neural network. other participants made use of ensemble learning techniques with majority voting on neural network approaches. one example is presented by @xcite , who used ensemble learning of two lstm models, one exploiting character n-grams and the other word n-grams. the third ranking approach also used ensemble learning, but instead opted for traditional machine learning classifiers (logistic regression and support vector machines) with word embeddings and manually extracted features @xcite . while research into english sarcasm and irony detection is thriving and receives a lot of attention, other languages are lagging behind (cakebread-andrews, 2021). semeval 2022 again includes a sub-task specifically for sarcasm detection in english and arabic (isarcasmeval), which aims to both improve the state-of-the-art methodology for english and expand the scope to less-researched languages. irony detection for dutch is still in its infancy. ever since @xcite and @xcite collected and analyzed dutch irony corpora, and gathered some initial insights into irony detection for dutch, no new research on the topic has been presented (to the best of our knowledge). one way to overcome the lack of languagespecific research is the use of multilingual language models, like multilingual bert @xcite , which makes sense as irony and sarcasm are assumed to be language-universal. multilingual approaches that utilize the exact same feature set and language models, have shown promising results, but generally do not aim to outperform language-specific models and rather attempt to catch up to their performance levels. an example is the language-universal approach presented by cignarella et al. (2020), who successfully created a syntactically informed bert model for english, french, italian and spanish social media data. dutch was not included in this research. ## experimental corpus and data description for this research, we made use of the dutch data set for irony detection collected by @xcite . the balanced corpus consists of 5,566 annotated tweets and was gathered in two ways. one part (3,179 tweets) contains irony-related hashtags (i.e. #sarcasme, #ironie, #not) and was annotated with a fine-grained annotation scheme. remaining tweets, which balance out the corpus, were posted by the same users as the ironic tweets and were confirmed to be non-ironic by annotators. out of the 3,179 tweets with irony-related hashtags, 6% were found to be non-ironic. this is notably lower than in the english data set, which was collected and labeled for semeval 2018 using the same annotation guidelines and methods @xcite , where 19% of the hashtag-containing tweets are non-ironic. the inclusion of "#not" as an irony-related hashtag was found to make the ironic data set less noisy for dutch compared to english, where 'not' had sometimes been used as a negating particle rather than an irony marker. there are a couple of reasons why we selected this data set for our research. the first and most important reason is that the tweets containing ironyrelated hashtags were manually checked to confirm if they are actually ironic, and to define the type of irony being used. as noted by @xcite and @xcite , tweets containing irony-related hashtags can still be non-ironic and introduce noise in the corpus. having them checked by annotators ensures a better corpus quality compared to hashtag-based approaches that are common in this research field. the fine-grained annotation guidelines are another useful aspect of the data set, because they allow for more insight and understandability in how irony is linguistically realized. each ironic tweet receives a label indicating the type of irony: ironic with sentiment clash, situational irony or other irony. a first traditional distinction is made between verbal and situational irony. situational irony happens when a situation fails to meet our expectations @xcite . a common example of this is when firefighters have a fire in their kitchen while they are out to answer a fire alarm @xcite . verbal irony, here represented by the labels ironic by clash and other irony, is defined as expressions that convey the opposite meaning of what is said @xcite and implies the expression of a feeling, attitude or evaluation @xcite . ironic by clash occurs when a text expresses an evaluation whose literal polarity is opposite to the intended polarity. any other forms of verbal irony are categorized as other irony. the distribution of the data is as follows: the english equivalent of this dutch data set is the foundation of the irony detection shared task of semeval2018 @xcite and often used as one of the go-to data sets for irony or sarcasm detection (included in cignarella et al. (2020); potamias et al. (2020); ahuja and sharma (2021); chowdhury and chaturvedi ( 2021 )). as the dutch counterpart is collected, annotated in the same manner by native speakers and shows an acceptable level of agreement between the annotators with scores ranging from moderate to substantial (see ## system setup and features the baseline support vector classifier (svc) system leverages a core set of lexical character and word n-gram features, which are augmented with more elaborate syntactic, semantic and sentiment lexicon features. syntactic features include partof-speech frequencies, temporal clash and named entity features. the temporal clash feature indicates whether two different verb tenses occur in the same tweet. named entity features include both a binary feature (whether or not there is a named entity), and a frequency feature (counting the number of named entities in the tweet). semantic features are binary features based on word2vec @xcite clusters of semantically related words, generated from a large twitter background corpus. such a feature could, for example, check whether the tweet contains a word in the semantic cluster [school, dissertation, presentation, degree, classes, papers, etc.] @xcite . lastly, the sentiment lexicon features count the number of positive and negative token occurrences in each lexicon and take the sum of the sentiment values. we used a variety of sentiment lexicons, including the nrc word-emotion lexicon (mohammad and turney, 2013), pattern @xcite , the duoman lexicon @xcite , the hogenboom emoticon lexicon @xcite and the emoji sentiment ranking (kralj @xcite . all svm models were trained using libsvm @xcite to stick as closely as possible to the methodology of @xcite . in the same way, we optimized the hyperparameters of the svm on the train set through grid search 3 . in addition to this baseline model, a dutch transformer language model (de vries et al., 2019), built from diverse corpora containing 2.4 billion tokens, was fine-tuned on the training data for irony detection. the methodology used for fine-tuning and the number of positive, neutral and negative sentiments for each target, the most common of the three is assumed as the target's implicit sentiment. for the target "drilled awake", for example, 44% of the tweets were classified as negative, 22% as neutral and 33% as positive. because the negative partition is the largest, we assign a negative polarity to that target. after determining the implicit sentiment of the target using the method described above, we looked for the presence of a sentiment clash by searching the remainder of each tweet (without the target string) for any positive or negative sentiment token in any of our sentiment lexicons (nrc, pattern @xcite , duoman @xcite , hogenboom @xcite and emoji @xcite ), considering it the tweet's explicit sentiment, and compared it to the implicit sentiment of the target. this method was able to cover 756 out of 939 (81%) of all annotated targets (in test and train set). in such manner, the correct implicit sentiment was predicted for 636 out of 756 targets (84%). if the explicit sentiment in the tweet contradicts the implicit sentiment of the irony target, we call this an implicit sentiment clash. this feature only occurs when we were able to determine an implicit sentiment for the annotated target, meaning there are no targets for non-ironic tweets. despite the high coverage and accurate analysis for implicit sentiment, only 16% of our ironic test tweets and 17% in the training set received the sentiment clash feature. this might seem surprisingly low considering 79% of all ironic tweets have been annotated with the label 'ironic by clash'. however, we should keep in mind that only about a third of the tweets with the label 'ironic by clash' received an annotated irony target. a closer look reveals that the use of lexicons as indicators for explicit sentiment works quite well foot_4 . we believe this could still be improved. in some cases, the explicit sentiment that causes the clash was annotated as part of the irony target foot_5 . besides the clash between implicit and explicit sentiment, we implemented another feature to indicate a contrast among explicitly mentioned sentiments. this time, the explicit sentiments were gathered across all lexicons collectively instead of per lexicon as was done for the baseline svm. an explicit clash occurs, for example, when a text contains a word like "lovely" and an angry emoji or other word like "disgusting". this explicit clash occurs in 22% of all test tweets and co-occurs with the irony label in 58% of the cases. although this feature did not show a high information gain in the data set (0.005), we still considered it worthwhile to combine it with the implicit clash feature for our experiments. for the evaluation of the features and svm models, we developed separate svm systems containing (1) the baseline feature set, (2) all mentioned features including the implicit and explicit sentiment clash and (3) the baseline feature set with the implicit sentiment clash, but without the explicit clash foot_6 . for each of the systems with implicit clash as a feature, we evaluated two versions of the feature: one with the automatically predicted implicit sentiment and one with the annotated implicit sentiment. ## experimental results and analysis first of all, we noticed that all models reached fscores above 70% (see our svm system containing the automatically generated clash feature successfully leverages the implicit sentiment of irony target strings and is able to improve the baseline f-score with another percentage. this might seem a modest improvement, but we should stress that this feature was only one out of the 15,845 features and could have been 'undersnowed' by the many lexical features. these results are further confirmed when comparing the performance of both implicit clash models. our automatic implicit clash model without the explicit feature even slightly outperformed the model with manually annotated implicit sentiment. we hypothesize this is because of the nature of some of the annotated strings. the annotation guidelines did not include any length or format restrictions for the irony targets, which causes them sometimes to be exceedingly long and therefore noisy. some of the targets already contain a sentiment clash. the most obvious explanation would be a mistake during annotation, which makes it impossible to detect a clash between the target and the remainder of the tweet. however, it could just as well be a nested clash. in that case there would be two clashes in the tweet, one inside the target and one between the target and the rest of the tweet. others contain common sense that is strongly connected to the physical world and require the understanding of price values for certain goods or the duration of some activities, etc. unreasonably large amounts should generally be considered negative, but there are no methods yet to explain a machine how many agents should man a station or what the appropriate price for a t-shirt is. many of these kind of appreciations and opinions even depend on personal, geographical or cultural preferences and characteristics. below we present some of the targets without an implicit sentiment prediction (original dutch tweet with english translation): evaluation on a subset containing only the tweets with an annotated target leads to some fascinating outcomes (see a cursory manual analysis of the wrong predictions of our best system reveals that many contain a the svm with automatic implicit sentiment still attains the best results when looking at the accuracy of each model per label, as shown in ## conclusion and future research this paper presents a set of experiments for irony detection on dutch tweets. the proposed svm models obtained good classification scores, considerably outperformed our baseline transformer model (bertje) and were able to exploit the sentiment clash feature to achieve more accurate results. for the task of irony detection, the results confirmed that feature-based approaches, although requiring a lot of effort, obtain good results and give more insight into feature relevance and possible future improvements. although the dutch data set has until now remained uncharted and has no comparable results yet, the applied methodology in this replication experiment has shown 7% to 9% higher f-scores compared to the english data set. implicit sentiment was successfully inferred for irony targets by running sentiment analysis on a large background corpus containing these targets. our approach using sentiment lexicons for the detection of explicit sentiment to clash with our detected implicit seems to be efficient. our feature indicating a clash between implicit and explicit sentiment has proven to be a valuable addition to the feature set, even when it can only be activated in a portion of the tweets identified as 'ironic by clash'. it is somewhat unusual that our automatic prediction for implicit sentiment achieved better results than the feature with manually annotated sentiment. further manual analysis of the results will be necessary to better understand this discrepancy. a brief inspection of the misclassifications has led to the presumption that our best models have recognized hyperbolic or 'exaggerated' positive sentiment as a feature. we believe this occasionally causes misclassification of very positive texts as ironic. this might be because many tweets used to have an irony-related hashtag, which was indicated as essential to detect irony by human annotators. nonetheless, confirming this would also require more thorough analysis of the data and predictions. we consider the results of these exploratory experiments to be insightful but we have only scratched the surface. testing has indicated both improvements (coverage and sentiment analysis of implicit sentiment of targets) and highlighted some weaknesses. the major challenge that remains is the automatic detection of 'irony targets', the topics or concepts people are ironic or sarcastic about. hence, we will investigate this as the subject of our future research. on top of that, our implicit clash feature was only one of the 15,000 features, which might cause it to be 'undersnowed' by the many lexical features. therefore, we will also experiment with ensemble learning to increase the weight of this feature. in the large scope of irony or sarcasm detection, there are still many paths to pursue. one would be the incorporation of implicit sentiment features into other systems that exploit word embeddings, in the same way as cignarella et al. (2020) used n-gram features. another direction is to further expand the coverage of implicit sentiment of irony targets. this can be achieved by connecting related phrases or words like surgeon -doctor -dentist when there are no exact matches. alternatively, graph knowledge bases, such as senticnet @xcite ) can be leveraged for more advanced connections between concepts and already include sentiment related to a concept. experiments with older versions of senticnet as a sentiment lexicon, however, did provide worse results for sentiment analysis than our data-intensive tweetbased approach @xcite .