You Told Me That Joke Twice: A Systematic Investigation of Transferability and Robustness of Humor Detection Models (2023)

Abstract: In this study, we focus on automatic humor detection, a highly relevant task for conversational AI. To date, there are several English datasets for this task, but little research on how models trained on them generalize and behave in the wild. To fill this gap, we carefully analyze existing datasets, train RoBERTa-based and Naïve Bayes classifiers on each of them, and test on the rest. Training and testing on the same dataset yields good results, but the transferability of the models varies widely. Models trained on datasets with jokes from different sources show better transferability, while the amount of training data has a smaller impact. The behavior of the models on out-of-domain data is unstable, suggesting that some of the models overfit, while others learn non-specific humor characteristics. An adversarial attack shows that models trained on pun datasets are less robust. We also evaluate the sense of humor of the chatGPT and Flan-UL2 models in a zero-shot scenario. The LLMs demonstrate competitive results on humor datasets and a more stable behavior on out-of-domain data. We believe that the obtained results will facilitate the development of new datasets and evaluation methodologies in the field of computational humor. We’ve made all the data from the study and the trained models publicly available at https://github.com/Humor-Research/Humor-detection.

## introduction in robert heinlein's science fiction novel, the main character, mannie, uses a collection of jokes to track the development of self-awareness in a supercomputer nicknamed mike. this reflects a common belief that a sense of humor is an innate human trait. since alan turing proposed his imitation game @xcite , a computer's ability to carry on a conversation has been one of the criteria of its intelligence. @xcite conducted a series of interviews to understand what people expect from conversations with artificial agents, and what characteristics they consider important in conversations in general. participants mentioned humor as an important attribute that adds substance to discussions and can be a key driver of conversations. they also described humor as a desirable feature that can make conversations with agents more engaging and entertaining. humor theories can be traced back to aristotle and have been elaborated by various disciplines, including semantics, psychology, and linguistics @xcite @xcite . the development of conversational ai makes computational humor methods highly relevant and in demand for practical applications. however, the difficulty of developing automatic methods in this area is determined by humor properties such as diversity and unpredictability. humor is an umbrella term for many related yet distinct phenomena. in this study, we focus on verbal humor, but it can be still quite diverse. for example, puns are based on wordplay, while satire refers to real-life contexts, irony may appear serious at first glance, canned jokes have a relatively stable form and circulate among a wide audience, unlike spontaneous jokes, which are improvised and context-dependent. to date, there are several english datasets for binary humor detection task that vary widely in size and sources of humorous content. however, there is little research on how models trained on them generalize and behave in the wild. to fill this gap, we carefully collected nine datasets suitable for humor detection, analyzed and cleaned them, and formatted uniformly. we fine-tuned the roberta models on each of the datasets and tested them on the rest. we also compared the results with those of naïve bayes (nb) classifiers. in addition, we evaluated the humor capabilities of two llms: chatgpt and flan-ul2. we also conducted an adversarial attack and applied the models to supplementary collections of figurative language, ironic tweets, and utterances from fiction and tv series. our analysis confirms that standard robertabased models overfit on particular training sets. on several datasets, the transformer-based models don't improve over the unigram-based nb classifier. based on the obtained results, we can conclude that the crucial factor for a better generalization is the diversity of humor types in the training, not the size of the dataset. this is confirmed by the experiments on a medium-sized collection compiled from different sources. all but three models are quite resistant to an adversarial attack based on word substitution. the humor detection capabilities of the llms are quite competitive across different datasets, although don't show superior results on individual collections; these results are not based on memorization of jokes. the behavior of the models on supplementary collections is mixed. in general, the models assign texts to the humorous class more often than we expected, which may indicate overfitting. some models violate our assumed behavior based on the genres of the novels and tv series. the llms show a more stable behavior on supplementary collections, consistent with our expected behavior of a 'good humor detector'. based on the results obtained, we can conclude that humor is a complex and multifaceted phenomenon, and the task of humor detection must try to reflect this complexity. we have shown that one way to improve the generalizability of humor detection models is to train them on diverse collections. conversational systems are probably the most promising application for humor detection. accordingly, it is desirable to build new humorrelated datasets that reflect the peculiarities of this application domain, taking into account different dialog scenarios and user characteristics. we believe that the results obtained will facilitate the development of new datasets and evaluation methods in the field of computational humor. we have made the datasets in a unified format and the trained models freely available. ## related work an early work on automatic humor detection was specifically targeted at wordplay in 'knock-knock' jokes @xcite . @xcite addressed the humor detection task in a broader context and automatically compiled a large and diverse dataset of humorous and non-humorous short texts for training and testing humor classifiers. they conducted extensive experiments with different humor-specific features such as alliteration, rhyme, antonymy and adult slang, as well as lexical features using naïve bayes and svm classifiers. @xcite introduced the notion of 'humor anchors' -lexical units that enable humorous effect. they also addressed humor detection employing gradient boosting regression trees with a wide variety of features, including humor-specific features and word2vec embeddings. @xcite demonstrated utility of syntax features for humor detection. @xcite survey and summarize different features used in humor classification studies. there are studies concerned with humor detection in specific domains, such as twitter @xcite or product question answering systems @xcite . with the advent of neural networks, computational humor research has shifted from feature engineering to experiments with different neural architectures on larger collections. @xcite applied convolutional neural network (cnn) to humor detection. @xcite demonstrated that the bert-based classifier achieves superior results in humor recognition task on several existing datasets, and also proposed a new dataset composed of humorous reddit posts. there are studies leveraging the setup-punchline structure of jokes for humor detection task using bert senetence embeddings (annamoradnejad and zoghi, 2020) or gpt-2 @xcite . @xcite applied various models -three transformer models (bert, distilbert, and roberta), fasttext-based representation, gpt-2, and lstm, -to a collection of aligned humorous/non-humorous sentence pairs. the authors discovered that humor classification occurs in the last transformer layers meaning that the model relies on semantic, rather than lexical features, and one of the transformer heads specialized in attending to the humorous part of the input text. this study is complement to ours: we don't perform probing and extensive comparison over different models, but pay the main attention to one transformer-based model and various datasets. in a concurrent paper, @xcite improved humor detection by using several datasets containing different humor species at once, although didn't investigate generalization of models trained on individual datasets. several humor-related shared tasks organized in recent years have contributed to the nlp community's interest in computational humor and facilitated progress in the field @xcite @xcite @xcite . in this paper, we focus on verbal humor, but there are multimodal humor detection studies leveraging images, video, or audio along with texts @xcite @xcite . besides english data, there are humor-related datasets for italian @xcite , spanish @xcite , dutch (winters and delobelle, 2020), russian @xcite , and chinese @xcite . modern large language models (llms) are impressive in their ability to engage in meaningful, context-aware conversation, answer questions, generate text, and program code. however, as a systematic evaluation showed, llms tend to perform worse than specialized models at certain tasks @xcite . @xcite reports that chatgpt 'has some understanding of humor', but provides examples of failure and points out the need for a comprehensive examination of the humor capabilities of llms. @xcite elaborated a set of prompts to evaluate the gpt-4's ability to judge the funniness of jokes; one of the variants showed 'a weak but encouraging positive correlation with human judges.' a recent study investigated chatgpt's ability to generate and interpret jokes @xcite . the authors found out that chatgpt's humorous repertoire was quite limited and identified 25 top jokes. ## conclusions based on the results obtained, we can conclude that fine-tuned humor detection models perform well in laboratory settings. their transferability depends more on the diversity of the training data than on its volume. the results of the models on out-ofdomain data are unstable, suggesting that they are not yet mature enough to be used in practical applications. an adversarial attack showed that models trained on pun datasets are less robust. the results of llms on humorous datasets are quite competitive and don't rely on memorizing jokes, although they lag behind the results of the models trained specifically for humor detection. llms exhibit a more stable behavior on out-of-domain data, consistent with what is expected from a good humor detector. the results obtained on complementary datasets provide interesting insights and warrant further investigation. in order to obtain a more realistic evaluation of humor detection models, we propose to include conversational data and to complement the evaluation with user studies. we believe that the obtained results will facilitate the development of new datasets and improved evaluation methodology in the field of computational humor.