What do Humor Classifiers Learn? An Attempt to Explain Humor Recognition Models (2023)

Abstract: Towards computational systems capable of dealing with complex and general linguistic phenomena, it is essential to understand figurative language, which verbal humor is an instance of. This paper reports state-of-the-art results for Humor Recognition in Portuguese, specifically, an F1-score of 99.64% with a BERT-based classifier. However, following the surprising high performance in such a challenging task, we further analyzed what was actually learned by the classifiers. Our main conclusions were that classifiers based on content-features achieve the best performance, but rely mostly on stylistic aspects of the text, not necessarily related to humor, such as punctuation and question words. On the other hand, for humor-related features, we identified some important aspects, such as the presence of named entities, ambiguity and incongruity.

## introduction as part of usual human language, dealing with complex deep linguistic knowledge, such as figurative language, is an important element of research on natural language processing (nlp). verbal humor is a large instance of figurative language, whose understanding and generation are crucial for language fluency and the comprehension of deeper nuances of language @xcite . additionally, computational systems capable of processing humor might give other fields of research (e.g. linguistics, psychology, philosophy, to name a few) insights about how this phenomenon works and how it is conceived through language. regarding such computational models, we must always call in question their trustworthiness before drawing any conclusion that they are suitable to solve any specific problem, especially in tasks deemed as extremely complex, such as humor recognition, fake news detection, irony recognition, and the like @xcite . thus, it is essential to question if it is possible to really understand what the machine is learning and if it is actually capturing information relevant to the phenomenon being studied. in this way, we can find flaws with the methods or resources used, which drives to further research on the subject to develop better models. within this context, we present a study on humor recognition with a special focus on identifying which features and pieces of information are mostly used by supervised machine learning (ml) models for this task, including classical ml classification algorithms and deep learning large language models (llms). we further highlight that the entirety of this work was made for the portuguese language, much more underdeveloped on this task when compared to languages like english. towards our goal, we first replicated the current state-of-the-art methods for humor recognition in portuguese @xcite . in addition, we fine-tuned a bert model pretrained for portuguese @xcite for the same task. results were further analyzed with shap @xcite , a tool for machine learning explainability. shap provided scores for each feature, word, or sub-word used by the respective models, which, together with careful manual analysis, helped in understanding what exactly the models had learned from the provided data. all experiments were carried out on the corpus created by gonçalo @xcite , which is, to the best of our knowledge, the only corpus in portuguese created for the task of humor recognition. our results show that the bert model outperformed all other ml methods in terms of f1-score, achieving a score of 99.6%. however, through careful analysis, we discovered that this model, alongside other methods based on content-features, based their decisions primarily on stylistic aspects of the texts, such as punctuation, and other phenomena not necessarily related to humor, for instance the presence of questions. we also noted aspects of the set of humor-related features proposed by gonçalo oliveira et al. (2020) that might not have been expected by their original authors, such as the relation between concreteness and humor, and the association of people named entities with humorous texts. however, some of their interpretations were reinforced by the ml models, for example, the connection of ambiguity and incongruity to humorousness. the remainder of this paper is organized as follows: some relevant related work about humor recognition and ml explainability is presented in section 2, followed by an overall description of our methodology, in section 3. later, the results are presented and discussed in section 4, with the final remarks and future work mentioned in section 5. in the end of the paper, we note some limitations of the current work, as well as ethical aspects that should be considered in the future. ## related work this paper has relations with two main areas of research: general humor recognition, usually interpreted as a ml classification task, and ml explainability, which aims at creating explanations for computational models, in order to inspect what information the model actually uses for inference. ## methodology our work has two main fronts of research: first, the implementation of humor recognition systems for the portuguese language; then, a deeper analysis of their performance, to assess their weaknesses and stimulate further research. this includes a discussion on how to overcome some challenges, followed by what could be developed towards improved systems. ## results this work has two main results. first, a new finetuned bert model for humor recognition in portuguese, which outperforms the current state-of-theart for this task in terms of automatic evaluation metrics. secondly, a deeper analysis of such models, identifying how well they are suited for the task in general. ## conclusion in this paper, we presented a re-implementation of the previous state-of-the-art method for humor recognition in the portuguese language, alongside a novel fine-tuned bert model for the same task, reaching a nearly-perfect f1 score of 99.64%. 9 however, a deeper analysis of the models using a machine learning explainability method, shap, enabled us to understand which pieces of information the models were relying on to do such classification. we came into the conclusion that bert and models based on tf-idf counts did not learn specific mechanisms of humor, but were instead leveraging mainly stylistic characteristics of the texts, such as punctuation and the presence of whquestions. furthermore, the analysis of how humor-related features were interpreted by the ml model led to interesting observations not considered by their proposers @xcite . for example, the association of humor with person named entities or higher levels of concreteness; however, some of their reasoning can also be reinforced by how the model used some of the knowledge provided, e.g. humor was considered related to higher levels of ambiguity and incongruity within the text, which is up to par with linguistic descriptions of verbal humor @xcite @xcite . as a final conclusion, we emphasize how challenging it is to create a text classification corpus for supervised ml in such a way that the model actually learns about the linguistic phenomenon in question, rather than resorting to specific attributes and shortcuts not directly related to the problem being studied. we find that humor is a specially difficult task to create such a corpus, as it is a largely di-verse phenomenon (verbal humor can be conveyed in many different ways), with an equally large universe of negative examples (non-humorous texts also present themselves in various formats). from these conclusions, we can draw some fruitful paths for future research. first, we mention the creation of a new corpus for humor recognition in portuguese, taking into account some of the flaws found in the corpus by gonçalo oliveira et al. (2020). however, as it is -to the best of our knowledge -the only available corpus for this task, it can still be evaluated if it is fit after some process of normalization, starting with punctuation, e.g., by adding full stops to the humor examples, which would be a less expensive process; some early experiments in this sense show a decrease of 4 percentage points in f-score obtained by the bert model when discarding or normalizing punctuation. another point to be considered for the creation of a new corpus is the responsibility of which texts to include; as we mention later in our ethics statement, the corpus used in this work contains texts annotated as jokes that contain rather problematic content, e.g. riddles that are openly racist. another possibility for future work is to change the models and how they work. one could use methods, such as the one proposed by @xcite , that are not based on ml, but rather on formalizing linguistic theories of humor to a computational environment. we also find it appealing to explicitly include linguistic knowledge into the ml models, so that they are powered with some information beyond the textual surface, argued by other researchers as vital to deal with complex phenomena such as humor @xcite . this goal could also be achieved by exploring further the humor-related features, which were proposed originally from a linguistic point-of-view; other extra-linguistic aspects of humor could also be studied, for instance how different cultural backgrounds affect the perception and definition of humorousness.