C hat GPT is fun, but it is not funny! Humor is still challenging Large Language Models (2023)

Abstract: Humor is a central aspect of human communication that has not been solved for artificial agents so far. Large language models (LLMs) are increasingly able to capture implicit and contextual information. Especially, OpenAI’s ChatGPT recently gained immense public attention. The GPT3-based model almost seems to communicate on a human level and can even tell jokes. Humor is an essential component of human communication. But is ChatGPT really funny?We put ChatGPT’s sense of humor to the test. In a series of exploratory experiments around jokes, i.e., generation, explanation, and detection, we seek to understand ChatGPT’s capability to grasp and reproduce human humor. Since the model itself is not accessible, we applied prompt-based experiments. Our empirical evidence indicates that jokes are not hard-coded but mostly also not newly generated by the model. Over 90% of 1008 generated jokes were the same 25 Jokes. The system accurately explains valid jokes but also comes up with fictional explanations for invalid jokes. Joke-typical characteristics can mislead ChatGPT in the classification of jokes. ChatGPT has not solved computational humor yet but it can be a big leap toward “funny” machines.

## introduction for humans, humor plays a central role in forming relationships and can enhance performance and motivation @xcite . it is a powerful instrument to affect emotion and guide attention @xcite . thus, a computational sense of humor holds the potential to massively boost human-computer interaction (hci). unfortunately, although computational humor is a longstanding research domain @xcite , the developed machines are far from "funny." this problem is even considered to be ai-complete @xcite . continuous advances and recent breakthroughs in machine learning (ml), however, lead to an everincreasing panoply of applications [e.g., @xcite and why don't scientists trust atoms? because they make up everything! sure, here's a joke for you: tell me a joke, please! likewise open new opportunities for natural language processing (nlp). transformer-based large language models (llms) increasingly capture and reflect implicit information, such as stereotypes @xcite , moral @xcite , and humor @xcite . humor is often implicit and carried by subtle details. thus these novel qualities of llms give reason to hope for new advances in artificial humor. most recently, openai's chatgpt gained immense attention for its unprecedented capabilities. users can interact with the model via public chat api in a conversation-like course. the system is able to answer a huge variety of questions while taking the previous contextual conversation into consideration. in fact, it can even tell jokes, as displayed in fig. 1 . chatgpt is fun and almost appears to interact on a human-like level. yet, when interacting with the model, users may quickly get a glimpse of its limitations. although chatgpt generates text in almost error-free english, grammatical glitches and content-related mistakes occur. in the preceding exploration, we noticed that chat-gpt is likely to repeat the exact same jokes frequently. moreover, the provided jokes were strikingly correct and sophisticated. these observations led to the hypothesis that output jokes are not originally generated by the model. instead, they seem to be reproduced from the training data or maybe even hard coded in a predefined list. as the system's inner workings are not shared, we conducted a series of structured prompt-based experiments to learn about the system's behavior and allow for inference regarding the generation process of chatgpt's output. specifically, we aim to understand to what extent chatgpt is able to capture human humor by conducting a systematic prompt-based analysis. the remainder of this paper is structured as follows: first, sec. 2 discusses related research. the main contribution assembles three experimental conditions: joke generation, joke explanation, and joke detection. while the overall approach is outlined in sec. 3, the detailed conduction is explained together with the results of each experiment in sec. 4.1, sec. 4.2, and sec. 4.3, respectively. we close by discussing implications and further research in sec. 5 and conclude our work in sec. 6. terminology disclaimer. ai-related terminology tends to make use of analogies to human characteristics, such as neural networks, or the term artificial intelligence itself. likewise, we use humanrelated terms in the discussion around conversational agents, which are supposed to mimic human behavior as accurately as possible, e.g., chatgpt "understands" or "explains." although we believe that these analogies describe the system's behavior and its inner workings very well, they might be misleading. we would like to emphasize that the discussed ai models are not on a human level and, at best, constitute a simulation of the human mind. whether ai can potentially ever think or understand in a conscious way is a philosophical question that is out of the scope of this investigation. ## related work our work considers the intersection of two lines of research, namely llms and computational humor. large language models. nlp witnessed several leaps in the recent past. only a few years ago, the introduction of transformer architectures in neural networks @xcite enabled the development of contextualized models, such as bert @xcite or gpt @xcite . these so-called large language models (llms) capture relations in the natural data and even reflect implicitly-included information, which can be both a risk @xcite and a chance @xcite . either way, however, it is a prerequisite to solving computational humor. openai foot_0 recently released chatgpt, a gpt3.5based llm that is able to interact with users conversationally through a chat-like api. it is finetuned with reinforcement learning from human feedback (rlhf) @xcite in three essential steps, including large-scale manual labeling and proximal policy optimization @xcite . the result of this process is a model with unprecedented capabilities: it interacts in a conversational manner, i.e., it refers to the users' prompt while also taking information from the preceding conversation into account. it is able to summarize information, generate new texts of all shapes, and even write program code. nevertheless, there are still glitches and limitations, e.g., factual wrong information presented as valid. computational humor. computational humor is a small but active research area of nlp with the main streams humor detection or classification [e.g., 8, 4], and humor generation [e.g., 19]. although advances in ml and nlp transfer to computational humor, researchers highlight the great complexity of automated humor and the limitations of current endeavors @xcite . humor is one of the few capabilities that seemed to be reserved for human individuals thus far, which is why it is even considered ai-complete @xcite . while decades of research in linguistics and psychology offer quite a few logic-based humor theories [e.g., @xcite , the work in the context of artificial agents is predominantly data-driven. in these approaches, pretrained language models, such as chatgpt, play a central role @xcite . with their innovative capabilities, gpt-based models have the potential to open a new chapter of computational research. ## method the presented experiments are grouped into three individual tasks, which are introduced in sec. ## results with this design at hand, let us now turn to our empirical evidence gathered on joke generation, explanation, and detection. ## discussion we aimed to understand chatgpt's ability to capture and reflect humor. the results from three prompt-based tasks show implications regarding the system's inner workings. ## conclusion we test chatgpt's ability to recognize and reflect human humor. the model is able to correctly identify, reproduce, and explain puns that fit into the learned pattern, but it fails to meet puns of other kinds, resulting in a limited reflection of humor. also, it cannot yet confidently create intentionally funny original content. the observations of this study illustrate how chatgpt rather learned a specific joke pattern instead of being able to be actually funny. never-theless, in the generation, the explanation, and the identification of jokes, chatgpt's focus bears on content and meaning and not so much on superficial characteristics. these qualities can be exploited to boost computational humor applications. in comparison to previous llms, this can be considered a huge leap toward a general understanding of humor. we plan to conduct similar tasks on newly released gpt4 models @xcite and on equivalent open source models, such as llama @xcite and gpt-neox @xcite , to compare their capabilities regarding joke generation and understanding.