“A good pun is its own reword”: Can Large Language Models Understand Puns? (2024)

Abstract: Puns play a vital role in academic research due to their distinct structure and clear definition, which aid in the comprehensive analysis of linguistic humor. However, the understanding of puns in large language models (LLMs) has not been thoroughly examined, limiting their use in creative writing and humor creation. In this paper, we leverage three popular tasks, i.e., pun recognition, explanation and generation to systematically evaluate the capabilities of LLMs in pun understanding. In addition to adopting the automated evaluation metrics from prior research, we introduce new evaluation methods and metrics that are better suited to the in-context learning paradigm of LLMs. These new metrics offer a more rigorous assessment of an LLM’s ability to understand puns and align more closely with human cognition than previous metrics. Our findings reveal the “lazy pun generation” pattern and identify the primary challenges LLMs encounter in understanding puns.

## introduction pun, as a form of wordplay, cleverly exploits double or multiple meanings of words @xcite . for example, for a pun sentence, "a good pun is its own reword", it plays on the similar sounds of "reword" and "reward", suggesting that the intrinsic value or reward of a good pun lies in its clever use of language or its inventive rephrasing. in most cases, the use of puns can produce humorous effects, as it creates a lexical-semantic ambiguity @xcite and a context-shift surprise @xcite . compared to other forms of humor, such as jokes @xcite and comedies @xcite , puns are appropriate for linguistic humor study as they have a more precise defini-figure 1 : toy examples of achieving three representative tasks related to pun understanding with llms, including pun recognition, explanation and generation. we explore the primary difficulties (e.g., paradoxical response, missing alternative word and lazy pattern) in these tasks. previous research on pun exploration primarily concentrated on developing specific language models or complex frameworks to recognize @xcite , explain @xcite , or generate @xcite puns. with the advancement of large language models (llms), recent studies have explored using llms for detecting jokes @xcite and identifying humor in images @xcite and videos @xcite . exploring llms' comprehension of puns could further enhance their values on creative text creation and humor generation. unfortunately, there are still no studies evaluating llms' capability of understanding puns systematically. there-fore, in this paper, we aim to systematically evaluate the capabilities of llms on pun understanding. as illustrated in figure 1 , to provide comprehensive assessments, we focus on three tasks from previous work, i.e., pun recognition, pun explanation, and pun generation. to adapt these tasks to the in-context learning (icl) paradigm of llms, we develop some new methods and metrics to ensure rigorous evaluation. for pun recognition, we create dual-biased prompts to gauge the confidence level of llms' responses. these prompts explicitly incorporate the terms "pun" or "non-pun" to interfere with the model's judgment. for pun explanation, we employ both a fine-grained punchline check and a coarse-grained pairwise comparison. these methods help identify llms' shortcomings and assess the overall quality of llms' explanations. for pun generation, we introduce two novel settings, i.e. free and constrained generation, which demonstrate the llms' ability to create puns under varying conditions. moreover, we introduce an overlap metric to measure the originality of the puns generated by llms. our research has demonstrated that most llms are easily influenced by prompt bias in recognizing puns. they also struggle to explain puns which are based on phonetic similarities. in addition, we observe that llms often resort to a low-quality and incorrect pattern in pun generation, separating the double meanings instead of combining them. we term this pattern as "lazy pun generation". despite all these issues, some powerful llms still exhibit impressive performance across the three tasks. specifically, llms are competitive with humans in pun explanation and surpass the state-of-the-art models in pun generation. the main contributions of this paper are summarized as follows: ## related work studies on puns puns, recognized as a significant linguistic art form, have garnered attention in ai research @xcite @xcite ,. previous work mainly collects various types of puns @xcite from literature and the internet and proposes diverse tasks to evaluate the pun understanding capabilities of lms. these tasks can be divided into three categories: 1) pun recognition @xcite @xcite , which involves the detection of puns and localization of pun words. 2) pun explanation @xcite , which clarifies why the puns are funny by natural language explanations. 3) pun generation, which mainly requests small lms to either rewrite retrieved sentences into puns @xcite or create puns more flexibly using acquired context words @xcite @xcite . for evaluation metrics, some work analyses pun from multiple quantifiable dimensions like ambiguity and distinctiveness @xcite , as well as surprise and unusualness @xcite . however, these studies mostly focus on training small models in pun tasks. our research is the first to systematically evaluate the capabilities of llms to recognize, explain, and generate puns. llms for humors with vastly improved understanding and creativity, llms not only excel in traditional humor tasks such as detection and rating @xcite @xcite but also demonstrate exciting potential in humor explanation and generation @xcite . some works aid llms in joke generation with humor algorithms @xcite or feedback-driven techniques @xcite , while others focus on comprehending and explaining punchlines in images @xcite or videos @xcite . our work is the first to focus on pun understanding, a vital part of the humor. ## preliminaries in this paper, we focus on two primary types of puns: homographic pun (hom-pun) and heterographic pun (het-pun) @xcite . in the above two examples, the underlined parts represent the pun-alternative word pair @xcite , with the alternative word in (parentheses). for hom-puns, the pun word w p and the alternative word w a are identical. for het-puns, these two words have a similar pronunciation, but only the former appears in the sentence. both w p and w a have their respective meanings: pun sense s p and alternative sense s a , which are supported by the clever use of contextual words c w . in the first instance, the c w are "friend" and "to piece", and in the second example, are "life" and "puzzle". following the notation of @xcite , we refer to w p , w a , s p , and s a together as the pun pair, denoted as p @xmath3, w a , s p , s a >. ## probing protocol in this section, we design an evaluation protocol consisting of three progressive tasks to assess whether llms can understand puns well. ## results and analysis we design two types of prompts for pun recognition: the first type is the basic prompt, which only includes test data and biased instructions. the second type is the enhanced prompt, which adds to the basic prompt with the definition of puns and some examples (3 puns and 3 non-puns). 5 as shown in this may reveal that llms capture the core feature (i.e., dual meanings) of puns and use it as the main criterion for judgment. 5) gpt-4-turbo and claude-3-opus demonstrate exceptional performance, exhibiting satisfactory pun recognition capabilities. cot prompting although we primarily use cot to obtain explanations of puns from llms, it also offers an opportunity to explore its impact on the pun recognition task. we differentiate between two response methods based on the enhanced prompt: answering directly and answering with cot, while keeping the prompt's bias towards pun. then, we select four models and chart their performance in figure 2 . it is observable that, except for llama2-7b-chat, the remaining three llms showed an overall improvement in accuracy after using cot. notably, gemini-pro and gpt-4-turbo's weak spots in recognizing non-pun text are compensated for through cot response, showcasing a stronger ability to distinguish between puns and non-puns. ## conclusion in this paper, we examine the ability of large language models (llms) to understand puns. we employ three tasks: pun recognition, pun explanation, and pun generation, and develop various metrics to systematically assess the capabilities of llms in these areas. experiments indicate that although llms perform satisfactorily in recognizing and explaining puns, there is still room for improvement in their ability to generate creative and humorous puns. we also suggest that het-pun explanation and generation are more difficult than those of hompun. we believe our evaluation methods and findings will contribute to advancing research on pun understanding.