Chumor 2.0: Towards Better Benchmarking C hinese Humor Understanding from (Ruo Zhi Ba) (2025)

Abstract: Existing humor datasets and evaluations predominantly focus on English, leaving limited resources for culturally nuanced humor in non-English languages like Chinese. To address this gap, we construct **Chumor**, the first and the largest Chinese humor explanation dataset. **Chumor** is sourced from Ruo Zhi Ba (RZB, 弱智吧), a Chinese Reddit-like platform known for sharing intellectually challenging and culturally specific jokes. We test ten LLMs through direct and chain-of-thought prompting, revealing that **Chumor** poses significant challenges to existing LLMs, with their accuracy slightly above random and far below human. In addition, our analysis highlights that human-annotated humor explanations are significantly better than those generated by GPT-4o and ERNIE4-turbo. We release **Chumor** at https://huggingface.co/datasets/MichiganNLP/Chumor , our project page is at https://github.com/MichiganNLP/Chumor-2.0 , our leaderboard is at https://huggingface.co/spaces/MichiganNLP/Chumor-leaderboard , and our codebase is at https://github.com/MichiganNLP/Chumor-2.0 .

## introduction humor is an intrinsic human trait that touches the core of our social and emotional lives, making it a rich field of study across various disciplines @xcite @xcite . with the advent of large language models (llms), researchers have evaluated llms' performance on diverse tasks @xcite @xcite and observed llms' extraordinary performance on many @xcite . in contrast, researchers have observed that llms still fail to understand humor @xcite . however, with all these studies on humor, † corresponding author of this work. most evaluations remain in english @xcite . this presents a significant gap, particularly for non-english languages like chinese, where culturally nuanced humor understanding is unexamined. in this paper, we try to bridge this gap by constructing chumor, a funny and challenging chinese humor understanding dataset sourced from ruo zhi ba (rzb, "弱智吧" in chinese), a chinese version of reddit platform known for sharing intellectually challenging and culturally specific jokes. this platform provides a set of unique chinese jokes that incorporate the subtleties and intricacies of chinese humor. unlike existing datasets that focus on tasks such as humor detection, punchline identification, or humor generation, chumor addresses the challenge of humor explanation. this involves not just identifying humor but understanding the reasoning behind it, a task that requires both linguistic and cultural knowledge. specifically, chumor tasks the llms with determining whether an explanation fully explains the joke. we source the explanations from gpt-4o and ernie 4-turbo , and have the entire dataset manually annotated by five native chinese speakers. we evaluate ten llms from various model families, and reveal that all models perform poorly, lagging significantly behind humans on chumor. we observe that chain-ofthought prompting does not necessarily improve models performance and can sometimes confuse their reasoning process. in addition, we conduct a case study in which one of the authors annotates the entire dataset, followed by a/b testing conducted by six native chinese speakers to compare explanations from gpt-4o versus human, and cultural desc. require knowledge of specific historical, social, or linguistic contexts. ## chumor dataset data collection. we construct our dataset by including rzb jokes from "best annual threads" between 2018 and 2021 that have been previously crawled † . in addition, we directly collect all threads in the "moderator's recommendation" section from rzb. each thread in rzb consists of "标题" (title), "一楼" (content), and several "跟帖" (follow-up posts). for threads from best annual threads, the jokes are listed in the follow-up posts, which are selected by the forum moderator. for threads from moderator's recommendation, the jokes consist of the title and the content of each thread. we remove the content if it repeats the title. humor explanation classification. we design a humor explanation classification task that can be easily used to test llms' capabilities in humor understanding. specifically, we use two llms, gpt-4o and ernie 4-turbo to generae explanations for our collected jokes. we manually annotate the generated explanations as either "fully explain the joke" (good) or "partially explain or not explain the joke" (bad) based on a majority vote among five of the authors who are native chinese speakers. each joke, along with its explanation, forms an individual instance in chumor, leading to a total of 3,339 instances. among these, 1,454 items are labeled as good and 1,887 as bad explanations. data examples from chumor. we present examples from chumor in humor categorization in chumor. we categorize the jokes in rzb into six types in label source (zh) "你 自 己 掰 了 一 千 吨 玉 米?!" "哈哈我瞎掰的" (zh) 这个笑话的幽默之处在于对话中的双关语和出人意料的回答。"掰了一千吨 玉米"看似是一个惊人的壮举，而"哈哈我瞎掰的"则瞬间揭露了这其实是一个夸 张的玩笑，这种巧妙的反转和语言的戏谑带来了幽默感。 good ernie4-turbo (en) "you shucked a thousand tons of corn all by yourself?!" "haha, let's just say there's been plenty of shuckin' going on." gemini 1.5-pro @xcite ) from google, glm-4 plus (bigmodel, 2024) from tsinghua university, gpt-4 turbo , gpt-4o (openai, 2023, 2024) from openai, ernie 4-turbo @xcite from baidu. for all the open-source llms, we use the instruction-tuned version in our evaluation. ## results and discussions overall model performance. figure 1 presents the accuracy of different llms on chumor in dp and cot settings. appendix h presents additional results and analysis. overall, we observe that all models perform poorly on chinese humor comprehension, with accuracy scores ranging between 44.6% and 60.3%. ernie 4-turbo and gemini 1.5-pro achieve the highest accuracy of 60.3%, and are just 10 points above the random baseline and far below human performance of 78.3%, highlighting the difficulty of chumor and the limitations of these llms in understanding chinese humor. ## conclusion we introduce chumor, a chinese humor understanding dataset that captures intellectually challenging and culturally specific humor in chinese. our analysis reveals that chumor remains difficult even for advanced llms, with a significant performance gap between llms and humans. furthermore, we find that chain-of-thought reasoning does not improve llms' humor comprehension and, in some cases, leads to over-analysis and incorrect interpretations. additionally, models such as gpt-4o and ernie 4-turbo struggle to explain jokes as effectively as humans, highlighting fundamental challenges in humor reasoning. these findings un-derscore the unique difficulties that chinese humor presents to llms. we hope that chumor can advance non-english humor research and contribute to evaluating llms' reasoning abilities across diverse cultural backgrounds.