Getting Serious about Humor: Crafting Humor Datasets with Unfunny Large Language Models (2024)

Abstract: Humor is a fundamental facet of human cognition and interaction. Yet, despite recent advances in natural language processing, humor detection remains a challenging task that is complicated by the scarcity of datasets that pair humorous texts with similar non-humorous counterparts. We investigate whether large language models (LLMs) can generate synthetic data for humor detection via editing texts. We benchmark LLMs on an existing human dataset and show that current LLMs display an impressive ability to “unfun” jokes, as judged by humans and as measured on the downstream task of humor detection. We extend our approach to a code-mixed English-Hindi humor dataset where we find that GPT-4’s synthetic data is highly rated by bilingual annotators and provides challenging adversarial examples for humor classifiers.

## introduction despite their success on natural language tasks, large language models (llms) struggle to reliably detect and explain humor @xcite @xcite , and generate novel jokes @xcite . notably, humans also struggle to write jokes; even at satirical newspapers like the onion, less than 3% of proposed headlines are printed @xcite . in contrast, humans are able to consistently edit jokes to unfun them, an insight which motivated @xcite to host a game where internet users competed to edit satirical headlines to make them serious. the resulting dataset, the unfun corpus @xcite , has been a valuable tool for advancing computational humor research. the dataset has been used to study properties of both humor and transformer architectures @xcite and even to generate * equal contribution. figure 1 : outputs from gpt-4. we leverage language models to edit away (or "unfun") humor in existing human-written jokes, resulting in aligned datasets that pair humorous texts with non-humorous counterparts. while useful contributions, unfun and other aligned humor datasets @xcite are limited in both size and scope, due to their reliance on human annotation. we investigate the alternative of using llms to create datasets of aligned humorous and non-humorous texts. 1 previous work @xcite @xcite has found that llms are limited in their ability to create synthetic humor. we take a new approach, exploiting the asymmetrical difficulty @xcite of synthetic humor generation. rather than only testing whether llms can generate humor, we explore their ability to edit away humor in existing jokes. validating and harnessing this capability could provide large paired datasets and support future work on improving humor detection and even generation. our contributions include benchmarking against human-curated data in the unfun corpus, where we find that llms like gpt-4 and gpt-3.5 @xcite can (1) outperform humans at removing humor from texts and that (2) this ability can be harnessed to generate high quality synthetic data for training humor classifiers. while these models can also be prompted to modify unfunny headlines to craft satire, we find that this ability is more inconsistent and lags behind satirical writers. finally, we consider a code-mixed english-hindi humor dataset to evaluate whether gpt-4's "unfunning" ability generalizes to other domains and languages. we find that the resulting synthetic unfunny dataset is rated highly by bilingual annotators and poses challenging adversarial data for models trained on the original corpus. ## getting serious with language models we first revisit the unfun task and resulting dataset, but with language models as players. ## extending unfun to other languages recent work has found that gpt-4 exhibits strong multilingual capabilities @xcite @xcite . motivated by these findings, we investigate whether its ability to edit away humor generalizes to other languages and forms of joke. ## 858 original dataset source unfuns balanced accuracy humor non-humor original 22.6 (3.7) 67.9 (0.9) 80.3 (3.5) 56.9 (5.1) (25%) synth unfuns 34.0 (8.4) 67.7 (1.7) 78.4 (3.3) 55.4 (5.9) (50%) synth unfuns 57.7 (6.0) 62.1 (0.6) 68.4 (5.7) 55.9 (4.7) ## discussion our results indicate that current llms struggle to generate humor, but can outperform crowd-workers at editing away (or unfunning) humor. we hypothesize that maximum likelihood training, combined with autoregressive sampling techniques, does not endow models with the creative spark required for joke writing, and instead lends itself to making high probability, reasonable substitutions to replace incongruous twists. our evaluations on code-mixed english hindi twitter data indicate that, for gpt-4, this ability can impressively generalize to other languages and settings to create novel unfun-like datasets. we are excited for future work that harnesses this capability and resulting data to improve humor detection and generation systems, and also to demystify fundamental properties of humor. ## limitations we consider two settings, english satirical headlines and code-mixed english-hindi tweets. humor practices and references vary by culture @xcite , and we leave investigating cultural impacts on llms and humor to future work. in both of our evaluations, the subjectivity of humor presents a challenge for our evaluations @xcite . we see evidence of this in another concern is data contamination @xcite , and that a portion of the text from the unfun corpus could have been trained on and memorized by the llms we evaluated. we investigate this concern in appendix a.6. we note that our results on english-hindi data show that gpt-4's abilities generalize to a dataset where these pairs do not already exist on the internet. ## ethical statement humor brings joy to people and plays a critical role in building and maintaining social relationships @xcite . however, its importance presents a double-edged sword; offensive and hurtful humor can cause real harms, and reinforce prejudice @xcite . as a result, with their widespread adoption, it will be paramount for ai systems to be more capable of identifying and appropriately navigating jokes. we believe that our work on benchmarking llm humor abilities and building challenging detection datasets is an important step in this direction. however, one possible concern is that malicious actors could leverage our unfunning approach to circumvent existing safeguards. in our experimentation, we found numerous settings where gpt-4 refused to generate jokes for offensive topics, but had no trouble editing texts to remove humor and offensiveness. this could enable building large parallel datasets of (offensivetext, non-offensive counterparts) that could then be used to train models for offensive joke generation.