Do Androids Laugh at Electric Sheep? Humor “Understanding” Benchmarks from The New Yorker Caption Contest (2023)

Abstract: Large neural networks can now generate jokes, but do they really “understand” humor? We challenge AI models with three tasks derived from the New Yorker Cartoon Caption Contest: matching a joke to a cartoon, identifying a winning caption, and explaining why a winning caption is funny. These tasks encapsulate progressively more sophisticated aspects of “understanding” a cartoon; key elements are the complex, often surprising relationships between images and captions and the frequent inclusion of indirect and playful allusions to human experience and culture. We investigate both multimodal and language-only models: the former are challenged with the cartoon images directly, while the latter are given multifaceted descriptions of the visual scene to simulate human-level visual understanding. We find that both types of models struggle at all three tasks. For example, our best multimodal models fall 30 accuracy points behind human performance on the matching task, and, even when provided ground-truth visual scene descriptors, human-authored explanations are preferred head-to-head over the best machine-authored ones (few-shot GPT-4) in more than 2/3 of cases. We release models, code, leaderboard, and corpus, which includes newly-gathered annotations describing the image’s locations/entities, what’s unusual in the scene, and an explanation of the joke.

## introduction humor can be dissected, as a frog can, but the thing dies in the process and the innards are discouraging to any but the pure scientific mind. explanation generation task 3 can you please pass the cow? ❌ i'd kill for some cream cheese. vs. ✅ can you please pass the cow? ❌ welcome to insomniacs anonymous vs. 🏆 can you please pass the cow? ## experiments we split the 704 cartoons into 5 cross-validation splits such that entire contests are held out at test time. task construction details are in appendix c; modeling details (e.g., hyperparameter sweeps, task formatting) are in appendix b. ## conclusion we demonstrate that today's vision and language models still cannot recognize caption relevance, evaluate (at least in the sense of reproducing crowdsourced rankings), or explain the new yorker caption contest as effectively as humans can. however, the partial capacity of today's ai is still substantial, and may be sufficient for models to serve as creative collaborators, e.g., as brainstorming assistants for humorists/cartoonists. specifically: 1) our matching/quality ranking models could help entrants receive quantitative feedback on the relevance/predicted quality of their submissions, and 2) the annotated corpus+explanations we introduce could be repurposed for generation (we explore generation of novel cartoons/captions in appendix g). finally, a promising avenue for future work focused on generating humorous captions (c.f. our focus of humor "understanding" benchmarks) would be to operationalize the feedback provided by our matching/ranking models in an reinforcement learning from human feedback (rlhf) loop. a last remark. we cannot claim to know whether the human-machine 'humor understanding gap' will be closed sooner or later. 16 but we encourage other researchers to have as much fun with the topic as we did!