Humor in Pixels: Benchmarking Large Multimodal Models Understanding of Online Comics (2025)

Abstract: Understanding humor is a core aspect of social intelligence, yet it remains a significant challenge for Large Multimodal Models (LMMs). We introduce PixelHumor, a benchmark dataset of 2,800 annotated multi-panel comics designed to evaluate LMMs’ ability to interpret multimodal humor and recognize narrative sequences. Experiments with state-of-the-art LMMs reveal substantial gaps: for instance, top models achieve only 61% accuracy in panel sequencing, far below human performance. This underscores critical limitations in current models’ integration of visual and textual cues for coherent narrative and humor understanding. By providing a rigorous framework for evaluating multimodal contextual and narrative reasoning, PixelHumor aims to drive the development of LMMs that better engage in natural, socially aware interactions.

## introduction motivation. humor is a quintessential element of human communication and intelligence, reflecting our ability to perceive, interpret, and appreciate complex social and cultural nuances @xcite . it plays a pivotal role in social interaction, creativity, and even cognitive development. the capacity to understand and generate humor is often considered a hallmark of advanced intelligence, involving sophisticated processes such as abstract thinking, contextual reasoning, and emotional perception. in the realm of artificial intelligence, large language models (llms) like gpt-4o @xcite have demonstrated remarkable proficiency in various natural language processing tasks, including text generation, translation, and questionanswering @xcite . however, their ability to understand or generate humor remains limited @xcite . humor comprehension involves not only linguistic understanding @xcite but also the interpretation of subtle cues, double meanings, and cultural references, which are challenging for ai systems primarily trained on textual data. ## related works humor has been explored through multiple disciplinary lenses, including linguistic @xcite , psychological @xcite , and philosophical perspective @xcite , as well as its physiological benefits @xcite . in this work, we focus on insights from the social science perspective to inform the design of benchmarks for lmms. humor, as a form of communication @xcite and social intelligence @xcite , plays a pivotal role in various social and cultural contexts @xcite @xcite , making its comprehension an essential aspect of human-centric ai development. despite advances in ai, lmms struggle to replicate humor's intricate interplay of linguistic, cultural, and contextual elements @xcite . this challenge is compounded by the overlap of online humor with potentially harmful con-tent, prompting research on detecting @xcite , clustering @xcite and explaining @xcite @xcite toxic memes @xcite , alongside efforts in meme generation @xcite @xcite and humorous image captioning @xcite @xcite . key humor understanding tasks-detection, classification, interpretation, and generation-have been identified, yet existing datasets address these in isolation. textbased datasets like talkfunny @xcite , one-liners @xcite , pun of the day @xcite , and ted-laughter @xcite focus on written jokes, while humordb @xcite targets static visual humor classification and explanation. multimodal datasets to address satire include yesbut @xcite and memes such as memotion 3.0 @xcite , mermaid @xcite , and totaldefmeme @xcite . audiotext datasets like big bang theory @xcite detect punchlines, whereas audiovisual datasets (mustard @xcite , ur-funny @xcite , mumor @xcite ) analyze dynamic humor. however, these datasets lack integrated evaluation of narrative sequencing and multi-panel humor, limiting their ability to assess lmms' temporal reasoning (see appendix a for details). to address these gaps, we introduce pixelhumor, a benchmark comprising 2,800 annotated multi-panel online comics. unlike previous data sets, which rely on single panel images or static contexts, pixelhumor uniquely integrates humor detection, classification in eight styles, open-ended interpretation, and sequence recognition within a unified framework. this multi-panel focus enables evaluation of narrative and temporal reasoning, critical for humor delivery in visual storytelling, revealing lmm limitations in contextual integration. by holistically assessing these dimensions, pixel-humor provides a comprehensive tool to advance the research on lmm humor comprehension. ## experiment settings in this section, we outline the experiment settings for evaluating lmms' ability to comprehend humor in comics using the pixelhumor dataset. ## experiment results in this section, we present the experimental results across the four humor evaluation tasks: humor identification, humor classification, humor interpretation, and sequence recognition. overall, closedsource models such as gpt-4o and gemini-1.5-pro consistently outperform open-source counterparts, particularly on tasks requiring fine-grained narrative understanding and multimodal reasoning. while all models excel at detecting the presence of humor, they struggle significantly with identifying key narrative moments, explaining humor coherently, and reconstructing comic sequences. these findings highlight fundamental challenges in current lmms' abilities to integrate visual and textual information over extended contexts. we report both quantitative metrics and qualitative analyses to provide a comprehensive assessment of model performance and limitations. ## discussion this work introduced pixelhumor, a novel benchmark designed to evaluate lmms ability to understand humor in online comics. through comprehensive experiments across four core tasks, humor identification, humor classification, humor interpretation, and sequence recognition, we provided the first in-depth assessment of lmms' multimodal humor comprehension. our results reveal that while lmms achieve nearperfect accuracy in detecting the presence of humor, they struggle with deeper aspects such as narrative sequencing, subtle humor style classification, and modality attribution. sequence recognition results show that even top models often rely on conventional reading heuristics rather than accurately re-constructing narrative flow. in humor classification, models consistently excel at explicit styles like personification but perform poorly on nuanced categories such as sarcasm and dark humor. additionally, humor interpretation tasks highlight significant degradation in reasoning quality as narrative complexity increases. these findings have important implications for the development of future multimodal ai systems. our results suggest that current lmms rely heavily on surface-level heuristics-such as favoring the textual modality-rather than engaging in true multimodal contextual and sequential reasoning. to genuinely comprehend humor, models must move beyond pattern recognition to understand causal, temporal, and multimodal relationships that underpin comic narratives. this will require advances such as hierarchical modeling of narrative structures, enabling models to track setups, twists, and punchlines across multiple panels and modalities. moreover, improved cross-modal fusion mechanisms are needed to better integrate visual and textual cues, particularly for humor that emerges from their interaction rather than from either modality alone. future work could focus on developing models that better track long-range multimodal dependencies, distinguish finer-grained humor mechanisms, and adapt to diverse cultural humor norms. extending pixelhumor to cover broader linguistic and stylistic variations could further stress-test lmms' reasoning limits. future work could also cover web comics in other languages to assess an lmm's ability to comprehend humor in multilingual settings. ## conclusion pixelhumor is a challenging benchmark, comprised of web comics, to assess an lmm's ability to comprehend and interpret humor. our experiments highlight fundamental limitations in lmms, and show that they are still far behind humans in complex humor-related tasks: in particular, humor interpretation where human written explanations are still largely preferred; humor style classification where models struggle to identify darker and sarcastic styles; and sequence recognition which is required in the build-up to a humorous punchline. this work lays the foundation for building ai systems that can engage more socio-intelligently with one of the most complex forms of human communication: humor.