Can Language Models Laugh at Y ou T ube Short-form Videos? (2023)

Abstract: As short-form funny videos on social networks are gaining popularity, it becomes demanding for AI models to understand them for better communication with humans. Unfortunately, previous video humor datasets target specific domains such as speeches or sitcoms, and mostly focus on verbal cues. We curate a user-generated dataset of 10K multimodal funny videos from YouTube, called ExFunTube. Using a video filtering pipeline with GPT-3.5, we verify both verbal and visual elements contributing to humor. After filtering, we annotate each video with timestamps and text explanations for funny moments. Our ExFunTube is unique over existing datasets in that our videos cover a wide range of domains with various types of humor that necessitate a multimodal understanding of the content. Also, we develop a zero-shot video-to-text prompting to maximize video humor understanding of large language models (LLMs). With three different evaluation methods using automatic scores, rationale quality experiments, and human evaluations, we show that our prompting significantly improves LLMs’ ability for humor explanation.

## introduction today, a huge number of short-form funny videos are popularly circulated on social media platforms. although humor often triggers instant laughter, understanding humor is not a straightforward process. numerous studies @xcite @xcite @xcite @xcite have explored the cognitive process of humor appreciation. for instance, @xcite and @xcite propose the incongruity theory, asserting that incongruity provokes laughter. @xcite further develops the idea by defining the discrepancy between expectation and content, such as punchlines or cartoons. @xcite suggests the incongruity-resolution theory, positing that humor arises only when the incongruity is resolved by hey luke, sit. luke, dandelion. ## the exfuntube dataset the exfuntube dataset comprises 10,136 videos, each annotated with timestamps of funny moments and corresponding explanations describing why each moment is humorous. the purpose of this dataset is to evaluate the models' ability to explain why a given video is funny as a measure of understanding video humor. ## approach we explore an approach to explain video humor. our idea is first to convert the video content into fine-grained text and then take advantage of recent powerful llms in a zero-shot manner. we design to extract as much information from videos into text as possible. figure 3 shows a zero-shot videoto-text prompting that converts the video content into a text input to llms. ## experiments we experiment with different models to see how well they explain the humor in the exfuntube videos. we evaluate the models in three different ways of model-based automatic scores, rationale quality experiments, and human evaluation. ## conclusion we introduced exfuntube, a dataset consisting of 10,136 user-generated videos annotated with timestamps and explanations of funny moments. our dataset aims to assess how well ai models understand and explain video humor. we devised a zero-shot video-to-text prompting to make existing llms better explain the video content. with three different evaluation methods, we demonstrated that the humor in our dataset is multimodal, and our prompting maximized llms' ability to generate explanations. however, as the performance still falls short of human levels, our dataset remains sufficiently challenging and calls for future research. furthermore, we can consider the training of the model using user feedback for personalized humor understanding.