SMILE : Multimodal Dataset for Understanding Laughter in Video with Language Models (2024)

Abstract: Despite the recent advances in artificial intelligence, building social intelligence remains a challenge.Among social signals, laughter is one of the distinctive expressions that occurs during social interactions between humans.In this work, we tackle a new challenge for machines to understand the rationale behind laughter in video, Video Laugh Reasoning.We introduce this new task to explain why people laugh in a particular video and a dataset for this task.Our proposed dataset, SMILE, comprises video clips and language descriptions of why people laugh. We propose a baseline by leveraging the reasoning capacity of large language models (LLMs) with textual video representation. Experiments show that our baseline can generate plausible explanations for laughter. We further investigate the scalability of our baseline by probing other video understanding tasks and in-the-wild videos. We release our dataset, code, and model checkpoints on https://github.com/postech-ami/SMILE-Dataset.

## introduction in this work, we take the first stepping stone to tackle the challenge of understanding laughter by introducing a task, video laugh reasoning that aims to interpret the reasons behind laughter in a video. for this task, we curate a new dataset, smile, consisting of video clips and corresponding text annotations explaining reasons for laughter. we probe through the question "why do people laugh?" and reason through the answer in a language form; thus, we define the task as a free-form text generation task in which the model generates an explanation for the laughter with a given video clip (see figure 1 ). while reasoning laughter by answering the question is an effective way of probing the level of understanding, laughter itself has an inherently complex nature which can be influenced by diverse factors @xcite @xcite , e.g., the subjectivity @xcite , context knowledge @xcite , and multimodality @xcite . to build a clearer resource for understanding laughter and its social norm behind it, we design the dataset to focus on audience laughter, a cohesive form from social influence in distinct contexts @xcite , and thereby alleviating the subjectivity associated with individual laughter. also, for our task, we propose a baseline based on large language models (llms) with multimodal why the audience laugh? the audience laughed because speaker made a humorous between the serious topic of quantum mechanics and the ridiculous cartoonish image displayed, possibly a playful representation of schrödinger's cat scenario. ## task definition and dataset in this section, we introduce our video laugh reasoning task and our dataset for it. ## discussion in this section, we discuss the scalability of utilizing large language models with textual video representation by conducting evaluations on other tasks and on in-the-wild videos. ## conclusion in this paper, we aim to understand the reason behind laughter by introducing laugh reasoning task, accompanied with smile dataset. while the model did not surpass human capabilities, we show that the model can generate plausible explanations about laughter reason, underlining that multimodal cues in our dataset nicely infuse the laugh reasoning capacity to the model. we also show the results applied to other tasks and other types of video, hinting at the scalability of utilizing llm with multimodal textual representation. limitation & future direction. our llm-based baseline serves as the initial method for laugh reasoning task and has a margin to improve. for the multimodal textual representation, as it is a primitive form for capturing human social interaction in the video, we can enhance it with diverse attributes such as gesture, eye gaze, and relationship or use other representations such as scene graph. our work mainly focuses on audience laughter as the first stepping stone toward understanding laughter due to its distinct and cohesive signal, while there are diverse mechanisms behind laughter. recognizing this, enriching our work with diverse video types like vlogs, movies, and talk shows is a promising direction to capture a broader range of laughter, as we show the possibility in § 5.2. potential application & broader impact. our work can be regarded as a stepping stone toward developing socially intelligent agents that understand and appropriately create non-verbal cues, such as laughter, playing a crucial role in building rapport, expressing emotions, and creating deep emotional exchanges @xcite . such advancement moves us beyond the capabilities of current dialogue agents, e.g., chatgpt or alexa, which mostly focus on verbal signals. incorporating 3d talking head methods @xcite could offer the way agents are visualized, enabling more expressive and multimodal interactions with users.