VIBE : Can a VLM Read the Room? (2025)

Abstract: Understanding human social behavior such as recognizing emotions and the social dynamics causing them is an important and challenging problem. While LLMs have made remarkable advances, they are limited to the textual domain and cannot account for the major role that non-verbal cues play in understanding social situations. Vision Language Models (VLMs) can potentially account for this gap, however their ability to make correct inferences over such social cues has received little attention. In this paper, we explore the capabilities of VLMs at social reasoning. We identify a previously overlooked limitation in VLMs: the Visual Social-Pragmatic Inference gap. To target this gap, we propose a new task for VLMs: Visual Social-Pragmatic Inference. We construct a high quality dataset to test the abilities of a VLM for this task and benchmark the performance of several VLMs on it.

## introduction in recent years there has been increasing interest in ai systems that can achieve human levels of social intelligence. llms in particular have shown remarkable promise in such tasks, though they struggle when presented with complex scenarios @xcite @xcite and are limited to the textual domain. the recent emergence of vlms @xcite @xcite provides an exciting opportunity to study social scenes by connecting textual and visual information and leveraging the strong reasoning capabilities of llms. however, while some works have explored the efficacy of using vlms for social intelligence tasks @xcite @xcite , this area still remains relatively underexplored. in this paper, we conduct a thorough investigation of vlms' ability to perform social reasoning tasks. we focus on emotion-related inferences drawn from videos that depict a scene's social dynamics. our goal is to test whether vlms can "read the room", i.e., assess the alignment between their inferences and human-level understanding of the scene. our assessment is done at two levels of analysis. first, in terms of the visual cues the model identifies as relevant for understanding the social situation, and second in terms of the pragmatic inferences made on top of these cues. most works have focused on the correctness of the first step, i.e., testing whether models incorrectly hallucinate a visual pattern @xcite @xcite @xcite . in contrast, we focus on the second and as a result expose a previously overlooked limitation in vlms: even when the vlm correctly identifies a relevant visual cue, it struggles to interpret it correctly. we refer to this as the visual social-pragmatic (vsp) inference gap. to further elucidate what we mean by this, we use figure 1 (a frame representing a short video clip) as an example: while a man is clearly smiling, it does not indicate joy, as evident by his slumped posture and teary eyes. furthermore, the second person in the scene (a pregnant woman in hospital gown) looks at him in sympathy and comforts him. the smile should be interpreted as sad or bittersweet. however, while the vlm correctly identifies the smile, it misinterprets it as evidence for joy. our diagnostic process proceeds in three steps. first we conduct a diagnostic analysis over the emotion recognition task in conversational videos (see sec. 4) by evaluating the contribution of the visual modality. while offering limited (often negative) contribution when fusing the modalities directly, we note an interesting finding -when visual cues derived from the vlm augment the conversational transcript, they can lead to improved performance. this improvement is not guaranteed and careful tuning is needed, indicating that these cues or inferences made over them are noisy. second, through human evaluation we separate between noisy cues (i.e., hallucinated) vs. misinterpreted cues (see sec. 5.3). we curate a dataset, vibe (vsp inference of behavior and emotion), con-sisting of 994 unique instances of the vsp inference task. we benchmark several vlms on our dataset, and we compare model performances to human performance on the dataset, revealing that humans outperform the best vlm by 17.2% in accuracy. this indicates that vsp inference-and not only hallucination-is an important limitation that vlms struggle with. we perform analyses to elucidate exactly which kinds of social visual cues are hallucinated, misinterpreted, or uninformative. specifically, we see that the most common failure mode on a downstream emotion recognition task is misinterpretation, particularly among subtle facial movements like gaze and eye behavior and furrowed brows (sec. 5.3). our contributions are: ## task definitions in this section we briefly define the two tasks that the paper addresses: the diagnostic task of emotion prediction (in multi-party conversation videos) and the novel proposed task of visual social-pragmatic inference (vsp inference). we also define vocabulary used in the paper. ## diagnostic task: emotion prediction the conversations and video clips we used were sourced from the mc-eiu dataset @xcite which consists of text transcripts of conversations and video clips of the utterances. since our focus was videos that require vsp inferences, we used only the videos explicitly labeled with a non neutral emotion. additionally, it is known that vlms struggle with long context videos @xcite , so we only used videos that were under 4 seconds. the final dataset included 3,536 chinese video clips (from three different shows) and 5,589 english clips (from two shows). this formed a robust, diverse set of over 9000 videos spanning multiple languages, cultures, and genres. as illustrated in figure 2 , we had two main diagnostic experimental settings: (1) the vlm acts independently to do the task, and (2) the vlm acts as a perception agent only, and a stronger llm (gpt-4o-mini) reasons over the visual information. for both settings, we compare performance with the very strong baseline of an llm (gpt-4o-mini) operating over the text transcript. the motivation behind having the two settings was to probe two different capabilities of the vlm. in setting (1) the goal was to test the vlm's ability to directly identify the emotion of the speaker. it was not obvious how the vlm would perform compared to the stronger llm; while the llm has stronger reasoning capabilities, the vlm is aided by rich visual data in addition to the text. in contrast, it was reasonable to expect a boost in performance when the vlm had both modalities, compared to either one alone. in setting (2) the goal was to test the ability of the vlm to identify, accurately interpret and communicate what it saw to the llm (gpt-4omini). in this setting, it was reasonable to expect that the additional visual information would boost the performance of the llm compared to the text only baseline. for both settings, to give the vlms the best chance, we sampled the maximum possible frames that our compute resources allowed (30 frames). this decision was based on prior research that showed that performance generally increases as more frames are sampled @xcite . ## proposed task: visual social-pragmatic inference the vsp inference task is a new task we propose in this paper. given a visual cue and a video, the task is to correctly interpret the pragmatic meaning of the cue. in the following sections we describe the process of creating the dataset and formalize the inputs and outputs. we benchmark select vlms on the dataset and present the results and analysis. ## conclusion we introduced the vsp inference task, exposing a previously overlooked limitation in vlms. through diagnostic analyses, human annotation and the vibe dataset, we showed that current vlms struggle with vsp inference. our results underscore the need for improved social reasoning in vlms and provide a benchmark for future research.