V - ALPHASOCIAL : Benchmark and Self-Reflective Chain-of-Thought Generation for Visual Social Commonsense Reasoning (2025)

Abstract: Social commonsense reasoning naturally involves both the verbal and non-verbal cues of a social interaction. It is important for Large Vision-Language Models (VLMs) to leverage both textual and visual information in performing tasks like social understanding and reasoning. However, while current LLMs have shown good social reasoning capabilities in textual context, whether they can effectively incorporate visual information in social comprehension remains under-explored. To narrow the gap, we first construct and propose a benchmark: V-Social, featuring well-aligned text and visual content, tailored to assess visual social commonsense for multimodal foundation models. Through experimenting with V-Social, we find that even the most advanced VLM, GPT-4o, often falls short in social commonsense reasoning. This highlights the critical need to enhance the social grounding of VLMs. One major obstacle for improving this is the lack of high-quality data with good reasoning process. To overcome this obstacle, we introduce V-AlphaSocial, a novel method that generates high-quality chain-of-thought reasoning paths from unlabeled data. We design a visual reasoning reward model to improve VLM, and then iteratively refine both the VLM and the reward model. Our extensive analysis showcases how our method enhances social commonsense reasoning, proposing an effective approach that facilitates deeper exploration into field.

## introduction recent advances in large language models (llms) have significantly enhanced their ability to understand social commonsense and mimic humans in generating socially acceptable response @xcite . however, social interactions are inherently multimodal @xcite , incorporating not only textual but also visual cues such as gestures, facial expressions, and actions. integrating these features from visual modalities into social commonsense reasoning tasks is crucial for understanding and improving models' social commonsense reasoning holistically. by incorporating visual encoder with llm, vision-language models has demonstrated decent performance on a wide range of tasks such as image & video captioning and understanding @xcite . these models show potential in processing nuanced and contextrich social interactions. however, there is a lack of comprehensive benchmarks specifically designed to evaluate their abilities in visual social commonsense reasoning. current benchmarks are often constrained by high-quality and aligned multi-modal data @xcite . moreover, they do not fully capture the nuanced context and reasoning depth necessary for a comprehensive understanding of social commonsense. @xcite . to narrow the gap, we propose a comprehensive benchmark: v-social, covering six curated dimensions of social commonsense reasoning. our dataset consists of data with well-aligned text and visual modality. to simulate complex social situations, we collect long videos engaging multiple characters and propose questions that require deep reasoning from interactions between characters, which makes the task more challenging. after evaluations of the stateof-the-art vlms on v-social, we find that there are many failure cases such as reasoning errors and social understanding errors. these findings highlight the critical need to extend the general reasoning abilities of current vlms to nuanced social commonsense reasoning tasks, which has been illustrated in figure 1 . moreover, addressing this need presents two key challenges: the lack of high-quality training data with social reasoning processes, and reasoning errors in complex social situations. as such, we propose a novel framework: v-alphasocial, which leverages data with no reasoning path to generate high-quality reasoning processes and augments existing data with "thought process" like human beings. our approach starts by using a single in-context data sample, grounded in social commonsense with a complete reasoning path, to generate reasoning processes on unlabeled data. a multi-modal discriminator is then trained to provide rewards for selecting optimal reasoning processes. to enhance the diversity of the reasoning process generated by vlms, we explicitly augment the context with visual information using surrogate models such as blip and facial expression recognition (fer) models. this method not only enriches the training process by ensuring high fidelity in the generated reasoning across modalities. we conduct comprehensive experiments on v-social and socialiq2 @xcite to demonstrate the effectiveness of the proposed v-alphasocial, achieving a performance gain of 14% compared to the baselines on v-social. additionally, we find that incorporating visual information significantly improves the performance of social commonsense reasoning, underscoring the importance of this task. in summary, our contributions are threefold: (1) we curate v-social, a novel and high-quality benchmark comprising aligned text, image, and video data to comprehensively evaluate multimodal foundation models' ability to conduct visual social commonsense reasoning. (2) we introduce v-alphasocial, an innovative self-evolving approach to iteratively synthesize high-quality reasoning processes from unlabeled multimodal data, ## experiments for this part, we conduct extensive experiments to answer the following questions: q1: does visual modality brings more information that can help vlms to perform better in social commonsense reasoning (see section 4.2) q2: how does different model perform on our benchmark (see section 4.2)? q3: how about the effectiveness of v-alphasocial? does our model generalize well to our collected v-social? (see section 4.2) q4: what are the failure modes of the baseline vlms and how our method mitigates them? (see section 4.2) ## conclusion our research addresses the critical challenge of integrating multimodal data for the enhancement of social commonsense reasoning in visual language models (vlms). by introducing a dataset, v-social, we provide the necessary protocols for more effective benchmarking of visual social commonsense reasoning. furthermore, our novel method, v-alphasocial, leverages self-training with context augmentation and a multimodal discriminator to significantly improve the generation of high-quality reasoning paths. through our extensive analysis, we demonstrate the efficacy of v-alphasocial in enhancing the social commonsense capabilities of vlms. for future work, we plan to generalize our method to other video question answering and explore more in-depth alignment algorithm for boosting vlms with lowresource data.