E go N ormia: Benchmarking Physical-Social Norm Understanding (2025)

Abstract: Human activity is moderated by norms; however, supervision for normative reasoning is sparse, particularly where norms are physically- or socially-grounded. We thus present EgoNormia \lVert \epsilon \rVert , comprising 1,853 (200 for EgoNormia-verified) multiple choice questions (MCQs) grounded within ego-centric videos of human interactions, enabling the evaluation and improvement of normative reasoning in vision-language models (VLMs). spans seven norm categories: safety, privacy, proxemics, politeness, cooperation, coordination/proactivity, and communication/legibility. To compile this dataset at scale, we propose a novel pipeline to generate grounded MCQs from raw egocentric video. Our work demonstrates that current state-of-the-art VLMs lack robust grounded norm understanding, scoring a maximum of 54% on EgoNormia and 58% on EgoNormia-verified, with performance across norm categories indicating significant risks of safety and privacy when VLMs are used in real-world agents. We additionally explore methods for improving normative understanding, demonstrating a naive retrieval-based generation (RAG) method using can enhance normative reasoning in VLMs.

## introduction humans have a long history of expecting ai to adhere to human-defined norms @xcite @xcite . this is because norms are a fundamental regulator of human activities and interactions @xcite , with even children being able to operate within norm-regulated environments @xcite . given the importance of norms to embodied action-taking, and the increasing capabilities and prevalence of model-driven embodied agents, we ask: can vision-language models (vlms) can understand norms grounded in the physical world and make human-aligned, norminformed decisions? the answer to this question is critical if vlm-based agents are expected to collaborate and coordinate with humans @xcite , safely @xcite and responsibly @xcite . current sota vlms are neither optimized for, nor evaluated on, physical-normative reasoning. while they excel at mathematical, scientific, and abstract reasoning @xcite @xcite , they are unlikely to have the same strong understanding of human normative dynamics in the physical world. this is because, unlike humans, who learn norms through active feedback and trial-and-error exploration @xcite , vision-language models are trained on massive-scale corpuses @xcite , where examples of physically-grounded normative reasoning are sparse @xcite . to comprehensively measure vlm normative reasoning ability, we introduce egonormia, 1 a challenging qa benchmark that is physically grounded in 1k+ egocentric social interaction clips from ego4d @xcite . egonormia spans 100 distinct settings across a wide range of activities, cultures, and interactions. unlike similarly visually-grounded spatiotemporal, predictive, or causal reasoning benchmarks @xcite , egonormia evaluates models' ability to reason about what should be done under social norms. egonormia highlights cases where these norm-related objectives conflict-the richest arena for evaluating normative decision-making. we further introduce egonormia-verified, a split of 200 egonormia tasks, to enable quicker evaluations. 1 ## egonormia egonormia is designed to achieve several goals: ## evaluation accuracy is used in the first two subtasks with a single ground-truth answer; intersection over union (iou) is used on the third subtask, where multiple contextually-sensible action choices exist. we evaluated the following state-of-the-art foundation models: gemini 1.5/2.0/2.5 flash/pro @xcite gpt-4o @xcite , claude 3.5 sonnet @xcite , o3/o4-mini foot_2 (openai, 2024), deepseek r1 (guo et al., 2025), internvl 2.5 @xcite , qwen 2.5 vl @xcite . to characterize the impact of visual priors on model performance, egonormia benchmarking was performed across three settings: (a) blind (no input), where only the questions are provided to the models; (b) pipeline (text-only), where a rich text description of the scene generated by gemini 1.5 flash is provided as part of the questions; and (c) video, where both video and questions are provided. for compatibility, videos are sampled at one frame per second and concatenated ltr foot_3 into a single image, as this yields the best performance of all alternatives; results of ablation of input format are tabulated in appendix j. we use cot prompting @xcite ) across all non-reasoning models in evaluation and provide results in ## conclusion we introduce egonormia, a novel benchmark and dataset designed to rigorously evaluate the ability of vlms to understand physical social norms (psn) in egocentric embodiments. we demonstrate that, despite sota models' strong visual recognition and abstract reasoning capabilities, they remain inferior to humans in psn understanding, primarily due to norm sensibility and prioritization errors. we demonstrate egonormia's direct utility in augmenting normative understanding by testing a retrieval-based method, demonstrating improvements across out-of-domain and out-of-embodiment videos. finally, we identify opportunities for future work in embodied norm understanding, suggesting post-training on large norm datasets as a promising direction for study.