Does Visual Grounding Enhance the Understanding of Embodied Knowledge in Large Language Models? (2025)

Abstract: Despite significant progress in multimodal language models (LMs), it remains unclear whether visual grounding enhances their understanding of embodied knowledge compared to text-only models. To address this question, we propose a novel embodied knowledge understanding benchmark based on the perceptual theory from psychology, encompassing visual, auditory, tactile, gustatory, olfactory external senses, and interoception. The benchmark assesses the models’ perceptual abilities across different sensory modalities through vector comparison and question-answering tasks with over 1,700 questions. By comparing 30 state-of-the-art LMs, we surprisingly find that vision-language models (VLMs) do not outperform text-only models in either task. Moreover, the models perform significantly worse in the visual dimension compared to other sensory dimensions. Further analysis reveals that the vector representations are easily influenced by word form and frequency, and the models struggle to answer questions involving spatial perception and reasoning. Our findings underscore the need for more effective integration of embodied knowledge in LMs to enhance their understanding of the physical world.

## introduction embodied knowledge is acquired through experience and contextualized in relation to the body @xcite . it is inherently sensory, encompassing sights, sounds, smells, touch, and taste @xcite . humans rely on both sensory experiences and language experience to represent knowledge @xcite @xcite @xcite @xcite @xcite . similarly, embodied ## method this paper proposes an embodied knowledge understanding benchmark consisting of two tasks: sensoryvec and perceptualqa, which evaluate the model's capability to represent sensory information in vector form and answer perception-related questions, respectively. ## discussion experimental results on both datasets demonstrate that existing models perform poorly in representing and applying embodied knowledge, with no clear or systematic improvements observed in vlms over ## conclusion in this paper, we investigate how visual grounding influences models' understanding of embodied knowledge. to this end, we introduce a benchmark grounded in the psychological framework of the perceptual system, covering external senses (visual, auditory, tactile, gustatory, olfactory) as well as interoception. our benchmark comprises two tasks-sensoryvec and perceptualqa-with over 1,700 questions designed to systematically evaluate the models. our findings reveal that existing models perform suboptimally in embodied knowledge understanding, with vision-language models showing no significant advantage over text-only models. this suggests that current visual grounding approaches do not effectively enhance embodied knowledge comprehension. further analysis shows that models' vector representations are susceptible to surface form and frequency bias, and they struggle with spatial perception and reasoning tasks. these insights underscore the importance of continued advancement in embodied knowledge understanding for agi development. future research should explore developing diverse forms of training data, novel training tasks and architectures that better integrate multimodal perceptual information to advance embodied knowledge understanding. specifically, in addition to text and image data, several categories of multimodal information may need to be incorporated, including dynamic sensory data such as video sequences, perception-related brain neural data, and feedback signals from non-human sensors such as haptic sensors and motion capture systems. these diverse data modalities may be essential to fully support the acquisition of comprehensive embodied knowledge. furthermore, future models may require joint training with embodied agents on foundational tasks that involve interaction with simulated or real-world environments, which would enable them to actively explore, ground concepts through experiential learning, and acquire causal understanding beyond statistical correlations.