Can Language Models Understand Physical Concepts? (2023)

Abstract: Language models (LMs) gradually become general-purpose interfaces in the interactive and embodied world, where the understanding of physical concepts is an essential prerequisite. However, it is unclear whether LMs can understand physical concepts in the human world. To investigate this, we design a benchmark VEC that covers the tasks of (i) Visual concepts, such as the shape and material of objects, and (ii) Embodied Concepts, learned from the interaction with the world such as the temperature of objects. Our zero (few)-shot prompting results show that the understanding of certain visual concepts emerges as scaling up LMs, but there are still basic concepts to which the scaling law does not apply. For example, OPT-175B performs close to humans with a zero-shot accuracy of 85% on the material concept, yet behaves like random guessing on the mass concept. Instead, vision-augmented LMs such as CLIP and BLIP achieve a human-level understanding of embodied concepts. Analysis indicates that the rich semantics in visual representation can serve as a valuable source of embodied knowledge. Inspired by this, we propose a distillation method to transfer embodied knowledge from VLMs to LMs, achieving performance gain comparable with that by scaling up parameters of LMs 134\times . Our dataset is available at https://github.com/TobiasLee/VEC.

## introduction with the emergent capabilities such as arithmetic @xcite and multi-step reasoning @xcite brought by large-scale pre-training, language models (lms) are gradually becoming unified interfaces @xcite , capable of instructing embodied robots for high-level tasks such as cleaning the spilled coke in interactive and embodied environments @xcite . understanding physical concepts is an essential prerequisite for these tasks, e.g., producing correct instructions for cleaning the coke requires understanding the visual characteristics of a coke can, as well as physical properties such as hardness. however, it still remains unclear whether current lms can understand basic physical concepts @xcite . to answer the question, we first define an evaluation suite of physical concepts covering visual and embodied concepts. specifically, visual concepts examine knowledge that can be gained via visual perception, including generic visual concepts, such as color, shape, and material of common objects, and spatial perception, which focuses on the relationship between visual stimuli, i.e., relative size and height of objects. the ability to deal with visual concepts serves as the basis for understanding real-world scenes to perform further instruction. embodied concepts examine knowledge that requires more interaction and multimodal sensory experience in the embodied world, including knowledge about the mass, temperature, and hardness of objects, e.g., ice is colder than water. understanding embodied concepts is essential for an embodied agent to make correct choices when translating language into actions @xcite . we compose a visual and embodied concepts evaluation benchmark vec, with examples shown in with the benchmark, we examine a wide range of lms. we cover masked language models and causal language models in text-only lms, including bert @xcite and roberta @xcite , gpt (opt)-family @xcite with parameters ranging from 125m to 175b, llama-1/2 @xcite and vicuna @xcite . furthermore, as humans understand the world by learning from multiple modalities, especially using the visual modality @xcite , we are interested in whether the vision supervision in recent vision-augmented language models (vlms) @xcite @xcite ; ## vec benchmark our vec benchmark aims to evaluate the understanding of physical concepts of lms. inspired by the world scope definitions by @xcite , we divide physical knowledge into visual knowledge and embodied knowledge. the former are visual properties that can be acquired via visual perception, while the latter focus on knowledge that requires multimodal sensory interaction. ## prompting methods recent studies have shown that prompting methods that fit the pre-training paradigm are more effective than other possible prompting methods @xcite . following these studies, we design specific prompts for lms with different objectives. prompting masked language models following pet (schick and schütze, 2021a,b), we probe the masked language models by converting knowledge facts into a question-answering form. for example, a size knowledge fact (coin, smaller than, table) is converted into a sentence with a special mask token: question: is a coin smaller than a table? answer: @xcite . we also explored other prompts, such as is a coin [mask] than table. however, our experiments show that a question-answering form can better induce models to generate answers and avoid the influence of tokenization of different lms. given masked inputs, the model is asked to predict the probabilities of the mask token over two choices, opt bert s 1 : the water is colder than the frying oil. question: is the water colder than the frying oil? answer: @xcite . ## conclusion in this paper, we introduce vec for evaluating the understanding of physical concepts in lms. our results show that large lms understand specific visual concepts but struggle with embodied knowledge. vlms instead perform much better in both the visual and the embodied world, indicating that visual signals are vital for understanding physical concepts. further analysis suggests that transferring the vlm representations to lms effectively boosts embodied concepts understanding, shedding light on directions for improving lms.