NEWTON : Are Large Language Models Capable of Physical Reasoning? (2023)

Abstract: Large Language Models (LLMs), through their contextualized representations, have been empirically proven to encapsulate syntactic, semantic, word sense, and common-sense knowledge. However, there has been limited exploration of their physical reasoning abilities, specifically concerning the crucial attributes for comprehending everyday objects. To address this gap, we introduce NEWTON, a repository and benchmark for evaluating the physics reasoning skills of LLMs. Further, to enable domain-specific adaptation of this benchmark, we present a pipeline to enable researchers to generate a variant of this benchmark that has been customized to the objects and attributes relevant for their application. The NEWTON repository comprises a collection of 2800 object-attribute pairs, providing the foundation for generating infinite-scale assessment templates. The NEWTON benchmark consists of 160K QA questions, curated using the NEWTON repository to investigate the physical reasoning capabilities of several mainstream language models across foundational, explicit, and implicit reasoning tasks. Through extensive empirical analysis, our results highlight the capabilities of LLMs for physical reasoning. We find that LLMs like GPT-4 demonstrate strong reasoning capabilities in scenario-based tasks but exhibit less consistency in object-attribute reasoning compared to humans (50% vs. 84%). Furthermore, the NEWTON platform demonstrates its potential for evaluating and enhancing language models, paving the way for their integration into physically grounded settings, such as robotic manipulation. Project site: https://newtonreasoning.github.io

## introduction natural language processing (nlp) has made remarkable progress using contextualized representations trained on extensive unprocessed text data figure 1 . as works begin leveraging llms in physically grounded contexts, it is crucial to understand whether such models possess the ability to reason about everyday scenarios. newton, a repository, pipeline, and benchmark, facilitates evaluation of various llms in a physical reasoning context. @xcite . as we start using llms in physically embodied pipelines @xcite @xcite , it is crucial to comprehensively understand the extent that llms can perform physical reasoning. some studies have proposed generalized frameworks to assess language model performance @xcite , while others have designed question answering and reading comprehension datasets to probe llms @xcite @xcite . however, few have explored the physical reasoning ability of llms. physical reasoning involves the cognitive process of comprehending and predicting the dynamics of physical systems based on observable phenomena and fundamental principles @xcite . it encompasses the capacity to make sense of the world by applying knowledge of attributes, such as brittleness, malleability, etc. by considering brittleness, for instance, we recognize the need to handle an object with caution, while malleability suggests that an object can be easily reshaped under pressure without fracturing. these abstract concepts enables reason-ing about the response of objects to interactions or changes in the environment, as shown in figure 1 . nevertheless, creating an evaluation framework for physical reasoning is difficult, primarily due to the lack of paired object-attribute data. humans possess a wealth of knowledge regarding the internal structure of objects and their interactions with the physical world. however, this knowledge is often implicitly acquired, making it difficult to explicitly represent such information. previous studies in the field of physical reasoning, such as @xcite ; @xcite , have focused on common-sense reasoning or small-scale validation. as a result, the need for a comprehensive and systematic assessment of physical reasoning remains an open problem. to address this gap, we propose newton, a repository, pipeline, and benchmark designed to evaluate the physical reasoning capability of llms. the newton repository consists of labeled object-centric data, crowd-sourced for 700+ objects across 8 physical attributes. the newton pipeline introduces a method for systematically generating infinite evaluation questions tailored to specific use cases. the newton benchmark consists of 160k pre-generated questions of progressive difficulty, spanning tasks of foundational attribute comprehension, explicit application, and implicit scenario-based analysis. extensive empirical findings demonstrate the unique contributions of newton, revealing its usefulness for evaluating llms' understanding of underlying physics principles that dictate the behavior and properties of objects in everyday scenarios. moreover, new-ton effectively compliments the existing repertoire of reasoning benchmarks and datasets, further enhancing the potential to assess and refine the physical reasoning capabilities of llms. ## newton repository at the core of constructing an evaluation framework for physical reasoning lies the need for a representation that captures the essential attributes of objects. in this section, we highlight the new-ton repository, including the identification and a shortlist of objects and attributes, and obtaining a set of consistent object-attribute annotations. ## newton pipeline leveraging the object-attribute repository, we introduce a pipeline which enables systematic generation of diverse questions based on pre-defined personalized templates, as shown in figure 2 . the process starts with crowd-sourcing human annotations on object attributes, which is then filtered based on inter-annotator agreement. the filtered data is then used to populate template-based questions in a systematic manner, a process which is extensible to a diverse range of personalized scenarios, beyond those established in the paper. ## newton benchmark we introduce the newton benchmark, a tool to assess the cognitive ability of language models to understand and reason about physical attributes of everyday objects. the newton benchmark comprises of three progressively challenging tracks, and have a combined 160k questions covering 700+ objects and 8 unique attribute categories. ## results in this section, we evaluate the performance of state-of-the-art models on newton. specifically, we quantitatively analyze the performance of the models on the tasks of foundational attribute comprehension, explicit application, and implicit analysis. we also qualitatively examine the patterns in errors made by the models. ## conclusion in this work, we present newton, a repository, pipeline, and benchmark to support the assessment and refinement of llms in the context of physical attribute understanding of everyday objects. the newton repository offers a large collection of object-attribute pairs, enabling the generation of infinite test scenarios for evaluating language models, using the newton pipeline. to demonstrate its usefulness and potential, we introduce the newton benchmark, which consists of 160k questions involving 700+ objects and 8 attributes. these questions cover three distinct tracks: fundamental object understanding, explicit application, and implicit analysis. newton is built to enable comprehensive and automatic evaluation of language models across an array of scenarios, and support the reliable incorporation of llms into physically grounded contexts and applications.