Navigate Complex Physical Worlds via Geometrically Constrained LLM (2024)

Abstract: This study investigates the potential of Large Language Models (LLMs) for reconstructing and understanding the physical world based solely on textual knowledge. It explores the impact of model performance on spatial understanding abilities by introducing a set of geometric conventions and developing a workflow based on multi-layer graphs and multi-agent systems. The study examines how LLMs achieve multi-step and multi-objective geometric inference in a spatial environment, using unified geometric conventions and a graph-driven framework. A genetic algorithm, inspired by large-scale model knowledge, is employed to solve geometric constraint problems, enhancing the spatial reasoning capabilities of LLMs. This work innovatively explores the feasibility of using text-based LLMs as builders of the physical world and designs a workflow to enhance their spatial comprehension and construction capabilities.

## introduction llms acquire extensive world knowledge embedded in textual data through pre-training. this raises an intriguing question: can llms reconstruct and simulate the physical world using this textual knowledge? the physical world, characterized by complex geometric and physical constraints, can be abstracted into fundamental geometric shapes. utilizing a custom-designed engine, we simplify the 3d world's geometric content into basic cube combinations. this work pioneers the exploration of text-only llms as potential builders of the physical world, leveraging their pre-trained knowledge to understand and generate 3d spatial representations purely from textual descriptions. some preliminary work on world-building has explored constructing 3d worlds at the image level. techniques like 3d-vae-gan @xcite and pix2vox @xcite variational autoencoders (vaes) @xcite and generative adversarial networks (gans) @xcite to generate high-quality 3d models with precise shape and pose control. atlasnet @xcite approximates 3d surfaces by learning a set of 2d textures, effectively handling irregular topologies. despite their impressive quality, these models struggle with simulating complex physical interactions and maintaining spatial consistency due to intricate and dynamic geometric constraints @xcite . some methods rely on high-precision geometric libraries or external knowledge bases for humanlevel prior knowledge. for instance, @xcite and @xcite use llms to generate 3d scene images by calling blender apis based on user requirements. @xcite proposes combining external knowledge bases to generate 3d scenes from sketches. however, these methods heavily depend on external libraries and interfaces, which lack flexibility and face challenges like resource maintenance, copyright disputes, and network security issues @xcite . we explored how to leverage llm pre-training knowledge to autonomously guide complex geometric constraints. our evaluation compared the spatial construction and geometric relationship understanding abilities of gpt-3.5-turbo and gpt-4, revealing that gpt-4 excels in spatial construction tasks due to its superior performance. we also introduced an innovative multi-agent approach for 3d scene construction, establishing geometric conventions at three levels (center, axis, and surface) to standardize the spatial relationships of 3d objects as understood by llms. this multi-level graphdriven approach enhances the spatial understanding and reasoning capabilities of llms. the workflow ensures information consistency and uniformity, mitigating data silos and redundancy issues, while enabling llms to explore their ability to understand geometric relationships of physical world. 1 2 related work ## experiment in this section, we will discuss the factors affecting the quality of the 3d scene graph generated by the llm from two aspects. the first influencing factor is the model's ability. we test the generation performance of the base models gpt-3.5-turbo and gpt-4 without using the framework. the second influencing factor is the degree of integration with the work framework. we set up three sets of experiments to explore the complete use of the work framework, including ablation experiments to analyze the impact of removing certain components. ## conclusion and limitation our research provides an intuitive demonstration of the spatial understanding capabilities of llms and quantitatively evaluates the spatial comprehension of two distinct models. additionally, we enhance the geometric understanding and spatial reasoning abilities of llms in complex physical environments by implementing well-defined geometric conventions and a graph-driven framework. this study is conducted using a customdeveloped sandbox platform, designed to present the spatial concepts understood by llms in a more intuitive and flexible manner. however, due to resource constraints, we are unable to test higherperforming models, which limits our ability to fully showcase the framework's potential in improving the spatial understanding of llms.