L 3 GO : Language Agents with Chain-of-3 D -Thoughts for Generating Unconventional Objects (2025)

Abstract: Diffusion-based image generation models such as DALL-E 3 and Stable Diffusion-XL demonstrate remarkable capabilities in generating images with realistic and unique compositions. Yet, these models are not robust in precisely reasoning about physical and spatial configurations of objects, especially when instructed with unconventional, thereby out-of-distribution descriptions, such as “a chair with five legs”. In this paper, we propose a language agent with chain-of-3D-thoughts (L3GO), an inference-time approach that can reason about part-based 3D construction of unconventional objects that current data-driven diffusion models struggle with. More concretely, we use large language models as agents to compose a desired object via trial-and-error within the 3D simulation environment. To facilitate our investigation, we develop a new benchmark, Unconventionally Feasible Objects (UFO), as well as SimpleBlenv, a wrapper environment built on top of Blender where language agents can build and compose atomic building blocks via API calls. Human and automatic GPT-4V evaluations show that our approach surpasses the standard GPT-4 and other language agents (e.g., ReAct and Reflexion) for 3D mesh generation on ShapeNet. Moreover, when tested on our UFO benchmark, our approach outperforms other state-of-the-art text-to-2D image and text-to-3D models based on human evaluation.

## introduction ai applications that generate 2d images @xcite @xcite and 3d models @xcite * work conducted while at yale university. correspondence: yutaroyamada@sakana.ai † work conducted while at allen institute for ai 1 https://www.blender.org/ l3go dall-e 3 prompt: "a chair with five legs" from text instructions have opened up significant possibilities for creators. however, these tools lack precise output controls as they often produce unexpected or "hallucinatory" results @xcite not loyal to the input prompts. additionally, early versions of stable diffusion @xcite had difficulty in combining multiple concepts in one image or would mix up different attributes. previous efforts have improved performance on object-attribute attachment, missing objects, etc. by steering attention layers @xcite @xcite , or, by training larger models with detailed captions on a vast scale (stablediffusion-xl (sdxl) @xcite and dall-e-3 @xcite .) however, even the most performant diffusion model, dall-e 3, still fails to generate objects that require precise 3d spatial understanding like "a chair with five legs" (figure 1 ). this difficulty persists even after repeated attempts to adjust dall-e-3's outputs with human feedback directly, e.g., "the chair you generated has seven legs. please make a chair with exactly five legs." we posit that the sophisticated text-based reasoning abilities inherent in llms can compensate for shortcomings in the 3d spatial comprehension of text-to-2d image and text-to-3d models. we present l3go, an inference agent capable of iteratively soliciting feedback from llms to integrate corrections and enhance the precision of rendering a 3d mesh used as a skeleton to generate 2d image. we conduct our experiments within blender -a widely used 3d modeling software. we create and release an environment called simpleblenv, based on blender, to systematically evaluate text-to-3d mesh generation performance of llm agents. state of the art llms such as gpt-4 @xcite , despite being trained on only text has decent spatial reasoning capabilities. figure 2 shows mixed results when gpt-4 is prompted to write a python script that can run in blender to create 3d meshes of basic objects solely based on the object name. on the one hand, text-only gpt-4 demonstrates surprising proficiency in creating simple 3d objects like lamps (2a) comprising of three basic shapes. however, as object complexity increases to more than four parts (ex: four legs of a table) or complex objects like airplane (2b, 2c), gpt4's success in perfectly assembling them is limited. our l3go agent bridges these gaps by breaking down the constructing complex objects by employing a more structured, part-by-part approach into: (1) identifying relevant parts specifications and critiquing them, (2) identifying spatial specifications and placement, (3) running the current action to critique the spatial placement and completion. this setup iteratively seeks feedback from simpleblenv and the specifications and critiques are generated from llm. finally, we render the mesh into an image, and feed it into controlnet @xcite with canny edge detection @xcite to generate a textured and more natural looking image. we conduct human evaluations to compare the performance of llm-based mesh creation using 13 popular object categories from shapenet. l3go outperforms basic gpt-4, react-b, and reflexion-b according to both human and auto evaluation. we also show that mesh quality evaluation using gpt-4v (openai, 2023) yields a metric with high correlation to human judgement. finally, we introduce unconventionally feasible objects, named ufo with unconventional yet feasible objects. we show that l3go surpasses current state-of-the-art text-to-2d image and text-to-3d mesh models on ufo. collectively, our findings indicate the promising role of integrating language agents in diffusion model pipelines, particularly for constructing objects with specific attribute requirements in the future applications of generative ai. large language models as agents llm agents @xcite @xcite @xcite represent a new category of artificial intelligence systems built upon large models. these agents are capable of acting, reasoning, and interacting with external environments. although llms have limited executable skills on their own, when integrated with external apis and knowledge sources, they can tackle a wide range of tasks @xcite ). an iterative approach has shown to be beneficial to solve natural language processing tasks, as evidenced by react @xcite and embod-ied tasks applied to games @xcite , web navigation @xcite , and robot navigation @xcite . our approach, which utilizes llms for creating 3d meshes, contributes to the expanding research in this developing field. ## experiments on shapenet in this section, we detail the baseline methods ( §3.1) to compare text-to-2d image and text-to-3d mesh generation to compare with our l3go. we demonstrate the effectiveness of llm-based methods in the generation of simple 3d objects, specifically focused on 13 well-known categories sourced from shapenet. for automatic evaluation, we use gpt-4v as an evaluator for recognizing 3d meshes ( §3.2) of simple objects. we also show that human assessments and gpt-4v's automated ## conclusion we introduced l3go, a language agent designed to generate 3d objects from text instructions through an api we developed for blender, a 3d modeling software. our evaluation using 13 largest object categories from shapenet shows that l3go's superior 2 https://platform.openai.com/docs/guides/images capabilities in comparison to other models such as gpt-4, react, and reflexion. additionally, we devised ufo, a set of challenges aimed at testing the ability of generative ai models in creating common objects with unconventional characteristics. the performance of l3go marks a significant advancement in the application range of language models. for instance, diffusion models could be further improved with unconventional data generated by structured prompting. moreover, analyzing how language models process spatial information with internal model representations may yield valuable insights into understanding and improving their 3d modeling abilities.