U nreal LLM : Towards Highly Controllable and Interactable 3 D Scene Generation by LLM -powered Procedural Content Generation (2025)

Abstract: The creation of high-quality 3D scenes is essential for applications like video games and simulations, yet automating this process while retaining the benefits of Procedural Content Generation (PCG) remains challenging. In this paper, we introduce UnrealLLM, a novel multi-agent framework that connects natural language descriptions with the professional PCG system (Unreal Engine 5) to automate scene generation. UnrealLLM constructs a comprehensive knowledge base to translate text into executable PCG blueprints and a diverse asset library that guarantees high-quality scene generation. Additionally, it also introduces a text-based blueprint system with a spline-based control mechanism for geometric arrangement, enabling natural language interaction and enhancing interactivity in 3D environments using UE5’s advanced capabilities. Through extensive experiments, we show that UnrealLLM achieves competitive performance in technical metrics and aesthetic quality, offering unique advantages in generation scale and interactivity. This work makes a valuable contribution to automated 3D content creation, benefiting both novice users and professional designers.

## introduction the creation of high-quality 3d scenes plays a crucial role in various applications, including video games, simulation, and visual effects production. these applications demand not only visually appealing scenes but also functionally complete 3d environments that can support interaction, simulation, and real-world deployment. while recent advances in ai-driven content generation have shown promising results, creating complex, productionready 3d scenes still faces various challenges. to be concrete, current approaches to 3d scene generation broadly fall into two categories, explicit and implicit methods. explicit 3d generation directly generates 3d meshes from descriptions @xcite @xcite @xcite . although showing promise in generating individual 3d objects, they face significant challenges in terms of quality and scalability when dealing with scenes comprising multiple assets. on the other hand, implicit scene generation methods @xcite @xcite @xcite , despite recent advances in interactivity demonstrated by approaches like gamengen @xcite and gamegenx @xcite , primarily focus on 2d visual rendering. while these methods can produce visually impressive results, the lack of explicit 3d representations renders the objects indecomposable, leading to reduced flexibility and applicability. given these limitations in current approaches, procedural content generation (pcg) offers a promising solution by leveraging rule-based spatial assembly mechanisms and existing high-quality assets. these systems excel at creating complex, realistic environments through sophisticated procedural rules, from natural landscapes with realistic ecological patterns to structured urban environments with precise spatial arrangements. however, pcg requires a deep understanding of generation rules and professional software knowledge, making it challenging for beginners and time-consuming to use. to tackle the aforementioned challenges, recent works like 3d-gpt @xcite and scenex @xcite have proposed an instruction-driven approach to 3d scene generation by combining a large language model (llm) agent with procedural generation tools in blender. while current methods have effectively collaborated with human designers to establish procedural generation frameworks, they encounter some limitations. firstly, they depend on infinigen @xcite , which offers a restricted variety of scene types and limited styles of 3d resources. additionally, their approach of using llms to primarily orchestrate pre-defined infinigen apis places inherent limitations on creative potential. in contrast, unreal engine 5 (ue5) foot_0 offers a powerful and flexible ecosystem for procedural content generation (pcg), enabling users to utilize a diverse range of professional assets from the marketplace and leverage advanced technologies like nanite, lumen, and chaos for high-quality visuals. moreover, ue5's native pcg system facilitates direct access to node-level scene composition, presenting opportunities for more dynamic and innovative content creation. however, how llms can interface with node-level scene generation in ue5 also raises challenges. first, constructing the ue5-specific blueprint for scene generation requires extensive expert knowledge, needing a semantic translation mechanism between natural language and node-level pcg workflows. second, the fine granularity and high diversity of assets in ue5 present efficiency challenges for context-aware retrieval across various artistic styles. finally, ensuring stability and quality in generation demands the systematic integration of expert knowledge. the preceding analysis motivates our design of a ue5-native generation framework, named unre-alllm, which redefines the interaction between large language models (llms) and procedural content generation (pcg) processes. this novel multiagent framework bridges the gap between natural language interaction and pcg-based scene generation. as demonstrated in figure 1 , our framework not only facilitates text-to-scene generation but also fully interactive environments that support complex gameplay mechanics like character walking, swimming, and object destruction, leveraging ue5's real-time physics and gameplay systems. a key innovation is the development of a text-based blueprint representation that allows ai agents to systematically process and manipulate pcg graphs, capturing both structural composition and parametric configurations in a machine-processable and human-interpretable format. to enhance asset utilization, we construct a rich 3d asset library with multi-modal representations for context-aware retrieval, and we develop a comprehensive pcg knowledge base encompassing node operations, parameter configurations, and proven generation strategies to ensure reliable, high-quality scene generation. additionally, we introduce a spline-based control enhancement that provides precise spatial control over the generation process, enabling users to achieve specific aesthetic goals while maintaining procedural efficiency. our main contributions can be summarized as follows: text-to-3d scene generation recent advances in text-to-image synthesis have catalyzed significant developments in text-guided 3d content creation. early approaches explored vision-language models for generation guidance @xcite , followed by the development of score distillation sampling techniques that enabled the adaptation of 2d generative models for 3d content synthesis @xcite . the field has witnessed diverse explorations in representation formats, ranging from neural implicit representations @xcite to optimized mesh-based approaches @xcite . researchers have addressed generation consistency through various strategies, including multi-view synthesis methods @xcite and advanced distillation techniques @xcite . parallel efforts have focused on single-view reconstruction @xcite and specialized applications in shape synthesis @xcite and domain-specific generation @xcite . however, existing approaches face challenges in creating productionquality 3d scenes, particularly in terms of structural organization and asset quality. our work addresses these limitations by integrating natural language understanding with established pcg systems and high-fidelity asset libraries. ## experiments metrics. we use multiple metrics to evaluate the performance of our system. for technical assessment, we adopt executability rate (er@1) and success rate (sr@1), where er@1 measures the proportion of proposed actions that can be executed in the pcg system, and sr@1 evaluates the correctness of these actions @xcite . to quantify aesthetic quality, we employ the gpt aesthetic score (gas), extending the aesthetic evalua-method er@1 sr@1 gpt-3.5 @xcite 58.6 41 gpt-4o @xcite 68.2 63.4 o1-mini @xcite 83.0 78 o1 @xcite 75.6 73.2 claude3.5 sonnet(anthropic, 2024) 80.1 70.8 ## conclusion in this paper, we present a novel multi-agent framework that bridges natural language interaction with professional pcg systems in unreal engine 5. our approach combines large language models with pro-cedural generation techniques through text-based blueprint representation and comprehensive knowledge bases, enabling intuitive control over scene generation. experimental results demonstrate our system's effectiveness in technical reliability, aesthetic quality, and gameplay interactivity. we believe our work represents a significant step toward democratizing professional pcg systems, benefiting both novice users and professional designers in automated 3d content creation.