LEGENT : Open Platform for Embodied Agents (2024)

Abstract: Despite advancements in Large Language Models (LLMs) and Large Multimodal Models (LMMs), their integration into language-grounded, human-like embodied agents remains incomplete, hindering complex real-life task performance in 3D environments. Existing integrations often feature limited open-sourcing, challenging collective progress in this field. We introduce LEGENT, an open, scalable platform for developing embodied agents using LLMs and LMMs. LEGENT offers a dual approach: a rich 3D environment with interactive, communicable, and actionable agents, paired with a user-friendly interface, and a sophisticated data generation pipeline utilizing advanced algorithms to exploit supervision from simulated worlds at scale. In our experiments, an embryonic vision-language-action model trained on LEGENT-generated data surpasses GPT-4V in embodied tasks, showcasing promising generalization capabilities. The demo video is available at the following link https://video.legent.ai.

## introduction large language models (llms) @xcite @xcite and large multimodal models (lmms) @xcite @xcite present inspiring capabilities in understanding and generating human-like text and realistic images. however, their direct application in embodied ai, where agents interact in physical or simulated environments, is still primitive. llms and lmms lack the necessary grounding @xcite in physical interactions to operate in these settings effectively. research in embodied intelligence has evolved significantly, leading to more realistic and sophisticated environments @xcite @xcite and increasingly challenging tasks @xcite @xcite . however, these traditional environments and approaches are typically incompatible with current llms and lmms, which hinders the seamless integration of task execution via language interaction. consequently, these approaches do not leverage the extensive generalizable knowledge present in llms and lmms. to achieve generalizable embodied intelligence, two key factors are crucial: language grounding to utilize the extensive knowledge in lmms, and the expansion of training data for embodied ai. there have been noteworthy efforts in combining embodied ai with lmms @xcite . they collect large-scale training data from embodied scenes and train end-to-end mod-els that interpret both language and visual inputs and perform corresponding actions. however, the lack of open-source access to these environments and datasets restricts open-source community-wide progress in this field. therefore, the academic community urgently requires an open-source platform that facilitates the integration of language grounding in embodied environments and schemes to generate large-scale training data for embodied agents based on llms and lmms. towards this aspiration, we introduce legent, an open and user-friendly platform that enables scalable training of embodied agents based on llms and lmms. legent contains two parts. first, it provides a 3d embodied environment with the following features: (1) diverse, realistic, and interactive scenes; (2) human-like agents with egocentric vision capable of executing actions and engaging in direct language interaction with users; to demonstrate the potential of legent, we train a basic vision-language-action model based on lmms with generated data on two tasks: navigation and embodied question answering. the model processes textual and egocentric visual input and produces controls and textual responses directly. the prototype model outperforms gpt-4v (openai, 2023), which lacks training in an embodied setting. the generalization experiment reveals the legent-trained model's ability to generalize to unseen settings. legent platform and its documentation are publicly available at https://legent.ai . ## legent in this section, we introduce our platform leg-ent. the design of legent involves scene, agent, and interface. all three components are specially tailored for the integration of llms and lmms, and ensure scalability. ## data generation the second part of legent is a scalable data generation pipeline. it aims at exhaustively exploiting the inherent supervision from simulated worlds and supporting large-scale training of general-purpose embodied agents. here we elaborate on the implementation of our data generation framework. ## conclusion and future work in this work, we present legent, an open platform for developing embodied agents, focusing on integrating lmms with scalable embodied training. by bridging the gap between embodied ai and lmm's development, we hope legent inspires research in this field. we are committed to the ongoing development of legent, making it more scalable and user-friendly. in our future releases, we prioritize: (1) building a more diverse data generation pipeline.