L ang S uit· E : Planning, Controlling and Interacting with Large Language Models in Embodied Text Environments (2024)

Abstract: Recent advances in Large Language Models (LLMs) have shown inspiring achievements in constructing autonomous agents that rely onlanguage descriptions as inputs. However, it remains unclear how well LLMs can function as few-shot or zero-shot embodied agents in dynamic interactive environments. To address this gap, we introduce LangSuit·E, a versatile and simulation-free testbed featuring 6 representative embodied tasks in textual embodied worlds. Compared with previous LLM-based testbeds, LangSuit·E (i) offers adaptability to diverse environments without multiple simulation engines, (ii) evaluates agents’ capacity to develop “internalized world knowledge” with embodied observations, and (iii) allows easy customization of communication and action strategies. To address the embodiment challenge, we devise a novel chain-of-thought (CoT) schema, EmMem, which summarizes embodied states w.r.t. history information. Comprehensive benchmark results illustrate challenges and insights of embodied planning. LangSuit·E represents a significant step toward building embodied generalists in the context of language models.

## introduction creating embodied agents that can perceive, think, and interact with simulated environments from the agents' own perspective is a long-standing yet imperative challenge within the realms of robots and artificial intelligence. recent striking progress of large language models (llms) has ushered promising avenues for constructing autonomous agents, wherein llms serve as their central planners or controllers, evidenced by several intriguing demos and benchmarks within simulated environments @xcite @xcite or even realistic * equal contribution † correspondence to zilong @xmath0.@xmath1, spanning from web browsing @xcite , language games @xcite and householding @xcite , etc. these language agents, reliant solely upon language descriptions as their inputs, can benefit from llms' exhibited commonsense knowledge, emergent semantic reasoning and problem-solving skills. despite the promising advances of prior llm-asagent tasks, it remains unclear that: without perception error caused by visual models, can llms behave as embodied agents to solve complicated tasks within dynamic environments? previous works mostly treat llm agents as task planners, providing them with the global observations and all required information (e.g., entire history) in natural language to produce the next step, which contrasts the nature of chat-based language models that can potentially form their internal map and history across conversational turns. moreover, the action spaces across tasks are specially designed and can only be evaluated within specific environments, which limits the potential capacity of evaluating the generalizability across tasks. in stark contrast, research on embodied agents and environments has gone through many successful attempts -numerous open-source embodied systems developed with realistic simulation engines, such as ai2thor @xcite , virtualhome @xcite , spurring the trend of learning embodied instruction-following challenges, such as navigation @xcite , manipulations @xcite , question answering @xcite ,etc. models learned within these environments, nevertheless, struggle with their dependency on simulation engines and are therefore limited to specific tasks. other prominent research works such as palm-e @xcite and rt @xcite sought to powerful vision-language models (vlms) for embod- [system][→bot1]: you can see a table, two chairs in front. interactive qa how many vases are around here? i will go and explore the room... answer is [2] ## the langsuit•e testbed the goal of langsuit•e is to provide a unified testbed thereby evaluating the diversified ability of llm-as-agents in a dynamic embodied environment with textualized perceptual representation and realistic physical limitation. moreover, by unifying various embodied tasks in one environment, agents could improve the ability of knowledge transfer and construct consistent world cognition from interactions with the environment to solve different kinds of tasks. ## conclusion we introduce langsuit•e, a lightweight and simulation-free testbed in textual embodied worlds. designed to be fully embodied, langsuit•e aims to test llm agent's embodied planning capacity without errors from visual perceptions. to further address the embodiment challenge, we devise a novel cot schema, emmem, which summarizes embodied states w.r.t. history information. although benchmark results indicate the effectiveness of em-mem, there still remains a huge performance gap compared with sft results on individual tasks. as such, we call for future exploration of embodied generalists on langsuit•e.