OPE x: A Component-Wise Analysis of LLM -Centric Agents in Embodied Instruction Following (2024)

Abstract: Embodied Instruction Following (EIF) is a crucial task in embodied learning, requiring agents to interact with their environment through egocentric observations to fulfill natural language instructions. Recent advancements have seen a surge in employing large language models (LLMs) within a framework-centric approach to enhance performance in embodied learning tasks, including EIF. Despite these efforts, there exists a lack of a unified understanding regarding the impact of various components—ranging from visual perception to action execution—on task performance. To address this gap, we introduce OPEx, a comprehensive framework that delineates the core components essential for solving embodied learning tasks: Observer, Planner, and Executor. Through extensive evaluations, we provide a deep analysis of how each component influences EIF task performance. Furthermore, we innovate within this space by integrating a multi-agent design into the Planner component of our LLM-centric architecture, further enhancing task performance. Our findings reveal that LLM-centric design markedly improves EIF outcomes, identify visual perception and low-level action execution as critical bottlenecks, and demonstrate that augmenting LLMs with a multi-agent framework further elevates performance.

## introduction embodied learning, particularly through tasks like embodied instruction following (eif) @xcite , stands at the forefront of artificial intelligence research. eif, where agents must interpret natural language instructions to navigate and act within their environment using egocentric observations, epitomizes the challenge of integrating cognitive understanding with physical action. this intersection is crucial for developing autonomous agents capable of nuanced interaction with complex, real-world environments, marking a significant stride towards more advanced and versatile ai systems. as the research community harnesses advancements in deep learning, we edge closer to this ambition @xcite @xcite . traditional approaches to embodied instruction following (eif) often rely on expert-generated annotations, a process that can be both expensive and challenging to scale for real-world applications. in contrast, large language models (llms), such as those cited in recent studies @xcite @xcite @xcite @xcite @xcite , have emerged as a potent alternative, showcasing exceptional capabilities in natural language understanding and generation. these models, enriched by extensive textual datasets, demonstrate significant commonsense reasoning abilities. as a result, there's a growing trend towards leveraging llm-centric architectures for embodied learning tasks including eif, which promise to simplify planning and execution tasks through a few-shot learning paradigm. however, despite their potential, the implementation of eif systems introduce a variety of designs and components across different studies @xcite @xcite @xcite . there remains a notable gap in systematically understanding how these disparate elements influence overall task performance, underscoring the need for a thorough analysis of llm-centric methods within the context of eif. in addressing the complexities of embodied instruction following (eif), we introduce opex, a novel framework designed to systematically outline the essential components for mastering embodied learning tasks. opex is segmented into three core parts: observer, planner, and executor. the observer component is tasked with processing and interpreting sensory inputs, primarily visual, to construct an actionable understanding of the agent's immediate environment. the planner dynamically devises strategic plans as subtasks to complete the tasks based on perceptual inputs, effectively bridging the gap between perception and action. lastly, the executor is responsible for implementing these plans with a skill library, which translates several re-useable skills into precise, context-aware actions within the environment, ensuring the agent's interactions are both relevant and goal-oriented. this tripartite structure provides a clear delineation of roles within the system, facilitating a granular analysis of how each contributes to the overarching performance of eif tasks. to understand the impact of each opex component on performance in eif tasks, we conducted an in-depth analysis. by experimenting with different versions of the observer, planner, and executor components, we assessed how each contributes to and influences overall success. this approach allowed us to identify the key attributes and design choices that enhance the system's ability to tackle complex embodied tasks, providing clear insights into optimizing embodied learning agents. to further unlock the potential of llms in embodied learning, we eliminate the influence of visual perception and low-level action execution of the system utilizing a pure-text counterpart environment @xcite and further adopt a multi-agent llm communication strategy, splitting the instruction-following challenge into distinct reasoning and grounding roles handled by a reasoner agent and an actor agent, respectively. this approach simplifies the task into decision-making processes, where both agents utilize world knowledge obtained from an explorer. this explorer gathers insights either through direct interaction with the environment or from human contributions, thereby enriching the collaborative problem-solving capabilities of the reasoner and actor with more grounded and informed decision-making. our experimental evaluation was conducted using the alfred @xcite and alf-world @xcite benchmarks, providing a comprehensive testing ground for our extensive evaluation. the core analysis of our experiments underscores significant advancements: the llm-centric approach notably enhances perfor-mance in eif tasks. we pinpoint visual perception and low-level action execution as pivotal bottlenecks. moreover, our results affirm that incorporating a multi-agent llm communication strategy into an llm-centric task solver significantly boosts overall task performance on aflworld, showcasing the effectiveness of our proposed methodology in addressing the complexities of embodied learning tasks. ## task formulation we benchmark our method with alfred @xcite and its textworld counterpart alfworld @xcite . both contain a set of environments associated with longhorizon household tasks specified by natural language instructions. the language instruction @xmath0, l low } consists of instructions at two different levels: a high-level instruction goal l high that summarizes the task and a sequence of low-level instructions l low that depict the specific actions required. at the time step t, alfred also provides a visual egocentric observation v t (text observation l t if on alfworld) represents the world state w t . given the language instruction l and an initial observation v 0 (l 0 if on alfworld), the agent's objective is to generate an execution trajectory @xmath1, a 0 , v 1 , a 1 , . . . , v t , a t ⟩, where a t is the action taken by the agent at time step t, and v t+1 is the observation of the world state w t+1 caused by that action. the action space a can be categorized into two classes: navigation actions a n and interaction actions a i , respectively foot_0 . in practice, we follow the setting of film @xcite , where the navigation actions a n are constrained to discrete values, and a pixel-wise interaction mask of the target object must be specified for interaction actions a i . there are seven types of household tasks, of which each episode is terminated either if an agent meets the goal conditions specified in l (success) or reaches the maximum number of steps (fail). see appendix. a for a task example in alfred. ## methodology we first provide an overview of the proposed llmcentric framework (opex) in figure 1 . opex consists of six components: (1) a semantic mapping executor output -------thought: i will explore the room to have a general idea of the environment. ## conclusion we introduce opex, an llm-centric framework tailored for embodied instruction following (eif), and undertake extensive evaluations to dissect the influence of its distinct components. building on this foundation, we further improve opex by integrating world knowledge with a multi-agent llm communication strategy to further harness llms' potential in addressing eif challenges. our comprehensive analysis reveals that an llm-centric design significantly enhances eif performance, pinpointing visual perception and low-level action execution as crucial bottlenecks. additionally, our findings demonstrate that integrating a multiagent llm communication mechanism within llms markedly boosts their effectiveness, offering promising directions for future research in embodied learning.