E ^2 CL : Exploration-based Error Correction Learning for Embodied Agents (2024)

Abstract: Language models are exhibiting increasing capability in knowledge utilization and reasoning. However, when applied as agents in embodied environments, they often suffer from misalignment between their intrinsic knowledge and environmental knowledge, leading to infeasible actions. Traditional environment alignment methods, such as supervised learning on expert trajectories and reinforcement learning, encounter limitations in covering environmental knowledge and achieving efficient convergence, respectively. Inspired by human learning, we propose Exploration-based Error Correction Learning (E ^2 CL), a novel framework that leverages exploration-induced errors and environmental feedback to enhance environment alignment for embodied agents. E ^2 CL incorporates teacher-guided and teacher-free explorations to gather environmental feedback and correct erroneous actions. The agent learns to provide feedback and self-correct, thereby enhancing its adaptability to target environments. Extensive experiments in the VirtualHome environment demonstrate that E ^2 CL-trained agents outperform those trained by baseline methods and exhibit superior self-correction capabilities.

## introduction language models (lms) are becoming increasingly capable of knowledge utilization and reasoning across various knowledge-intensive tasks @xcite @xcite . this success motivates researchers to build lm-based agents in embodied environments, which similarly requires the use of reasoning and planning upon environmental knowledge @xcite . in this case, lm-based agents are asked to plan appropriate actions based on the given environmental information and the history of actions already taken. however, the knowledge acquired by these lm-based agents comes *equal contribution. from general-purpose corpora during pre-training, and as a result the intrinsic knowledge of these models often misalign with environmental knowledge. such environmental knowledge involves physical constraints that lms have not yet explored. for example, if the embodied agent holds two objects, it is prohibited to grab one more other object. this misalignment causes lm-based agents to frequently produce actions that cannot be executed in the environment, hindering their applications in real-world environments. to address the above issue, two primary types of environment alignment methods have been explored. the first type involves having lm-based agents undergo supervised learning on expert trajectories @xcite , which are human-labeled sequences of observations and actions. nevertheless, these trajectories often fail to fully cover the knowledge within the environment, such as scenarios where certain actions cannot be executed. the second type is based on reinforcement learning (rl), which allows agents to freely explore the environment, collect trajectories that comprehensively cover the environment's knowledge, and obtain rewards based on these trajectories' success or failure @xcite . however, the rewards are sparsely obtained because the performance evaluation of the agent is based on a complete trajectory. this makes the learning process difficult to converge. human learning is not comprehensive nor efficient if it relies solely on imitating experts' behavior or merely knowing whether an action is correct. instead, by collecting and understanding feedback from the environment via exploration and learning to correct errors based on the feedback, humans can learn comprehensively and efficiently. inspired by this, we propose a novel exploration framework for lm-based agents to align with environments, which is called exploration-based error correction learning (e 2 cl). as depicted in figure 1 , our framework incorporates exploration-induced errors and environmental feedback, leading to a comprehensive alignment with target environments. in detail, we adopt a pretrained model as the agent to perform predefined tasks and explore the environment to collect experiences in both efficient and comprehensive manners. this is achieved by two different proposed schemes, namely teacherguided exploration and teacher-free exploration. the former prompts the agent to perform onestep exploration given sliced expert trajectories, whereas the latter allows the agent to continue exploring until it infers a stop. in these two exploration phases, we collect the feedback given by the environment when the agent makes errors, as well as the correct actions corresponding to these error actions. having these exploration trajectories with additional correction, we train the agent to provide feedback on their trajectories and correct their error actions based on the feedback. to utilize the learned self-correction ability, we further propose a speculative inference algorithm, which performs corrections if the initially planned actions are inferred to be errors according to the feedback from the agent. we evaluate the agent trained by e 2 cl in vir-tualhome @xcite , a household embodied environment. e 2 cl-trained agent surpasses the agents trained by other baseline methods in all agentic metrics, demonstrating its effectiveness. furthermore, our analysis reveals that the small models constructed using our method outperform larger models of the same series that have only undergone behavior cloning. in addition, in evaluations based on feedback-driven re-planning, our models demonstrate self-correction capabilities that are comparable to llms. in summary, our main contributions are as follows. ( 1 ) we introduce the exploration-based error correction learning (e 2 cl) framework, enabling lm-based agents to align with environments through effective feedback-driven exploration and correction. (2) we propose two novel exploration schemes, teacher-guided and teacher-free explorations, which facilitate the collection of correction and feedback via agent-environment interactions. (3) we introduce a novel action inference algorithm called speculative inference, which effectively avoids executable errors. (4) we demonstrate the superior performance of e 2 cl-trained agents in the virtualhome environment, surpassing baseline methods and showcasing the potential of our approach for real-world deployment. ## method in this section, we propose an exploration-based error correction learning (e 2 cl) framework. this framework focuses on equipping lm-based agents with self-feedback and self-correction capabilities. the overview of e 2 cl is depicted in figure 2 . ## conclusion in this work, we aim to align the embodied agent with the environment to enhance its task-solving performance. firstly, we present e foot_1 cl, a novel framework that leverages exploration-induced errors and environmental feedback to enhance environment alignment for lm-based agents during teacher-guided and teacher-free exploration. furthermore, we introduce speculative inference, a process in which the agent utilizes learned abilities for self-feedback and self-correction to reduce execution errors. extensive experiments show that our method outperforms many existing baseline methods.