LLM -Based Offline Learning for Embodied Agents via Consistency-Guided Reward Ensemble (2024)

Abstract: Employing large language models (LLMs) to enable embodied agents has become popular, yet it presents several limitations in practice. In this work, rather than using LLMs directly as agents, we explore their use as tools for embodied agent learning. Specifically, to train separate agents via offline reinforcement learning (RL), an LLM is used to provide dense reward feedback on individual actions in training datasets. In doing so, we present a consistency-guided reward ensemble framework (CoREN), designed for tackling difficulties in grounding LLM-generated estimates to the target environment domain. The framework employs an adaptive ensemble of spatio-temporally consistent rewards to derive domain-grounded rewards in the training datasets, thus enabling effective offline learning of embodied agents in different environment domains. Experiments with the VirtualHome benchmark demonstrate that CoREN significantly outperforms other offline RL agents, and it also achieves comparable performance to state-of-the-art LLM-based agents with 8B parameters, despite CoREN having only 117M parameters for the agent policy network and using LLMs only for training.

## introduction developing embodied agents capable of understanding user instructions and executing tasks in physical environments represents a crucial milestone in the pursuit of general ai. recent advancements in large language models (llms) have demonstrated their remarkable reasoning capabilities, paving the way for their application in embodied agents @xcite @xcite @xcite . yet, deploying an llm directly as part of an embodied agent presents several inefficiencies, * equally contributed to this work † corresponding author such as the need for sophisticated environmentspecific prompt design, substantial computational resource demands, and inherent model inference latency @xcite . these factors can limit the practical application of llms, particularly in scenarios where embodied agents are required to respond rapidly and efficiently. in the literature of reinforcement learning (rl), data-centric offline learning approaches have been explored @xcite . these offline rl approaches are designed to establish efficient agent structures, necessitating datasets that include wellannotated agent trajectories with reward information. however, the characteristics of instructionfollowing tasks assigned to embodied agents, particularly their long-horizon goal-reaching nature, often conflict with such dense data requirements of offline rl. embodied agents normally can produce trajectories with sparse reward feedback, because their instruction-following tasks are evaluated based on binary outcomes of success or failure, which directly align with the specific goals of the instructions. in offline rl, this sparse reward setting poses significant challenges in achieving effective agent policies @xcite . in this work, we explore llms for offline rl. by employing capable llms as a reward estimator that provides immediate feedback on agent actions, we augment the trajectory dataset with dense reward information. this method, llm-based reward estimation is capable of significantly enhancing the effectiveness of offline rl for embodied agents. to do so, we address the limitations inherent in llm-based reward estimation. a primary challenge arises from the limited interaction with the environment in an offline setting, which complicates the llms' ability to acquire essential environmental knowledge. the offline setting makes it difficult to ensure that the generated rewards are properly grounded in the specific domain of the environment. for instance, without explicit knowledge that a flowerpot is typically stored in a living room in the target environment, an llm might struggle to accurately assign rewards for actions like "go to living room" versus "go to balcony" when tasked with watering plants. while both actions might seem reasonable from a commonsense perspective, the optimal action depends on specific conditions of the target environment that the llm may not have access to in the offline setting. these challenges, unique to the offline context, differentiate our work from previous works on online llm-based reward estimation, where llms can be fine-tuned or prompts can be refined through repeated interaction with environment or human @xcite @xcite . since these interactions are not available in offline settings, improving the llm's insufficient spatial reasoning for accurate reward estimation requires a fundamentally different approach. in response, we present coren, a consistencyguided reward ensemble framework, specifically designed for robust llm-based reward estimation and effective agent offline learning. it adopts a twostaged reward estimation process, as depicted in figure 1 . (i) an llm is first queried to estimate several types of rewards for actions, each considering a distinct spatio-temporal consistency criterion of the llm to have coherent and domain-grounded rewards. (ii) then, these rewards are further orchestrated, being unified into domain-specifically tuned rewards via an alignment process with the sparse rewards of given trajectories. the resulting agent, trained on the unified dense rewards by offline rl, is capable of performing instructionfollowing tasks with high efficiency and minimal latency at deployment. this offline rl scheme, enhanced by llm-based reward estimation, overcomes the limitations faced by the agents that rely on the online exploitation of llms. the contributions of our work can be summarized as follows: (i) addressing a practical yet challenging problem of embodied agent offline learning using llms for the first time; (ii) proposing a twostaged reward estimation algorithm guided by a spatio-temporal consistency ensemble; and (iii) extensive evaluation on the virtualhome benchmark, demonstrating performance comparable to state-ofthe-art llm-based online agents. ## conclusion we presented the reward ensemble framework coren to achieve robust llm-based reward estimation for offline rl, specifically tailored for embodied instruction-following tasks. the framework utilizes a spatio-temporal consistency-guided ensemble method for reward estimation. it generates multiple stepwise rewards on offline trajectories, with each reward focusing on a specific con-sistency related to contextual, structural, or temporal aspects, and then it integrates the multiple rewards into more domain-grounded ones via the sparse reward-aligned ensemble. as this work is the first to adopt llms for offline learning of embodied agents, we hope it can provide valuable insights into the development of llm-driven training acceleration techniques. this is particularly significant for embodied agents involved in longhorizon instruction-following tasks, which are typically constrained by sparse reward signals.