EPO : Hierarchical LLM Agents with Environment Preference Optimization (2024)

Abstract: Long-horizon decision-making tasks present significant challenges for LLM-based agents due to the need for extensive planning over multiple steps. In this paper, we propose a hierarchical framework that decomposes complex tasks into manageable subgoals, utilizing separate LLMs for subgoal prediction and low-level action generation. To address the challenge of creating training signals for unannotated datasets, we develop a reward model that leverages multimodal environment feedback to automatically generate reward signals. We introduce Environment Preference Optimization (EPO), a novel method that generates preference signals from the environment’s feedback and uses them to train LLM-based agents. Extensive experiments on ALFRED demonstrate the state-of-the-art performance of our framework, achieving first place on the ALFRED public leaderboard and showcasing its potential to improve long-horizon decision-making in diverse environments.

## introduction long-horizon decision-making/planning remains a formidable challenge for large language model(llm)-based agents @xcite @xcite . these tasks require extensive planning over multiple steps, maintaining coherence and goal orientation, which is difficult for llms that are typically designed for more immediate and localized predictions. moreover, a key issue of finetuning llms for embodied agents is the need of large scale labeled data @xcite . the same issue is reflected in researchers' effort in building reward models from vision foundation models as we might need to obtain "internet-scale" data of task demonstrations @xcite . to tackle the first challenge, a straightforward way is to first let the llm decompose the longhorizon task into shorter horizon subtasks, and then use different llms as the policies at different levels, i.e., use one llm-based policy to generate subgoals, and use another llm generate low-level actions given the subgoals, both of which require significantly fewer planning steps. this decomposition facilitates more effective planning and execution by leveraging the predictive power of llms at both the subgoal and action levels. however, the problem of how to efficiently train these llm-based agents remains. in this paper, we consider the setting where only part of the dataset are annotated with ground-truth actions and subgoals, and we need to find a way to create training signals for the unannotated dataset. the common training signals for decision-making agents are based on the rewards received during interactions with the environment @xcite . but the manual design of reward functions is both time-consuming and prone to inaccuracies, which hinders the scalability and adaptability of llm-based agents in dynamic and diverse environments. consequently, there is a growing need for methods that can automatically generate reward signals from the environment, thus bypassing the complexities associated with human-engineered rewards. this motivation drives us to explore reward modeling approaches that can leverage multimodal feedback from the environment, such as visual and interaction data, to guide the learning process of llm-based agents by leveraging the public pretrained foundation models. on the other hand, recent advancements in preference optimization techniques, such as direct preference optimization (dpo) @xcite , have shown that llms can be effectively trained using preference-based signals rather than explicit reward functions. dpo leverages the inherent capabilities of llms to model preferences between different outputs, facilitating a more intuitive and flexible training paradigm. this insight inspires us to develop a novel method that combines the strengths of preference optimization with automatic reward modeling to enhance the performance of llm-based agents in long-horizon decision-making tasks. in this paper, we propose a hierarchical llmsbased framework for long-horizon decision making problems. our agent decomposes complex tasks into manageable subtasks by training two llms to predict the subgoal decomposition and low-level actions respectively. to retrieve enough training signals from the unannotated dataset, we propose a llm-based reward model that is able to integrate the multimodal environment feedback information and automatically generate reward signals for the unannotated dataset. then, we introduce environment preference optimization (epo), a method that generates preference signals automatically from the environment's feedback. epo ranks the proposed actions and subgoals based on the estimated rewards and constructs a preference dataset that guides the training of llm-based agents. this approach leverages both annotated and unannotated datasets, significantly expanding the training data available for improving agent performance. to validate our framework design, we conduct extensive experiments on alfred @xcite , a popular household simulation environment for embodied agents. our method achieves the state-of-the-art performance on alfred. we also find that unified environment feedback significantly help decision-making agents in both subgoal decomposition level and environment interaction level. moreover, in the setup where there exists a large dataset of task specifications but only a small annotated task and demonstrations, our framework allows agent to benefit from the unannotated new tasks while significantly outperforming supervised training, indicating the potential of our framework. to sum up, we make the following contributions: reward modeling with foundational models. foundation models with their capability in encoding generic representations of a modality have motivated researchers to use them to generate reward signals in order to bypass human reward engineering. among these efforts, sontakke et al. ( 2023 use language models to generate rewards to help robot learn skills based on the symbolic states. for embodied agents, ellm @xcite propose a framework to use llms to guide agents' exploration and generate reward based on the task goals in 2d games and robotic simulators. compared to existing works, we fill in the blank by proposing a generic framework that use llms to synthesize reward from multimodal environment feedback. preference-based learning for language models. aligning language models to human preference @xcite has greatly improved language models to follow human instructions. recent development such as direct preference optimization @xcite , self-rewarding language models @xcite in preference alignment allows the language model to directly learn the preference relation and also learn from its own synthesized data. inspired by these work, we extend the definition of "preference" into the alignment between environment feedback and agent actions with respect to the task specification. we leverage the algorithmic advantage demonstrated in dpo and the idea of self data synthesis @xcite to train llm-based embodied agents to ground language to environment feedback. ## method we first describe the problem setup in 3.1 and then introduce our hierarchical llms-based decisionmaking agent in 3.2. then we present our approaches for generating reward signals from multimodal environment feedback in 3.3. lastly, we explain how we train the hierarchical agents with environment preference optimization in 3.4. ## results in this section, we first compare the overall performance of our framework with the state-of-the-art methods on alfred public leaderboard and then modularly study the components of our framework. we obtain all the results following the standard setting in alfred where we first let the agent learn from the given dataset offline, and then test the the online rollout performance of the learned policies (modules) on the given set of new test tasks. ## conclusion in this paper, we presented a hierarchical llmbased framework for long-horizon decision-making tasks, addressing the inherent challenges of extensive planning and the need for scalable training signals. by leveraging a reward model that integrates multimodal environment feedback, and introducing environment preference optimization (epo), we successfully generated training signals for unannotated datasets. our framework demonstrated stateof-the-art performance on the alfred benchmark. future work will focus on exploring the integration of additional types of multimodal feedback to further enhance the agent's decision-making capabilities, as well as extending our framework to real world robotics tasks.