LL a MA -Rider: Spurring Large Language Models to Explore the Open World (2024)

Abstract: Recently, various studies have leveraged Large Language Models (LLMs) to help decision-making and planning in environments and try to align the LLMs’ knowledge with the world conditions. Nonetheless, the capacity of LLMs to continuously acquire environmental knowledge and adapt in an open world remains uncertain. In this paper, we propose an approach to spur LLMs to explore the open world, gather experiences, and learn to improve their task-solving capabilities. In this approach, a multi-round feedback-revision mechanism is utilized to encourage LLMs to actively select appropriate revision actions guided by feedback information from the environment. This facilitates exploration and enhances the model’s performance. Besides, we integrate sub-task relabeling to assist LLMs in maintaining consistency in sub-task planning and help the model learn the combinatorial nature between tasks, enabling it to complete a wider range of tasks through training based on the acquired exploration experiences. By evaluation in Minecraft, an open-ended sandbox world, we demonstrate that our approach LLaMA-Rider enhances the efficiency of the LLM in exploring the environment, and effectively improves the LLM’s ability to accomplish more tasks through fine-tuning with merely 1.3k instances of collected data, showing minimal training costs compared to the baseline using reinforcement learning. The code is available at https://github.com/PKU-RL/LLaMA-Rider.

## introduction recently, significant advancements and successes have been achieved in the performance of large language models (llms) in attaining human-like intelligence @xcite . given the powerful capability of llms, many research works have started utilizing their abilities to assist intelligent † corresponding author. agents in decision-making @xcite @xcite , and have found that llms possess a certain level of abilities for planning and accomplishing various tasks @xcite . however, the knowledge that llms rely on comes from the language corpus used during pre-training, and there may be discrepancies between this knowledge and specific environments @xcite . to ground llms to environments, some studies design specific mechanisms through prompt engineering to provide information from environments for llms @xcite @xcite @xcite . however, llms do not improve or acquire new knowledge in environments. additionally, for more complex tasks, more complicated mechanisms and prompts are required, which results in high costs of llm generation and reliance on strong models like @xcite with enough knowledge @xcite . some other studies ground llms with finetuning @xcite @xcite , but they usually require task-dependent datasets. reinforcement learning (rl) methods are also studied in the literature @xcite , but these methods train llms as task-specific policies, and we found that rl methods are difficult to scale up to larger models or more complex tasks (see section 5.2.2). in this paper, we aim to enhance llms through their exploration in open-ended environments (figure 1 ), like humans can adapt to new situations through practice. previous studies have tried to update llms in embodied environments like babyai @xcite and virtualhome @xcite , but these world sizes are rather limited. whether llms can improve their knowledge in more complicated open-ended worlds like minecraft is still unknown @xcite . we think there are two major challenges here. first, in an environment like minecraft, tasks are often complex and may involve many subtasks. at the same time, these long-horizon tasks often require each step to be carried out precisely, and a single error in the middle sometimes can negate previous progress. besides, due to the high level of freedom, the action space can be large, while many actions may be invalid in different states. these reasons make it hard to collect successful task trajectories in the environment using random exploration as in previous works @xcite . the second challenge is that there can be a significant amount of tasks in such an open world, so training policies for specific tasks are not applicable in these environments. we hope that llms have the ability to perform multiple tasks and generalize to new tasks. in response to these challenges, we propose llama-rider, a two-stage learning framework consisting of an exploration stage and a learning stage (figure 2 ). we investigate how to spur llms to explore the environment themselves and collect experiences for learning. compared to random exploration or search methods that can hardly work in complex environments, allowing llms to explore on their own can harness the inherent capabilities of the models, thereby enabling more effective discovery of successful experiences. we propose a multi-round feedback mechanism, which allows the llm to revise its decisions with information about failed actions. this feedback-revision exploration mechanism is more efficient due to the capability of llms, as the draft decisions made are often related to task completion at first, and llms can understand feedback information effectively. additionally, we use sub-task relabeling to help llms maintain consistency in sub-task planning. in the learning stage, we process the collected experiences into datasets and use supervised finetuning (sft) to train the llm. in addition to the experience gained from successful tasks, we also collect experiences from partially completed subtasks, as some tasks are too difficult to accomplish in the exploration stage. numerous tasks in openended environments often have compositionality, which means experiences from past tasks can frequently assist in completing other tasks. we propose to use sub-task relabeling of the collected experiences to improve data utilization while helping llms learn the compositionality between tasks. we evaluate our method in minedojo @xcite , a simulation platform for minecraft. we use the basic skills trained by plan4mc @xcite as the action space since the skills possess more semantics compared with primitive actions and are better aligned with llms. our experiments show that llama-rider can explore the environment efficiently with our feedback-revision mechanism, and can learn to complete tasks more effectively by finetuning on a collected dataset of only 1.3k skill execution steps in size, demonstrating much higher sample efficiency compared to rl methods. we also show the generalization ability of llama-rider in novel hard tasks. ## methodology our method is illustrated in figure 2 , which is a two-stage framework. we introduce the exploration and learning stage respectively in the following. ## conclusion in this paper, we introduce llama-rider, a learning framework that spurs the llm to explore the open world with a feedback-revision mechanism and then use the collected experiences to update itself for task planning. we also propose to use subtask relabeling for long-horizon tasks. experiments in the open world minecraft show the effectiveness and efficiency of our method which helps the llm to adapt to the embodied environment and improve the capability to solve multiple tasks. we also find that llama-rider can use past experiences to solve novel hard tasks, showing a life-long exploration and learning potential.