A uto P lan: Automatic Planning of Interactive Decision-Making Tasks With Large Language Models (2023)

Abstract: Recent large language models (LLMs) are promising for making decisions in grounded environments. However, LLMs frequently fail in complex decision-making tasks due to the misalignment between the pre-trained knowledge in LLMs and the actual rules in the environment. Existing methods require either costly gradient computation or lengthy in-context demonstrations. In this paper, we propose AutoPlan, an approach to guide LLM-based agents to accomplish interactive decision-making tasks. AutoPlan augments the LLM prompt with a task-solving plan and optimizes it through iterative experience collection and reflection. Our experiments show that AutoPlan, though using no in-context demonstrations, achieves success rates on par with the baselines using human-written demonstrations on ALFWorld and even outperforms them by 8% on HotpotQA. The code is available at https://github.com/owaski/AutoPlan.

## introduction the ability to make decisions lies at the core of human intelligence, enabling us to navigate through a multitude of choices and select the best possible actions based on available information. recent large language models, trained with trillions of tokens, have gained impressive reasoning ability and now have the potential to act as autonomous agents for decision-making tasks in grounded environments @xcite @xcite . decision-making tasks in grounded environments can be as simple as calculating mathematical problems with an external calculator or as complex as doing housework. current llm can easily use an external calculator by decomposing the formula into atomic function calls @xcite . however, llms frequently fail in more complex tasks in an environment with many objects and prerequisite dependencies. considering the heat task figure 1 : problem illustration: planning for decisionmaking tasks. given the description of an environment, legit actions, and a task instance, the goal is to guide an llm to generate a sequence of thoughts and actions (highlighted in yellow) reacting to observations provided by the environment (highlighted in blue) to solve the task instance. an llm may fail in complex tasks due to the lack of prior knowledge. in alfworld @xcite ), llm agents struggle to find the correct action sequence within the maximum number of actions (figure 1 ). the primary reason behind such failures is the misalignment between llm's pre-trained knowledge (e.g., generating fluent sentences) and the concrete rule of the grounded environment (e.g., household item functionality in alfworld). in the alfworld environment, the agent can only heat an object with a microwave instead of a toaster. however, the llm does not learn such knowledge during pretraining, eventually failing the task. existing methods aligning llms to desired environments either employ reinforcement learning (rl) and imitation learning (il) methods @xcite , or provide a few demonstrations to conduct in-context learning (icl) @xcite . on the one hand, rl and il methods require computationally costly gradient computation for existing llms. on the other hand, the performance of icl methods highly depends on the selection of demonstrations. in this work, we propose autoplan, a purely 1. go to the location of target object 2. take the object from the receptacle 3. go to the microwave 4. heat the target object with the microwave 5. go to the target receptacle 6. place the heated object in/on the target receptacle task plan 𝒳 !"# experience collection llm prompt wrong correct 1. go to the location of target object 2. take the object from the receptacle 3. go to the toaster 4. heat the target object with the toaster 5. go to the target receptacle 6. place the heated object in/on the target receptacle reflection observation task description: you need to interact with a simulated household to solve a job by interleaving think[], action[] and observation[]. you can do following actions: 1) go to receptacle 2) take object from receptacle 3) put object in/on receptacle 4) heat object with receptacle task instance: put a hot bowl in sidetable task plan: 𝓧𝒊 task starts. you are in the middle of a room. looking around you, you see a countertop 1, microwave 1, toaster 1, side ## autoplan in this section, we describe autoplan in detail. we first describe the general procedure of using llm to solve an interactive decision-making task. then we present autoplan that solves the task by a textbased plan, obtained by an iterative three-stage process: autoplan 1) collects interaction experiences using the task plan at the current step, 2) reflects on the collected experiences, and 3) updates the plan. ## experiment we aim to answer the following questions: 1) does autoplan improve upon baselines? 2) is autoplan efficient during inference? 3) does batching stabilize the optimization? 4) does trio reflection ensure steady progression? 4.1 data alfworld is a text-based game enabling agents to navigate and interact with a simulated household to accomplish six types of tasks. each task instance comes with a high-level objective (e.g., put a hot tomato on the desk), and the agent can achieve the objective through low-level actions described in text (e.g., heat tomato 1 with microwave 2, go to desk 1). since the environment feedback of invalid actions provided in the original alfworld is too primitive, we manually augment the feedback ( we randomly sample 24 task instances for each type of task from the training games to optimize the task-specific plan and, following prior works @xcite , use 134 unseen validation games foot_0 to evaluate our method. alfworld evaluates the success/failure of a task instance by checking if the agent is in the goal state (e.g. if the hot mug is already on the desk). wikipedia pages. as in @xcite , the llm agent is required to answer questions by interacting with a wikipedia api. the api supports three types of actions: (1) search[entity]: returns the first five sentences from the wikipedia page of the entity if it exists or suggests top-5 similar entities 2 . we randomly sample 50 hard (question, answer, supporting facts) triples from the official training set to optimize the plan and sample 200 questions from the official development set as the test set 3 . the final answer is evaluated by three external human annotators rather than exact-match (em) since the answer provided by the agent and the gold answer can differ drastically in form but share the same meaning. we include the complete annotation instruction in the appendix b.2 and take the majority vote of 3 annotators as the final evaluation result. the agreement rate (all three annotators agree with each other) is above 90% for all considered models. 2 we notice that this api retrieves the latest information instead of the wikipedia dump (2017-10-01) used to build hotpotqa dataset, so we modify it to return the historical page of entities before 2017-10-01. ## conclusion we propose autoplan, a prompt-based method, to enable llm to solve interactive decision-making tasks without gradient computation or in-context demonstrations. autoplan conditions llm on an additional task plan described in natural language, which is obtained through an iterative three-stage process. experiments show that autoplan achieves better results than baselines and is also efficient during inference. the ablation study further confirms the effectiveness of batching and explicit reflection original plan summary only new plan new plan full reflection summary: found bread 1 on countertop 1. take it and go to the microwave 1. found an egg inside. place the bread inside, close the microwave. but failed to heat the bread.