Learning to Plan by Updating Natural Language (2024)

Abstract: Large Language Models (LLMs) have shown remarkable performance in various basic natural language tasks. For completing the complex task, we still need a plan for the task to guide LLMs to generate the specific solutions step by step. LLMs can directly generate task plans, but these plans may still contain factual errors or are incomplete. A high-quality task plan contains correct step-by-step solutions for solving all situations and behavioral instructions for avoiding mistakes. To obtain it, we propose the Learning to Plan method, which involves two phases: (1) In the first learning task plan phase, it iteratively updates the task plan with new step-by-step solutions and behavioral instructions, which are obtained by prompting LLMs to derive from training error feedback. (2) In the subsequent test phase, the LLM uses the learned task plan to guide the inference of LLM on the test set. We demonstrate the effectiveness of our method on the five different reasoning type tasks (8 datasets). Further, our analysis experiment shows that the task plan learned by one LLM can directly guide another LLM to improve its performance, which reveals a new transfer learning paradigm.

## introduction large language models (llms), such as chat-gpt and @xcite , have recently achieved strong zero-shot/few-shot performance on various natural language tasks, such as generating passages @xcite , generating code @xcite , and solving grade school math problems @xcite . llms can further learn new basic abilities by connecting them with millions of apis like taskmatrix.ai @xcite or new tools like tool-former @xcite . however, llms still struggle to complete complex tasks, such as writing a long novel @xcite , coding for a large project @xcite , and solving complex math problems @xcite . this indicates that task instruction (prompt): calculating the sin/cos value of an angle based on the length of legs and other conditions in a triangle. input: in a triangle abc, ac is 8 units, and bc is 15 units, what is @xmath0: 15 zero-shot chain of thought: let's think step by step. corresponding solution of the input: apply the cosine rule: cos( ̸ abc) = 8 2 +15 2 -15 2 2•8•15 = 64 240 simplify: cos( ̸ abc) = 4 15 task plan learned by our method: there are two solutions for finding the value of an angle in a triangle. solution 1 involves using the law of cosines to calculate cos( ̸ abc). solution 2 involves using the pythagorean theorem to find the length of the third side of the triangle and then using the definition of sine to calculate sin( ̸ abc). to find the length of the hypotenuse of a right triangle using the pythagorean theorem, use the formula (ac) 2 +(bc) 2 = (ab) 2 . python functions can also be used to calculate these values. corresponding solution of the input: using the pythagorean theorem, we can have @xmath1. then, based on the definition of cosines, we can have @xmath2: we list the typical elements of a task. specifically, we first collect the wrong samples that can not be solved with the guidance of the current task plan. this step is called the finding wrong samples with current task plan step. next, in the computing plan update step, we prompt the llm to generate multiple new step-by-step solutions for solving wrong samples and samples similar to them. we choose the solution that achieves the best performance on a validation set as the plan update. text compression is also considered for deduplication. in the updating plan step, we update the task plan by directly adding the best new solution (text) to the original task plan (text). we repeat the learning process until we can not find a new solution to improve the task plan's performance. in the test phase, we add the learned task plan into the prompt to guide the llm. experiments on 8 reasoning datasets with five reasoning types: mathematical reasoning, causal reasoning, logical reasoning, symbolic reasoning task, and combinatorial reasoning show that our method improves the llm's performance of these tasks significantly. for example, our method's average performance of 10 tasks from the amps mathematical dataset @xcite outperforms the performance directly measured in the zero-shot/few-shot chain of thought setting by 18.3%/7% respectively ( the paper's main conclusions are as follows: our method works well for both closed-source and open-source llms. prompt engineering @xcite @xcite further improves the performance of llms by inserting a prompt into the input. discrete prompts consist of natural language phrases. @xcite uses the mining-based methods to extract middle words and words in the dependency path as the potential discrete prompts. prompt-paraphrasing methods @xcite create the potential prompts by paraphrasing the original prompt into a set of other prompts. @xcite uses pre-trained t5 to generate discrete prompts. however, discrete prompts can not update the previous prompt to make the prompt better and do not induce the task information into their method. continuous prompts @xcite @xcite perform prompting directly in the embedding space of the model. their templates have their own parameters that can be tuned in the training dataset. but humans can not directly understand them. task planning methods can be divided into two main approaches: one approach prompts the llm to break the problem down into sub-problems without feedback. chain of thought (cot) prompting and zero-shot cot @xcite ) generate reasoning intermediate processes for complex problems by prompting the model to "think step by step. least-to-most prompting @xcite prompts llms to decouple a complex problem into a list of sub-problems and solve them sequentially. another approach uses feedback to decouple complex tasks step by step. self-refine method and related methods @xcite @xcite and tree of thoughts @xcite improve llms' specific solution iteratively by utilizing the feedback provided by llm itself. react @xcite , progprompt @xcite , and inner monologue @xcite uses environmental feedback to generate specific action plan step by step. ape @xcite and instruction induction @xcite update task instruction by iteratively generating instruction candidates and selecting the one with the best validation performance. ## experiment datasets mathematical reasoning, we select 10 challenging mathematical tasks from the amps dataset @xcite (see ## conclusion large language models (llms) have shown impressive performance in various natural language tasks. but they still need a task plan when processing complex tasks. in this paper, we introduce the learning to plan method, which explicitly learns the task plan based on the error feedback from the training dataset and uses the learned task plan to solve the task. our proposed method's effectiveness has been verified through multiple reasoning tasks, and we conclude that it is a promising approach for improving llms' complex task performance.