Reverse Chain: A Generic-Rule for LLM s to Master Multi- API Planning (2024)

Abstract: While enabling large language models to implement function calling (known as APIs) can greatly enhance the performance of Large Language Models (LLMs), function calling is still a challenging task due to the complicated relations between different APIs, especially in a context-learning setting without fine-tuning. This paper introduces “Reverse Chain”, a controllable, target-driven approach designed to empower LLMs with the capability to operate external APIs only via prompts. Recognizing that most LLMs have limited tool-use capabilities, Reverse Chain limits LLMs to executing simple tasks, e.g., API Selection and Argument Completion. Furthermore, to manage a controllable multi-function calling, Reverse Chain adopts a generic rule-based on a backward reasoning process. This rule determines when to do API selection or Argument completion. To evaluate the multi-tool-use capability of LLMs, we have released a compositional multi-tool task dataset, available at https://github.com/zhangyingerjelly/reverse-chain. Extensive numerical experiments validate the remarkable proficiency of Reverse Chain in managing multiple API calls.

## introduction recently, there has been an impressive wave in the progress made in large language models (llms), due to their excellent performance in a variety of tasks @xcite @xcite @xcite . however, llms still face difficulties with some specialized tasks due to their fundamental limitation on the information they stored and learned, which can become outdated and may not be suitable for all applications. a practical solution is to augment llms with external tools (known * equal contributions † this author is the corresponding author as apis). in this setup, llms act as controllers, not only to understand user intents but crucially to select and orchestrate the appropriate tools to complete tasks. unfortunately, llms still lack the sophistication to fully understand human instructions and effectively implement function calling. many works are dedicated to enhancing the function calling abilities of llms through fine-tuning or in-context learning methods @xcite @xcite @xcite @xcite @xcite . compared to fine-tuning, in-context learning approaches offer a more straightforward and scalable solution, as they eliminate the need to train an entirely new model for each new api. consequently, the primary goal of this paper is to enhance the api planning capabilities of llms within the in-context learning setting. different from the aforementioned studies which focus on simpler tasks, such as single-tool task or independent multi-tool task (detailed in in the realm of tool-use, various prompting techniques have been explored. one-step planning algorithms are introduced in @xcite , but its accuracy is often low in complex, ambiguous scenarios. the chain of thought (cot) approach @xcite counters this by task type example api planning single-tool what's the weather in new york ? getwearther(@xmath0, could you find a flight and book it to my destination ? query goal api api api api api api api api api api api api wrong goal (b) reverse chain (a) planning (c) react api api call in path execution path potential path reverse chain of thought fail to enter the next step step-by-step planning with intermediate reasoning. known as cot planning, this technique decomposes tasks into several simpler sub-tasks, thereby boosting reasoning and accuracy. nevertheless, as illustrated in figure 1 (a), a limitation of these planning methods is their potential for errors in the intermediate stages. while the final step of the plan is intended to achieve the ultimate goal, errors in the intermediate planning steps can lead to execution failures. for instance, as illustrated in the compositional multi-tool case of in summary, both one-step/cot planning and react are forward reasoning solutions, so they encounter similar control challenges: each step exhibits a high level of unpredictability and uncertainty, especially at the beginning when the search space is large. errors can propagate from a wrong thought or action, leading to incorrect solution paths or final goals. this issue arises because these methods start from scratch and progress forward towards the final target, with the llm bearing the entire burden of planning. to address these issues, we propose a controllable yet general framework called reverse chain. this framework consists of a generic rule and two key modules: api selection and argument completion, both centered on prompting an llm. specifically, the generic rule in reverse chain performs a multi-api planning task in a backward manner: it starts by selecting the final api for a task, and then completes the required arguments, drawing values from the query and context, or by outputs of other apis. when a new api is selected during the argument completion stage, this process repeats. the procedure continues iteratively until all arguments of all apis are filled. reverse chain distinguishes itself from previous work with the following three main advantages: 1. backward reasoning, starting from the final goal, preventing planning from deviating into a wrong direction, thus ensuring the correctness of the final goal. 2. the step-by-step decomposition dominated by the rule makes the process controllable, with each stage being forward-executable, effectively avoiding errors such as incorrect intermediate stage. 3. the tasks of llms are simplified to just selecting apis and filling arguments, avoiding complex planning. this strategy effectively utilizes the strengths and capabilities of the existing llms without depending on extensive reasoning abilities. in summary, the contributions of this paper are: tool learning the discussion of tool usage in llms has grown significantly, with models like toolformer leading the way @xcite . current approaches can be divided into two categories. the first category focuses on enhancing the tool-specific capabilities of language models through fine-tuning with specialized datasets @xcite @xcite @xcite @xcite . the second category directly leverages the capabilities of llms, prompting them to interact with various tools, ranging from ai models @xcite to more versatile tool sets @xcite @xcite . generally, the prompting approach is simpler and more scalable, but it still has a significant gap compared to fine-tuning method, so this work is proposed to enhance the api planning capability of prompting methods. it is notable that while the previously mentioned studies introduced numerous tool-learning datasets, they primarily encompass relatively simple tasks, focusing on single-tool task or independent multi-tool task. in contrast, this paper targets a more complex task called compositional task, where multiple dependent apis are needed. prompting llms various methods, like cot @xcite for task decomposition and react @xcite for melding reasoning with action, enhance general prompting capabilities. additionally, numerous planning methods are tailored for tool-use. @xcite start by generating a direct solution outline, followed by selecting and executing relevant apis. dfsdt @xcite can be seen as an improved version of react, enables llms to evaluate different reasoning paths and select the most promising one. restgpt's @xcite workflow involves an iterative "plan and execute" cycle. meanwhile, @xcite employs a sequential planning approach, feeding the outcome of each step into the subsequent one. all these works require an llm to perform either full or step-by-step planning based on the task. however, the reverse chain proposed in this work simplifies this by having the llm focus on just two tasks: api selection and argument completion, thereby greatly simplifying the task complexity. furthermore, unlike previous methods that progress from scratch to the final goal, reverse chain starts from the end goal and reasons backwards, enhancing controllability. ## reverse chain: a multi-api planning approach the objective of this work is to generate effective api planning based on user queries and api candidates. figure 2 provides a detailed example: a user query could be a natural language request like "please help jack book a meeting room from 9:00 am to 10:00 am". each api in the api pool is characterized by its description, arguments, and output. e.g., the api recommendroom has a functionality description of "recommend the id of an available meeting room", arguments "start_time" and "end_time", and an output of "room_id". a successful api planning consists of two parts: selecting the proper api and filling in all the argu- api description arguments output name2id convert user name to user id person_name person_id recommendroom recommend the id of an available meeting room start_time, end_time room_id bookroom book a meeting room person_id, room_id start_time, end_time room_info api bookroom person_id room_id start_time end_time api recommendroom api name2id api selection argument completion person_name value: jack start_time end_time value: 9am value: 10am value: 9am value: 10am 1213 005 reverse chain of thought forward execution filled argument unfilled argument ⍉ ⍉ ⍉ argument completion reverse chain bookroom (person @xmath4, room @xmath5, end @xmath6, start @xmath7, end @xmath8, where the argument values can come from the query or context, or from the output of another api. section 3.1 outlines the reverse chain process, while section 3.2 specifically discusses the two modules that interact with llm: api selection and argument completion. ## experiments in this section, extensive experiments are conducted to investigate the performance of reverse chain. we start with generating an evaluation dataset automatically, benchmarking different incontext learning methods on function calling and defining the evaluation metrics. in section 4.1, to benchmark reverse chain, we compare its api planning capabilities with the current state-of-theart in-context learning solutions on chatgpt. section 4.2, details a set of ablation experiments designed to elucidate the underlying principles of reverse chain. finally, section 4.3 analyzes the factors contributing to the effectiveness of reverse chain. dataset we construct a dataset for evaluating compositional multi-tool tasks. guided by the selfinstruct paradigm @xcite , this dataset is generated automatically based on gpt-4 and chatgpt (gpt-3.5-turbo), involving the following steps: ## conclusion this paper proposed reverse chain, a concise, target-driven approach developed to empower llms with the capability to interact with external apis in an in-context learning setting. by implementing a backward reasoning strategy and a generic rule, reverse chain effectively broke down complex function-calling challenges into two fundamental tasks for llms: api selection and argument completion. additionally, we collected a compositional multi-tool dataset for evaluation. extensive experiments revealed that reverse chain markably enhances the tool-use proficiency of the existing llm chatgpt, achieving superior performance compared to methods like cot and react. although the current work concentrates on compositional multi-tool tasks, it can also be easily extended to other types of tasks. for instance, in the case of independent multi-tool tasks, after identifying sub-intents at the beginning of the task (known as intent detection, a well-established problem in nlp with numerous robust solutions), we could employ the reverse chain process for each identified sub-task separately.