TPTU -v2: Boosting Task Planning and Tool Usage of Large Language Model-based Agents in Real-world Industry Systems (2024)

Abstract: Large Language Models (LLMs) have demonstrated proficiency in addressing tasks that necessitate a combination of task planning and the usage of external tools, such as weather and calculator APIs. However, real-world industrial systems present prevalent challenges in task planning and tool usage: numerous APIs in the real system make it intricate to invoke the appropriate one, while the inherent limitations of LLMs pose challenges in orchestrating an accurate sub-task sequence and API-calling order. This paper introduces a comprehensive framework aimed at enhancing the Task Planning and Tool Usage (TPTU) abilities of LLM-based agents in industry. Our framework comprises three key components designed to address these challenges: (1) the API Retriever selects the most pertinent APIs among the extensive API set; (2) the Demo Selector retrieves task-level demonstrations, which is further used for in-context learning to aid LLMs in accurately decomposing subtasks and effectively invoking hard-to-distinguish APIs; (3) LLM Finetuner tunes a base LLM to enhance its capability for task planning and API calling. We validate our methods using a real-world industry system and an open-sourced academic dataset, demonstrating the efficacy of each individual component as well as the integrated framework. The code is available at here.

## introduction large language models (llms) have exhibited remarkable prowess in various domains of natural language processing (nlp) @xcite @xcite , encompassing language understanding @xcite , reasoning (wei et al., 2022; † these authors contribute equally to this work. ‡ these authors work as interns at sensetime research. the corresponding authors: hy.mao@pku.edu.cn; zlibn@wiso.uni-koeln.de @xcite , and program synthesis @xcite . however, leveraging llms for complex tasks presents formidable challenges. on the one hand, llms inherently exhibit limitations in their capabilities, struggling with logical problem-solving, such as mathematics, and facing the risk of stored knowledge quickly becoming outdated as the world evolves. instructing llms to utilize external tools including calculators, calendars, or search engines can prevent them from generating inaccurate information and aid them in effectively addressing problems. on the other hand, integrating these tools into complex systems transcends mere task understanding. it demands the ability to break down intricate tasks, manipulate various tools, and engage with users in effective interactions. several research endeavors, known as llm-based ai agents @xcite @xcite , such as autogpt @xcite , @xcite , and @xcite , have made advancements by employing llms as central controllers. these endeavors automatically decompose user queries into sub-tasks, execute low-level tool (i.e., api) calls for these sub-tasks, and ultimately resolve the overarching problem. despite these advances, llm-based agents still grapple with pressing challenges in real-world industry applications. firstly, real-world systems usually have a vast number of apis, making it impractical to input descriptions of all apis into the prompt of llms due to the token length limitations. secondly, the real system is designed for handling complex tasks, and the base llms (i.e., general llms without finetuning on these tasks) often struggle to correctly plan sub-task orders and api-calling sequences for such tasks. thirdly, beyond the sheer quantity of apis, a more substantial challenge is that real systems are primarily designed around a core purpose, resulting in the fact that certain apis may overlap and exhibit similar semantics and functionality, while different in usage. for instance, variations in required input parameters create difficulties in distinguishing and invoking these apis for llms and even humans. how to address these issues could be the critical step for llm-based agents towards omniscience and omnipotence and being implemented in realworld industry scenarios. in this paper, we propose a framework to improve the task planning and tool usage (tptu) abilities of llm-based agents in the real-world systems, which is composed of three key components to address the above three challenges: (1) api retriever that is based on contrastive learning, to accurately recall the apis that are most relevant to the user's task from all apis. the descriptions of these filtered apis can then be input into llm as prompts, allowing the llm to understand and make accurate choices within the filtered api set. (2) demo selector that is based on our selfestablished knowledge base using the logs when performing tasks, it adaptively retrieves different demonstrations related to hard-to-distinguish apis, which is further used for in-context learning so that llm can distinguish the subtle differences in the functions and usages of different apis. (3) llm finetuner tunes a base llm with a dataset of three essential criteria, i.e., real-world, structured prompt, and diversity, so that the finetuned llm can be more capable of task planning and api calls, especially for domain-specific tasks. our main contributions can be summarized as follows: ## methodology in response to the typical challenges of deploying llms within intricate real-world systems, we propose a comprehensive framework that fundamentally bolsters the capabilities of llms in task planning and tool usage (tptu), which are the cornerstone of the agent's abilities. in this paper, task planning entails generating a step-by-step subtask sequences for the complex task, while tool usage requires the agent to select appropriate apis and execute them correctly to obtain the answer. ## experiments we present comprehensive experiments to rigorously evaluate the efficacy of our proposed framework. experiments are structured to assess the framework's performance in both a real-world scenario and an open-source academic challenge to analysis our framework's generalization. ## conclusion in this paper, we present a comprehensive framework to augment the capabilities of llms in realworld scenarios, with a specific focus on task planning and tool usage (tptu). our approach, which integrates api retriever, llm finetuner, and demo selector, has been validated in both a real-world scenario and an open-source setting. the results demonstrate that fine-tuning llms with a curated dataset can significantly improve their effectiveness in executing real-world tasks. the api retriever and demo selector also prove indispensable, particularly in improving the model's decision-making accuracy and adaptability. this research not only highlights the potential of llms in practical applications but also establishes a foundation for future advancements in this field.