Toolink: Linking Toolkit Creation and Using through Chain-of-Solving on Open-Source Model (2024)

Abstract: Large Language Models (LLMs) have demonstrated remarkable progress in utilizing tools, but their closed-source nature and high inference costs pose limitations on their adaptability, necessitating a valid method that leverages smaller, open-sourced models. In this paper, we introduce Toolink, a comprehensive framework that performs task-solving by first creating a toolkit and then integrating the planning and calling of tools through a chain-of-solving (CoS) approach. We first validate the efficacy of Toolink in harnessing the model’s creativity and CoS ability on ChatGPT. Subsequently, we curate CoS-GPT, a chain-of-solving dataset designed for tool-using, and finetune the LLaMA-7B model. It results in LLaMA-CoS, a powerful open-source model with advanced tool-planning and tool-calling capabilities. Evaluation of diverse tasks from BIG-bench demonstrates its CoS ability matches that of ChatGPT while its performance surpasses the chain-of-thought approach. Further studies highlight the generalization of LLaMA-CoS to unseen tasks and showcase its capability in using toolkits not explicitly tailored for the target task, affirming its robustness in real-world scenarios. All codes and data are released.

## introduction large language models (llms) such as codex @xcite , @xcite , and gpt4 (openai, 2023) have made significant strides in code generation, in-context learning, and logical reasoning. however, they still struggle with precise calculations and accessing current information @xcite @xcite . to address these issues, research has focused on equipping llms with tools such as calculators @xcite @xcite , search engines @xcite ., 1 https://github.com/qiancheng0/toolink toolkit generation update_orientation() multiply() add() subtract_time() navigation arithmetic matrix date chain-of-solving the useful tools are [tool1] and [tool2]. we first to use [tool1] to achieve @xmath0. and then we use the [tool2] to achive @xmath1. 2022; schick et al., 2023), scratch pads @xcite , calendars @xcite , and image retrievers @xcite to enhance their capabilities, thus benefiting various tasks including question-answering, math calculations, and long-form generation. recent studies have also explored how llms can devise plans, make decisions, and perform tool invocations @xcite @xcite . by combining them into a pipeline, these frameworks aim to construct more advanced nlp systems for improved task performance. however, current tool-using pipelines heavily rely on closed-source models with inaccessible parameters. it poses challenges particularly as follows: i) limited adaptability: the closed-source nature of major llms prevents them from customization, resulting in a lack of flexibility to adapt to tasks with specific requirements. ii) low efficiency and high inference cost: many existing llms can only be accessed online, which imposes limitations on the inference rate and leads to high expense. iii) privacy and security concerns: each query must be submitted to these closed-source llms to obtain a tool-using solution, which raises concerns regarding potential privacy breaches and compromises data security. to address these challenges, we propose toolink, a comprehensive framework to boost the tool-using ability of open-source models with the help of closed-source models. as shown in figure 1 , toolink first decomposes the target task by creating a toolkit for problem-solving, and then leverages the open-source model to use tools to answer queries in a chain-of-solving (cos) approach. specifically, cos disentangles the model's reasoning through two stages: cos-planning, which selects useful tools from the created toolkit and plans their usages based on the specific query; and cos-calling, which focuses on deriving the answer by performing tool invocations in code format according to the plan devised. to effectively train the open-source model in these abilities, we employ chatgpt to curate cos-gpt, a training dataset that aims to inspire the tool-using ability of open-source models through cos. specifically, we finetune llama-7b @xcite into llama-cos, which is equipped with strong toolusing capabilities by linking toolkit creation with the chain of problem-solving. llama-cos can solve the queries offline without uploading queries to closed-source models, ensuring data security and privacy. experiments further illustrate that toolink outperforms the chainof-thought (cot) @xcite on diverse tasks from @xcite and enables llama-cos to showcase comparable cos ability to that of chatgpt. in addition, llama-cos can generalize to unseen tasks by planning and calling tailored tools, and solve the target task with a toolkit not specifically tailored for it. these findings further affirm our framework's robustness in solving queries under real-world scenarios. more recent studies, such as hugginggpt @xcite , chameleon-llm @xcite , visualgpt @xcite and taskmatrix.ai @xcite , focus on assembling plannings, execution, and reasoning about tools into a robust pipeline. in addition to tool-using, art @xcite builds toolkits based on retrieved tasks from the manually built library, while latm @xcite and cre-ator @xcite involve the llms' toolmaking ability to offload their reasoning burden and raise task performance. in contrast to their prevalent use of closed-source llms to leverage tools, toolink offers unique advantages of tool use for smaller, open-source models. ## toolink framework as shown in figure 2 , toolink first adopts toolkit creation to break down the target task through generating potential tools for task-solving ( § 3.1). ## experiments to evaluate the effectiveness of toolink, we first conduct a validation test using chatgpt. we select eight distinct tasks from the big-bench dataset @xcite to investigate whether toolink can effectively leverage chatgpt's creativity and tool-using capability to improve task performance. subsequently, we finetune the open-source llama-7b model following the adaptation process outlined in § 3.3. this results in llama-cos, a model capable of linking the created toolkit with specific tool use through cos. we evaluate the effectiveness of llama-cos in utilizing tools on the same set of eight tasks and showcase its excellence. ## conclusions we present toolink, a tool-training framework that effectively applies toolkits to solve problems leveraging small, open-source language models. toolink offers increased flexibility in adapting to diverse downstream tasks while addressing concerns related to high inference costs and privacy. our main contributions include i) empirically implementing a framework that can effectively leverage open-source models' tool-using ability, ii) devising the chain-of-solving (cos) method that links toolkit creation and tool use through robust planning and calling, and iii) releasing the cos-gpt dataset that aims to enhance the model's cos capabilities. specifically, our llama-cos model outperforms traditional cot and achieves a comparable performance to chatgpt concerning tool-using. we believe our study provides a solid foundation for future researchers to explore and enhance the tool-using capabilities of open-source models.