CREATOR : Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models (2023)

Abstract: Large Language Models (LLMs) have made significant progress in utilizing tools, but their ability is limited by API availability and the instability of implicit reasoning, particularly when both planning and execution are involved. To overcome these limitations, we propose CREATOR, a novel framework that enables LLMs to create their own tools using documentation and code realization. CREATOR disentangles abstract tool creation and concrete decision execution, resulting in improved performance. We evaluate CREATOR on MATH and TabMWP benchmarks, respectively consisting of challenging math competition problems and diverse tabular contents. Remarkably, CREATOR outperforms existing chain-of-thought, program-of-thought, and tool-using baselines. Additionally, we introduce the Creation Challenge dataset, featuring 2K diverse questions, to emphasize the necessity and benefits of LLMs’ tool creation ability. Further research demonstrates that leveraging LLMs as tool creators facilitates knowledge transfer, and LLMs exhibit varying levels of tool creation abilities, enabling them to adapt to diverse situations. The tool creation ability revolutionizes the LLM’s problem-solving paradigm, driving us closer to the next frontier of artificial intelligence.

## introduction in recent years, notable progress has been made in large language models (llms) like gpt-3 @xcite , codex @xcite , palm @xcite , llama @xcite , @xcite , and the latest @xcite . these models exhibit impressive capabilities in in-context learning, code generation, and various natural language processing (nlp) tasks @xcite . however, there are still limitations to address, such as the inability to handle up-to-date information @xcite , provide accurate mathematical results, or reason over long chains of logic @xcite @xcite @xcite . to overcome these concerns, researchers have explored equipping llms with external tools to alleviate their memory burden and enhance their expertise @xcite . for instance, integrating tools such as question-answering systems or web search engines enables llms to learn how and when to access external resources for problemsolving @xcite . recent studies have also incorporated additional tools for llms, such as github resources, neural network models (e.g., huggingface library), and code interpreters (e.g., python interpreter), aiming to enhance their capabilities @xcite @xcite @xcite . these tools require llms to provide detailed plans before utilizing them to solve complex problems. however, tool-augmented llms still encounter challenges @xcite @xcite , particularly in the following aspects. (1) limitation in scope: current approaches focus on a limited number of tools, making it difficult to find an appropriate existing tool for new problem types. in this paper, we propose a novel approach to address these challenges. rather than treating llms as users of tools, we empower them to be creators of tools, enabling them to solve problems with higher accuracy and flexibility. we introduce our tool creation framework, creator, which leverages llms' ability to create and modify tools based on the problem at hand. figure 1 illustrates the differences between creator and a general tool-using framework. while the tool-using framework focuses on reasoning to select and plan api usage, our framework emphasizes diversifying tool choices, disentangling abstract and concrete reasoning, and improving robustness and accuracy. specifically, creator consists of four stages: to evaluate our design's effectiveness, we test creator on two existing benchmarks: math @xcite and tabmwp @xcite , as well as the creation challenge dataset we create. the math dataset contains diverse and challenging math competition problems, while tabmwp includes a wide range of tabular contexts for problemsolving. notably, chatgpt built on creator achieves remarkable average accuracies of 59.7% and 94.7% on math and tabmwp respectively, surpassing the standard chain-of-thought (cot) @xcite , program-of-thought (pot) @xcite , and tool-using baselines by significant margins. as existing benchmarks do not specifically evaluate tool creation, we further introduce the creation challenge dataset, which consists of novel and challenging problems that are inadequately solved using existing tools or code packages. this dataset highlights the necessity and advantages of llms' tool creation ability. in addition, we show experimental results that provide evidence of how tool creation plays a crucial role in promoting knowledge transfer across similar queries that possess common core knowledge but differ in specific scenarios. we also present case studies highlighting the varying levels of tool creation ability observed in llms, allowing them to better adapt to diverse problem settings. tool use and tool creation. as an emerging field within nlp, the active interaction of llms with environments is facilitated through tools that serve as the medium @xcite . recent studies address constraints of llms, such as the limited real-time responsiveness and inaccurate calculations, by incorporating external tools @xcite @xcite . these studies augment llms with tools like scratch pads, search engines, qa systems, and calculators @xcite @xcite to improve task performance. more recent efforts integrate llms' tool-using abilities into a pipeline for task planning, tool calling, and result synthesis @xcite @xcite . in contrast, our work goes further by enabling llms to create tools instead of relying solely on existing tools. as our concurrent works, tool creation ability is also investigated under latm framework @xcite and llm customization @xcite . ## design of creator distinct from previous frameworks for tool-using, creator leverages the tool creation ability of llms by incorporating four special stages: cre- method create tools utilize tools apply codes emphasize reusability reasoning pattern cot ----linear pot --✓ -linear tool use -✓ ✓ -linear creator ✓ ✓ ✓ ✓ non-linear ## experiments to evaluate the effectiveness of creator, we conduct experiments on two established benchmarks: math @xcite and tabmwp @xcite . additionally, we perform experiments on a newly introduced dataset, creation challenge, comprising 2k diverse questions that are inadequate to solve using existing tools or code packages. this enables us to further demonstrate the necessity and advantages of the llm's tool creation ability. ## conclusion we propose the concept of automatic tool creation through llms and empirically devise creator that harnesses the capabilities of llms as tool creators. by disentangling llm's abstract and concrete reasoning, creator enables clearer logic and enhances overall performance. through comprehensive evaluations on established benchmarks and creation challenge, we demonstrate the superiority and indispensability of creator compared to existing cot, pot, and tool-using approaches. we anticipate our study will inspire the development of more sophisticated ai systems leveraging llm's tool creation potential.