Tool Zero: Training Tool-Augmented LLM s via Pure RL from Scratch (2025)

Abstract: Training tool-augmented LLMs has emerged as a promising approach to enhancing language models’ capabilities for complex tasks. The current supervised fine-tuning paradigm relies on constructing extensive domain-specific datasets to train models. However, this approach often struggles to generalize effectively to unfamiliar or intricate tool-use scenarios. Recently, reinforcement learning (RL) paradigm can endow LLMs with superior reasoning and generalization abilities. In this work, we address a key question: Can the pure RL be used to effectively elicit a model’s intrinsic reasoning capabilities and enhance the tool-agnostic generalization? We propose a dynamic generalization-guided reward design for rule-based RL, which progressively shifts rewards from exploratory to exploitative tool-use patterns. Based on this design, we introduce the Tool-Zero series models. These models are trained to enable LLMs to autonomously utilize general tools by directly scaling up RL from Zero models (i.e., base models without post-training). Experimental results demonstrate that our models achieve over 7% performance improvement compared to both SFT and RL-with-SFT models under the same experimental settings. These gains are consistently replicated across cross-dataset and intra-dataset evaluations, validating the effectiveness and robustness of our methods.

## introduction integrating llms with external tools has emerged as a pivotal advancement, significantly enhances their ability to address complex tasks @xcite . it opens up many practical uses across different fields. for example, it supports the automation of reasoning tasks @xcite , and enables agent applications @xcite . a tool-augmented model can respond to * corresponding author. email: xding@ir.hit.edu.cn a user's query by invoking and executing external tools. in this paper, tools are used interchangeably with apis, functions, and plugins. current approaches to enhance tool-use capability involve synthesizing extensive tool-use trajectories with advanced language models, followed by sft on the generated data @xcite @xcite . under this paradigm, models achieves satisfactory performance in the scenarios that have the same distribution as the training data @xcite . however, these sft-trained models primarily engage in imitating surface-level patterns rather than internalizing the reasoning process, tend to memorize the training trajectories rather than developing robust, intrinsic reasoning capabilities @xcite . consequently, they exhibit limited generalization ability when applied to unseen scenarios, as elaborated in preliminary study in section 3. an illustrative example in figure 1 demonstrates that while the model correctly addresses a natural language translation task, it fails to appropriately invoke tools for code transpilation. specifically, the model interprets "translation" solely at the natural language level, failing to recognize the code transpilation scenario implicit in the user's query. this mismatch underscores a critical limitation: current models lack the intrinsic reasoning abilities to discern nuanced task contexts. enabling llms with genuine reasoning capabilities to overcome such generalization barriers has thus become an urgent research imperative. recent studies have demonstrated that simple rule-based r1-style rl @xcite , even without sft, can significantly enhance llms' complex reasoning capabilities @xcite @xcite . this paradigm inspires us to extend pure rl to the tool learning domain, aiming to address the generalization limitations by eliciting models' intrinsic reasoning abilities. to this end, we propose a dynamic generalization-guided reward design for rulebased rl. this approach employs a progressive reward strategy: it first promotes early-stage exploratory behavior to cultivate intrinsic reasoning, then refines these capabilities into tool-use patterns focused on final-task precision. this design effectively resolves the exploration-exploitation dilemma in open-domain tool learning, bridging the gap between reasoning generalization and taskspecific tool use. to evaluate generalization, we conducted extensive experiments across diverse function-calling benchmarks. results demonstrate that our proposed tool-zero 7b/32b models, trained using our method, significantly outperform both sft models and rl-with-sft baselines. for example, tool-zero-7b achieves a 7.14% performance improvement compared to sft model toolace-8b. notably, it also surpasses the rl-with-sft model toolrl-7b by 7.18%. additionally, these gains are consistently replicated in both cross-dataset and intra-dataset evaluations. ## related work to contextualize our approach, we survey prior research on tool learning and its integration with large language models. ## problem statement and analysis problem formulation. we first provide the problem formulation of reasoning in tool augmented models. it formalizes the integration of external tools into the inference process to solve complex tasks. given a tool set @xmath0, t 2 , . . . , t n } and a user query q, the reasoning trajectory up to step k is defined as: ## method in this section, we provide a detailed introduction to our method. figure 3 shows the overall architecture of our proposed dynamic generalization-guided reward strategy for grpo (gg-grpo). ## experiments in this section, we show the superiority of our method in performance and robustness across various benchmarks, and in-depth analysis to verify the effectiveness of our method. ## conclusion this study firstly extends a pure rule-based rl paradigm in tool-augmented models. designing a dynamic generalization-guided reward to tackle the generalization limitations. by fostering intrinsic reasoning through progressive explorationexploitation strategies, our approach reduces reliance on task-specific data and enhances toolagnostic adaptability. across diverse benchmarks, tool-zero models outperform sft and rl-with-sft baselines. these results validate rl's potential for scalable, autonomous tool learning in llms, advancing versatile ai agents for open-domain tasks.