TL -Training: A Task-Feature-Based Framework for Training Large Language Models in Tool Use (2025)

Abstract: Large language models (LLMs) achieve remarkable advancements by leveraging tools to interact with environments, a critical step toward generalized AI. However, the standard supervised fine-tuning (SFT) approach, which relies on large-scale datasets, often overlooks task-specific characteristics in tool use, leading to performance bottlenecks. To address this issue, we analyze three existing LLMs and uncover key insights: training data can inadvertently impede tool-use behavior, token importance is distributed unevenly, and errors in tool calls fall into a small set of categories. Building on these findings, we propose TL-Training , a task-feature-based framework that mitigates the effects of suboptimal training data, dynamically adjusts token weights to prioritize key tokens during SFT, and incorporates a robust reward mechanism tailored to error categories, optimized through proximal policy optimization. We validate TL-Training by training CodeLLaMA-2-7B and evaluating it on four open-source test sets. Our results demonstrate that the LLM trained by our method matches or surpasses both open- and closed-source LLMs in tool-use performance using only 1,217 training data points. Additionally, our method enhances robustness in noisy environments and improves general task performance, offering a scalable and efficient paradigm for tool-use training in LLMs. Code and data are available at https://github.com/Junjie-Ye/TL-Training.

## introduction large language models (llms) @xcite @xcite excel in natural language understanding due to pre-training on extensive datasets @xcite . by incorporating tool-use capabilities, llms can extend beyond text generation to interact * corresponding author. with the external environment, enabling tasks such as web searches and email management @xcite . furthermore, these capabilities are essential for addressing real-world user needs and advancing the development of general-purpose ai @xcite . current approaches to training llms for tool use rely heavily on large-scale datasets generated from trajectories of interactions with tools @xcite . standard supervised fine-tuning (sft) is then applied to pre-trained models. while effective in some cases, these methods overlook key task-specific characteristics, leading to performance bottlenecks. for instance, toolllama-2-7b-v2 @xcite achieves only 80% of gpt-4's performance on tool-use benchmarks @xcite , indicating room for significant improvement. to fill this gap, we conduct an in-depth analysis of three tool-using llms, uncovering several key phenomena. notably, over 17% of the training data for rotllama @xcite ) contains tool-calling errors (figure 1 ), primarily due to reliance on data from gpt series models, which are not entirely error-free with complex tools. training on such flawed data can hinder model performance. additionally, our tests of toolllama-2-7b-v2 and nexusraven-13b-v2 (team, 2023) reveal that incorrect tool selections often share a common prefix with the correct ones ( based on these insights, we propose tl-training, a task-feature-based framework for training llms in tool use. tl-training mitigates the negative impact of training data by identifying erroneous interaction paths and excluding them from gradient updates. it prioritizes key tokens through adaptive weighting during sft and incorporates tool feedback into a robust reward mechanism for reinforcement learning using the proximal policy optimization (ppo) @xcite . we validate our approach by training codellama-2-7b (rozière et al., 2023) on a curated dataset of 1,217 tool-call trajectories generated with gpt-4o. evaluations on four open-source test sets demonstrate that the model trained with tl-training matches or surpasses the tool-use performance of leading open-and closedsource llms, despite requiring significantly less training data. additionally, tl-training improves robustness in noisy environments and enhances general task performance. in summary, our contributions are as follows: blocked for tool hallucination tool name getpaperbydetail is not defined assistant getpaperdetails(@xmath0: {"title": "toolllm: facilitating large language models to master 16000+ real-world apis"}} assistant finish(@xmath1, whose title is "toolllm: facilitating large language models to master 16000+ real-world apis."') ## preliminaries task formulation given a model m, a user query q, and a collection of tools t, the task of tool use requires m to iteratively select the appropriate tool t @xmath2, process its feedback o s , and continue selecting subsequent tools t s+1 until the query is resolved and a final answer is obtained. formally, this can be represented as t s+1 = m(•|q, t, t 0..s , o 0..s ). this task is distinct from traditional natural language processing tasks, as it requires the model to invoke tools repeatedly and interpret their feedback dynamically. despite its importance, to the best of our knowledge, there has been no systematic examination of the intrinsic properties of tool use. thus, we aims to fill this gap by conducting an in-depth analysis focusing on both the training data and model performance. ## approaches building on the analysis in section 3, we propose tl-training, a novel training paradigm for llms in tool use. as shown in figure 2 , this paradigm incorporates three core techniques: mitigating the adverse effects of suboptimal data by preventing its back-propagation (section 4.1), prioritizing key tokens using adaptive weight adjustments (section 4.2), and implementing a reward mechanism tailored to tool invocation error categories to enable effective reinforcement learning (section 4.3). 1 ## conclusion in this paper, we introduce tl-training, a novel paradigm for training llms specifically for tool use. our approach mitigates the impact of erroneous interaction data, adaptively adjusts token weights, and introduces a reward mechanism tailed for tool use to facilitate ppo-based reinforcement learning. this methodology not only enhances llms' tool-use capabilities but also improves their robustness in noisy environments, all while preserving strong general performance across a range of tasks. our findings demonstrate the effectiveness of tl-training in addressing realworld challenges in tool use, offering a promising direction for future research in improving llm interaction capabilities and adaptability.