TTPA : Token-level Tool-use Preference Alignment Training Framework with Fine-grained Evaluation (2025)

Abstract: Existing tool-learning methods usually rely on supervised fine-tuning, they often overlook fine-grained optimization of internal tool call details, leading to limitations in preference alignment and error discrimination. To overcome these challenges, we propose **T**oken-level **T**ool-use **P**reference **A**lignment Training Framework (TTPA), a training paradigm for constructing token-level tool-use preference datasets that align LLMs with fine-grained preferences using a novel error-oriented scoring mechanism. TTPA first introduces reversed dataset construction, a method for creating high-quality, multi-turn tool-use datasets by reversing the generation flow. Additionally, we propose _Preference Oriented Tool-use Dataset Construction_ to capture fine-grained preferences by modeling token-level differences during generation. To address biases in scoring, we introduce the _Error-oriented Scoring Mechanism_, which quantifies tool-call errors and can be used as a training signal. Extensive experiments on three diverse benchmark datasets demonstrate that TTPA significantly improves tool-using performance while showing strong generalization ability across models and datasets.

## introduction in recent years, large language models (llms) have demonstrated significant potential across a wide range of domains @xcite , and enabling llms @xcite to interact with external environments is critical for enhancing their ability to solve complex real-world problems, such as calling search engines to access real-time information @xcite and travel planning @xcite . as llms continue to evolve, integrating external tools is essential not only to address practical user needs but also to advance toward artificial general intelligence @xcite @xcite . current approaches primarily employ supervised fine-tuning (sft) to improve the tool-use capabilities of llm @xcite @xcite @xcite . although sft improves tool call quality and facilitates structured outputs, it may still struggle with fine-grained cases where even minor token-level errors can lead to wrong tool calls, such as missing braces. moreover, they typically rely on synthetic data generated in a forward manner, first creating queries and then generating answers, which may cause many issues, such as unanswerable queries and leakage of tool names or arguments, resulting in low-quality samples that require costly filtering. to address the former issue, recent work explores reinforcement learning (rl) methods for tool learning, such as tl-training @xcite , which uses complex reward functions with proximal policy optimization @xcite , and dpo @xcite , which leverages trajectory-level preference sampling that focuses on the overall correctness of the full sequence of tool calls. while rl-based approaches aim to achieve preference alignment to help models prefer correct tool use, they face key challenges in implementation and stability @xcite : @xcite existing methods often overlook fine-grained preference discrepancies within individual tool calls. in highly structured outputs like tool calls, even a single token error can lead to complete failure, highlighting the necessity for more precise preference alignment. (2) furthermore, existing preference data sampling methods typically rely on llm or human evaluations at the trajectory level, rather than assessing each individual tool call. this coarse-grained assessment introduces biases due to overlooking fine-grained errors and relying on ambiguous criteria, often resulting in preference data with low discriminative quality and high noise levels, which limits the effectiveness of alignment strategies. to overcome the above challenges, we propose token-level tool-use preference alignment training framework (ttpa), a tool-use training paradigm that first constructs token-level preference datasets that align llms with fine-grained preferences, and then employs an error-oriented reward mechanism to train the model. the proposed ttpa contains two main components: (1)preference oriented tool-use dataset construction, including reversed dataset construction and tokenlevel preference sampling, (2)error-oriented scoring mechanism. in the first component, we first propose a reversed data construction approach, which introduces a novel paradigm for creating multi-turn tool-use datasets to address the latter issue of sft methods. unlike conventional methods @xcite ) that start with queries, our approach reverses the process: we first leverage llms to rehearse a sequence of tool calls and a final answer within a predefined toolusing scenario. the query is then constructed based on the generated answer. this reversed strategy avoids complex and inefficient filtering by deriving queries from scenarios and answers, ensuring each query is answerable and preventing data leakage. moreover, it maintains question difficulty since the model must use multiple tools, with combinedtool tasks considered more challenging. to capture the fine-grained preference in the tool calls, we propose token-level preference sampling. unlike trajectory-level methods @xcite that incorporate complete tool-calling sequences, our approach explicitly captures models' preferences by sampling top-k candidate tokens from the probability distribution during tool-call generation by llm. when training the tool-use llm, existing models employ llms to grade the outputs as the training signal that usually introduces biases caused by coarse-grained evaluation and ambiguous criteria @xcite . thus, we propose the error-oriented scoring mechanism, which defines a taxonomy of tool-call errors. and then we use it to construct a preference alignment dataset and fine-tune the llm. extensive experiments on three benchmark datasets show that ttpa notably improves tool selection, parameter filling, and return value parsing capabilities. moreover, the model fine-tuned with ttpa demonstrates strong generalization and transferability across datasets, enhanc-ing the reliability and applicability of llms in real-world applications. our contributions are summarized as follows: ## related work tool learning. tool learning enhances llms by integrating external tools, enabling them to select tools, generate parameters, and parse results to respond to user queries @xcite @xcite @xcite . approaches include tuningfree methods, which use in-context learning or algorithmic design @xcite @xcite , and tuning-based methods, which fine-tune on tool-use datasets @xcite @xcite . tuning-free methods are often limited by the foundation model's capabilities, while tuning-based methods face challenges with noisy data. our framework addresses this by employing reversed dataset construction and token-level preference sampling to produce high-quality, lownoise datasets, ensuring better alignment with tooluse tasks and addressing fine-grained discrepancies in tool calls. additionally, our approach introduces an error-oriented scoring mechanism to refine the alignment process and improve model robustness in complex scenarios. tool-use datasets. tool learning has driven the creation of datasets to improve llms' tool-use capabilities @xcite @xcite . toolbench @xcite leverages llms to compile large datasets, while apigen @xcite uses an automated pipeline to generate diverse datasets across multiple api categories. toolace @xcite further advances this by integrating tool synthesis and dialogue generation, enhancing dataset diversity and complexity. however, these datasets often suffer from noise, single-turn limitations, or high resource costs, and few address the growing need for preference-based datasets. our framework uses reversed dataset construction and token-level preference sampling to construct high-quality preference datasets, aligning tool-use preferences and improving fine-grained alignment for structured outputs, ensuring better generalization across diverse tool-use scenarios. ## conclusion in this paper, we present token-level tooluse preference alignment training framework (ttpa), an automated method for constructing high-quality tool-use preference datasets to enhance the tool-use capability of large language models. the ttpa employs preference oriented tool-use dataset construction, which incorporates two key components: (1) reversed data construction for generating diverse tool-use dataset, and (2) token-level preference sampling for capturing token-level preference, to construct a rich and fine-grained tool-use preference dataset that better aligns with real-world usage scenarios. additionally, we develop an error-oriented scoring mechanism that enables precise alignment of llms with fine-grained user preferences during tool-usage. experiment results demonstrate that the tool learning model fine-tuned with ttpa can achieve stateof-the-art performance, thereby advancing the field of tool usage in large language models.