Tool learning via Inference-time Scaling and Cycle Verifier (2025)

Abstract: In inference-time scaling, Chain-of-Thought (CoT) plays a crucial role in enabling large language models (LLMs) to exhibit reasoning capabilities. However, in many scenarios, high-quality CoT data is scarce or even unavailable. In such cases, STaR-like methods can help LLMs synthesize CoT based on user queries and response, but they inevitably suffer from the risk of compounding errors. In this work, we tackle an even more challenging scenario: tool learning in the absence of user queries. We design a data scaling method using back-translation, which establishes an inference cycle to synthesize both user queries and CoT data. To reudce the compounding error of inference time, we introduce two rule-based verifiers to assess the validity of the synthesized CoT data. In particular, the Cycle Verifier facilitates performance improvement by continuously accumulating new data over multiple iterations. Our approach achieves a 75.4% pass rate and a 79.6% win rate using small models (7B) in StableToolBench. Notably, these results are obtained exclusively from self-synthesized high-quality data, without relying on external supervision or expert trajectories for warm-up.

## introduction tool use is a critical capability for llms, enabling them to perform complex reasoning tasks through interacting with real-world environments @xcite @xcite @xcite . imitation learning is one of the most efficient ways to acquire new skills, as it allows llms to benefit from expert trajectories. however, collecting high-quality expert data @xcite is often time-consuming and inherently non-scalable. in real-world scenarios, human experts use tools in a sequential manner: they first determine the query intent, select the appropriate tool, and then provide a cot-based solution. in contrast, instruction backtranslation @xcite follows an inverse process, inferring the user's intent based on a given solution. as shown in figure 1 , we integrate these two approaches to establish an inference cycle by leveraging tool environments and llms. this cycle ensures data scalability by continuously generating new user queries and interacting with the environment to produce cot data. star-like methods @xcite provide a potential approach to bootstrapping reasoning capability in inference-time scaling @xcite . however, they rely on oracle responses foot_0 to filter low-quality data using rejection sampling, which is unavailable in our scenario. in multi-intent tool use, llms often require multiple rounds of reasoning and interaction. it is not suitable to obtain reward estimation through frequent trial-and-error, which is inefficient and unscalable. in other words, without a deterministic reward modeling, data augmentation is highly susceptible to compounding errors @xcite , a phenomenon known as llm hallucination and error propagation in inference trajectories. thus, we introduce two predefined rule-based verifiers, which provide process-level reward signals to evaluate each api invocation action. in particular, the format verifier aims to filter out parameter value inconsistencies by ensuring that the provided parameters align with the function's predefined types. the cycle verifier ensures the consistency of api invocation trajectories, preventing deviations from the conditions of the initially synthesized user query. notably, these reward signal can be distributed across multiple reasoning steps, ensuring the effectiveness of process supervision. furthermore, we introduce the preference tree @xcite to enhance sampling efficiency, which reduces the cost of redundant reasoning in multi-step inference tasks. additionally, the preference tree provides pairwise samples (chosen and rejected) for direct preference optimization (dpo) training. in our implementation, we adopt iterative training to perform multiple stage data synthesis, a process we call infcycle, effectively unlocking the potential of inferencetime computation. we evaluate infcycle on a diverse set of benchmarks, including the interactive stabletoolbench and the non-interactive berkeley function calling dataset. moreover, our evaluations across multiple foundation models, from the weakly toolcapable mistral-7b-instruct to the more advanced llama3-8b-instruct and qwen2.5-7b-instruct. empirical results indicate that infcycle not only enhances model performance but also outperforms robust baselines such as gpt-4 on stabletoolbench. additionally, on berkeley function calling, infcycle achieves relative improvements of +16 and +40 points, further demonstrating its effectiveness. our main contributions are as follows: inference-time scaling llms can utilize techniques such as cot @xcite or reflection @xcite to enhance their reasoning capabilities during testing. however, many studies show that these methods often have limited effectiveness for complex tasks @xcite @xcite . nevertheless, this research direction remains crucial for the future, particularly in exploring the trade-offs between inference time and pre-training computing. @xcite demonstrate that scaling inference computing through repeated sampling leads to significant improvements in coverage across various tasks and models. @xcite introduce a compute-optimal strategy that enhances the efficiency of test-time compute scaling compared to a best-of-n methods. in our scenario, no expert data trajectories or proprietary models are available, meaning the model improves purely through self-improvement. under such strict conditions, obtaining explicit reward signals is highly challenging. therefore, this work focuses on exploring the effectiveness of prior knowledge and consistency constraints as alternative forms of reward. ## preliminaries we first provide the task definition of tool use and describe the scenarios in our works. given a user query q and a set of candidate api functions @xmath0, api 1 , . . . , api |a| }, llms require to fulfill the user's intent by executing a specific sequence of api function calls. the decision pro- ## methods in scenarios lacking tool-specific training data, we first collect an available candidate toolset and synthesize data. then, we filter high-quality cot data for model training and inference, as shown in figure 3 . considering the capability limitations of open-source small size llms, the entire process requires multiple iterative refinements. ## experiments we chose stabletoolbench @xcite and the berkeley function-calling @xcite benchmark to evaluate the effectiveness of our proposed method. stabletoolbench requires real-time interaction with the rapidapi 5 to gather feedback, primarily evaluating the model's performance in a dynamic environment. in contrast, berkeley function-calling employs a static evaluation set that emphasizes the model's ability to extract complex apis and parameters. to achieve this, we collected 6k apis from rapidapi and converted 2k code problems from leetcode into usable apis. for additional statistics and experimental details, please refer to the appendix a. ## discussion and conclusion in this work, we enhance the model's tool use capabilities without relying on external supervision. inspired by inference-time scaling, which increases the sampling space to enhance performance, this approach is particularly suitable for small size models to facilitate self-improvement. we demonstrate that infcycle effectively synthesizes high-quality data using the llms and cycle consistency acting as process verifiers. in the future, we plan to incorporate more tools and parameters into our research. the trade-off between the number of parameters and inference cost remains a key focus, as it directly impacts the efficiency and applicability. additionally, we plan to conduct more experiments to investigate further scalability factors.