S table T ool B ench- M irror API : Modeling Tool Environments as Mirrors of 7,000+ Real-World API s (2025)

Abstract: The rapid advancement of large language models (LLMs) has spurred significant interest in tool learning, where LLMs are augmented with external tools to tackle complex tasks. However, existing tool environments face challenges in balancing stability, scale, and realism, particularly for benchmarking purposes. To address this, we propose MirrorAPI, a novel framework that trains specialized LLMs to accurately simulate real API responses, effectively acting as “mirrors” to tool environments. Using a comprehensive dataset of request-response pairs from 7,000+ APIs, we employ supervised fine-tuning and chain-of-thought reasoning to enhance simulation fidelity. MirrorAPI achieves superior accuracy and stability compared to state-of-the-art methods, as demonstrated by its performance on the newly constructed MirrorAPI-Bench and its integration into StableToolBench.

## introduction recently, the remarkable reasoning capabilities exhibited by large language models (llms) @xcite @xcite have catalyzed growing interest in tool learning @xcite @xcite . analogous to the way humans leverage tools in the physical world to enhance their cognitive and physical capacities, llms augmented with external tools @xcite demonstrate unprecedented potential in addressing diverse and complex challenges such as general task solving @xcite , web search @xcite , and multimodal scenarios @xcite ). † project leader * equal contribution github: thunlp-mt/stabletoolbench dataset and models: https://huggingface.co/ stabletoolbench stability scalability realness ## environment simulation for stabletoolbench an important application of the simulation model is its role as an environment model in a tool learning benchmark. to improve the reliability and stability of simulation on stabletoolbench, we further finetune mirrorapi on the cache data, named mirrorapi-cache. to evaluate mirrorapi and mirrorapi-cache on stabletoolbench, we compare the real environment with our mirrorapi in the stabletoolbench setting, with both the solvable pass rate (sopr) and our proposed final answer completeness (fac) scores. additionally, we present the performance of several baseline models when mirrorapi is used as the environment model, demonstrating its reliability. ## analysis 5.1 ablation studies to evaluate the effectiveness of training in cot mode, we compare several models on the ood succ test set using the observation-following metric. the models tested include: mirrorapi, mirrorapi without cot training, mirrorapi without augmented and cot data, and mirrorapi trained sequentially with sft and cot. note that no additional data was created specifically for cot compared to sft. for inference, we use the direct sft prompt for all models except the one trained sequentially with cot, which uses the cot prompt to achieve better performance. the results are presented in ## conclusion in this work, we present mirrorapi, a novel framework that trains specialized llms to act as "mirrors" to tool environments by accurately replicating real api responses. the simulator is trained on request-response data pairs from over 7,000 apis sourced from rapidapi, with performance enhanced through a novel api mechanism reasoning framework. experimental results demonstrate that mirrorapi achieves strong documentation and instruction following capability. responses provided by it are also the most similar to the real environment compared to baseline prompting methods. furthermore, when deployed as an environment model in stabletoolbench, mirrorapi delivers reliable outcomes comparable to those obtained from real-world apis, validating its practical utility. beyond the immediate scope of this work, general tool simulators may be beneficial to advancing llms' tool-using capabilities. first, such simulators can enable execution feedback during training or inference without costly real-environment interactions, paving the way for novel online training algorithms and adaptive inference strategies @xcite @xcite . second, they facilitate the generation of diverse training trajectories through the integration of synthetic apis, addressing data scarcity challenges in toollearning scenarios. finally, while our current focus lies in simulating functional and reliable tools, the framework can be extended to simulate tools with operational failures, safety-critical vulnerabilities, or ethical concerns. this capability could significantly expand the scope of tool-learning research by enabling risk-aware training and robustness testing in controlled environments.