Toolscaler: Scalable Generative Tool Calling via Structure-Aware Semantic Tokenization (2025)

Abstract: Enhancing large language models (LLMs) with external tools has become a promising approach for solving complex tasks. As the number of available tools grows, context-based prompting methods increasingly rely on retrieval mechanisms. A common solution is to represent each tool with a unique token and train LLMs to generate the corresponding token during inference. However, this approach suffers from linear growth in representation space, leading to scalability challenges. It also limits generalization to novel or rare tools and underutilizes collaborative signals among tools in downstream tasks. In this paper, we propose SGTC, a generative tool invocation framework that introduces structure-aware semantic tokenization to encode tools as discrete code sequences. This method ensures similar tools share subtokens, enabling compression of the representation space and facilitating token sharing for new tools. We further introduce a post-guided, multistage iterative training strategy on a shared backbone model, where collaborative signals from downstream tasks guide the dynamic refinement of tool representations. Extensive experiments on the ToolBench dataset, which includes over 47,000 APIs, demonstrate the effectiveness of SGTC across various tasks, showcasing its potential as a scalable and generalizable generative tool-using paradigm in large-scale tool usage scenarios. The code is available at https://github.com/OPilgrim/Toolscaler.

## introduction large language models (llms) improve their ability to interact with the real world through integration with tools, such as calculators, databases, etc. @xcite @xcite , and are proficient in handling external input, performing actions, and autonomously completing tasks @xcite . however, as the number of tools grows to tens of thousands, existing methods for tool retrieval and execution struggle to scale effectively. while various approaches have been proposed to integrate tools into llms @xcite , including context-based prompting @xcite @xcite and fine-tuning with tool description @xcite @xcite , they still face challenges in large-scale tool settings. context-based prompting methods are inherently constrained by the input length limitation of llms, making it infeasible to include all tools within a single prompt and requiring external retrievers to select a small subset of candidate tools. on the other hand, fine-tuning-based methods that integrate tools into model parameters @xcite often rely on assigning each tool a unique identifier (id) @xcite , which introduces several limitations in large-scale scenarios. first, the vocabulary size grows linearly with the number of tools, resulting in higher memory consumption and a larger decoding space, which increases the inference burden @xcite . second, the data sparsity and long-tail distribution of tool usage not only hinder the learning of reliable representations for infrequent tools, but also make it difficult to incorporate newly introduced tools without additional retraining or architectural changes. third, since id embeddings are learned independently, they fail to capture functional similarities or collaborative relationships among tools, further limiting generalization and reuse across tasks. to address these limitations, we propose toolscaler, a unified generative framework that provides a scalable and semantically structured rep-resentation of large-scale tools, enabling simultaneous tool retrieval and calling during generation. first, we introduce structure-aware semantic tokenization, which assigns each tool a compact sequence of discrete codes @xcite @xcite derived from its semantic embedding. these semantic embeddings are obtained by compressing tool knowledge into a small number of special tokens that encode functional and behavioral information. to generate the code sequences, we employ a lightweight deep residual k-means algorithm over the semantic embedding space for centroid assignment, and use the resulting centroids to initialize the embeddings of code tokens. the discrete codes are then dynamically refined via post-guided training to ensure that semantically or functionally similar tools share similar subtokens. this codebased tokenization facilitates representation compactness and encourages knowledge sharing across tools, while also enabling approximate similarity estimation (e.g., via hamming distance) without additional model training-thus offering scalability to newly added or unseen tools. its hierarchical structure enables a logarithmic compression of the tool vocabulary space, significantly reducing the decoding overhead compared to linear id-based indexing. second, we unify semantic tokenization, retrieval, and calling into a single generative modeling framework. this design allows for multistage iterative training, where the model progressively integrates tool knowledge-from basic documents to usage contexts and invocation workflows-across stages. finally, in later training iterations, downstream collaboration signals are leveraged to refine tokenization strategies, allowing the model to dynamically adapt to tool usage patterns and improve its generative capabilities over time. in summary, our work contributes the following key aspects: thanks to the powerful contextual learning ability @xcite , it is possible to enable llms to use tools simply by displaying examples within the prompt, without the need for training @xcite . therefore, most methods focus on guiding llms to mimic human task solving processes and generate plans @xcite @xcite @xcite , and improving plans by incorporating execution feedback @xcite , thus combining reasoning with action. however, context-based learning methods are prone to hallucinations and are limited by inadequate context capacity when faced with large-scale tools. although the tool retrieval stage is widely used, including trained additional retriever to rank top-k candidates from a large number of tools based on similarity to the query to enhance the generation process @xcite @xcite . such strategies do not improve the model's understanding of external tool knowledge, and maintaining dense retrieval databases and document indices can lead to inefficiency and difficulties in optimizing within an end-to-end agent framework. ## preliminaries existing agents based on llms that use tools typically involve four stages @xcite : given a query/task q, (1) generating a plan p, (2) determining the tool @xmath0, (3) generating tool parameters c, (4) and collecting feedback f from tool execution. the model iteratively repeats the process (p i , d i , c i , f i ) until it generates a stopping symbol or reaches the maximum number of iterations, ultimately generating the answer a and completing the task. the entire process forms an interaction trajectory t @xmath1, (p 1 , d 1 , c 1 , f 1 ), ...., (p t , d t , c t , f t ), a], while t is the total round, and @xmath2. a specific example can be found in figure 8 . we unify the four phases through a generative framework and focus on improving the second phase. during the generative tool determination phase, y i+1 = logp (idx(d)|q, @xmath3, embd(d)), where idx(d) is the tool tokens. when the candidate toolset |d| = n is large, existing unique identifier schemes @xcite suffer from sparse supervision and poor generalization. instead, if tool representations share substructures, we can reduce representation space and enhance inter-tool correlation. to this end, we adopt a codebook-based semantic tokenization @xcite , where a codebook with l layers and k codes per layer enables tools to share semantic components. two similar tools will share the same code at layer @xmath4. this yields a representation capacity of k l , allowing compact encoding even when n ≫ k. compared to unique identifiers requiring n × d memory, our method compresses into logarithmic space k × l × d, where d is the embedding dimension. @xmath5; a j ] figure 1 : overview of the toolscaler framework. toolscaler employs a three-stage training paradigm with multiround iterative refinement to progressively optimize tool representations. initially, the llm generates clustered, structure-aware semantic code sequences that replace tool names in the corpus, forming a compact and expressive representation space. the model then learns tool usage from query-tool pairs, procedural logic from execution trajectories, and collaboration patterns from multi-tool interactions. throughout iterations, tool knowledge and clustering co-evolve, refining code sequences for better alignment with downstream retrieval and invocation. finally, a generative agent is trained to perform end-to-end tool calling. the content block contains textual information extracted from the tool documentation, such as functional descriptions, and is represented as {a j } m @xmath6, where m denotes the number of distinct pieces of information. the token block consists of a sequence of v gist tokens @xcite , each with learnable embeddings designed to extract and aggregate information from the content block. the task block contains a special indicator token t j and the corresponding target output a j ; specifically, for 0 < @xmath7, whereas for @xmath8. finally, the placeholder block is employed to ensure that the task block can be effectively guided by the output of the token block. in practice, its embedding is initialized with the output embedding of the token block, thereby facilitating the reconstruction or generation process. similarly, we adopt a cascaded attention masking scheme to restrict task output generation solely to the token (and subsequent placeholder) blocks. each block applies a causal mask to capture internal sequential dependencies, while only the content block fully attends to the token block and the placeholder block to the task block; all other inter-block attention is disabled. ## conclusions we propose toolscaler, a model-agnostic framework that leverages a single llm to perform generative tool retrieval and calling, thereby eliminating the need for external retrievers. toolscaler introduces structure-aware semantic code sequences to concisely and effectively represent large-scale toolsets, while maintaining adaptability to the continual expansion of new tools. our method integrates basic tool knowledge and inter-tool coordination signals, and dynamically refines code sequences through multistage iterative training. extensive experiments demonstrate the effectiveness of toolscaler, particularly in multi-tool scenarios. it consistently outperforms strong baselines in ac-curacy, pass rate, and win rate, and further shows clear advantages over other tokenization strategies through structure-aware semantic modeling. our study provides a promising direction for large-scale generative tool execution and lays the groundwork for future extensions, such as combining generative agents with reinforcement learning to further enhance tool-use autonomy in llms.