Self Knowledge-Tracing for Tool Use ( SKT -Tool): Helping LLM Agents Understand Their Capabilities in Tool Use (2025)

Abstract: Large Language Models (LLMs) enhanced with tool use and APIs improve task performance but often misuse them, leading to inefficiency and unnecessary cost. We propose Self Knowledge-Tracing for Tool Use (SKT-Tool), a method enabling LLMs to assess their capabilities and make informed API usage decisions using knowledge tracing (KT). Our teacher-student framework helps LLMs optimize API calls in real-time without fine-tuning. Experiments across multiple datasets show that SKT-Tool significantly reduces API calls while maintaining accuracy, offering a scalable and cost-effective solution for tool-augmented LLMs. We conclude by analyzing shortcomings in this method and identifying directions for future work.

## introduction large language models (llms) have recently achieved a great number of remarkable abilities when incorporated with external tools like application programming interfaces (apis) @xcite . however, two problems emerge: current llm agents often struggle choosing which tools to use on different problems @xcite , and also that much of current research focuses solely on the api choice @xcite @xcite , where an llm indiscriminately calls tools during tests, which may not align with real-world scenarios @xcite . this can result in redundant api calls, leading to extra computation and even possible monetary loss in cases where every api call costs money, thus substantially reducing the usefulness of tool-augmented llm agents in real-world applications. furthermore, the introduction of tools tends to decrease the performance of llms across general datasets * lead author † senior author figure 1 : example dialogue from the synthetic generation process. this is a sample of the interaction string that is passed onto our student model for reference. when they have to decide whether or not tool use is appropriate @xcite . existing approaches fine-tune llms to choose apis @xcite @xcite but overlook the model's inherent capabilities. in this paper, we propose self knowledge-tracing for tool use (skt-tool) which offers a solution involving a student-teacher framework. trice @xcite introduces a method of teaching llms to use tools through continuous feedback. their findings also introduce the problem of llms dropping in question-answering accuracy when introduced to tools due to confusion over how to use them, and they employ an extensive feedback mechanism that relies on instructiontuning and reinforcement learning to align the llms behavior towards appropriate tool use. skt-tool differs by employing knowledge tracing, where the teacher llm generates targeted questions to assess the student's capabilities, unlike prior methods relying solely on task feedback. it optimizes through inference-time optimization instead of fine-tuning or reinforcement learning. unlike trice, which focuses only on accuracy, skt-tool also reduces api calls while aiming to maintain accuracy. ## method in this section, we detail our technical approach to creating skt-tool. our framework is divided into 2 subsections: (1) the teacher model and synthetic data generation; (2) the student model, retrieval-augmented generation (rag) and knowledge tracing, fig. 2 provides a visual overview of the system architecture. as seen in fig. 2 , we set up two language models to converse with each other, where one acts as the teacher and the other is the student. our framework aims to improve the student's model ability to decide whether or not it should call an api and answer qa datasets correctly with apis. ## conclusion we introduce a framework that helps llms assess their api usage, reducing calls by up to 50% while maintaining accuracy. after running multiple tests on the interaction history generated by the teacher model, we found that each interaction history appended incurred an average of 300 additional input tokens. the additional input tokens for 5 questions through rag are not high enough to exceed smaller models' context windows. though accuracy improvements are limited, our approach enables smaller models to use tools efficiently, lowering costs and expanding practical applications, particularly for weaker, newly-developed llms. we also believe that future work in refining the prompting methods for teacher feedback will help improve the student's final accuracy.