M e NT i: Bridging Medical Calculator and LLM Agent with Nested Tool Calling (2025)

Abstract: Integrating tools into Large Language Models (LLMs) has facilitated the widespread application. Despite this, in specialized downstream task contexts, reliance solely on tools is insufficient to fully address the complexities of the real world. This particularly restricts the effective deployment of LLMs in fields such as medicine. In this paper, we focus on the downstream tasks of medical calculators, which use standardized tests to assess an individual’s health status. We introduce MeNTi, a universal agent architecture for LLMs. MeNTi integrates a specialized medical toolkit and employs meta-tool and nested calling mechanisms to enhance LLM tool utilization. Specifically, it achieves flexible tool selection and nested tool calling to address practical issues faced in intricate medical scenarios, including calculator selection, slot filling, and unit conversion. To assess the capabilities of LLMs for quantitative assessment throughout the clinical process of calculator scenarios, we introduce CalcQA. This benchmark requires LLMs to use medical calculators to perform calculations and assess patient health status. CalcQA is constructed by professional physicians and includes 100 case-calculator pairs, complemented by a toolkit of 281 medical tools. The experimental results demonstrate significant performance improvements with our framework. This research paves new directions for applying LLMs in demanding scenarios of medicine.

## introduction large language models (llms) have achieved milestone successes, demonstrating exceptional multi-tasking potential that includes reasoning, planning, tool usage, and code generation @xcite @xcite . recent studies indicate that by deeply integrating llms with large-scale toolsets, their task-solving efficiency can be significantly enhanced, further expanding the functional boundaries of the llms @xcite @xcite . additionally, developing customized tools tailored to specific application scenarios has become a key approach to optimize performance and enhance adaptability, showing remarkable results in numerous downstream tasks @xcite @xcite . in this paper, we focus on the issue of medical calculators, a typical and representative downstream task for llms tool application. medical calculators are standardized tools used to quantitatively assess individual health status, functional levels, disease severity, and treatment outcomes. they are extensively applied in clinical diagnostics, therapeutic monitoring, and research fields, aiding medical professionals in precisely evaluating and guiding personalized medical decisions @xcite @xcite . instances of calculator applications in medical scenarios are shown in figure 1 . calculators can help physicians quantify a patient's condition, providing a basis for formulating the next steps in the treatment plan. however, given the complexity of medical calculators in practical applications, the current methods of tool application are insufficient. specifically, selecting the appropriate tool is challenging. firstly, there are currently over 700 calculators in use foot_1 , and this vast number makes it difficult to choose the appropriate one. secondly, each calculator is developed for different conditions, and the medical knowledge associated with these conditions, as well as the specific rules embedded within each calculator, make it hard to master them. lastly, the continuous publication of new calculators every year adds ongoing pressure to keep tools up-to-date. consequently, we introduce the meta-tool mechanism, aimed at promoting flexible tool selection in vast toolkit. the meta-tool standardizes the timing of tool usage in medical calculator scenarios and refines the tool selection process, specifically catering to the selection within large-scale toolkits and complex scenarios. moreover, after selecting the tools, their application proves to be challenging due to locating information within lengthy case histories, slots filling, and unit conversion. medical scenarios involve complex case histories, and selecting values from these contexts tests the capacity for long-context processing. furthermore, the process demands substantial medical expertise to avoid errors in slot filling and to handle the challenges of unit conversion. for instance, llms may conflate rheumatic heart disease with congestive heart failure, or require conversions between units of total cholesterol from 8.3 mmol/l to 320.92 mg/dl. consequently, we introduce the nested calling mechanism. nested tool calling allows llm to introduce additional tools when the current tools and information are insufficient for the task at hand. this mechanism is particularly valuable in scenarios requiring extra unit conversion tools, ensuring robust handling of complex medical calculation tasks. we introduce a new benchmark, calcqa, to validate the practicality of the llm agent throughout the entire clinical process of medical calculator scenarios. in collaboration with board-certified physicians, we develop this benchmark: physicians select calculators based on real patient cases and provide corresponding diagnostic pathways, then use gpt-4 to simulate dialogue scenarios. alongside this benchmark, we develop a specialized medical calculator toolkit. we compile 44 medical calculators that are widely used in medical practice and systematically organize conversion methods for 237 common medical units. our contributions are as follows: (1) we introduce the calcqa benchmark, a new benchmark to assess the capabilities of llms in clinical calculator scenarios, including 100 calculator pairs based on real patients' cases, along with 281 medical calculator tools. (2) we develop the generalized menti agent architecture, which expands the llms' ability to address real-world medical calculator tasks through meta-tool and nested calling mechanisms. (3) our research indicates that menti represents the first full-process implementation of llms for calculator assessments in real medical scenarios, demonstrating exceptional performance in executing such tasks. ## methodology in section 3.1, we introduce calcqa, a novel benchmark and a toolkit with specialized medical tools, to assess the ability of llms to perform end-to-end calculator assessment in clinical scenarios. we develop this benchmark by having professional physicians diagnose real cases and then structuring these cases and diagnostic results into question-answer pairs. in sections 3.2 and 3.3, we introduce menti, which employs the meta-tool for tool selection and nested calling mechanism to assist llms in tool utilization. by enabling flexible tool selection and nested tool calling, menti effectively addresses intricate scenario tasks with the specialized toolkit. specifically, after the physician conducts a primary assessment of the patient and identifies the next steps for quantification, the task is handed over to menti. after preliminary diagnosis, menti uses the meta-tool (section 3.2) to select an appropriate medical calculator. menti then proceeds to fill in the slots of calculator parameters. if menti detects a mismatch in parameter units, it employs the nested tool calling (section 3.3). this mechanism enables iterative tool selection and utilization to resolve unit mismatches caused by complex toolchains requiring multiple tools. once the conversion tool's results are integrated, menti continues filling the calculator parameters and completes the final computation. this workflow, illustrated with a concrete example in figure 2 , is further detailed with complete patient information and case examples in the appendix d. ## experiment in this section, we test the performance of menti on calcqa and compare it with other methods (4.1 & 4.2). additionally, we conduct further analysis (4.3), showing the rational design of menti. ## conclusion in this work, we introduce menti, an agent framework that achieves flexible tool selection and nested tool calling, specifically designed for medical calculator scenarios. we develop and validate a medical calculator toolkit. additionally, we introduced calcqa, a benchmark consisting of 100 case-assessment pairs, to assess the capabilities of llms in medical calculator tasks. extensive experiments have shown that our method achieves superior performance compared to existing methods. we hope our work will further inspire the application of llms in medical contexts.