T ool S pectrum: Towards Personalized Tool Utilization for Large Language Models (2025)

Abstract: While integrating external tools into large language models (LLMs) enhances their ability to access real-time information and domain-specific services, existing approaches focus narrowly on functional tool selection following user instructions while overlooking the critical role of context-aware personalization in tool selection. This oversight leads to suboptimal user satisfaction and inefficient tool utilization, particularly when overlapping toolsets require nuanced selection based on contextual factors. To bridge this gap, we introduce ToolSpectrum, a benchmark designed to evaluate LLMs’ capabilities in personalized tool utilization. Specifically, we formalize two key dimensions of personalization, user profile and environmental factors, and analyze their individual and synergistic impacts on tool selection. Through extensive experiments on ToolSpectrum, we demonstrate that personalized tool selection significantly improves user experience across diverse scenarios. However, even state-of-the-art LLMs exhibit the limited ability to reason jointly about user profiles and environmental factors, often prioritizing one dimension at the expense of the other. Our findings underscore the necessity of context-aware personalization in tool-augmented LLMs and reveal critical limitations for current models. Our data and code will be released soon.

## introduction integrating external tools into large language models (llms) enables them to transcend inherent knowledge boundaries by dynamically accessing specialized functionalities, driving unprecedented progress in task automation and problem-solving @xcite @xcite @xcite ; qian † equal contributions. work done during the first author's internship at beihang university. ‡ corresponding author gender: male height: 174 age: 15 profession: student, … preferences: prefers budget-friendly options, seeks lowest prices, compares brands for best deals. i need to book a flight to xian, could you help me book the tickets? weather: thunderstorm time of day: 9:00 date: 2024-08-07, …. ## toolspectrum: towards personalized tool utilization to accurately evaluate the ability of llms to utilize tools in a personalized way, we introduce toolspectrum, the first benchmark that considers both user profiles and environmental factors. in this section, we first provide a formal definition of personalized tool utilization and then describe our data collection pipeline, designed to efficiently and effectively gather the necessary information for evaluation. ## analysis in this section, we conduct a comprehensive analysis to answer three research questions rq1: is personalization truly necessary when llms utilize tools? (sec 5. ## conclusion this paper introduces personalized tool utilization, which considers functionally similar tools and userspecific factors to optimize tool utilization in realworld scenarios. specifically, we define two critical factors: user profile and environment, and present toolspectrum, a benchmark designed to evaluate personalized tool utilization. through extensive experiments conducted on toolspectrum, we demonstrate that incorporating personalized tool utilization significantly enhances its effectiveness. however, current llms face challenges in performing effectively on this new task.