Auto- SLURP : A Benchmark Dataset for Evaluating Multi-Agent Frameworks in Smart Personal Assistant (2025)

Abstract: In recent years, multi-agent frameworks powered by large language models (LLMs) have advanced rapidly. Despite this progress, there is still a notable absence of benchmark datasets specifically tailored to evaluate their performance. To bridge this gap, we introduce Auto-SLURP, a benchmark dataset aimed at evaluating LLM-based multi-agent frameworks in the context of smart personal assistants. Auto-SLURP extends the original SLURP dataset—initially developed for natural language understanding tasks—by relabeling the data and integrating simulated servers and external services. This enhancement enables a comprehensive end-to-end evaluation pipeline, covering language understanding, task execution, and response generation. Our experiments demonstrate that Auto-SLURP presents a significant challenge for current state-of-the-art frameworks, highlighting that truly reliable and intelligent multi-agent personal assistants remain a work in progress.

## introduction multi-agent frameworks built on large language models (llms) have seen rapid development in recent years @xcite @xcite @xcite . these frameworks provide general-purpose infrastructures that facilitate the construction of multiagent systems through modular architectures, communication protocols, and coordination strategies. despite the rapid progress, there remains a noticeable gap in standardized benchmarks tailored to evaluate the effectiveness of these frameworks. while a number of benchmarks have been proposed to assess the tool-use capabilities of llms (qin et al., 2023; chen et al., 2023c; zhu * corresponding author et al., 2023; zhuang et al., 2024; ye et al., 2024), they primarily focus on individual llms and address only a narrow slice of functionality. as a result, they do not adequately reflect the complexity, interactivity, and coordination challenges inherent in real-world multi-agent scenarios. to capture broader dimensions of agent behavior, several social and interactive benchmarks have recently been proposed. for example, cooperation @xcite , sotopia @xcite , agentsense @xcite , and socialbench @xcite create social environments to evaluate agents' interpersonal and collaborative abilities. in parallel, agentbench @xcite targets reasoning and decision-making skills in domains such as coding, web navigation, and e-commerce. other works, including magic @xcite , cuisineworld @xcite , bat-tleagentbench @xcite , civrealm @xcite , and legalagentbench @xcite , introduce game-based or domain-specific settings to assess multi-agent interaction. meanwhile, benchmarks in embodied environments-such as agentboard @xcite , alfworld @xcite , the threedworld transport challenge @xcite , and wah @xcite )-focus on grounding agents in physical or simulated worlds. however, these efforts are typically designed to evaluate individual llms' task execution and interaction capabilities in multi-agent systems, rather than to assess the performance or flexibility of open-source multi-agent frameworks. moreover, the highly integrated nature of game-based and embodied environments often makes them difficult to adapt for evaluating general-purpose frameworks, limiting their reusability and extensibility @xcite . taken together, although significant progress has been made in benchmarking agent capabilities, ex-user could you please email john saying i'm on leave re-labeled original intent email_sendemail email_sendemail slots to_person: john, content: i'm on leave person : john one particularly compelling application is the smart personal assistant-an ai system capable of understanding natural language and performing tasks on behalf of users. this vision has long captured the imagination of both researchers and the public @xcite . despite significant progress in ai and the emergence of powerful llm-based multi-agent systems, this vision remains underexplored in the context of multiagent evaluation. personal assistants are expected to handle a wide range of tasks, such as checking the weather, sending emails, managing calendars, and controlling iot devices. achieving this level of functionality demands not only natural language understanding (nlu), but also sophisticated capabilities in decision-making, reasoning, tool use, coordination, and adaptability @xcite . to help fill this gap, we introduce auto-slurp, a benchmark designed to evaluate the effectiveness of llm-based multi-agent frameworks in building smart personal assistants. auto-slurp is built upon the slurp dataset @xcite , originally created for natural language understanding in smart home scenarios. we extend slurp's original intent-slot structure to support comprehensive end-to-end evaluation: from language understanding and intent interpretation, to task execution and response generation. to better reflect the complexity of real-world interactions, we relabel the slots and restructure the data to align with complete user-interaction pipelines. auto-slurp simulates realistic assistant interactions by integrating external services and simulated servers, enabling thorough evaluation of a framework's ability to handle complex, multi-step operations. these operations include api access, state management across modules, and coordina-tion between agents with specialized responsibilities. this setup allows us to assess not just whether multi-agent frameworks can interpret user commands, but also whether they can effectively orchestrate the backend processes needed to carry them out. the dataset spans a wide range of task domains, such as calendar management, media playback, transportation scheduling, and information retrieval. this diversity ensures that auto-slurp serves as a robust and representative benchmark for evaluating both the flexibility and reliability of multi-agent frameworks in realistic scenarios. our experimental results demonstrate that auto-slurp presents significant challenges even for state-of-the-art multi-agent frameworks. these findings underscore the complexity involved in achieving seamless, intelligent assistant behavior and reveal that we are still some distance away from building fully dependable ai-based personal assistants. to adapt slurp for our specific use case, we slot_name description descriptor additional constraints mentioned in the utterance content the content of an email or a search query from_person the sender in an email to_person the recipient in an email from_relation the sender's relationship to the user to_relation the recipient's relationship to the user setting an action that changes the state or configuration of a device the dataset encompasses a wide range of tasks, from straightforward actions like setting calendars or playing music, to more complex operations such as information retrieval or handling transportationrelated commands. we randomly select 1,000 samples from the slurp training set and 100 samples from the testing set. based on our experimental results, this subset is considered sufficient for training and testing llm-based multi-agent frameworks. detailed statistics on the domains and tasks in the test set are provided in appendix a. ## conclusion we present auto-slurp, a dataset designed to evaluate llm-based multi-agent frameworks. we assess the end-to-end execution tasks, not just the nature language understanding tasks. by incorporating simulated servers and external services, we evaluate the capacity of the frameworks to complete the entire process. the dataset proves to be sufficiently challenging to test the state-of-the-art multi-agent frameworks.