Rethinking Stateful Tool Use in Multi-Turn Dialogues: Benchmarks and Challenges (2025)

Abstract: Existing benchmarks that assess Language Models (LMs) as Language Agents (LAs) for tool use primarily focus on stateless, single-turn interactions or partial evaluations, such as tool selection in a single turn, overlooking the inherent stateful nature of interactions in multi-turn applications. To fulfill this gap, we propose DialogTool, a multi-turn dialogue dataset with stateful tool interactions considering the whole life cycle of tool use, across six key tasks in three stages: 1) tool creation ; 2) tool utilization : tool awareness, tool selection, tool execution; and 3) role-consistent response : response generation and role play. Furthermore, we build VirtualMobile – an embodied virtual mobile evaluation environment to simulate API calls and assess the robustness of the created APIs. Taking advantage of these artifacts, we conduct comprehensive evaluation on 13 distinct open- and closed-source LLMs and provide detailed analysis at each stage, revealing that the existing state-of-the-art LLMs still cannot perform well to use tools over long horizons .

## introduction large language models (llms) often rely on various tools to engage with external environments @xcite , in order to overcome their inherent limitations by using upto-date information @xcite , domainspecific information @xcite or external knowledge, such as knowledge graphs @xcite , as well as named tool learning @xcite . therefore, there are many previous studies that have been devoted to constructing benchmarks to evaluate the ability of llms to use different tools on various downstream environments/tasks @xcite @xcite , 2024b). however, these efforts predominantly focus on stateless single-turn interaction, while overlooking the stateful tool use in the multi-turn interactions @xcite @xcite . for instance, when a user fails to provide all the required arguments to use a tool in a single turn or requests details about a previous tool call, it becomes infeaible to provide detailed response without the tracking of tool states. in addition, most of existing benchmarks or environments fail to address the complexities of real-world interactions across the entire lifecycle of tool use, encompassing tool creation, selection, execution, and integration of final responses, especially for tools with varying numbers and types of arguments @xcite . ## analysis to offer a comprehensive evaluation of whole lifecycle of stateful tool use, we conduct error analyses for tool creation and tool utilization. in addition, we also explore the effects of different selection ## conclusion we introduce dialogtool, which is the first multiturn interactive benchmark considering the whole life cycle of tool utilization of dialogue agents, spanning across three distinct stages, six different tasks. furthermore, we build virtualmobile, an virtual mobile evaluation environment to simulate api calls and return corresponding results. we hope these benchmark and environment will provide a comprehensive platform for evaluating and advancing dialogue agents' tool utilization capabilities, fostering future research in multi-turn interactions.