DICE - BENCH : Evaluating the Tool-Use Capabilities of Large Language Models in Multi-Round, Multi-Party Dialogues (2025)

Abstract: Existing function-calling benchmarks focus on single-turn interactions. However, they overlook the complexity of real-world scenarios. To quantify how existing benchmarks address practical applications, we introduce DICE-SCORE, a metric that evaluates the dispersion of tool-related information such as function name and parameter values throughout the dialogue. Analyzing existing benchmarks through DICE-SCORE reveals notably low scores, highlighting the need for more realistic scenarios. To address this gap, we present DICE-BENCH, a framework that constructs practical function-calling datasets by synthesizing conversations through a tool graph that maintains dependencies across rounds and a multi-agent system with distinct personas to enhance dialogue naturalness. The final dataset comprises 1,607 high-DICE-SCORE instances. Our experiments on 19 LLMs with DICE-BENCH show that significant advances are still required before such models can be deployed effectively in real-world settings. Our code and data are all publicly available.

## introduction function-calling refers to the ability of llms to execute predefined external functions (or apis) through generating structured calls from natural language input @xcite @xcite . while early virtual assistants (vas) relied on rigid rule-based systems, llmintegrated vas now combine reasoning with external data retrieval @xcite . as interactions grow more complex, there is a growing need for vas to support multi-party and multi-turn dialogues @xcite . when exactly are we planning to ? pick it up i believe we were aiming for . mid-november mid-november works. shall we say around the ? 15th the sounds perfect. are we set on as our starting point? 15th san francisco san francisco seems like a convenient spot. we can make a more informed decision about our plans. let's proceed with . ai, please handle the details for us. finding a rental car find_car (" ", " ") 11-15 san francisco ai: i've found a toyota camry available for rental on november 15th in san francisco. single round -four party dialogue despite advancements, most function-calling benchmarks assume all api parameters are present in a single user utterance, overlooking real-world group chat scenarios @xcite @xcite . for example, when people in a group chat decide where to go and which flight to take, a va must be able to track multiple turns of dialogue to book a hotel and flight ticket. such complexities remain largely unaddressed by existing benchmarks. we therefore present dice-bench (dialoguebased interactive calling evaluation benchmark), a framework designed to evaluate function-calling performance in realistic multi-party, multi-round dialogues. in our paper, round is defined as a complete dialogue cycle consisting of multiple user utterances and system responses, and dependency as the condition where the current round's context benchmark # instances tool dialogue dice-score # tools dependency multi-party multi-round @xcite 17002 1645 ✗ ✗ ✗ 0.7895 toolalpaca @xcite 3938 400 ✗ ✗ ✗ 0.5660 toolllm @xcite 12657 16464 ✓ ✗ ✗ 0.5989 toolbench @xcite 2746 8 ✓ ✗ ✗ 0.7225 api-bank @xcite 2202 2211 ✗ ✗ ✗ 1.6318 metatool @xcite 21127 199 ✗ ✗ ✗ 0.5437 taskbench @xcite 17331 103 ✓ ✗ ✗ 0.6415 rotbench @xcite 945 568 ✗ ✗ ✗ 0.5651 dice-bench (ours) 1607 124 ✓ ✓ ✓ 3.6444 we also report dice-score for every dataset, showing that dice-bench handles more realistic tasks. in real-world group chats, key details often emerge across multiple turns, requiring accurate tracking for coherent interactions. to address this, we generate diverse dialogues using a multi-agent system, where each agent has a distinct persona. then, we refine the dataset through automated, rulebased, and human criteria-based filtering. after rigorous validation, our benchmark includes 1, 607 instances covering both single-round and multiround dialogues. existing benchmarks do not assess functioncalling in multi-round, multi-party dialogues, which makes accurate execution challenging due to the tool-related information being dispersed across turns. to quantify this complexity, we propose dice-score (dialogue information coverage evaluation score), which measures how fragmented tool-related details are within the input context. a higher dice-score indicates greater dispersion, requiring llms to integrate scattered information across turns. experiments on various llms show a significant performance drop as dice-score increases, underscoring the need for improved dialogue-tracking and contextintegration strategies. our contributions are as follows. ## dice-bench in this section, we introduce dice-bench, a benchmark designed to evaluate the functioncalling capabilities of llms in multi-round, multiparty dialogues. unlike previous approaches that concentrate on one-on-one human-llm interactions, dice-bench presents dialogue-based inputs in which multiple speakers provide scattered pieces of information over several turns. as shown in figure 2 , we also explicitly model inter-round dependencies using tool graph. this approach builds upon the concept introduced in taskbench @xcite . ## conclusion we introduce dice-bench, a benchmark for evaluating tool-calling in realistic multi-round, multiparty dialogues. by constructing and validating 1,607 dialogue instances, we demonstrate that current models struggle when critical information is scattered across multiple rounds and speakers. dice-score quantifies this dispersion and correlates with significantly lower model performance at higher scores. we intend for this dataset to encourage further research on integrating context across complex multi-party, multi-turn interactions, paving the way for more effective and realistic aipowered virtual assistants.