clem:todd: A Framework for the Systematic Benchmarking of LLM -Based Task-Oriented Dialogue System Realisations (2025)

Abstract: The emergence of instruction-tuned large language models (LLMs) has advanced the field of dialogue systems, enabling both realistic user simulations and robust multi-turn conversational agents. However, existing research often evaluates these components in isolation, either focusing on a single user simulator or a specific system design, limiting the generalisability of insights across architectures and configurations. In this work, we propose clem:todd (chat-optimized LLMs for task-oriented dialogue systems development), a flexible framework for systematically evaluating dialogue systems under consistent conditions. clem:todd enables detailed benchmarking across combinations of user simulators and dialogue systems, whether existing models from literature or newly developed ones. To the best of our knowledge, clem:todd is the first evaluation framework for task-oriented dialogue systems that supports plug-and-play integration and ensures uniform datasets, evaluation metrics, and computational constraints. We showcase clem:todd’s flexibility by re-evaluating existing task-oriented dialogue systems within this unified setup and integrating three newly proposed dialogue systems into the same evaluation pipeline. Our results provide actionable insights into how architecture, scale, and prompting strategies affect dialogue performance, offering practical guidance for building efficient and effective conversational AI systems.

## introduction task-oriented dialogue systems @xcite @xcite @xcite @xcite help users complete specific goals, such as booking travel or making reservations, through multi-turn natural language interactions. these systems must accurately interpret user intent, manage ambiguity, and respond coherently. with the rise of llms, these systems @xcite have significantly improved in handling task-oriented, goal-driven conversations. in addition to enhancing dialogue system capabilities, llms now play dual roles: as end-to-end dialogue systems @xcite and as user simulators @xcite for training and evaluation. however, evaluation remains a challenge @xcite . current evaluations often suffer from inconsistent datasets, metrics, and compute settings, making it difficult to compare models or draw conclusions about system design. a recent paradigm for evaluating llms as agents, involves self-play in conversational "games". frameworks such as clembench @xcite , gameeval @xcite , spag @xcite , and textarena @xcite assess the capabilities of llms in instruction following, logical reasoning, and context retention across multi-turn settings. however, they do not address the evaluation of interactive systems built on llms, such as task-oriented dialogue (tod) systems. to address this gap, we introduce clem:todd 1 (see figure 1 ), an evaluation framework that builds on the self-play framework setup for tod. the user simulator and dialogue system act as the two players in the self-play setup, with a central controller (game master) managing turn-taking, task flow, and interactions with external tools such as databases. this setup supports consistent benchmarking, stress testing, and detailed analysis under shared datasets, metrics, and resource constraints. using clem:todd, we explore three research questions: (1) can llm-based self-play be adapted to evaluate interactive tod systems? (2) on the multiwoz2.2 benchmark, a widely used dataset for multi-domain tod, how do user simulators and dialogue systems perform across different models and architectures? (3) can clem:todd facilitate evaluation beyond multiwoz to plug in new instances, either derived from other datasets or generated synthetically, to enhance robustness testing? we benchmark both existing dialogue systems and three newly developed ones within our framework, providing systematic comparisons and insights across a range of configurations. our contributions are as follows: (a) we propose clem:todd, a self-play-based evaluation framework for systemlevel benchmarking of tod systems; (b) we reevaluate existing systems and our proposed dialogue systems under the same benchmarking setup; and (c) we analyse trade-offs across architectures, model sizes, and user simulators, with a focus on task performance and computational cost. ## related work our work builds on research in: llm-based tod systems, user simulators, task evaluation, and selfplay frameworks for multi-turn interaction. llm-based tod systems leverage llms in diverse architectural setups to enable successful task completion. @xcite proposed a modular system using llms across functional components, while instructtod @xcite and protod @xcite extended this design with performance-oriented enhancements. in contrast, autotod @xcite introduced a zero-shot monolithic system, showcasing the potential of llms as end-to-end systems. dard @xcite proposed a multi-agent framework, coordinating domain-specific llms via a central llm-based dialogue manager. llm-based task evaluation explores using llms as evaluators to assess dialogue quality and task success. recent work has prompted llms to act as judges in both open-ended @xcite and task-specific contexts @xcite @xcite @xcite . @xcite explored automatic dialogue evaluation using llms, and @xcite introduced llm-rubric, a framework for structured, rubric-based evaluation of language outputs. frameworks for dialogue system evaluation @xcite propose an evaluation framework for dialogue-system self-play in opendomain dialogue systems. @xcite discuss the importance of using a goal-oriented user simulator for improved evaluation. @xcite showcase the evaluation of different dialogue systems against a single user simulator. additionally convlab @xcite and parlai foot_0 offer infrastructure for developing dialogue agents, while dialevalmetrics @xcite focuses on evaluation. however, none provide a unified setup for controlled, plug-and-play experimentation across llm-based user simulators and dialogue systems, evaluated with consistent metrics and compute constraints. clem:todd fills this gap by enabling systematic exploration of architectural choices, simulator-system interactions, and scaling trade-offs. llm-based self-play frameworks self-play has recently been used to evaluate llms on multiturn tasks in frameworks like clembench @xcite , gameeval @xcite , spag @xcite , and textarena @xcite . these systems focus on model-level evaluation, while clem:todd adapts self-play for benchmarking full dialogue systems. building on these foundations, we propose clem:todd an evaluation framework that integrates llm-based user simulators, dialogue systems, and automatic evaluation. it supports flexible combinations of monolithic, modular, and multi-agent architectures and evaluates them under consistent metrics and compute settings. this design enables comprehensive benchmarking and reveals system interaction dynamics in controlled conditions. ## methodology we frame task-oriented dialogue as a form of "assistance game" @xcite , where one player needs assistance completing a task, and the other must work toward recovering their reward function. while @xcite address the learning problem, we use this framing to develop an evaluation framework for testing different approaches to building dialogue systems. ## experimental setup dataset we conduct experiments using the widely used multiwoz 2.2 dataset @xcite , which contains annotated dialogues across multiple domains such as restaurant, hotel, and train booking. importantly for our purposes, each dialogue comes with a verbally specified goal that was given to the 'customer' (e.g., "you want to book a chinese restaurant in the south of the city, for a party of 4"), and a structured representation of the dialogue outcome. for evaluation, we use only the test split (1000 tasks). to ensure a focus on end-to-end task completion, we filter the dataset to include only tasks that end in booking actions, resulting in 117 tasks (60 singledomain and 57 multi-domain dialogues) consisting of three domains (restaurant, hotel, and train). these filtered dialogues goals, after pre-processing to remove html artifacts, are directly used as input in the user simulator prompt. this filtering is specific to our evaluation goals and does not reflect a limitation of the clem:todd framework, which supports arbitrary task types and domains (more details in section 5.3). ## results we organize our results around the research questions introduced in section 1, providing quantitative comparisons and visualizations. ## conclusion we propose clem:todd a framework designed for the systematic evaluation of llm-based taskoriented dialogue systems. by leveraging a structured self-play paradigm, clem:todd enables consistent and modular assessment across diverse dialogue system architectures and user simulators. our experiments on standard (multiwoz) benchmarks, demonstrate that while large-scale monolithic models offer better performance at low computational cost, modular architectures, especially modular-llm variants, achieve reasonable tradeoffs between performance and efficiency. furthermore, we show that clem:todd can be adapted to evaluate dialogue systems in unseen and unrealistic domains beyond fixed datasets.