A uto E val- T o D : Automated Evaluation of Task-oriented Dialog Systems (2025)

Abstract: Task-oriented Dialog systems (ToD) are essential in automating user interactions, but their complex design and dynamic nature make evaluation particularly challenging. Current evaluation methodologies heavily depend on human annotators, which can be inefficient, subjective, and expensive to scale. To advance the field, there is a pressing need for a reliable, scalable, and systematic evaluation framework that can provide comprehensive insights into ToD system performance. In this paper, we propose, AutoEval-TOD, an automated end-to-end evaluation framework using large language models (LLMs). Our framework first interacts with the ToD system and then assesses its performance across key dimensions by analyzing both the ToD’s responses and internal states. We validate our approach by applying it to multiple ToD systems, highlighting its adaptability and potential for widespread use in both research and industrial settings.

## introduction task-oriented dialog systems (tod) play a pivotal role in various industries, automating interactions across domains like customer service, technical support, and recommendation. these systems are designed to understand user input, manage multiturn conversations, and ultimately fulfill specific tasks, such as booking a flight or troubleshooting an issue. despite their widespread adoption, evaluating tod systems remains a significant challenge, owing to their complexity and the broad range of performance metrics that need to be considered. evaluating a tod system is far more intricate than simply checking whether a task is completed. it requires assessing several dimensions, such as response quality, dialogue coherence, retrieval accuracy, user satisfaction, and efficiency. moreover, many of these metrics are subjective, adding further complexity to evaluation processes. as a result, most companies and research teams rely on human annotators to evaluate tod systems. this reliance on human evaluation introduces several limitations. first, the subjective nature of human assessment can lead to variability in results, reducing the reliability of performance benchmarks. second, manual evaluation does not scale well as the volume of interactions increases, making it unfeasible for large-scale deployments. finally, human evaluators may introduce unconscious biases, potentially skewing results based on factors such as cultural background or personal preferences. to address these limitations, we propose an automated evaluation framework for tod systems that leverages the power of large language models (llms). as illustrated in figure 1 , the framework operates in three key stages: it first interacts with the tod system to simulate real user scenarios, and then it evaluates the system across multiple dimensions by analyzing both its responses and criterion ragas ragchecker raddle botsim god-eval autoeval-tod ## related work building a tod system is a very crucial application. recent developments in building tod leverage large language models (llms) to enhance tod capabilities. for instance, @xcite proposed a single-prompt technique for building entire tod systems, while @xcite explored function and tool calling capabilities of llms for tod construction. while frameworks for creating tod systems have become more accessible, their evaluation remains challenging due to the complexity of their components. ## proposed evaluation autoeval-tod is designed to provide a holistic evaluation by focusing on 5 critical dimensions essential for effective tod development: (1) retrieval performance, (2) user experience, (3) turn optimization, (4) domain compliance, and (5) response quality. these aspects were carefully selected to cover the full spectrum of capabilities required for tod to function effectively in realworld scenarios. autoeval-tod provides a comprehensive evaluation that goes beyond traditional metrics, offering insights into both the technical performance and practical usability of tod. ## dynamic user simulator to comprehensively evaluate tod systems, we utilized a llm-based conversation simulator by building upon the framework introduced by @xcite . it is designed to interact with the tod system while capturing both conversation logs and backend metadata for detailed analysis. this setup enhances the realism of our evaluations by incorporating dynamic conversations that simulate realworld interactions. in this simulation, a user llm is tasked with emulating a user (via prompt i.1) who has purchased a product and is now experiencing a seeded issue, drawn from the knowledge store, that requires resolution. the user llm engages naturally with the tod system, seeking solutions while the conversation and accompanying metadata are saved for evaluation. example of one such conversation is shown in figure 6 in appendix k. additionally, an oracle llm (with prompt h.2) plays a crucial role in ensuring unbiased evaluation. the oracle is assigned an 'oracle solution'-the correct resolution to the issue the user is facing. when the tod system recommends a solution, the user llm consults the oracle llm to verify whether the recommended solution aligns with the oracle's. this mechanism prevents the user llm from unin-tentionally leaking hints or solution details to the tod system, as shown in figure 2 , ensuring a fair evaluation. while the conversations are used for end-to-end evaluations based on both manual and automated metrics, the backend metadata helps gather specific data points for measuring the performance of individual modules. this combination provides a robust framework for evaluating the tod system's capabilities, ensuring consistency and adaptability to various user llm interactions. while our focus here has been on the troubleshooting domain in this section, it's important to note that this framework is highly versatile and can be readily adapted to other domains with minimal modifications, underscoring its broad applicability in various conversational ai contexts. ## implementation details to enable seamless integration, the framework is designed as a library that can be easily imported and used to evaluate any tod system. the system developer can configure the tod endpoint and database details, allowing the dynamic user simulator to select a seed issue (and its corresponding oracle solution) and trigger the tod endpoint with simulated user messages. the framework automatically handles high-level evaluation criteria, such as user experience, domain compliance, and response quality. for more granular metrics like retrieval performance and turn optimization, developers can activate specific functions within the tod system. the framework also offers flexible api access, enabling developers to log custom data and monitor system behavior as needed. this apidriven approach allows for tailored logging to meet the unique requirements of any tod system under evaluation. we utilize the claude-3-sonnet model for all llm-based evaluations in our framework. to validate llm performance, we employ a diverse group of human annotators. annotators are provided with detailed task-specific guidelines and asked to annotate a sample input for each task in the framework. ## conclusion we introduce autoeval-tod, a comprehensive automated evaluation framework for task-oriented dialogue (tod) systems. leveraging llms, our approach enables scalable, consistent, and multifaceted assessment of tod performance across key dimensions. by utilizing user and oracle llms to simulate real-world conversations, autoeval-tod provides nuanced insights into system behavior. experiments on multiple tod systems and datasets demonstrate its versatility and effectiveness, with strong alignment between automated evaluations and human judgments. autoeval-tod offers a standardized, reproducible evaluation methodology that can accelerate progress in tod research and development, facilitating rapid iteration and improvement of these critical systems.