TD - EVAL : Revisiting Task-Oriented Dialogue Evaluation by Combining Turn-Level Precision with Dialogue-Level Comparisons (2025)

Abstract: Task-oriented dialogue (TOD) systems are experiencing a revolution driven by Large Language Models (LLMs), yet the evaluation methodologies for these systems remain insufficient for their growing sophistication. While traditional automatic metrics effectively assessed earlier modular systems, they focus solely on the dialogue level and cannot detect critical intermediate errors that can arise during user-agent interactions. In this paper, we introduce **TD-EVAL** (**T**urn and **D**ialogue-level **Eval**uation), a two-step evaluation framework that unifies fine-grained turn-level analysis with holistic dialogue-level comparisons. At turn-level, we assess each response along three TOD-specific dimensions: *conversation cohesion*, *backend knowledge consistency*, and *policy compliance*. Meanwhile, we design **TOD Agent Arena** that uses pairwise comparisons to provide a measure of dialogue-level quality. Through experiments on MultiWOZ 2.4 and Tau-Bench, we demonstrate that TD-EVAL effectively identifies the conversational errors that conventional metrics miss. Furthermore, TD-EVAL exhibits better alignment with human judgments than traditional and LLM-based metrics. These findings demonstrate that TD-EVAL introduces a new paradigm for TOD system evaluation, efficiently assessing both turn and system levels with an easily reproducible framework for future research.

## introduction task-oriented dialogue (tod) systems are conversational agents that help users complete specific tasks such as booking hotels, ordering food, or scheduling appointments. advances in large language models (llms) have significantly enhanced the capabilities and flexibility of modern tod systems @xcite i'm looking for an expensive restaurant in the center of town you are traveling to cambridge and looking forward to try local restaurants. you are looking for a restaurant. the restaurant should be in the centre and should be in the expensive price range. the restaurant should serve caribbean food. if there is no such restaurant, how about one that serves european food. ## preliminaries while various aspects can contribute to the satisfactory evaluation of dialogue systems, we focus on task completion scenarios, reviewing automatic metrics used in tod and analyzing their limitations in the context of modern dialogue systems. ## td-eval to comprehensively evaluate llm performance on tod tasks, it is critical to assess both turn-level and dialogue-level performances @xcite . we propose a two-step evaluation protocol, td-eval, as shown in figure 3 . the first step introduces a turn-level metric to evaluate tod responses across three key dimensions: conversation cohesion, backend knowledge consistency, and policy compliance. we chose these dimensions to specifically target the tod domain. the second step extends this evaluation from turn to dialogue-level with a pairwise ranking method. for each dialogue pair, the llm indicates a preference for the stronger conversation following prompted policies. this dual-step protocol offers a holistic view of tod performance, capturing both localized (per turn) and global (wholedialogue) qualities through a scalable, automated llm-as-judge approach. ## conclusion we present td-eval, a simple yet powerful framework for tod evaluation that combines finegrained turn-level checks with a holistic dialoguelevel ranking. by adopting an llm-as-judge paradigm, td-eval goes beyond standard metrics to reveal subtle yet critical errors, such as inconsistent database usage and policy violations, which often remain undetected by final-turn or dialogue-level summaries. through elo-based ranking and targeted turn-level scoring, our experiments on multiwoz 2.4 and ω -bench demonstrate td-eval's alignment with human judgments. this work opens a new path for llmdriven tod evaluation-one that is both flexible and transparent-ensuring greater accountability and accuracy in developing next-generation dialogue systems. we intend to release our framework, system responses, and human evaluations to foster reproducibility and community adoption.