CMT -Eval: A Novel C hinese Multi-turn Dialogue Evaluation Dataset Addressing Real-world Conversational Challenges (2025)

Abstract: Multi-turn dialogue is a key paradigm for interaction between users and Large Language Models (LLMs). However, existing evaluation benchmarks fail to capture users’ evolving needs and how their diverse conversation styles affect the dialogue flow. To address these limitations, we propose CMT-Eval, the first dedicated dataset for fine-grained evaluation of Chinese multi-turn dialogue systems. Built upon a linguistic theory-driven Speech Act Framework, diverse user personas, and varied conversational challenges, CMT-Eval comprises 596 high-quality dialogues with 4,431 turns, simulating realistic, multifaceted, and challenging conversations. Experiments reveal that models struggle with specific speech acts, user personas, and complex scenarios, highlighting the effectiveness of CMT-Eval in assessing LLMs’ multi-turn dialogue capabilities and providing valuable insights for their enhancement. The dataset, code, and prompts are available at https://github.com/hejaida/CMT-Eval .

## introduction as large language models (llms) become increasingly capable, people rely on them for a wide range of tasks, from everyday help to thoughtprovoking discussions, where interactions predominantly take the form of multi-turn dialogues. recent work @xcite @xcite has highlighted critical areas for improvement in complex real-world tasks involving multi-turn instructions, underscoring the need to evaluate models' multi-turn dialogue capabilities. however, existing evaluation benchmarks primarily employ multiple-choice or single-turn questions @xcite . recent efforts have made pioneering progress toward multi-turn dialogue evaluation, especially in exploring diverse interaction patterns @xcite * corresponding author. and hierarchical, fine-grained assessment of conversational skills @xcite . meanwhile, this direction has also expanded into diverse domains, including agents, math, coding, role-playing, medical llms, and intelligent tutors @xcite @xcite . despite these advancements, current efforts still face challenges, such as limited dialogue turns (typically 2-3) @xcite @xcite , single-task focus @xcite , and standardized datasets that may not fully capture the evolving nature of user needs and the diversity of expression styles in natural, longer conversations. moreover, most benchmarks focus on english evaluation, underscoring the lack of datasets in chinese and other languages. in light of these issues, we propose cmt-eval, the first dedicated dataset for fine-grained evaluation of chinese multi-turn dialogue systems. it is designed to closely mirror real-world conversational scenarios, with a user-centered design that captures how users naturally express and adapt their needs in multi-turn interactions. as illustrated in figure 1 , our three-step data construction pipeline integrates diverse user personas into data collected from real-world sources @xcite , reflecting the heterogeneity of real-world users. guided by @xcite 's linguistic theory, we introduce a hierarchical speech act framework that simulates the evolving needs of users throughout the dialogue. to impose greater challenges on dialogue systems, we construct two additional subsets featuring more complex constraints and long-text questions. through this pipeline, cmt-eval comprises 596 high-quality multi-turn dialogues across 4,431 turns, organized into three distinct subsets. we assess the multi-turn dialogue capabilities of a series of llms on cmt-eval. results reveal that models exhibit relative weaknesses in information synthesis during longer conversations. they also struggle with specific speech acts, such as topic shift, and exhibit poor adaptability to certain user personas. furthermore, their performance deteriorates in more challenging scenarios, such as those involving global constraints and ambiguous expressions. this demonstrates the value of cmt-eval in mimicking real-world user-llm interactions and enabling in-depth, fine-grained evaluation of model performance. in summary, this paper makes three main contributions: • we present cmt-eval, the first systematic chinese multi-turn dialogue evaluation dataset, filling a critical gap in the current landscape. ## related work llms have demonstrated impressive capabilities in language understanding, generation, and reasoning, spurring the development of various evaluation benchmarks, such as mmlu @xcite , helm @xcite , cmmlu @xcite , and c-eval @xcite . however, most of them assess llm performance using multiple-choice and single-turn questions, overlooking the fact that the primary mode of user-llm interaction is multi-turn dialogues. recently, a few studies have focused on assessing the dialogue generation capabilities of llms, e.g. botchat @xcite ). an increasing number of works have also begun exploring multi-turn dialogue evaluation. @xcite identifies eight common categories of user prompts and constructs 80 two-turn dialogues. mint @xcite emphasizes models' ability to leverage tools and natural language feedback for problem-solving across multi-turn dialogues. mt-eval @xcite analyzes human-llm conversations and proposes four interaction patterns: recollection, expansion, refinement, and follow-up. it extends dialogues to an average of seven turns. however, each dialogue is constrained to evaluate a single pattern within traditional nlp tasks, thus failing to fully capture user intents in multi-turn dialogues. mt-bench-101 @xcite advances the field by introducing a three-tier hierarchical taxonomy with 13 fine-grained evaluation tasks and constructs a largescale dataset. nevertheless, these conversations are directly generated by models, differing from realworld user queries, and each conversation averages only three turns. similar to mt-eval @xcite , it evaluates one task per dialogue. in addition, multi-turn dialogue evaluation in chinese remains underexplored. superclue-open (xu et al., 2023) includes 300 two-turn dialogues by manually adding a follow-up question to single-turn queries, while @xcite proposes a framework for evaluating multiturn psychological counseling. these limitations highlight the need for a systematic multi-turn dialogue evaluation dataset especially in chinese. ## methods this section outlines the construction of cmt-eval, covering the speech act framework, user personas, and data pipeline (see figure 1 ), and concludes with the evaluation method. ## discussions in this section, we further examine model performance with respect to different speech acts and user personas, offering deeper insights into model behavior. figure 6 presents the mean scores of different speech acts across dialogue turns on standard foot_1 . among the speech acts, topic shift consistently yields the lowest scores, although it exhibits a gradual improvement as the dialogue progresses. specifically, the topic transition pattern allows each dialogue two topic shifts: one from the original topic to a new one and another returning to the original topic, with the new topic spanning an average of 2.5 turns. our analysis reveals that the second topic shift receives a higher average score than the first (4.43 vs. 3.99), suggesting that models perform better with the original topic compared to the challenge of introducing a new one. ## conclusion we propose cmt-eval, the first systematic dataset for fine-grained evaluation of chinese multi-turn dialogue systems. it comprises 596 high-quality dialogues with 4,431 turns, addressing a critical gap in this field. built upon a linguistic theory-driven speech act framework, diverse user personas, and varied conversational challenges, cmt-eval simulates realistic, multifaceted, and challenging inter-actions that closely align with real-world scenarios. beyond dataset construction, we conduct experiments showing that models struggle with certain speech acts and user personas, especially under global constraints or ambiguity, and suggest several directions for improvement such as better tailoring responses to user characteristics. overall, cmt-eval enables effective multi-turn evaluation, informs dataset design, and offers guidance for dialogue strategies in dialogue systems. future work may include enriching the diversity and coverage of user personas, scaling the dataset to support supervised fine-tuning, and adapting both the dataset and construction methodology for other languages.