MT -Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Models (2024)

Abstract: Large language models (LLMs) are increasingly used for complex multi-turn conversations across diverse real-world applications. However, existing benchmarks mainly focus on single-turn evaluations, overlooking the models’ capabilities in multi-turn interactions. To address this gap, we introduce , a comprehensive benchmark to evaluate the multi-turn conversational abilities of LLMs. By analyzing human-LLM conversations, we categorize interaction patterns into four types: recollection, expansion, refinement, and follow-up. We construct multi-turn queries for each category either by augmenting existing datasets or creating new examples using GPT-4 with a human-in-the-loop process to avoid data leakage. To study the factors impacting multi-turn abilities, we create single-turn versions of the 1170 multi-turn queries and compare performance. Our evaluation of 10 well-known LLMs shows that while closed-source models generally surpass open-source ones, certain open-source models exceed GPT-3.5-Turbo in specific tasks. We observe significant performance degradation in multi-turn settings compared to single-turn settings in most models, which is not correlated with the models’ fundamental capabilities. Moreover, we identify the distance to relevant content and susceptibility to error propagation as the key factors influencing multi-turn performance.

## introduction the rise of large language models (llms) is transforming our daily lives and professional endeavors with their growing capabilities. individuals increasingly rely on llm-based ai assistants for diverse tasks, such as coding assistance, summarizing text from documents, and devising business strategies @xcite . these tasks often require understanding complex instructions and adapting to evolving needs through multiple user interactions. moreover, it is crucial for llms to generate contextually coherent responses by retaining and recalling historical information. the ability of llms to engage in multi-turn conversations is often overlooked in existing evaluation frameworks. for instance, mmlu @xcite evaluates language understanding in multiple tasks using single queries, and mt-bench @xcite evaluates conversational ability using twoturn interactions without considering more turns and various conversation types. to bridge the gap, we propose mt-eval, an evaluation benchmark to measure the capabilities of llms to conduct coherent multi-turn conversations. our analysis of interactions in lmsys-chat-1m @xcite reveals four predominant patterns when users interact with ai assistants: recollection, where the assistant must recall information from earlier turns; expansion, involving the exploration of varied topics within the main subject; refinement, where initial instructions are clarified or revised; and follow-up, consisting of questions based on the assistant's previous responses (see figure 1 ). these patterns are reflective of the majority of real-world multi-turn interactions with assistants. we then construct evaluation sets for each interaction type by augmenting existing datasets or creating new ones to cover real-world applications. we employ the gpt-4 with a human-in-the-loop process for generating new instances to avoid data contamination. the human annotators rigorously review and revise each instance to ensure quality, difficulty, relevance, and originality. a performant multi-turn conversational model should perform well in multi-turn interactions and demonstrate minimal performance difference from the corresponding single-turn scenario. therefore, we compare models using both response quality in the multi-turn setting and performance differences for identical queries in single vs. multi-turn settings. combining these two results provides a comprehensive view of their multi-turn conversational capabilities. we evaluate 10 popular llms, including both open-source and closed-source models. beyond general evaluation, we conduct in-depth analysis and ablation studies revealing how llms conduct multi-turn interactions and what affects their performance. our findings include: 1) the closed-source models still dominate in multi-turn conversational abilities, but some open-source models have comparable performance to gpt-3.5-turbo in some tasks. 2) most llms perform worse in the multiturn setting than in single-turn. the performance gap between the two settings is not related to the model's fundamental capacities. 3) increasing distance to relevant content negatively impacts performance. 4) models are prone to error propagation due to sensitivity to dialogue history. we summarize our contributions as follows: ## related work recent advancements in llms @xcite @xcite have significantly improved their ability to engage in human-like, multi-turn conversations. these models can now understand instructions, intentions, and context from human prompts, offering valuable responses @xcite . however, a limited number of studies have delved into the multi-turn conversation capabilities of llms. @xcite developed mt-bench, a dataset comprising 80 meticulously crafted multi-turn questions designed to evaluate the conversational flow and instruction-following capabilities of llms. nevertheless, the dataset's limited sample size poses a challenge, with each conversation consisting of only two turns. this constraint hinders the ability to broaden the evaluation scope or capture the intricacies of more extended conversational contexts. @xcite proposed halie, a framework for evaluating human-ai interaction. but its reliance on human participation limits its scalability and efficiency across different tasks. in specific domains, @xcite ## mt-eval mt-eval is designed to comprehensively evaluate the multi-turn conversation capabilities of llms across a wide range of real-world application contexts. by reviewing the existing authentic aihuman conversation datasets (like sharegpt and lmsys-chat-1m dataset @xcite )), we have identified and categorized four primary modes of engagement in user-assistant interactions: recollection: users present queries or tasks that necessitate the assistant's capacity to retrieve information from prior interactions, relying on the assistant's global context awareness and long-term memory capabilities. for instance, a user may in- follow-up 1. "the hobbit" is a children's fantasy novel written by j.r.r. tolkien. it was published . . . 2. . . . what are some other books written by j.r.r. tolkien? refinement format your answer in json. recollection tell me five facts about the book "the hobbit". begin all your responses in the upcoming conversation with the letter c. ok. expansion tell me five facts about the book "the hobbit". tell me five facts about the book "the great gatsby". 1. creation: "the hobbit" was created by j.r.r. tolkien as a standalone children's story . . . 2. . . . 1. "the hobbit" is a children's . . . 1. "the great gatsby" is a novel written by f. scott fitzgerald . . . 2. . . . tell me five facts about the book "the hobbit". { "facts": [ "\"the hobbit\" is a children's . . .", . . ., ] } tell me five facts about the book "the hobbit". "the lord of the rings" trilogy . . . 1. "the hobbit" is a children's . . . recollection accesses the model's ability to recall information from previous conversations. expansion evaluates the model's capacity to address queries surrounding the same topic. refinement gauges the model's adherence to progressively complex instructions. follow-up examines the model's proficiency in responding to queries that build upon its preceding response. a more detailed description of these tasks can be found in section 3. refinement: users clarify or modify their previous instructions. for instance, users might add more detailed constraints, like specifying the desired output format, or provide feedback to clarify their instructions. this requires the assistant's ability to keep track of the instruction changes and leverage feedback to refine its responses. follow-up: users ask questions that build upon the assistant's last response, often referencing specific details or opinions mentioned in that response. for instance, a user may seek additional information about a person mentioned in the assistant's prior response. this assesses the assistant's capacity to engage in coherent conversations. mt-eval includes test instances targeting these four conversation categories (see figure 1 ), while mirroring everyday scenarios of document processing, content creation, and information retrieval @xcite . it comprises 168 dialogue sessions with 1,170 turns to assess models' competence in handling such realistic multi-turn interactions. ## conclusion mt-eval represents an important first step in systematically evaluating and understanding llms' multi-turn conversational abilities. our experiment shows a pronounced gap between single-turn versus multi-turn performance across current models, a phenomenon that persists irrespective of the underlying capabilities of the models. our comprehensive analysis reveals that the distance to relevant content and susceptibility to error propagation are the key factors that cause a decline in multi-turn performance. we believe this work not only sheds light on the current limitations of llm's multiturn conversational abilities, it also paves the way for further efforts to close the identified gap and develop robust conversational models capable of multi-turn interactions.