M ulti C hallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLM s (2025)

Abstract: We present MultiChallenge, a pioneering benchmark evaluating large language models (LLMs) on conducting multi-turn conversations with human users, a crucial yet underexamined capability for their applications. MultiChallenge identifies four categories of challenges in multi-turn conversations that are not only common and realistic among current human-LLM interactions, but are also challenging to all current frontier LLMs. All 4 challenges require accurate instruction-following, context allocation, and in-context reasoning at the same time.We also develop LLM as judge with instance-level rubrics to facilitate an automatic evaluation method with fair agreement with experienced human raters. Despite achieving near perfect scores on existing multi-turn evaluation benchmarks, all frontier models have less than 50% accuracy on MultiChallenge, with the top-performing Claude 3.5 Sonnet (October 2024) achieving just a 41.4% average accuracy.

## introduction evaluation benchmarks have played a critical role in the evolution of large language models (ziyu et al., 2023; porcu and havlínová, 2024), with benchmarks being continuously released to test large language models (llms) on various tasks @xcite @xcite . one such task is multi-turn conversation with humans, a common yet challenging endeavor. this is because conducting multi-turn conversations requires not only accurate instruction following, but also careful attention allocation to conversation context, and more importantly, near or surpassing human in-context reasoning capability. despite the growing demand and complexity of multi-turn llm-human conversations, comprehensive evaluation frameworks for this task remain limited @xcite to bridge this gap, we build multichallenge, a realistic and challenging benchmark designed to evaluate llms' ability to respond appropriately in multi-turn conversations. the benchmark consists of conversation histories of up to 10 turns between two parties, ending in a final user turn that contains a requirement or question. llms must generate an appropriate response to this final turn, taking into account the entire conversation context. we identify four challenge categories in multi-turn conversations that are not only common and realistic among current human-llm interactions, but also difficult for current frontier llms. they include instruction retention, inference memory of user information, reliable versioned editing, and self-coherence. instruction retention evaluates whether llms are able to follow instructions specified in the first user turn throughout the entire multi-turn conversation. inference memory of user information evaluates llms on recalling and connecting relevant details scattered in previous user turns when they are implicitly required to respond to the final user turn. reliable versioned editing evaluates whether llms can properly help humans revise existing materials through back-and-forth iterations with human users. finally, self-coherence evaluates whether llms can be reasonably coherent with model responses in the conversation history and avoid sycophancy (unconditionally agreeing to human users). we elaborate on the 4 challenges in section 3 and show examples of them in figures 1 and 2 . all 4 challenges require not only instruc-tion following, but accurate context allocation and in-context reasoning at the same time. we adopt a hybrid approach in which human raters and llms work together, to produce diverse, realistic, and challenging multi-turn test examples. we also develop an llm-as-a-judge system with instance-level rubrics to facilitate an automatic evaluation method with good agreement with experienced human raters. results from both human and automatic evaluation show that all frontier models have less than 50% accuracy on multichallenge, despite reaching near perfect scores on existing multi-turn evaluation benchmarks (achiam et al., 2023; yang et al., 2024), indicating the difficulty and usefulness of multichallenge in multi-turn conversation evaluation. among them, claude 3.5 sonnet (october 2024) achieves the best general performance, reaching a 41.4% average accuracy score on multichallenge. our experiments also show high correlation between human raters and our automatic judgments with instance-level rubrics, demonstrating the potential for quick and accurate multi-turn evaluations. we open source the data and code of this benchmark. 1 2 related work evaluation benchmarks on multi-turn conversations. while benchmarks have rapidly emerged to assess frontier capabilities of llms, such as ai as agents @xcite existing benchmarks, although widely adopted, no longer discriminate effectively among frontier llms, with models such as gpt-4, claude 3, and llama3 achieving near-perfect scores on them. for instance, mt-bench (zheng et al., 2023), which assesses general capabilities such as extraction and knowledge assessment, is now largely saturated. mt-eval (kwan et al., 2024) demonstrates a similar saturation, as evidenced by near-perfect scores from gpt-4 (achiam et al., 2023) in its original publication, and its "expansion" and "follow-up" challenges are now less demanding for modern models. more recently, multi-if (he et al., 2024) has been built to extend the ifeval benchmark @xcite to multi-turn and multilingual sce-narios. it focuses more on the auto-verifiable format and instruction following in multi-turn settings, such as maintaining all uppercase text, instead of the mixed set of model capabilities required in realistic multi-turn conversations such as in-context reasoning. in contrast to these works, multichallenge is specifically designed to be challenging for current models across this wider set of capabilities. it evaluates relevant skills separately, providing a more detailed analysis and revealing performance nuances that are otherwise obscured. appendix a.1 provides a detailed comparison of the benchmarks; multi-agent synthetic data generation. multiagent frameworks have emerged as a popular means to create diverse, high-quality synthetic datasets. agentinstruct (mitra et al., 2024) demonstrates this by utilizing multiple llm-powered agents in three agentic flows to generate synthetic text data with minimal human intervention. similar frameworks in other domains include splunk's mag-v (sengupta et al., 2024) for customer data and matrix @xcite for social dynamics data. in this work, we employ a multi-agent method to generate initial synthetic multi-turn conversations for 4 multichallenge challenges, followed by a hybrid approach with human editing and reviewing to produce the final benchmark. ## multichallenge in multichallenge, we focus on four challenge categories commonly encountered in realistic multiturn human-llm interactions, which we found difficult for current frontier llms. in this section, we provide specific definitions and test examples for each category and also outline our automatic evaluation method, enabling efficient and trustworthy adoption of multichallenge. comprehensive benchmark statistics are presented in ## the hybrid approach to build multichallenge producing realistic, diverse, and challenging test examples for multichallenge that can make most frontier llms fail, is difficult and time-consuming even for human experts. to facilitate this process and reduce costs while maintaining quality, we use a hybrid approach: initially generating synthetic data, then having experts review and edit it. in this section, we first describe our multi-agent synthetic data generation system and then illustrate the human editing process. ## experiments in this section, we first present the human evaluated performance of 6 frontier models, o1 these 6 llms are the same ones that we used to collect test examples. we analyze their performance across 4 challenge categories. we also use our automatic evaluation method for most opensource llms. lastly, we provide analysis and case studies on multichallenge, such as difficulty distribution and auto-eval trustworthiness. ## conclusion our introduction of multichallenge marks a significant advancement in evaluating large language models' (llms) capability at conducting multi-turn conversations with human users, a crucial but previously overlooked aspect. through a novel ai-human collaborative construction, multichallenge offers a comprehensive benchmark covering 4 multi-turn challenges that are realistic and currently challenging to frontier llms.