MT -Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues (2024)

Abstract: The advent of Large Language Models (LLMs) has drastically enhanced dialogue systems. However, comprehensively evaluating the dialogue abilities of LLMs remains a challenge. Previous benchmarks have primarily focused on single-turn dialogues or provided coarse-grained and incomplete assessments of multi-turn dialogues, overlooking the complexity and fine-grained nuances of real-life dialogues. To address this issue, we introduce MT-Bench-101, specifically designed to evaluate the fine-grained abilities of LLMs in multi-turn dialogues. By conducting a detailed analysis of real multi-turn dialogue data, we construct a three-tier hierarchical ability taxonomy comprising 4208 turns across 1388 multi-turn dialogues in 13 distinct tasks. We then evaluate 21 popular LLMs based on MT-Bench-101, conducting comprehensive analyses from both ability and task perspectives and observing differing trends in LLMs performance across dialogue turns within various tasks. Further analysis indicates that neither utilizing common alignment techniques nor chat-specific designs has led to obvious enhancements in the multi-turn abilities of LLMs. Extensive case studies suggest that our designed tasks accurately assess the corresponding multi-turn abilities. The data and code are available at https://github.com/mtbench101/mt-bench-101.

## introduction large language models (llms) based chatbots have made remarkable advances and significantly enhanced dialogue systems. several benchmarks have been introduced to assess the capabilities of large language models (llms) in single-turn dialogues, e.g., @xcite , bbh @xcite , and alpacaeval @xcite . however, daily dialogues between users and chatbots usually involve multi-turn conversations @xcite figure 1 : mt-bench-101 encompasses three overarching abilities and thirteen distinct tasks within multi-turn dialogue scenarios, facilitating a granular benchmarking from basic perceptivity to advanced interactivity. on the right, a model with a broader range of abilities is considered better in multi-turn scenarios. in this paper, we introduce mt-bench-101, a new benchmark designed specifically for evaluating the chat capabilities of llms in multi-turn dialogues, as shown in figure 1 . during the ability modeling process for multi-turn dialogue, we undertake a systematical analysis combining realworld multi-turn dialogue data @xcite with the teaching taxonomy from educational psychology @xcite . this integrated approach has culminated in the formulation of a three-tier hierarchical ability taxonomy which is both datadriven and rooted in psychological frameworks. figure 2 illustrates the overall framework of our ability taxonomy. the first layer outlines three progressive overarching abilities which are depicted in figure 1 . perceptivity is the most fundamental ability, reflecting the model's accuracy in understanding context. adaptability is built upon this foundation, indicating the model's ability to respond effectively to user feedback. finally, interactivity captures the capacity of models for proactive engagement with humans, which is crucial for excelling in multi-turn interactions. the second tier specifies seven detailed abilities, while the third tier further decomposes these abilities into 13 distinct tasks. this taxonomy provides evaluation results across three levels from general to detailed, allowing for the identification of deficiencies in models at varying levels of granularity. for each third-tier task, we meticulously design specific prompts and utilize gpt-4 for data generation. in total, mt-bench-101 encompasses 4208 turns within 1388 multi-turn dialogues. for evaluation, we utilize golden context as dialogue history allowing llms to generate more fluid and rational dialogues, and we follow @xcite @xcite to utilize gpt-4 as a stand-in for human raters to score for each turn. we design unique scoring guidelines for each task and use the lowest round score as the total score for the dialogue to allow for a more rational assessment. we then perform extensive experiments on mt-bench-101 to assess the multi-turn chat ability of existing llms, including 2 close-sourced llms and 19 open-sourced llms. our findings include: ## mt-bench-101 this section begins with a detailed description of the three-tier hierarchical ability taxonomy for multi-turn dialogues. following that, we explain the methodology used to collect the dataset. finally, we present an analysis of the dataset's statistics. ## limitations with llm technologies rapidly evolving, new multi-turn capabilities are likely to emerge. consequently, the findings of this study may not encompass all multi-turn abilities. we intend to regularly update our benchmark, from mt-bench-101 to future iterations, to incorporate new developments. ## ethics statement we collected and annotated data using gpt-4 and had it reviewed by humans. additionally, we obtained the participants' informed consent and ensured their privacy and autonomy. all participants were fully aware of and consented to the annotation process. we have taken rigorous steps to ensure that the dataset is devoid of offensive content or personal identity information. however, there might still be residual errors or biases due to inadvertent mistakes by gpt-4 or oversights by annotators. we have made our best effort to rectify these issues, but it's challenging to eliminate them entirely. these issues may be present in all similar datasets. furthermore, the dataset will be publicly available and could be misused for training, which might make our benchmark less effective. as llms continue to evolve, the current capability taxonomy for multiturn dialogues might be incomplete. in response, we will continue to release updated versions of the dataset to address data leaks and extend capabilities. lastly, the dataset released in this work is intended solely for research and may not be suitable for commercial use without additional verification.