B ot C hat: Evaluating LLM s’ Capabilities of Having Multi-Turn Dialogues (2024)

Abstract: In the realm of modern Large Language Models (LLMs), facilitating high-quality, multi-turn dialogues with humans represents a cornerstone feature. However, human-based evaluation of such a capability involves substantial manual effort. This study offers a formative assessment of current LLMs’ proficiency in emulating human-like, multi-turn conversations using an LLM-centric approach. The evaluation encompasses three key elements in the evaluation pipeline: utterance generation, evaluation protocol, and judgement, and we delve deeply into each aspect. GPT-4, both as an utterance generator and as a judge, exhibits exceptional performance. As a generator, GPT-4 crafts dialogues indistinguishable from human interactions in terms of style and flow. When judging, it shows a heightened alignment with human evaluative standards and consistency. Conversely, other LLMs face challenges in producing quality multi-turn dialogues, hindered by inadequate instruction-following abilities, a propensity for prolix utterances, and overall limited capabilities. Notably, generating extensive dialogues (e.g., spanning tens of turns) remains a formidable task for most LLMs, particularly in Chinese contexts. We hope that our work can serve as a valuable resource for evaluating the multi-turn chatting capabilities of LLMs. Related resources are available at https://github.com/open-compass/BotChat.

## introduction the evolution of large language models (llms) @xcite @xcite ) marks a transformative phase in artificial intelligence, significantly surpassing traditional language models @xcite @xcite in figure 1 : botchat evaluates multi-turn dialogue performance by prompting chatbots to generate dialogues from initial human utterances, followed by evaluation through a judge llm. despite their advanced capabilities, not all llms consistently deliver satisfactory performance in multi-turn human interactions. in practical applications, it has been observed that dialogues generated by certain llms frequently fail to meet user satisfaction criteria. the issues manifest in multiple aspects, including poor adherence to user instructions, undesirable tone, lengthy utterances, and the generation of repetitive content. the evaluation of these conversational capabilities remains a complex challenge. traditional approaches, primarily human-based @xcite , heavily involve manual labor for human-bot conversation generation and quality assessment. this paper proposes a more efficient paradigm, named botchat, to evaluate multi-turn chatting capability. botchat is an entirely llm-based approach, eliminating the need for manual labor. the methodology comprises two stages: dialogue generation and quality assessment. initially, we use the very first utterances (chatseed) from multilingual realworld conversations @xcite for utterance-by-utterance dialogue generation by chatbots. in each step, a chatbot generates one utterance based on all history utterances. this process autonomously generates dialogues with varying numbers of turns. in the second stage, we assess dialogues using different judge llms and a suite of llm-based evaluation protocols. our experiments demonstrate that gpt-4 excels in human alignment and self-consistency compared to other llms. we introduce three evaluation protocols: unieval (individual dialogue evaluation), paireval (comparative evaluation of two dialogues), and gteval (comparison with a corresponding human dialogue). while unieval and paireval are applicable to dialogues with an arbitrary number of turns, gteval is limited by the extent of the ground-truth dialogue. additionally, addressing the unique challenges posed by repetitive utterances, which are common in chinese conversational scenarios, we present dupdetect for preprocessing unnatural dialogue evaluations, thereby also reducing evaluation costs. with the evaluation protocols, we compare representative llms, ranging from the state-of-theart closed-source @xcite to smallscale open-source llms @xcite . during the evaluation, three evaluation protocols draw substantially identical conclusions. gpt-4 generates human-style multi-turn conversations with impressive quality, outperforming all other llms. for all llms, the quality of generated dialogues declined quickly as the number of dialogue turns increases. such degradation is particularly evident for small-scale open-source llms, compared to the top-tier llm, gpt-4. in particular, this phenomenon is more pronounced in the chinese context than in the english one. with qualitative assessment, we find that llms fail to generate multi-turn conversations with desirable quality primarily due to: poor instruction-following capability, a tendency to generate lengthy utterances, and limited general capability. ## botchat in this section, we delineate the evaluation paradigms incorporated into botchat. the primary aim of this framework is to evaluate llms' conversational abilities in alignment with human subjective preferences. we begin with a comprehensive overview of the workflow for generating multi-turn dialogues. subsequently, we introduce three distinct evaluation strategies: unitary evaluation (unieval), pairwise evaluation (paireval), and ground-truth evaluation (gteval). additionally, in response to the unique challenges posed by repetitive utterances (which frequently occur in chinese conversational scenarios), we introduce dupdetect. ## conclusion in this paper, we design a proxy evaluation paradigm botchat to measure the multi-turn conversational capabilities of large language models. botchat evaluate chatbot generated dialogues with an llm judge, to emancipate heavy human labor from the evaluation. we design multiple evaluation protocols and adopt them to evaluate dialogues generated by 14 modern llms. we find that a large proportion of llms excel at having dialogues of limited turns. however, when the turn number is large, only a few llms (gpt-4, vicuna-v1.5-13b, etc.) achieve satisfying performance. we hope that botchat can serve as a valuable resource on the journey towards automated evaluation of multi-turn conversational capability.