C hat M atch: Evaluating Chatbots by Autonomous Chat Tournaments (2022)

Abstract: Existing automatic evaluation systems of chatbots mostly rely on static chat scripts as ground truth, which is hard to obtain, and requires access to the models of the bots as a form of “white-box testing”. Interactive evaluation mitigates this problem but requires human involvement. In our work, we propose an interactive chatbot evaluation framework in which chatbots compete with each other like in a sports tournament, using flexible scoring metrics. This framework can efficiently rank chatbots independently from their model architectures and the domains for which they are trained.

## introduction evaluation of dialogue systems is an open problem. existing automatic evaluation metrics for chitchat systems are similar to those for other text generation tasks (e.g., machine translation @xcite , question-answering @xcite , summarization @xcite ), which depends on calculating word overlaps with reference responses. however, for chitchats, there are usually many alternative but plausible responses given a situation, perhaps more than any other text generation task mentioned above. a limited number of reference responses are not sufficient to determine how good a generated response is. moreover, such static settings are not good at assessing an interactive, context-sensitive system. interactive human evaluation metrics usually involve a likert scale evaluation after a multi-turn conversation with the bot to be assessed. while this method is a step up from the previous static evaluation, it is difficult for human judges to give a concrete score to any bot. comparing the performance of two bots is easier. thus acute-eval @xcite asks the judges to make a binary judgment of who is better in conversations between two identical bots or between a human and a bot. a more advanced version of that is spot the bot @xcite which models the human evaluation of a conversation after the turing test. however, such a process is still time-consuming and costly, compared with automatic evaluations. in our opinion, a good method for evaluating multi-turn conversational model/system should satisfy the following requirements: i) be as efficient and inexpensive as possible; ii) can truly reflect a model's ability to conduct a human conversation; iii) evaluation results should correlate well with human judgments; iv) can be used to compare and rank the capabilities of a set of models/systems. toward that goal, in this work, we propose an automatic interactive evaluation framework, which is called chatmatch(cm) for chitchat agents. this framework can be used to rank a number of bots with little time and minimum human effort. above all, we want to emphasize the significance of direct interactions between bots in the evaluation. people tend to believe that human-bot conversations are more reliable and produce more comprehensive evaluations of chatbots' capabilities. this is not always true. as human annotators know their counterpart is a robot, they tend to ask common and goal-directed questions. on the other hand, some bot-bot chat logs in our experiments show that, surprisingly, conversations between different bots may expose their strengths and weaknesses never seen in human-bot conversations. figure 1 gives two small chat fragments, illustrating such differences. while talking about hobbies, human keeps asking the bot some blunt questions, which leads to dull responses from the bot. however, in a bot-bot setting, two bots, including the same bot in the previous conversation, start explaining their hobbies to each other, producing a more interesting conversation. human: what's your hobby? plato-2: i've been playing the drums. human: why? plato-2: it's just something i've always done. our framework consists of two components: competition and scoring, which interoperate with each other. the competition is modeled after most sports tournaments such as soccer or ping pong. there are three levels of competitions. from bottom up, they are: game-level, match-level and tournament-level. each match consists of several games. during a game, two bots will converse freely with each other and a virtual judge will score their performances according to a set of userdefined criteria such as consistency and fluency, etc. these criteria are flexible and extensible. the main contributions of this paper are: ## approach in this section, we first introduce the general framework of chatmatch, which is modeled as a sports tournament, then discuss some possible scoring metrics (or dimensions) that can be used by the virtual judges in these matches. ## experimental setup this section describes the chatbots that we experiment with, the set of baseline approaches that are compared to chatmatch, and some implementation details used in the following experiments. foot_1 ## results and analysis in this section, we first present the end-to-end results from our automatic framework and other baselines. then we show that cm can generalize to different set of bots by applying cm on any 4-bot combination of the bot pool and check the results. finally, we analyze the design of the framework with ablation tests in detail. ## related work two major approaches are used for evaluating chatbots or dialogue systems. some evaluate a single turn at a time by comparing the response with a ground truth utterance from a static script of real human dialogue or get a score by combining the response with the context. others evaluate the interaction between human and bot or between bot and bot by some scoring metrics. we will present some typical systems from each of these approaches below and discuss their pros and cons. most existing evaluation systems are based on static scripts. traditional metrics such as bleu @xcite , rouge @xcite and meteor @xcite are widely used for evaluating text generation systems. more recently, automatic evaluation methods based on static scripts are gradually moving toward using pre-trained language models. pang et al. (2020) uses gpt-2 @xcite and @xcite uses roberta @xcite to automatically evaluate the generation in an unsupervised way with a higher correlation to human evaluation than traditional ones. however, these static evaluation metrics which need fixed contexts are not flexible enough to assess chatbot's ability, in need of adapting to dynamic and changing contexts. we argue that interactive systems like chatmatch are more promising as they test chatbots in a real conversation mode. interactive evaluation systems attract increasing attention lately. @xcite and @xcite use dialogues between a bot and itself, which is called self-talk, to evaluate the bot in a more automatic manner. but it often leads to a lot of repeated chat context. @xcite designed spot the bot, a framework that enables a group of bots to chat with each other and then asks humans to annotate if the bots talk more like a human or more like a bot. prior to our work, there was still no interactive and automatic evaluation framework that works without any participation of human annotators. our work fills this gap and moreover, it's very flexible since more complex metrics or algorithms can be plugged in as scoring functions. our results also show that there is a strong correlation between automatic evaluation results and human judgments. ## conclusion in this work, we present a new automatic evaluation framework called chatmatch. we first make the chatbots converse directly with each other. then we use a three-level rule-based scoring framework to rank their performances which mimics the process of a double round-robin tournament. our framework shows a good correlation with human judges, better than state-of-the-art automatic and semi-automatic chatbot evaluation frameworks. another remarkable advantage of our framework is that it's totally automatic and time-saving which costs 2 min 57 secs on average to get the final ranking results among 7 chatbots, much faster than manual evaluation. we believe that this kind of automatic interactive evaluation framework opens up new opportunities for future research on dialogue system evaluation.