Spot The Bot: A Robust and Efficient Framework for the Evaluation of Conversational Dialogue Systems (2020)

Abstract: The lack of time efficient and reliable evalu-ation methods is hampering the development of conversational dialogue systems (chat bots). Evaluations that require humans to converse with chat bots are time and cost intensive, put high cognitive demands on the human judges, and tend to yield low quality results. In this work, we introduce Spot The Bot, a cost-efficient and robust evaluation framework that replaces human-bot conversations with conversations between bots. Human judges then only annotate for each entity in a conversation whether they think it is human or not (assuming there are humans participants in these conversations). These annotations then allow us to rank chat bots regarding their ability to mimic conversational behaviour of humans. Since we expect that all bots are eventually recognized as such, we incorporate a metric that measures which chat bot is able to uphold human-like be-havior the longest, i.e.Survival Analysis. This metric has the ability to correlate a bot’s performance to certain of its characteristics (e.g.fluency or sensibleness), yielding interpretable results. The comparably low cost of our frame-work allows for frequent evaluations of chatbots during their evaluation cycle. We empirically validate our claims by applying Spot The Bot to three domains, evaluating several state-of-the-art chat bots, and drawing comparisonsto related work. The framework is released asa ready-to-use tool.

## introduction evaluation is a long-standing issue in developing conversational dialogue systems (i.e., chatbots). the underlying difficulty in evaluation lies in the problem's open-ended nature, as chatbots do not solve a clearly-defined task whose success can be measured in relation to an a priori defined ground truth. automatic metrics have so far failed to show high correlation with human evaluations @xcite @xcite . human evaluation approaches are mainly classified according to the following: single-turn vs. multi-turn evaluation, and direct user evaluation vs. expert judge evaluation. single-turn analysis is usually performed by a human judge that rates a single response of the bot to a given context, whereas multi-turn analysis is often performed by a user that interacts with the bot and rates the interaction. single-turn ratings disregard the multiturn nature of a dialogue @xcite . although more and more multi-turn evaluations are performed, most of them are based on human-bot conversations, which are costly to obtain and tend to suffer from low quality @xcite . the instructions to be followed by annotators are often chosen ad-hoc and there are no unified definitions. compounded with the use of often criticized likert scales @xcite , these evaluations often yield a low agreement. the required cost and time efforts also inhibit the widespread use of such evaluations, which raises questions on the replicability, robustness, and thus significance of the results. in this work, we present the spot the bot framework, a cost-efficient evaluation methodology that can be used to rank several bots with regard to their ability to disguise as humans. it works as a multiturn-based evaluation with human judges. spot the bot is based on two observations: first, chatbots are trained on conversations between humans, and thus, they should be evaluated regarding their ability to mimic human behavior. second, the longer a conversation is, the more likely it is that a bot exhibits non-human-like behavior. spot the bot works by generating conversations between bots, then mixing these bot-bot conversations with human-human conversations and letting human judges decide for each entity in the conversations if it is a human or a bot. the conversations are rated at different points in time, which introduces the time-dependent component. this setting allows for two different analyses: a ranking based on pairwise comparisons of bots, and the application of the survival analysis, which computes the survival rate for each bot at different conversation lengths. furthermore, the human judges annotate the entities with respect to more fine-grained features, which can be chosen based on characteristics that the bots are expected to exhibit (e.g. fluency or informativeness). the survival analysis further allows to pin down the features that contribute to a dialogue system's survival, enabling interpretable results. we show that our framework produces reliable, repeatable results, while being quicker and more cost-effective to run than related approaches, as it does not rely on human-bot conversations and generally requires fewer annotations. furthermore, we show that disagreement between human annotators can be interpreted as a feature of a system's performance, rather than a weakness in the evaluation approach. we apply the framework to three well-known domains and common baselines and state-of-the-art systems to produce a stable ranking among them. we release the framework as a ready-to-use tool for evaluating dialogue systems into which different systems can be plugged and compared foot_0 . ## related work there exist various methods to evaluate dialogue systems, both automatic and human-based, but no single evaluation metric is widely agreed upon in the scientific community @xcite . automatic evaluation metrics for chatbots are known to correlate poorly with human ratings @xcite @xcite ), so we focus on human-based approaches, which can be classified in two dimensions: 1) single-turn vs. multi-turn approaches, and 2) approaches where the dialogue systems are judged by the user directly (interactive) or where judgments are made by objective experts, who do not participate in the dialogue (static). ## spot the bot in this section, we first provide an overview of the spot the bot framework and then describe the evaluation process's individual steps. ## conclusion in this work, we introduced spot the bot, a robust and time-efficient approach for evaluating conversational dialogue systems. it is based on conversations between bots rated by humans with respect to the bots' ability to mimic human behavior. we show that spot the bot yields robust and significant results while reducing the evaluation time compared to other evaluation frameworks. a team of researchers who would like to benchmark their system against four competing chatbots could do that for the cost of fewer than 3 hours of crowdsourced annotations. spot the bot facilitates developers making real progress based on frequent manual evaluations data, avoiding the use of noisy automatic metrics or once-in-a-year costly manual evaluations. we make the framework as well as the data publicly available.