Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approach (2021)

Abstract: Reliable automatic evaluation of dialogue systems under an interactive environment has long been overdue. An ideal environment for evaluating dialog systems, also known as the Turing test, needs to involve human interaction, which is usually not affordable for large-scale experiments. Though researchers have attempted to use metrics for language generation tasks (e.g., perplexity, BLEU) or some model-based reinforcement learning methods (e.g., self-play evaluation) for automatic evaluation, these methods only show very weak correlation with the actual human evaluation in practice. To bridge such a gap, we propose a new framework named ENIGMA for estimating human evaluation scores based on recent advances of off-policy evaluation in reinforcement learning. ENIGMA only requires a handful of pre-collected experience data, and therefore does not involve human interaction with the target policy during the evaluation, making automatic evaluations feasible. More importantly, ENIGMA is model-free and agnostic to the behavior policies for collecting the experience data, which significantly alleviates the technical difficulties of modeling complex dialogue environments and human behaviors. Our experiments show that ENIGMA significantly outperforms existing methods in terms of correlation with human evaluation scores.

## introduction one of the fundamental research bottlenecks for developing dialog systems falls in evaluation, namely how to measure the performance of these systems in an automatic and scalable manner. different from supervised natural language understanding tasks (e.g., text classification and machine translation), an ideal environment for evaluating dialog systems, also known as the turing test, involves multi-turn human interaction @xcite @xcite . while online platforms such as amazon mechanical turk can provide human-based evaluation, they are often expensive and not scalable. researchers have adopted language quality metrics for single-turn response generation given a fixed context (e.g., bleu score and perplexity) to implement automatic dialog systems evaluation @xcite @xcite @xcite . however, these metrics only weakly correlate to human evaluation in practice @xcite . one cause of such weak correlation is that language quality metrics rely on the exact match between generated text and ground-truth, which generally do not fully overlap. while certain embeddingbased metrics have been developed to combat this lack of coverage @xcite , they are only post-hoc judgments based on static experience data, and does not necessarily reflect the dynamic quality of multi-turn interactive dialog well @xcite . moreover, evaluation of goal-oriented dialog systems should be based on how well dialog systems collect information from users and whether the goal is completed; language quality metrics are thus unable to meet these requirements. to overcome the limitations of the aforementioned static evaluation methods, another line of work has proposed to model the interactive process of a conversation as a markov decision process (mdp) @xcite @xcite @xcite . accordingly, automatic evaluation of dialog systems can be formulated as an off-policy evaluation (ope) problem, where a human subject is the so-called "environment" in the reinforcement learning (rl) literature. for instance, @xcite propose a model-based approach for goaloriented dialog systems. they first learn an envi-ronment/human model from the experience data consisting of human response, and then evaluate a dialog agent/policy by executing the policy within the learned environment. this procedure is known as "self-play evaluation". such a model-based approach requires an accurate estimation of an environment/human. however, both the input and output of the environment are in a combinatorially large space, i.e., the trained model needs to be able to mimic complex human behavior of generating meaningful sentences from huge vocabulary. unfortunately, such a requirement is far beyond the current capability of model-based rl algorithms. as a result, evaluations that require accurate modeling of the environment are often unreliable. a similar model-based approach is proposed @xcite to evaluate open-domain chit-chat dialog systems. in addition to modeling human behavior, they also model the reward function (for mimicking the complex mechanism behind human ratings) based on handcrafted features, which makes evaluation even more unreliable. in this paper, we propose a general ope framework named enigma (evaluating dialog systems automatically) for estimating human evaluation score (i.e., how a human would rate a dialog system). different from the existing model-based approaches, which rely on complex modeling of human behavior given combinatorially large vocabulary, enigma takes advantage of recent advances in model-free ope and avoids direct modeling of dynamic transitions and reward functions in a complex environment. moreover, enigma overcomes several limitations of existing ope methods in order to evaluate dialog systems: (i) existing ope methods only apply to infinite or fixed horizon settings (where horizon length corresponds to number of turns in a conversation), while conversations, on the other hand, often have varying horizon lengths; we conduct thorough experiments on evaluating goal-oriented (airdialog, @xcite ) and chit-chat @xcite ) dialog systems to demonstrate the superiority of enigma. specifically, we follow the experimental settings similar to @xcite ; @xcite in section 4), and show enigma significantly outperforms the existing static and self-play evaluation methods. ## background an agent π is essentially a policy that maps s to p(a), where p(•) denotes the set of probability measures over the action space. a transition kernel p (•|s t , a t ) returns s t+1 as the state at turn t + 1, and an environment e generates a reward r @xmath0, a t ) ∈ [0, 1]. note that s t+1 essentially concatenates s t and a t with e t . the initial state s 1 = {e 0 } is randomly sampled from some distribution µ 0 . we follow the sparse reward setting, timators still require modeling behavior policies, they are still significantly easier than model-based ope, which has to model the underlying dialog environment. when interaction with human is prohibited, human-free automatic evaluation is required and off-policy evaluation (ope) @xcite is an appealing choice. in particular, ope can estimate ρ(π) based solely on pre-collected tuples {(s, a, r, s ope has been considered as one of the most fundamental problems in rl. a straightforward approach is to first directly learn an environment model (r and p ) from experience data and then estimate ρ(π) by executing the policy within the learned environment. such model-based ope exactly corresponds to the so-called "self-play evaluation" in the dialog system literature @xcite . unfortunately, it is notoriously difficult to specify a proper model for highly complicated environments such as a dialog environment (i.e., a human), where the state and action spaces are combinatorially large due to huge vocabulary size and complex transitions. as a result, the estimation error of the environment accumulates as interaction proceeds, and modelbased self-play evaluation of dialog systems often becomes unreliable @xcite . to address the challenge above, many modelfree ope methods that avoid direct modeling of the environment have been proposed. model-free ope can be categorized into behavior-aware and behavior-agnostic methods. specifically, behavioraware methods rely on either knowing or accurately estimating the probabilistic model of the behavior policies used for collecting the experience data (e.g., inverse propensity scoring, @xcite ). unfortunately, behavior policies are often unknown in practice. estimating their probabilistic models is also quite challenging, as it requires modeling human behaviors or complex dialog systems. behavior-agnostic methods, on the other hand, do not require explicit knowledge or direct modeling of behavior policies, and are therefore more favorable when experience data is collected by multiple (potentially unknown) behavior policies. unfortunately, most of the existing model-free behavior-agnostic ope methods focus on either infinite-horizon @xcite @xcite or fixed-horizon settings @xcite , and cannot be applied to evaluating dialog systems whose horizon (number of turns) vary between conversations. while lstdq @xcite can be adopted to handle varying horizons, it has been shown to not work well under the sparse reward setting @xcite . ## enigma we present the enigma framework for automatically evaluating dialog systems. in particular, enigma is model-free and agnostic to behavior policies for generating the experience data. enigma has three components: (1) pseudo-state padding for converting a dialog into an infinitehorizon mdp, (2) distribution-correction estimation (dice, @xcite ) with postnormalization for estimating the value of the target policy based on experience data, and (3) function approximation with pre-trained language models. ## experiments we empirically evaluate enigma on two dialog datasets: airdialog @xcite for goaloriented tasks and convai2 @xcite for open-domain chit-chat respectively. see details of experimental setup in appendix b. 2 4.1 policy training data and experience data as mentioned in section 3.2, there exists an information theoretic limit for all off-policy evaluation methods: no method can perform well when the state-action density ratio between the target and behavior policy is too large. to avoid such a circumstance, we need to ensure that the experience data collected by a behavior policy do not deviate too much from data induced by the target policy. unfortunately, both datasets used in our experiments do not satisfy such a requirement. air-dialog, for example, consists of dialog between humans, which are near-perfect golden samples as human agents almost always successfully book tickets for customers. dialog system agents, on the other hand, have many failure modes (i.e., the target policy/agent does not book the correct ticket for a human customer). hence, directly using human dialog as the behavior data to evaluate dialog agents is subject to limitations. in order to properly evaluate an imperfect target policy in the presence of the information theoretic limit, we refer to @xcite ; @xcite , and collect experience data using behavior policies similar to the target policy. to avoid confusion, we call data collected by the behavior policy "experience data" and data used to train an agent "policy training data". more details are elaborated below for each dataset. it is worth noting that existing work on dialog systems evaluation also enforces similar requirements. for example, @xcite show higher pearson correlation coefficient (0.37) be2 we release our source code for enigma algorithm here: https://github.com/google-research/ google-research/tree/master/dialogue_ ope/airdialogue_ope and dataset here https: //github.com/hmjianggatech/dialogue_ope_ data. ## discussions and conclusion existing research on automatic evaluation of dialog systems can be categorized into static vs. dynamic evaluation. most of existing research falls into static evaluation focusing on language quality of single-turn response or on task-completion given fixed dialog, while few literature emphasizes dynamic properties of an interactive environment and tries to considers the sequential interaction between a human and an agent, and thus it is more challenging. we note that in both static and dynamic evaluations, the algorithms rely on the assumption of sufficient data coverage (explicitly or implicitly) to ensure reliable evaluation. for example, in static evaluation, bleu score requires all reasonably good responses to be exactly covered by the experience data. more recently, @xcite show that their method only works when the behavior policies include the target policy. dynamic evaluation also assumes the sufficient coverage. we emphasize that it is the information-theoretic limit of all ope methods @xcite , which requires the experience data to cover sufficient target policy behaviors to ensure accurate estimation. therefore, we suggest the broader research community to release human-model interaction evaluation data to further promote research in automatic dialog systems evaluation. in this paper, we develop a model-free dynamic evaluation framework, enigma, which adopts the current state-of-the-art ope method in reinforcement learning. different from existing singleturn language quality metrics and model-based reinforcement learning methods, enigma naturally takes into consideration the interactive and dynamic nature of conversations, while avoiding the difficulty of modeling complex human conversational behaviors. our thorough experimental results demonstrate that enigma significantly outperforms existing methods in terms of correlation with human evaluation scores. one potential future direction is to extend enigma from offpolicy evaluation to off-policy improvement, which aims to learn a dialog system based on experience data @xcite .