Dialogue Evaluation with Offline Reinforcement Learning (2022)

Abstract: Task-oriented dialogue systems aim to fulfill user goals through natural language interactions. They are ideally evaluated with human users, which however is unattainable to do at every iteration of the development phase. Simulated users could be an alternative, however their development is nontrivial. Therefore, researchers resort to offline metrics on existing human-human corpora, which are more practical and easily reproducible. They are unfortunately limited in reflecting real performance of dialogue systems. BLEU for instance is poorly correlated with human judgment, and existing corpus-based metrics such as success rate overlook dialogue context mismatches. There is still a need for a reliable metric for task-oriented systems with good generalization and strong correlation with human judgements. In this paper, we propose the use of offline reinforcement learning for dialogue evaluation based on static data. Such an evaluator is typically called a critic and utilized for policy optimization. We go one step further and show that offline RL critics can be trained for any dialogue system as external evaluators, allowing dialogue performance comparisons across various types of systems. This approach has the benefit of being corpus- and model-independent, while attaining strong correlation with human judgements, which we confirm via an interactive user trial.

## introduction with the rise of personal assistants, task-oriented dialogue systems have received a surge in popularity and acceptance. task-oriented dialogue systems are characterized by a user goal which motivates the interaction, e.g., booking a hotel, searching for a restaurant or calling a taxi. the dialogue agent is considered successful if it is able to fulfill the user goal by the end of the interaction. ideally, success rates are obtained via interaction with a real user in-the-wild. unfortunately, with a handful of exceptions, e.g., letsgo @xcite and alexa challenge @xcite , that is often out of reach. the closest approximation is human trials with paid users such as amazon mechanical turk workers, which has also been adopted as final evaluation in recent incarnations of the dialogue state tracking challenge (dstc) @xcite . however, such evaluations are highly time-and cost-intensive, making them impractical for optimization during an iterative development. the third alternative is to use a user simulator to conduct online dialogue simulation, however the result is subject to the quality of the user simulator itself. furthermore, developing such simulators is far from straightforward and requires significant amounts of handcrafting @xcite . only recently we have seen datadriven user simulators that can compete with handcoded ones @xcite . while there has been considerable progress towards more meaningful automatic evaluation metrics for dialogues, there remains a number of limitations as highlighted by the recent nsf report @xcite : the metrics 1) measure only a limited set of dialogue qualities, which mostly focus on subjective aspects such as fluency and coherence, 2) lack generalization across datasets and models, and 3) are not yet strongly correlated with human judgements. these limitations hinder a more widespread use of newly proposed metrics for benchmarking and comparison, especially with prior works. further, in particular for task-oriented dialogue systems, the need for reliable automatic evaluation of dialogue success is still unanswered. being able to automatically evaluate the success rate of any policy using static data offers a number of benefits in terms of required resources, generalizability, and reproducibility. furthermore, it is not only suitable for the final evaluation of a dialogue policy, but can also be utilized as an objective for iterative optimization. the corpus-based success rate is one such method, which has become the standard metric for state-of-the-art comparisons of policy optimization approaches today @xcite . unfortunately, this metric is computed based on pseudo-dialogues that may contain context mismatch. therefore, we believe it should be treated more as an approximation: it is insufficient at best, and misleading at worst, in reflecting real performance of dialogue systems. in addition, the rules used to check the goal completion need to be handcrafted based on the ontology, making this method data-or ontology-dependent. in this paper, we propose to use offline reinforcement learning (rl) to train a policy evaluator, also known as a critic, based on a static collection of dialogue data foot_0 . we show that an offline critic addresses the limitations of current automatic metrics: 1) it can be trained to evaluate any dialogue system architecture after-the-fact, allowing comparisons across various types of systems from prior works, 2) it can be utilized in the iterative development phase to optimize a dialogue policy, 3) it is theoretically grounded, solving the problems that standard corpus-based success rate has due to context mismatch, and 4) it strongly correlates with the performance of the system when interacting with human users, which we confirm via a user trial. ## related work for a long time, the research in dialogue policy has focused on user-centered criteria such as user satisfaction @xcite @xcite . the most reliable way to obtain these scores is to have users interact directly with the system and let them subjectively rate the system afterwards. due to the time and resource requirements to carry out such evaluations, human trials are usually done only as the final evaluation after the system development is finished. as the line between policy and natural language generation (nlg) tasks becomes blurred, we see the introduction of metrics such as bleu @xcite and perplexity. however, these have been labeled early on to be potentially misleading, as they correlate poorly with human judgement @xcite . this circumstance motivates automatic metrics that are highly correlated with human ratings @xcite . however, these metrics are designed to measure subjective quality of a dialogue response, making them more suitable for evaluating chat-based systems. despite the availability of toolkits that facilitate user simulation (us) evaluation @xcite , corpus-based match and success rates are the default benchmark for works in task-oriented dialogue systems today @xcite . these metrics are practical to compute, reproducible, and scalable. current standard corpus-based metrics are computed on a pseudo-dialogue constructed using user utterances from data and responses generated by the system. a set of rules then checks whether the system provides all information requested by the user. unfortunately, they do not take into account context mismatches that may originate from the pseudo-dialogue construction and therefore does not reflect other aspects of dialogue quality as the resulting dialogue flow is completely overlooked. there has been few applications of offline rl to dialogue systems. @xcite explores various language-based criteria, e.g., sentiment and semantic similarity, as reward signals for opendomain dialogue, paired with a kullback-leibler (kl) control for exploration within the support of the data. @xcite proposed using finetuned language models to utilize unlabeled data for learning the critic function. the method is however only demonstrated on a very small state and action space, and it is therefore unclear whether it generalizes to more complex set ups. @xcite applied offline rl with a pair-wise reward learning model based on preference learning, however it still utilizes the corpus-based success rate for choosing the preferred rollout. to the best of our knowledge, offline rl has not previously been deployed for dialogue evaluation. ## offline critic for dialogue policy evaluation and optimization the architecture of our proposed critic is depicted in figure 1(b) . we utilize recurrency to let the critic take dialogue context into account. we encode the word-level user utterance with an rnn and concatenate it with the binary belief state to obtain s t . on the other hand, the critic has the flexibility of taking any form of action. with latent actions, the action can be used as input directly by concatenating it with the state. when word-level or semantic actions are considered, a separate encoder can be used before concatenating it with the state. in addition, to leverage the available data as much as possible, we incorporate the user goal for estimating the return. the mdp then becomes the dynamic parameter mdp (dp-mdp) as described by @xcite , where a set of task parameters @xmath0, a t ; g) and reward function r(s t , a t ; g). it is safe to incorporate the user goal for learning, because the critic is only used for policy evaluation and not needed to run the policy. if the user goal is not given in the data, it can be automatically derived from the dialogue state. to maintain the correctness of the dialogue context, when predicting q(s t , a t ), all actions @xmath1. only a t is taken from the output of the policy. this is in contrast to the existing corpus-based success rate computation, where all @xmath2. to keep the critic pessimistic in the face of uncertainty, we implement a dropout layer and do k forward passes for each state-action pair and the lowest value is then taken as the final prediction, i.e., q(s t , a t ) = min k @xmath3. in this way, prediction with high variance, i.e., high uncertainty, is punished by taking the lower bound. this mechanism replaces the use of double critic in plas. ## conclusion we propose the use of offline rl for dialogue evaluation based on static corpus. while offline rl critics are typically utilized for policy optimization, we show that they can be trained for any dialogue system as external evaluators that are corpus-and model-independent, while attaining strong correlation with human judgements, which we confirm via an interactive user trial. not only does the offline rl critic provide a corpus-based metric that is reliable and efficient to compute, it also addresses a number of issues highlighted in the recently published nsf report @xcite . it is important to note that the proposed framework does not depend on the definition of states, action and rewards. so in principle, one could apply this method beyond task-oriented dialogue systems. for example, one could evaluate a number of chat-bots considering a corpus annotated only with level of engagement achieved in each dialogue and thus measure the level of engagement of the evaluated chat-bots.