Learning an Unreferenced Metric for Online Dialogue Evaluation (2020)

Abstract: Evaluating the quality of a dialogue interaction between two agents is a difficult task, especially in open-domain chit-chat style dialogue. There have been recent efforts to develop automatic dialogue evaluation metrics, but most of them do not generalize to unseen datasets and/or need a human-generated reference response during inference, making it infeasible for online evaluation. Here, we propose an unreferenced automated evaluation metric that uses large pre-trained language models to extract latent representations of utterances, and leverages the temporal transitions that exist between them. We show that our model achieves higher correlation with human annotations in an online setting, while not requiring true responses for comparison during inference.

## introduction recent approaches in deep neural language generation have opened new possibilities in dialogue generation @xcite . most of the current language generation efforts are centered around language modelling or machine translation @xcite , which are evaluated by comparing directly against the reference sentences. in dialogue, however, comparing with a single reference response is difficult, as there can be many reasonable responses given a context that have nothing to do with each other @xcite . still, dialogue research papers tend to report scores based on word-overlap metrics from the machine translation literature (e.g. bleu @xcite , meteor @xcite ). however word-overlap metrics aggressively penalize the generated response based on lexical differences with the ground truth and correlate poorly to human judgements @xcite . one can build dialogue evaluation metrics in two ways: referenced metrics, which compare the generated response with a provided ground-truth response (such as the above word-overlap metrics), or an unreferenced metrics, which evaluate the generated response without any such comparison. @xcite propose a learned referenced metric named adem, which learns an alignment score between context and response to predict human score annotations. however, since the score is trained to mimic human judgements, it requires collecting large-scale human annotations on the dataset in question and cannot be easily applicable to new datasets @xcite . recently, @xcite proposed a hybrid referenced-unreferenced metric named ruber, where the metric is trained without requiring human responses by bootstrapping negative samples directly from the dataset. however, referenced metrics (including ruber, as it is part referenced) are not feasible for evaluation of dialogue models in an online setting-when the model is pitched against a human agent (model-human) or a model agent (model-model)-due to lack of a reference response. in this setting, models are usually eval-uated directly by humans, which is costly and requires careful annotator training @xcite . the contributions of this paper are (1) a completely unsupervised unreferenced metric maude (metric for automatic unreferenced dialogue evaluation), which leverages state-of-the-art pretrained language models @xcite , combined with a novel discoursestructure aware text encoder and contrastive training approach; and (2) results showing that maude has good correlation with human judgements. ## background we consider the problem of evaluating the response of a dialogue system, where an agent is provided with a sequence of sentences (or utterances) @xmath0, u 2 , ..., u n } (termed as context) to generate a response @xmath1. each utterance, u i , can be represented as a set of words u @xmath2, w 2 , ..., w n }. an utterance u i can be represented as a vector as h @xmath3, where f e is an encoder that encodes the words into a fixed vector representation. this work focuses on the evaluation of generative neural dialogue models, which typically consist of an encoder-decoder style architecture that is trained to generate u n+1 word-by-word @xcite . the response of a generative model is typically evaluated by comparing with the ground-truth response using various automatic word-overlap metrics, such as bleu or meteor. these metrics, along with adem and ruber, are essentially single-step evaluation metrics, where a score is calculated for each contextresponse pair. if a dialogue d i contains n utterances, we can extract n -1 context-response pairs : (c 1 : in this paper, we are interested in devising a scalar metric that can evaluate the quality of a contextresponse pair: score(c i , r i ) = @xmath4, 1). a key benefit of this approach is that this metric can be used to evaluate online and also for better training and optimization, as it provides partial credit during response generation. ## proposed model we propose a new model, maude, for online unreferenced dialogue evaluation. we first describe the general framework behind maude, which is inspired by the task of measuring alignment in natural language inference (nli) @xcite . it involves training text encoders via noise contrastive estimation (nce) to distinguish between valid dialogue responses and carefully generated negative examples. following this, we introduce our novel text encoder that is designed to leverage the unique structural properties of dialogue. maude is designed to output a scalar score(c i , r i ) = @xmath5, 1), which measures how appropriate a response r i is given a dialogue context c i . this task is analogous to measuring alignment in nli, but instead of measuring entailment or contradiction, our notion of alignment aims to quantify the quality of a dialogue response. as in nli, we approach this task by defining encoders f θ e (c) and f θ e (r) to encode the context and response, a combination function f comb (.) to combine the representations, and a final classifier f t (.), which outputs the alignment score: the key idea behind an unreferenced dialogue metric is the use of noise contrastive estimation (nce) @xcite for training. specifically, we train the model to differentiate between a correct response (score(c, r) → 1), and a negative response (score(c, r) → 0), where r represents a candidate false response for the given context c. the loss to minimize contains one positive example and a range of negative examples chosen from a sampling policy p (r): however, the standard text encoders used in these traditional nli approaches ignore the temporal structure of dialogues, which is critical in our setting where the context is composed of a sequence of distinct utterances, with natural and stereotypical transitions between them. (see appendix a for a qualitative analysis of these transitions). thus we propose a specialized text encoder for maude, which uses a bert-based encoder f bert e but additionally models dialogue transitions using a recurrent neural network: the final representation of the dialogue context is learned by pooling the individual hidden states of the rnn using max-pool (equation 4 ). this context representation is mapped into the response vector space using weight w, to obtain c i . we then learn the alignment score between the context c i and response r i 's representation h r i following equation 1, by using the combination function f comb being the same as in equation 3 . ## experiments to empirically evaluate our proposed unreferenced dialogue evaluation metric, we are interested in answering the following key research questions: ## conclusion in this work, we explore the feasibility of learning an automatic dialogue evaluation metric by leveraging pre-trained language models and the temporal structure of dialogue. we propose maude, which is an unreferenced dialogue evaluation metric that leverages sentence representations from large pretrained language models, and is trained via noise contrastive estimation. maude also learns a recurrent neural network to model the transition between the utterances in a dialogue, allowing it to correlate better with human annotations. this is a good indication that maude can be used to evaluate online dialogue conversations. since it provides immediate continuous rewards and at the singlestep level, maude can be also be used to optimize and train better dialogue generation models, which we want to pursue as future work.