Evaluating Open-Domain Dialogues in Latent Space with Next Sentence Prediction and Mutual Information (2023)

Abstract: The long-standing one-to-many issue of the open-domain dialogues poses significant challenges for automatic evaluation methods, i.e., there may be multiple suitable responses which differ in semantics for a given conversational context. To tackle this challenge, we propose a novel learning-based automatic evaluation metric (CMN), which can robustly evaluate open-domain dialogues by augmenting Conditional Variational Autoencoders (CVAEs) with a Next Sentence Prediction (NSP) objective and employing Mutual Information (MI) to model the semantic similarity of text in the latent space. Experimental results on two open-domain dialogue datasets demonstrate the superiority of our method compared with a wide range of baselines, especially in handling responses which are distant to the “golden” reference responses in semantics.

## introduction open-domain dialogue generation is a prominent research direction in conversational ai due to a wide range of useful applications that it can facilitate, such as for personal digital assistants and customer service @xcite @xcite . while evaluating natural language generation (nlg) systems is notoriously difficult, evaluation of open-domain dialogue generation introduces an extra layer of complexity, as a variety of responses can be generated, each semantically different and yet valid in the given context @xcite @xcite . for example, given the conversational context "iverson is my all-time favourite player.", responses such as "he is my favourite player too." or "yes, his quickness is amazing!" are both contextually relevant, yet semantically different. existing approaches for evaluating open-domain dialogue systems can be broadly divided into two different categories: reference-based and referencefree approaches. the reference-based metrics typically score a system by computing how similar an output response is compared to the goldstandard reference. popular metrics under this category may rely on surface-form similarity by counting the n-gram overlap between the response candidate and the reference (e.g., bleu @xcite , rouge @xcite , and me-teor @xcite ), or by calculating the similarity based on embedding representations such as embedding-average @xcite , or even via high-dimensional representations learned for the response and the reference such as bertscore @xcite . one noticeable limitation of reference-based metrics is that they are reference centric and do not take the context of the conversation into consideration. furthermore, due to the well-known one-to-many issue in open-domain dialogue @xcite @xcite , a good response that matches well to its context could express significantly different semantics to its reference, for which the aforementioned metrics will be inadequate to handle. to tackle the one-to-many issue, some works @xcite @xcite have proposed reference-free metrics to evaluate generated responses by measuring their similarity with the corresponding conversational context, by designing discriminative models trained on the context and the reference to judge whether the generated response matches the context well. as these discriminative metrics are typically trained using a single relevant (aka. positive) response and multiple negative samples, @xcite argue that such metrics should be trained with multiple rele-vant and irrelevant responses for any given context to allow for robust evaluation. however, most existing datasets do not contain multiple references due to high cost of acquisition, rendering this recommendation impractical. @xcite take a different approach to the problem by evaluating generated responses in the latent space produced by conditional variational autoencoders (cvaes), as it can encode discrete text data into a smooth latent space @xcite . specifically, they proposed to use the prior distribution to approximate the conditional distribution for all the feasible responses to tackle the one-to-many issue with limited data. however, there is no guarantee that the prior distribution can represent a rich set of feasible responses @xcite . @xcite proposed a self-training framework for multi-domain dialogue evaluation. the model performance was boosted by training on augmented datasets of four different domains, which are first automatically labelled by a teacher model and then followed by a manual annotation process. to our knowledge, no prior works have attempted to model the intra-relation between a context and a response through the next sentence prediction (nsp) task and mutual information (mi) directly, which can provide a strong signal for indicating the sequential and semantic dependencies between the context and response. to tackle the one-to-many issue, we design a reference-based automatic evaluation metric (cmn), which can robustly evaluate open-domain dialogues with a single gold-standard reference. our method consists of a training stage and an evaluation stage. in the training stage, the cvaes are augmented with a nsp objective @xcite , which plays a crucial role in addressing the one-to-many issue in dialogue evaluation, especially when the semantics of the generated response are distant from the reference but still relate well to the context. in the evaluation phase, we score a response candidate by calculating the mi of the contextresponse and response-reference pairs in the latent space, which are then combined through a weighting controlled by the nsp probability. however, it is intractable to calculate mi directly as we only have access to samples instead of the prior and posterior distributions @xcite . to tackle this challenge, we pro-pose to employ a contrastive learning method based on noise contrastive estimation (nce) @xcite to calculate the lower bound of mi. overall, introducing the nsp objective and mi strengthens our model's ability to capture the sequential dependencies between the context and response, as well as to better leverage the information from references. experimental results on two open-domain dialogue datasets show the superiority of our method compared to a wide range of baseline metrics based on both pearson and spearman correlations with human annotations. in addition, we provide a detailed analysis of the effectiveness of our proposed method in solving the one-to-many issue in open-domain dialogue evaluation. our code is available at https://github.com/ bernard-yang/cmn-acl2023 . unlike metrics comparing the word overlap directly, embedding metrics first convert sentences into a high dimensional representation and calculate the semantic similarity between them. with the development of large-scale pre-training models, embedding metrics such as bertscore @xcite and bleurt @xcite have been shown to effectively enhance sentence representation. however, these automatic reference-based metrics cannot handle the wellknown one-to-many problem in the open-domain dialogue. reference-free metrics. existing reference-free metrics attempt to design discriminative models to solve the one-to-many issue by calculating the similarity between the context and the response candidate. ruber @xcite is an unsupervised metric that calculates the similarity of the generated response with both the context and the response. maude @xcite employs a large-scale pre-trained model to convert sentences into hidden representations and leverage the temporal transitions between them. @xcite argued that such models should be trained on datasets contain-ing multiple responses. however, most existing datasets only contain a single relevant reference and making this recommendation impractical. ems @xcite first attempted to utilise cvaes to learn the reference information with limited data and approximate all feasible responses with the prior distribution. however, their model's prior distribution and sampled variables do not necessarily contain all the feasible response information for a given context, as ems is only trained with a single reference. we propose a reference-based method by augmenting cvaes with the nsp objective and employing mi to evaluate the response candidates. zhang et al. (2022a) tackled multi-domain evaluation by training a teacher model with humanannotated data in a particular domain. the model then labels the data from dialogue datasets in four other domains. this teacher-annotated data is then used to introduce a new evaluator, which can generalise across multiple domains. however, this method requires human labelling and additional training data, which are not required by our method. ## results in this section, we evaluate our model's performance on evaluating open-domain dialogues under both standard and diverse settings.