USR : An Unsupervised and Reference Free Evaluation Metric for Dialog Generation (2020)

Abstract: The lack of meaningful automatic evaluation metrics for dialog has impeded open-domain dialog research. Standard language generation metrics have been shown to be ineffective for evaluating dialog models. To this end, this paper presents USR, an UnSupervised and Reference-free evaluation metric for dialog. USR is a reference-free metric that trains unsupervised models to measure several desirable qualities of dialog. USR is shown to strongly correlate with human judgment on both Topical-Chat (turn-level: 0.42, system-level: 1.0) and PersonaChat (turn-level: 0.48 and system-level: 1.0). USR additionally produces interpretable measures for several desirable properties of dialog.

## introduction the lack of meaningful automatic evaluation metrics is a significant impediment for open-domain dialog generation research. standard language generation metrics have been shown to be ineffective for dialog evaluation @xcite . without well-accepted, meaningful automatic metrics, open-domain dialog researchers have come to rely on human evaluation. due to its time-and cost-intensive nature, human evaluation is typically only used for the final dialog model. as such, during development dialog systems are generally optimized for poorly-correlated automatic metrics (e.g., f-1, bleu, ppl) which can result in sub-par human evaluation scores @xcite . to facilitate development of opendomain dialog models with meaningful automatic metrics, this paper presents the unsupervised and reference free (usr) evaluation metric for dialog. standard automatic metrics for evaluating dialog generation (e.g., bleu, f-1, meteor, rouge) have several shortcomings that make them unsuitable for dialog evaluation: (1) the one-to-many nature of dialog @xcite makes wordoverlap metrics ineffective for scoring valid system output that deviates from the ground-truth response @xcite . (2) human evaluation of dialog typically measures multiple properties (e.g., appropriate, interesting, consistent). automatic metrics on the other hand, condense the multi-faceted nature of dialog quality to a single uninterpretable metric. (3) there are many definitions of what a good dialog is and, as such, it is not feasible to construct a "one size fits all" metric. depending on the task and the data, the desired qualities of a dialog system may differ @xcite . usr is a reference-free metric that consists of several interpretable sub-metrics which are combined in a configurable manner. rather than relying on a ground-truth reference response, unsupervised models are trained to measure desired qualities of dialog (e.g., interesting, natural). as such, usr (1) alleviates the one-to-many issue of standard metrics, (2) produces interpretable measures for desirable properties of dialog, and (3) provides a configurable mechanism for combining several submetrics into an overall quality score. to evaluate the performance of usr, human quality annotations were collected for models trained on the topical-chat @xcite and the personachat corpora @xcite . usr is shown to strongly correlate with human judgment on both topical-chat (turn-level spearman: 0.42, system-level spearman: 1.0) and personachat (turn-level spearman: 0.48 and system-level spearman: 1.0). the strong correlation with human judgment across two datasets and a variety of model types shows that usr is a valuable tool for the dialog community. further, since usr does not require any explicit supervision, it has the potential to generalize to several dialog tasks and datasets. the contributions of this paper as as follows: (1) a strongly-correlated, unsupervised and reference free metric is proposed for evaluating open-domain dialog systems, (2) a thorough human quality annotation is carried out and is released 1 to facilitate future benchmarking of dialog evaluation metrics. ## related work standard automatic metrics for language generation correlate poorly with human judgement of dialog @xcite @xcite . for example, the f-1 score can be gamed by outputting the most frequent words, regardless of the context @xcite . the poor performance of present metrics is largely due to the one-to-many nature of dialog @xcite . to avoid comparing to a single reference response, several authors have proposed using multiple reference responses. multiple reference responses can be obtained with retrieval models @xcite or through data collection @xcite . these multi-reference metrics show improvement in performance, but it is infeasible to thoroughly cover the space of potential responses. as such, this paper addresses the one-to-many issue of dialog by presenting a reference-free metric. @xcite train adem to produce a quality score conditioned on the dialog context, the reference response and the generated response. @xcite present a framework for evaluation of alexa prize conversations, which attains moderate correlation with user ratings. both of these methods are trained on explicit quality annotations. in contrast, usr requires no explicit supervision and will more easily generalize to new datasets and tasks. @xcite proposes a reference-free dialog evaluator which is trained to discriminate between human and generated responses. this work is similar to usr in that it evaluates the quality of a response without a reference or quality annotation training data. using the evaluation model as a reward during reinforcement learning exhibited strong performance. however, correlation with human judgement was not evaluated. intuitively, it appears insufficient to rely on a discriminator as a meaningful evaluation of dialog since this assumes that all human responses are perfect and all generated responses are imperfect. ## human quality annotation to evaluate the correlation of automatic metrics with human judgment, human quality annotation was carried out across two open-domain dialog corpora. generated responses were obtained from several models described in section 3.3. for each dialog context, an additional human response was also written. human annotation was then carried out on sixty dialog contexts, with six responses per context for topical-chat (four system outputs, one newly-annotated human output, one original ground-truth response) and five for personachat (one less system output). each response was given six different scores: understandable (0-1), natural (1-3), maintains context (1-3), interesting (1-3), uses knowledge (0-1), overall quality (1-5). three annotators labeled each response. the task instructions were very detailed in order to minimize subjectivity in the quality annotations. for example, individuals may differ in their definition of interesting (e.g., some individuals find football interesting, others do not). thus, the instructions contained a clear, albeit somewhat rigid definition, of interesting. the instructions for overall quality annotation, however, were less rigid and therefore those annotations contain some amount of annotator-specific subjectivity. the data collection and experiments with per-sonachat were carried out to assess the generality of the usr metric. as such, the annotation questions used were specifically tailored to topical-chat, but are still suitable for personachat. ## automatic metrics this section describes the automatic metrics explored for evaluating generated responses. section 4.1 describes several existing metrics that were studied. section 4.2 presents usr, a novel unsupervised and reference-free metric. ## results this section evaluates all of the automatic metrics described in section 4, by comparing them to human judgement. the best sub-metrics for each dialog quality are used as input for the regression model of the usr metric. while the best performing sub-metrics are not consistent across the two datasets, the usr metric nonetheless exhibits strong results. the annotations for the original ground-truth are not used for evaluation, in order to accurately compare referenced and reference-free metrics. the ability of mlm and dr to accurately quantify qualities of a generated response without a reference response, and (2) the ability of usr to effectively combine mlm and dr into a better correlated overall metric. usr shows a similar improvement over all other metrics on personachat, as shown in we compute the system-level correlation between all automatic metrics and the overall quality ratings. usr significantly (@xmath3.01) outperforms all other metrics with a spearman correlation of 1.0 on both datasets and pearson correlations of 0.92 (topical-chat) and 0.82 (personachat). the full set of system-level correlations can be found in the appendix. these results demonstrate usr's effectiveness. it strongly outperforms other metrics on both turnlevel and system-level correlations. @xcite use the f-1 score as their primary automatic evaluation metric when presenting topical-chat. the results demonstrate a significant difference between usr and the f-1 score, suggesting that usr is a better metric for the topical-chat corpus. ## discussion usr achieves statistically significant correlations with human judgement. the results hold across two datasets, topical-chat @xcite and personachat @xcite . usr is configurable. notably it is composed of several specific dialog quality sub-metrics. these sub-metrics are combined in a configurable manner, using a regression. for other tasks, datasets or even users, this regression can be adjusted, allowing qualities to be removed or re-weighted. additional sub-metrics could be added. usr should be used alongside human evaluation. usr was created to facilitate development and tuning of dialog models. as such, usr can be used for model selection and hyperparameter tuning. usr should not be used to claim superior performance over another method. usr may not work with non-generative models, which were not addressed here. responses produced by a model that is too similar to the evaluation models (e.g., to dr) are a particular concern. ## conclusions this paper presents usr, an unsupervised and reference-free evaluation metric for dialog. to address the shortcomings of standard metrics for language generation, usr (1) is reference-free, (2) is composed of multiple sub-metrics that evaluate specific qualities of dialog, (3) has a definition of good dialog that is configurable. thus the metric may be adapted to different tasks and datasets. usr is shown to strongly correlate with human judgment on topical-chat (turn-level: 0.42, system-level: 1.0) and personachat (turn-level: 0.48, systemlevel: 1.0).