Designing Precise and Robust Dialogue Response Evaluators (2020)

Abstract: Automatic dialogue response evaluator has been proposed as an alternative to automated metrics and human evaluation. However, existing automatic evaluators achieve only moderate correlation with human judgement and they are not robust. In this work, we propose to build a reference-free evaluator and exploit the power of semi-supervised training and pretrained (masked) language models. Experimental results demonstrate that the proposed evaluator achieves a strong correlation (> 0.6) with human judgement and generalizes robustly to diverse responses and corpora. We open-source the code and data in https://github.com/ZHAOTING/dialog-processing .

## introduction evaluation of conversational systems has been one major obstacle in dialogue research. particularly for open-domain dialogues, automated metrics have been shown to correlate poorly with human judgement @xcite . although human evaluation provides the most accurate assessment, they are slow and expensive. an alternative is to train an evaluator that learns to predict a human-like score. @xcite proposed adem, a supervised regression model, for automatic response evaluation and reported 0.436 pearson's and 0.428 spearman's correlations with human judgement. though better than automated metrics, the scores only indicate moderate correlations. another criticism from @xcite further pointed out that adem produces scores of low deviation and lacks robustness under adversarial attack. an ideal evaluator should be precise such that its predictions have a strong correlation with human judgement. it should also be robust such that it generalizes to new dialogues unseen during training. we explored three methods to improve the precision and robustness of response evaluators. 1) we propose building referencefree evaluator since reference-dependent metrics cause the problem of low deviation described by @xcite . we also find that the referencedependent evaluators' performance degrades significantly when we remove ground-truth responses from test data. 2) @xcite proposed an unsupervised model (ruber) that outperforms supervised adem by training on a next sentence prediction (nsp) task. we show that ruber can be further improved by supervised training on a small amount of annotated data. 3) we make use of strong pretrained models such as roberta @xcite to obtain better text representations. by combining the three methods, a reference-free, semi-supervised, robertabased evaluator has better correlation and robustness. experimental results also show that the model can maintain good performances in crossdomain and low-resource settings. next sentence prediction is to predict whether a sentence is a true continuation given a preceding context, where a positive sample is the ground-truth subsequent sentence and a negative sample is a different piece of text. nsp benefits not only evaluation @xcite , but also language understanding @xcite and language generation @xcite . dialogue response evaluation can also be improved with better automated metrics and approximation to response quality. examples of successful attempts to improve automated metrics include exploiting multiple references for comparison @xcite and combining human judgement with automated metrics @xcite . @xcite demonstrated that single-turn human judgement is not reliable as expected and proposed multi-turn human evaluation. @xcite approximated sentiment, semantic similarity, and engagement with new automated metrics and used a hybrid metric in a multi-turn evaluation setting. @xcite showed that entailment is also an option to approximate dialogue coherence and quality. ## background adem is a regression model that takes as inputs a dialogue context vector c, a hypothesis response vector r, and a reference response vector r. its output is the sum of a referenced metric and an unreferenced metric: ruber unref (c, r) = mlp([c; r; c t mr]; θ), (4) where [•; •] denotes the concatenation of vectors and mlp is a multi-layer perceptron with nonlinear activation functions. m and θ are trainable parameters. besides the differences in metric computation, they are different in training strategy. adem uses supervised training to minimize the mean square error between predictions and human scores, while ruber uses unsupervised training on an nsp task to minimize a margin ranking loss. in section 5, we combine their advantages to build a better response evaluator. ## data collection for assessing dialogue response evaluators, we sample 100 dialogues from the test split of the dailydialog corpus @xcite which contains 13,118 open-domain and human-written conversations. we expand them with extra response hypotheses and collect human annotations of response quality. collection of extra responses. besides the ground-truth response, we add responses from different sources for each dialogue context, including 1) a negative-sampling response randomly selected from a different dialogue and 2) responses generated by generative models trained on the training split. we combine 6 generative models (s2s @xcite , attentional s2s, hred @xcite , vhred @xcite , gpt2-sm, and gpt2-md @xcite ) with 3 decoding methods (greedy decoding, ancestral sampling, and nucleus sampling @xcite ). the resulting response pool for each dialogue context contains 20 responses of various qualities. ## experimental evaluations ## conclusion automatic dialogue response evaluators have problems in robustness and correlation with human judgement. we investigated three methods to alleviate them: 1) using reference-free metrics, 2) applying semi-supervised training, and 3) exploiting powerful pretrained text encoders. experimental results demonstrated that our proposed evaluator achieved strong correlation (> 0.6) with human judgement and showed robustness in dealing with diverse responses and a new domain. it can also be trained efficiently with less than 100 annotated samples. ilya sutskever, oriol vinyals, and quoc v le. 2014. sequence to sequence learning with neural networks. in nips 2014, advances in neural information processing systems 27: annual conference on neural information processing systems 2014, pages 3104-3112. chongyang tao, lili mou, dongyan zhao, and rui yan. 2018. ruber: an unsupervised method for automatic evaluation of open-domain dialog systems. in aaai 2018, the 32nd aaai conference on artificial intelligence. thomas wolf, victor sanh, julien chaumond, and clement delangue. 2019. transfertransfo: a transfer learning approach for neural network based conversational agents. arxiv preprint arxiv:1901.08149. saizheng zhang, emily dinan, jack urbanek, arthur szlam, douwe kiela, and jason weston. 2018. personalizing dialogue agents: i have a dog, do you have pets too? in acl 2018, the 56th annual meeting of the association for computational linguistics, volume 1, pages 2204-2213. tianyu zhao and tatsuya kawahara. 2019. effective incorporation of speaker information in utterance encoding in dialog. arxiv preprint arxiv:1907.05599.