Towards Quantifiable Dialogue Coherence Evaluation (2021)

Abstract: Automatic dialogue coherence evaluation has attracted increasing attention and is crucial for developing promising dialogue systems. However, existing metrics have two major limitations: (a) they are mostly trained in a simplified two-level setting (coherent vs. incoherent), while humans give Likert-type multi-level coherence scores, dubbed as “quantifiable”; (b) their predicted coherence scores cannot align with the actual human rating standards due to the absence of human guidance during training. To address these limitations, we propose Quantifiable Dialogue Coherence Evaluation (QuantiDCE), a novel framework aiming to train a quantifiable dialogue coherence metric that can reflect the actual human rating standards. Specifically, QuantiDCE includes two training stages, Multi-Level Ranking (MLR) pre-training and Knowledge Distillation (KD) fine-tuning. During MLR pre-training, a new MLR loss is proposed for enabling the model to learn the coarse judgement of coherence degrees. Then, during KD fine-tuning, the pretrained model is further finetuned to learn the actual human rating standards with only very few human-annotated data. To advocate the generalizability even with limited fine-tuning data, a novel KD regularization is introduced to retain the knowledge learned at the pre-training stage. Experimental results show that the model trained by QuantiDCE presents stronger correlations with human judgements than the other state-of-the-art metrics.

## introduction dialogue coherence, which requires a response to be fluent, consistent and context-related, is an essential property for developing promising dialogue systems @xcite . however, it is still challenging to evaluate the coherence of a response generated by a dialogue system. although human evaluation is always considered as the most accurate way to evaluate the coherence, it is expensive and high-latency, which cannot meet the evaluation demand of the frequent development of dialogue systems. therefore, automatic evaluation metrics are developed to serve as human proxies that can rapidly compute the dialogue coherence and return relatively accurate results. the current widely used metrics measure the lexical word-overlap between generated responses and reference responses, such as bleu @xcite and rouge @xcite . however, they have been demonstrated to be biased and correlate poorly with human judgements since no semantic information is considered @xcite . to overcome this issue, researchers turned to develop learnable metrics based on neural networks that incorporate the semantic information, such as ruber @xcite , @xcite and grade @xcite . however, these metrics deviate from the actual human rating due to two limitations. first, they simplify the coherence evaluation task in a two-level setting, i.e., coherent or incoherent, by maximizing the differences between the positive coherent dialogues and the negative incoherent ones obtained by some negative sampling strategies. in contrast, humans usually adopt likert scaling and give coherence scores from multiple levels like 1 to 5, as shown in figure 1 . second, to avoid relying on large-scale human-annotated data, they are mostly trained in a purely unsupervised manner and cannot align with the human rating due to the absence of introducing the actual human rating standards during training. to address the above limitations, we propose a novel dialogue coherence metric training framework, named as quantifiable dialogue coherence evaluation (quantidce). this framework consists of two training stages: multi-level ranking (mlr) pre-training and knowledge distillation (kd) finetuning. at the mlr pre-training stage, a new multilevel ranking (mlr) loss is proposed for learning the coarse judgement of coherence degrees. specifically, the mlr loss separates the context-response pairs with different coherence levels and compacts the pairs within the same level in one-dimensional score space. as a result, the pretrained model is able to distinguish different coherence-level dialogue responses for a given context and predicts more accurate coherence scores. at the kd finetuning stage, the pretrained model is further finetuned to learn the actual human rating standards with only very few human-annotated coherence scores. to mitigate overfitting into the scarce annotated data during fine-tuning, a novel knowledge distillation regularization loss is introduced to retain the knowledge learned at the pre-training stage, where the pretrained model (teacher) provides the soft targets for the model during fine-tuning (student). experimental results show that the metric trained by our quantidce obviously outperforms the other state-of-the-art metrics in terms of the pearson, spearman and kendall correlations with human judgements by around 5% points on average. to summarize our contributions: knowledge distillation. knowledge distillation (kd) is a method that transfers the knowledge from a large trained teacher model to a smaller student model by using the soft targets provided by the teacher @xcite . in recent years, kd has been applied to many specific tasks @xcite @xcite . unlike these previous works, we use kd to retain knowledge learned at the pre-training stage one: mlr pre-training 𝓛 𝑚𝑙𝑟 bert mlp bert mlp human rating stage two: kd fine-tuning 𝓁 𝑘𝑑 𝓁 𝑚𝑠𝑒 teacher model kd student model level-1 centroid score level-2 centroid score level-3 centroid score ＜ ＜ level-1 scores 𝒊 𝒕𝒉 dialogue example context level-1 level-2 level-3 response separation compactness level-2 scores level-3 scores 𝓛 kd_𝑚𝑠𝑒 ＜ ＜ separation loss compactness loss ordering loss concat …… …… …… concat concat figure 2: the overall pipeline of our quantidce, consisting of two training stages which are marked by the blue and the black one-way arrows. each input dialogue example contains one context with three-level candidate responses and five responses for each level, shown as red, orange and green rectangles respectively. the solid circle represents the centroid score for each level of the i th dialogue. at mlr pre-training stage, the contextresponse pairs are encoded with bert and transformed into the coherence scores through the mlp prediction network, and then mlr loss is applied to optimize the network. the dotted two-way arrows indicate that both ends should be separated, while the solid two-way arrows indicate that both ends should be compact. and at the kd fine-tuning stage, the student model is first initialized with the teacher model and optimized by kd-mse loss. ## quantidce framework in this section, we present quantidce, a two-stage framework for dialogue coherence metric learning, consisting of multi-level ranking (mlr) pre-training and knowledge distillation (kd) finetuning. as illustrated in figure 2 , given a metric model m (section 3.1), quantidce enables m to learn multi-level representations for contextresponse pairs with different levels of coherence degrees during the pre-training stage (section 3.2), and further to learn the rating standards of humans with only a fraction of data during the fine-tuning stage (section 3.3). after these two training stages, the quantifiable gap between automatic metrics and humans can be obviously reduced. ## conclusion in this paper, we propose quantidce, a novel training framework aiming to bridge the gap between the training objective and the actual human rating and train a quantifiable dialogue coherence metric. in general, quantidce includes two training stages, mlr pre-training for learning the coarse human judgements of dialogue coherence degrees, and kd fine-tuning for learning the actual human rating standards. experimental results show that the metric trained by quantidce presents strong correlations with human judgements. for future work, it is interesting to investigate a more efficient way to obtain multi-level data and extend the multilevel setting into the general evaluation for natural language generation. samples and both level-1 and level-2 pairs as negative samples, except that the former use crossentropy loss while the latter use ranking loss; c) bleurt was initialized with the official recommended checkpoint bleurt-base and finetuned on dailydilaogeval by following the office guidelines 11 .