S el F -Eval: Self-supervised Fine-grained Dialogue Evaluation (2022)

Abstract: This paper introduces a novel Self-supervised Fine-grained Dialogue Evaluation framework (SelF-Eval). The core idea is to model the correlation between turn quality and the entire dialogue quality. We first propose a novel automatic data construction method that can automatically assign fine-grained scores for arbitrarily dialogue data. Then we train SelF-Eval with a multi-level contrastive learning schema which helps to distinguish different score levels. Experimental results on multiple benchmarks show that SelF-Eval is highly consistent with human evaluations and better than the state-of-the-art models. We give a detailed analysis of the experiments in this paper. Our code is available on GitHub.

## introduction dialogue systems (ds) aim to satisfy human needs @xcite @xcite such as information, communication, entertainment, etc. appraising the quality of the ds responses reflects the system's capability and provides insights into required further improvements @xcite . among the commonly used evaluation metrics, human evaluation is of high reliability but expensive to conduct, automatic metrics used in language generation (perplexity @xcite ) or machine translation (bleu @xcite , rouge @xcite , etc.) are easy to conduct but ineffective to reflect the dialogue quality @xcite . therefore, researchers have made great efforts to find more reliable automatic evaluation metrics that are highly correlated with human evaluation @xcite @xcite . the current automatic dialogue evaluation metrics leverage semantic information @xcite @xcite to measure dialogue quality. for example, when evaluating the response quality, they either compute the semantic similarity between dialogue context and the generated response @xcite @xcite or measure the soft semantic overlap between ground-truth response and the modelgenerated one @xcite @xcite @xcite . when evaluating @xcite the overall dialogue quality, they either learn a dialogue-level representation for rating directly @xcite or calculate the score with the help of other indirect assists @xcite . however, recent studies @xcite @xcite show that current models can only work well for measuring the response or evaluating the entire dialogue. they could not perform well in both situations at the same time. it means that the dialogue representation they learned @xcite could not reflect both turn quality and the entire dialogue quality. the dialogue quality is affected by all turns' qualities in it @xcite and this effect is accumulated in a multi-turn dialogue @xcite . figure 1 shows these quality correlations between turns and dialogue. each turn is marked with a serial number. three dialogue examples are starting with the same user turn (1). the left example (1/2/3/4/5) shows that two good agent responses result in good overall quality. the middle (1/6/7/8/9) and right (1/6/7/10/11) examples show how lower-quality agent responses result in different dialogue qualities (passable or poor). the current open-domain dialogue evaluation methods fail to model the fine-grained correlations between turn quality and dialogue quality, which entail a poor dialogue representation for evaluation. in this paper, we introduce an evaluation method that explicitly models the correlations between turn quality and dialogue quality. specifically, we aim to learn a dialogue representation that can reflect each turn's contribution, so that the evaluation score obtained by this representation aligns turn quality with the dialogue quality. to this end, we need to first obtain large amounts of dialogue data that reflects fine-grained correlations between turns quality and dialogue quality, then train an evaluation model to measure the fine-grained correlations. the contributions of this paper are: our code and data are publicly available: https://github.com/royny/self-eval . ## related work we first survey evaluation metrics in open-domain dialogue (sections 5.1 and 5.2), then compare related work in task-oriented dialogue (section 5.3). ## experimental results and analysis we aim to answer the following questions about self-eval: ## conclusion we propose to measure dialogue quality by modeling the fine-grained correlations between turns and the entire dialogue. we introduce our data construction method and self-eval model. experiments show that self-eval builds fine-grained correlations and gets better correlation scores with human ratings than sota models. we think our method may have two potential applications: 1) it can be used alone in the evaluation of dialogue tasks after training the self-eval model with a large amount of in-domain data; 2) it can be combined with other evaluation models (such as fed) to evaluate the dialogue task by integrating the advantages of different evaluation models. in the future, we would like to improve our method by 1) employing multi-granularity turn-level scores; 2) modeling the nonlinear relationships between replacement numbers and dialogue quality.