Dial- M : A Masking-based Framework for Dialogue Evaluation (2023)

Abstract: In dialogue systems, automatically evaluating machine-generated responses is critical and challenging. Despite the tremendous progress in dialogue generation research, its evaluation heavily depends on human judgments. The standard word-overlapping based evaluation metrics are ineffective for dialogues. As a result, most of the recently proposed metrics are model-based and reference-free, which learn to score different aspects of a conversation. However, understanding each aspect requires a separate model, which makes them computationally expensive. To this end, we propose Dial-M, a Masking-based reference-free framework for Dialogue evaluation. The main idea is to mask the keywords of the current utterance and predict them, given the dialogue history and various conditions (like knowledge, persona, etc.), thereby making the evaluation framework simple and easily extensible for multiple datasets. Regardless of its simplicity, Dial-M achieves comparable performance to state-of-the-art metrics on several dialogue evaluation datasets. We also discuss the interpretability of our proposed metric along with error analysis.

## introduction dialogue systems research has seen massive advancements in recent years. it is not surprising to see models generating high-quality human-like meaningful responses nowadays. despite this enormous progress, the evaluation of machinegenerated dialogues remains a concern. although many automatic metrics have been proposed, we still have to rely on human evaluation, which is tedious and costly. thus, improving the quality of automatic dialogue evaluation is essential for the overall development of this evolving area. the evaluation metrics for dialogue generation can be broadly divided into two classes: referencebased and reference-free. in reference-based metrics, the generated dialogue is evaluated with respect to one more reference utterance(s). the most popular reference-based metrics used in dialogue systems are standard word-overlapping based metrics like bleu @xcite , nist @xcite , meteor @xcite , rouge @xcite , diversity @xcite , and entropy @xcite . however, these metrics have been shown to be ineffective because of the one-to-many nature of dialogues @xcite . as a result, people started adopting learning-based referenced metrics like adem @xcite , ru-ber @xcite , @xcite , pone @xcite , bertscore @xcite , bleurt @xcite , fbd @xcite , deep am-fm @xcite , etc. however, referencebased metrics are not feasible for evaluation in an online setting where the reference response is unavailable. also, collecting good-quality candidate responses is costly and requires human annotation. hence, most of the recent efforts are being made in the direction of reference-free metrics. in reference-free metrics, the generated dialogue is evaluated without any references. here, most of the methods formulate the dialogue evaluation problem as one or more classification tasks and use the classification scores as the metric or submetrics. metrics like maude @xcite and deb @xcite learn to differentiate between correct and incorrect responses given the context. grade @xcite and dy-naeval @xcite leverage graph-based methods, while @xcite relies on abstract meaning representation (amr) to evaluate dialogue coherence. mdd-eval @xcite addresses the issue of multi-domain evaluation by introducing a teacher evaluator. the quality of a generated dialogue depends on multiple factors such as understandability, informativeness, coherence, etc. metrics like usr @xcite , usl-h @xcite , fed @xcite , holisticeval @xcite , d-score @xcite , qual-ityadapt @xcite learn to compute various sub-metrics and then combine them to give a final score. for further improvement, im foot_0 @xcite combines multiple metrics that are good at measuring different dialog qualities to generate an aggregate score. however, modeling different sub-metric requires a separate model or adapter, increasing the computational cost. moreover, the decision boundary of the classification-based metrics depends on the quality of negative sampling @xcite , inducing training data bias. in this work, we aim to address these issues by proposing dial-m 1 , a masking-based referencefree framework for dialogue evaluation. the central idea of dial-m is to mask the keywords of the current utterance and use the cross-entropy loss while predicting the masked keywords as the evaluation metric. doing so avoids the requirement for multiple models and negative sampling, making the framework simple and easily extensible to multiple datasets. the keywords in the current utterance are obtained in an unsupervised manner. we show that dial-m achieves comparable performance to various state-of-the-art metrics on several evaluation datasets, especially knowledge-grounded datasets like topical-chat. we observe that dial-m can capture different aspects of a conversation. we also show that the dial-m score can be interpreted by inspecting the masked words, which enables the scope for error analysis. ## experimental setup we use dailydialog @xcite , persona-chat @xcite , wizard-of-wikipedia @xcite , and topical-chat @xcite ) for both pre-training and finetuning dial-m. we show our results on usr @xcite , predictiveengage @xcite , and holisticeval @xcite datasets for dialogue evaluation. usr is based on topical-chat and persona-chat, while predic-tiveengage and holisticeval are based on dailydialog. we call the topical-chat and persona-chat datasets of usr as usr-topical and usr-persona, respectively. we use spacy @xcite pos tagger along with yake! to find the keywords during evaluation. we analyzed the pos tags of co-occurring words in response (u t ) knowledge (c t ) pair in topical-chat train data and selected the most frequent pos tags (nn, nnp, nns, jj, cd, vb, vbn, vbd, vbg, rb, vbp, vbz, nnps, and jjs) for our purpose. the rest of the details are provided in appendix a.2. ## result and analysis rows 13-16 of ## discussion in this section, we discuss the interpretability and error analysis of dial-m scores. good cases (responses 1-3). we can observe that dial-m has given a low score to response 1 in comparison to responses 2 and 3, which correlates with the human scores. the reason for this low score can be deduced by looking at the masked words of response 1, which are connected to both context and condition (persona). in response 2, masked words like red and blue are out of context, resulting in a higher dial-m score. the masked words of response 3 are slightly out of context in comparison to response 1, resulting in an average score that is reflected in the human scores as well. let us now analyze response 4, which can be treated as a bad case because dial-m finds it superior even though it is not the best response. the possible reason for the lower human score of response 4 than response 1 is the usage of "i get up early everyday", which is not mentioned in the persona. however, the phrase "i get up early" is very common. since dial-m is pre-trained on mlm task, the prediction of "early" given "i get up" becomes easy, resulting in the lowest score. this is how we can interpret and perform error analysis of the dial-m scores by inspecting the masked words. we observed that dial-m generally assigns a low score to short, generic, and frequently used sentences where the masked word can be easily predicted from its neighbors. we aim to address this issue in our future work. ## conclusion in conclusion, we propose dial-m, a maskingbased reference-free framework for dialogue eval-uation. we mask the keywords of the current utterance and use the cross-entropy loss while predicting the masked keywords as the evaluation metric. formulating the problem as a keyword masking task avoids the requirement for multiple models and negative sampling, making the framework simple and easily extensible to multiple datasets. dial-m achieves comparable performance to stateof-the-art metrics on several dialogue evaluation datasets. we also show the utility of keyword masking in capturing various aspects of a conversation and discuss the interpretability and error analysis of dial-m scores. we want to explore better keyword extraction strategies in future work. we also want to investigate better techniques to handle the cases where no keywords are detected in the current utterance.