Deconstruct to Reconstruct a Configurable Evaluation Metric for Open-Domain Dialogue Systems (2020)

Abstract: Many automatic evaluation metrics have been proposed to score the overall quality of a response in open-domain dialogue. Generally, the overall quality is comprised of various aspects, such as relevancy, specificity, and empathy, and the importance of each aspect differs according to the task. For instance, specificity is mandatory in a food-ordering dialogue task, whereas fluency is preferred in a language-teaching dialogue system. However, existing metrics are not designed to cope with such flexibility. For example, BLEU score fundamentally relies only on word overlapping, whereas BERTScore relies on semantic similarity between reference and candidate response. Thus, they are not guaranteed to capture the required aspects, i.e., specificity. To design a metric that is flexible to a task, we first propose making these qualities manageable by grouping them into three groups: understandability, sensibleness, and likability, where likability is a combination of qualities that are essential for a task. We also propose a simple method to composite metrics of each aspect to obtain a single metric called USL-H, which stands for Understandability, Sensibleness, and Likability in Hierarchy. We demonstrated that USL-H score achieves good correlations with human judgment and maintains its configurability towards different aspects and metrics.

## introduction evaluating a dialogue response is crucial for the development of open-domain dialogue systems. it allows comparison between different systems, which is similar to how the machine translation community uses bleu @xcite to evaluate the overall quality of the translation and determines whether a system is the state-of-the-art @xcite @xcite . without automatic evaluation metrics, many studies tend to rely on either expert or crowdsourced platform to score the responses, which are both time-consuming and cost-ineffective @xcite @xcite . to cope with this, various metrics have been proposed to score the overall quality of a dialogue response. word overlap-based metrics, which were adopted from the mt community to measure the overlapping words between reference and candidate sentences, have been used to evaluate the dialogue responses @xcite . however, @xcite showed that these metrics, i.e., bleu @xcite , meteor @xcite , or rouge score @xcite , do not correlate well with human judgements, because there are many possible responses to reply to a given context. recently, learning-based metrics, which aim to predict the overall quality of a response, have a better correlation with human judgment compared with word overlap-based metrics. various training settings have also been explored. for example, adem @xcite is trained to predict the score by learning to regress on human judgments, whereas pone @xcite is trained with next utterance prediction task with sophisticated samplings. however, these metrics are not configurable and may suffer from several limitations. first, they may not capture a certain quality that is essential for a particular task. as shown in responses: ground-truth: next week? it is too late. i need it urgently. candidate 1: yes. that would be fine. figure 1 : decomposition the structure of a response quality. and bert-ruber tend to assign a relatively high score to the unspecific response. this might be due to the complexity of the overall score. generally, a single overall score is usually comprised of different qualities, such as readability, specificity, and empathy, and the importance of each aspect differs according to the task. for example, specificity is preferred in food-ordering chatbots, whereas fluency is preferred in language-teaching chatbots. however, the existing metrics are not flexible to such changes. bertscore @xcite , for example, relies on using pre-trained bert embedding @xcite to compute similarity between reference and candidate responses; thus this does not guarantee good correlation for the specificity quality ( aside from evaluating a response using only a single overall score, some studies evaluate the response on multiple aspects, i.e., fluency, relevancy, specificity, and empathy @xcite @xcite . the limitation of this approach is that with multiple scores to consider, it becomes unclear to determine which response is better. is a specific response more preferable than an empathetic one? to address these issues, we first propose simplifying the various qualities by grouping them into three main aspects: understandability @xcite , sensibleness @xcite , and likability. we assume these groups have hierarchical properties in the following way: (i) a response first needs to be understandable; (ii) then it needs to make sense to the context; (iii) other qualities, i.e., specificity, are just additional qualities that make an acceptable response more likable for a given task (figure 1 ). if we want the score to capture empathy instead, we only need to replace specificity in the top layer with empathy. in other words, the likability aspect does not need to implicitly capture the understandability or sensibleness, as it will be checked by the lower layers in the hierarchy. based on these properties, we propose a simple method to combine scores of each aspect to obtain usl-h score, which stands for understandability, sensibleness, and likability in hierarchy. usl-h can be modified to add or remove a quality and to replace a metric with a more optimal alternative. this configurability removes the barrier of requiring a single complicated model and instead enables a combination of multiple sub-metrics. for simplicity, we demonstrate the configurability using only specificity as our likability aspect. experimenting on the dailydialog dataset @xcite , we show that valid utterance prediction, next utterance prediction, and masked language models have good correlations with human judgments on understandability, sensibleness, and specificity, respectively. moreover, combining these sub-metrics as a single metric using our usl-h also correlates well with overall quality on both pearson and spearman correlations. alternatively, we replace specificity with the empathy aspect, recombine the sub-metrics, and put it up against other metrics to select the most empathetic responses. we find that this new configuration can detect better empathetic responses compared to the rest. through various experiments, we show that usl-h is configurable to capture certain qualities of a response and can be improved further upon replacing a sub-metric with a better performing alternative. the main contributions of this paper are the following: (i) the grouping of various qualities of dialogue responses into three main aspects: understandability, sensibleness, and likability, (ii) introducing a configurable hierarchical evaluation metric that can be modified to work with a set of response's qualities and sub-metrics according to the task while achieving good correlation with human judgments. learning-based metrics have been explored recently, especially with the next utterance prediction setting @xcite @xcite . the model learns to determine whether or not a context-response pair is valid. it is typically trained with context-response pairs that appear in the dialogue as the positive examples. then, negative sampling is used to obtain negative examples. training using this setting demonstrates a moderate correlation with human judgment. however, since such learning-based metrics rely on the positive and negative examples, a sophisticated sampling technique is required to obtain appropriate examples that reflect a certain quality. score composition some studies have attempted to develop metrics by combining scores from different aspects into a single score @xcite . @xcite proposed a metric that combines scores from semantic and syntactic sub-metrics with a weighted average. this metric had a moderate correlation with human judgment and outperformed all of the word-overlapbased metrics. however, it does not consider qualities, such as specificity or diversity. alternatively, @xcite proposed a human evaluation metric that considers both sensibleness and specificity, where specificity is dependent on sensibleness and is scored only if the response is sensible; otherwise, it is zero by default. then, they obtained the final human overall score by averaging them together. unlike @xcite , they did not use any metric for any aspect. instead, they suggest using perplexity as the evaluation metric to capture both qualities. however, using a single metric, like perplexity, to monitor multiple aspects is not configurable when we want to evaluate another aspect, for instance, sensibleness and empathy. ## experiment training corpus the corpus used in this study is dailydialog @xcite , which is about dayto-day communication on everyday topics. this dataset consists of 11,118/1,000/1,000 dialogues for train/valid/test sets with explicit textual information of five dialogue acts and seven emotion labels. we split this dataset evenly into two parts: (i) for training generative and retrieval models to generate candidate responses, and (ii) for training automatic evaluation metrics for scoring each aspect. ## conclusion and future work this study demonstrated a bottom-up approach to building an automatic evaluation metric by deconstructing the overall quality of a response into three fundamental aspects (understandability, sensibleness, and likability), exploring a suitable metric for each aspect, and reconstructing them back to obtain a single metric. however, we restricted the likability aspect to only specificity or empathy. for our future work, we intend to investigate other likability scores, such as engagement or diversity, to ensure that this metric is usable across different tasks and datasets.