IM 2: an Interpretable and Multi-category Integrated Metric Framework for Automatic Dialogue Evaluation (2022)

Abstract: Evaluation metrics shine the light on the best models and thus strongly influence the research directions, such as the recently developed dialogue metrics USR, FED, and GRADE. However, most current metrics evaluate the dialogue data as isolated and static because they only focus on a single quality or several qualities. To mitigate the problem, this paper proposes an interpretable, multi-faceted, and controllable framework IM^2 (Interpretable and Multi-category Integrated Metric) to combine a large number of metrics which are good at measuring different qualities. The IM^2 framework first divides current popular dialogue qualities into different categories and then applies or proposes dialogue metrics to measure the qualities within each category and finally generates an overall IM^2 score. An initial version of IM^2 was submitted to the AAAI 2022 Track5.1@DSTC10 challenge and took the 2^nd place on both of the development and test leaderboard. After the competition, we develop more metrics and improve the performance of our model. We compare IM^2 with other 13 current dialogue metrics and experimental results show that IM^2 correlates more strongly with human judgments than any of them on each evaluated dataset.

## introduction because human evaluation for natural language generation (nlg) systems is both expensive and time-consuming, relevant and meaningful automatic evaluation metrics that strongly correlate with human judgments are crucial. however, as the one-to-many natures of dialogue makes standard automatic language evaluation metrics (e.g., bleu and meteor) ineffective for evaluating open-domain dialogue systems @xcite , many automatic evaluation metrics specifically designed for dialogue have been recently proposed @xcite @xcite @xcite @xcite @xcite . although these dialogue metrics correlate with human evaluation, they focus on a single quality or a few qualities, thus evaluating the dialogue data as isolated and static, e.g., grade @xcite evaluates the topic coherence of dialogue and predictiveengage @xcite estimates the user engagement. therefore, multi-quality metrics are preferred, e.g., fed @xcite ) measures 9 turn-level qualities and 11 dialogue-level qualities for predicting the overall impression score. however, the generalization capability of existing multiquality metrics is questionable, e.g., fed correlates poorly with human judgments when scoring other dialogues outside its own data. recently, the track5.1@dstc10 challenge @xcite just ended, whose purpose is to develop effective automatic open-ended dialogue evaluation metrics that perform robustly across a range of dialogue tasks. no individual metric will be competitive. therefore, recent work attempted to combine dialogue evaluation metrics: 1) combining usr @xcite , grade @xcite , pone @xcite and @xcite through simple-averaging has been reported in a comprehensive assessment of dialogue evaluation metrics @xcite ; 2) usl-h @xcite divides dialogue qualities into three categories (viz. u, s, l) and linearly combines them; 3) the track5.1@dstc10 baseline, deep am-fm @xcite , is a simply combined metric which measures the adequacy metric (am) and the fluency metric (fm) simultaneously. however, the above combinations are straightforward, and thus exploring more sophisticated combination mechanisms has been claimed as an important direction for future work @xcite . on that ground, this paper proposes a novel metric framework named im 2 (interpretable and m ulti-category integrated m etric), which first divides current dialogue qualities into three categories, and then applies or proposes dialogue metrics (named sub-metrics) to measure the qualities within each category, and finally generates an overall evaluation score. the three quality categories are: 1) nuf (natural, understandable, and fluent), which measures the basic quality of the response; 2) cr (coherent and relevant), which measures the response's quality conditioned on the context; 3) ies (interesting, engaging, and specific), which measures the special property of the response. particularly, im 2 leverages the multi-level integration, i.e., first producing categorical metrics by integrating on sub-metrics and then producing the overall metric by integrating on categorical metrics. the contribution of this paper is two-fold: 1.we proposed a novel framework for combing automatic dialogue evaluation metrics. the proposed im 2 is: 1) reference-free, which does not need reference responses; 2) interpretable, which integrates fine-grained sub-metrics and meaningful categorical metrics; 3) flexible, which allows categorical metrics to be used independently. 2.we submitted an early version of im 2 to the aaai 2022 track5.1@dstc10 challenge and obtained a high average spearman correlation coefficient 0.3937 on the development datasets and 0.2819 on the test datasets foot_0 . after the competition, we further improved the correlation score to 0.4645 and 0.3510 respectively, via developing more metrics. ## problem statement the proposed framework is reference-free, which scores the system response without human reference(s). formally, given a dialogue context c and a system response r, the goal is to learn a scoring function f : (c, r) → s that evaluates the generated response. dialogue metrics are assessed by comparing them to human judgments. concretely, a human annotator or several annotators score the quality of a response conditioned on the dialogue context: (c, r) → q. given the scores produced by a metric, @xmath0, ..., s k }, and the corresponding human quality annotations, @xmath1, ..., q k }, we can measure the performance of the metric by calculating the correlation between s and q. ## conclusion this paper explores the sophisticated mechanism for combining dialogue metrics and proposed a novel framework, im 2 . the experimental results show that im 2 strongly correlates with human judgments and outperforms all compared metrics. further, our work reveals that training a perfect metric model for all dialogue datasets is difficult, but selecting the most appropriate metric for different dialogues is promising. there are many future works. first, we will pay more attention to challenge dialogue datasets, such as those with lengthy context. second, we will merge qualities for newest competition tasks, such as track4@dstc11 (robust and multilingual automatic evaluation metrics for open-domain dialogue systems) 7 . third, we will attempt more powerful dialogue systems, such as plato-2 @xcite which directs towards building an open-domain chatbot, and with the help from the im 2 evaluation scores more human-style responses might be generated.