D ial S umm E val: Revisiting Summarization Evaluation for Dialogues (2022)

Abstract: Dialogue summarization is receiving increasing attention from researchers due to its extraordinary difficulty and unique application value. We observe that current dialogue summarization models have flaws that may not be well exposed by frequently used metrics such as ROUGE. In our paper, we re-evaluate 18 categories of metrics in terms of four dimensions: coherence, consistency, fluency and relevance, as well as a unified human evaluation of various models for the first time. Some noteworthy trends which are different from the conventional summarization tasks are identified. We will release DialSummEval, a multi-faceted dataset of human judgments containing the outputs of 14 models on SAMSum.

## introduction neural network based approaches and sizable datasets have led to significant progress in researches towards conventional summarization tasks such as news and scientific papers @xcite . compared with conventional summarization tasks, dialogue summarization has received increasing attention from researchers due to its great difficulty and unique application value @xcite . with the proposal of dialogue summary datasets such as samsum @xcite , dialogsum @xcite and medi-asum @xcite , a number of models for automatic generation of dialogue summaries have emerged @xcite @xcite @xcite @xcite . there is no denying that these studies have made promising progress, but it remains a challenge to evaluate these advances comprehensively. current studies generally use the samsum dataset and adopt rouge @xcite , an n-grambased automatic evaluation metric using reference summaries, as the overall evaluation criterion for summary quality, complemented by manual evaluation. @xcite and @xcite illustrate the limitations of rouge in evaluating summarization tasks. also the manual evaluation protocols vary from one research to another based on our observations. we argue that the inadequate evaluation mechanism may have become a major obstacle to the progress of dialogue summarization researches. many studies, such as @xcite and @xcite , have pointed out that the current dialogue summarization models still have many shortcomings, such as wrong references, incorrect reasoning and improper gender pronouns, and rouge may not reflect these problems effectively. for example, @xcite note that rouge-1 and rouge-l fail to accurately measure factual inconsistency across domains. our case study in although there are automatic evaluation metrics for measuring the quality of all aspects of summaries on conventional summarization tasks, especially for factual consistency @xcite , it is difficult to guarantee that they will still perform well on dialogue summarizarion. recently proposed automatic metrics for evaluating generic natural language generation tasks such as bertscore @xcite , bartscore @xcite have also not been experimented on dialogue summarization. the high abstraction level, low extraction rate, and the requirement for complex reasoning power of the ola should be free by 8. kurt wants her to call him. ola will be late. she should be free by 8. kurt will call her. 0.69 0.42 0.67 analysis and evaluation for dialogue summarization models @xcite and chen and yang (2020) sampled the output of models on samsum and analyzes the error types when proposing a new model. due to the different manual evaluation protocols and the small number of models included, it is difficult to comprehensively compare the strengths and weaknesses of different models. @xcite designed several tricks to address the special challenges in dialogue summarization and analysized their effects, such as using name substitution to cope with the presence of multiple speakers in dialogues. @xcite focused on the problem of lengthy input and relevant information location in long dialogue summarization, and compared the performance of some models and strategies. no manual evaluation was involved in these studies. ## preliminaries in this section, we introduce the involved dataset, metrics and models. ## metric evaluation in this section, we will introduce several definitions in meta-evaluation and re-evaluate the metrics mentioned in section 3.2. ## model evaluation in each dimension, we evaluate each model mentioned in section 3.3 using the average of the human evaluation scores of all summaries. analyzing extractive models produce summaries that differ in style from abstractive models, and many conversations contain ungrammatical utterances, which can affect the reading experience and impair their fluency and coherence. in particular, longest-3, which extracts some potentially discontinuous sentences from dialogues, has low coherence. however, since they do not modify the content, they still perform well in terms of con-sistency. since the average length of dialogues in samsum is small, extracting a few sentences from it can generally include important contents, so the relevance is also high. the evaluation of the extractive models raises a qustion: what kind of summaries do readers actually want? the early neural summrization models represented by pgn and transformer perform relatively poorly in all dimensions compared to the reference summaries, especially consistency and relevance. this is to be expected because of the high difficulty of dialogue summarization and the small size of samsum dataset. an important finding is that the generic pretrained language models represented by bart, pegasus and unilm, and various recently proposed models specifically designed on the dialogue summarization task do not have significant differences in each dimension. they are already comparable, and in some cases better, in terms of coherence and fluency compared to the reference summaries. they have improved dramatically compared to earlier neural summarization models with respect to consistency and relevance, but there is still some room for enhancement. on the one hand, this finding affirms the capability of these models; on the other hand, it urges us to reflect on how much these recently proposed complex models or fancy techniques are an improvement over the generic pre-trained language mod-els. ## conclusion we point out the problems with the evaluation in the dialogue summarization and introduce di-alsummeval, a multi-faceted dataset containing the output of various models and the corresponding human judgments. based on this dataset, we provide a comprehensive re-evaluation and analysis of the performance of widely used automatic evaluation metrics and each model. there are three important findings: 1) few metrics are excellent in all dimensions, and the recently proposed bartscore and qa-based metrics are comparatively outstanding and worth exploring. 2) the automatic evaluation metrics and their variants present some trends that differ from conventional summarization. 3) a variety of models specifically designed for dialogue summarization perform comparably to reference summaries in terms of coherence and fluency, but still have shortcomings in consistency and relevance. we hope that researchers in the field recognize the importance of evaluation in current research, choose some other metrics in addition to rouge when evaluating models, propose automatic evaluation metrics that can be better adapted to the field of dialogue summarization based on our work.