Original: DialSummEval: Revisiting Summarization Evaluation for Dialogues
Revised: DialSummEval: Revisiting Summarization Evaluation for Dialogues

Original: Abstract
Revised: Abstract

Original: Dialogue summarization is receiving increasing attention from researchers due to its extraordinary difﬁculty and unique application value.
Revised: Dialogue summarization is receiving increasing attention from researchers due to its extraordinary difﬁculty and unique application value.

Original: We observe that current dialogue summarization models have ﬂaws that may not be well exposed by frequently used metrics such as ROUGE.
Revised: We observe that current dialogue summarization models have ﬂaws that may not be well exposed by frequently used metrics such as ROUGE.

Original: In our paper, we re-evaluatecategories of metrics in terms of four dimensions: coherence, consistency, ﬂuency and relevance, as well as a uniﬁed human evaluation of various models in dialogue summarization for the ﬁrst time.
Revised: In our paper, we re-evaluatecategories of metrics in terms of four dimensions: coherence, consistency, ﬂuency and relevance, as well as a uniﬁed human evaluation of various models in dialogue summarization for the ﬁrst time.

Original: Some noteworthy trends which are different from the conventional summarization tasks are identiﬁed.
Revised: Some noteworthy trends which are different from the conventional summarization tasks are identiﬁed.

Original: We will release
Revised: We will release

Original: DialSummEval, a multi-faceted dataset of human judgments containing the outputs ofmodels on SAMSum.
Revised: DialSummEval, a multi-faceted dataset of human judgments containing the outputs ofmodels on SAMSum.

Original:  Introduction
Revised:  Introduction

Original: Neural network based approaches and sizable datasets have led to signiﬁcant progress in researches towards conventional summarization tasks such as news and scientiﬁc papers
Revised: Neural network based approaches and sizable datasets have led to signiﬁcant progress in researches towards conventional summarization tasks such as news and scientiﬁc papers

Original: ( Lin and
Revised: ( Lin and

Original: Ng, 2019)
Revised: Ng, 2019)

Original: Compared with conventional summarization tasks, dialogue summarization has received increasing attention from researchers due to its great difﬁculty and unique application value ( Feng et al., 2021a).
Revised: Compared with conventional summarization tasks, dialogue summarization has received increasing attention from researchers due to its great difﬁculty and unique application value ( Feng et al., 2021a).

Original: With the proposal of dialogue summary datasets such as SAMSum (Gliwa et al., 2019), DialogSum (Chen et al., 2021) and MediaSum (Zhu et al., 2021), a number of models for automatic generation of dialogue summaries have emerged (Feng et al., 2021b; Liu and Chen, 2021;
Revised: With the proposal of dialogue summary datasets such as SAMSum (Gliwa et al., 2019), DialogSum (Chen et al., 2021) and MediaSum (Zhu et al., 2021), a number of models for automatic generation of dialogue summaries have emerged (Feng et al., 2021b; Liu and Chen, 2021;

Original: Zou et al., 2021;
Revised: Zou et al., 2021;

Original: Qi et al., 2021; Chen and Yang, 2020; Chen and Yang, 2021;
Revised: Qi et al., 2021; Chen and Yang, 2020; Chen and Yang, 2021;

Original: Zhao et al., 2020; Liu et al., 2021).
Revised: Zhao et al., 2020; Liu et al., 2021).

Original: There is no denying that these studies have made promising progress, but it remains a challenge to evaluate these advances comprehensively.
Revised: There is no denying that these studies have made promising progress, but it remains a challenge to evaluate these advances comprehensively.

Original: Current studies generally use the SAMSum dataset and adopt ROUGE (Lin, 2004), an n-grambased automatic evaluation metric using reference summaries, as the overall evaluation criterion for summary quality, complemented by manual evaluation.
Revised: Current studies generally use the SAMSum dataset and adopt ROUGE (Lin, 2004), an n-grambased automatic evaluation metric using reference summaries, as the overall evaluation criterion for summary quality, complemented by manual evaluation.

Original: Schluter (2017) and Graham (2015) illustrate the limitations of ROUGE in evaluating summarization tasks.
Revised: Schluter (2017) and Graham (2015) illustrate the limitations of ROUGE in evaluating summarization tasks.

Original: Also the manual evaluation protocols vary from one research to another based on our observations.
Revised: Also the manual evaluation protocols vary from one research to another based on our observations.

Original: We argue that the inadequate evaluation mechanism may have become a major obstacle to the progress of dialogue summarization researches.
Revised: We argue that the inadequate evaluation mechanism may have become a major obstacle to the progress of dialogue summarization researches.

Original: Many studies, such as Chen and Yang (2020) and Tang et al.
Revised: Many studies, such as Chen and Yang (2020) and Tang et al.

Original: (2021), have pointed out that the current dialogue summarization models still have many shortcomings, such as wrong references, incorrect reasoning and improper gender pronouns, and ROUGE may not reﬂect these problems effectively.
Revised: (2021), have pointed out that the current dialogue summarization models still have many shortcomings, such as wrong references, incorrect reasoning and improper gender pronouns, and ROUGE may not reﬂect these problems effectively.

Original: For example, Gabriel et al.
Revised: For example, Gabriel et al.

Original: (2021) note that ROUGE-1 and ROUGE-L fail to accurately measure factual inconsistency across domains.
Revised: (2021) note that ROUGE-1 and ROUGE-L fail to accurately measure factual inconsistency across domains.

Original: Our case study in Table 1 also illustrates this point.
Revised: Our case study in Table 1 also illustrates this point.

Original: However, it is impractical to perform frequent time-consuming and costly manual evaluation.
Revised: However, it is impractical to perform frequent time-consuming and costly manual evaluation.

Original: The alternative is to introduce or propose more reliable automatic evaluation metrics to evaluate the models in a more comprehensive and ﬁnegrained manner.
Revised: The alternative is to introduce or propose more reliable automatic evaluation metrics to evaluate the models in a more comprehensive and ﬁnegrained manner.

Original: Although there are automatic evaluation metrics for measuring the quality of all aspects of summaries on conventional summarization tasks, especially for factual consistency (Huang et al., 2021), it is difﬁcult to guarantee that they will still perform well on dialogue summarizarion.
Revised: Although there are automatic evaluation metrics for measuring the quality of all aspects of summaries on conventional summarization tasks, especially for factual consistency (Huang et al., 2021), it is difﬁcult to guarantee that they will still perform well on dialogue summarizarion.

Original: Recently proposed automatic metrics for evaluating generic natural language generation tasks such as
Revised: Recently proposed automatic metrics for evaluating generic natural language generation tasks such as

Original: BERTScore (Zhang* et al., 2020), BARTScore (Yuan et al., 2021) have also not been experimented on dialogue summarization.
Revised: BERTScore (Zhang* et al., 2020), BARTScore (Yuan et al., 2021) have also not been experimented on dialogue summarization.

Original: The high abstraction level, low extraction rate, and the requirement for complex reasoning power of the
Revised: The high abstraction level, low extraction rate, and the requirement for complex reasoning power of the

Original: Dialogue
Revised: Dialogue

Original: Reference Summary
Revised: Reference Summary

Original: Generated Summary
Revised: Generated Summary

Original: RR-
Revised: RR-

Original: R-L
Revised: R-L

Original: Kirsten : Youth group this Friday, don’t be late.
Revised: Kirsten : Youth group this Friday, don’t be late.

Original: Alex : What time?
Revised: Alex : What time?

Original: Kirsten : 7 pm.
Revised: Kirsten : 7 pm.

Original: We’re going bowling, so we’ll meet up and then all go together.
Revised: We’re going bowling, so we’ll meet up and then all go together.

Original: Alex : Cool.
Revised: Alex : Cool.

Original: See you.
Revised: See you.

Original: Kirsten : Bye
Revised: Kirsten : Bye

Original: Kirsten reminds Alex that the youth group meets this friday at 7 pm and go bowling.
Revised: Kirsten reminds Alex that the youth group meets this friday at 7 pm and go bowling.

Original: Kirsten is going bowling with her youth group this
Revised: Kirsten is going bowling with her youth group this

Original: Friday at 7 pm.
Revised: Friday at 7 pm.

Original: Ola :
Revised: Ola :

Original: Hey running late
Revised: Hey running late

Original: Ola : I should be free by
Revised: Ola : I should be free by

Original: Kurt :
Revised: Kurt :

Original: Sure no prob, call me
Revised: Sure no prob, call me

Original: Ola should be free by 8.
Revised: Ola should be free by 8.

Original: Kurt wants her to call him.
Revised: Kurt wants her to call him.

Original: Ola will be late.
Revised: Ola will be late.

Original: She should be free by 8.
Revised: She should be free by 8.

Original: Kurt will call her.
Revised: Kurt will call her.

Original: dialogue summarization task present new challenges to automatic evaluation metrics.
Revised: dialogue summarization task present new challenges to automatic evaluation metrics.

Original: There have been a number of manual evaluation datasets and analytical studies for conventional summarization tasks (( Dang and Owczarzak, 2008); Fabbri et al., 2021b;
Revised: There have been a number of manual evaluation datasets and analytical studies for conventional summarization tasks (( Dang and Owczarzak, 2008); Fabbri et al., 2021b;

Original: Bhandari et al., 2020), but very little work has been done on systematic analysis of dialogue summarization models and evaluation metrics.
Revised: Bhandari et al., 2020), but very little work has been done on systematic analysis of dialogue summarization models and evaluation metrics.

Original: Our work will ﬁll the gap in this area and includes the following contributions: 1) We identify evaluation problems in the ﬁeld of dialogue summarization and point out the urgent need of automatic evaluation metrics that better adapt to dialogue summarization.
Revised: Our work will ﬁll the gap in this area and includes the following contributions: 1) We identify evaluation problems in the ﬁeld of dialogue summarization and point out the urgent need of automatic evaluation metrics that better adapt to dialogue summarization.

Original: We collect and provide a sizable, multi-faceted dataset of manual evaluations for dialogue summarization, which contains the output of 14 models, and the dataset will be released.
Revised: We collect and provide a sizable, multi-faceted dataset of manual evaluations for dialogue summarization, which contains the output of 14 models, and the dataset will be released.

Original: 3) We re-evaluate the performance oftypes of automatic evaluation metrics on dialogue summarization.
Revised: 3) We re-evaluate the performance oftypes of automatic evaluation metrics on dialogue summarization.

Original: We evaluate a variety of dialogue summarization models (extractive, abstractive, and recently based on pre-trained language models) in a uniﬁed manner.
Revised: We evaluate a variety of dialogue summarization models (extractive, abstractive, and recently based on pre-trained language models) in a uniﬁed manner.

Original:  Related Work
Revised:  Related Work

Original: Meta-Evaluation with Human Judgments Automatic evaluation Metrics such as ROUGE (Lin, 2004) and BERTScore (Zhang* et al., 2020)) were compared with other metrics when proposed.
Revised: Meta-Evaluation with Human Judgments Automatic evaluation Metrics such as ROUGE (Lin, 2004) and BERTScore (Zhang* et al., 2020)) were compared with other metrics when proposed.

Original: However, they are basically not using the dialogue summarization dataset as an experimental corpus, and rarely provide new human judgments data.
Revised: However, they are basically not using the dialogue summarization dataset as an experimental corpus, and rarely provide new human judgments data.

Original: Bhandari et al.
Revised: Bhandari et al.

Original: (2020) used pyramid (Nenkova and Passonneau, 2004), a widely used human evaluation method on several conventional summarization datasets to obtain relevance scores for some of the system outputs and re-evaluated the metrics in 6 categories.
Revised: (2020) used pyramid (Nenkova and Passonneau, 2004), a widely used human evaluation method on several conventional summarization datasets to obtain relevance scores for some of the system outputs and re-evaluated the metrics in 6 categories.

Original: Similarly, Fabbri et al.
Revised: Similarly, Fabbri et al.

Original: (2021b) used CNN/DailyMail dataset (Hermann et al. , 2015) and the output of some models for human evaluation covering four facets of relevance, consistency, ﬂuency, and coherence, and then reevaluated the metrics in 14 categories.
Revised: (2021b) used CNN/DailyMail dataset (Hermann et al. , 2015) and the output of some models for human evaluation covering four facets of relevance, consistency, ﬂuency, and coherence, and then reevaluated the metrics in 14 categories.

Original: None of these involved dialogue summarization datasets.
Revised: None of these involved dialogue summarization datasets.

Original: Pagnoni et al.
Revised: Pagnoni et al.

Original: (2021) made a careful categorization of factual errors and benchmarked factuality metrics using human annotations they collected on
Revised: (2021) made a careful categorization of factual errors and benchmarked factuality metrics using human annotations they collected on

Original: CNN/DailyMail and XSum dataset (Narayan et al., 2018).
Revised: CNN/DailyMail and XSum dataset (Narayan et al., 2018).

Original: Notably, Gabriel et al.
Revised: Notably, Gabriel et al.

Original: (2021) is one of the few current studies using the dialogue summarization
Revised: (2021) is one of the few current studies using the dialogue summarization

Original: dataset SAMSum ( Gliwa et al., 2019) for meta-evaluation, but it focuses on factual consistency and selects a small number of metrics.
Revised: dataset SAMSum ( Gliwa et al., 2019) for meta-evaluation, but it focuses on factual consistency and selects a small number of metrics.

Original: Analysis and Evaluation for Dialogue Summarization
Revised: Analysis and Evaluation for Dialogue Summarization

Original: Models Tang et al.
Revised: Models Tang et al.

Original: and Chen and Yang (2020) sampled the output of models on SAMSum and analyzes the error types when proposing a new model.
Revised: and Chen and Yang (2020) sampled the output of models on SAMSum and analyzes the error types when proposing a new model.

Original: Due to the different manual evaluation protocols and the small number of models included, it is difﬁcult to comprehensively compare the strengths and weaknesses of different models.
Revised: Due to the different manual evaluation protocols and the small number of models included, it is difﬁcult to comprehensively compare the strengths and weaknesses of different models.

Original: Khalifa et al.
Revised: Khalifa et al.

Original: (2021) designed several tricks to address the special challenges in dialogue summarization and analysized their effects, such as using name substitution to cope with the presence of multiple speakers in dialogues.
Revised: (2021) designed several tricks to address the special challenges in dialogue summarization and analysized their effects, such as using name substitution to cope with the presence of multiple speakers in dialogues.

Original: Zhang et al.
Revised: Zhang et al.

Original: (2021) focused on the problem of lengthy input and relevant information location in long dialogue summarization, and compared the performance of some models and strategies.
Revised: (2021) focused on the problem of lengthy input and relevant information location in long dialogue summarization, and compared the performance of some models and strategies.

Original: No manual evaluation was involved in these studies.
Revised: No manual evaluation was involved in these studies.

Original:  Preliminaries
Revised:  Preliminaries

Original: In this section, we introduce the involved dataset, metrics and models.
Revised: In this section, we introduce the involved dataset, metrics and models.

Original: 3. Dataset
Revised: 3. Dataset

Original: SAMSum (Gliwa et al., 2019) is the ﬁrst manually annotated, high-quality chat summarization dataset, containing over 16k dialogues.
Revised: SAMSum (Gliwa et al., 2019) is the ﬁrst manually annotated, high-quality chat summarization dataset, containing over 16k dialogues.

Original: We use it in this study as it is most widely used and has greatly promoted the research in the ﬁeld of dialogue summarization, and we are able to collect the outputs of various models on this dataset.
Revised: We use it in this study as it is most widely used and has greatly promoted the research in the ﬁeld of dialogue summarization, and we are able to collect the outputs of various models on this dataset.

Original: 3. Evaluation Metrics
Revised: 3. Evaluation Metrics

Original: We selected a number of evaluation metrics that are frequently used on summarization or other natural language generation tasks.
Revised: We selected a number of evaluation metrics that are frequently used on summarization or other natural language generation tasks.

Original: Some are for overall quality; others are speciﬁc to a particular aspect.
Revised: Some are for overall quality; others are speciﬁc to a particular aspect.

Original: Some require reference summaries or source documents; some only need the summary itself.
Revised: Some require reference summaries or source documents; some only need the summary itself.

Original: Here is a brief categorization and description.
Revised: Here is a brief categorization and description.

Original: Metrics based on n-gram overlap include:
Revised: Metrics based on n-gram overlap include:

Original: ROUGE ( Lin, 2004) is the most widely used automatic evaluation metric in summarization.
Revised: ROUGE ( Lin, 2004) is the most widely used automatic evaluation metric in summarization.

Original: Researchers mainly adopt ROUGE-1, ROUGE-2 and
Revised: Researchers mainly adopt ROUGE-1, ROUGE-2 and

Original: ROUGE-L, which measure the unigram-overlap, bigram-overlap and longest common sequence between two texts respectively.
Revised: ROUGE-L, which measure the unigram-overlap, bigram-overlap and longest common sequence between two texts respectively.

Original: BLEU (Papineni et al., 2002) is the primary evaluation metric for machine translation.
Revised: BLEU (Papineni et al., 2002) is the primary evaluation metric for machine translation.

Original: It calculates n-gram overlap between texts using precision scores and includes a brevity penalty.
Revised: It calculates n-gram overlap between texts using precision scores and includes a brevity penalty.

Original: METEOR (Banerjee and Lavie, 2005) computes an alignment by mapping unigrams in two texts, based on surface forms, stemmed forms, and meanings.
Revised: METEOR (Banerjee and Lavie, 2005) computes an alignment by mapping unigrams in two texts, based on surface forms, stemmed forms, and meanings.

Original: CHRF (Popovi´c, 2015) computes character based n-gram overlap between two texts.
Revised: CHRF (Popovi´c, 2015) computes character based n-gram overlap between two texts.

Original: Metrics based on pre-trained language models include:
Revised: Metrics based on pre-trained language models include:

Original: BERTScore (Zhang* et al., 2020) measures the soft-overlap between two texts at token level using contextual embeddings from BERT.
Revised: BERTScore (Zhang* et al., 2020) measures the soft-overlap between two texts at token level using contextual embeddings from BERT.

Original: MoverScore (Zhao et al., 2019) applies the semantic distance between two texts at n-gram level using n-gram embeddings pooled from BERT.
Revised: MoverScore (Zhao et al., 2019) applies the semantic distance between two texts at n-gram level using n-gram embeddings pooled from BERT.

Original: BARTScore (Yuan et al., 2021)
Revised: BARTScore (Yuan et al., 2021)

Original: treats evaluation as a nature language generation task and assumes that when the quality of generated text is better, the conditional language model has a higher probability of generating it from the source text or the reference, or is more likely to generate the reference from it.
Revised: treats evaluation as a nature language generation task and assumes that when the quality of generated text is better, the conditional language model has a higher probability of generating it from the source text or the reference, or is more likely to generate the reference from it.

Original: It can be ﬂexibly applied to evaluation of text from different perspectives using BART.
Revised: It can be ﬂexibly applied to evaluation of text from different perspectives using BART.

Original: BLANC (Vasilyev et al., 2020) is a referenceless metric.
Revised: BLANC (Vasilyev et al., 2020) is a referenceless metric.

Original: It hypothesizes that a good summary is beneﬁcial for a pre-trained language model to conduct language understanding tasks on the source document.
Revised: It hypothesizes that a good summary is beneﬁcial for a pre-trained language model to conduct language understanding tasks on the source document.

Original: Speciﬁcally, it measures the performance boost of the masked language modeling for BERT utilizing the summary in two different ways.
Revised: Speciﬁcally, it measures the performance boost of the masked language modeling for BERT utilizing the summary in two different ways.

Original: PPL , namely perplexity, is often used to evaluate the quality of a language model or the ﬂuency of an utterance.
Revised: PPL , namely perplexity, is often used to evaluate the quality of a language model or the ﬂuency of an utterance.

Original: We adopt GPT-

Revised: We adopt GPT-

Original: ( Radford et al., 2019) as the language model for computing the perplexity for the whole summary.
Revised: ( Radford et al., 2019) as the language model for computing the perplexity for the whole summary.

Original: Metrics based on word embeddings include:
Revised: Metrics based on word embeddings include:

Original: SMS (Clark et al., 2019), namely Sentence
Revised: SMS (Clark et al., 2019), namely Sentence

Original: Mover Similarity, extends Word Movers Distance (Kusner et al., 2015) to measure the distance between two texts which are represented as a bag of sentence embeddings.
Revised: Mover Similarity, extends Word Movers Distance (Kusner et al., 2015) to measure the distance between two texts which are represented as a bag of sentence embeddings.

Original: Embedding average (Landauer and Dumais, 1997) is an embedding based metric computing the cosine similarity between the embeddings of two texts.
Revised: Embedding average (Landauer and Dumais, 1997) is an embedding based metric computing the cosine similarity between the embeddings of two texts.

Original: A sentence-level embedding is represented by averaging the embeddings of the words composing the sentence.
Revised: A sentence-level embedding is represented by averaging the embeddings of the words composing the sentence.

Original: Vector extrema (Forgues et al., 2014) is also an embedding based metric similar to Embedding av- erage.
Revised: Vector extrema (Forgues et al., 2014) is also an embedding based metric similar to Embedding av- erage.

Original: The metric computes a sentence-level embedding by taking the most extreme value of the embeddings of the words composing the sentence for each dimension of the embedding.
Revised: The metric computes a sentence-level embedding by taking the most extreme value of the embeddings of the words composing the sentence for each dimension of the embedding.

Original: Greedy matching (Rus and Lintean, 2012) is another embedding based metric.
Revised: Greedy matching (Rus and Lintean, 2012) is another embedding based metric.

Original: The metric does not compute a sentence-level embedding.
Revised: The metric does not compute a sentence-level embedding.

Original: It directly compares the embeddings of words in the two sentences using a greedy matching algorithm to calculate similarity.
Revised: It directly compares the embeddings of words in the two sentences using a greedy matching algorithm to calculate similarity.

Original: Metrics based on question-answering include:
Revised: Metrics based on question-answering include:

Original: FEQA (Durmus et al., 2020) employs a BERTbased question-answering model to answer questions using source document.
Revised: FEQA (Durmus et al., 2020) employs a BERTbased question-answering model to answer questions using source document.

Original: Questions are generated by a ﬁne-tuned BART model using generated summaries with masked named entities as inputs.
Revised: Questions are generated by a ﬁne-tuned BART model using generated summaries with masked named entities as inputs.

Original: The metric reports F1 scores against the gold answer, which are often regarded as a measure of factual consistency.
Revised: The metric reports F1 scores against the gold answer, which are often regarded as a measure of factual consistency.

Original: SummaQA
Revised: SummaQA

Original: (Scialom et al., 2019) is also a QAbased metric.
Revised: (Scialom et al., 2019) is also a QAbased metric.

Original: Unlike FEQA, it generates questions from source documents instead of summaries to be evaluated and then uses summaries to answer them.
Revised: Unlike FEQA, it generates questions from source documents instead of summaries to be evaluated and then uses summaries to answer them.

Original: The F1 overlap score and QA-model conﬁdence are reported.
Revised: The F1 overlap score and QA-model conﬁdence are reported.

Original: QuestEval (Scialom et al., 2021) is another a
Revised: QuestEval (Scialom et al., 2021) is another a

Original: QA-based metric.
Revised: QA-based metric.

Original: This metric can be considered as a combination of FEQA and SummaQA.
Revised: This metric can be considered as a combination of FEQA and SummaQA.

Original: It takes into account the scores obtained from both styles.
Revised: It takes into account the scores obtained from both styles.

Original: For comparison purposes, We use the reference-less mode.
Revised: For comparison purposes, We use the reference-less mode.

Original: Metrics based on entailment classiﬁcation include:
Revised: Metrics based on entailment classiﬁcation include:

Original: FactCC ( Kryscinski et al., 2020) is a metric based on entailment classiﬁcation.
Revised: FactCC ( Kryscinski et al., 2020) is a metric based on entailment classiﬁcation.

Original: We follow the way Pagnoni et al.
Revised: We follow the way Pagnoni et al.

Original: (2021) used it.
Revised: (2021) used it.

Original: Each sentence of the summary is fed into the classiﬁer together with the document to determine whether the facts are consistent, and the proportion of consistent sentences is used to indicate how consistent the summary is.
Revised: Each sentence of the summary is fed into the classiﬁer together with the document to determine whether the facts are consistent, and the proportion of consistent sentences is used to indicate how consistent the summary is.

Original: DAE (Goyal and Durrett, 2020; Goyal and Durrett, 2021) is an entailment classiﬁcation metric based on dependencies.
Revised: DAE (Goyal and Durrett, 2020; Goyal and Durrett, 2021) is an entailment classiﬁcation metric based on dependencies.

Original: We use it in a similar way to FactCC.
Revised: We use it in a similar way to FactCC.

Original: When a sentence cannot be parsed by the metric, we default it factually inconsistent.
Revised: When a sentence cannot be parsed by the metric, we default it factually inconsistent.

Original: 3. Summarization Models
Revised: 3. Summarization Models

Original: We select some representative models and get the outputs of them on the test set of SAMSum.
Revised: We select some representative models and get the outputs of them on the test set of SAMSum.

Original: We choose LEAD-3 and LONGEST-3 as representatives of the simple extractive approaches.
Revised: We choose LEAD-3 and LONGEST-3 as representatives of the simple extractive approaches.

Original: PGN (See et al., 2017) and Transformer (Vaswani et al., 2017) are selected as representatives of the earlier neural summarization models.
Revised: PGN (See et al., 2017) and Transformer (Vaswani et al., 2017) are selected as representatives of the earlier neural summarization models.

Original: For generic pretrained generative models, we use BART (Lewis et al., 2020), PEGASUS (Zhang et al., 2020) and
Revised: For generic pretrained generative models, we use BART (Lewis et al., 2020), PEGASUS (Zhang et al., 2020) and

Original: UniLM (Dong et al., 2019).
Revised: UniLM (Dong et al., 2019).

Original: We retrain these models above to obtain the outputs and the automatic evaluation results are close to Gliwa et al.
Revised: We retrain these models above to obtain the outputs and the automatic evaluation results are close to Gliwa et al.

Original: and Wu et al.
Revised: and Wu et al.

Original: (2021) in default settings.
Revised: (2021) in default settings.

Original: For models speciﬁcally designed for dialogue summarization, we choose CODS (Wu et al., 2021), ConvoSumm (Fabbri et al., 2021a), MV-BART (Chen and Yang, 2020), PLM-BART (Feng et al., 2021c),
Revised: For models speciﬁcally designed for dialogue summarization, we choose CODS (Wu et al., 2021), ConvoSumm (Fabbri et al., 2021a), MV-BART (Chen and Yang, 2020), PLM-BART (Feng et al., 2021c),

Original: Ctrl-DiaSumm ( Liu and Chen, 2021), S-BART (Chen and Yang, 2021) and the outputs are all provided by their authors.
Revised: Ctrl-DiaSumm ( Liu and Chen, 2021), S-BART (Chen and Yang, 2021) and the outputs are all provided by their authors.

Original: We also regard the reference summary as a kind of model output.
Revised: We also regard the reference summary as a kind of model output.

Original:  Data Annotation 4. Annotation Setup
Revised:  Data Annotation 4. Annotation Setup

Original: Since human evaluation is expensive and timeconsuming, we decide to randomly sample 100 dialogues from the test set of SAMSum and evaluate the summaries generated by all models on these dialogues.
Revised: Since human evaluation is expensive and timeconsuming, we decide to randomly sample 100 dialogues from the test set of SAMSum and evaluate the summaries generated by all models on these dialogues.

Original: To comprehensively evaluate each metric and model, we perform human evaluation in four aspects, as in Kryscinski et al.
Revised: To comprehensively evaluate each metric and model, we perform human evaluation in four aspects, as in Kryscinski et al.

Original: Coherence measures the quality of all sentences in the summary as a whole.
Revised: Coherence measures the quality of all sentences in the summary as a whole.

Original: It focuses on whether the summary is coherent and natural.
Revised: It focuses on whether the summary is coherent and natural.

Original: Consistency measures how well the summary aligns with the dialogue in facts.
Revised: Consistency measures how well the summary aligns with the dialogue in facts.

Original: It focuses on whether the summary contains factual errors.
Revised: It focuses on whether the summary contains factual errors.

Original: Fluency measures the quality of individual sentences in the summary compared to Coherence.
Revised: Fluency measures the quality of individual sentences in the summary compared to Coherence.

Original: It focuses on whether the sentences are well-written and grammatically correct.
Revised: It focuses on whether the sentences are well-written and grammatically correct.

Original: Relevance measures how well the summary captures the key points of the dialogue.
Revised: Relevance measures how well the summary captures the key points of the dialogue.

Original: It focuses on whether all and only the important aspects are contained in the summary.
Revised: It focuses on whether all and only the important aspects are contained in the summary.

Original: To ensure the quality of the annotation, we tried to annotate some of the data ourselves at the beginning to judge the difﬁculty of the task and the approximate time spent.
Revised: To ensure the quality of the annotation, we tried to annotate some of the data ourselves at the beginning to judge the difﬁculty of the task and the approximate time spent.

Original: 4. Annotation Process
Revised: 4. Annotation Process

Original: We initially tried to annotate the data using crowdsourcing platforms.
Revised: We initially tried to annotate the data using crowdsourcing platforms.

Original: We published the annotation task on Amazon Mechanical Turk 16 .
Revised: We published the annotation task on Amazon Mechanical Turk 16 .

Original: The interface contained instructions and deﬁnitions of the four aspects.
Revised: The interface contained instructions and deﬁnitions of the four aspects.

Original: A dialogue and a corresponding summary were included in the interface, and the summaries of different models on the same dialogue were presented to the annotators in a sequence to facilitate comparison.
Revised: A dialogue and a corresponding summary were included in the interface, and the summaries of different models on the same dialogue were presented to the annotators in a sequence to facilitate comparison.

Original: For each dimension/aspect, annotators were asked to rate the summary on a Likert scale from 1 to 5.
Revised: For each dimension/aspect, annotators were asked to rate the summary on a Likert scale from 1 to 5.

Original: Each summary was evaluated by 5 different annotators, and
Revised: Each summary was evaluated by 5 different annotators, and

Original: For each dimension we would receive a total of 100 × 14 × 5 = 7000 human annotations.
Revised: For each dimension we would receive a total of 100 × 14 × 5 = 7000 human annotations.

Original: The annotation was done quickly in one day, but the quality was not satisfactory.
Revised: The annotation was done quickly in one day, but the quality was not satisfactory.

Original: We calculated the average score of each model in each aspect based on these annotation data and found that the scores of the models are close in each dimension, which is not in the accordance with the reality.
Revised: We calculated the average score of each model in each aspect based on these annotation data and found that the scores of the models are close in each dimension, which is not in the accordance with the reality.

Original: For example, in terms of consistency, the reference summary and the extractive approaches should have had a deﬁnite advantage, but this failed to be reﬂected from the data.
Revised: For example, in terms of consistency, the reference summary and the extractive approaches should have had a deﬁnite advantage, but this failed to be reﬂected from the data.

Original: The result is shown in Table 5.
Revised: The result is shown in Table 5.

Original: For reliability reasons, we do not use these annotations for our analysis.
Revised: For reliability reasons, we do not use these annotations for our analysis.

Original: Then, we decided to recruit annotators from the school forum who are required to be capable of reading daily conversations and articles in English ﬂuently.
Revised: Then, we decided to recruit annotators from the school forum who are required to be capable of reading daily conversations and articles in English ﬂuently.

Original: We recruited three annotators, using a similar annotation interface and approach as in the crowd-sourcing platforms.
Revised: We recruited three annotators, using a similar annotation interface and approach as in the crowd-sourcing platforms.

Original: These annotators were college students and they are ﬂuent in English.
Revised: These annotators were college students and they are ﬂuent in English.

Original: The differences with the crowd-sourcing platform annotation are as follows: 1) For a student who wanted to participate in the annotation, we would ask him to annotate all models on the ﬁrst 10 conversations ( 10 × 14 = 140 annotations), and let her/him continue the annotation only when these annotation results were checked by us to conﬁrm that the annotator had understood the task correctly and could ﬁnish the annotation responsibly.
Revised: The differences with the crowd-sourcing platform annotation are as follows: 1) For a student who wanted to participate in the annotation, we would ask him to annotate all models on the ﬁrst 10 conversations ( 10 × 14 = 140 annotations), and let her/him continue the annotation only when these annotation results were checked by us to conﬁrm that the annotator had understood the task correctly and could ﬁnish the annotation responsibly.

Original: Otherwise, we paid the annotator directly for this part and terminated his annotation task. 2)
Revised: Otherwise, we paid the annotator directly for this part and terminated his annotation task. 2)

Original: We required each annotator to annotate all data ( 100 × 14 = 1400 annotations) to ensure the consistency within the annotator.
Revised: We required each annotator to annotate all data ( 100 × 14 = 1400 annotations) to ensure the consistency within the annotator.

Original: During the annotation process, we kept in touch with the annotators via email or instant messaging app to answer their questions at any time.
Revised: During the annotation process, we kept in touch with the annotators via email or instant messaging app to answer their questions at any time.

Original: It took around 10 days to ﬁnish the annotation.
Revised: It took around 10 days to ﬁnish the annotation.

Original: We received 100 × 14 × 3 = 4200 annotations for each perspective.
Revised: We received 100 × 14 × 3 = 4200 annotations for each perspective.

Original: For each aspect of each summary, if two scores were the same and the other was different from them, we considered the different one as noise.
Revised: For each aspect of each summary, if two scores were the same and the other was different from them, we considered the different one as noise.

Original: For each dimension, we removed the noise separately and calculated the the Krippendorff’s alpha coefﬁcient (Krippendorff, 2011).
Revised: For each dimension, we removed the noise separately and calculated the the Krippendorff’s alpha coefﬁcient (Krippendorff, 2011).

Original: We found the inter-annotator interval metric to be within an acceptable range - from 0.5621 to 0.7564, as detailed in Table 2.
Revised: We found the inter-annotator interval metric to be within an acceptable range - from 0.5621 to 0.7564, as detailed in Table 2.

Original: The raw annotated data will be released and we use the cleaned data for analysis.
Revised: The raw annotated data will be released and we use the cleaned data for analysis.

Original: At last, we use the average of the cleaned data to represent the human evaluation score of an summary on a dimension.
Revised: At last, we use the average of the cleaned data to represent the human evaluation score of an summary on a dimension.

Original:  Metric Evaluation
Revised:  Metric Evaluation

Original: In this section, we will introduce several deﬁnitions in meta-evaluation and re-evaluate the metrics mentioned in Section 3.2.
Revised: In this section, we will introduce several deﬁnitions in meta-evaluation and re-evaluate the metrics mentioned in Section 3.2.

Original: 5. Task Formulation
Revised: 5. Task Formulation

Original: As mentioned by Bhandari et al.
Revised: As mentioned by Bhandari et al.

Original: (2020), there are two common ways to measure the correlation of automatic evaluation metrics to manual evaluation: system-level and summary-level.
Revised: (2020), there are two common ways to measure the correlation of automatic evaluation metrics to manual evaluation: system-level and summary-level.

Original: Assuming there are N dialogues, the i -th dialogue is represented as d i .
Revised: Assuming there are N dialogues, the i -th dialogue is represented as d i .

Original: For a dialogue d i , there are J summaries generated by J models, and we denote each of them as s ij , j = 1 · · · J .
Revised: For a dialogue d i , there are J summaries generated by J models, and we denote each of them as s ij , j = 1 · · · J .

Original: There are K evaluation metrics (or human evaluation) in total, and m k refers to an automatic evaluation metric or human evaluation of a certain dimension.
Revised: There are K evaluation metrics (or human evaluation) in total, and m k refers to an automatic evaluation metric or human evaluation of a certain dimension.

Original: m k
Revised: m k

Original: ( s ij ) means the score of k -th metric towards a pair of dialogue and summary ( d i , s ij ) .
Revised: ( s ij ) means the score of k -th metric towards a pair of dialogue and summary ( d i , s ij ) .

Original: We use
Revised: We use

Original: R ( m i , m j ) to denote the correlation coefﬁcient between two metrics m i and m j .
Revised: R ( m i , m j ) to denote the correlation coefﬁcient between two metrics m i and m j .

Original: System-level correlation is deﬁned as follows.
Revised: System-level correlation is deﬁned as follows.

Original: The corresponding p-value which indicates statistical signiﬁcance can be obtained:
Revised: The corresponding p-value which indicates statistical signiﬁcance can be obtained:

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Summary-level correlation is deﬁned as follows, and the p-value cannot be derived here because
Revised: Summary-level correlation is deﬁned as follows, and the p-value cannot be derived here because

Original: the
Revised: the

Original: Summary-level correlation is an average value:
Revised: Summary-level correlation is an average value:

Original: [Equation]
Revised: [Equation]

Original: 5. Discussion
Revised: 5. Discussion

Original: Comparing the performance of various metrics reveals some trends in Table 3.
Revised: Comparing the performance of various metrics reveals some trends in Table 3.

Original: In each dimension, metrics which are strongly correlated with human judgments exist, but few metrics show signiﬁcant strengths in all four dimensions.
Revised: In each dimension, metrics which are strongly correlated with human judgments exist, but few metrics show signiﬁcant strengths in all four dimensions.

Original: Of all the metrics, QuestEval has the most comprehensive capabilities at the system level.
Revised: Of all the metrics, QuestEval has the most comprehensive capabilities at the system level.

Original: Generally metrics that perform better on coherence and ﬂuency perform worse on consistency and relevance, and vice versa.
Revised: Generally metrics that perform better on coherence and ﬂuency perform worse on consistency and relevance, and vice versa.

Original: This can be attributed to the deﬁnition of the dimensions, i.e. there is some correlation between the four dimensions themselves, which is shown in Figure 4.
Revised: This can be attributed to the deﬁnition of the dimensions, i.e. there is some correlation between the four dimensions themselves, which is shown in Figure 4.

Original: In all dimensions, automatic evaluation metrics based on pre-trained language models generally outperform metrics based on n-gram overlap and context-independent word embedding.
Revised: In all dimensions, automatic evaluation metrics based on pre-trained language models generally outperform metrics based on n-gram overlap and context-independent word embedding.

Original: Among them, the recently proposed BARTScore and the increasingly popular QA-based metrics perform the best.
Revised: Among them, the recently proposed BARTScore and the increasingly popular QA-based metrics perform the best.

Original: This suggests that both directions have the potential to be explored in terms of evaluation for dialogue summarization.
Revised: This suggests that both directions have the potential to be explored in terms of evaluation for dialogue summarization.

Original: Across dimensions, almost all metrics correlate better with human judgments at the system level than at the summary level, and both showed good agreement with each other.
Revised: Across dimensions, almost all metrics correlate better with human judgments at the system level than at the summary level, and both showed good agreement with each other.

Original: This indicates that the summarylevel correlations are also worth referring to when enough data are not available for system-level analysis.
Revised: This indicates that the summarylevel correlations are also worth referring to when enough data are not available for system-level analysis.

Original: In addition, metrics such as BLEU and
Revised: In addition, metrics such as BLEU and

Original: CHRF, which are frequently used in other natural language generation tasks (e.g., machine translation, dialogue, etc.), do not show advantages on dialogue summarization.
Revised: CHRF, which are frequently used in other natural language generation tasks (e.g., machine translation, dialogue, etc.), do not show advantages on dialogue summarization.

Original: The characteristics presented by the automatic evaluation metrics on the dialogue summarization differ from those of the conventional summarization tasks.
Revised: The characteristics presented by the automatic evaluation metrics on the dialogue summarization differ from those of the conventional summarization tasks.

Original: For ROUGE, we ﬁnd that increasing the size of n in ROUGE-n is not better in almost all dimensions, which is different from the ﬁndings of Rankel et al.
Revised: For ROUGE, we ﬁnd that increasing the size of n in ROUGE-n is not better in almost all dimensions, which is different from the ﬁndings of Rankel et al.

Original: and Fabbri et al.
Revised: and Fabbri et al.

Original: (2021b).
Revised: (2021b).

Original: The ability of ROUGE to reﬂect content selection, i.e., relevance, as we usually believe, is also questionable.
Revised: The ability of ROUGE to reﬂect content selection, i.e., relevance, as we usually believe, is also questionable.

Original: Compared to the results of Fabbri et al.
Revised: Compared to the results of Fabbri et al.

Original: (2021b), metrics based on n-gram overlap such as ROUGE and CHRF perform worse on dialogue summarization, while some metrics that use source documents such as BLANC perform better.
Revised: (2021b), metrics based on n-gram overlap such as ROUGE and CHRF perform worse on dialogue summarization, while some metrics that use source documents such as BLANC perform better.

Original: We need to focus on the limitations of ROUGE and the role of the source dialogues in evaluating dialogue summaries.
Revised: We need to focus on the limitations of ROUGE and the role of the source dialogues in evaluating dialogue summaries.

Original: We have also observed some interesting phenomena.
Revised: We have also observed some interesting phenomena.

Original: Entailment classiﬁcation metrics such as FactCC and DAE outperform many metrics in terms of consistency, but not as well as
Revised: Entailment classiﬁcation metrics such as FactCC and DAE outperform many metrics in terms of consistency, but not as well as

Original: BARTScore and QA-based metrics.
Revised: BARTScore and QA-based metrics.

Original: This may be due to the large gap between the corpus used in training and dialogues, and the need to slice the summaries by sentence when using them.
Revised: This may be due to the large gap between the corpus used in training and dialogues, and the need to slice the summaries by sentence when using them.

Original: FEQA, which is designed for factual consistency, however, performs best in coherence and ﬂuency, and rather poorly in consistency and relevance.
Revised: FEQA, which is designed for factual consistency, however, performs best in coherence and ﬂuency, and rather poorly in consistency and relevance.

Original: Comparing its performance with QuestEval and SummaQA, generating questions from the original dialogue may be more reliable in measuring consistency, which corroborates with the points of Gabriel et al.
Revised: Comparing its performance with QuestEval and SummaQA, generating questions from the original dialogue may be more reliable in measuring consistency, which corroborates with the points of Gabriel et al.

Original: Coherence
Revised: Coherence

Original: Consistency
Revised: Consistency

Original: [Table]
Revised: [Table]

Original: 0.56 ∗ 0.
Revised: 0.56 ∗ 0.

Original: BARTScore-r-h 0.67 ∗∗ 0.
Revised: BARTScore-r-h 0.67 ∗∗ 0.

Original: 0.67 ∗∗ 0.
Revised: 0.67 ∗∗ 0.

Original: BLANC-help + -0.
Revised: BLANC-help + -0.

Original: 0.60 ∗ 0.
Revised: 0.60 ∗ 0.

Original: BLANC-tune + -0.
Revised: BLANC-tune + -0.

Original: 0.56 ∗ 0.
Revised: 0.56 ∗ 0.

Original: FEQA + 0.82 ∗∗ 0.
Revised: FEQA + 0.82 ∗∗ 0.

Original: 0.84 ∗∗ 0.
Revised: 0.84 ∗∗ 0.

Original: QuestEval + 0.
Revised: QuestEval + 0.

Original: 0.85 ∗∗ 0.
Revised: 0.85 ∗∗ 0.

Original: 0.75 ∗∗ 0.
Revised: 0.75 ∗∗ 0.

Original: 0.83 ∗∗ 0.
Revised: 0.83 ∗∗ 0.

Original: SummaQA-conf
Revised: SummaQA-conf

Original: + -0.
Revised: + -0.

Original: 0.64 ∗ 0.
Revised: 0.64 ∗ 0.

Original: 0.67 ∗∗ 0.
Revised: 0.67 ∗∗ 0.

Original: SummaQA-fscore + -0.
Revised: SummaQA-fscore + -0.

Original: 0.58 ∗ 0.
Revised: 0.58 ∗ 0.

Original: 0.62 ∗ 0.
Revised: 0.62 ∗ 0.

Original: PPL - -0.
Revised: PPL - -0.

Original: CHRF 0.
Revised: CHRF 0.

Original: BLEU0. BLEU0.
Revised: BLEU0. BLEU0.

Original: BLEU0.
Revised: BLEU0.

Original: [Table]
Revised: [Table]

Original: summary and then conducting nature language understanding tasks on source dialogues, while BLANC-help refers to the way of inferring with a generated summary concatenated together.
Revised: summary and then conducting nature language understanding tasks on source dialogues, while BLANC-help refers to the way of inferring with a generated summary concatenated together.

Original: For SummaQA, SummaQA-fscore measures the average overlap between predictions and ground truth answers, and SummaQA-conf corresponds to the conﬁdence of the predictions.
Revised: For SummaQA, SummaQA-fscore measures the average overlap between predictions and ground truth answers, and SummaQA-conf corresponds to the conﬁdence of the predictions.

Original: [Table]
Revised: [Table]

Original: PEGASUS
Revised: PEGASUS

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: BART
Revised: BART

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: BART
Revised: BART

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: It is surprising that metrics based on the language model such as PPL, BARTScore-h performs poorly in measuring both coherence and ﬂuency.
Revised: It is surprising that metrics based on the language model such as PPL, BARTScore-h performs poorly in measuring both coherence and ﬂuency.

Original: The exact reasons for this need further investigation.
Revised: The exact reasons for this need further investigation.

Original:  Model Evaluation
Revised:  Model Evaluation

Original: In each dimension, we evaluate each model mentioned in Section 3.3 using the average of the human evaluation scores of all summaries.
Revised: In each dimension, we evaluate each model mentioned in Section 3.3 using the average of the human evaluation scores of all summaries.

Original: Analyzing Table 4, we conclude the following.
Revised: Analyzing Table 4, we conclude the following.

Original: The reference summaries in SAMSum are not perfect, and the annotators felt that they also contained some factual inconsistencies compared to the source dialogues, as well as important elements of the dialogues that were not all captured by them.
Revised: The reference summaries in SAMSum are not perfect, and the annotators felt that they also contained some factual inconsistencies compared to the source dialogues, as well as important elements of the dialogues that were not all captured by them.

Original: However, comparing the human evaluation scores of the reference summaries in CNNDM (Fabbri et al., 2021b), the quality is already superior.
Revised: However, comparing the human evaluation scores of the reference summaries in CNNDM (Fabbri et al., 2021b), the quality is already superior.

Original: Extractive models produce summaries that differ in style from abstractive models, and many conversations contain ungrammatical utterances, which can affect the reading experience and impair their ﬂuency and coherence.
Revised: Extractive models produce summaries that differ in style from abstractive models, and many conversations contain ungrammatical utterances, which can affect the reading experience and impair their ﬂuency and coherence.

Original: In particular,
Revised: In particular,

Original: LONGEST-3, which extracts some potentially discontinuous sentences from dialogues, has low coherence.
Revised: LONGEST-3, which extracts some potentially discontinuous sentences from dialogues, has low coherence.

Original: However, since they do not modify the content, they still perform well in terms of consistency.
Revised: However, since they do not modify the content, they still perform well in terms of consistency.

Original: Since the average length of dialogues in SAMSum is small, extracting a few sentences from it can generally include important contents, so the relevance is also high.
Revised: Since the average length of dialogues in SAMSum is small, extracting a few sentences from it can generally include important contents, so the relevance is also high.

Original: The evaluation of the extractive models raises a qustion: what kind of summaries do readers actually want?
Revised: The evaluation of the extractive models raises a qustion: what kind of summaries do readers actually want?

Original: The early neural summrization models represented by PGN and Transformer perform relatively poorly in all dimensions compared to the reference summaries, especially consistency and relevance.
Revised: The early neural summrization models represented by PGN and Transformer perform relatively poorly in all dimensions compared to the reference summaries, especially consistency and relevance.

Original: This is to be expected because of the high difﬁculty of dialogue summarization and the small size of SAMSum dataset.
Revised: This is to be expected because of the high difﬁculty of dialogue summarization and the small size of SAMSum dataset.

Original: An important ﬁnding is that the generic pretrained language models represented by BART,
Revised: An important ﬁnding is that the generic pretrained language models represented by BART,

Original: PEGASUS and UniLM, and various recently proposed models speciﬁcally designed on the dialogue summarization task do not have signiﬁcant differences in each dimension.
Revised: PEGASUS and UniLM, and various recently proposed models speciﬁcally designed on the dialogue summarization task do not have signiﬁcant differences in each dimension.

Original: They are already comparable, and in some cases better, in terms of coherence and ﬂuency compared to the reference summaries.
Revised: They are already comparable, and in some cases better, in terms of coherence and ﬂuency compared to the reference summaries.

Original: They have improved dramatically compared to earlier neural summarization models with respect to consistency and relevance, but there is still some room for enhancement.
Revised: They have improved dramatically compared to earlier neural summarization models with respect to consistency and relevance, but there is still some room for enhancement.

Original: On the one hand, this ﬁnding afﬁrms the capability of these models; On the other hand, it urges us to reﬂect on how much these recently proposed complex models or fancy techniques are an improvement over the generic pre-trained language mod- els.
Revised: On the one hand, this ﬁnding afﬁrms the capability of these models; On the other hand, it urges us to reﬂect on how much these recently proposed complex models or fancy techniques are an improvement over the generic pre-trained language mod- els.

Original:  Conclusion
Revised:  Conclusion

Original: We point out the problems with the evaluation in the dialogue summarization and introduce DialSummEval, a multi-faceted dataset containing the output of various models and the corresponding human judgments.
Revised: We point out the problems with the evaluation in the dialogue summarization and introduce DialSummEval, a multi-faceted dataset containing the output of various models and the corresponding human judgments.

Original: Based on this dataset, we provide a comprehensive re-evaluation and analysis of the performance of widely used automatic evaluation metrics and each model.
Revised: Based on this dataset, we provide a comprehensive re-evaluation and analysis of the performance of widely used automatic evaluation metrics and each model.

Original: There are three important ﬁndings: 1) Few metrics are excellent in all dimensions, and the recently proposed
Revised: There are three important ﬁndings: 1) Few metrics are excellent in all dimensions, and the recently proposed

Original: BARTScore and QA-based metrics are comparatively outstanding and worth exploring.
Revised: BARTScore and QA-based metrics are comparatively outstanding and worth exploring.

Original: The automatic evaluation metrics and their variants present some trends that differ from conventional summarization.
Revised: The automatic evaluation metrics and their variants present some trends that differ from conventional summarization.

Original: A variety of models specifically designed for dialogue summarization perform comparably to reference summaries in terms of coherence and ﬂuency, but still have shortcomings in consistency and relevance.
Revised: A variety of models specifically designed for dialogue summarization perform comparably to reference summaries in terms of coherence and ﬂuency, but still have shortcomings in consistency and relevance.

Original: We hope that researchers in the ﬁeld recognize the importance of evaluation in current research, choose some other metrics in addition to ROUGE when evaluating models, propose automatic evaluation metrics that can be better adapted to the ﬁeld of dialogue summarization based on our work.
Revised: We hope that researchers in the ﬁeld recognize the importance of evaluation in current research, choose some other metrics in addition to ROUGE when evaluating models, propose automatic evaluation metrics that can be better adapted to the ﬁeld of dialogue summarization based on our work.

Original:  Ethical Considerations
Revised:  Ethical Considerations

Original: Whether recruiting annotators through Amazon
Revised: Whether recruiting annotators through Amazon

Original: Mechanical Turk or campus, we paid them 15 dollars per hour, more than the local average minimum wage.
Revised: Mechanical Turk or campus, we paid them 15 dollars per hour, more than the local average minimum wage.

Original: We removed all content in the dataset that might contain personal information about the annotators.
Revised: We removed all content in the dataset that might contain personal information about the annotators.

Original: Acknowledgements
Revised: Acknowledgements

Original: We thank all authors for providing the summaries generated by their models.
Revised: We thank all authors for providing the summaries generated by their models.

Original: We thank
Revised: We thank

Original: Baizhou Huang for his help in the process of retraining some models.
Revised: Baizhou Huang for his help in the process of retraining some models.

Original: This work was supported by National Science Foundation of China (No. 62161160339), National Key RD Program of China (No.2018YFB1005100), State Key Laboratory of Media Convergence Production Technology and Systems and Key Laboratory of Science, Technology and Standard in Press Industry (Key Laboratory of Intelligent Press Media Technology).
Revised: This work was supported by National Science Foundation of China (No. 62161160339), National Key RD Program of China (No.2018YFB1005100), State Key Laboratory of Media Convergence Production Technology and Systems and Key Laboratory of Science, Technology and Standard in Press Industry (Key Laboratory of Intelligent Press Media Technology).

Original: We appreciate the anonymous reviewers for their helpful comments.
Revised: We appreciate the anonymous reviewers for their helpful comments.

Original: Xiaojun Wan is the corresponding author.
Revised: Xiaojun Wan is the corresponding author.

Original: A Annotation Interface
Revised: A Annotation Interface

Original: Figure 1 and Figure 2 show the instructions for annotation and deﬁnition of each aspect.
Revised: Figure 1 and Figure 2 show the instructions for annotation and deﬁnition of each aspect.

Original: They were read by all annotators.
Revised: They were read by all annotators.

Original: Figure 3 shows a source dialogue and a summary to be evaluated.
Revised: Figure 3 shows a source dialogue and a summary to be evaluated.

Original: B Correlation between different dimensions
Revised: B Correlation between different dimensions

Original: Figure 4 shows the system-level correlation between coherence, consistency, ﬂuency and relevance.
Revised: Figure 4 shows the system-level correlation between coherence, consistency, ﬂuency and relevance.

Original: Consistency is strongly correlated with relevance.
Revised: Consistency is strongly correlated with relevance.

Original: C Correlation between different metrics
Revised: C Correlation between different metrics

Original: Figure 5 shows the system-level correlation between different metrics.
Revised: Figure 5 shows the system-level correlation between different metrics.

Original: D Reasons for discarding data from Amazon Mechanical Turk
Revised: D Reasons for discarding data from Amazon Mechanical Turk

Original: Table 5 shows the result of model evaluation using annotations from Amazon Mechanical Turk.
Revised: Table 5 shows the result of model evaluation using annotations from Amazon Mechanical Turk.

Original: The performance of the models is indistinguishable, which is not consistent with our observation.
Revised: The performance of the models is indistinguishable, which is not consistent with our observation.

Original: E The evaluation results for the models we reproduced
Revised: E The evaluation results for the models we reproduced

Original: Table 6 shows the value of ROUGE-1, ROUGE
Revised: Table 6 shows the value of ROUGE-1, ROUGE

Original: 2 and ROUGE-L on the test set of SAMSum for the models we reproduced.
Revised: 2 and ROUGE-L on the test set of SAMSum for the models we reproduced.

Original: The results is close to those in Gliwa et al.
Revised: The results is close to those in Gliwa et al.

Original: and Wu et al.
Revised: and Wu et al.

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Figure]

Revised: C o h e r e n c e

Revised: C o n s i s t e n c y

Revised: F l u e n c y

Revised: R e l e v

Revised: [Figure]

Revised: n c e

Revised: [Figure]

Revised: r o u g e

Revised: [Table]

Revised: 1 r o u g e

Revised: [Table]

Revised: 2 r o u

Revised: [Table]

Revised: e

Revised: [Table]

Revised: 3 r o u

Revised: [Table]

Revised: e

Revised: [Table]

Revised: 4 r o u

Revised: [Table]

Revised: e

Original: [Table]
Revised: [Table]

Revised: l b e r t s c o r e _ p b e r t s c o r e _ r b

Revised: e r t s c o r e _ fm o v e r s c o r e s m s b a r t s c o r e _ s _ h b a r t s c

Revised: o r e _

Revised: h b a r t s c o r e _ h _ r b a r t s c

Revised: o r e _ r _ h b l a n c _

Revised: h e l p b l a n c _ t un e f e q a _ s c o r e q u e s t e v a l s u m m a q a _ p r o b s u mm a q a _ f s c

Revised: o r e pp l c h r f

Revised: B l e u _ 1 B l e u _ 2 B

Revised: l

Revised: e

Revised: u _ 3 B l e u _

Revised: M E T E

Revised: O

Revised: R

Revised: E m b e dd i n g A v e r a g e

Revised: V e c t

Revised: o r

Revised: E x t r e m a

Revised: G r ee d y

Revised: M a t c h i n

Revised: [Table]

Revised: [Figure]