Original: WSSEMR: Weakly Supervised Summary Evaluation by Mutating References
Revised: SueNes: A Weakly Supervised Approach to Evaluating Single-Document Summarization via Negative Sampling

Original: Abstract
Revised: Abstract

Original: Canonical automatic summary evaluation metrics, such as ROUGE, focus on lexical similarity which cannot well capture semantics nor linguistic quality and require a reference summary which is costly to obtain.
Revised: Canonical automatic summary evaluation metrics, such as ROUGE, focus on lexical similarity which cannot well capture semantics nor linguistic quality and require a reference summary which is costly to obtain.

Original: Recently, there have been a growing number of efforts to alleviate either or both of the two drawbacks.
Revised: Recently, there have been a growing number of efforts to alleviate either or both of the two drawbacks.

Original: In this paper, we  study  weakly supervised summary evaluation by mutating references (WSSEMR).
Revised: In this paper, we present a proof-of-concept study to a weakly supervised summary evaluation approach without the presence of reference summaries.

Original: Massive data in existing summarization datasets are transformed for training by pairing documents with corrupted reference summaries.
Revised: Massive data in existing summarization datasets are transformed for training by pairing documents with corrupted reference summaries.

Original: In cross-domain tests, our strategy outperforms baselines with promising improvements, and show a great advantage in gauging linguistic qualities over all metrics.
Revised: In cross-domain tests, our strategy outperforms baselines with promising improvements, and show a great advantage in gauging linguistic qualities over all metrics.

Original:  Introduction
Revised:  Introduction

Original: In natural language processing, the problem of summarization studies generating a summary from a source document which is longer than the summary.
Revised: In natural language processing, the problem of summarization studies generating a summary from a source document which is longer than the summary.

Original: De facto metrics to judge a generated summary include ROUGE (Lin, 2004), BLEU (Papineni et al., 2002) and METEOR (Banerjee and Lavie, 2005).
Revised: De facto metrics to judge a generated summary include ROUGE (Lin, 2004), BLEU (Papineni et al., 2002) and METEOR (Banerjee and Lavie, 2005).

Original: Previous work (Ng and Abrecht, 2015; Liu and Liu, 2008; Liu et al., 2016; Shang et al., 2018) agrees on two major drawbacks of them: 1) they favor lexical similarity, falling short on semantic similarity or linguistic quality, and 2) they require a reference summary which is often expensive to obtain (Zopf, 2018).
Revised: Previous work (Ng and Abrecht, 2015; Liu and Liu, 2008; Liu et al., 2016; Shang et al., 2018) agrees on two major drawbacks of them: 1) they favor lexical similarity, falling short on semantic similarity or linguistic quality, and 2) they require a reference summary which is often expensive to obtain (Zopf, 2018).

Original: Initially, the first drawback is partially alleviated by replacing lexicons with their word embeddings (Ng and Abrecht, 2015; Ellouze et al., 2017;
Revised: Initially, the first drawback is partially alleviated by replacing lexicons with their word embeddings (Ng and Abrecht, 2015; Ellouze et al., 2017;

Original: Ruseti et al., 2018; Xia et al., 2019).
Revised: Ruseti et al., 2018; Xia et al., 2019).

Original: After the birth of transformers (Vaswani et al., 2017), this effort has expanded to sentence or document level, including reference-based (Zhang* et al., 2020;
Revised: After the birth of transformers (Vaswani et al., 2017), this effort has expanded to sentence or document level, including reference-based (Zhang* et al., 2020;

Original: Zhao et al., 2019), and reference-free ones (Vasilyev et al., 2020;
Revised: Zhao et al., 2019), and reference-free ones (Vasilyev et al., 2020;

Original: Scialom et al., 2019; Gao et al., 2020).
Revised: Scialom et al., 2019; Gao et al., 2020).

Original: The main difference between the two groups is whether a reference summary is needed when evaluating a machine-generated summary.
Revised: The main difference between the two groups is whether a reference summary is needed when evaluating a machine-generated summary.

Original: The two groups have complementary pros and cons.
Revised: The two groups have complementary pros and cons.

Original: Reference-based metrics have a better performance, but they are impractical when summarization is used industrially, such as in customer support (Liu et al., 2019), team conversation (Zhang and Cranshaw, 2018), and bug reporting (Rastkar et al., 2014), where it is too costly to manually craft an equally massive amount of reference summaries.
Revised: Reference-based metrics have a better performance, but they are impractical when summarization is used industrially, such as in customer support (Liu et al., 2019), team conversation (Zhang and Cranshaw, 2018), and bug reporting (Rastkar et al., 2014), where it is too costly to manually craft an equally massive amount of reference summaries.

Original: In contrast, without human written reference summaries, reference-free approaches generally perform poorer.
Revised: In contrast, without human written reference summaries, reference-free approaches generally perform poorer.

Original: Modern transformer-based referencefree approaches often rely on non-summarization tasks, such as QA (Vasilyev et al., 2020;
Revised: Modern transformer-based referencefree approaches often rely on non-summarization tasks, such as QA (Vasilyev et al., 2020;

Original: Scialom et al., 2019).
Revised: Scialom et al., 2019).

Original: Such fact-focused approach makes them excel on content/fact aspects (still worse than reference-based ones) but not on linguistic ones.
Revised: Such fact-focused approach makes them excel on content/fact aspects (still worse than reference-based ones) but not on linguistic ones.

Original: The non-summarization tasks also introduce noises.
Revised: The non-summarization tasks also introduce noises.

Original: Therefore, in this paper, as a proof of concept, we explore a hybrid or middle approach to combine the best of both worlds.
Revised: Therefore, in this paper, as a proof of concept, we explore a hybrid or middle approach to combine the best of both worlds.

Original: Using documentsummary pairs in existing summarization datasets, our weakly supervised approach mutates * reference summaries and pair them with documents to form training data and then use the trained model to evaluate unseen summaries in the presence of documents without corresponding reference sum- maries.
Revised: Using documentsummary pairs in existing summarization datasets, our weakly supervised approach mutates * reference summaries and pair them with documents to form training data and then use the trained model to evaluate unseen summaries in the presence of documents without corresponding reference summaries.

Original: In this way, we make use of human written summaries, which are very precious, in training, but we do not need them in summary evaluation.
Revised: In this way, we make use of human written summaries, which are very precious, in training, but we do not need them in summary evaluation.

Original: We call our approach WSSEMR , which stands for “Weakly Supervised Summary Evaluation by Mutating References.”
Revised: We call our approach SueNes , which stands for “ Su mmary e valuation by Ne gative s ampling.”

Original: The quality of a summary is usually evaluated on two facets: content/fact aspects and linguistic qualities.
Revised: The quality of a summary is usually evaluated on two facets: content/fact aspects and linguistic qualities.

Original: Experiments later will show that a value of our approach is that we can use the same model architecture to build models that excel on different tasks by feeding training data from the same source but mutated in different strategies.
Revised: Experiments later will show that a value of our approach is that we can use the same model architecture to build models that excel on different tasks by feeding training data from the same source but mutated in different strategies.

Original: For example, deleting words is the best for linguistic qualities while deleting sentences is the best for content/fact coverage.
Revised: For example, deleting words is the best for linguistic qualities while deleting sentences is the best for content/fact coverage.

Original: Our approach is empirically compared against an array of existing metrics on three human summary evaluation datasets.
Revised: Our approach is empirically compared against an array of existing metrics on three human summary evaluation datasets.

Original: Despite being training-based, our approach exhibits consistent results across various training domains which are all different from the test domain.
Revised: Despite being training-based, our approach exhibits consistent results across various training domains which are all different from the test domain.

Original: It outperforms reference-free baselines with promising improvements on content/fact aspects, and further outperforms all existing metrics in gauging linguistic qualities.
Revised: It outperforms reference-free baselines with promising improvements on content/fact aspects, and further outperforms all existing metrics in gauging linguistic qualities.

Original: In summary, our contributions or merits are:
Revised: In summary, our contributions or merits are:

Original: • a simple but effective approach to referencefree † summary quality assessment, • data mutation for preparing training data from the unlabeled, • one task/framework for multi-aspect judging, • extensive cross-domain experiments to validate the effectiveness and domain robustness of our approach.
Revised: • a simple but effective approach to referencefree † summary quality assessment, • negative sampling for preparing training data from the unlabeled, • one task/framework for multi-aspect judging, • extensive cross-domain experiments to validate the effectiveness and domain robustness of our approach.

Original: We hope our study can inspire more research into hybridizing reference-free and reference-based summary evaluation.
Revised: We hope our study can inspire more research into hybridizing reference-free and reference-based summary evaluation.

Original: Our code is at https:// github.com/forrestbao/WSSEMR/
Revised: Our code is at https:// github.com/forrestbao/SueNes/

Original:  The Approach 2. Model Architecture
Revised:  The Approach 2. Model Architecture

Original: A reference-free single-document summary quality assessor can be formulated as a regression function f
Revised: A reference-free single-document summary quality assessor can be formulated as a regression function f

Original: ( d, s ) ∈ [0 , 1] of an input document d = [ t 1 , t 2 , · · · ] , and a machine-generated summary s =
Revised: ( d, s ) ∈ [0 , 1] of an input document d = [ t 1 , t 2 , · · · ] , and a machine-generated summary s =

Original: [ t ′ 1 , t ′ 2 , · · · ] , where t i ’s and t ′
Revised: [ t ′ 1 , t ′ 2 , · · · ] , where t i ’s and t ′

Original: i ’s are text tokens.
Revised: i ’s are text tokens.

Original: As a proof of concept, we explore an extremely lean implementation of f : first d and s are jointly transformed into a vector representation e
Revised: As a proof of concept, we explore an extremely lean implementation of f : first d and s are jointly transformed into a vector representation e

Original: = g ( d, s ) , and then it is mapped to a summary quality score via a fully-connected layer, i.e., f ( d, s )
Revised: = g ( d, s ) , and then it is mapped to a summary quality score via a fully-connected layer, i.e., f ( d, s )

Original: =
Revised: =

Original: σ ( W e ) .
Revised: σ ( W e ) .

Original: The function g can be implemented in the
Revised: The function g can be implemented in the

Original: BERT (Devlin et al., 2019) style with an input sequence [[CLS] , t 1 , t 2 , · · · , [SEP] , t ′ 1 , t ′ 2 , · · · , [SEP]] .
Revised: BERT (Devlin et al., 2019) style with an input sequence [[CLS] , t 1 , t 2 , · · · , [SEP] , t ′ 1 , t ′ 2 , · · · , [SEP]] .

Original: The output on the [CLS] token is a joint representation of both the document d and the summary s .
Revised: The output on the [CLS] token is a joint representation of both the document d and the summary s .

Original: While the human evaluation to a summary may cover multiple aspects, such as content/fact coverage and linguistics, a model of us will only yield one number.
Revised: While the human evaluation to a summary may cover multiple aspects, such as content/fact coverage and linguistics, a model of us will only yield one number.

Original: But by using different data mutation strategies, we can get models (different f ’s) adept at different aspects of a summary.
Revised: But by using different data mutation strategies, we can get models (different f ’s) adept at different aspects of a summary.

Original: 2. Data mutation
Revised: 2. Negative Sample Generation

Original: It is impractical to train f with existing summarization datasets, such as CNN/DailyMail (Hermann et al., 2015; Nallapati et al., 2016), because they contain only high-quality, reference-class summaries written manually and thus are all of label 1.
Revised: It is impractical to train f with existing summarization datasets, such as CNN/DailyMail (Hermann et al., 2015; Nallapati et al., 2016), because they contain only high-quality, reference-class summaries written manually and thus are all of label 1.

Original: Some summary evaluation datasets, such as RealSumm (Bhandari et al., 2020), Newsroom (Grusky et al., 2018), and TAC2010 (NIST, 2010), do contain human ratings to system-generated summaries of various qualities.
Revised: Some summary evaluation datasets, such as RealSumm (Bhandari et al., 2020), Newsroom (Grusky et al., 2018), and TAC2010 (NIST, 2010), do contain human ratings to system-generated summaries of various qualities.

Original: But they are too small, containing no more than 100 news articles or article groups each.
Revised: But they are too small, containing no more than 100 news articles or article groups each.

Original: Therefore, training against human ratings or in a supervised approach is impractical.
Revised: Therefore, training against human ratings or in a supervised approach is impractical.

Original: To work around, we propose a weakly supervised solution as depicted in Figure 1(a).
Revised: To work around, we propose a weakly supervised solution as depicted in Figure 1(a).

Original: Existing summarization datasets contain many documentsummary pairs.
Revised: Existing summarization datasets contain many documentsummary pairs.

Original: For each pair ⟨ d, s ⟩ , the reference summary s is mutated into K new summaries of different extents s 1 , s 2 , · · · , s K , which are then paired with the document to form new pairs
Revised: For each pair ⟨ d, s ⟩ , the reference summary s is mutated into K new summaries of different extents s 1 , s 2 , · · · , s K , which are then paired with the document to form new pairs

Original: [Equation]
Revised: [Equation]

Original: which are finally assigned targets to form the training data
Revised: which are finally assigned targets to form the training data

Original: [Equation]
Revised: [Equation]

Original: As illustrated in Figure 2, the training target y k ∈ [1 ..K ] is the percentage of intact content.
Revised: As illustrated in Figure 2, the training target y k ∈ [1 ..K ] is the percentage of intact content.

Original: For example, if 30% of tokens in a mutated summary are not original, then the label is 0.7.
Revised: For example, if 30% of tokens in a mutated summary are not original, then the label is 0.7.

Original: In addition, the original document-summary pair ⟨ d, s ⟩ is also used in training with a target of 1.
Revised: In addition, the original document-summary pair ⟨ d, s ⟩ is also used in training with a target of 1.

Original: Documents
Revised: Documents

Original: Summaries
Revised: Summaries

Original: Generated (mutated or ref.)
Revised: Generated (mutated or ref.)

Original: Scores
Revised: Scores

Original: A supervised model f
Revised: A supervised model f

Original: I npu t(a) Training.
Revised: I npu t(a) Training.

Original: On massive pairs of documents and (reference) summaries without human ratings .
Revised: On massive pairs of documents and (reference) summaries without human ratings .

Original: Targets generated by our heuristics.
Revised: Targets generated by our heuristics.

Original: Dataset: BillSum, arXiv, or BigPatent.
Revised: Dataset: BillSum, arXiv, or BigPatent.

Original: I npu t
Revised: I npu t

Original: T a r ge t
Revised: T a r ge t

Original: Documents Summaries
Revised: Documents Summaries

Original: Human ratings on 1 or N aspects
Revised: Human ratings on 1 or N aspects

Original: Copy
Revised: Copy

Original: I npu t
Revised: I npu t

Original: I npu t(b) Testing.
Revised: I npu t(b) Testing.

Original: On machine-generated summaries and their human ratings.
Revised: On machine-generated summaries and their human ratings.

Original: Dataset: TAC2010, RealSumm, or Newsroom.
Revised: Dataset: TAC2010, RealSumm, or Newsroom.

Original: Documents
Revised: Documents

Original: Summaries (reference)
Revised: Summaries (reference)

Original: The supervised model f
Revised: The supervised model f

Original: R a t i
Revised: R a t i

Original: ng s ( 1D ) on a s pe c t...
Revised: ng s ( 1D ) on a s pe c t...

Original: R a t i
Revised: R a t i

Original: ng s ( 1D ) on a s pe c t
Revised: ng s ( 1D ) on a s pe c t

Original: R a t i
Revised: R a t i

Original: ng s ( 1D ) on a s pe c t
Revised: ng s ( 1D ) on a s pe c t

Original: N P r ed i c t i on s ( 1 - D v e c t o r)
Revised: N P r ed i c t i on s ( 1 - D v e c t o r)

Original: Correlation coefficients vs
Revised: Correlation coefficients vs

Original: S S
Revised: S S

Original: S N ... S ours
Revised: S N ... S ours

Original: Mutation
Revised: Mutation

Original: Mutations can happen at the token or sentence level, where tokens or sentences are randomly selected for mutation.
Revised: Mutations can happen at the token or sentence level, where tokens or sentences are randomly selected for mutation.

Original: A selected token or sentence is mutated in one of the three methods:
Revised: A selected token or sentence is mutated in one of the three methods:

Original: inserting a token/sentence from other summaries behind it, 2.
Revised: inserting a token/sentence from other summaries behind it, 2.

Original: deleting it, or 3.
Revised: deleting it, or 3.

Original: replacing it with a token/sentence from other summaries.
Revised: replacing it with a token/sentence from other summaries.

Original: We do not mix different mutation levels nor mix different mutation methods when preparing the training data.
Revised: We do not mix different mutation levels nor mix different mutation methods when preparing the training data.

Original: Instead, our experiments study one combination of a mutation level and a mutation method, denoted as a mutation strategy, each time.
Revised: Instead, our experiments study one combination of a mutation level and a mutation method, denoted as a mutation strategy, each time.

Original: a document its reference summary the document 15% mutated summary the document 40% mutated summary ⋮ ⋮ 0.
Revised: a document its reference summary the document 15% mutated summary the document 40% mutated summary ⋮ ⋮ 0.

Original: Samples:
Revised: Samples:

Original: Labels:
Revised: Labels:

Original:  Experiments 3. Test data
Revised:  Experiments 3. Test data

Original: The ground truth of a summary’s quality is human ratings to it.
Revised: The ground truth of a summary’s quality is human ratings to it.

Original: A model trained (Fig. 1(a)) is tested (Fig. 1(b)) against human ratings.
Revised: A model trained (Fig. 1(a)) is tested (Fig. 1(b)) against human ratings.

Original: Three test datasets are chosen below.
Revised: Three test datasets are chosen below.

Original: Due to the limited number and sizes of human evaluation datasets, they are all in the news domain.
Revised: Due to the limited number and sizes of human evaluation datasets, they are all in the news domain.

Original: The human evaluation protocols can be found in their respective references.
Revised: The human evaluation protocols can be found in their respective references.

Original: TAC2010 (NIST, 2010) is a multi-document (ten-document) summarization task reporting both factual and linguistic aspects.
Revised: TAC2010 (NIST, 2010) is a multi-document (ten-document) summarization task reporting both factual and linguistic aspects.

Original: We use
Revised: We use

Original: [Equation]
Revised: [Equation]

Original: i ∈ [1 .. 10] f
Revised: i ∈ [1 .. 10] f

Original: ( d i , s ) to approximate the score of the summary s composed from ten documents d 1 to d 10 .
Revised: ( d i , s ) to approximate the score of the summary s composed from ten documents d 1 to d 10 .

Original: We only use Set A of TAC2010 because Set B is not for regular summarization.
Revised: We only use Set A of TAC2010 because Set B is not for regular summarization.

Original: Newsroom (Grusky et al., 2018) also covers both factual (in INFormativeness and RELevance) and linguistic (in COHerence and FLUency) aspects.
Revised: Newsroom (Grusky et al., 2018) also covers both factual (in INFormativeness and RELevance) and linguistic (in COHerence and FLUency) aspects.

Original: For human ratings, three human annotators rate one pair of a document and machine-generated summary.
Revised: For human ratings, three human annotators rate one pair of a document and machine-generated summary.

Original: The mean of their ratings on each aspect is used in our experiments.
Revised: The mean of their ratings on each aspect is used in our experiments.

Original: RealSumm (Bhandari et al., 2020) focuses on only factual coverage.
Revised: RealSumm (Bhandari et al., 2020) focuses on only factual coverage.

Original: It covers 14 abstractive and 11 extractive summarizers published after 2018 and conducts human evaluation on the two groups separately.
Revised: It covers 14 abstractive and 11 extractive summarizers published after 2018 and conducts human evaluation on the two groups separately.

Original: Note that we do not and cannot train a model against the labels in a test set, as mentioned in § 2.1.
Revised: Note that we do not and cannot train a model against the labels in a test set, as mentioned in § 2.1.

Original: If a test set rates on multiple aspects, we do not train one model for each aspect.
Revised: If a test set rates on multiple aspects, we do not train one model for each aspect.

Original: Nor do we train models for individual or a collection of test sets.
Revised: Nor do we train models for individual or a collection of test sets.

Original: We compute correlation between the predictions from our model and human ratings on each aspect of each test set.
Revised: We compute correlation between the predictions from our model and human ratings on each aspect of each test set.

Original: 3. Training data
Revised: 3. Training data

Original: Three widely used summarization datasets from three different domains are chosen for training: Billsum (Kornilova and Eidelman, 2019),
Revised: Three widely used summarization datasets from three different domains are chosen for training: Billsum (Kornilova and Eidelman, 2019),

Original: Scientific-Papers/arXiv (Cohan et al., 2018), and
Revised: Scientific-Papers/arXiv (Cohan et al., 2018), and

Original: Big-Patent (Sharma et al., 2019).
Revised: Big-Patent (Sharma et al., 2019).

Original: Datasets from the news domain are avoided on purpose because the test data is in the news domain.
Revised: Datasets from the news domain are avoided on purpose because the test data is in the news domain.

Original: This crossdomain setting allows us to examine whether a model is prone to domain differences.
Revised: This crossdomain setting allows us to examine whether a model is prone to domain differences.

Original: For each reference summary, K = 5 mutated summaries are generated.
Revised: For each reference summary, K = 5 mutated summaries are generated.

Original: The percentage of intact content is measured by the number of tokens and the number of characters for token-level and sentence-level mutations, respectively.
Revised: The percentage of intact content is measured by the number of tokens and the number of characters for token-level and sentence-level mutations, respectively.

Original: 3. Baselines and upper bounds
Revised: 3. Baselines and upper bounds

Original: To fairly compare, four recent metrics:
Revised: To fairly compare, four recent metrics:

Original: BLANC (Vasilyev et al., 2020),
Revised: BLANC (Vasilyev et al., 2020),

Original: SummaQA
Revised: SummaQA

Original: (Scialom et al., 2019), SUPERT (Gao et al., 2020) and LS-Score (Wu et al., 2020) , are used as baselines because like our approach, they do not need a reference summary to judge a machine-generated summary, i.e., reference-free.
Revised: (Scialom et al., 2019), SUPERT (Gao et al., 2020) and LS-Score (Wu et al., 2020) , are used as baselines because like our approach, they do not need a reference summary to judge a machine-generated summary, i.e., reference-free.

Original: Human crafted reference summaries give reference-based metrics advantages.
Revised: Human crafted reference summaries give reference-based metrics advantages.

Original: The results of reference-based metrics are included as soft upper bounds: ROUGE-1, ROUGE-2 and ROUGEL (Lin, 2004),
Revised: The results of reference-based metrics are included as soft upper bounds: ROUGE-1, ROUGE-2 and ROUGEL (Lin, 2004),

Original: MoverScore (Zhao et al., 2019),
Revised: MoverScore (Zhao et al., 2019),

Original: BertScore (Zhang* et al., 2020), BLEU (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005), and S 3 (Peyrard et al., 2017).
Revised: BertScore (Zhang* et al., 2020), BLEU (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005), and S 3 (Peyrard et al., 2017).

Original: 3. Settings
Revised: 3. Settings

Original: Because the baselines use BERT, we use BERT as well for a fair comparison.
Revised: Because the baselines use BERT, we use BERT as well for a fair comparison.

Original: Specifically, BERTbase uncased (L=12, H=768) is fine-tuned, with a learning rate of 1e-5, three epochs, and a batch size of 14.
Revised: Specifically, BERTbase uncased (L=12, H=768) is fine-tuned, with a learning rate of 1e-5, three epochs, and a batch size of 14.

Original: The input sequence is limited to 512 tokens using the round robin trimmer.
Revised: The input sequence is limited to 512 tokens using the round robin trimmer.

Original: The training loss is
Revised: The training loss is

Original: MSE as this problem is regression.
Revised: MSE as this problem is regression.

Original: 3. Results
Revised: 3. Results

Original: We use the summary-level (Peyrard et al., 2017) meta-evaluation strategy to report an approach’s average correlation with human ratings.
Revised: We use the summary-level (Peyrard et al., 2017) meta-evaluation strategy to report an approach’s average correlation with human ratings.

Original: Summary evaluation usually covers two types of aspects, contents/facts and linguistics.
Revised: Summary evaluation usually covers two types of aspects, contents/facts and linguistics.

Original: They are reported separately in Tables 1 and 2.
Revised: They are reported separately in Tables 1 and 2.

Original: Due to space limit, only the best mutation strategy is reported for each aspect group.
Revised: Due to space limit, only the best mutation strategy is reported for each aspect group.

Original: On content/fact aspects , the best mutation strategy is sentence deletion and our best models outperform baselines on all test datasets.
Revised: On content/fact aspects , the best mutation strategy is sentence deletion and our best models outperform baselines on all test datasets.

Original: Our approach makes the most improvement over baselines on RealSumm, a dataset much bigger than Newsroom and more modern than TAC2010, and the least improvement on TAC2010, the oldest dataset.
Revised: Our approach makes the most improvement over baselines on RealSumm, a dataset much bigger than Newsroom and more modern than TAC2010, and the least improvement on TAC2010, the oldest dataset.

Original: [Table]
Revised: [Table]

Original: On linguistic aspects
Revised: On linguistic aspects

Original: [Table]
Revised: [Table]

Original: the best mutation strategy is word deletion.
Revised: the best mutation strategy is word deletion.

Original: Here, even our worst model cannot be outperformed by any baseline nor upper bound.
Revised: Here, even our worst model cannot be outperformed by any baseline nor upper bound.

Original: As mentioned earlier, canonical metrics are lexical-based while modern reference-based and reference-free approaches focus on facts.
Revised: As mentioned earlier, canonical metrics are lexical-based while modern reference-based and reference-free approaches focus on facts.

Original: Through mutating reference summaries, our approach can create summaries of different linguistic qualities.
Revised: Through mutating reference summaries, our approach can create summaries of different linguistic qualities.

Original: Although our approach makes big improvements over baselines on TAC2010 and Newsroom’s FLUency
Revised: Although our approach makes big improvements over baselines on TAC2010 and Newsroom’s FLUency

Original: [Table]
Revised: [Table]

Original: its edge is smaller on Newsroom’s COHerence.
Revised: its edge is smaller on Newsroom’s COHerence.

Original: A sentence-level scrambling mutation may improve our approach’s performance on COHerence in the future
Revised: A sentence-level scrambling mutation may improve our approach’s performance on COHerence in the future

Original: [Table]
Revised: [Table]

Original: reportedthesameissueat https://github.com/whl97/LS-Score/issues .
Revised: reportedthesameissueat https://github.com/whl97/LS-Score/issues .

Original: Itis furtherexcludedfromtherankingbecauseitistrainedonthesamedomainasthetestdomainwhereasallotherbaselinesandourmodelsarenot.
Revised: Itis furtherexcludedfromtherankingbecauseitistrainedonthesamedomainasthetestdomainwhereasallotherbaselinesandourmodelsarenot.

Original: [Table]
Revised: [Table]

Original: 3. Discussions
Revised: 3. Discussions

Original: What is the best mutation?
Revised: What is the best mutation?

Original: Across datasets, deletion-based mutations are most effective.
Revised: Across datasets, deletion-based mutations are most effective.

Original: The two kinds of deletions happen to be complementarily effective for two aspect groups: sentence deletion for content/fact aspects vs. word deletion for linguistic aspects.
Revised: The two kinds of deletions happen to be complementarily effective for two aspect groups: sentence deletion for content/fact aspects vs. word deletion for linguistic aspects.

Original: This is an advantage of our approach that under a uniformed framework, different summary quality aspects can be gauged by designing different mutation options .
Revised: This is an advantage of our approach that under a uniformed framework, different summary quality aspects can be gauged by designing different mutation options .

Original: The complementariness of sentence deletion and word deletion can be well explained as that removing a sentence from a reference summary reduces a great amount of key information while removing a word from a sentence changes it syntactically.
Revised: The complementariness of sentence deletion and word deletion can be well explained as that removing a sentence from a reference summary reduces a great amount of key information while removing a word from a sentence changes it syntactically.

Original: We found that word-level mutations are less useful for content/fact aspects, probably because of the inertia of the context after words are altered.
Revised: We found that word-level mutations are less useful for content/fact aspects, probably because of the inertia of the context after words are altered.

Original: Which training domain/dataset should I use?
Revised: Which training domain/dataset should I use?

Original: Due to the composition of summarizers and the limited data size in human evaluation, it is very hard to get a consistent ranking of metrics on different datasets (Bhandari et al., 2020).
Revised: Due to the composition of summarizers and the limited data size in human evaluation, it is very hard to get a consistent ranking of metrics on different datasets (Bhandari et al., 2020).

Original: For example, in
Revised: For example, in

Original: Table 1, Billsumm outperforms all baselines and its peers on TAC2010 but not the case on Newsroom and RealSumm.
Revised: Table 1, Billsumm outperforms all baselines and its peers on TAC2010 but not the case on Newsroom and RealSumm.

Original: Still, the impact of training domain seems manageable.
Revised: Still, the impact of training domain seems manageable.

Original: The average absolute deviations across the training datasets/domains are given at the bottom of Tables 1 and 2.
Revised: The average absolute deviations across the training datasets/domains are given at the bottom of Tables 1 and 2.

Original: They mostly below 3.5%.
Revised: They mostly below 3.5%.

Original: A qualitative analysis shows that the variation seems more due to the characteristics of the text than the domain.
Revised: A qualitative analysis shows that the variation seems more due to the characteristics of the text than the domain.

Original: Legislative bills (Billsum) have lots of short, hierarchical clauses and thus differ from common English greatly.
Revised: Legislative bills (Billsum) have lots of short, hierarchical clauses and thus differ from common English greatly.

Original: Scientific papers have many equations and cross-references.
Revised: Scientific papers have many equations and cross-references.

Original: There are also many occurrences of L A TEX or MathML in the dataset arXiv.
Revised: There are also many occurrences of L A TEX or MathML in the dataset arXiv.

Original: On top of that, all our experiments use different training and test domains.
Revised: On top of that, all our experiments use different training and test domains.

Original: Hence we would say that the impact of domain variation is very small.
Revised: Hence we would say that the impact of domain variation is very small.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we propose a weakly supervised approach to summary quality evaluation.
Revised: In this paper, we propose a weakly supervised approach to summary quality evaluation.

Original: A few mutation methods are introduced to make use of the massive, precious human written summaries in summarization datasets.
Revised: A few mutation methods are introduced to make use of the massive, precious human written summaries in summarization datasets.

Original: In cross-domain experiments, our approach achieves better performance than baselines, especially on linguistic aspects.
Revised: In cross-domain experiments, our approach achieves better performance than baselines, especially on linguistic aspects.

Original: We hope this proof-of-concept study can inspire more reference-free summary evaluation.
Revised: We hope this proof-of-concept study can inspire more reference-free summary evaluation.

Original: Acknowledgments
Revised: Acknowledgments

Original: Bao, Luo, Li, and He’s work in this paper is partially supported by National Science Foundation (NSF) grants No.
Revised: Bao, Luo, Li, and He’s work in this paper is partially supported by National Science Foundation (NSF) grants No.

Original: MCB-1821828 and No. CNS1817089.
Revised: MCB-1821828 and No. CNS1817089.

Original: The authors would also like to thank reviewers who have given precious feedback on improving this work.
Revised: The authors would also like to thank reviewers who have given precious feedback on improving this work.

Original: A Dataset statistics
Revised: A Dataset statistics

Original: For test sets:
Revised: For test sets:

Original: • TAC2010 Guided Summarization
Revised: • TAC2010 Guided Summarization

Original: Task Set A consists of 46 topics, each of which is associated with a set of 10 documents.
Revised: Task Set A consists of 46 topics, each of which is associated with a set of 10 documents.

Original: We evaluate the metrics over summaries generated by 43 systems.
Revised: We evaluate the metrics over summaries generated by 43 systems.

Original: • Newsroom contains human-rated summaries generated by 7 systems for 60 documents.
Revised: • Newsroom contains human-rated summaries generated by 7 systems for 60 documents.

Original: • RealSumm sampled 100 documents from the CNN/DailyMail test set, and collected human ratings for summaries generated by 11 extrative systems and 14 abstractive systems.
Revised: • RealSumm sampled 100 documents from the CNN/DailyMail test set, and collected human ratings for summaries generated by 11 extrative systems and 14 abstractive systems.

Original: For training sets, the numbers of pairs of documents and reference summaries in the train split are:
Revised: For training sets, the numbers of pairs of documents and reference summaries in the train split are:

Original: • Billsum : 18, • Scientific papers/arXiv : 203, • Big-Patent : 1,207,
Revised: • Billsum : 18, • Scientific papers/arXiv : 203, • Big-Patent : 1,207,

Original: For each dataset, we use the entire (except for BigPatent, 10% due to its huge size) train split in
Revised: For each dataset, we use the entire (except for BigPatent, 10% due to its huge size) train split in

Original: Google Tensorflow Datasets for training.
Revised: Google Tensorflow Datasets for training.

Original: B Computational environment and cost
Revised: B Computational environment and cost

Original: All experiments were carried out on one RTX
Revised: All experiments were carried out on one RTX

Original: GPU installed on a desktop computer.
Revised: GPU installed on a desktop computer.

Original: The training takes about a week for all three training datasets.
Revised: The training takes about a week for all three training datasets.

Original: C Another type of mutation
Revised: C Another type of mutation

Original: In addition to the three mutation methods mentioned already, we have another method called crosspairing.
Revised: In addition to the three mutation methods mentioned already, we have another method called crosspairing.

Original: [Figure]
Revised: [Figure]

Original: Illustrated in Figure 3, it is inspired by the nextsentence prediction (NSP) task in original BERT training.
Revised: Illustrated in Figure 3, it is inspired by the nextsentence prediction (NSP) task in original BERT training.

Original: Given a document and its reference summary, we create negative data by pairing the document with reference summaries of other documents.
Revised: Given a document and its reference summary, we create negative data by pairing the document with reference summaries of other documents.

Original: We assign the label 0 to a mismatching documentsummary pair, and the label 1 to any original pair of a document and its reference summary.
Revised: We assign the label 0 to a mismatching documentsummary pair, and the label 1 to any original pair of a document and its reference summary.

Original: D Complete empirical results
Revised: D Complete empirical results

Original: Due to space limit, we were only able to present the result of the best mutation method in § 3.5.
Revised: Due to space limit, we were only able to present the result of the best mutation method in § 3.5.

Original: Full results are given in Tables 3, 4,
Revised: Full results are given in Tables 3, 4,

Original: [Table]
Revised: [Table]

Original: and 6.
Revised: and 6.

Original: Pearson’s for LS-Score is unable to be produced due to reasons explained in the footnote on page
Revised: Pearson’s for LS-Score is unable to be produced due to reasons explained in the footnote on page

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]