Original: SMART: S ENTENCES AS B ASIC U NITS FOR T EXT
Revised: SMART: S ENTENCES AS B ASIC U NITS FOR T EXT

Original: E VALUATION
Revised: E VALUATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Widely used evaluation metrics for text generation either do not work well with longer texts or fail to evaluate all aspects of text quality.
Revised: Widely used evaluation metrics for text generation either do not work well with longer texts or fail to evaluate all aspects of text quality.

Original: In this paper, we introduce a new metric called SMART to mitigate such limitations.
Revised: In this paper, we introduce a new metric called SMART to mitigate such limitations.

Original: Speciﬁcally, we treat sentences as basic units of matching instead of tokens, and use a sentence matching function to soft -match candidate and reference sentences.
Revised: Speciﬁcally, we treat sentences as basic units of matching instead of tokens, and use a sentence matching function to soft -match candidate and reference sentences.

Original: Candidate sentences are also compared to sentences in the source documents to allow grounding (e.g., factuality) evaluation.
Revised: Candidate sentences are also compared to sentences in the source documents to allow grounding (e.g., factuality) evaluation.

Original: Our results show that system-level correlations of our proposed metric with a model-based matching function outperforms all competing metrics on the SummEval summarization meta-evaluation dataset, while the same metric with a string-based matching function is competitive with current modelbased metrics.
Revised: Our results show that system-level correlations of our proposed metric with a model-based matching function outperforms all competing metrics on the SummEval summarization meta-evaluation dataset, while the same metric with a string-based matching function is competitive with current modelbased metrics.

Original: The latter does not use any neural model, which is useful during model development phases where resources can be limited and fast evaluation is required.
Revised: The latter does not use any neural model, which is useful during model development phases where resources can be limited and fast evaluation is required.

Original: SMART also outperforms all factuality evaluation metrics on the
Revised: SMART also outperforms all factuality evaluation metrics on the

Original: TRUE benchmark.
Revised: TRUE benchmark.

Original: Finally, we also conducted extensive analyses showing that our proposed metrics work well with longer summaries and are less biased towards speciﬁc models.
Revised: Finally, we also conducted extensive analyses showing that our proposed metrics work well with longer summaries and are less biased towards speciﬁc models.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original:  One major obstacle in the progress of text generation tasks (e.g., document summarization, longform question answering, data-to-text generation, etc.) is automatic evaluation.
Revised: [Figure] More details in One major obstacle in the progress of text generation tasks (e.g., document summarization, longform question answering, data-to-text generation, etc.) is automatic evaluation.

Original: Traditionally, automatic metrics that rely on discrete token-level matching such as ROUGE (Lin, 2004) and BLEU (Papineni et al., 2002) have been utilized to check whether system outputs are of high quality across four dimensions (Kryscinski et al., 2019; Fabbri et al., 2021): coherence, factuality, ﬂuency, and informativeness.
Revised: Traditionally, automatic metrics that rely on discrete tokenlevel matching such as ROUGE (Lin, 2004) and BLEU (Papineni et al., 2002) have been utilized to check whether system outputs are of high quality across four dimensions (Kryscinski et al., 2019; Fabbri et al., 2021): coherence, factuality, ﬂuency, and informativeness.

Original: These metrics do not correlate well with human judgments on all four dimensions of text quality (Fabbri et al., 2021).
Revised: These metrics do not correlate well with human judgments on all four dimensions of text quality (Fabbri et al., 2021).

Original: Because of this, the evaluation is usually coupled with human elicitation studies that ask humans to rate texts.
Revised: Because of this, the evaluation is usually coupled with human elicitation studies that ask humans to rate texts.

Original: These studies can be expensive and nearly impossible to reproduce.
Revised: These studies can be expensive and nearly impossible to reproduce.

Original: More recently, pretrained language models are leveraged for automatically evaluating systemgenerated texts (Zhang* et al., 2020;
Revised: More recently, pretrained language models are leveraged for automatically evaluating systemgenerated texts (Zhang* et al., 2020;

Original: Sellam et al., 2020; Yuan et al., 2021), which have shown improvements on correlation with human judgments.
Revised: Sellam et al., 2020; Yuan et al., 2021), which have shown improvements on correlation with human judgments.

Original: Nevertheless, both ROUGE and LM-based metrics have three major drawbacks.
Revised: Nevertheless, both ROUGE and LMbased metrics have three major drawbacks.

Original: Firstly, these metrics are not good at evaluating long and multi-sentence texts.
Revised: Firstly, these metrics are not good at evaluating long and multi-sentence texts.

Original: Figure 1 illustrates system-level rank correlations of ROUGE in different text lengths, which shows that after a certain length, ROUGE drastically decreases its evaluative power.
Revised: Figure 1 illustrates system-level rank correlations of ROUGE in different text lengths, which shows that after a certain length, ROUGE drastically decreases its evaluative power.

Original: By design, ROUGE is also not robust to evaluating possibly shufﬂed information in long outputs, hurting its performance on evaluating coherence.
Revised: By design, ROUGE is also not robust to evaluating possibly shufﬂed information in long outputs, hurting its performance on evaluating coherence.

Original: On the other hand, LM-based metrics such as the state-of-the-art BARTScore (Yuan et al., 2021), are constrained to the length limitation of the pretrained LM used, thus they are not able to evaluate outputs longer than this limit.
Revised: On the other hand, LM-based metrics such as the state-of-the-art BARTScore (Yuan et al., 2021), are constrained to the length limitation of the pretrained LM used, thus they are not able to evaluate outputs longer than this limit.

Original: Secondly, most of these metrics only use reference texts during evaluation.
Revised: And secondly, most of these metrics only use reference texts during evaluation.

Original: This restricts the capability of the metrics from evaluating dimensions of text quality that requires grounding to the source.
Revised: This restricts the capability of the metrics from evaluating dimensions of text quality that requires grounding to the source.

Original: Yuan et al.
Revised: Yuan et al.

Original: (2021) suggested to use the source document during evaluation, however their evaluation is still limited to short documents because of length limitations in LMs.
Revised: (2021) suggested to use the source document during evaluation, however their evaluation is still limited to short documents because of length limitations in LMs.

Original: Finally, LM-based metrics can be very slow, which hinders usage in the model development stages, where quick evaluation is necessary. As such, most practitioners use ROUGE in these stages, which can lead to suboptimal modeling. < < < <65 Number of Tokens in Reference 0.
Revised: In this paper, we propose an automatic metric called SMART ( S entence MA tching for R ating T ext).

Original: S y s t e m - L e v e l C o rr e l a t i o n SMART 
Revised: SMART is motivated by the pyramid method of human evaluation for summarization (Nenkova et al., 2007), where they transform text into semantic content units (SCUs), or sentences that contain a single fact.

Original: ROUGE BartScore In this paper, we propose an automatic metric called SMART ( S entence MA tching for R ating
Revised: Since this kind of transformation cannot be done automatically, we use sentences as a proxy to SCUs, and treat them as basic units of matching instead of tokens.

Original: T ext). To support long and multi-sentence texts, we treat sentences as basic units of matching instead of tokens.
Revised: This additionally allows the metric to effectively support long and multi-sentence texts.

Original: Since sentences most likely do not have exact matches, we use a soft-matching function that returns a matching score between 0 and 1, given a pair of sentences.
Revised: Since sentences most likely do not have exact matches, we use a soft-matching function that returns a matching score between 0 and 1, given a pair of sentences.

Original: Moreover, to allow grounded evaluation, we also include the source in the calculation of the metric.
Revised: Moreover, to allow grounded evaluation, we also include the source in the calculation of the metric.

Original: Similar to ROUGE, we introduce multiple SMART versions using sentence n-gram overlap and longest common subsequence.
Revised: Similar to ROUGE, we introduce multiple SMART versions using sentence n-gram overlap and longest common subsequence.

Original: We show that SMART with BLEURT (Sellam et al., 2020) as a soft-matching function outperforms all the competing approaches on all four dimensions of quality in the SummEval dataset (Fabbri et al., 2019).
Revised: Our experiments show that SMART with BLEURT (Sellam et al., 2020) as a soft-matching function outperforms all the competing approaches on all four dimensions of quality in the SummEval dataset (Fabbri et al., 2019).

Original: We also show that SMART with T5-ANLI (Honovich et al., 2022) outperforms all competing factuality-based evaluation metrics on the TRUE benchmark (Honovich et al., 2022).
Revised: We also show that SMART with T5-ANLI (Honovich et al., 2022) outperforms all competing factuality-based evaluation metrics on the TRUE benchmark (Honovich et al., 2022).

Original: Moreover, a faster variant of SMART, which does not use any neural model for text matching, shows competitive correlations with human judgments.
Revised: Moreover, a faster variant of SMART, which does not use any neural model for text matching, shows competitive correlations with human judgments.

Original: Finally, our extensive analyses show that SMART works better with longer summaries and is less biased towards speciﬁc models.
Revised: Finally, our extensive analyses show that SMART works better with longer summaries and is less biased towards speciﬁc models.

Original: We will release and maintain a toolkit containing scripts to easily run evaluation using SMART.
Revised: We will release and maintain a toolkit containing scripts to easily run evaluation using SMART.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Evaluation in conditional generation tasks such as machine translation and document summarization is a long-standing problem.
Revised: Evaluation in conditional generation tasks such as machine translation and document summarization is a long-standing problem.

Original: Traditionally, evaluation involves human elicitation studies that score texts based on different metrics of quality, such as adequacy, ﬁdelity, and ﬂuency in machine translation (Hovy, 1999), and coherence, conciseness, ﬂuency, readability, and content relevance in summarization (Mani, 2001).
Revised: Traditionally, evaluation involves human elicitation studies that score texts based on different metrics of quality, such as adequacy, ﬁdelity, and ﬂuency in machine translation (Hovy, 1999), and coherence, conciseness, ﬂuency, readability, and content relevance in summarization (Mani, 2001; Nenkova et al., 2007).

Original: Automatic metrics based on token n-gram matching have been developed to replace these expensive and time-consuming studies, in whichROUGE (Lin, 2004) and
Revised: Automatic metrics based on token n-gram matching have been developed to replace these expensive and time-consuming studies, in which

Original:  BLEU (Papineni et al., 2002) are most widely used in summarization and translation, respectively.
Revised: ROUGE (Lin, 2004) and BLEU (Papineni et al., 2002) are most widely used in summarization and translation, respectively.

Original: Several extensions to token n-gram matching have been proposed, such as using paraphrases, synonyms (Lavie & Agarwal, 2007), and word embeddings (Ng & Abrecht, 2015) to handle cases that are semantically equivalent, and downweighting common n-grams to focus more on salient ones (Vedantam et al., 2015).
Revised: Several extensions to token n-gram matching have been proposed, such as using paraphrases, synonyms (Lavie & Agarwal, 2007), and word embeddings (Ng & Abrecht, 2015) to handle cases that are semantically equivalent, and downweighting common n-grams to focus more on salient ones (Vedantam et al., 2015).

Original: Popovi´c (2015) instead use character-level n-gram matching to also match words that are conjugated differently and support morphologically rich languages.
Revised: Popovi´c (2015) instead use character-level n-gram matching to also match words that are conjugated differently and support morphologically rich languages.

Original: With the introduction and success of pretrained language models such as BERT (Devlin et al., 2019) and BART (Lewis et al., 2020), evaluation metrics that leverage them have been proposed.
Revised: With the introduction and success of pretrained language models such as BERT (Devlin et al., 2019) and BART (Lewis et al., 2020), evaluation metrics that leverage them have been proposed.

Original: BERTScore (Zhang* et al., 2020) leverages contextualized token embeddings from BERT and obtains pairwise matching of tokens from reference and system summaries.
Revised: BERTScore (Zhang* et al., 2020) leverages contextualized token embeddings from BERT and obtains pairwise matching of tokens from reference and system summaries.

Original: MoverScore
Revised: MoverScore

Original: (Zhao et al., 2019) extends BERTScore by instead having many-to-one soft alignments using Word Mover’s Distance (WMD; Kusner et al., 2015).
Revised: (Zhao et al., 2019) extends BERTScore by instead having many-to-one soft alignments using Word Mover’s Distance (WMD; Kusner et al., 2015).

Original: BLEURT (Sellam et al., 2020) ﬁne-tunes BERT to predict human scores with large-scale synthetic training data.
Revised: BLEURT (Sellam et al., 2020) ﬁne-tunes BERT to predict human scores with large-scale synthetic training data.

Original: BARTScore (Yuan et al., 2021) uses BART and treats evaluation as a text generation problem, using likelihood of predicting the system summary given the source document or the reference summary.
Revised: BARTScore (Yuan et al., 2021) uses BART and treats evaluation as a text generation problem, using likelihood of predicting the system summary given the source document or the reference summary.

Original: Clark et al.
Revised: Clark et al.

Original: and Zhao et al.
Revised: and Zhao et al.

Original: (2019) also explored sentence-level matching with WMD using (contextualized) sentence embeddings, however they show no concrete improvements over other model-based metrics (Fabbri et al., 2021).
Revised: (2019) also explored sentence-level matching with WMD using (contextualized) sentence embeddings, however they show no concrete improvements over other model-based metrics (Fabbri et al., 2021).

Original: In contrast, we show that our metric which uses sentence-level matching correlates better with human judgments than all competing models.
Revised: In contrast, we show that our metric which uses sentence-level matching correlates better with human judgments than all competing models.

Original: Factuality in summarization (Falke et al., 2019; Maynez et al., 2020) is usually evaluated separately since most automatic metrics are focused on informativeness and do not include the source document in the metric calculation.
Revised: Factuality in summarization (Falke et al., 2019; Maynez et al., 2020) is usually evaluated separately since most automatic metrics are focused on informativeness and do not include the source document in the metric calculation.

Original: Factuality-speciﬁc metrics can be divided into three approaches: natural language inference (NLI) based approaches (Falke et al., 2019;
Revised: Factuality-speciﬁc metrics can be divided into three approaches: natural language inference (NLI) based approaches (Falke et al., 2019;

Original: Maynez et al., 2020; Laban et al., 2022), where a summary is considered factual if all its facts are entailed by the source document, model-based approaches (Kryscinski et al., 2020;
Revised: Maynez et al., 2020; Laban et al., 2022), where a summary is considered factual if all its facts are entailed by the source document, model-based approaches (Kryscinski et al., 2020;

Original: Deng et al., 2021), where a model is trained to detect factual errors in the summary, and question answering (QA) based approaches (Durmus et al., 2020; Wang et al., 2020; Honovich et al., 2021), where questions generated in a factual summary should be answerable using the source.
Revised: Deng et al., 2021), where a model is trained to detect factual errors in the summary, and question answering (QA) based approaches (Durmus et al., 2020; Wang et al., 2020; Honovich et al., 2021), where questions generated in a factual summary should be answerable using the source.

Original: While we also compare correlations of automatic metrics with human judgments on factuality, the goal of our work is to ﬁnd holistic metrics for evaluation that can also effectively evaluate other dimensions of text quality.
Revised: While we also compare correlations of automatic metrics with human judgments on factuality, the goal of our work is to ﬁnd holistic metrics for evaluation that can also effectively evaluate other dimensions of text quality.

Original: Our results show that other dimensions such as coherence, ﬂuency, and informativeness also beneﬁt in the use of the source documents in the metric.
Revised: Our results show that other dimensions such as coherence, ﬂuency, and informativeness also beneﬁt in the use of the source documents in the metric.

Original: Finally, we also show that our metric improves the state-of-the-art of factual consistency evaluation based on the TRUE benchmark Honovich et al.
Revised: Finally, we also show that our metric improves the state-of-the-art of factual consistency evaluation based on the TRUE benchmark Honovich et al.

Original:  P ROBLEM D EFINITION
Revised:  P ROBLEM D EFINITION

Original: We primarily use document summarization – the task of generating concise and accurate summaries of input documents (Mani, 2001) – to explain our metric, but the proposed metric can be easily adapted to other tasks, as shown in Section 5.2.
Revised: We primarily use document summarization – the task of generating concise and accurate summaries of input documents (Mani, 2001) – to explain our metric, but the proposed metric can be easily adapted to other tasks, as shown in Section 5.2.

Original: Let S be a list of source documents, C be a list of summaries generated for S by a candidate system, and R be a list of reference summaries produced by human annotators for S .
Revised: Let S be a list of source documents, C be a list of summaries generated for S by a candidate system, and R be a list of reference summaries produced by human annotators for S .

Original: Note that R can be a nested list, i.e., for each example, there can be multiple references.
Revised: Note that R can be a nested list, i.e., for each example, there can be multiple references.

Original: Moreover, let Q be a list of dimensions of summary quality, and let H q be a list of human-annotated scores for C in terms of a certain summary quality q .
Revised: Moreover, let Q be a list of dimensions of summary quality, and let H q be a list of human-annotated scores for C in terms of a certain summary quality q .

Original: For each summary quality q ∈ Q , the problem is to devise an evaluation metric f q ( S , R , C ) that outputs a list of scores that correlates well with H q .
Revised: For each summary quality q ∈ Q , the problem is to devise an evaluation metric f q ( S , R , C ) that outputs a list of scores that correlates well with H q .

Original: Note that, unlike most of previous work on summarization evaluation (Lin, 2004;
Revised: Note that, unlike most of previous work on summarization evaluation (Lin, 2004;

Original: Clark et al., 2019; Bhandari et al., 2020;
Revised: Clark et al., 2019; Bhandari et al., 2020;

Original: Fabbri et al., 2021), we also take into account source documents S when calculating metric f q ( · ) .
Revised: Fabbri et al., 2021), we also take into account source documents S when calculating metric f q ( · ) .

Original: This ensures that the metric can evaluate quality dimensions that require looking at the source.
Revised: This ensures that the metric can evaluate quality dimensions that require looking at the source.

Original: We deﬁne the list of summary quality Q as the following four dimensions of summary quality, deﬁned as follows (based on deﬁnitions in Fabbri et al., 2021 and Yuan et al., 2021):
Revised: We deﬁne the list of summary quality Q as the following four dimensions of summary quality, deﬁned as follows (based on deﬁnitions in Fabbri et al., 2021 and Yuan et al., 2021):

Original: • Coherence : The summary should be well-structured and well-organized.
Revised: • Coherence : The summary should be well-structured and well-organized.

Original: The summary should not just be a heap of related information, but should build from sentence to sentence to a coherent body of information about a topic.
Revised: The summary should not just be a heap of related information, but should build from sentence to sentence to a coherent body of information about a topic.

Original: • Factuality : The summary should only contain statements that are entailed by the source document.
Revised: • Factuality : The summary should only contain statements that are entailed by the source document.

Original: The summary should not contain hallucinated facts that either do not exist in the source or contradict facts from the source.
Revised: The summary should not contain hallucinated facts that either do not exist in the source or contradict facts from the source.

Original: • Fluency :
Revised: • Fluency :

Original: The summary should have no formatting problems, capitalization errors or obviously ungrammatical sentences (e.g., fragments, missing components) that make the text difﬁcult to read.
Revised: The summary should have no formatting problems, capitalization errors or obviously ungrammatical sentences (e g., fragments, missing components) that make the text difﬁcult to read.

Original: • Informativeness : The summary should include only important information from the source document.
Revised: • Informativeness : The summary should include only important information from the source document.

Original: The summary should not include redundant information and information that are considered excessive and non-salient.
Revised: The summary should not include redundant information and information that are considered excessive and non-salient.

Original:  S ENTENCE M ATCHING FOR R ATING T EXT
Revised:  S ENTENCE M ATCHING FOR R ATING T EXT

Original: We now describe our proposed metric, SMART ( S entence MA tching for R ating T ext), which has two key ideas.
Revised: We now describe our proposed metric, SMART ( S entence MA tching for R ating T ext), which has two key ideas.

Original: Firstly, we treat sentences as basic units of matching between system and reference summaries, instead of tokens.
Revised: Firstly, we treat sentences as basic units of matching between system and reference summaries, instead of tokens.

Original: At sentence-level, exactly matching sentences are most likely nonexistent (in datasets with abstractive reference summaries), thus we instead utilize soft-matching functions to compare sentences.
Revised: At sentence-level, exactly matching sentences are most likely nonexistent (in datasets with abstractive reference summaries), thus we instead utilize soft-matching functions to compare sentences.

Original: Similar to ROUGE, we present two types of SMART: n-gram overlap (SMART-N) and longest common subsequence (SMART-L).
Revised: Similar to ROUGE, we present two types of SMART: n-gram overlap (SMART-N) and longest common subsequence (SMART-L).

Original: Secondly, SMART allows to compare the candidate system summary with both the reference summary and the source document.
Revised: Secondly, SMART allows to compare the candidate system summary with both the reference summary and the source document.

Original: This is particularly important when evaluating dimensions of summary quality that rely on the source document such as factuality.
Revised: This is particularly important when evaluating dimensions of summary quality that rely on the source document such as factuality.

Original: SMART-N
Revised: SMART-N

Original: In order for SMART to work, summaries should be split into sentences.
Revised: In order for SMART to work, summaries should be split into sentences.

Original: Let R =
Revised: Let R =

Original: [ r i ] and C =
Revised: [ r i ] and C =

Original: [ c j ] be the sequence of sentences of the reference and the candidate system summary. 
Revised: [ c j ] be the sequence of sentences of the reference and the candidate system summary. [Equation]

Original:  SMART-N ﬁnds pairs of sentence n-grams in R and C that have the highest matching scores given by a sentence matching function that returns a score between 0 and 1 (detailed description in Section
Revised: [Figure] SMART-N ﬁnds pairs of sentence n-grams in R and C that have the highest matching scores given by a sentence matching function that returns a score between 0 and 1 (detailed description in Section

Original: A.1).
Revised: A.1).

Original: Formally, given N as the number of sentences in the sentence n-gram, SMART-N can be calculated as follows: 
Revised: Formally, given N as the number of sentences in the sentence n-gram, SMART-N can be calculated as follows: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where match ( · , · ) is the sentence matching function, prec , rec , and f are precision, recall, and f-measure, respectively.
Revised: where match ( · , · ) is the sentence matching function, prec , rec , and f are precision, recall, and f-measure, respectively.

Original: Note that unlike ROUGE, the numerators of precision and recall are different due to the use of a soft-matching function, thus they are calculated separately.
Revised: Note that unlike ROUGE, the numerators of precision and recall are different due to the use of a soft-matching function, thus they are calculated separately.

Original: In our experiments, we used SMART-1 and SMART-2 1 , but SMART-N can be easily extended to work with larger N s.
Revised: In our experiments, we used SMART-1 and SMART-2 1 , but SMART-N can be easily extended to work with larger N s.

Original: SMART-L SMART-L is essentially the Longest Common Subsequence (LCS) of sentences in the reference and the candidate system summary.
Revised: SMART-L SMART-L is essentially the Longest Common Subsequence (LCS) of sentences in the reference and the candidate system summary.

Original: However, the original LCS algorithm requires an exact match to work.
Revised: However, the original LCS algorithm requires an exact match to work.

Original: We instead use a soft version of LCS, where the task is deﬁned as:
Revised: We instead use a soft version of LCS, where the task is deﬁned as:

Original: Given two sequences X =
Revised: Given two sequences X =

Original: [ x i ] and Y =
Revised: [ x i ] and Y =

Original: [ y j ] and a matching function match ( x i , y j ) , ﬁnd two softsubsequences x i 1 , ..., x i l and y j 1 , ..., y j l of length l with i k − 1 ≤ i k ≤ i k +1 and j k − 1 ≤ j k ≤ j k +1 , maximizing the sum (cid:80) lk
Revised: [ y j ] and a matching function match ( x i , y j ) , ﬁnd two softsubsequences x i 1 , ..., x i l and y j 1 , ..., y j l of length l with i k − 1 ≤ i k ≤ i k +1 and j k − 1 ≤ j k ≤ j k +1 , maximizing the sum (cid:80) lk

Original: =1 match ( x i k , y j k ) .
Revised: =1 match ( x i k , y j k ) .

Original: Unlike normal subsequences, soft-subsequences allow repetition of sentences as long as they do not go back to previous sentences (hence the use of ≤ operator).
Revised: Unlike normal subsequences, soft-subsequences allow repetition of sentences as long as they do not go back to previous sentences (hence the use of ≤ operator).

Original: This relaxation helps in cases where the meaning of a sentence on one side spans over multiple sentences on the other side.
Revised: This relaxation helps in cases where the meaning of a sentence on one side spans over multiple sentences on the other side.

Original: Furthermore,
Revised: Furthermore,

Original: Soft-LCS is similar but different from a simple sequence alignment problem since the weight of the match depends on both the positions of the items in the sequence and the items themselves.
Revised: Soft-LCS is similar but different from a simple sequence alignment problem since the weight of the match depends on both the positions of the items in the sequence and the items themselves.

Original: It is a less-restricted version of the Heaviest Common Subsequence (HCS; Jacobson & Vo, 1992) since the matching function is relaxed to allow the use of a soft match (which is essentially an exact mismatch) in the subsequence.
Revised: It is a less-restricted version of the Heaviest Common Subsequence (HCS; Jacobson & Vo, 1992) since the matching function is relaxed to allow the use of a soft match (which is essentially an exact mismatch) in the subsequence.

Original: It turns out that Soft-LCS can be solved using a dynamic programming algorithm similar to that of
Revised: It turns out that Soft-LCS can be solved using a dynamic programming algorithm similar to that of

Original: LCS, which is illustrated as a pseudocode in Figure 2.
Revised: LCS, which is illustrated as a pseudocode in Figure 2.

Original: The main difference is that since we do not require an exact match, we always take the maximum among three cases: (1) choosing tosoft-match x i and y j , (2) choosing to soft-match x i and y j − 1 , and (3) choosing to skip x i . [Equation] [Figure] [Equation]
Revised: The main difference is that since we do not require an exact match, we always take the maximum among three cases: (1) choosing to

Original: [Figure]

Original: Given the Soft-LCS function, we can then calculate SMART-L as follows:
Revised: Given the Soft-LCS function, we can then calculate SMART-L as follows:

Original: [Equation]
Revised: [Equation]

Original: Comparing with Source Some dimensions of summary quality require access to source to be effectively evaluated.
Revised: Comparing with Source Some dimensions of summary quality require access to source to be effectively evaluated.

Original: To cover those dimensions, SMART also compares the candidate system summary with the source, in addition to comparison with the reference summary.
Revised: To cover those dimensions, SMART also compares the candidate system summary with the source, in addition to comparison with the reference summary.

Original: Let S =
Revised: Let S =

Original: [ s k ] be the sequence of sentences of the source document.
Revised: [ s k ] be the sequence of sentences of the source document.

Original: SMART that uses both source and reference is calculated as follows.
Revised: SMART that uses both source and reference is calculated as follows.

Original: We ﬁrst calculate two SMART scores that (1) compares candidate system summary C with reference summary R , and (2) compares C with source document S .
Revised: We ﬁrst calculate two SMART scores that (1) compares candidate system summary C with reference summary R , and (2) compares C with source document S .

Original: Then, we aggregate the scores by taking their maximum.
Revised: Then, we aggregate the scores by taking their maximum.

Original: For example, SMART-N ( S, R, C ) is calculated as:
Revised: For example, SMART-N ( S, R, C ) is calculated as:

Original: [Equation]
Revised: [Equation]

Original: Multiple References Finally, when there are multiple reference summaries, we calculate SMART for each reference, and aggregate them by taking their maximum, as also commonly done in previous work (Fabbri et al., 2021).
Revised: Multiple References Finally, when there are multiple reference summaries, we calculate SMART for each reference, and aggregate them by taking their maximum, as also commonly done in previous work (Fabbri et al., 2021).

Original: This is intuitive since the candidate system summary only needs to match with at least one of the reference.
Revised: This is intuitive since the candidate system summary only needs to match with at least one of the reference.

Original: Shorter Acronym We use the following template to describe SMART variants in a space-efﬁcient manner: S[1|2|L]-m , where m is the sentence matching function of choice.
Revised: Shorter Acronym We use the following template to describe SMART variants in a space-efﬁcient manner: S[1|2|L]-m , where m is the sentence matching function of choice.

Original: For example, SMART1 with a BLEURT (Sellam et al., 2020) matching function can be shortened into S1-BLEURT.
Revised: For example, SMART1 with a BLEURT (Sellam et al., 2020) matching function can be shortened into S1-BLEURT.

Original:  E XPERIMENTS AND R ESULTS
Revised:  E XPERIMENTS AND R ESULTS

Original: Sentence Matching Functions
Revised: Sentence Matching Functions

Original: One advantage of SMART is that it is easily extensible by changing the matching functions depending on the task.
Revised: One advantage of SMART is that it is easily extensible by changing the matching functions depending on the task.

Original: We investigated and compared six different matching functions, three model-based and three string-based, in Sections A.1 and A.5 of the Appendix and use the ones that perform the best for each task.
Revised: We investigated and compared six different matching functions, three model-based and three string-based, in Sections A.1 and A.5 of the Appendix and use the ones that perform the best for each task.

Original: Speciﬁcally, for document summarization, we use
Revised: Speciﬁcally, for document summarization, we use

Original: BLEURT (Sellam et al., 2020) as the best model-based function and CHRF (Popovi´c, 2015) as the best string-based function.
Revised: BLEURT (Sellam et al., 2020) as the best model-based function and CHRF (Popovi ´ c, 2015) as the best string-based function.

Original: For factual consistency, we use ANLI (Honovich et al., 2022) as the matching function.
Revised: For factual consistency, we use ANLI (Honovich et al., 2022) as the matching function.

Revised: Using these matching functions for the corresponding tasks is intuitive; BLEURT has been speciﬁcally trained to rank texts using objectives that correspond to multiple dimensions of text quality, while ANLI is an NLI classiﬁer that outputs a binary prediction (entailed or not).

Original: Implementation Details
Revised: Implementation Details

Original: The sentence matching functions are implemented as follows.
Revised: The sentence matching functions are implemented as follows.

Original: For
Revised: For

Original: BLEURT (Sellam et al., 2020), we used the BLEURT-20 checkpoint 2 suggested by the authors which also supports non-English languages.
Revised: BLEURT (Sellam et al., 2020), we used the BLEURT-20 checkpoint 2 suggested by the authors which also supports non-English languages.

Original: For ANLI, we used the same implementation as in
Revised: For ANLI, we used the same implementation as in

Original: Honovich et al.
Revised: Honovich et al.

Original: (2022), where T5-11B is ﬁne-tuned with 25K training steps on ANLI (Nie et al., 
Revised: (2022), where T5-11B is ﬁne-tuned with 25K training steps on ANLI (Nie et al., 2020), treating both contradiction and neutral pairs as not entailed. Sentences are split using nltk 3 .

Original: For the document summarization experiments, we report f-measure scores whenever available, such as in ROUGE, BERTScore, and SMART.
Revised: For the document summarization experiments, we report f-measure scores whenever available, such as in ROUGE, BERTScore, and SMART.

Original: For the factual consistency experiments, we report precision scores.
Revised: For the factual consistency experiments, we report precision scores.

Original: We also report the version of SMART that considers both source and reference as in Eq 11.
Revised: We also report the version of SMART that considers both source and reference as in Eq 11.

Original: We tested our metric in two different tasks: document summarization and factual consistency.
Revised: We tested our metric in two different tasks: document summarization and factual consistency.

Original: We discuss them in detail in the next sections.
Revised: We discuss them in detail in the next sections.

Original: 5. D OCUMENT S UMMARIZATION
Revised: 5. D OCUMENT S UMMARIZATION

Original: Dataset and Evaluation We conducted experiments on the SummEval dataset (Fabbri et al., 2021), a document summarization meta-evaluation suite consisting of summaries from the CNN/DM dataset  (Hermann et al., 2015).
Revised: Dataset and Evaluation We conducted experiments on the SummEval dataset (Fabbri et al., 2021), a document summarization meta-evaluation suite consisting of summaries from the CNN/DM dataset nltk/ (Hermann et al., 2015).

Original: Annotation is done in two stages and using experts to ensure high quality and high agreement across annotators.
Revised: Annotation is done in two stages and using experts to ensure high quality and high agreement across annotators.

Original: There are 1600 data points (16 systems × 100 examples) in total, each of which includes a score between 1 to 5 for each of the four dimensions of summary quality, which represents the average score given by three experts.
Revised: There are 1600 data points (16 systems × 100 examples) in total, each of which includes a score between 1 to 5 for each of the four dimensions of summary quality, which represents the average score given by three experts.

Original: Each data point also includes 11 reference summaries: the original summary from the CNN/DM dataset and 10 human-written abstractive summaries from Kryscinski et al.
Revised: Each data point also includes 11 reference summaries: the original summary from the CNN/DM dataset and 10 human-written abstractive summaries from Kryscinski et al.

Original: For evaluation, we use system-level correlation using Kendall tau, where we ﬁrst take the average score for each system and take the correlation.
Revised: For evaluation, we use system-level correlation using Kendall tau, where we ﬁrst take the average score for each system and take the correlation.

Original: Comparison Metrics We compared SMART with three types of metrics:  string-based metrics,  source-free and source-dependent model-based metrics.
Revised: Comparison Metrics We compared SMART with four types of metrics: source-free and sourcedependent string-based metrics, and source-free and source-dependent model-based metrics.

Original: String-based metrics include: (1-3) ROUGE-1/2/L (Lin, 2004) measures token-level overlap between reference and output summaries; (4) BLEU (Papineni et al., 2002) measures token-level overlap between reference and output summaries with a focus on precision; (5) CHRF
Revised: Source-free string-based metrics include: (1-3) ROUGE-1/2/L (Lin, 2004) measures token-level overlap between reference and output summaries; (4) BLEU (Papineni et al., 2002) measures tokenlevel overlap between reference and output summaries with a focus on precision; and(5)

Original: (Popovi ´ c, 2015) measures character-based n-gram overlap between reference and output summaries; and (6-8) S1/2/LCHRF, our best SMART metric using a string-based matching function, as shown in §A.1 ofthe
Revised: CHRF (Popovi´c, 2015) measures character-based n-gram overlap between reference and output summaries. Source-dependent string-based metrics include: (6-9) src-ROUGE-1/2/L and src-CHRF, versions of

Original:  Appendix.
Revised: ROUGE and CHRF that compare the output summary with the source instead of with the reference; and (10-12) S1/2/ L-CHRF, our best SMART metric using a string-based matching function, as shown in §A.1 of the Appendix.

Original: Source-free model-based metrics include: (9) BERTScore (Zhang* et al., 2020):
Revised: Source-free model-based metrics include: (13) BERTScore (Zhang* et al., 2020):

Original: A metric that relies on BERT(-like) models (Devlin et al., 2019) and computes an aggregation of the token-level similarity scores; (10) MoverScore
Revised: A metric that relies on BERT(-like) models (Devlin et al., 2019) and computes an aggregation of the token-level similarity scores; (14) MoverScore

Original: (Zhao et al., 2019): measures the semantic distance between BERT n-gram embeddings of reference and candidate summaries using Word Mover’s Distance (WMD; Kusner et al., 2015); (11) BLEURT (Sellam et al., 2020): ﬁnetunes BERT using a combination of real and synthetic training data with gold-/silver-standard human judgment scores; and (12) Sentence Mover’s
Revised: (Zhao et al., 2019): measures the semantic distance between BERT n-gram embeddings of reference and candidate summaries using Word Mover’s Distance (WMD; Kusner et al., 2015); (15) BLEURT (Sellam et al., 2020): ﬁnetunes BERT using a combination of real and synthetic training data with gold-/silver-standard human judgment scores; and (16) Sentence Mover’s

Original: Similarity (SMS; Clark et al., 2019): uses an extension of WMD that works with sentences instead of tokens.
Revised: Similarity (SMS; Clark et al., 2019): uses an extension of WMD that works with sentences instead of tokens.

Original: Source-dependent model-based metrics include: (13) PRISM (Thompson & Post, 2020): leverages a zero-shot paraphrasing model and uses probabilities from force-decoding the candidate summary given the source as input; (14) Q 2 (Honovich et al., 2021): employs question generation and question answering models and checks whether answers from the summary are entailed by answers from source; (15) ANLI (Honovich et al., 2022): ﬁne-tunes T5 (Raffel et al., 2020) using the ANLI dataset (Nie et al., 2020) to produce an entailment score given the source as premise and the summary as hypothesis; (16-17) BARTScore(+CNN)
Revised: Source-dependent model-based metrics include: (17) src-BLEURT, a version of BLEURT that compare the output summary with the source instead of with the reference; (18) PRISM (Thompson & Post, 2020): leverages a zero-shot paraphrasing model and uses probabilities from force-decoding the candidate summary given the source as input; (19) Q 2 (Honovich et al., 2021): employs question generation and question answering models and checks whether answers from the summary are entailed by answers from source; (20) ANLI (Honovich et al., 2022): ﬁne-tunes T5 (Raffel et al., 2020) using the ANLI dataset (Nie et al., 2020) to produce an entailment score given the source as premise and the summary as hypothesis; (21-22) BARTScore(+CNN)

Original: (Yuan et al., 2021): evaluates text using probabilities from force-decoding the candidate summary given the source as input using BART (Lewis et al., 2020) without (with) ﬁnetuning with CNN/DM summarization dataset (Hermann et al., 2015); and (18-20) S1/2/
Revised: (Yuan et al., 2021): evaluates text using probabilities from force-decoding the candidate summary given the source as input using BART (Lewis et al., 2020) without (with) ﬁnetuning with CNN/DM summarization dataset (Hermann et al., 2015); and (23-25) S1/2/

Original: L-BLEURT: Our best SMART metric using a model-based matching function, as shown in §A. of the Appendix.
Revised: L-BLEURT: Our best SMART metric using a model-based matching function, as shown in §A.1 of the Appendix.

Original: Results Table 1 reports the system-level correlations of different metrics for each quality dimension.
Revised: Results Table 1 reports the system-level correlations of different metrics for each quality dimension.

Original: For all quality dimensions, SMART with BLEURT matching function has the highest correlation, where SL-BLEURT evaluates coherence and informativeness better, and S1-BLEURT and S2BLEURT evaluate factuality and ﬂuency better, respectively.On average, SL-BLEURT performs best, [Table] outperformed by any other metric (using William’s Test; Graham &
Revised: For all quality dimensions, SMART with BLEURT matching function has the highest correlation, where SL-BLEURT evaluates coherence and informativeness better, and S1-BLEURT and S2BLEURT evaluate factuality and ﬂuency better, respectively.

Original: Baldwin, 2014) for that speciﬁc dimension are boldfaced .

Original: We also show the average scores in the µ column, where the best values for each block are underlined. Note that BARTScore+CNN uses BART that is ﬁne-tuned on CNN/DM, the same data source as SummEval, thus direct comparison with the other metrics is not possible. followed by S1-BLEURT and S2-BLEURT, all three of which outperforming BARTScore+CNN, which is ﬁnetuned on the same summarization dataset as SummEval.
Revised: On average, SL-BLEURT performs best, followed by S1-BLEURT and S2-BLEURT, all three of which outperforming BARTScore+CNN, which is ﬁnetuned on the same summarization dataset as SummEval.

Original: S2-BLEURT also performs comparably with Q 2 in factuality evaluation.
Revised: S2-BLEURT also performs comparably with Q 2 in factuality evaluation.

Original: Given that each of the SMART metrics are better at evaluating different quality dimensions, it is therefore recommended to use them as a set, similar to how ROUGE metrics are used.
Revised: Given that each of the SMART metrics are better at evaluating different quality dimensions, it is therefore recommended to use them as a set, similar to how ROUGE metrics are used.

Original: We can also see in the table that source-dependent metrics are better than source-free ones, signifying the importance of the use of source documents during summary evaluation.
Revised: We can also see in the table that source-dependent metrics are better than source-free ones, signifying the importance of the use of source documents during summary evaluation.

Revised: However, just simply using the source does not improve the metric; src-ROUGE1/2/L, src-CHRF, and src-BLEURT obtain worse correlation scores compared to their reference-only counterparts.

Original: Among source-free metrics, SMS performs the best, showing the superiority of sentencelevel matching metrics against the token-level ones.
Revised: Among source-free metrics, SMS performs the best, showing the superiority of sentence- [Table] level matching metrics against the token-level ones.

Original: Among string-based metrics, SL-CHRF performs the best, with a competitive system-level correlation when compared with BARTScore+CNN on average.
Revised: Among string-based metrics, SL-CHRF performs the best, with a competitive system-level correlation when compared with BARTScore+CNN on average.

Original: This shows that SMART achieves comparable evaluation power with previous LM-based metrics even without using any pretrained language models.
Revised: This shows that SMART achieves comparable evaluation power with previous LM-based metrics even without using any pretrained language models.

Original: Ablation Studies We present various ablation studies on the different components of SMART in
Revised: Ablation Studies We present various ablation studies on the different components of SMART in

Original: Table 2.
Revised: Table 2.

Original: For simplicity, we report on SMART- X , an average of SMART-l, SMART-2 and SMART-L.
Revised: For simplicity, we report on SMART- X , an average of SMART-l, SMART-2 and SMART-L.

Original: The ﬁrst block contains SMART- X that uses only either precision or recall, both of which have substantially lower system-level correlation than f-measure.
Revised: The ﬁrst block contains SMART- X that uses only either precision or recall, both of which have substantially lower system-level correlation than f-measure.

Original: The recall variant performs better, which follows results from traditionally recall-oriented evaluation metrics in summarization (Lin, 2004).
Revised: The recall variant performs better, which follows results from traditionally recall-oriented evaluation metrics in summarization (Lin, 2004).

Original: The second block contains different ways to aggregate SMART scores that compare candidate summaries (1) to the source documents and (2) to the reference summaries.
Revised: The second block contains different ways to aggregate SMART scores that compare candidate summaries (1) to the source documents and (2) to the reference summaries.

Original: When using only one of the two scores (i.e., reference- or source-only), SMART scores signiﬁcantly perform worse, which implies that using both source and reference is necessary for SMART to work.
Revised: When using only one of the two scores (i.e., reference- or source-only), SMART scores signiﬁcantly perform worse, which implies that using both source and reference is necessary for SMART to work.

Original: We also tried 
Revised: We also tried aggregation through taking the average or the minimum, both of which perform worse than taking the maximum as in Eq 11. Taking the minimum notably does not increase the correlation, and this

Original: [Table]
Revised: [Table]

Original: aggregation through taking the average or the minimum, both of which perform worse than taking the maximum as in Eq 11.
Revised: is because it restricts giving high scores to summaries that match both the source and the reference, which is not necessary.

Original: One interesting ﬁnding is that using only the reference summaries to calculate SMART gives negative correlation scores on all dimensions.
Revised: One interesting ﬁnding is that using only the reference summaries to calculate SMART gives negative correlation scores on all dimensions.

Revised: This can be explained in two-folds.

Revised: Firstly, human annotations in SummEval is done by only looking at the source document, which means they are not biased towards summaries that look closer to the given reference summaries.

Original: Given the ﬁndings from Fabbri et al.
Revised: And secondly, Fabbri et al.

Original: (2021)  that CNN/DM summaries are worse than system summaries in terms of human ratings, we further investigated using different reference summaries for SMART.
Revised: (2021) showed that CNN/DM summaries are worse than system summaries in terms of human ratings. Given these [Table] we further investigated using different reference summaries for SMART.

Original: Speciﬁcally, we tried (1) replacing the set of reference summaries from SummEval into the original summary from CNN/DM, and (2) treating the best system summary according to the average human score as the reference.
Revised: Speciﬁcally, we tried (1) replacing the set of reference summaries from SummEval into the original summary from CNN/DM, and (2) treating the best system summary according to the average human score as the reference.

Original: As can be seen in the third block of Table 2, CNN/DM reference summaries correlate negatively on all dimensions.
Revised: As can be seen in the third block of Table 2, CNN/DM reference summaries correlate negatively on all dimensions.

Original: When using the best system summary as the reference, SMART now obtains positive correlations across all dimensions.
Revised: When using the best system summary as the reference, SMART now obtains positive correlations across all dimensions.

Original: The correlation further improves when SMART uses both the source and the new reference.
Revised: The correlation further improves when SMART uses both the source and the new reference.

Original: Further Analyses We provide several analyses in the Appendix to further understand our metrics.
Revised: Further Analyses We provide several analyses in the Appendix to further understand our metrics.

Original: In §A.2, we show that SMART works better with longer summaries, and in §A.3, we also show that it is less biased towards speciﬁc models compared to other competing metrics.
Revised: In §A.2, we show that SMART works better with longer summaries, and in §A.3, we also show that it is less biased towards speciﬁc models compared to other competing metrics.

Original: Finally, in §A.4, we show that source-only SMART works well as a reference-free metric and correctly ranks systems that popular metrics such as ROUGE (Lin, 2004), BERTScore (Zhang* et al., 2020), and QAEval (Deutsch et al., 2021) cannot, according to Goyal et al.
Revised: Finally, in §A.4, we show that source-only SMART works well as a reference-free metric and correctly ranks systems that popular metrics such as ROUGE (Lin, 2004), BERTScore (Zhang* et al., 2020), and QAEval (Deutsch et al., 2021) cannot, according to Goyal et al.

Original: 5. F ACTUAL C ONSISTENCY
Revised: 5. F ACTUAL C ONSISTENCY

Original: Dataset and Evaluation We conducted experiments on the TRUE benchmark (Honovich et al., 2022), a meta-evaluation suite for factuality, consisting of source and candidate texts from different kinds of text generation tasks: abstractive summarization, dialogue generation, and paraphrase detection.
Revised: Dataset and Evaluation We conducted experiments on the TRUE benchmark (Honovich et al., 2022), a meta-evaluation suite for factuality, consisting of source and candidate texts from different kinds of text generation tasks: abstractive summarization, dialogue generation, and paraphrase detection.

Original: We note that there is no reference text given and the candidate text is compared only to the source.
Revised: We note that there is no reference text given and the candidate text is compared only to the source.

Original: In this benchmark, factuality scores are binary (positive/negative, in contrast to the ﬁve-scale scoring in SummEval), and the generated text needs to be fully factual to get a positive score.
Revised: In this benchmark, factuality scores are binary (positive/negative, in contrast to the ﬁve-scale scoring in SummEval), and the generated text needs to be fully factual to get a positive score.

Original: Given this, the metrics are evaluated using the Received Operating Characteristic Area Under the Curve (ROC AUC) to remove the necessity of setting a speciﬁc decision threshold.
Revised: Given this, the metrics are evaluated using the Received Operating Characteristic Area Under the Curve (ROC AUC) to remove the necessity of setting a speciﬁc decision threshold.

Original: Comparison Metrics
Revised: Comparison Metrics

Original: The best matching function for factual consistency is ANLI, as shown in §A.5 of the Appendix.
Revised: The best matching function for factual consistency is ANLI, as shown in §A.5 of the Appendix.

Original: Since there is no reference text, we used the version of SMART that compares only with the source.
Revised: Since there is no reference text, we used the version of SMART that compares only with the source.

Original: Moreover, we aggregated S1/2/
Revised: Moreover, we aggregated S1/2/

Original: L-ANLI into S X
Revised: L-ANLI into S X

Original: -ANLI by taking their mean forbrevity (results of individual metrics are reported in §A.5). We compared (1) S X -ANLI with three factuality-speciﬁc evaluation metrics: (2) Q 2 (Honovich et al., 2021); (3) ANLI (Honovich et al.,
Revised: -ANLI by taking their mean for

Original: [Table]
Revised: [Table]

Original: FRANK
Revised: FRANK

Original: [Table]
Revised: [Table]

Original: ; and ( SC ZS (Summary Consistency; Laban et al., 2022) that splits the source  texts into sentences, computes entailment probabilities on all sentence pairs, and aggregates them into a single score.
Revised: Results can be from ensembles of multiple metrics, and are grouped by the number of metrics used in the ensemble brevity (results of individual metrics are reported in §A.5). We compared (1) S X -ANLI with three factuality-speciﬁc evaluation metrics: (2) Q 2 (Honovich et al., 2021); (3) ANLI (Honovich et al., 2022); and (4) SC ZS (Summary Consistency; Laban et al., 2022) that splits the source and candidate texts into sentences, computes entailment probabilities on all sentence pairs, and aggregates them into a single score.

Original: The last metric is similar to S1-ANLI, but with two key differences: (a) The aggregation is different; and (b) Training is done using both NLI and fact veriﬁcation datasets.
Revised: The last metric is similar to S1-ANLI, but with two key differences: (a) The aggregation is different; and (b) Training is done using both NLI and fact veriﬁcation datasets.

Original: Finally, following Honovich et al.
Revised: Finally, following Honovich et al.

Original: we also compared three ensembles of multiple metrics by averaging their scores: (5) ANLI + Q 2 , (6) ANLI + S X
Revised: we also compared three ensembles of multiple metrics by averaging their scores: (5) ANLI + Q 2 , (6) ANLI + S X

Original: -ANLI, and (7) ANLI
Revised: -ANLI, and (7) ANLI

Original: + Q 2 + SZ ZS .
Revised: + Q 2 + SZ ZS .

Original: Results Table 3 shows the ROC AUC scores of different metrics.
Revised: Results Table 3 shows the ROC AUC scores of different metrics.

Original: Among single metrics, S X -ANLI performs the best, substantially improving over the second best metric.
Revised: Among single metrics, S X -ANLI performs the best, substantially improving over the second best metric.

Original: Moreover, ensembling
Revised: Moreover, ensembling

Original: S X -ANLI with ANLI provides another boost, performing superiorly compared to the ANLI + Qensemble metric and comparably with the ANLI + Q 2 + SZ ZS ensemble metric.
Revised: S X -ANLI with ANLI provides another boost, performing superiorly compared to the ANLI + Qensemble metric and comparably with the ANLI + Q 2 + SZ ZS ensemble metric.

Original: Our ensemble metric is also more cost-effective since it only requires a single model, while the other ensemble metric requires at least two more (e.g., question answering and generation models for Q 2 ).
Revised: Our ensemble metric is also more cost-effective since it only requires a single model, while the other ensemble metric requires at least two more (e.g., question answering and generation models for Q 2 ).

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this paper, we proposed SMART (Sentence Matching for Rating Text), a new metric for evaluating generated text given both a source document and a reference text.
Revised: In this paper, we proposed SMART (Sentence Matching for Rating Text), a new metric for evaluating generated text given both a source document and a reference text.

Original: SMART makes use of a sentencelevel soft-matching function to match sentences, which can either be string-based or model-based depending on the available resources.
Revised: SMART makes use of a sentencelevel soft-matching function to match sentences, which can either be string-based or model-based depending on the available resources.

Original: This function can easily be replaced with new and better ones, which allows us to create a better SMART metric.
Revised: This function can easily be replaced with new and better ones, which allows us to create a better SMART metric.

Original: We provided two types of SMART based on n-gram overlap and longest common subsequence, and our extensive experiments showed that
Revised: We provided two types of SMART based on n-gram overlap and longest common subsequence, and our extensive experiments showed that

Original: SMART evaluates document summaries better in terms of all four dimensions of summary quality: coherence, factuality, ﬂuency, and informativeness.
Revised: SMART evaluates document summaries better in terms of all four dimensions of summary quality: coherence, factuality, ﬂuency, and informativeness.

Original: Our analyses also showed that SMART is better as summary length increases and is less biased than other competing metrics.
Revised: Our analyses also showed that SMART is better as summary length increases and is less biased than other competing metrics.

Original: 6. L IMITATIONS AND F UTURE W ORK
Revised: 6. L IMITATIONS AND F UTURE W ORK

Original: We acknowledge several limitations of our work.
Revised: We acknowledge several limitations of our work.

Original: Firstly, SMART currently assumes single-source inputs, which makes it non-trivial to use for evaluating texts in tasks with multi-source inputs.
Revised: Firstly, SMART currently assumes single-source inputs, which makes it non-trivial to use for evaluating texts in tasks with multi-source inputs.

Original: While this can be mitigated by concatenating documents in chronological order, there may be cases where such ordering is not possible.
Revised: While this can be mitigated by concatenating documents in chronological order, there may be cases where such ordering is not possible.

Original: Ultimately, ﬁnding an efﬁcient way to extend the (soft version of) longest common subsequence algorithm into multiple sources is necessary, which we plan to explore in the future.
Revised: Secondly, we plan to develop a system to automatically transform text into a list of semantic content units (SCUs), or sentences that contain a single fact.

Revised: This would improve SMART by supporting texts with sentences that have more than one atomic fact.

Original: Moreover, we plan to maintain an open-source toolkit which includes scripts to easily run evaluation using SMART, and is updated accordingly when new features, such as better matching functions or support for multi-source inputs.
Revised: Finally, we plan to maintain an open-source toolkit which includes scripts to easily run evaluation using SMART, and is updated accordingly with new support, such as better matching functions or support for multi-source inputs.

Original: A A PPENDIX A. S ENTENCE M ATCHING F UNCTIONS
Revised: A A PPENDIX A. S ENTENCE M ATCHING F UNCTIONS

Original: The scores we get from SMART depend on the sentence matching function match .
Revised: The scores we get from SMART depend on the sentence matching function match .

Original: We investigate six different sentence matching functions widely used in both machine translation and document summarization.
Revised: We investigate six different sentence matching functions widely used in both machine translation and document summarization.

Original: Speciﬁcally, we compare three string-based and three model-based matching functions.
Revised: Speciﬁcally, we compare three string-based and three model-based matching functions.

Original: The former do not rely on accelerators (GPUs/TPUs) while the latter leverage pretrained neural models.
Revised: The former do not rely on accelerators (GPUs/TPUs) while the latter leverage pretrained neural models.

Original: Thus, we expect string-based matching functions to be inferior, but they are good alternatives for faster evaluation or when accelerators are not available.
Revised: Thus, we expect string-based matching functions to be inferior, but they are good alternatives for faster evaluation or when accelerators are not available.

Original: ROUGE (Lin, 2004)
Revised: ROUGE (Lin, 2004)

Original: A popular document summarization evaluation metric, it measures the number of overlapping textual units.
Revised: A popular document summarization evaluation metric, it measures the number of overlapping textual units.

Original: As with most summarization work, we explored three types of textual units: unigrams (ROUGE-1), bigrams (ROUGE-2), and longest common subsequence (ROUGE-L).
Revised: As with most summarization work, we explored three types of textual units: unigrams (ROUGE-1), bigrams (ROUGE-2), and longest common subsequence (ROUGE-L).

Original: BLEU (Papineni et al., 2002)
Revised: BLEU (Papineni et al., 2002)

Original: A popular machine translation evaluation metric, it is a precisionfocused metric that calculates n-gram overlap between two texts and also includes a brevity penalty.
Revised: A popular machine translation evaluation metric, it is a precisionfocused metric that calculates n-gram overlap between two texts and also includes a brevity penalty.

Original: CHRF (Popovi´c, 2015)
Revised: CHRF (Popovi´c, 2015)

Original: Another machine translation evaluation metric, it calculates characterbased n-gram overlap between system and reference sentences.
Revised: Another machine translation evaluation metric, it calculates characterbased n-gram overlap between system and reference sentences.

Original: Unlike ROUGE and BLEU which operate at the token level, CHRF is more effective especially in morphologically-rich languages as it does not require any tokenization, lemmatization, and stemming (Mathur et al., 2020; Kocmi et al., 2021).
Revised: Unlike ROUGE and BLEU which operate at the token level, CHRF is more effective especially in morphologically-rich languages as it does not require any tokenization, lemmatization, and stemming (Mathur et al., 2020; Kocmi et al., 2021).

Original: BERTScore (Zhang* et al., 2020)
Revised: BERTScore (Zhang* et al., 2020)

Original: A model-based metric that leverages contextualized token embeddings from BERT(-like models).
Revised: A model-based metric that leverages contextualized token embeddings from BERT(-like models).

Original: It computes similarity scores by aligning tokens from reference and candidate summaries, and token alignments are computed greedily to maximize cosine similarity.
Revised: It computes similarity scores by aligning tokens from reference and candidate summaries, and token alignments are computed greedily to maximize cosine similarity.

Original: ANLI (Honovich et al., 2022)
Revised: ANLI (Honovich et al., 2022)

Original: Another model-based metric mainly used to evaluate factuality, which uses T5 (Raffel et al., 2020) ﬁne-tuned on the ANLI dataset (Nie et al., 2020) to produce a score between 0 (not entailed) and 1 (entailed) given a premise and a hypothesis.
Revised: Another model-based metric mainly used to evaluate factuality, which uses T5 (Raffel et al., 2020) ﬁne-tuned on the ANLI dataset (Nie et al., 2020) to produce a score between 0 (not entailed) and 1 (entailed) given a premise and a hypothesis.

Original: We use the source/reference as premise and the candidate summary as hypothesis.
Revised: We use the source/reference as premise and the candidate summary as hypothesis.

Original: BLEURT (Sellam et al., 2020)
Revised: BLEURT (Sellam et al., 2020)

Original: A supervised model-based metric that uses BERT that is trained to predict human judgment scores using a small-scale dataset.
Revised: A supervised model-based metric that uses BERT that is trained to predict human judgment scores using a small-scale dataset.

Original: To make it more robust, the model is ﬁrst pretrained with a large-scale synthetic dataset.
Revised: To make it more robust, the model is ﬁrst pretrained with a large-scale synthetic dataset.

Original: Moreover, it is optimized using several objectives including ROUGE, BLEU, BERTScore, and entailment.
Revised: Moreover, it is optimized using several objectives including ROUGE, BLEU, BERTScore, and entailment.

Original: BLEURT has been shown to be effective in evaluating sentence match in machine translation, thus we expect it to be the better matching function among all matching functions.
Revised: BLEURT has been shown to be effective in evaluating sentence match in machine translation, thus we expect it to be the better matching function among all matching functions.

Original: One advantage of SMART is that it is easily extensible by changing the matching functions to better ones.
Revised: One advantage of SMART is that it is easily extensible by changing the matching functions to better ones.

Original: This means that a more domain-speciﬁc matching function can be used for evaluation towards speciﬁc domains, or a better-performing sentence matching metric can be used to improve overall evaluation.
Revised: This means that a more domain-speciﬁc matching function can be used for evaluation towards speciﬁc domains, or a better-performing sentence matching metric can be used to improve overall evaluation.

Original: SMART with Different Matching Functions
Revised: SMART with Different Matching Functions

Original: We compare different variants of SMART using the six matching functions described in Section A.1.
Revised: We compare different variants of SMART using the six matching functions described in Section A.1.

Original: Table 4 shows their system-level correlations, where
Revised: Table 4 shows their system-level correlations, where

Original: Coh, Fac, Flu, and Inf stand for coherence, factuality, ﬂuency, and informativeness, respectively.
Revised: Coh, Fac, Flu, and Inf stand for coherence, factuality, ﬂuency, and informativeness, respectively.

Original: Among string-based matching functions, CHRF performs the best in terms of average correlation, followed by BLEU.
Revised: Among string-based matching functions, CHRF performs the best in terms of average correlation, followed by BLEU.

Original: This shows that machine translation metrics are better sentence matchers.
Revised: This shows that machine translation metrics are better sentence matchers.

Original: Among model-based matching functions, BLEURT performs the best by a large margin; SMART with
Revised: Among model-based matching functions, BLEURT performs the best by a large margin; SMART with

Original: BLEURT signiﬁcantly outperforms all the other model-based variants on all dimensions of summary quality.
Revised: BLEURT signiﬁcantly outperforms all the other model-based variants on all dimensions of summary quality.

Original: We believe that this is because BLEURT is optimized to match sentences, as well as to predict ROUGE, BLEU, BERTScore, and entailment scores.
Revised: We believe that this is because BLEURT is optimized to match sentences, as well as to predict ROUGE, BLEU, BERTScore, and entailment scores.

Original: Interestingly, ANLI as a matching function underperforms even in the factuality dimension.
Revised: Interestingly, ANLI as a matching function underperforms even in the factuality dimension.

Original: We posit that this is because sentences are passed to the matching function without their neighboring context.
Revised: We posit that this is because sentences are passed to the matching function without their neighboring context.

Original: [Table] ANLI
Revised: Finally, when compared against the no-SMART variant (e.g., BLEURT vs. S1/2/L-BLEURT), SMART improves the correlation

Original: [Table]
Revised: [Table]

Original: BLEURT
Revised: overall, except for three matching functions: ROUGE-1/2 and ANLI.

Original: [Table]
Revised: We believe this is because they are not effective as a sentence matching function.

Original: [Figure] [Table] [Figure]
Revised: ROUGE-1/2 only matches unigrams/bigrams, in contrast to BLEU (where all n-grams up to n = 4 are considered collectively) and BertScore (where the n-grams are contextualized).

Original: [Table] [Figure]
Revised: ANLI is an NLI classiﬁer that produces predictions that are close to 0 or 1, which is not ideal for ranking two systems with different degrees of errors.

Original: A. SMART W ORKS W ELL WITH L ONGER S UMMARIES
Revised: A. SMART W ORKS W ELL WITH L ONGER S UMMARIES

Original: We divided the datasets into four buckets based on the average number of tokens in the reference summary, where the ﬁrst bucket contains the shortest reference summaries.
Revised: We divided the datasets into four buckets based on the average number of tokens in the reference summary, where the ﬁrst bucket contains the shortest reference summaries.

Original: For each bucket, we then calculated system-level correlation for all competing metrics.
Revised: For each bucket, we then calculated system-level correlation for all competing metrics.

Original: For each quality dimension, we report the relative increase in correlation with respect to ROUGE- X , which is illustrated in Figure 3.
Revised: For each quality dimension, we report the relative increase in correlation with respect to ROUGE- X , which is illustrated in Figure 3.

Original: As can be seen in the ﬁgure, in general, all metrics perform better relative to ROUGE as the number of tokensincreases from 43 tokens, which shows that ROUGE is not suitable for long summary evaluation. Interestingly, ROUGE also underperforms in the ﬁrst bucket, which means that it is also not good at [Figure]
Revised: As can be seen in the ﬁgure, in general, all metrics perform better relative to ROUGE as the number of tokens

Original: [Figure]
Revised: [Figure]

Original: ) and the pairwise ranking accuracy of different metrics for different summary quality.
Revised: GovReport arXiv Metric relevance factuality geomean relevance factuality geomean ROUGE0. ROUGE0.

Original: Best scores are boldfaced
Revised: ROUGE-L 0. BERTScore 0. BLEURT 0. BARTScore 0.

Revised: S X -BLEURT 0.

Original:  evaluating short summaries.
Revised: S X -BLEURT-precision -0. S X -BLEURT-recall 0. increases from 43 tokens, which shows that ROUGE is not suitable for long summary evaluation. Interestingly, ROUGE also underperforms in the ﬁrst bucket, which means that it is also not good at evaluating short summaries.

Original: Among the competing metrics, S X -BLEURT (and S X -CHRF) correlate the best (and second best) when the there are more tokens in the source/reference.
Revised: Among the competing metrics, S X -BLEURT (and S X -CHRF) correlate the best (and second best) when the there are more tokens in the source/reference.

Revised: We also compared SMART with different metrics on two long-form summarization datasets, arXiv (Cohan et al., 2018) and GovReport (Huang et al., 2021).

Revised: We used the meta-evaluation suite curated in Koh et al.

Revised: (2022), where they annotate scores in terms of relevance (measures whether a summary contains the salient information found in the reference) and factuality (assesses whether a summary is factually consistent with the source).

Revised: Table 5 reports the Spearman rank correlation scores of competing metrics, including ROUGE-1/2/L, BERTScore, BLEURT, BARTScore, and S X -BLEURT.

Original: A. SMART IS L ESS B IASED TOWARDS S PECIFIC M ODELS
Revised: In terms of the geometric average of both relevance and factuality, S X -BLEURT performs the best among all metrics. All previous metrics except BARTScore only positively correlates with relevance. BARTScore is positively correlated with both relevance and factuality, however it is substantially worse than SMART. When dissecting SMART into its precision and recall variants, we see that the precision variant is correlated well with factuality while the recall variant is correlated well with relevance.

Revised: This shows that SMART can be used as a single metric to evaluate both relevance and factuality, which can be useful for hill-climbing during model development.

Revised: A. SMART IS L ESS B IASED TOWARDS S PECIFIC M ODELS

Original: While we acknowledge that all automatic metrics are not perfect as shown in Table 1, their rankings should not be hugely different from human rankings.
Revised: While we acknowledge that all automatic metrics are not perfect as shown in Table 1, their rankings should not be hugely different from human rankings.

Original: Moreover, they should not be biased towards a single summarization model.
Revised: Moreover, they should not be biased towards a single summarization model.

Original: To this end, we get the difference in rankings given by humans and by the automatic metrics for each summarization model for each quality dimension.
Revised: To this end, we get the difference in rankings given by humans and by the automatic metrics for each summarization model for each quality dimension.

Original: Figure 4 illustrates the resulting differences averaged over all dimensions of summary quality, in whichwe have two interesting observations.
Revised: Figure 4 illustrates the resulting differences averaged over all dimensions of summary quality, in which

Original: Firstly, all automatic metrics are in general biased towards
Revised: [Figure]

Original: [Table]
Revised: [Figure]

Original:  ranking extractive systems higher.
Revised: ) and the pairwise ranking accuracy of different metrics for different summary quality. Best scores are boldfaced . we have two interesting observations. Firstly, all automatic metrics are in general biased towards ranking extractive systems higher.

Original: This suggests that a separate extractive and abstractive model evaluation is necessary using current automatic metrics.
Revised: This suggests that a separate extractive and abstractive model evaluation is necessary using current automatic metrics.

Original: We leave exploration of metrics that are equally unbiased towards both extractive and abstractive for future work.
Revised: We leave exploration of metrics that are equally unbiased towards both extractive and abstractive for future work.

Original: Secondly, we found that
Revised: Secondly, we found that

Original: BARTScore scores BART (M22 in Figure 4; Lewis et al., 2020) signiﬁcantly higher than all the other models (BARTScore of BART is − 1 . 398 vs. − 2 .
Revised: BARTScore scores BART (M22 in Figure 4; Lewis et al., 2020) signiﬁcantly higher than all the other models (BARTScore of BART is − 1 . 398 vs. − 2 .

Original: 303 ± 0 .
Revised: 303 ± 0 .

Original: 313 on average without BART).
Revised: 313 on average without BART).

Original: This problem is ampliﬁed when BARTScore is ﬁnetuned using the CNN/DM dataset ( − 0 .
Revised: This problem is ampliﬁed when BARTScore is ﬁnetuned using the CNN/DM dataset ( − 0 .

Original: 488 vs.
Revised: 488 vs.

Original: − 1 .
Revised: − 1 .

Original: 702 ± 0 . 248 ).
Revised: 702 ± 0 . 248 ).

Original: This shows that using pretrained encoder-decoder models for summary evaluation induces bias towards summarization models ﬁnetuned on the same model.
Revised: This shows that using pretrained encoder-decoder models for summary evaluation induces bias towards summarization models ﬁnetuned on the same model.

Original: In Figure 4, we can see that S X -BLEURT is the least biased since its rank differences with human scores are closer to zero.
Revised: In Figure 4, we can see that S X -BLEURT is the least biased since its rank differences with human scores are closer to zero.

Original: To quantitatively measure if the above statement is true, we use two  measures.
Revised: To quantitatively measure if the above statement is true, we use two [Table] measures.

Original: The ﬁrst one is the standard deviation of the rank difference, where the score closest to zero can be considered the least biased.
Revised: The ﬁrst one is the standard deviation of the rank difference, where the score closest to zero can be considered the least biased.

Original: The second measure is the pairwise ranking accuracy, where for all pairs of system, we check whether human and metric rankings are equivalent.
Revised: The second measure is the pairwise ranking accuracy, where for all pairs of system, we check whether human and metric rankings are equivalent.

Original: Table 5 shows these numbers, which show that S X -BLEURT has the lowest standard deviation of the rank difference and the highest pairwise rank accuracy across all quality dimensions.
Revised: Table 6 shows these numbers, which show that S X -BLEURT has the lowest standard deviation of the rank difference and the highest pairwise rank accuracy across all quality dimensions.

Original: This entails that the metric is the least biased among the competing metrics.
Revised: This entails that the metric is the least biased among the competing metrics.

Original: A. SMART IS E FFECTIVE E VEN W ITHOUT R EFERENCE S UMMARIES
Revised: A. SMART IS E FFECTIVE E VEN W ITHOUT R EFERENCE S UMMARIES

Original: In this section, we show that SMART does not require reference summaries to effectively evaluate system summaries.
Revised: In this section [Table] we show that SMART does not require reference summaries to effectively evaluate system summaries.

Original: We make use of two sets of newswire data from BBC and CNN from Goyal et al.
Revised: We make use of two sets of newswire data from BBC and CNN from Goyal et al.

Original: Each dataset consists of a source news article and three candidate summaries generated by (1)
Revised: Each dataset consists of a source news article and three candidate summaries generated by (1)

Original: GPT3-D2: a GPT-3 model (Brown et al., 2020) which is trained to follow user instructions (Ouyang et al., 2022); (2) BRIO (Liu et al., 2022): a summarization model ﬁne-tuned on both CNN/DM (Hermann et al., 2015) and XSum (Narayan et al., 2018) which currently achieves state-of-the-art results; and (3) T0 (Sanh et al., 2022): a prompt-based model ﬁne-tuned on multiple tasks which include summarization datasets.
Revised: GPT3-D2: a GPT-3 model (Brown et al., 2020) which is trained to follow user instructions (Ouyang et al., 2022); (2) BRIO (Liu et al., 2022): a summarization model ﬁne-tuned on both CNN/DM (Hermann et al., 2015) and XSum (Narayan et al., 2018) which currently achieves state-of-the-art results; and (3) T0 (Sanh et al., 2022): a prompt-based model ﬁne-tuned on multiple tasks which include summarization datasets.

Original: Goyal et al.
Revised: Goyal et al.

Original: (2022) showed that reference-dependent metrics such as ROUGE (Lin, 2004), BERTScore (Zhang* et al., 2020), and QAEval (Deutsch et al., 2021) do not correlate well with human judgments.
Revised: (2022) showed that reference-dependent metrics such as ROUGE (Lin, 2004), BERTScore (Zhang* et al., 2020), and QAEval (Deutsch et al., 2021) do not correlate well with human judgments.

Original: We therefore compared reference-free versions of ﬁve metrics: ROUGE 1/2/L, BLEURT, and S XBLEURT, where we calculate the scores by instead comparing the candidate summary with the source.
Revised: We therefore compared reference-free versions of ﬁve metrics: ROUGE 1/2/L, BLEURT, and S XBLEURT, where we calculate the scores by instead comparing the candidate summary with the source.

Original: Table 6 reports the scores of these metrics on the BBC and CNN datasets.
Revised: Table 7 reports the scores of these metrics on the BBC and CNN datasets.

Original: As can be seen in the table, only S X -BLEURT is able to rank the systems correctly on both datasets, according to the given human rankings.
Revised: As can be seen in the table, only S X -BLEURT is able to rank the systems correctly on both datasets, according to the given human rankings.

Original: Both ROUGE and BLEURT scores produce incorrect rankings on one dataset, however GPT3-D2 is ranked the highest in most cases for all metrics.
Revised: Both ROUGE and BLEURT scores produce incorrect rankings on one dataset, however GPT3-D2 is ranked the highest in most cases for all metrics.

Original: This entails that using reference summaries may not be necessary (and can be detrimental) to evaluate system summaries, and our results demonstrate that SMART is better equipped than other metrics.[Table]
Revised: This entails that using reference summaries may not be necessary (and can be detrimental) to evaluate system summaries, and our results demonstrate that SMART is better equipped than other metrics.

Original: A. SMART FOR A SSESSING F ACTUAL C ONSISTENCY
Revised: A. SMART FOR A SSESSING F ACTUAL C ONSISTENCY

Original: Table 7 reports the ROC AUC scores of different individual SMART metrics using three different kinds of matching functions: ANLI, BLEURT, and CHRF.
Revised: Table 8 reports the ROC AUC scores of different individual SMART metrics using three different kinds of matching functions: ANLI, BLEURT, and CHRF.

Original: Overall, SMART with ANLI performs signiﬁcantly better than the other matching functions.
Revised: Overall, SMART with ANLI performs signiﬁcantly better than the other matching functions.

Original: This is expected since ANLI produces scores that are extremely polarized (i.e., close to 0 or 1) even for slightly nonfactual pairs, which is more appropriate for this task.
Revised: This is expected since ANLI produces scores that are extremely polarized (i.e., close to 0 or 1) even for slightly nonfactual pairs, which is more appropriate for this task.

Original: BLEURT and CHRF, on the other hand, produce partial scores for such pairs, which make them more appropriate in ranking systems, such as in SummEval. 
Revised: BLEURT and CHRF, on the other hand, produce partial scores for such pairs, which make them more appropriate in ranking systems, such as in SummEval. [Table]