Original: JAUNE:
Revised: JAUNE:

Original: J USTIFIED
Revised: J USTIFIED

Original: A ND
Revised: A ND

Original: U NIFIED
Revised: U NIFIED

Original: N EURAL
Revised: N EURAL

Original: LANGUAGE E VALUATION
Revised: LANGUAGE E VALUATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We review the limitations of BLEU and ROUGE – the most popular metrics used to assess reference summaries against hypothesis summaries, and introduce
Revised: We review the limitations of BLEU and ROUGE – the most popular metrics used to assess reference summaries against hypothesis summaries, and introduce

Original: JAUNE: a set of criteria for what a good metric should behave like and propose concrete ways to use recent Transformers-based Language Models to assess reference summaries against hypothesis summaries.
Revised: JAUNE: a set of criteria for what a good metric should behave like and propose concrete ways to use recent Transformers-based Language Models to assess reference summaries/translations against hypothesis summaries/translations.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Evaluation metrics play a central role in the machine learning community.
Revised: Evaluation metrics play a central role in the machine learning community.

Original: They directthe efforts of the research community andare used to deﬁne the state of the art models.
Revised: They direct research efforts and deﬁne the state of the art models.

Original: In machine translation and summarization, the two most common metrics used for evaluating similarity between candidate and reference texts are BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004).
Revised: In machine translation and summarization, the two most common metrics used for evaluating similarity between candidate and reference texts are BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004).

Original: Both approaches rely on counting the matching n-grams in the candidates summary to n-grams in the reference text.
Revised: Both approaches rely on counting the matching n-grams in the candidate text to n-grams in the reference text.

Original: BLEU is precision focused while ROUGE is recall focused.
Revised: BLEU is precision focused while ROUGE is recall focused.

Original: These metrics have posed serious limitations and have already been criticized by the academic community (Reiter, 2018)
Revised: These metrics have posed serious limitations and have already been criticized by the academic community (Reiter, 2018)

Original: (Callison-Burch et al., 2006) (Sulem et al., 2018)
Revised: (Callison-Burch et al., 2006) (Sulem et al., 2018)

Original: (Novikova et al., 2017).
Revised: (Novikova et al., 2017).

Original: In this work, we formulate an empirical criticism of BLEU and ROUGE, establish JAUNE: a set of criteria that a sound evaluation metric should pass to justify its working and propose concrete ways to use recent advances in NLP to design data-driven metric addressing the weaknesses found in BLEU and ROUGE and scoring high on the criteria for a sound evaluation metric.
Revised: In this work, we formulate an empirical criticism of BLEU and ROUGE, establish JAUNE: a set of criteria that a sound evaluation metric should pass. Furthermore we propose concrete ways to use recent advances in NLP to design data-driven metrics addressing the weaknesses found in BLEU and ROUGE while scoring high on the criteria for a sound evaluation metric.

Original:  R ELATED W ORK 2. BLEU, ROUGE AND N - GRAM MATCHING APPROACHES
Revised:  R ELATED W ORK 2. BLEU, ROUGE AND N - GRAM MATCHING APPROACHES

Original: BLEU (Bilingual Evaluation Understudy) (Papineni et al., 2002) and ROUGE (Recall-Oriented Understudy for Gisting Evaluation) (Lin, 2004) have been used to evaluate many NLP tasks for almost two decades.
Revised: BLEU (Bilingual Evaluation Understudy) (Papineni et al., 2002) and ROUGE (Recall-Oriented Understudy for Gisting Evaluation) (Lin, 2004) have been used to evaluate many NLP tasks for almost two decades.

Original: The general acceptance of these methods depend on many factors including their simplicity and intuitive interpretability.
Revised: The general acceptance of these methods depend on many factors including their simplicity and intuitive interpretability.

Original: Moreover, the main factor is the claim that they highly correlate with human judgement (Papineni et al., 2002).
Revised: Moreover, the main factor is the claim that they highly correlate with human judgement (Papineni et al., 2002).

Original: The shortcomings of these methods have been widely criticised and studied.
Revised: The shortcomings of these methods have been widely criticised and studied.

Original: Reiter (Reiter, 2018), in his structured review of BLEU, ﬁnds a low correlation between BLEU and human judgment.
Revised: Reiter (Reiter, 2018), in his structured review of BLEU, ﬁnds a low correlation between BLEU and human judgment.

Original: Callison et al (Callison-Burch et al., 2006) examines BLEU in the context of machine translation and ﬁnd that BLEU neither correlate with human judgment on adequacy(whether the hypothesis sentence adequately captures the meaning of the reference sentence) nor on ﬂuency(the quality of language in a sentence).
Revised: Callison et al (Callison-Burch et al., 2006) examine BLEU in the context of machine translation and ﬁnd that BLEU neither correlates with human judgment on adequacy (whether the hypothesis sentence adequately captures the meaning of the reference sentence) nor on ﬂuency(the quality of language in a sentence).

Original: Sulem et al (Sulem et al., 2018) examine BLEU – in the context of text simpliﬁcation – on grammaticality, meaning preservation and simplicity, and report a very low, and in some cases, negative correlation with human judgment.
Revised: Sulem et al (Sulem et al., 2018) examine BLEU – in the context of text simpliﬁcation – on grammaticality, meaning preservation and simplicity. They report a very low, and, in some cases, negative correlation with human judgment.

Original: Considering these results it is a natural step to pursue new avenues for natural language evaluation and with the advent of deep learning using neural networks for this task is a promising step forward.
Revised: Considering these results, it is a natural step to pursue new avenues for natural language evaluation and, with the advent of deep learning, using neural networks for this task is a promising step forward.

Original: 2. T RANSFORMERS , BERT AND GPT
Revised: 2. T RANSFORMERS , BERT AND GPT

Original: Language modeling has become an important NLP technique thanks to the ability to apply it to various NLP tasks as explained in Radford 
Revised: Language modeling has become an important NLP technique, thanks to the ability to apply it to various NLP tasks as explained in Radford et al

Original: et al (Radford et al., 2019).
Revised: (Radford et al., 2019).

Original: There are two leading architectures for language modeling Recurrent Neural Networks (RNNs)(Mikolov et al., 2010) and
Revised: There are two leading architectures for language modeling: Recurrent Neural Networks (RNNs)(Mikolov et al., 2010) and

Original: Transformers (Vaswani et al., 2017) .
Revised: Transformers (Vaswani et al., 2017).

Original: RNNs handle the input tokens, words or characters, one by one through time to learn the relationship between them, whereas, transformers receive a segment of tokens and learn the dependencies between them using an attention mechanism.
Revised: RNNs handle the input tokens, words or characters, one by one through time to learn the relationship between them, whereas, transformers receive a segment of tokens and learn the dependencies between them using an attention mechanism.

Original: 2. M ODEL - BASED METRICS
Revised: 2. M ODEL - BASED METRICS

Original: While BLEU and ROUGE are deﬁned in a discrete space new evaluation metric can be deﬁned in this continuous space.
Revised: While BLEU and ROUGE are deﬁned in a discrete space new evaluation metric can be deﬁned in this continuous space.

Original: BERTscore (Zhang et al., 2019) uses word embeddings and cosine similarity to create a score array and use greedy matching to maximize the similarity score.
Revised: BERTscore (Zhang et al., 2019) uses word embeddings and cosine similarity to create a score array and uses greedy matching to maximize the similarity score.

Original: Sentence Mover’s
Revised: Sentence Mover’s

Original: Similarity (Clark et al., 2019) uses the mover similarity, Wasserstein distance, between sentence embedding generated from averaging the word embeddings in a sentence.
Revised: Similarity (Clark et al., 2019) uses the mover similarity, Wasserstein distance, between sentence embedding generated from averaging the word embeddings in a sentence.

Original: Both of these methods report stronger correlations with human judgment and better results when compared to BLEU and ROUGE.
Revised: Both of these methods report stronger correlations with human judgment and better results when compared to BLEU and ROUGE.

Original: While they are using word embeddings (Mikolov et al., 2013) to transfer their sentence in a continuous space they are still using distance metrics to evaluate that sentence.
Revised: While they are using word embeddings (Mikolov et al., 2013) to transfer their sentence in a continuous space they are still using distance metrics to evaluate that sentence.

Original: While BLEND (Ma et al., 2017) uses an SVM to combine different existing evaluation metrics.
Revised: While BLEND (Ma et al., 2017) uses an SVM to combine different existing evaluation metrics.

Original: One other evaluation methodproposed is RUSE (Shimanaka et al., 2018) this method proposes embedding both sentences separately and pooling them to a given size.
Revised: Another proposed evaluation method is RUSE (Shimanaka et al., 2018) this method embeds both sentences separately and pool them to a given size.

Original: After that they use a pre trained MLP to predict on different tasks.
Revised: After the method uses a pre-trained MLP to predict on different tasks.

Original: This quality estimator metric is then proposed to be used in language evaluation.
Revised: This quality estimator metric is then proposed to be used in language evaluation.

Original: Our proposed methodology is to take neural language evaluation beyond architecture speciﬁcations.
Revised: Our proposed methodology is to take neural language evaluation beyond architecture speciﬁcations.

Original: We are proposing a framework in which an evaluators success can be determined.
Revised: We are proposing a framework in which an evaluator’s success can be determined.

Original: 2. GLUE B ENCHMARK
Revised: 2. GLUE B ENCHMARK

Original: The GLUE Benchmark is a tool for evaluating and analyzing the performance of models across a diverse range of existing NLU tasks (Wang et al., 2018).
Revised: The GLUE Benchmark is a tool for evaluating and analyzing the performance of models across a diverse range of existing NLU tasks (Wang et al., 2018).

Original: The recent introduction of this benchmark has catalyzed the development of architectures scoring well on a wide variety of tasks and encouraged the NLP community to move away from specialized models doing well on a single task to models performing well across benchmarks.
Revised: The recent introduction of this benchmark has catalyzed the development of architectures scoring well on a wide variety of tasks and encouraged the NLP community to move away from specialized models doing well on a single task to models performing well across benchmarks.

Original: The variety of tasks introduced in the GLUE Benchmark are linguistic acceptability, sentiment analysis, semantic similarity, question answering, logical inference and reading comprehension.
Revised: The variety of tasks introduced in the GLUE Benchmark are linguistic acceptability, sentiment analysis, semantic similarity, question answering, logical inference and reading comprehension.

Original: To be assessed according to that benchmark, models such as
Revised: To be assessed according to that benchmark, models such as

Original: Transformers are usually pre-trained on a large corpus in an unsupervised manner and ﬁne-tuned on a dataset used for the speciﬁc task of the benchmark.
Revised: Transformers are usually pre-trained on a large corpus in an unsupervised manner and ﬁne-tuned on a dataset used for the speciﬁc task of the benchmark.

Original:  C HALLENGES WITH BLEU AND ROUGE
Revised:  C HALLENGES WITH BLEU AND ROUGE

Original: In this part, we will discuss the limitations of BLEU and ROUGE.
Revised: In this part, we will discuss the limitations of BLEU and ROUGE.

Original: There are simple ways to attack these n-gram based metrics like adding a single word negation or changing all possible words with synonyms.
Revised: There are simple ways to attack these n-gram based metrics like adding a single word negation or changing all possible words with synonyms.

Original: Although these are theoretically plausible scenarios we also wanted to analyze which cases forced these metrics to fail in real life.
Revised: Although these are theoretically plausible scenarios we also wanted to analyze which cases forced these metrics to fail in real life.

Original: We took 100 examples from the STS-B dataset (Cer et al., 2017) where the absolute difference between the BLEU/ROUGE score and normalized label was the biggest.
Revised: We took 100 examples from the STS-B dataset (Cer et al., 2017) where the absolute difference between the BLEU/ROUGE score and normalized label was the biggest.

Original: This does not necessarily capture all failure cases of BLEU/ROUGE but a variety of failure cases can be observed.
Revised: This does not necessarily capture all failure cases of BLEU/ROUGE but a variety of failure cases can be observed.

Original: Through this analysis, we see that there are systematically recurring real life examples, just like in our theo- retical examples, where BLEU and ROUGE are failing to assess the level of similarity between two sentences.
Revised: Through this analysis, we see that there are systematically recurring real life examples, just like in our theo- retical examples, where BLEU and ROUGE are failing to assess the level of similarity between two sentences.

Original: We also observe that though some of the shortcomings of unigram metrics are mitigated through higher order n-grams, they open the door for different problems.
Revised: We also observe that though some of the shortcomings of unigram metrics are mitigated through higher order n-grams, they open the door for different problems.

Original: Some of the most common failure cases that we have encountered are listed in table 1.
Revised: Some of the most common failure cases that we have encountered are listed in table 1.

Original: 3. I DIOMS AND ADDING DETAILS
Revised: 3. I DIOMS AND ADDING DETAILS

Original: One quite common problem we have is when idioms are used or extra examples/detailsare given in one of the sentences.
Revised: A commonly encountered failure mode involves the use of idioms and the addition of extra examples/details in one of the sentences.

Original: These types of errors are especially common in more natural conversations.
Revised: These types of errors are especially common in more natural conversations.

Original: These types of errors also made 25 % of our analysis.
Revised: These types of errors also made 25 % of our analysis.

Original: Here we characteristically see humans giving high scores to these sentences because they are aware of which part holds the core meaning of the sentence where BLEU/ROUGE lack this ability.
Revised: Here we characteristically see humans giving high scores to these sentences because they are aware of which part holds the core meaning of the sentence while BLEU/ROUGE lack this ability.

Original: One example from the dataset is “You should take this animal to a vet right away.”
Revised: An example from the dataset is “You should take this animal to a vet right away.”

Original: and “As covered in the other answers, your only option is to see a vet in order to have surgery done.”
Revised: and “As covered in the other answers, your only option is to see a vet in order to have surgery done.”

Original: while the true score for this sentence pair is 3.6 out of 5 our BLEU score is 0.46 out of 5.
Revised: while the true score for this sentence pair is 3.6 out of 5, BLEU gives this sentence a 0.46 out of 5.

Original: 3. C HANGING WORDS
Revised: 3. C HANGING WORDS

Original: Anotherof the most common error cases of BLEU and ROUGE is where one or a few important words of a sentence is changed while the rest of the structure is kept the same.There are many examples to this case and the common thing is we see the sentence structure preserved but words changed to alter the meaning remarkably.
Revised: Another common failure mode of BLEU and ROUGE occurs when one or a few important words of a sentence is changed while the rest of the structure is kept the same.

Original: One example from STS-B is “a man is speaking.”
Revised: An example from STS-B is “a man is speaking.”

Original: and “a man is spitting.”
Revised: and “a man is spitting.”

Original: while human judges give these two sentences a similarity score of 0.64 out of 5 our BLEU-1 score is a 3.75 out of 5.
Revised: while human judges give these two sentences a similarity score of 0.64 out of 5, BLEU-1 gives this sentence a 3.75 out of 5.

Original: 3. G ENERAL PARAPHRASE
Revised: 3. G ENERAL PARAPHRASE

Original: Wealso frequently see BLEU and ROUGE failing with general paraphrases which both include the above mentioned changes but also extend to words being replaced by synonyms, 12 % of cases, reordering of sub-sentences, around 10%, using different tenses or even simple spelling errors or differences, around 15%.
Revised: We frequently see BLEU and ROUGE failing in the context of: • General paraphrases where words are replaced by synonyms. This accounts for 12 % of cases. • Sentences are reordered.

Original: These are not adversarial methods that are formulated to exploit any evaluation metric but direct results of the very nature of language.
Revised: This accounts for around 10% of observed errors. • Different verb tenses are used. This accounts for 15% of observed errors.

Original: While the higher order n-grams are supposed to preserve the intelligibility of the sentence and not reward a model that outputs words in a random order they also punish valid reorderings of sub sentences or words.
Revised: While the higher order n-grams are supposed to preserve the intelligibility of the sentence and not reward a model that outputs words in a random order they also punish valid re-orderings of sub sentences or words.

Original: In these smoothed methods changing a word with a synonym will also result in a much higher penalty.
Revised: In these smoothed methods, changing a word with a synonym will also result in a much higher penalty.

Original: This higher penalty is supposed to work as a layer of protection for when only a single word is changed that shift the meaning of the sentence completely but this also results in a low similarity score if the meaning of the sentence stays exactly the same.

Original: Since this metric is oblivious to the meaning of the word that replaces the former let alone having a sense of context for this word. BLEU and ROUGE are methods that are much more frequently under scoring sentence pairs than over scoring them hence similar to Reiter (Reiter, 2018) we conclude that BLEU/ROUGE can be fruitful in deciding whether a model is bad but not whetherif it is good.
Revised: BLEU and ROUGE are methods that are much more frequently under scoring sentence pairs than over scoring them. Similarly to Reiter (Reiter, 2018) we conclude that BLEU/ROUGE can be fruitful in deciding whether a model is bad but not whether it is good.

Original: While this analysis has been conducted in more detail the complete extend is beyond the purpose of this paper.

Original: The examples given above are fairly enough to show the main type of errors BLEU/ROUGE are facing and why they are falling short in the very task they were designed to do.

Original: We also provide detailed examples of these failure cases in the appendix.
Revised: We also provide detailed examples of these failure cases in the appendix.

Original: 3. E XPERIMENTS 3.4. S OME EXAMPLES FROM STS-B SENTENCES PAIRS
Revised: 3. E XPERIMENTS 3.4. S OME EXAMPLES FROM STS-B SENTENCES PAIRS

Original: To illustrate our argument, we will give some examples from the dataset with their BLEU/ROUGE scores as well as a score generated from a RoBERTa model ﬁne tuned on the STS-B dataset. 
Revised: To illustrate our argument, we will give some examples from the dataset with their BLEU/ROUGE scores as well as a score generated from a RoBERTa model ﬁne tuned on the STS-B dataset. We refer to this model as RoBERTa-STS.

Original: Note that in this paper the BLEU* and ROUGE* scores are not between 0 and 1 but are scaled with  to be more understandable with the scale of the scoring metric used in the dataset.
Revised: Note that in this paper the BLEU* and ROUGE* scores are not between 0 and 1 but are scaled with 5 to be more understandable with the scale of the scoring metric used in the dataset.

Original: That is why we refer to them as BLEU* and ROUGE*
Revised: That is why we refer to them as BLEU* and ROUGE*

Original: [Table]
Revised: [Table]

Original: The last time the survey was conducted, in 1995, those numbers matched. In 1995, the last survey, those numbers were equal.
Revised: The last time the survey was conducted, in 1995, those numbers matched. In 1995, the last survey, those numbers were equal.

Original: [Table]
Revised: [Table]

Original: A band is performing on a stage.
Revised: A band is performing on a stage.

Original: A band is playing onstage.
Revised: A band is playing onstage.

Original: [Table]
Revised: [Table]

Original: Two white dogs are swimming in the water.
Revised: Two white dogs are swimming in the water.

Original: The birds are swimming in the water.
Revised: The birds are swimming in the water.

Original: [Table]
Revised: [Table]

Original: A man plays the piano. A man is playing a piano.
Revised: A man plays the piano. A man is playing a piano.

Original: [Table]
Revised: [Table]

Original: Pardon the brevity of this answer, but I would say ”named” is preferred within the context of your example.
Revised: Pardon the brevity of this answer, but I would say ”named” is preferred within the context of your example.

Original: Named is preferred in your example, since you are formally giving a name to your method.
Revised: Namedis preferred in your example, since you are formally giving a name to your method.

Original: [Table]
Revised: [Table]

Original: In the above section we tried to give a sense of the mechanism of the failure of BLEU and ROUGE but in order to get a comprehensive view will also look at the general performance of these scoring methods.
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Revised: S EMANTIC SIMILARITY EXPERIMENTS

Original: In ﬁgure 1 we can see that development set scores of RoBERTa in ﬁgure 1a and BLEU* in ﬁgurue 1b compared to the gold labels.
Revised: In ﬁgure 1 we can see that development set scores of RoBERTa-STS in ﬁgure 1a and BLEU* in ﬁgure 1b compared to the gold labels.

Original: While we will look at the correlation scores we can also see that the average error in BLEU and RoBERTa are remarkably different.This also clearly shows the difference in robustness between these two models and also emphasises that the failure cases we have given above are not exceptions but on the contrary are quite frequent.
Revised: While we will look at the correlation scores, we can also see that the average errors in BLEU and RoBERTa-STS are remarkably different.

Original: [Table]
Revised: [Table]

Original: In table 2 we also look at the Pearson correlation between human judgment and different evaluators in the STS-B challenge development set.  T OWARDS A ROBUST , DATA - DRIVEN EVALUATION APPROACH 4. M ETRIC S CORECARD In our methodology to design new evaluation metrics for comparing reference summaries/translations to hypothesis ones, we established ﬁrst-principles criteria on what a good evaluator should do. Key contributions of this section include separating the criteria of a good evaluator from the implementation of those criteria and moving away from uni-dimensional, human-designed evaluators towards multi-dimensional, model-based evaluators.
Revised: [Table]

Original: Most metrics currently used have a stated goal of automatically determining the quality of a summary but fail to exactly mention the components of a high quality summary. This ambiguity contributes to these metrics not being robust as heuristic modiﬁcations made to ,for example, account for bi-grams and trigrams overlap fail to take into account other syntactic modiﬁcations such as changing a sentence from the active voice to the passive voice. Even a data-driven evaluator such as a BERT semantic similarity score can be attacked if fed sentences which do not make grammatical sense but are assigned a higher than expected similarity score.
Revised: [Table]

Original:  For example, let’s consider sentences s1: “A man is carrying a canoe with a dog” and s2 “A dog is carrying a man in a canoe with”. BERT-based similarity scores give s1 and s2 a similarity score of 5 whereas sentence s2 does not make grammatical sense. To account for the ﬂaws of any particular dimension and the evaluators only assessing for that dimension, we propose a modular, multi-dimensional criteria. The advantage of breaking down the deﬁnition of a high quality summary into modular criteria is that it simpliﬁes the assessment of evaluation metrics and the models used to score each dimension can be recomposed and combined back into a single metric.
Revised:  A SSESSING E VALUATION M ETRICS 4. M ETRIC S CORECARD

Original: The ﬁrst criteria of a good evaluation metric to compare reference translations/summaries against hypothesis summary is that it should be highly correlated with human judgement of semantic similarity. These automatic metrics were ﬁrst designed because human evaluation of summary/translation is expensive and all state having a high correlation with human judgement of similarity as an objective (Lin, 2004) (Papineni et al., 2002)
Revised: To overcome the previously highlighted challenges and provide a framework in which metrics comparing reference summaries/translation can be assessed and improved, we establish ﬁrst-principles criteria on what a good evaluator should do:

Original: (Denkowski & Lavie, 2011). As a complement to semantic similarity, having an evaluator able to distinguish linguistically similar sentences which are in logical contradiction will make it robust against ﬂaws observed in previous section where sentences can have a high number of shared words but a few words which dramatically change their meaning as in the “NOT” attack case. The third criteria comes from the observation that solely optimizing for BLEU/ROUGE has led to models able to achieve a high score without necessarily generating legible sentences (Paulus et al., 2017). As also shown above, solely optimizing for semantic similarity according to a model can also still lead to producing sentences which do not make grammatical sense.
Revised: • The ﬁrst one is that it should be highly correlated with human judgement of semantic similarity.

Original: Having an evaluator able to penalize and identify hypothesis summaries which do not make grammatical sense will make evaluation robust against this ﬂaw. A fourth criteria of robust evaluators is that they should be difﬁcult to game. That is, they should assign a high score to a reference summary against a hypothesis summary if, and only if these summaries have similar meaning and should assign a low score to a reference summary if, and only if, those summaries/translation have dissimilar meaning. Although obvious, we have empirically and theoretically shown that ROUGE and BLEU do not currently ﬁt this criteria.
Revised: • The second one is that it should be able to distinguish sentences which are in logical contradiction, logically unrelated or in logical agreement.

Original: We envision two primary use cases for the metric scorecard. It can serve as a benchmark to objectively assess and compare evaluation metrics such as ROUGE, BLEU, METEOR. It can also be used to leverage recent NLP models todesign evaluation metrics as we explore in the next section.
Revised: • The third one is that given s1, s2 which are semantically similar, eval(s1,s2) ≥ eval(s1,s2(corrupted) ≥ eval(s1,s2(more corrupted)) where corruption here includes removing words, adding noise to the word order or including grammatical mistakes.

Original: 4. I MPLEMENTING MODEL - BASED EVALUATORS SATISFYING SCORECARD
Revised: 4. I MPLEMENTING THE METRIC SCORECARD

Original: After presenting the motivation for our multidimensional scorecard, we nowproceed to show how it can be used to design summary evaluators. The scenario is that we have a reference summary s and a hypothesis summary s2.
Revised: We will now show how the scorecard can be implemented. For every dimension of the scorecard the experiments are done with three metrics.

Original: We want to know whether s2 is a high quality summary similar to s1. Until now, that deﬁnition was vague and mainly captured by the BLEU and ROUGE score as evaluators. In the previous section, we established 4 criteria for a good evaluators. 3 of those criteria can be used to assess how close s2 is to s1 given that the fourth criteria (robustness) is a property of the evaluator itself.
Revised: BLEU with equal weights between 1 to 4 grams.

Original: The 3 criteria we have identiﬁed which are applicable to designing model-based evaluators include:

Original: • Eval(s1,s2) should have one dimension assessing the semantic similarity of s1 and s2.
Revised: ROUGE with averaging ROUGE-1 and ROUGE-2 and a neural evaluator.

Original: This dimension should have high correlation with human judgement of similarity • Eval(s1,s2) should have one dimension able to identify if s1 and s2 are in contradiction, unrelated or agreement. • Eval(s2) should be able to identify is s2 makes grammatical sense and is not a gibberish sentence. For these three criteria, we will show how we can use recent advances in language modelling to design evaluators which are either signiﬁcantly outperforming BLEU/ROUGE on these dimensions or adding a criteria that BLEU and ROUGE could not previously control for when assessing hypothesis summaries.
Revised: The evaluator is the RoBERTaSTS.

Original: 4.2. S EMANTIC S IMILARITY
Revised: 4.2. S EMANTIC S IMILARITY

Original: The ﬁrst criteria of a good evaluator is that it should have a high correlation with human judgement of  semantic similarity.
Revised: The ﬁrst expectation from a good similarity metric is to correlate highly with human judgment in terms of assessing semantic similarity.

Original: To develop an evaluator corresponding to that dimension, we picked the best performing model on a semantic similarity task of the GLUE benchmark as of the writing of this paper. Starting from the RoBERTa large pre-trained model (Liu et al., 2019), we ﬁnetune it to predict sentence similarity on the STS-B benchmarkdataset (Cer et al., 2017). Given two sentences of text, s1 and s2, the system needs tocompute how similar s1 and s2 are and returns a  similarity score between 0 and 5. The dataset comprises naturally occurring pairs of sentences drawn from several domains and genres, annotated by crowdsourcing.
Revised: Here we assessed BLEU and ROUGE on the STS-B benchmark and compared their performance to a RoBERTa model ﬁne tuned for semantic similarity (Table 3).

Original: The benchmark comprises 8628 sentence pairs with 5700 pairs in the training set, 1500 in the development set and 1379 in the test set.

Original: 4.2. L OGICAL E QUIVALENCE
Revised: 4.2. L OGICAL E NTAILMENT

Original: The second criteria of a good evaluator isthat it should be able to accurately detect whether s1 is in contradiction with s2 or not. To develop an evaluator corresponding to that dimension, we picked the best performing model on a logical inference task of the GLUE benchmark as of the writing of this paper.
Revised: Another characteristic of a good metric is to differentiate the argument, core meaning, in a sentence and take it into account when assessing hypothesis text with references.

Original: For logical inference, we start with a pretrained RoBERTa model and ﬁnetune it using the MultiGenre Natural Language Inference Corpus (Nangia et al., 2017) . It is a crowdsourced collection of sentence pairs with textual entailment annotations.
Revised: Here we used the MNLI dataset where for each text we have three hypothesis text representing contradiction, neutral and entailment.

Original: Given a premise sentence and a hypothesis sentence, the task is to predict whether the premise entails the hypothesis, contradicts the hypothesis, or neither (neutral).
Revised: We expect a good metric to rank entailment higher than neutral, and both of them higher than contradiction.

Original: The training set includes 393k sentence pairs, development set includes 20k and test set includes 20k.
Revised: To assess the quality of a metric, we propose to use the Spearman’s ranked correlation and in Table 5 we also experiment with Kendall’s τ .

Original: The accuracy of the pre-trained model on the development set is 0.9060 .
Revised: Here we observe that the RoBERTa-STS model remarkably outperforms BLEU and ROUGE and both of these metrics show very little correlation with human judgment.

Original: 4.2. S ENTENCE I NTELLIGIBILITY
Revised: 4.2. R OBUSTNESS TO G RAMMATICAL E RRORS

Original: The third criteria of a good evaluator is that it should be able to detect sentences which do not make grammatical sense.
Revised: For assessing the third criteria.

Original: To develop an evaluator corresponding to that dimension, we explored two possibilities: directly using a state of the art language model or using an architecture ﬁne tuned on a task such as linguistic acceptability. For a language model based evaluator of sentence intelligibility, we can use an architecture such as 
Revised: We start with 3479 sentence pairs from the MNLI dataset that are labelled as entailment.

Original: GPT-2 (Radford et al., 2018) and use the perplexity score. The rationale for using the perplexity score of a large scale language models is that one way to frame a sentence which is not intelligible is that the order of words is surprising to a reader.
Revised: We introduce random corruptions such as random insertion, deletion and grammatical errors as in (Zhao et al., 2019). We use two different set of parameters for different

Original: Being surprising to a reader means having words not expected.
Revised: [Table]

Original: Since language models trained to predict the next word maintain a probability distribution over an upcoming word given previous words, an unlikely word would surprise the language model. This general insight is captured by the perplexity score assigned by a language model to a sentence.
Revised: [Table]

Original: The current challenge is thatperplexity scores alone can not inform whether a sentence makes grammatical sense. Assuming that the reference sentence has a correct syntax, one way of implementing the intelligibility criteria can be comparing the relative perplexity score of reference and hypothesis summaries. For our ﬁne tuned model, we start with a pretrained RoBERTa model and ﬁnetune it using the 
Revised: corruption levels and expect that a good metric would rank the original similar sentence higher than the less corrupted and both higher than the more corrupted sentence. Here we also propose to use the Spearman’s ranked correlation and also experiment with Kendall’s τ .

Original: Corpus of Linguistic Acceptability (CoLA) . It consists of examples of expert English sentence acceptability judgments drawn from 22 books. Each example is a single string of English words annotated with whether it is grammatically possible sentence of English. The training set for CoLA has 10k sentences and the development set includes 1k sentences.
Revised: We report results on Table 5.

Original: The current model gets 67. percent accuracy on the test set showing that this is still an active area of research.
Revised: Where we see that the RoBERTa-STS model once more outperforms BLEU and ROUGE.

Original: 4.2. R ATIONALE FOR L ANGUAGE  M ODELS
Revised:  M ACHINE T RANSLATION E XPERIMENTS

Original: The overall rationale for using Transformer-based language models ﬁne tuned for speciﬁc aspects of the scorecard is that recent work has shown that language models are unsupervised multi task learners (Radford et al., 2019) and can rediscover the classical NLP pipeline (Tenney et al., 2019). More speciﬁcally, in these models, there are localizable regions associated with distinct types of linguistic decisions suggesting that they can directly encode a range of syntactic and semantic information. Furthermore, the information is encoded in a natural progression POS tags processed earliest, followed by constituents, dependencies, semantic roles, and coreference.
Revised: In the previous section, we outlined a series of criteria to assess evaluation metrics and shown how for each dimension, RoBERTa-STS signiﬁcantly outperformed ROUGE and BLEU.

Original: That is, it appears that basic syntactic information appears earlier in the network, while high-level semantic information appears at higher layers. By taking these pre-trained models and ﬁne tuning them on a speciﬁc task, we make them pay attention to the correct level of abstraction corresponding to the scorecard.
Revised: In the following section, we report results showing how BLEU, ROUGE and RoBERTa-STS correlate with human judgement of quality in the case of machine translation. Using the WMT2015 and WMT2016 datasets, we collected reference and hypothesis translations along with human judgment scores where the translation was from another language to English. This gave us 5360 sentence pairs.

Original: 4.2. C OMPOSING DIMENSIONS OF AN EVALUATOR
Revised: For each of these sentence pairs, we computed the BLEU, ROUGE-1, ROUGE-2, ROUGE-L scores. We also computed the RoBERTa-STS predicted semantic similarity.

Original: In this section, we have shown how we could use state of the art language models to implement our evaluator. As long as the evaluators output a numeric score for each criteria, they can be aggregated through a weighted sum or other composition function into an aggregate score that can be used to summarize the quality of a hypothesis summary and comparing it to a reference summary.
Revised: We performed a correlation of the various scores with human judgements and report the scores in Table 6. Here again, we can see Transformers based model outperforming BLEU and ROUGE to assess the quality of machine translation.

Original:  C ONCLUSION AND FUTURE WORK
Revised:  C ONCLUSION AND FUTURE WORK

Original: In this work, we have shown three main limitations of BLEU and ROUGE and proposed a path forward outlining why and how state of the art language models can be used as summary evaluators.
Revised: In this work, we have established a framework to assess metrics comparing the quality of reference and hypothesis summary/translations.

Original: While Transformers are currently the best performing architecture on the GLUE benchmark and have been used to implement each dimension of our evaluators, the framework and approach is independent current models and the family of evaluators it generates is meant to evolve with the ﬁeld. This work opens at least four follow up questions which can be explored in future work. Firstly, the proposed dimensions of the scorecard may be redundant and/or incomplete. Secondly, assuming a ﬁxed scorecard, there might be better ways to robustly implement an evaluator assessing that criteria.
Revised: Based on these criteria, we compare evaluators using recent

Original: For example, instead of using one model, ensembling approach may enable to overcome weaknesses of each ﬁne tuned model.

Original: Thirdly, assessing published summarization models using that scorecard is a next step and would provide another analysis of the state of the art. Finally, ﬁnding out whether the model-based evaluators can be used to design an objective function to optimize for and generate good summaries is also an open research question. An interactive or reinforcement learning scenario in which a generator takes as input a text, produces a summary and get feedback by the evaluators would be interesting to explore. Open ended questions include both the convergence of such a training procedure and ,assuming convergence, whether the summaries produced by such a procedure will be deemed high quality by human evaluators.
Revised: Transformers (in this case RoBERTa-STS) to BLEU and ROUGE and highlight their potential to replace BLEU and ROUGE

Original: A A PPENDIX
Revised: A A PPENDIX

Original: In the appendix we will discuss the failure cases of BLEU, ROUGE and RoBERTa in detail to provide a better understanding of how these models can fall short in language evaluation.
Revised: In the appendix we will discuss the failure cases of BLEU, ROUGE and RoBERTa-STSin detail to provide a better understanding of how these models can fall short in language evaluation.

Original: This is important because a good metric scorecard has to represent the quality of an evaluator and this experiments are to show that our metrics cover many of the failure cases and can assess them without the burden of manually evaluating the outputs of every evaluator.
Revised: This is important because a good metric scorecard has to represent the quality of an evaluator and this experiments are to show that our metrics cover many of the failure cases and can assess them without the burden of manually evaluating the outputs of every evaluator.

Original: We will start by taking examples from the BLEU and ROUGE dataset.
Revised: We will start by taking examples from the BLEU and ROUGE dataset.

Original: As in the paper the BLEU scores used are always a uniform average up to 4-grams and the ROUGE score is the average of
Revised: As in the paper the BLEU scores used are always a uniform average up to 4-grams and the ROUGE score is the average of

Original: ROUGE-1 and ROUGE-2.
Revised: ROUGE-1 and ROUGE-2.

Original: Both scores are scaled up to 5 to increase the interpretebility of the scores given that the labels in the similarty dataset are between 0 and 5.
Revised: Both scores are scaled up to 5 to increase the interpretebility of the scores given that the labels in the similarty dataset are between 0 and 5.

Original: [Table]
Revised: [Table]

Original: The company claims it’s the largest single Apple VAR Xserve sale to date.
Revised: The company claims it’s the largest single Apple VAR Xserve sale to date.

Original: The company claimed it is the largest sale of Xserves by an Apple retailer.[Table]
Revised: The company claimed it is the largest sale of Xserves by an Apple retailer.

Original: A woman puts ﬂour on a piece ofmeat.
Revised: A woman puts ﬂour on a piece of

Original:  A woman is putting ﬂour onto some meat.[Table]
Revised: meat. A woman is putting ﬂour onto some meat.

Original: He later learned that the incident was caused by the Concorde’s sonic boom.
Revised: He later learned that the incident was caused by the Concorde’s sonic boom.

Original: He later found out the alarming incident had been caused by Concorde’s powerful sonic boom.
Revised: He later found out the alarming incident had been caused by Concorde’s powerful sonic boom.

Original: [Table] It indeed appears the Andromeda galaxy (M31) and The Milky Way (MW) are en route to a collision.
Revised: It indeed appears the Andromeda galaxy (M31) and The Milky Way (MW) are en route to a collision.

Original: Ina few billion years, the Milky Way and Andromeda will collide.
Revised: Ina few billion years, the Milky Way and Andromeda will collide.

Original: [Table] You deﬁnitely do NOT want to be supporting your weight with your arms on the bike for normal riding.
Revised: You deﬁnitely do NOT want to be supporting your weight with your arms on the bike for normal riding.

Original: No, don’t support your weight on your arms Your hands simply aren’t really made for supporting all that weight.
Revised: No, don’t support your weight on your arms Your hands simply aren’t really made for supporting all that weight.

Original: 2.73/ 4.20/  7 detained for ’house sister’ scandal
Revised: 2.73/ 4.20/  7 detained for ’house sister’ scandal

Original: China detains 7 for ”house sister” scandal 0.
Revised: China detains 7 for ”house sister” scandal 0.

Original: A man plays the violin.
Revised: A man plays the violin.

Original: A man is playing violin.
Revised: A man is playing violin.

Original: It is simply the number of balls bowled divided by the number of wickets taken.
Revised: It is simply the number of balls bowled divided by the number of wickets taken.

Original: Bowling strike rate is deﬁned for a bowler as the average number of balls bowled per wicket taken.
Revised: Bowling strike rate is deﬁned for a bowler as the average number of balls bowled per wicket taken.

Original: Police helicopter crashes into pub in Glasgowseveral casualties
Revised: Police helicopter crashes into pub in Glasgowseveral casualties

Original: Helicopter crashes into roof of Glasgow club 0.
Revised: Helicopter crashes into roof of Glasgow club 0.

Revised: [Table]

Original: Oil falls in Asian trade
Revised: Oil falls in Asian trade

Original: Oil prices down in Asian trade 1.
Revised: Oil prices down in Asian trade [Table] A

Original: A skateboarder jumps off the stairs.
Revised: skateboarder jumps off the stairs

Original: A dog jumps off the stairs.
Revised: A dog jumps off the stairs

Original:  Wigan 3-2 Wolves: Match report, pictures & video highlights
Revised: [Table] [Table] Wigan 3-2 Wolves: Match report, pictures & video highlights

Original: Arsenal 0-0 Chelsea: Match report, pictures & video highlights3.
Revised: Arsenal 0-0 Chelsea: Match report, pictures & video highlights

Original:  In table 3 we see examples of many different error cases and in most sentences we also have more than one cause for the drastic difference between BLEU/ROUGE and the Label.
Revised: [Table] [Table] In table 7 we see examples of many different error cases and in most sentences we also have more than one cause for the drastic difference between BLEU/ROUGE and the Label.

Original: For instance in rows 1 and 6 we see that the cause for the error is the reordering of sub-sentences, spelling/punctuation and newly introduced words that don’t change the meaning but merely extend it.
Revised: For instance in rows 1 and 6 we see that the cause for the error is the reordering of sub-sentences, spelling/punctuation and newly introduced words that don’t change the meaning but merely extend it.

Original: While BLEU and
Revised: While BLEU and

Original: ROUGE are failing in these examples we see that the RoBERTa model scores similar to the label.
Revised: ROUGE are failing in these examples we see that the RoBERTa-STSmodel scores similar to the label.

Original: In line 7 we can see that the RoBERTa model score is above 5.
Revised: In line 7 we can see that the RoBERTa-STSmodel score is above 5.

Original: While this is because the model is designed as a regression model, as expected given the nature of the task.
Revised: While this is because the model is designed as a regression model, as expected given the nature of the task.

Original: In rows 2 and 7 we see that the main difference is the form or tense of the verb in a sentence.
Revised: In rows 2 and 7 we see that the main difference is the form or tense of the verb in a sentence.

Original: While using higher order n-grams enforces the goal of BLEU and ROUGE to not over score randomly shufﬂed sentences or remarkable meaning shift due to minimal changes in words such as in row 11.
Revised: While using higher order n-grams enforces the goal of BLEU and ROUGE to not over score randomly shufﬂed sentences or remarkable meaning shift due to minimal changes in words such as in row 11.

Original: This also makes BLEU severely under score simple changes with synonyms or valid re-orderings as seen in the examples below.
Revised: This also makes BLEU severely under score simple changes with synonyms or valid re-orderings as seen in the examples below.

Original: This characteristic of BLEU reinforces the point that BLEU and
Revised: This characteristic of BLEU reinforces the point that BLEU and

Original: ROUGE are not useful in tracking the state of the art and comparing the best methods but are tools to weed out bad models fairly simply.
Revised: ROUGE are not useful in tracking the state of the art and comparing the best methods but are tools to weed out bad models fairly simply.

Original: In rows 3 and 9 we see sentences that differ due to using descriptive phrases instead of a word or extending the sentence with more information.
Revised: In rows 3 and 9 we see sentences that differ due to using descriptive phrases instead of a word or extending the sentence with more information.

Original: These types of errors changes are also caught with language models since we know they have the ability to hold the meaning of multiple words and incorporate them to reach a related word as in the famous example of king - men + woman =
Revised: These types of errors changes are also caught with language models since we know they have the ability to hold the meaning of multiple words and incorporate them to reach a related word as in the famous example of king - men + woman =

Original: queen [cite word2vec].
Revised: queen [cite word2vec].

Original: In rows 4,8 and 5 we see general paraphrases with the same meaning represented in a generally different sentence.
Revised: In rows 4,8 and 5 we see general paraphrases with the same meaning represented in a generally different sentence.

Original: In all cases we see an drastic difference between BLEU/ROUGE and the label but these cases also unearth a speciﬁc characteristic of the neural evaluator.
Revised: In all cases we see an drastic difference between BLEU/ROUGE and the label but these cases also unearth a speciﬁc characteristic of the neural evaluator.

Original: In 4 and 8 we see that the error of the RoBERTa model comparatively lower than row 5.
Revised: In 4 and 8 we see that the error of the RoBERTa-STS model comparatively lower than row 5.

Original: While it is hard to determine the exact cause through only looking at these examples table 4 for the RoBERTa failure cases will make this case more compelling.
Revised: While it is hard to determine the exact cause through only looking at these examples table 8 for the RoBERTa-STS failure cases will make this case more compelling.

Original: While language models have a general sense of the context in a given sentence they still lack a general knowledge of the world.
Revised: While language models have a general sense of the context in a given sentence they still lack a general knowledge of the world.

Original: Hence in the second sentence of row 5, because the words riding, bike, bicycle are missing the model has a hard time recognising that the second sentence is also about the same topic.
Revised: Hence in the second sentence of row 5, because the words riding, bike, bicycle are missing the model has a hard time recognising that the second sentence is also about the same topic.

Original: To test this we added ”while riding” or ”on a bike” at the end of a sentence and the score immediately went up to 3.6/5 while barely changing the BLEU* and the ROUGE score.
Revised: To test this we added ”while riding” or ”on a bike” at the end of a sentence and the score immediately went up to 3.6/5 while barely changing the BLEU* and the ROUGE score.

Original: In row 4 and 8 however, the context of the sentence is deﬁned explicitly with the key phrases.
Revised: In row 4 and 8 however, the context of the sentence is deﬁned explicitly with the key phrases.

Original: We see this bias affecting RoBERTa scoring in the examples below.
Revised: We see this bias affecting RoBERTa-STS scoring in the examples below.

Original: [Table]It
Revised: [Table]

Original:   would be unusual for a snake to attack a stationary person.I’m no herpetologist, but in
Revised:  It would be unusual for a snake to attack a stationary person.

Original:  my experience , snakes are in the ”you don’t 
Revised: I’m no herpetologist, but in my experience, snakes are in the ”you don’t bug me, I won’t bug you” category.

Original: bug me

Original: , I won’t bug you” category. [Table]  New UN peacekeeping chief named for Central African Republic UN takes over peacekeeping in Central AfricanRepublic
Revised:  New UN peacekeeping chief named for Central African Republic UN takes over peacekeeping in Central African

Original: [Table]
Revised: Republic 1.

Original: From Broadway comedies like”The Seven Year Itch” (1952),
Revised: From Broadway comedies like

Original: ”Will
Revised: The

Original:  Success Spoil Rock Hunter
Revised: Seven Year Itch” (1952), ”Will Success Spoil Rock Hunter

Original: Playwright
Revised: Playwright

Original: George Axelrod, who anticipatedthe sexual revolution
Revised: George Axelrod, who anticipated

Original:  with The Seven
Revised: the sexual revolution with The Seven

Original: Year Itch and
Revised: Year Itch and

Original: Will Success
Revised: Will Success

Original: Spoil Rock Hunter? [Table]
Revised: Spoil Rock Hunter? 2.

Original:  a group of navy seals are singing A group of military personnel are playing in a brass quintet.[Table]
Revised:  a group of navy seals are singing A group of military personnel are playing in a brass quintet.

Original: In the above examples we will ﬁnd a two points that will helps us better understand the RoBERTa as a neural evaluator.
Revised: In the above examples we will ﬁnd a two points that will helps us better understand the RoBERTaSTS as a neural evaluator.

Original: First we see that the neural network sometimes lacks a sense of context that is not given in the sentence explicitly.
Revised: First we see that the neural network sometimes lacks a sense of context that is not given in the sentence explicitly.

Original: While these language models are trained on a large corpus and capture a sense of the words and language we still see that their performance is not perfect.
Revised: While these language models are trained on a large corpus and capture a sense of the words and language we still see that their performance is not perfect.

Original: We see these examples in row 4 where the model cannot relate a navy seal as a military personnel.
Revised: We see these examples in row 4 where the model cannot relate a navy seal as a military personnel.

Original: Or as in row 1 where the model cannot model an idiom.
Revised: Or as in row 1 where the model cannot model an idiom.

Original: The second and more critical place where we need further development is especially detecting whether the core argument/message in a sentence is the same beyond whether if they are talking about the same things.
Revised: The second and more critical place where we need further development is especially detecting whether the core argument/message in a sentence is the same beyond whether if they are talking about the same things.

Original: As in rows 2 and 3.
Revised: As in rows 2 and 3.

Original: We see the same landmark words and can clearly say that the sentences are talking about the same things but what a human can distinguish is that they are saying unrelated things.
Revised: We see the same landmark words and can clearly say that the sentences are talking about the same things but what a human can distinguish is that they are saying unrelated things.

Original: This is one of the key motivations in including the language inference task in the scorecard.
Revised: This is one of the key motivations in including the language inference task in the scorecard.

Original: Since detecting whether a pair of sentences are related on what level is a key part of detecting sentence similarity.
Revised: Since detecting whether a pair of sentences are related on what level is a key part of detecting sentence similarity.

Original: One last thing we will mention is that while RoBERTa and BLEU/ROUGE have different error cases their performance on these error cases is also remarkably different in favor of the former.
Revised: One last thing we will mention is that while RoBERTa-STS and BLEU/ROUGE have different error cases their performance on these error cases is also remarkably different in favor of the former.

Original: Tableshows the mean error of BLEU* and the RoBERTa model on each others top 500, which is one third of the development set, error cases.
Revised: Table 9 shows the mean error of BLEU* and the RoBERTa-STS model on each others top 500, which is one third of the development set, error cases.

Original: [Table]
Revised: [Table]

Original: We see in table 5 that in BLEU* has a remarkable error in both its failure cases and also the failure cases of RoBERTa while RoBERTa outperforms BLEU* in each category.
Revised: We see in table 9 that in BLEU* has a remarkable error in both its failure cases and also the failure cases of RoBERTa-STS while RoBERTa-STS outperforms BLEU* in each category.

Original: While neural evaluators have also room for improvement we can with conﬁdence say that they are outperforming classical methods and with a methodical way of improving them can bolster progress of NLP research.
Revised: While neural evaluators have also room for improvement we can with conﬁdence say that they are outperforming classical methods and with a methodical way of improving them can bolster progress of NLP research.