Original: JAUNE:
Revised: JAUNE:

Original: J USTIFIED
Revised: J USTIFIED

Original: A ND
Revised: A ND

Original: U NIFIED
Revised: U NIFIED

Original: N EURAL
Revised: N EURAL

Original: LANGUAGE E VALUATION
Revised: LANGUAGE E VALUATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We review the limitations of BLEU and ROUGE – the most popular metrics used to assess reference summaries against hypothesis summaries, and introduce
Revised: We review the limitations of BLEU and ROUGE – the most popular metrics used to assess reference summaries against hypothesis summaries, and introduce

Original: JAUNE: a set of criteria for what a good metric should behave like and propose concrete ways to use recent Transformers-based Language Models to assess reference summaries/translations against hypothesis summaries/translations.
Revised: JAUNE: a set of criteria for what a good metric should behave like and propose concrete ways to use recent Transformers-based Language Models to assess reference summaries/translations against hypothesis summaries/translations.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Evaluation metrics play a central role in the machine learning community.
Revised: Evaluation metrics play a central role in the machine learning community.

Original: They direct research efforts and deﬁne the state of the art models.
Revised: They direct research efforts and deﬁne the state of the art models.

Original: In machine translation and summarization, the two most common metrics used for evaluating similarity between candidate and reference texts are BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004).
Revised: In machine translation and summarization, the two most common metrics used for evaluating similarity between candidate and reference texts are BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004).

Original: Both approaches rely on counting the matching n-grams in the candidate text to n-grams in the reference text.
Revised: Both approaches rely on counting the matching n-grams in the candidate text to n-grams in the reference text.

Original: BLEU is precision focused while
Revised: BLEU is precision focused while

Original: ROUGE is recall focused.
Revised: ROUGE is recall focused.

Original: These metrics have posed serious limitations and have already been criticized by the academic community (Reiter, 2018)
Revised: These metrics have posed serious limitations and have already been criticized by the academic community (Reiter, 2018)

Original: (Callison-Burch et al., 2006) (Sulem et al., 2018)
Revised: (Callison-Burch et al., 2006) (Sulem et al., 2018)

Original: (Novikova et al., 2017).
Revised: (Novikova et al., 2017).

Original: In this work, we formulate an empirical criticism of BLEU and ROUGE, establish JAUNE: a set of criteria that a sound evaluation metric should pass.
Revised: In this work, we formulate an empirical criticism of BLEU and ROUGE, establish JAUNE: a set of criteria that a sound evaluation metric should pass.

Original: Furthermore we propose concrete ways to use recent advances in NLP to design data-driven metrics addressing the weaknesses found in BLEU and ROUGE while scoring high on the criteria for a sound evaluation metric.
Revised: Furthermore we propose concrete ways to use recent advances in NLP to design data-driven metrics addressing the weaknesses found in BLEU and ROUGE while scoring high on the criteria for a sound evaluation metric.

Original:  R ELATED W ORK 2. BLEU, ROUGE AND N - GRAM MATCHING APPROACHES
Revised:  R ELATED W ORK 2. BLEU, ROUGE AND N - GRAM MATCHING APPROACHES

Original: BLEU (Bilingual Evaluation Understudy) (Papineni et al., 2002) and ROUGE (Recall-Oriented Understudy for Gisting Evaluation) (Lin, 2004) have been used to evaluate many NLP tasks for almost two decades.
Revised: BLEU (Bilingual Evaluation Understudy) (Papineni et al., 2002) and ROUGE (Recall-Oriented Understudy for Gisting Evaluation) (Lin, 2004) have been used to evaluate many NLP tasks for almost two decades.

Original: The general acceptance of these methods depend on many factors including their simplicity and intuitive interpretability.
Revised: The general acceptance of these methods depend on many factors including their simplicity and intuitive interpretability.

Original: Moreover, the main factor is the claim that they highly correlate with human judgement (Papineni et al., 2002).
Revised: Moreover, the main factor is the claim that they highly correlate with human judgement (Papineni et al., 2002).

Original: The shortcomings of these methods have been widely criticised and studied.
Revised: The shortcomings of these methods have been widely criticised and studied.

Original: Reiter (Reiter, 2018), in his structured review of BLEU, ﬁnds a low correlation between BLEU and human judgment.
Revised: Reiter (Reiter, 2018), in his structured review of BLEU, ﬁnds a low correlation between BLEU and human judgment.

Original: Callison et al (Callison-Burch et al., 2006) examine BLEU in the context of machine translation and ﬁnd that BLEU neither correlates with human judgment on adequacy (whether the hypothesis sentence adequately captures the meaning of the reference sentence) nor on ﬂuency(the quality of language in a sentence).
Revised: Callison et al (Callison-Burch et al., 2006) examine BLEU in the context of machine translation and ﬁnd that BLEU neither correlates with human judgment on adequacy (whether the hypothesis sentence adequately captures the meaning of the reference sentence) nor on ﬂuency(the quality of language in a sentence).

Original: Sulem et al (Sulem et al., 2018) examine BLEU – in the context of text simpliﬁcation – on grammaticality, meaning preservation and simplicity.
Revised: Sulem et al (Sulem et al., 2018) examine BLEU – in the context of text simpliﬁcation – on grammaticality, meaning preservation and simplicity.

Original: They report a very low, and, in some cases, negative correlation with human judgment.
Revised: They report a very low, and, in some cases, negative correlation with human judgment.

Original: Considering these results, it is a natural step to pursue new avenues for natural language evaluation and, with the advent of deep learning, using neural networks for this task is a promising step forward.
Revised: Considering these results, it is a natural step to pursue new avenues for natural language evaluation and, with the advent of deep learning, using neural networks for this task is a promising step forward.

Original: 2. T RANSFORMERS , BERT AND GPT
Revised: 2. T RANSFORMERS , BERT AND GPT

Original: Language modeling has become an important NLP technique, thanks to the ability to apply it to various NLP tasks as explained in Radford et al
Revised: Language modeling has become an important NLP technique, thanks to the ability to apply it to various NLP tasks as explained in Radford et al

Original: (Radford et al., 2019).
Revised: (Radford et al., 2019).

Original: There are two leading architectures for language modeling: Recurrent Neural Networks (RNNs)(Mikolov et al., 2010) and
Revised: There are two leading architectures for language modeling: Recurrent Neural Networks (RNNs)(Mikolov et al., 2010) and

Original: Transformers (Vaswani et al., 2017).
Revised: Transformers (Vaswani et al., 2017).

Original: RNNs handle the input tokens, words or characters, one by one through time to learn the relationship between them, whereas, transformers receive a segment of tokens and learn the dependencies between them using an attention mechanism.
Revised: RNNs handle the input tokens, words or characters, one by one through time to learn the relationship between them, whereas, transformers receive a segment of tokens and learn the dependencies between them using an attention mechanism.

Original: 2. M ODEL - BASED METRICS
Revised: 2. M ODEL - BASED METRICS

Original: While BLEU and ROUGE are deﬁned in a discrete space new evaluation metric can be deﬁned in this continuous space.
Revised: While BLEU and ROUGE are deﬁned in a discrete space new evaluation metric can be deﬁned in this continuous space.

Original: BERTscore (Zhang et al., 2019) uses word embeddings and cosine similarity to create a score array and uses greedy matching to maximize the similarity score.
Revised: BERTscore (Zhang et al., 2019) uses word embeddings and cosine similarity to create a score array and uses greedy matching to maximize the similarity score.

Original: Sentence Mover’s
Revised: Sentence Mover’s

Original: Similarity (Clark et al., 2019) uses the mover similarity, Wasserstein distance, between sentence embedding generated from averaging the word embeddings in a sentence.
Revised: Similarity (Clark et al., 2019) uses the mover similarity, Wasserstein distance, between sentence embedding generated from averaging the word embeddings in a sentence.

Original: Both of these methods report stronger correlations with human judgment and better results when compared to BLEU and ROUGE.
Revised: Both of these methods report stronger correlations with human judgment and better results when compared to BLEU and ROUGE.

Original: While they are using word embeddings (Mikolov et al., 2013) to transfer their sentence in a continuous space they are still using distance metrics to evaluate that sentence.
Revised: While they are using word embeddings (Mikolov et al., 2013) to transfer their sentence in a continuous space they are still using distance metrics to evaluate that sentence.

Original: While BLEND (Ma et al., 2017) uses an SVM to combine different existing evaluation metrics.
Revised: While BLEND (Ma et al., 2017) uses an SVM to combine different existing evaluation metrics.

Original: Another proposed evaluation method is RUSE (Shimanaka et al., 2018) this method embeds both sentences separately and pool them to a given size.
Revised: Another proposed evaluation method is RUSE (Shimanaka et al., 2018) this method embeds both sentences separately and pool them to a given size.

Original: After the method uses a pre-trained MLP to predict on different tasks.
Revised: After the method uses a pre-trained MLP to predict on different tasks.

Original: This quality estimator metric is then proposed to be used in language evaluation.
Revised: This quality estimator metric is then proposed to be used in language evaluation.

Original: Our proposed methodology is to take neural language evaluation beyond architecture speciﬁcations.
Revised: Our proposed methodology is to take neural language evaluation beyond architecture speciﬁcations.

Original: We are proposing a framework in which an evaluator’s success can be determined.
Revised: We are proposing a framework in which an evaluator’s success can be determined.

Original: 2. GLUE B ENCHMARK
Revised: 2. GLUE B ENCHMARK

Original: The GLUE Benchmark is a tool for evaluating and analyzing the performance of models across a diverse range of existing NLU tasks (Wang et al., 2018).
Revised: The GLUE Benchmark is a tool for evaluating and analyzing the performance of models across a diverse range of existing NLU tasks (Wang et al., 2018).

Original: The recent introduction of this benchmark has catalyzed the development of architectures scoring well on a wide variety of tasks and encouraged the NLP community to move away from specialized models doing well on a single task to models performing well across benchmarks.
Revised: The recent introduction of this benchmark has catalyzed the development of architectures scoring well on a wide variety of tasks and encouraged the NLP community to move away from specialized models doing well on a single task to models performing well across benchmarks.

Original: The variety of tasks introduced in the GLUE Benchmark are linguistic acceptability, sentiment analysis, semantic similarity, question answering, logical inference and reading comprehension.
Revised: The variety of tasks introduced in the GLUE Benchmark are linguistic acceptability, sentiment analysis, semantic similarity, question answering, logical inference and reading comprehension.

Original: To be assessed according to that benchmark, models such as
Revised: To be assessed according to that benchmark, models such as

Original: Transformers are usually pre-trained on a large corpus in an unsupervised manner and ﬁne-tuned on a dataset used for the speciﬁc task of the benchmark.
Revised: Transformers are usually pre-trained on a large corpus in an unsupervised manner and ﬁne-tuned on a dataset used for the speciﬁc task of the benchmark.

Original:  C HALLENGES WITH BLEU AND ROUGE
Revised:  C HALLENGES WITH BLEU AND ROUGE

Original: In this part, we will discuss the limitations of BLEU and ROUGE.
Revised: In this part, we will discuss the limitations of BLEU and ROUGE.

Original: There are simple ways to attack these n-gram based metrics like adding a single word negation or changing all possible words with synonyms.
Revised: There are simple ways to attack these n-gram based metrics like adding a single word negation or changing all possible words with synonyms.

Original: Although these are theoretically plausible scenarios we also wanted to analyze which cases forced these metrics to fail in real life.
Revised: Although these are theoretically plausible scenarios we also wanted to analyze which cases forced these metrics to fail in real life.

Original: We took 100 examples from the STS-B dataset (Cer et al., 2017) where the absolute difference between the BLEU/ROUGE score and normalized label was the biggest.
Revised: We took 100 examples from the STS-B dataset (Cer et al., 2017) where the absolute difference between the BLEU/ROUGE score and normalized label was the biggest.

Original: This does not necessarily capture all failure cases of BLEU/ROUGE but a variety of failure cases can be observed.
Revised: This does not necessarily capture all failure cases of BLEU/ROUGE but a variety of failure cases can be observed.

Original: Through this analysis, we see that there are systematically recurring real life examples, just like in our theo- retical examples, where BLEU and ROUGE are failing to assess the level of similarity between two sentences.
Revised: Through this analysis, we see that there are systematically recurring real life examples, just like in our theo- retical examples, where BLEU and ROUGE are failing to assess the level of similarity between two sentences.

Original: We also observe that though some of the shortcomings of unigram metrics are mitigated through higher order n-grams, they open the door for different problems.
Revised: We also observe that though some of the shortcomings of unigram metrics are mitigated through higher order n-grams, they open the door for different problems.

Original: Some of the most common failure cases that we have encountered are listed in table 1.
Revised: Some of the most common failure cases that we have encountered are listed in table 1.

Original: 3. I DIOMS AND ADDING DETAILS
Revised: 3. I DIOMS AND ADDING DETAILS

Original: A commonly encountered failure mode involves the use of idioms and the addition of extra examples/details in one of the sentences.
Revised: A commonly encountered failure mode involves the use of idioms and the addition of extra examples/details in one of the sentences.

Original: These types of errors are especially common in more natural conversations.
Revised: These types of errors are especially common in more natural conversations.

Original: These types of errors also made 25 % of our analysis.
Revised: These types of errors also made 25 % of our analysis.

Original: Here we characteristically see humans giving high scores to these sentences because they are aware of which part holds the core meaning of the sentence while BLEU/ROUGE lack this ability.
Revised: Here we characteristically see humans giving high scores to these sentences because they are aware of which part holds the core meaning of the sentence while BLEU/ROUGE lack this ability.

Original: An example from the dataset is “You should take this animal to a vet right away.”
Revised: An example from the dataset is “You should take this animal to a vet right away.”

Original: and “As covered in the other answers, your only option is to see a vet in order to have surgery done.”
Revised: and “As covered in the other answers, your only option is to see a vet in order to have surgery done.”

Original: while the true score for this sentence pair is 3.6 out of 5, BLEU gives this sentence a 0.46 out of 5.
Revised: while the true score for this sentence pair is 3.6 out of 5, BLEU gives this sentence a 0.46 out of 5.

Original: 3. C HANGING WORDS
Revised: 3. C HANGING WORDS

Original: Another common failure mode of BLEU and ROUGE occurs when one or a few important words of a sentence is changed while the rest of the structure is kept the same.
Revised: Another common failure mode of BLEU and ROUGE occurs when one or a few important words of a sentence is changed while the rest of the structure is kept the same.

Original: An example from STS-B is “a man is speaking.”
Revised: An example from STS-B is “a man is speaking.”

Original: and “a man is spitting.”
Revised: and “a man is spitting.”

Original: while human judges give these two sentences a similarity score of 0.64 out of 5, BLEU-1 gives this sentence a 3.75 out of 5.
Revised: while human judges give these two sentences a similarity score of 0.64 out of 5, BLEU-1 gives this sentence a 3.75 out of 5.

Original: 3. G ENERAL PARAPHRASE
Revised: 3. G ENERAL PARAPHRASE

Original: We frequently see BLEU and ROUGE failing in the context of:
Revised: We frequently see BLEU and ROUGE failing in the context of:

Original: • General paraphrases where words are replaced by synonyms.
Revised: • General paraphrases where words are replaced by synonyms.

Original: This accounts for 12 % of cases.
Revised: This accounts for 12 % of cases.

Original: • Sentences are reordered.
Revised: • Sentences are reordered.

Original: This accounts for around 10% of observed errors.
Revised: This accounts for around 10% of observed errors.

Original: • Different verb tenses are used.
Revised: • Different verb tenses are used.

Original: This accounts for 15% of observed errors.
Revised: This accounts for 15% of observed errors.

Original: While the higher order n-grams are supposed to preserve the intelligibility of the sentence and not reward a model that outputs words in a random order they also punish valid re-orderings of sub sentences or words.
Revised: While the higher order n-grams are supposed to preserve the intelligibility of the sentence and not reward a model that outputs words in a random order they also punish valid re-orderings of sub sentences or words.

Original: In these smoothed methods, changing a word with a synonym will also result in a much higher penalty.
Revised: In these smoothed methods, changing a word with a synonym will also result in a much higher penalty.

Original: BLEU and ROUGE are methods that are much more frequently under scoring sentence pairs than over scoring them.
Revised: BLEU and ROUGE are methods that are much more frequently under scoring sentence pairs than over scoring them.

Original: Similarly to Reiter (Reiter, 2018) we conclude that BLEU/ROUGE can be fruitful in deciding whether a model is bad but not whether it is good.
Revised: Similarly to Reiter (Reiter, 2018) we conclude that BLEU/ROUGE can be fruitful in deciding whether a model is bad but not whether it is good.

Original: We also provide detailed examples of these failure cases in the appendix.
Revised: We also provide detailed examples of these failure cases in the appendix.

Original: 3. E XPERIMENTS 3.4. S OME EXAMPLES FROM STS-B SENTENCES PAIRS
Revised: 3. E XPERIMENTS 3.4. S OME EXAMPLES FROM STS-B SENTENCES PAIRS

Original: To illustrate our argument, we will give some examples from the dataset with their BLEU/ROUGE scores as well as a score generated from a RoBERTa model ﬁne tuned on the STS-B dataset.
Revised: To illustrate our argument, we will give some examples from the dataset with their BLEU/ROUGE scores as well as a score generated from a RoBERTa model ﬁne tuned on the STS-B dataset.

Original: We refer to this model as RoBERTa-STS.
Revised: We refer to this model as RoBERTa-STS.

Original: Note that in this paper the BLEU* and ROUGE* scores are not between 0 and 1 but are scaled with 5 to be more understandable with the scale of the scoring metric used in the dataset.
Revised: Note that in this paper the BLEU* and ROUGE* scores are not between 0 and 1 but are scaled with 5 to be more understandable with the scale of the scoring metric used in the dataset.

Original: That is why we refer to them as BLEU* and ROUGE*
Revised: That is why we refer to them as BLEU* and ROUGE*

Original: [Table]
Revised: [Table]

Original: The last time the survey was conducted, in 1995, those numbers matched. In 1995, the last survey, those numbers were equal.
Revised: The last time the survey was conducted, in 1995, those numbers matched. In 1995, the last survey, those numbers were equal.

Original: [Table]
Revised: [Table]

Original: A band is performing on a stage.
Revised: A band is performing on a stage.

Original: A band is playing onstage.
Revised: A band is playing onstage.

Original: [Table]
Revised: [Table]

Original: Two white dogs are swimming in the water.
Revised: Two white dogs are swimming in the water.

Original: The birds are swimming in the water.
Revised: The birds are swimming in the water.

Original: [Table]
Revised: [Table]

Original: A man plays the piano. A man is playing a piano.
Revised: A man plays the piano. A man is playing a piano.

Original: [Table]
Revised: [Table]

Original: Pardon the brevity of this answer, but I would say ”named” is preferred within the context of your example.
Revised: Pardon the brevity of this answer, but I would say ”named” is preferred within the context of your example.

Original: Namedis preferred in your example, since you are formally giving a name to your method.
Revised: Namedis preferred in your example, since you are formally giving a name to your method.

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: S EMANTIC SIMILARITY EXPERIMENTS
Revised: S EMANTIC SIMILARITY EXPERIMENTS

Original: In ﬁgure 1 we can see that development set scores of RoBERTa-STS in ﬁgure 1a and BLEU* in ﬁgure 1b compared to the gold labels.
Revised: In ﬁgure 1 we can see that development set scores of RoBERTa-STS in ﬁgure 1a and BLEU* in ﬁgure 1b compared to the gold labels.

Original: While we will look at the correlation scores, we can also see that the average errors in BLEU and RoBERTa-STS are remarkably different.
Revised: While we will look at the correlation scores, we can also see that the average errors in BLEU and RoBERTa-STS are remarkably different.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original:  A SSESSING E VALUATION M ETRICS 4. M ETRIC S CORECARD
Revised:  A SSESSING E VALUATION M ETRICS 4. M ETRIC S CORECARD

Original: To overcome the previously highlighted challenges and provide a framework in which metrics comparing reference summaries/translation can be assessed and improved, we establish ﬁrst-principles criteria on what a good evaluator should do:
Revised: To overcome the previously highlighted challenges and provide a framework in which metrics comparing reference summaries/translation can be assessed and improved, we establish ﬁrst-principles criteria on what a good evaluator should do:

Original: • The ﬁrst one is that it should be highly correlated with human judgement of semantic similarity.
Revised: • The ﬁrst one is that it should be highly correlated with human judgement of semantic similarity.

Original: • The second one is that it should be able to distinguish sentences which are in logical contradiction, logically unrelated or in logical agreement.
Revised: • The second one is that it should be able to distinguish sentences which are in logical contradiction, logically unrelated or in logical agreement.

Original: • The third one is that given s1, s2 which are semantically similar, eval(s1,s2) ≥ eval(s1,s2(corrupted) ≥ eval(s1,s2(more corrupted)) where corruption here includes removing words, adding noise to the word order or including grammatical mistakes.
Revised: • The third one is that given s1, s2 which are semantically similar, eval(s1,s2) ≥ eval(s1,s2(corrupted) ≥ eval(s1,s2(more corrupted)) where corruption here includes removing words, adding noise to the word order or including grammatical mistakes.

Original: 4. I MPLEMENTING THE METRIC SCORECARD
Revised: 4. I MPLEMENTING THE METRIC SCORECARD

Original: We will now show how the scorecard can be implemented.
Revised: We will now show how the scorecard can be implemented.

Original: For every dimension of the scorecard the experiments are done with three metrics.
Revised: For every dimension of the scorecard the experiments are done with three metrics.

Original: BLEU with equal weights between 1 to 4 grams.
Revised: BLEU with equal weights between 1 to 4 grams.

Original: ROUGE with averaging ROUGE-1 and ROUGE-2 and a neural evaluator.
Revised: ROUGE with averaging ROUGE-1 and ROUGE-2 and a neural evaluator.

Original: The evaluator is the RoBERTaSTS.
Revised: The evaluator is the RoBERTaSTS.

Original: 4.2. S EMANTIC S IMILARITY
Revised: 4.2. S EMANTIC S IMILARITY

Original: The ﬁrst expectation from a good similarity metric is to correlate highly with human judgment in terms of assessing semantic similarity.
Revised: The ﬁrst expectation from a good similarity metric is to correlate highly with human judgment in terms of assessing semantic similarity.

Original: Here we assessed BLEU and ROUGE on the STS-B benchmark and compared their performance to a RoBERTa model ﬁne tuned for semantic similarity (Table 2).
Revised: Here we assessed BLEU and ROUGE on the STS-B benchmark and compared their performance to a RoBERTa model ﬁne tuned for semantic similarity (Table 2).

Original: 4.2. L OGICAL E NTAILMENT
Revised: 4.2. L OGICAL E NTAILMENT

Original: Another characteristic of a good metric is to differentiate the argument, core meaning, in a sentence and take it into account when assessing hypothesis text with references.
Revised: Another characteristic of a good metric is to differentiate the argument, core meaning, in a sentence and take it into account when assessing hypothesis text with references.

Original: Here we used the MNLI dataset where for each text we have three hypothesis text representing contradiction, neutral and entailment.
Revised: Here we used the MNLI dataset where for each text we have three hypothesis text representing contradiction, neutral and entailment.

Original: We expect a good metric to rank entailment higher than neutral, and both of them higher than contradiction.
Revised: We expect a good metric to rank entailment higher than neutral, and both of them higher than contradiction.

Original: To assess the quality of a metric, we propose to use the Spearman’s ranked correlation and in Table 4 we also experiment with Kendall’s τ .
Revised: To assess the quality of a metric, we propose to use the Spearman’s ranked correlation and in Table 3 we also experiment with Kendall’s τ .

Original: Here we observe that the
Revised: Here we observe that the

Original: RoBERTa-STS model remarkably outperforms BLEU and ROUGE and both of these metrics show very little correlation with human judgment.
Revised: RoBERTa-STS model remarkably outperforms BLEU and ROUGE and both of these metrics show very little correlation with human judgment.

Original: 4.2. R OBUSTNESS TO G RAMMATICAL E RRORS
Revised: 4.2. R OBUSTNESS TO G RAMMATICAL E RRORS

Original: For assessing the third criteria.
Revised: For assessing the third criteria.

Original: We start with 3479 sentence pairs from the MNLI dataset that are labelled as entailment.
Revised: We start with 3479 sentence pairs from the MNLI dataset that are labelled as entailment.

Original: We introduce random corruptions such as random insertion, deletion and grammatical errors as in (Zhao et al., 2019).
Revised: We introduce random corruptions such as random insertion, deletion and grammatical errors as in (Zhao et al., 2019).

Original: We use two different set of parameters for different corruption levels and expect that a good metric would rank the original similar sentence higher than the less corrupted and both higher than the more corrupted sentence.
Revised: We use two different set of parameters for different corruption levels and expect that a good metric would rank the original similar sentence higher than the less corrupted and both higher than the more corrupted sentence.

Original: Here we also propose to use the Spearman’s ranked correlation and also experiment with Kendall’s τ .
Revised: Here we also propose to use the Spearman’s ranked correlation and also experiment with Kendall’s τ .

Original: We report results on Table 4.
Revised: We report results on Table 4.

Original: Where we see that the RoBERTa-STS model once more outperforms BLEU and ROUGE.
Revised: Where we see that the RoBERTa-STS model once more outperforms BLEU and ROUGE.

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  M ACHINE T RANSLATION E XPERIMENTS
Revised:  M ACHINE T RANSLATION E XPERIMENTS

Original: In the previous section, we outlined a series of criteria to assess evaluation metrics and shown how for each dimension, RoBERTa-STS signiﬁcantly outperformed ROUGE and BLEU.
Revised: In the previous section, we outlined a series of criteria to assess evaluation metrics and shown how for each dimension, RoBERTa-STS signiﬁcantly outperformed ROUGE and BLEU.

Original: In the following section, we report results showing how BLEU, ROUGE and RoBERTa-STS correlate with human judgement of quality in the case of machine translation.
Revised: In the following section, we report results showing how BLEU, ROUGE and RoBERTa-STS correlate with human judgement of quality in the case of machine translation.

Original: Using the WMT2015 and WMT2016 datasets, we collected reference and hypothesis translations along with human judgment scores where the translation was from another language to English.
Revised: Using the WMT2015 and WMT2016 datasets, we collected reference and hypothesis translations along with human judgment scores where the translation was from another language to English.

Original: This gave us 5360 sentence pairs.
Revised: This gave us 5360 sentence pairs.

Original: For each of these sentence pairs, we computed the BLEU, ROUGE-1,
Revised: For each of these sentence pairs, we computed the BLEU, ROUGE-1,

Original: ROUGE-2, ROUGE-L scores.
Revised: ROUGE-2, ROUGE-L scores.

Original: We also computed the RoBERTa-STS predicted semantic similarity.
Revised: We also computed the RoBERTa-STS predicted semantic similarity.

Original: We performed a correlation of the various scores with human judgements and report the scores in
Revised: We performed a correlation of the various scores with human judgements and report the scores in

Original: Table 5.
Revised: Table 5.

Original: Here again, we can see Transformers based model outperforming BLEU and ROUGE to assess the quality of machine translation.
Revised: Here again, we can see Transformers based model outperforming BLEU and ROUGE to assess the quality of machine translation.

Original:  C ONCLUSION AND FUTURE WORK
Revised:  C ONCLUSION AND FUTURE WORK

Original: In this work, we have established a framework to assess metrics comparing the quality of reference and hypothesis summary/translations.
Revised: In this work, we have established a framework to assess metrics comparing the quality of reference and hypothesis summary/translations.

Original: Based on these criteria, we compare evaluators using recent
Revised: Based on these criteria, we compare evaluators using recent

Original: Transformers (in this case RoBERTa-STS) to BLEU and ROUGE and highlight their potential to replace BLEU and ROUGE.
Revised: Transformers (in this case RoBERTa-STS) to BLEU and ROUGE and highlight their potential to replace BLEU and ROUGE.

Original: A A PPENDIX
Revised: A A PPENDIX

Original: In the appendix we will discuss the failure cases of BLEU, ROUGE and RoBERTa-STSin detail to provide a better understanding of how these models can fall short in language evaluation.
Revised: In the appendix we will discuss the failure cases of BLEU, ROUGE and RoBERTa-STSin detail to provide a better understanding of how these models can fall short in language evaluation.

Original: This is important because a good metric scorecard has to represent the quality of an evaluator and this experiments are to show that our metrics cover many of the failure cases and can assess them without the burden of manually evaluating the outputs of every evaluator.
Revised: This is important because a good metric scorecard has to represent the quality of an evaluator and this experiments are to show that our metrics cover many of the failure cases and can assess them without the burden of manually evaluating the outputs of every evaluator.

Original: We will start by taking examples from the BLEU and ROUGE dataset.
Revised: We will start by taking examples from the BLEU and ROUGE dataset.

Original: As in the paper the BLEU scores used are always a uniform average up to 4-grams and the ROUGE score is the average of
Revised: As in the paper the BLEU scores used are always a uniform average up to 4-grams and the ROUGE score is the average of

Original: ROUGE-1 and ROUGE-2.
Revised: ROUGE-1 and ROUGE-2.

Original: Both scores are scaled up to 5 to increase the interpretebility of the scores given that the labels in the similarty dataset are between 0 and 5.
Revised: Both scores are scaled up to 5 to increase the interpretebility of the scores given that the labels in the similarty dataset are between 0 and 5.

Original: [Table]
Revised: [Table]

Original:  The company claims it’s the largest single Apple VAR Xserve sale to date.
Revised:  The company claims it’s the largest single Apple VAR Xserve sale to date.

Original: The company claimed it is the largest sale of Xserves by an Apple retailer.
Revised: The company claimed it is the largest sale of Xserves by an Apple retailer.

Original: A woman puts ﬂour on a piece of meat.
Revised: A woman puts ﬂour on a piece of meat.

Original: A woman is putting ﬂour onto some meat.
Revised: A woman is putting ﬂour onto some meat.

Original: He later learned that the incident was caused by the Concorde’s sonic boom.
Revised: He later learned that the incident was caused by the Concorde’s sonic boom.

Original: He later found out the alarming incident had been caused by Concorde’s powerful sonic boom.
Revised: He later found out the alarming incident had been caused by Concorde’s powerful sonic boom.

Original: It indeed appears the Andromeda galaxy (M31) and The Milky Way (MW) are en route to a collision.
Revised: It indeed appears the Andromeda galaxy (M31) and The Milky Way (MW) are en route to a collision.

Original: Ina few billion years, the Milky Way and Andromeda will collide.
Revised: Ina few billion years, the Milky Way and Andromeda will collide.

Original: You deﬁnitely do NOT want to be supporting your weight with your arms on the bike for normal riding.
Revised: You deﬁnitely do NOT want to be supporting your weight with your arms on the bike for normal riding.

Original: No, don’t support your weight on your arms Your hands simply aren’t really made for supporting all that weight.
Revised: No, don’t support your weight on your arms Your hands simply aren’t really made for supporting all that weight.

Original: detained for ’house
Revised: detained for ’house

Original: sister’
Revised: sister’

Original: scandal
Revised: scandal

Original: China detains 7 for ”house sister” scandal 0.
Revised: China detains 7 for ”house sister” scandal 0.

Original: A man plays the violin.
Revised: A man plays the violin.

Original: A man is playing violin.
Revised: A man is playing violin.

Original: It is simply the number of balls bowled divided by the number of wickets taken.
Revised: It is simply the number of balls bowled divided by the number of wickets taken.

Original: Bowling strike rate is deﬁned for a bowler as the average number of balls bowled per wicket taken.
Revised: Bowling strike rate is deﬁned for a bowler as the average number of balls bowled per wicket taken.

Original: Police helicopter crashes into pub in Glasgowseveral casualties
Revised: Police helicopter crashes into pub in Glasgowseveral casualties

Original: Helicopter crashes into roof of Glasgow club 0.
Revised: Helicopter crashes into roof of Glasgow club 0.

Original: Oil falls in Asian
Revised: Oil falls in Asian

Original: trade
Revised: trade

Original: Oil prices down in Asian trade 1.
Revised: Oil prices down in Asian trade 1.

Original: A skateboarder jumps off the stairs.
Revised: A skateboarder jumps off the stairs.

Original: A dog jumps off the stairs.
Revised: A dog jumps off the stairs.

Original: Wigan 3-2 Wolves: Match report, pictures & video highlights
Revised: Wigan 3-2 Wolves: Match report, pictures & video highlights

Original: Arsenal 0-0 Chelsea: Match report, pictures & video highlights 3.
Revised: Arsenal 0-0 Chelsea: Match report, pictures & video highlights 3.

Original: In table 6 we see examples of many different error cases and in most sentences we also have more than one cause for the drastic difference between BLEU/ROUGE and the Label.
Revised: In table 6 we see examples of many different error cases and in most sentences we also have more than one cause for the drastic difference between BLEU/ROUGE and the Label.

Original: For instance in rows 1 and 6 we see that the cause for the error is the reordering of sub-sentences, spelling/punctuation and newly introduced words that don’t change the meaning but merely extend it.
Revised: For instance in rows 1 and 6 we see that the cause for the error is the reordering of sub-sentences, spelling/punctuation and newly introduced words that don’t change the meaning but merely extend it.

Original: While BLEU and
Revised: While BLEU and

Original: ROUGE are failing in these examples we see that the RoBERTa-STSmodel scores similar to the label.
Revised: ROUGE are failing in these examples we see that the RoBERTa-STSmodel scores similar to the label.

Original: In line 7 we can see that the RoBERTa-STSmodel score is above 5.
Revised: In line 7 we can see that the RoBERTa-STSmodel score is above 5.

Original: While this is because the model is designed as a regression model, as expected given the nature of the task.
Revised: While this is because the model is designed as a regression model, as expected given the nature of the task.

Original: In rows 2 and 7 we see that the main difference is the form or tense of the verb in a sentence.
Revised: In rows 2 and 7 we see that the main difference is the form or tense of the verb in a sentence.

Original: While using higher order n-grams enforces the goal of BLEU and ROUGE to not over score randomly shufﬂed sentences or remarkable meaning shift due to minimal changes in words such as in row 11.
Revised: While using higher order n-grams enforces the goal of BLEU and ROUGE to not over score randomly shufﬂed sentences or remarkable meaning shift due to minimal changes in words such as in row 11.

Original: This also makes BLEU severely under score simple changes with synonyms or valid re-orderings as seen in the examples below.
Revised: This also makes BLEU severely under score simple changes with synonyms or valid re-orderings as seen in the examples below.

Original: This characteristic of BLEU reinforces the point that BLEU and
Revised: This characteristic of BLEU reinforces the point that BLEU and

Original: ROUGE are not useful in tracking the state of the art and comparing the best methods but are tools to weed out bad models fairly simply.
Revised: ROUGE are not useful in tracking the state of the art and comparing the best methods but are tools to weed out bad models fairly simply.

Original: In rows 3 and 9 we see sentences that differ due to using descriptive phrases instead of a word or extending the sentence with more information.
Revised: In rows 3 and 9 we see sentences that differ due to using descriptive phrases instead of a word or extending the sentence with more information.

Original: These types of errors changes are also caught with language models since we know they have the ability to hold the meaning of multiple words and incorporate them to reach a related word as in the famous example of king - men + woman =
Revised: These types of errors changes are also caught with language models since we know they have the ability to hold the meaning of multiple words and incorporate them to reach a related word as in the famous example of king - men + woman =

Original: queen [cite word2vec].
Revised: queen [cite word2vec].

Original: In rows 4,8 and 5 we see general paraphrases with the same meaning represented in a generally different sentence.
Revised: In rows 4,8 and 5 we see general paraphrases with the same meaning represented in a generally different sentence.

Original: In all cases we see an drastic difference between BLEU/ROUGE and the label but these cases also unearth a speciﬁc characteristic of the neural evaluator.
Revised: In all cases we see an drastic difference between BLEU/ROUGE and the label but these cases also unearth a speciﬁc characteristic of the neural evaluator.

Original: In 4 and 8 we see that the error of the RoBERTa-STS model comparatively lower than row 5.
Revised: In 4 and 8 we see that the error of the RoBERTa-STS model comparatively lower than row 5.

Original: While it is hard to determine the exact cause through only looking at these examples table 7 for the RoBERTa-STS failure cases will make this case more compelling.
Revised: While it is hard to determine the exact cause through only looking at these examples table 7 for the RoBERTa-STS failure cases will make this case more compelling.

Original: While language models have a general sense of the context in a given sentence they still lack a general knowledge of the world.
Revised: While language models have a general sense of the context in a given sentence they still lack a general knowledge of the world.

Original: Hence in the second sentence of row 5, because the words riding, bike, bicycle are missing the model has a hard time recognising that the second sentence is also about the same topic.
Revised: Hence in the second sentence of row 5, because the words riding, bike, bicycle are missing the model has a hard time recognising that the second sentence is also about the same topic.

Original: To test this we added ”while riding” or ”on a bike” at the end of a sentence and the score immediately went up to 3.6/5 while barely changing the BLEU* and the ROUGE score.
Revised: To test this we added ”while riding” or ”on a bike” at the end of a sentence and the score immediately went up to 3.6/5 while barely changing the BLEU* and the ROUGE score.

Original: In row 4 and 8 however, the context of the sentence is deﬁned explicitly with the key phrases.
Revised: In row 4 and 8 however, the context of the sentence is deﬁned explicitly with the key phrases.

Original: We see this bias affecting RoBERTa-STS scoring in the examples below.
Revised: We see this bias affecting RoBERTa-STS scoring in the examples below.

Original: [Table]
Revised: [Table]

Original: It would be unusual for a snake to attack a stationary person.
Revised: It would be unusual for a snake to attack a stationary person.

Original: I’m no herpetologist, but in my experience, snakes are in the ”you don’t bug me, I won’t bug you” category.
Revised: I’m no herpetologist, but in my experience, snakes are in the ”you don’t bug me, I won’t bug you” category.

Original: [Table]
Revised: [Table]

Original: New UN peacekeeping chief named for Central African Republic
Revised: New UN peacekeeping chief named for Central African Republic

Original: UN takes over peacekeeping in Central African
Revised: UN takes over peacekeeping in Central African

Original: Republic
Revised: Republic

Original: [Table]
Revised: [Table]

Original: From Broadway comedies like ”The Seven Year
Revised: From Broadway comedies like ”The Seven Year

Original: Itch” (1952), ”Will Success Spoil Rock Hunter?”
Revised: Itch” (1952), ”Will Success Spoil Rock Hunter?”

Original: Playwright George Axelrod, who anticipated the sexual revolution with The Seven Year Itch and
Revised: Playwright George Axelrod, who anticipated the sexual revolution with The Seven Year Itch and

Original: Will Success Spoil Rock Hunter?
Revised: Will Success Spoil Rock Hunter?

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original:  a group of navy seals are singing A group of military personnel are playing in a brass quintet.
Revised:  a group of navy seals are singing A group of military personnel are playing in a brass quintet.

Original: [Table]
Revised: [Table]

Original: In the above examples we will ﬁnd a two points that will helps us better understand the RoBERTaSTS as a neural evaluator.
Revised: In the above examples we will ﬁnd a two points that will helps us better understand the RoBERTaSTS as a neural evaluator.

Original: First we see that the neural network sometimes lacks a sense of context that is not given in the sentence explicitly.
Revised: First we see that the neural network sometimes lacks a sense of context that is not given in the sentence explicitly.

Original: While these language models are trained on a large corpus and capture a sense of the words and language we still see that their performance is not perfect.
Revised: While these language models are trained on a large corpus and capture a sense of the words and language we still see that their performance is not perfect.

Original: We see these examples in row 4 where the model cannot relate a navy seal as a military personnel.
Revised: We see these examples in row 4 where the model cannot relate a navy seal as a military personnel.

Original: Or as in row 1 where the model cannot model an idiom.
Revised: Or as in row 1 where the model cannot model an idiom.

Original: The second and more critical place where we need further development is especially detecting whether the core argument/message in a sentence is the same beyond whether if they are talking about the same things.
Revised: The second and more critical place where we need further development is especially detecting whether the core argument/message in a sentence is the same beyond whether if they are talking about the same things.

Original: As in rows 2 and 3.
Revised: As in rows 2 and 3.

Original: We see the same landmark words and can clearly say that the sentences are talking about the same things but what a human can distinguish is that they are saying unrelated things.
Revised: We see the same landmark words and can clearly say that the sentences are talking about the same things but what a human can distinguish is that they are saying unrelated things.

Original: This is one of the key motivations in including the language inference task in the scorecard.
Revised: This is one of the key motivations in including the language inference task in the scorecard.

Original: Since detecting whether a pair of sentences are related on what level is a key part of detecting sentence similarity.
Revised: Since detecting whether a pair of sentences are related on what level is a key part of detecting sentence similarity.

Original: One last thing we will mention is that while RoBERTa-STS and BLEU/ROUGE have different error cases their performance on these error cases is also remarkably different in favor of the former.
Revised: One last thing we will mention is that while RoBERTa-STS and BLEU/ROUGE have different error cases their performance on these error cases is also remarkably different in favor of the former.

Original: Table 8 shows the mean error of BLEU* and the RoBERTa-STS model on each others top 500, which is one third of the development set, error cases.
Revised: Table 8 shows the mean error of BLEU* and the RoBERTa-STS model on each others top 500, which is one third of the development set, error cases.

Original: [Table]
Revised: [Table]

Original: We see in table 8 that in BLEU* has a remarkable error in both its failure cases and also the failure cases of RoBERTa-STS while RoBERTa-STS outperforms BLEU* in each category.
Revised: We see in table 8 that in BLEU* has a remarkable error in both its failure cases and also the failure cases of RoBERTa-STS while RoBERTa-STS outperforms BLEU* in each category.

Original: While neural evaluators have also room for improvement we can with conﬁdence say that they are outperforming classical methods and with a methodical way of improving them can bolster progress of NLP research.
Revised: While neural evaluators have also room for improvement we can with conﬁdence say that they are outperforming classical methods and with a methodical way of improving them can bolster progress of NLP research.