Original: B LON D E : An Automatic Evaluation Metric for Document-level Machine Translation
Revised: B LON D E : An Automatic Evaluation Metric for Document-level Machine Translation

Original: Abstract
Revised: Abstract

Original: Standard automatic metrics, e.g. BLEU , are not reliable for document-level MT evaluation.
Revised: Standard automatic metrics, e.g. BLEU , are not reliable for document-level MT evaluation.

Original: They can neither distinguish documentlevel improvements in translation quality from sentence-level ones, nor identify the discourse phenomena that cause context-agnostic translations.
Revised: They can neither distinguish documentlevel improvements in translation quality from sentence-level ones, nor identify the discourse phenomena that cause context-agnostic translations.

Original: This paper introduces a novel automatic metric B LON D E 1 to widen the scope of automatic MT evaluation from sentence to the document level.
Revised: This paper introduces a novel automatic metric B LON D E 1 to widen the scope of automatic MT evaluation from sentence to the document level.

Original: B LON D E takes discourse coherence into consideration by categorizing discourse-related spans and calculating the similarity-based F1 measure of categorized spans.
Revised: B LON D E takes discourse coherence into consideration by categorizing discourse-related spans and calculating the similarity-based F1 measure of categorized spans.

Original: We conduct extensive comparisons on a newly constructed document-level translation dataset BWB .
Revised: We conduct extensive comparisons on a newly constructed document-level translation dataset BWB .

Original: The experimental results show that B LON D E possesses better selectivity and interpretability at the document-level, and is more sensitive to document-level nuances.
Revised: The experimental results show that B LON D E possesses better selectivity and interpretability at the document-level, and is more sensitive to document-level nuances.

Original: In a large-scale human study, B LON D E also achieves signiﬁcantly higher Pearson’s r correlation with human judgments compared to previous metrics.
Revised: In a large-scale human study, B LON D E also achieves signiﬁcantly higher Pearson’s r correlation with human judgments compared to previous metrics.

Original:  Introduction
Revised:  Introduction

Original: Over the past few years, neural machine translation (NMT) models have become the models of choice in Machine Translation (MT) (Luong et al., 2015;
Revised: Over the past few years, neural machine translation (NMT) models have become the models of choice in Machine Translation (MT) (Luong et al., 2015;

Original: Vaswani et al., 2017; Zhang et al., 2018, inter alia ).
Revised: Vaswani et al., 2017; Zhang et al., 2018, inter alia ).

Original: Although some recent work (Hassan et al., 2018;
Revised: Although some recent work (Hassan et al., 2018;

Original: Popel, 2018; Bojar et al., 2018) suggest that NMT has achieved human parity at the sentence level, the reliability of these human-parity claims was quickly contested by Läubli et al.
Revised: Popel, 2018; Bojar et al., 2018) suggest that NMT has achieved human parity at the sentence level, the reliability of these human-parity claims was quickly contested by Läubli et al.

Original: (2018, 2020), showing that there is a larger difference between human and machine translation quality when intersentential context is taken into account.
Revised: (2018, 2020), showing that there is a larger difference between human and machine translation quality when intersentential context is taken into account.

Original: Therefore, document-level machine translation has received growing attention in the MT community.
Revised: Therefore, document-level machine translation has received growing attention in the MT community.

Original: However, despite various modeling advances, we still lack an efﬁcient and effective evaluation metric for document-level translation.
Revised: However, despite various modeling advances, we still lack an efﬁcient and effective evaluation metric for document-level translation.

Original: Standard evaluation metrics for MT (e.g., BLEU (Papineni et al., 2002), TER (Snover et al., 2006) and METEOR (Banerjee and Lavie, 2005)) focus on the quality of translations at the sentence level and do not consider discourse-level features.
Revised: Standard evaluation metrics for MT (e.g., BLEU (Papineni et al., 2002), TER (Snover et al., 2006) and METEOR (Banerjee and Lavie, 2005)) focus on the quality of translations at the sentence level and do not consider discourse-level features.

Original: Thus, test suites that perform context-aware evaluation by targeting characteristic discourse-level phenomena have been proposed (Hardmeier et al., 2015; Guillou and Hardmeier, 2016; Burchardt et al., 2017; Isabelle et al., 2017; Rios Gonzales et al., 2017; Müller et al., 2018; Bawden et al., 2018; Voita et al., 2019; Guillou and Hardmeier, 2018, inter alia ) for document-level MT.
Revised: Thus, test suites that perform context-aware evaluation by targeting characteristic discourse-level phenomena have been proposed (Hardmeier et al., 2015; Guillou and Hardmeier, 2016; Burchardt et al., 2017; Isabelle et al., 2017; Rios Gonzales et al., 2017; Müller et al., 2018; Bawden et al., 2018; Voita et al., 2019; Guillou and Hardmeier, 2018, inter alia ) for document-level MT.

Original: However, such test suites need to be re-created for new domains or even language pairs, and the construction of such test suites can be very labor-intensive.
Revised: However, such test suites need to be re-created for new domains or even language pairs, and the construction of such test suites can be very labor-intensive.

Original: We still lack an easy-to-use automatic metric that can reliably discriminate the quality of document-level translation.
Revised: We still lack an easy-to-use automatic metric that can reliably discriminate the quality of document-level translation.

Original: In this paper, we curate a large-scale documentlevel parallel corpus ( BWB ) from heterogeneous data sources, and quantify document-level translation mistakes by performing a large human study.
Revised: In this paper, we curate a large-scale documentlevel parallel corpus ( BWB ) from heterogeneous data sources, and quantify document-level translation mistakes by performing a large human study.

Original: We found that on this dataset, inconsistency 3 , ellipsis and ambiguity were the most noticeable phenomena critical for document-level MT, together amounting to 86.73% of MT mistakes.
Revised: We found that on this dataset, inconsistency 3 , ellipsis and ambiguity were the most noticeable phenomena critical for document-level MT, together amounting to 86.73% of MT mistakes.

Original: Based on this analysis, we propose B LON D E , an automatic metric that evaluates translation quality at the document level.
Revised: Based on this analysis, we propose B LON D E , an automatic metric that evaluates translation quality at the document level.

Original: At the core of the metric is the similarity-based bijection between subsets of reference and system categories (e.g. pronouns, inﬂected forms, discourse relations and lexicons) and phrases (e.g. named entities).
Revised: At the core of the metric is the similarity-based bijection between subsets of reference and system categories (e.g. pronouns, inﬂected forms, discourse relations and lexicons) and phrases (e.g. named entities).

Original: It then computes recall, precision and F-measure, along with the corresponding measure of n-grams.
Revised: It then computes recall, precision and F-measure, along with the corresponding measure of n-grams.

Original: Furthermore, B LON D E can incorporate human annotation easily by computing scores of human-annotated categories in the same way.
Revised: Furthermore, B LON D E can incorporate human annotation easily by computing scores of human-annotated categories in the same way.

Original: We compare B LON D E with 11 other metrics and demonstrate that B LON D E is better at distinguishing between context-aware and context-agnostic
Revised: We compare B LON D E with 11 other metrics and demonstrate that B LON D E is better at distinguishing between context-aware and context-agnostic

Original: MT systems.
Revised: MT systems.

Original: We also observe that the degree to which B LON D E correlates with sentence-level metrics (e.g. BLEU ) is lower than the degree to which the sentence-level metrics correlate with each other.
Revised: We also observe that the degree to which B LON D E correlates with sentence-level metrics (e.g. BLEU ) is lower than the degree to which the sentence-level metrics correlate with each other.

Original: This signals that B LON D E indeed captures additional aspects of translation quality beyond the sentence-level.
Revised: This signals that B LON D E indeed captures additional aspects of translation quality beyond the sentence-level.

Original: Finally, our human evaluation also reveals signiﬁcantly higher Pearson’s r correlation coefﬁcients between B LON D E and human assessments compared to other metrics.
Revised: Finally, our human evaluation also reveals signiﬁcantly higher Pearson’s r correlation coefﬁcients between B LON D E and human assessments compared to other metrics.

Original:  BWB : Bilingual Web Book Dataset
Revised:  BWB : Bilingual Web Book Dataset

Original: To design a metric that is sensitive to documentlevel phenomena, we ﬁrst curate a document-level
Revised: To design a metric that is sensitive to documentlevel phenomena, we ﬁrst curate a document-level

Original: Chinese–English parallel corpus, called BWB ( B ilingual W eb B ooks).
Revised: Chinese–English parallel corpus, called BWB ( B ilingual W eb B ooks).

Original: BWB consists of Chinese online novels across multiple genres (sci-ﬁ, romance, action, fantasy, comedy, etc.) and their corresponding English translations crawled from the Internet.
Revised: BWB consists of Chinese online novels across multiple genres (sci-ﬁ, romance, action, fantasy, comedy, etc.) and their corresponding English translations crawled from the Internet.

Original: Dataset Creation
Revised: Dataset Creation

Original: The novels are translated by professional native English speakers, and are corrected by editors.
Revised: The novels are translated by professional native English speakers, and are corrected by editors.

Original: The sentence alignment of the training set is done by Bluealign 4 (Sennrich and
Revised: The sentence alignment of the training set is done by Bluealign 4 (Sennrich and

Original: Volk, 2011).
Revised: Volk, 2011).

Original: We hired four bilingual graduate students to manually evaluate 163 randomly selected documents from the resulting BWB parallel corpus and observe an alignment accuracy rate of 93.1%.
Revised: We hired four bilingual graduate students to manually evaluate 163 randomly selected documents from the resulting BWB parallel corpus and observe an alignment accuracy rate of 93.1%.

Original: We further asked the same batch of annotators to correct such misalignments in both the development and the test set.
Revised: We further asked the same batch of annotators to correct such misalignments in both the development and the test set.

Original: Statistics
Revised: Statistics

Original: Table 1 summarizes the statistics of the
Revised: Table 1 summarizes the statistics of the

Original: BWB dataset.
Revised: BWB dataset.

Original: It is a much larger dataset, and contains longer documents and richer discourse phenomena compared to all previous document-level datasets (Lison and Tiedemann, 2016; Koehn and
Revised: It is a much larger dataset, and contains longer documents and richer discourse phenomena compared to all previous document-level datasets (Lison and Tiedemann, 2016; Koehn and

Original: Knowles, 2017; Barrault et al., 2019; Koehn, 2005;
Revised: Knowles, 2017; Barrault et al., 2019; Koehn, 2005;

Original: Liu and Zhang, 2020).
Revised: Liu and Zhang, 2020).

Original: To the best of our knowledge, this is the largest Chinese–English documentlevel translation dataset to date.
Revised: To the best of our knowledge, this is the largest Chinese–English documentlevel translation dataset to date.

Original: [Table]
Revised: [Table]

Original: Dataset Split
Revised: Dataset Split

Original: We treat chapters in our books as documents .
Revised: We treat chapters in our books as documents .

Original: The maximum, median, and minimum number of sentences per document are 46, 30 and 18, respectively.
Revised: The maximum, median, and minimum number of sentences per document are 46, 30 and 18, respectively.

Original: To prevent any train-test leakage, we split the dataset into a training, development and a test set such that chapters from the same book are part of the same split.
Revised: To prevent any train-test leakage, we split the dataset into a training, development and a test set such that chapters from the same book are part of the same split.

Original: We use 377 books for training, and randomly select 80 and 79 documents from the 3,018 documents in the remaining 6 books as the development and test set.
Revised: We use 377 books for training, and randomly select 80 and 79 documents from the 3,018 documents in the remaining 6 books as the development and test set.

Original:  Analyzing Discourse Errors
Revised:  Analyzing Discourse Errors

Original: Next, we conduct a human study on the test set of BWB , in which we identify and categorize the discourse errors made by MT systems that are not captured in sentence-level evaluation.
Revised: Next, we conduct a human study on the test set of BWB , in which we identify and categorize the discourse errors made by MT systems that are not captured in sentence-level evaluation.

Original: This human study is conducted by eight professional translators.
Revised: This human study is conducted by eight professional translators.

Original: The annotators are asked to classify translation errors into DOCUMENT -level and SENTENCE -level errors (some cases can be both).
Revised: The annotators are asked to classify translation errors into DOCUMENT -level and SENTENCE -level errors (some cases can be both).

Original: SENTENCE -level errors refer to those errors that cause the translations to be inadequate or not ﬂuent as stand-alone sentences, while DOCUMENT -level errors lead to
Revised: SENTENCE -level errors refer to those errors that cause the translations to be inadequate or not ﬂuent as stand-alone sentences, while DOCUMENT -level errors lead to

Original: [Table]
Revised: [Table]

Original: a coherence violation across multiple sentences in the document.
Revised: a coherence violation across multiple sentences in the document.

Original: DOCUMENT -level errors are further categorized according to the linguistic phenomena leading to the discrepancy in the context-dependent translation.
Revised: DOCUMENT -level errors are further categorized according to the linguistic phenomena leading to the discrepancy in the context-dependent translation.

Original: Table 2 shows the result of our error analysis.
Revised: Table 2 shows the result of our error analysis.

Original: A substantial proportion of translations have document-level errors (71.9%).
Revised: A substantial proportion of translations have document-level errors (71.9%).

Original: This veriﬁes that
Revised: This veriﬁes that

Original: BWB contains rich discourse phenomena that current common MT systems cannot address.
Revised: BWB contains rich discourse phenomena that current common MT systems cannot address.

Original: We observe that three categories, i.e. inconsistency (64.4%), ellipsis (20.3%) and ambiguity (7.3%), account for the vast majority of document-level errors.
Revised: We observe that three categories, i.e. inconsistency (64.4%), ellipsis (20.3%) and ambiguity (7.3%), account for the vast majority of document-level errors.

Original: Below we discuss these three categories of
Revised: Below we discuss these three categories of

Original: DOCUMENT -level errors and the design intuitions behind B LON D E .
Revised: DOCUMENT -level errors and the design intuitions behind B LON D E .

Original: Inconsistency
Revised: Inconsistency

Original: We consider two kinds of consistencies in translation: lexical and grammatical.
Revised: We consider two kinds of consistencies in translation: lexical and grammatical.

Original: Lexical consistency is deﬁned as a repetitive term that keeps the same translation throughout the whole document (Carpuat and Simard, 2012).
Revised: Lexical consistency is deﬁned as a repetitive term that keeps the same translation throughout the whole document (Carpuat and Simard, 2012).

Original: Inconsistent translation of named entities can signiﬁcantly impact translation output, although BLEU may not be adversely affected (Agrawal and Singla, 2012; Hermjakob et al., 2008).
Revised: Inconsistent translation of named entities can signiﬁcantly impact translation output, although BLEU may not be adversely affected (Agrawal and Singla, 2012; Hermjakob et al., 2008).

Original: Therefore, in the design of B LON D E , we also focus on the reiteration of named entities (e.g. Qiao in Figure 3).
Revised: Therefore, in the design of B LON D E , we also focus on the reiteration of named entities (e.g. Qiao in Figure 3).

Original: Typical grammatical consistency includes tense consistency and gender consistency.
Revised: Typical grammatical consistency includes tense consistency and gender consistency.

Original: Tense consistency refers to the tense being compatible (rather than keeping exactly the same tense) with the context.
Revised: Tense consistency refers to the tense being compatible (rather than keeping exactly the same tense) with the context.

Original: It is prominent when the source language is an isolating language, e.g. Chinese, and the target language is synthetic language, e.g. English ( teal in Figure 3).
Revised: It is prominent when the source language is an isolating language, e.g. Chinese, and the target language is synthetic language, e.g. English ( teal in Figure 3).

Original: In the same spirit, the same entity should maintain a consistent grammatical gender.
Revised: In the same spirit, the same entity should maintain a consistent grammatical gender.

Original: [Figure]
Revised: [Figure]

Original: Ellipsis
Revised: Ellipsis

Original: Ellipsis denotes the omission from a clause of one or more words that are nevertheless understood in the context of the remaining elements (Voita et al., 2019; Yamamoto and Sumita, 1998).
Revised: Ellipsis denotes the omission from a clause of one or more words that are nevertheless understood in the context of the remaining elements (Voita et al., 2019; Yamamoto and Sumita, 1998).

Original: Confusion arises when there are elliptical constructions in the source language while the target language does not allow the same types of ellipsis.
Revised: Confusion arises when there are elliptical constructions in the source language while the target language does not allow the same types of ellipsis.

Original: For example, the ellipsis of subjects or objects is very common in Chinese while it is ungrammatical in
Revised: For example, the ellipsis of subjects or objects is very common in Chinese while it is ungrammatical in

Original: English, especially for pronouns.
Revised: English, especially for pronouns.

Original: In Figure 3, she (Qiao) is omitted in Chinese.
Revised: In Figure 3, she (Qiao) is omitted in Chinese.

Original: However, it is hard to know the gender of Qiao from this stand-alone sentence: the correct pronoun choice can only be inferred from context (there is a her in the previous sentence).
Revised: However, it is hard to know the gender of Qiao from this stand-alone sentence: the correct pronoun choice can only be inferred from context (there is a her in the previous sentence).

Original: Another ellipsis that cannot be ignored is the omission of discourse markers, especially when the source language has more zero connective structures (Po-Ching and Rimmington, 2004) than the target language.
Revised: Another ellipsis that cannot be ignored is the omission of discourse markers, especially when the source language has more zero connective structures (Po-Ching and Rimmington, 2004) than the target language.

Original: In the example, However and So are ignored in SRC , which misleads the sentence-level system MTA to ignore the discourse relations between sentences.
Revised: In the example, However and So are ignored in SRC , which misleads the sentence-level system MTA to ignore the discourse relations between sentences.

Original: Ambiguity
Revised: Ambiguity

Original: Translation ambiguity occurs when a word in one language can be translated in more than one way into another language (Tokowicz and Degani, 2010).
Revised: Translation ambiguity occurs when a word in one language can be translated in more than one way into another language (Tokowicz and Degani, 2010).

Original: The cross-language ambiguity comes from several sources of within-language ambiguity including lexical ambiguity, polysemy, and near-synonymy.
Revised: The cross-language ambiguity comes from several sources of within-language ambiguity including lexical ambiguity, polysemy, and near-synonymy.

Original: A uniﬁed feature of these is that ambiguous terms satisfy the form of one-to-many mappings.
Revised: A uniﬁed feature of these is that ambiguous terms satisfy the form of one-to-many mappings.

Original: For the example in Figure 2, the word 看 (kan) can be translated to look , see , watch or view .
Revised: For the example in Figure 2, the word 看 (kan) can be translated to look , see , watch or view .

Original: Without the context indicating what refers to a movie, all the lexical choices are coherent.
Revised: Without the context indicating what refers to a movie, all the lexical choices are coherent.

Original:  B LON D E
Revised:  B LON D E

Original: The aforementioned document-level phenomena have little impact on the n -gram statistics of translations.
Revised: The aforementioned document-level phenomena have little impact on the n -gram statistics of translations.

Original: However, as is shown in Section 3, they can be key considerations for human readers when evaluating translations at the document level.
Revised: However, as is shown in Section 3, they can be key considerations for human readers when evaluating translations at the document level.

Original: Standard automatic metrics ignore the importance of contextual coherence of translations, causing the
Revised: Standard automatic metrics ignore the importance of contextual coherence of translations, causing the

Original: [Table]
Revised: [Table]

Original: TENSE V
Revised: TENSE V

Original: PRONOUN
Revised: PRONOUN

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: document-level nuances to be overlooked (Zhou et al., 2008; Xiong and Zhang, 2014).
Revised: document-level nuances to be overlooked (Zhou et al., 2008; Xiong and Zhang, 2014).

Original: In this section, we describe B LON D E , an automatic metric that explicitly tracks discourse phenomena.
Revised: In this section, we describe B LON D E , an automatic metric that explicitly tracks discourse phenomena.

Original: 4. Document-Level Evaluation
Revised: 4. Document-Level Evaluation

Original: We ﬁrst give the formulation of measuring discourse phenomena.
Revised: We ﬁrst give the formulation of measuring discourse phenomena.

Original: We deﬁne a document D =
Revised: We deﬁne a document D =

Original: [ S 1 , . . .
Revised: [ S 1 , . . .

Original: , S N ] as a sequence of N sentences.
Revised: , S N ] as a sequence of N sentences.

Original: We take a sentence S of length T to be a string of tokens t 1 · · · t T where each token t i is taken from the vocabulary V .
Revised: We take a sentence S of length T to be a string of tokens t 1 · · · t T where each token t i is taken from the vocabulary V .

Original: Let span ( S ) =
Revised: Let span ( S ) =

Original: { m 1 , m 2 , . . .
Revised: { m 1 , m 2 , . . .

Original: } be the set of spans in the sentence S .
Revised: } be the set of spans in the sentence S .

Original: Here, a span is a subsequence of the tokens in S = t 1 · · · t T .
Revised: Here, a span is a subsequence of the tokens in S = t 1 · · · t T .

Original: Let us assume that we are interested in K discourse categories .
Revised: Let us assume that we are interested in K discourse categories .

Original: These categories, each, capture a discourse phenomena of interest.
Revised: These categories, each, capture a discourse phenomena of interest.

Original: As shown in
Revised: As shown in

Original: Section 3, named entity inconsistency, tense inconsistency and pronoun ellipsis make up the majority of discourse errors (67.8%) on the data analyzed.
Revised: Section 3, named entity inconsistency, tense inconsistency and pronoun ellipsis make up the majority of discourse errors (67.8%) on the data analyzed.

Original: We, therefore, introduce three types of categories:
Revised: We, therefore, introduce three types of categories:

Original: ENTITY , TENSE and PRONOUN .
Revised: ENTITY , TENSE and PRONOUN .

Original: In addition, we introduce discourse markers DM as a category, which are the essential contextual links between the various discourse segments (See Figure 3).
Revised: In addition, we introduce discourse markers DM as a category, which are the essential contextual links between the various discourse segments (See Figure 3).

Original: For a certain discourse category of interest, k , we assume that there are L k features .
Revised: For a certain discourse category of interest, k , we assume that there are L k features .

Original: In our case, the features of ENTITY E are a list of named entities in D ; the features of TENSE are
Revised: In our case, the features of ENTITY E are a list of named entities in D ; the features of TENSE are

Original: V =
Revised: V =

Original: [ MD,VBD,VBN,VBP,VBZ,VBG,VB ] , 8 the features of PRONOUN are P =
Revised: [ MD,VBD,VBN,VBP,VBZ,VBG,VB ] , 8 the features of PRONOUN are P =

Original: [ masculine , feminine , neuter , epicene ] 9 ; the features of DM are M = [ contigency , temporal , expansion , comparison ] 10 .
Revised: [ masculine , feminine , neuter , epicene ] 9 ; the features of DM are M = [ contigency , temporal , expansion , comparison ] 10 .

Original: Note that different categories can have different numbers of features and the number of features can be dynamic:
Revised: Note that different categories can have different numbers of features and the number of features can be dynamic:

Original: E depends on D while V and P are ﬁxed.
Revised: E depends on D while V and P are ﬁxed.

Original: The intuition behind this is that we want to encourage the system output to keep consistent tense and pronouns as well as the consistent translation for a speciﬁc named entity.
Revised: The intuition behind this is that we want to encourage the system output to keep consistent tense and pronouns as well as the consistent translation for a speciﬁc named entity.

Original: Let us now deﬁne C k,l ( S ) ⊆ span ( S ) as the set of spans in S that share the l th feature in the k th discourse category.
Revised: Let us now deﬁne C k,l ( S ) ⊆ span ( S ) as the set of spans in S that share the l th feature in the k th discourse category.

Original: To give a concrete example, let us assume that TENSE is the k th category and VBD is the l th feature in this category.
Revised: To give a concrete example, let us assume that TENSE is the k th category and VBD is the l th feature in this category.

Original: The corresponding C k,l ( S ) is the set of the spans (in this case, unigrams) tagged with VBD in the sentence S .
Revised: The corresponding C k,l ( S ) is the set of the spans (in this case, unigrams) tagged with VBD in the sentence S .

Original: In Figure 3, all the spans are colored.
Revised: In Figure 3, all the spans are colored.

Original: We then let C k ( S ) refer to a vector of size
Revised: We then let C k ( S ) refer to a vector of size

Original: L k where each element of that vector is the set
Revised: L k where each element of that vector is the set

Original: C k,l ( S ) .
Revised: C k,l ( S ) .

Original: The sets of spans for ENTITY , TENSE ,
Revised: The sets of spans for ENTITY , TENSE ,

Original: PRONOUN and DM can be produced by a NER model, a POS tagger, a rule-based string match and a discourse marker, respectively.
Revised: PRONOUN and DM can be produced by a NER model, a POS tagger, a rule-based string match and a discourse marker, respectively.

Original: We also deﬁne a weight vector w k = [ w kl : ∀ l = 1 , 2 , . . .
Revised: We also deﬁne a weight vector w k = [ w kl : ∀ l = 1 , 2 , . . .

Original: L k ] for each discourse category k , where each entry w k,l corresponds to the weight given to a feature.
Revised: L k ] for each discourse category k , where each entry w k,l corresponds to the weight given to a feature.

Original: We then deﬁne the discourse representation of sentence S as the concatenation of all categories:
Revised: We then deﬁne the discourse representation of sentence S as the concatenation of all categories:

Original: [Equation]
Revised: [Equation]

Original: Similarity
Revised: Similarity

Original: Let sim : C ( S s ) × C
Revised: Let sim : C ( S s ) × C

Original: ( S r ) → R K represent a similarity vector which measures categorywise similarity between the discourse representations of two sentences S r and S s .
Revised: ( S r ) → R K represent a similarity vector which measures categorywise similarity between the discourse representations of two sentences S r and S s .

Original: Each entry of the vector sim takes non-negative values: the entry being zero if S s and S r have no common spans with that discourse category.
Revised: Each entry of the vector sim takes non-negative values: the entry being zero if S s and S r have no common spans with that discourse category.

Original: The similarity vector sim deﬁned here can be implemented in several ways.
Revised: The similarity vector sim deﬁned here can be implemented in several ways.

Original: A possible implementation of sim can be achieved by counting the number of functionally similar spans for each feature and then taking a weighted sum over all features.
Revised: A possible implementation of sim can be achieved by counting the number of functionally similar spans for each feature and then taking a weighted sum over all features.

Original: [Equation]
Revised: [Equation]

Original: where each entry sim k is deﬁned as follows:
Revised: where each entry sim k is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: count ( Ck
Revised: count [Equation] C

Original: [Equation]
Revised: [Equation]

Original: S s )) , count ( C k ( S r ))) where count ( C k ( · )) =
Revised: S s )) , count ( C k ( S [Equation] ))) where count ( C k ( · )) =

Original: [ | C k,l ( · ) | : ∀ l = 1 · · · L k ] denotes the cardinality of C k,l applied entry-wise and min denotes the minimum function applied element-wise.
Revised: [ | C k,l ( · ) | : ∀ l = 1 · · · L k ] denotes the cardinality of C k,l applied entry-wise and min denotes the minimum function applied element-wise.

Original: Intuitively, sim k , measures the number of functionally similar spans shared by S s and S r .
Revised: Intuitively, sim k , measures the number of functionally similar spans shared by S s and S r .

Original: Assume that TENSE is the k th category.
Revised: Assume that TENSE is the k th category.

Original: In Figure 3, sim k ( U MTA b , U REF b ) = 0 since MTA mistranslated all the verbs into the present tense due to the ignorance of context.

Original: The total similarity sim ( D MTA , D REF ) is the (weighted) total functionally similar spans across all features: (1, 2, 4, 0) for ( E , V , P , M ).

Original: Here, we assume the category weights to be uniformly 1.

Original: It is worth noting that there are many other reasonable ways to operationalize sim .
Revised: It is worth noting that there are many other reasonable ways to operationalize sim .

Original: For ENTITY , partial credit could be assigned to two named entities if they have overlapping tokens; for TENSE and PRONOUN , partial credit could be assigned to two similar categories, e.g. VBP and VB ; for DM , partial credit could be assigned according to the sense hierarchy and the conﬁdences in the detected discourse markers.
Revised: For ENTITY , partial credit could be assigned to two named entities if they have overlapping tokens; for TENSE and PRONOUN , partial credit could be assigned to two similar categories, e.g. VBP and VB ; for DM , partial credit could be assigned according to the sense hierarchy and the conﬁdences in the detected discourse markers.

Original: We leave the expansion of the sim deﬁnition to future work.
Revised: We leave the expansion of the sim deﬁnition to future work.

Original: Document-level Similarity Measure
Revised: Document-level Similarity Measure

Original: Now we turn from measuring the similarity at the sentence level to the document level.
Revised: Now we turn from measuring the similarity at the sentence level to the document level.

Original: We ﬁrst lift sim ( · , · ) to measure the similarity between two documents:
Revised: We ﬁrst lift sim ( · , · ) to measure the similarity between two documents:

Original: [Equation]
Revised: [Equation]

Original: where (cid:80) is applied element-wise.
Revised: where (cid:80) is applied element-wise.

Original: We then deﬁne sim ( · , · ) for a system document D s and a set of reference documents D r = { D r 1 , D r 2 , · · · } by aggregating the sim of all sentences in D s and D r :
Revised: We then deﬁne sim ( · , · ) for a system document D s and a set of reference documents D r = { D r 1 , D r 2 , · · · } by aggregating the sim of all sentences in D s and D r :

Original: [Equation]
Revised: [Equation]

Original: Here, ⊕ is a generic aggregator over multiple references, e.g., ⊕ = max , if we take the reference which has the maximum similarity with the system output; or ⊕ = (cid:80) if we sum up the similarity scores of all references.
Revised: Here, ⊕ is a generic aggregator over multiple references, e.g., ⊕ = max , if we take the reference which has the maximum similarity with the system output; or ⊕ = (cid:80) if we sum up the similarity scores of all references.

Original: Again, ⊕ is applied element-wise.
Revised: Again, ⊕ is applied element-wise.

Original: In Figure 3, since we only have one reference, sim ( D MTA , D r ) = sim

Original: [Equation]

Original: MTA

Original: [Equation] D REF ) = (1 , 2 , 4 , 0) . We also reuse the notation sim ( · , · ) for two sets of documents D s and D r :
Revised: We also reuse the notation sim ( · , · ) for two sets of documents D s and D r :

Original: [Equation]
Revised: [Equation]

Original: Note that the similarity vector can also be computed for the same (set of) documents.
Revised: Note that the similarity vector can also be computed for the same (set of) documents.

Original: For example, if sim is implemented as counting the number of functionally similar spans for each feature, then, sim ( D s , D s ) and sim
Revised: For example, if sim is implemented as counting the number of functionally similar spans for each feature, then, sim ( D s , D s ) and sim

Original: ( D r , D r ) denote the total number of spans of each category in the system output and the reference, respectively.
Revised: ( D r , D r ) denote the total number of spans of each category in the system output and the reference, respectively.

Original: In Figure 3, sim ( D MTA , D MTA ) =

Original: (1 , 7 , 6 , 0) and sim ( D r , D r )

Original: =

Original: sim ( D REF , D REF ) =

Original: Scoring
Revised: Scoring

Original: We are now ready to deﬁne the “goodness” of a system output with respect to our discourse phenomena of interest.
Revised: We are now ready to deﬁne the “goodness” of a system output with respect to our discourse phenomena of interest.

Original: We compute the precision, recall and F-measure for all the K discourse categories are deﬁned as follows:
Revised: We compute the precision, recall and F-measure for all the K discourse categories are deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: Here, p , r and F are all K -dimensional vectors; where, the k th element of these vectors represents the precision, recall and F-score for the k th category.
Revised: Here, p , r and F are all K -dimensional vectors; where, the k th element of these vectors represents the precision, recall and F-score for the k th category.

Original: Thus, the addition, multiplication and division operations above are also deﬁned elementwise.
Revised: Thus, the addition, multiplication and division operations above are also deﬁned elementwise.

Original: In Figure 3, recall that sim ( D MTA , D r ) =

Original: (1 , 2 , 4 , 0) , sim ( D MTA , D MTA ) =

Original: (1 , 7 , 6 , 0) and sim ( D r , D r )

Original: = (2 , 7 , 5 , 2) .

Original: Thus, we have: p

Original: [Equation]

Original: [Equation]

Original: B LON D- D
Revised: B LON D- D

Original: Further, we combine the scores of all categories into an overall score by a simple weighted averaging approach, named as B LON DD .
Revised: Further, we combine the scores of all categories into an overall score by a simple weighted averaging approach, named as B LON DD .

Original: By computing B LON D- D , one can distill the document-level translation quality from the sentence-level one.
Revised: By computing B LON D- D , one can distill the document-level translation quality from the sentence-level one.

Original: [Equation]
Revised: [Equation]

Original: where a k denotes the importance weight of the k th category, and p k and r k denote the k th entry of p and r , respectively.
Revised: where a k denotes the importance weight of the k th category, and p k and r k denote the k th entry of p and r , respectively.

Original: Therefore, B LON D-
Revised: Therefore, B LON D-

Original: D .F1 is deﬁned as:
Revised: D .F1 is deﬁned as:

Original: [Equation]
Revised: [Equation]

Revised: Whenever not speciﬁed, we simply use B LON DD to refer to B LON

Original: R
Revised: D- D .F1. F ( D MTA [Equation] [Equation]

Revised: [Equation]

Original: The B LOND-
Revised: [Equation] NA [Equation] B LON

Original:  D scores of MTA in Figure 3 are: ryan
Revised: [Equation] D [Equation] [Equation]

Original: This should be a footnote, I think.
Revised: [Equation] B LON D

Original: [Equation]
Revised: [Equation]

Original: Whenever not speciﬁed, we simply use B LON
Revised: D [Equation] MTA

Original: D-
Revised: [Equation] r [Equation]

Original: D to refer to B LON 
Revised: 12) 14 (27) 14 (45) 14 ( δ 2)= . B LON D [Equation] D

Revised: [Equation]

Original: D- D .F1.
Revised: F1( D MTA [Equation] D r ) = 2 · . 057 · . [Table]

Revised: 057 +

Revised: [Table]

Revised: = .

Original: [Table]
Revised: [Table]

Original: 4. B LON D E : Combining with N-Grams
Revised: 4. B LON D E : Combining with N-Grams

Original: However, focusing on discourse phenomena solely is not enough to provide comprehensive MT evaluation that correlates strongly with human judgments.
Revised: However, focusing on discourse phenomena solely is not enough to provide comprehensive MT evaluation that correlates strongly with human judgments.

Original: Consider the following example: [Equation]
Revised: Consider the following example: (1)

Original: REF Qiao lifted her heavy eyelids
Revised: REF Qiao lifted her heavy eyelids.

Original: MT Qiao scrunched her brows together.
Revised: MT Qiao scrunched her brows together.

Original: The output of MT is far from “good” in terms of adequacy, whereas B LON D- D ( MT ) = 1 , since MT translates both named entities and tenses correctly.
Revised: The output of MT is far from “good” in terms of adequacy, whereas B LON D- D ( MT ) = 1 , since MT translates both named entities and tenses correctly.

Original: Thus, in order to account for sentence-level adequacy of our ﬁnal metric B LON D E , we augment the set of categories and features to include each ngram (for a value of n) as a category and each span of n-tokens as a feature for the n-gram category.
Revised: Thus, in order to account for sentence-level adequacy of our ﬁnal metric B LON D E , we augment the set of categories and features to include each ngram (for a value of n) as a category and each span of n-tokens as a feature for the n-gram category.

Original: Formally, we have:
Revised: Formally, we have:

Original: [Equation]
Revised: [Equation]

Original: The calculation of B LON D E .P, B LON D E .R
Revised: The calculation of B LON D E .P, B LON D E .R

Original: and
Revised: and

Original: B LON D E .F1 is then done exactly in the same manner as B LON D- D .
Revised: B LON D E .F1 is then done exactly in the same manner as B LON D- D .

Original: Whenever not speciﬁed, we simply use B LON D E to refer to B LON D E .F1.
Revised: Whenever not speciﬁed, we simply use B LON D E to refer to B LON D E .F1.

Original: B LON D E covers both discourse coherence features and sentence-level adequacy, thus providing a comprehensive measurement of translation quality.
Revised: B LON D E covers both discourse coherence features and sentence-level adequacy, thus providing a comprehensive measurement of translation quality.

Original: Table 3 compares B LON D E with BLEU using the two MT outputs found in Figure 3.
Revised: Table 3 compares B LON D E with BLEU using the two MT outputs found in Figure 3.

Original: It is striking that BLEU rates MTA higher than MTB given that
Revised: It is striking that BLEU rates MTA higher than MTB given that

Original: MTB is clearly better than MTA to human readers.
Revised: MTB is clearly better than MTA to human readers.

Original: In sharp contrast, their B LON D E scores reﬂect the correct ranking in translation quality.
Revised: In sharp contrast, their B LON D E scores reﬂect the correct ranking in translation quality.

Original: 4. B LON D+: Combining with Human Annotations
Revised: 4. B LON D+: Combining with Human Annotations

Original: B LON D E is highly extensible and it is easy to incorporate human annotations: we can annotate spans related to discourse errors and treat them as categories.
Revised: B LON D E is highly extensible and it is easy to incorporate human annotations: we can annotate spans related to discourse errors and treat them as categories.

Original: The automated detected categories and human annotated categories is then combined by adopting the same weighted averaging approach.
Revised: The automated detected categories and human annotated categories is then combined by adopting the same weighted averaging approach.

Original: We name it as B LON D+.
Revised: We name it as B LON D+.

Original: We hired the same translators who analyzed discourse errors in Sectionto annotate ambiguous and omitted word/phrases on the test set of BWB .
Revised: We hired the same translators who analyzed discourse errors in Sectionto annotate ambiguous and omitted word/phrases on the test set of BWB .

Original:  Experiments
Revised:  Experiments

Original: In this section, we examine the effectiveness of
Revised: In this section, we examine the effectiveness of

Original: B LON D E at the document-level MT evaluation through experiments.
Revised: B LON D E at the document-level MT evaluation through experiments.

Original: We answer the following question:• Do differences in B LON D E reliably reﬂect differences in the document-level translation quality of different MT systems?
Revised: We answer the following question: Do differences in B LON D E reliably reﬂect differences in the document-level translation quality of different MT systems?

Original: To answer this question, we run several MT baselines and compare their B LON D E scores to eleven other metrics:Standard Sentence-level metrics BLEU (Papineni et al., 2002) , METEOR
Revised: To answer this question, we run several MT baselines and compare their B LON D E scores to eleven other metrics:

Original: (Banerjee and Lavie, 2005) , TER Snover et al., 2006)
Revised: Standard Sentence-level metrics

Original: , ROUGE

Original: L

Original: Lin , CIDE R Vedantam et al., 2015).
Revised: BLEU (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005), TER (Snover et al., 2006), ROUGE - L (Lin, 2004), CIDE R (Vedantam et al., 2015).

Original: Document-level metrics
Revised: Document-level metrics

Original: LC and RC (Wong and
Revised: LC and RC (Wong and

Original: Kit, 2012) – these are ratios between the number of lexical cohesion devices (repetition and collocation) and repeated content words over the total number of content words in a target document.
Revised: Kit, 2012) – these are ratios between the number of lexical cohesion devices (repetition and collocation) and repeated content words over the total number of content words in a target document.

Original: Theses are direct measurements of lexical cohesion.
Revised: Theses are direct measurements of lexical cohesion.

Original: Embedding- based
Revised: Embedding-based metrics

Original: metrics SKIP (SkipThought cosine similarity (Kiros et al., 2015)), AVER (Embedding average cosine similarity (Sharma et al., 2017)), VECTOR (Vector extrema cosine similarity (Forgues et al., 2014)), GREEDY (Greedy Match (Rus and Lintean, 2012)).
Revised: SKIP (SkipThought cosine similarity (Kiros et al., 2015)), AVER (Embedding average cosine similarity (Sharma et al., 2017)), VECTOR (Vector extrema cosine similarity (Forgues et al., 2014)), GREEDY (Greedy Match (Rus and Lintean, 2012)).

Original: 5. MT Systems
Revised: 5. MT Systems

Original: We test B LON D E on the following system outputs: a SMT system (Chiang, 2007), three wellknown commercial NMT systems ( OMT - A , OMT - B ,
Revised: We test B LON D E on the following system outputs: a SMT system (Chiang, 2007), three wellknown commercial NMT systems ( OMT - A , OMT - B ,

Original: OMT - C ), a sentence-level transformer-based system ( MT - S ) and a document-level system ( MTD ) trained on BWB .
Revised: OMT - C ), a sentence-level transformer-based system ( MT - S ) and a document-level system ( MTD ) trained on BWB .

Original: MT - D (Zhang et al., 2018) trains sentence-level model parameters and then estimates document-level model parameters while keeping the sentence-level Transformer model parameters ﬁxed.
Revised: MT - D (Zhang et al., 2018) trains sentence-level model parameters and then estimates document-level model parameters while keeping the sentence-level Transformer model parameters ﬁxed.

Original: We adopt Transformer Big (Vaswani et al., 2017) for both MT - S and MT - D .
Revised: We adopt Transformer Big (Vaswani et al., 2017) for both MT - S and MT - D .

Original: The ﬁnal “system” is a human post-editing ( PE ) on
Revised: The ﬁnal “system” is a human post-editing ( PE ) on

Original: OMT - C provided by professional translators, so it is supposed to be the strongest baseline.
Revised: OMT - C provided by professional translators, so it is supposed to be the strongest baseline.

Original: 5. The B LON D E Evaluation
Revised: 5. The B LON D E Evaluation

Original: Firstly, we leverage the test set of BWB and evaluate the above-mentioned systems by B LON D E and other metrics.
Revised: Firstly, we leverage the test set of BWB and evaluate the above-mentioned systems by B LON D E and other metrics.

Original: Figure 4 presents the means of all metrics along with the 95% conﬁdence interval estimated from bootstrap resampling.
Revised: Figure 4 presents the means of all metrics along with the 95% conﬁdence interval estimated from bootstrap resampling.

Original: We observe that the B LON D E scores demonstrate an "exponentially" increasing trend from sentence-level towards document-level and human post-editing, while the trends of standard metrics are mostly linear.
Revised: We observe that the B LON D E scores demonstrate an "exponentially" increasing trend from sentence-level towards document-level and human post-editing, while the trends of standard metrics are mostly linear.

Original: Specifically, the difference between the B LON D E scores of MT - S and MT - D (denoted as ∆ ( MT - S , MT - D )) is signiﬁcantly higher than the difference between the ∆ ( MT - S , MT - D ) in their BLEU scores.
Revised: Specifically, the difference between the B LON D E scores of MT - S and MT - D (denoted as ∆ ( MT - S , MT - D )) is signiﬁcantly higher than the difference between the ∆ ( MT - S , MT - D ) in their BLEU scores.

Original: An even larger ∆ between MT - D and PE in their B LON D E scores is observed, indicating MT - D is still far away from achieving human parity.
Revised: An even larger ∆ between MT - D and PE in their B LON D E scores is observed, indicating MT - D is still far away from achieving human parity.

Original: Note that the trend of B LON
Revised: Note that the trend of B LON

Original: D-
Revised: D-

Original: D scores is even more "exponential", in- dicating B LON
Revised: D scores is even more "exponential", in- dicating B LON

Original: D- D indeed distills document-level translation quality.
Revised: D- D indeed distills document-level translation quality.

Original: The paired t-statistics of individual documents are given in Table 4.
Revised: The paired t-statistics of individual documents are given in Table 4.

Original: Unlike BLEU , METEOR and other metrics, which either fails to distinguish human and machine translation or has lower discriminative power compared to distinguishing different machine translations, the B LON D E family maintain similar discriminative power across the three pair comparisons.
Revised: Unlike BLEU , METEOR and other metrics, which either fails to distinguish human and machine translation or has lower discriminative power compared to distinguishing different machine translations, the B LON D E family maintain similar discriminative power across the three pair comparisons.

Original: Interestingly but not surprisingly, the non-reference-based LC and RC fail to distinguish neither ( MT - S , MT - D ) nor ( MT - D , PE ), since sentence-level MT is by nature more repetitive than human translation, thus hard to distinguish accidental repetition from document-level cohesion.
Revised: Interestingly but not surprisingly, the non-reference-based LC and RC fail to distinguish neither ( MT - S , MT - D ) nor ( MT - D , PE ), since sentence-level MT is by nature more repetitive than human translation, thus hard to distinguish accidental repetition from document-level cohesion.

Original: In addition, the t-statistics of B LON
Revised: In addition, the t-statistics of B LON

Original: D- D categories provide rich diagnostic information.
Revised: D- D categories provide rich diagnostic information.

Original: As can be seen, although transformer-based NMT models have substantially higher BLEU scores than SMT systems, MT - S is not statistically superior to SMT in terms of named entity translation.
Revised: As can be seen, although transformer-based NMT models have substantially higher BLEU scores than SMT systems, MT - S is not statistically superior to SMT in terms of named entity translation.

Original: However, human post-editing scores signiﬁcant better machine translations in entity translation – meaning that named entity translation accounts for a substantial part of quality differences between machine and human.
Revised: However, human post-editing scores signiﬁcant better machine translations in entity translation – meaning that named entity translation accounts for a substantial part of quality differences between machine and human.

Original: In terms of TENSE and and DM translation,
Revised: In terms of TENSE and and DM translation,

Original: MT - D is not doing signiﬁcantly better than MT - S , which could be taken into consideration in future document-level MT model designs.
Revised: MT - D is not doing signiﬁcantly better than MT - S , which could be taken into consideration in future document-level MT model designs.

Original: We also show the pairwise Pearson correlations between different metrics in Figure 5.
Revised: We also show the pairwise Pearson correlations between different metrics in Figure 5.

Original: It illustrates the homogeneity/heterogeneity of different metrics.
Revised: It illustrates the homogeneity/heterogeneity of different metrics.

Original: We report the absolute value of correlation for TER as it aims for a strong negative correlation with human assessment.
Revised: We report the absolute value of correlation for TER as it aims for a strong negative correlation with human assessment.

Original: We see that while sentencelevel metrics ( BLEU , METEOR and ROUGE - L ) have strong correlations with each other, B LON D E correlates less well with those metrics, suggesting their heterogeneity.
Revised: We see that while sentencelevel metrics ( BLEU , METEOR and ROUGE - L ) have strong correlations with each other, B LON D E correlates less well with those metrics, suggesting their heterogeneity.

Original: 5. Human Evaluation
Revised: 5. Human Evaluation

Original: We then evaluate B LON D E along with other metrics in terms of their Pearson correlation with human assessment.
Revised: We then evaluate B LON D E along with other metrics in terms of their Pearson correlation with human assessment.

Original: Our human assessment is provided by four professional Chinese to English translators and four native English revisers.
Revised: Our human assessment is provided by four professional Chinese to English translators and four native English revisers.

Original: Two experimental units ( SENTENCE vs DOCUMENT ) are assessed independently in terms of FLUENCY and ADEQUACY , respectively.
Revised: Two experimental units ( SENTENCE vs DOCUMENT ) are assessed independently in terms of FLUENCY and ADEQUACY , respectively.

Original: In the SENTENCElevel evaluation, we show raters isolated sentences, while in the DOCUMENT -level evaluation, entire documents are presented and we only ask raters to evaluate the overall quality of sequential blocks of sentences (5 sentences per block) as used in the Relative Ranking (RR) evaluation (Bojar et al., 2016).
Revised: In the SENTENCElevel evaluation, we show raters isolated sentences, while in the DOCUMENT -level evaluation, entire documents are presented and we only ask raters to evaluate the overall quality of sequential blocks of sentences (5 sentences per block) as used in the Relative Ranking (RR) evaluation (Bojar et al., 2016).

Original: We use the Williams signiﬁcance test (Williams, 1959; Graham and Baldwin, 2014) following the practice adopted by WMT (Mathur et al., 2020) to identify differences in correlation that are statistically signiﬁcant.
Revised: We use the Williams signiﬁcance test (Williams, 1959; Graham and Baldwin, 2014) following the practice adopted by WMT (Mathur et al., 2020) to identify differences in correlation that are statistically signiﬁcant.

Original: The detailed protocol is presented in Appendix D.
Revised: The detailed protocol is presented in Appendix D.

Original: The results are shown in Table 5.
Revised: The results are shown in Table 5.

Original: B LON D E obtains the highest correlation with human assessment at both the sentence level and the document level.
Revised: B LON D E obtains the highest correlation with human assessment at both the sentence level and the document level.

Original: However, B LON D E correlates remarkably better with human assessment when context is taken into account, and it only signiﬁcantly outperforms all other metrics at document level.
Revised: However, B LON D E correlates remarkably better with human assessment when context is taken into account, and it only signiﬁcantly outperforms all other metrics at document level.

Original: It is worth noting that B LON D E also correlates well with FLUENCY assessment, even though it is, in essence, still a reference-based metric.
Revised: It is worth noting that B LON D E also correlates well with FLUENCY assessment, even though it is, in essence, still a reference-based metric.

Original: One possible explanation for this unexpected positive result is that it tracks span categories that directly relate to cohesion and coherence.
Revised: One possible explanation for this unexpected positive result is that it tracks span categories that directly relate to cohesion and coherence.

Original: Another important observation is that the recall-based B LON D E variants generally correlates better with human assessment, yet work worse in selectivity compared to the precision-based variants (see MT - D → PE in
Revised: Another important observation is that the recall-based B LON D E variants generally correlates better with human assessment, yet work worse in selectivity compared to the precision-based variants (see MT - D → PE in

Original: Table 4).
Revised: Table 4).

Original: This provides support for adopting the
Revised: This provides support for adopting the

Original: F-measure in order to get the best of both worlds.
Revised: F-measure in order to get the best of both worlds.

Original:  Related Work
Revised:  Related Work

Original: There have been a few works on automatic evaluation metrics for speciﬁc discourse phenomena.
Revised: There have been a few works on automatic evaluation metrics for speciﬁc discourse phenomena.

Original: For pronoun translation , Hardmeier and Fed-
Revised: For pronoun translation , Hardmeier and Fed-

Original: [Table] [Table]
Revised: [Table] [Table]

Original: erico (2010) measured the precision and recall of pronouns directly and Miculicich Werlen and
Revised: erico (2010) measured the precision and recall of pronouns directly and Miculicich Werlen and

Original: Popescu-Belis (2017) proposed to estimate the accuracy of pronoun translation (APT) by aligning source and target texts.
Revised: Popescu-Belis (2017) proposed to estimate the accuracy of pronoun translation (APT) by aligning source and target texts.

Original: However, as shown in Guillou and Hardmeier (2018), APT does not take the antecedents of an anaphoric pronoun into account.
Revised: However, as shown in Guillou and Hardmeier (2018), APT does not take the antecedents of an anaphoric pronoun into account.

Original: They cannot handle the mismatches in the numbers of pronouns either.
Revised: They cannot handle the mismatches in the numbers of pronouns either.

Original: Jwalapuram et al.
Revised: Jwalapuram et al.

Original: (2019) also proposed a specialized measure for pronoun evaluation which involves training.
Revised: (2019) also proposed a specialized measure for pronoun evaluation which involves training.

Original: Compared to those metrics, B LON D E does not rely on any alignment or training.
Revised: Compared to those metrics, B LON D E does not rely on any alignment or training.

Original: For lexical cohesion , Wong and Kit (2012) proposed LC and RC.
Revised: For lexical cohesion , Wong and Kit (2012) proposed LC and RC.

Original: Gong et al.
Revised: Gong et al.

Original: (2015) described a cohesion function to measure text cohesion via lexical chain and a gist consistency score based on topic model.
Revised: (2015) described a cohesion function to measure text cohesion via lexical chain and a gist consistency score based on topic model.

Original: However, they fail to distinguish accidental repetition from document-level cohesion.
Revised: However, they fail to distinguish accidental repetition from document-level cohesion.

Original: For discourse relations , Hajlaoui and PopescuBelis (2013) proposed to assessing the accuracy of connective translation (ACT).
Revised: For discourse relations , Hajlaoui and PopescuBelis (2013) proposed to assessing the accuracy of connective translation (ACT).

Original: However, it needs a bilingual dictionary of all possible DM translations, while B LON D E only demands a list of monolingual DMs.
Revised: However, it needs a bilingual dictionary of all possible DM translations, while B LON D E only demands a list of monolingual DMs.

Original: Moreover, B LON D E has higher tolerance of valid drop (Zufferey and Cartoni, 2012), where ACT suffers due to its recall-based exact match.
Revised: Moreover, B LON D E has higher tolerance of valid drop (Zufferey and Cartoni, 2012), where ACT suffers due to its recall-based exact match.

Original: Guzmán et al.
Revised: Guzmán et al.

Original: (2014) and Joty et al.
Revised: (2014) and Joty et al.

Original: (2014) exploited the discourse structure by computing a similarity measure between the discourse trees of reference and system output.
Revised: (2014) exploited the discourse structure by computing a similarity measure between the discourse trees of reference and system output.

Original: Those discourserepresentation-based metrics are indirect, and rely on discourse parsing tools, which are much more inaccurate than syntactic and semantic parsing tools used in B LON D E .
Revised: Those discourserepresentation-based metrics are indirect, and rely on discourse parsing tools, which are much more inaccurate than syntactic and semantic parsing tools used in B LON D E .

Original: Unlike previously proposed metrics, B LON D E does not only focus on one speciﬁc discourse phenomenon, thus having signiﬁcantly higher Pearson correlation coefﬁcients with human assessments.
Revised: Unlike previously proposed metrics, B LON D E does not only focus on one speciﬁc discourse phenomenon, thus having signiﬁcantly higher Pearson correlation coefﬁcients with human assessments.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we build a large-scale parallel dataset for document-level translation, BWB .
Revised: In this paper, we build a large-scale parallel dataset for document-level translation, BWB .

Original: We analyze it for common document-level translation errors in practice and propose B LON D E , an interpretable automatic metric for document-level MT evaluation.
Revised: We analyze it for common document-level translation errors in practice and propose B LON D E , an interpretable automatic metric for document-level MT evaluation.

Original: We further improve B LON D E by diagnosing and distilling discourse-related errors in MT outputs and human-annotations to obtain two improved metrics B LON D- D and B LON D+.
Revised: We further improve B LON D E by diagnosing and distilling discourse-related errors in MT outputs and human-annotations to obtain two improved metrics B LON D- D and B LON D+.

Original: These metrics were shown to have better selectivity than various sentence-level metrics and correlate better with human judgments.
Revised: These metrics were shown to have better selectivity than various sentence-level metrics and correlate better with human judgments.

Original: Ethical Considerations
Revised: Ethical Considerations

Original: The annotators were paid a fair wage and the annotation process did not solicit any sensitive information from the annotators.
Revised: The annotators were paid a fair wage and the annotation process did not solicit any sensitive information from the annotators.

Original: Finally, while our approach is not tuned for any speciﬁc real-world application, the approach could be used in sensitive contexts such as legal or health-care settings, and any work must use our approach undertake extensive quality-assurance and robustness testing before using it in their setting.
Revised: Finally, while our approach is not tuned for any speciﬁc real-world application, the approach could be used in sensitive contexts such as legal or health-care settings, and any work must use our approach undertake extensive quality-assurance and robustness testing before using it in their setting.

Original: Replicability: As part of our contributions, we will release the annotated BWB test set, and release the crawling script of the training set under
Revised: Replicability: As part of our contributions, we will release the annotated BWB test set, and release the crawling script of the training set under

Original: Fair Use rules.
Revised: Fair Use rules.

Original: The B LON D E package will also be made publically available.
Revised: The B LON D E package will also be made publically available.

Revised: Acknowledgements

Revised: Mrinmaya Sachan acknowledges support from an

Revised: ETH Zürich Research grant (ETH-19 21-1) and a grant from the Swiss National Science Foundation (project #201009) for partially supporting this work.

Original: A Dataset Creation The Background of Translators
Revised: A Dataset Creation The Background of Translators

Original: The original
Revised: The original

Original: Chinese books are translated by professional native
Revised: Chinese books are translated by professional native

Original: English speakers, and are corrected by editors.
Revised: English speakers, and are corrected by editors.

Original: Data Collection
Revised: Data Collection

Original: This process is implemented by a python web crawler, and certain data cleaning is also done in the process.
Revised: This process is implemented by a python web crawler, and certain data cleaning is also done in the process.

Original: We crawl the books chapter by chapter, and convert the text to UTF-8.
Revised: We crawl the books chapter by chapter, and convert the text to UTF-8.

Original: After deduplication, we remove the chapters with less than 5 sentences.
Revised: After deduplication, we remove the chapters with less than 5 sentences.

Original: We further remove the titles of each chapter, because most of them are neither translated properly nor in the document-level.
Revised: We further remove the titles of each chapter, because most of them are neither translated properly nor in the document-level.

Original: Alignment and Quality Control
Revised: Alignment and Quality Control

Original: After collecting the web books, we align the bilingual books chapter by chapter according to the indices, while removing those chapters without parallel data.
Revised: After collecting the web books, we align the bilingual books chapter by chapter according to the indices, while removing those chapters without parallel data.

Original: Then, we use Bluealign, which is an MT-based sentence alignment tool, to align the chapters into parallel sentences, while retaining the document-level information.
Revised: Then, we use Bluealign, which is an MT-based sentence alignment tool, to align the chapters into parallel sentences, while retaining the document-level information.

Original: We further deduplicate the parallel corpus and ﬁlter the pairs with a sequence ratio of 3.0.
Revised: We further deduplicate the parallel corpus and ﬁlter the pairs with a sequence ratio of 3.0.

Original: The scale of the ﬁnal corpus is 384 books with 9,581,816 sentence pairs (a total of 460 million words).
Revised: The scale of the ﬁnal corpus is 384 books with 9,581,816 sentence pairs (a total of 460 million words).

Original: To estimate the accuracy of this process, we hired 4 bilingual graduate students to manually evaluate 163 randomly selected documents from the resulting BWB parallel corpus.
Revised: To estimate the accuracy of this process, we hired 4 bilingual graduate students to manually evaluate 163 randomly selected documents from the resulting BWB parallel corpus.

Original: These students are native Chinese speakers who are proﬁcient in
Revised: These students are native Chinese speakers who are proﬁcient in

Original: English.
Revised: English.

Original: More speciﬁcally, they were asked to distinguish whether a document is well aligned at the sentence level by counting the number of misalignment.
Revised: More speciﬁcally, they were asked to distinguish whether a document is well aligned at the sentence level by counting the number of misalignment.

Original: For example, if Line 39 in English actually corresponds to Line 39 and Line 40 in Chinese, but the tool made a mistake that it combines the two sentences, it is identiﬁed as a misalignment.
Revised: For example, if Line 39 in English actually corresponds to Line 39 and Line 40 in Chinese, but the tool made a mistake that it combines the two sentences, it is identiﬁed as a misalignment.

Original: We observe an alignment accuracy rate of 93.1%.
Revised: We observe an alignment accuracy rate of 93.1%.

Original: We further asked the same batch of annotators to correct such misalignments in both the development and the test set.
Revised: We further asked the same batch of annotators to correct such misalignments in both the development and the test set.

Original: The annotation result shows that 7.3% lines are corrected.
Revised: The annotation result shows that 7.3% lines are corrected.

Original: B Error Analysis and B LON D+ Annotation
Revised: B Error Analysis and B LON D+ Annotation

Original: Error analysis and B LON D+ annotation are conducted together.
Revised: Error analysis and B LON D+ annotation are conducted together.

Original: This task is conducted by eight professional Chinese-English translators who are native in Chinese and ﬂuent in English.
Revised: This task is conducted by eight professional Chinese-English translators who are native in Chinese and ﬂuent in English.

Original: The guideline is as follows:
Revised: The guideline is as follows:

Original: • First, identify cases which have translation errors.
Revised: • First, identify cases which have translation errors.

Original: The annotators are instructed to mark examples as “translations with no error” only if it satisﬁes the criteria of both adequacy and ﬂuency as well as satisﬁes the criterion that it is coherent in the context.
Revised: The annotators are instructed to mark examples as “translations with no error” only if it satisﬁes the criteria of both adequacy and ﬂuency as well as satisﬁes the criterion that it is coherent in the context.

Original: • Second, identify whether the translation contains document-level error or sentence-level error (or both).
Revised: • Second, identify whether the translation contains document-level error or sentence-level error (or both).

Original: The annotators are instructed to mark examples as “cases with sentencelevel errors” when they are not adequate or ﬂuent as stand-alone sentences; while “document-level errors” mean those errors that cause the example violating the global criterion of coherence.
Revised: The annotators are instructed to mark examples as “cases with sentencelevel errors” when they are not adequate or ﬂuent as stand-alone sentences; while “document-level errors” mean those errors that cause the example violating the global criterion of coherence.

Original: • Third, categorize the examples with document-level errors according to the linguistic phenomena that lead to errors in MT outputs when considering context.
Revised: • Third, categorize the examples with document-level errors according to the linguistic phenomena that lead to errors in MT outputs when considering context.

Original: We ﬁrst conduct a test annotation and observe that the annotators categorize document-level errors into mainly into 3 categories, namely inconsistency, ellipsis, and ambiguity.
Revised: We ﬁrst conduct a test annotation and observe that the annotators categorize document-level errors into mainly into 3 categories, namely inconsistency, ellipsis, and ambiguity.

Original: According to this observation, we instruct annotators to mark documentlevel errors as inconsistency , ellipsis , and ambiguity , or other document-level error during the annotation process for the entire test set.
Revised: According to this observation, we instruct annotators to mark documentlevel errors as inconsistency , ellipsis , and ambiguity , or other document-level error during the annotation process for the entire test set.

Original: In the formal annotation process, we also added the requirement to annotate B LON D+ spans.
Revised: In the formal annotation process, we also added the requirement to annotate B LON D+ spans.

Original: The detailed requirement is as follows:
Revised: The detailed requirement is as follows:

Original: • Third, categorize the examples with document-level into 4 categories: inconsistency, ellipsis, and ambiguity, or other document-level error which cannot be categorized.
Revised: • Third, categorize the examples with document-level into 4 categories: inconsistency, ellipsis, and ambiguity, or other document-level error which cannot be categorized.

Original: Fourth, if the example is categorized as ambiguity , mark the speciﬁc word/phrase in the reference (English) that cause ambiguity and give the correct word/phrase.
Revised: Fourth, if the example is categorized as ambiguity , mark the speciﬁc word/phrase in the reference (English) that cause ambiguity and give the correct word/phrase.

Original: • Fifth, if the example is categorized as ellipsis and it is not related to pronouns or discourse markers, mark the omitted word/phrase in the reference (English).
Revised: • Fifth, if the example is categorized as ellipsis and it is not related to pronouns or discourse markers, mark the omitted word/phrase in the reference (English).

Original: C Human Post-Editing
Revised: C Human Post-Editing

Original: This task is conducted by the same eight professional Chinese-English translators who carry out the annotation in Appendix B.
Revised: This task is conducted by the same eight professional Chinese-English translators who carry out the annotation in Appendix B.

Original: We asked them to follow guidelines for achieving “good enough”
Revised: We asked them to follow guidelines for achieving “good enough”

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Barrault et al.,
Revised: Barrault et al.,

Original: [Table]
Revised: [Table]

Original: OpenSubtitles (Lison et al., 2018)
Revised: OpenSubtitles (Lison et al., 2018)

Original: [Table]
Revised: [Table]

Original: TED (Ansari et al.,
Revised: TED (Ansari et al.,

Original: [Table]
Revised: [Table]

Original: quality at the sentence-level (comprehensible, accurate but as not being stylistically compelling) but especially pay attention to document-level errors and correct them.
Revised: quality at the sentence-level (comprehensible, accurate but as not being stylistically compelling) but especially pay attention to document-level errors and correct them.

Original: D The Human Evaluation Protocol
Revised: D The Human Evaluation Protocol

Original: The human evaluation is conducted on outputs of ﬁve systems ( SMT , OMT - B , MT - S , CTX , PE ).
Revised: The human evaluation is conducted on outputs of ﬁve systems ( SMT , OMT - B , MT - S , CTX , PE ).

Original: We follow the protocol proposed by (Läubli et al., 2018, 2020).
Revised: We follow the protocol proposed by (Läubli et al., 2018, 2020).

Original: We conduct the evaluation experiment with a 2 × 2 mixed factorial design, carrying both DOCUMENT -level and SENTENCE -level evaluation in terms of ADEQUACY and FLUENCY .
Revised: We conduct the evaluation experiment with a 2 × 2 mixed factorial design, carrying both DOCUMENT -level and SENTENCE -level evaluation in terms of ADEQUACY and FLUENCY .

Original: In the SENTENCE -level evaluation, we show raters isolated sentences by random order; while in the DOCUMENT -level evaluation, entire documents are presented and we only ask raters to evaluate a sequence of 5 sequential sentences at a time in order.
Revised: In the SENTENCE -level evaluation, we show raters isolated sentences by random order; while in the DOCUMENT -level evaluation, entire documents are presented and we only ask raters to evaluate a sequence of 5 sequential sentences at a time in order.

Original: To avoid reference bias, the ADEQUACY evaluation is only based on source texts, while no source texts nor references are presented in the FLUENCY evaluation.
Revised: To avoid reference bias, the ADEQUACY evaluation is only based on source texts, while no source texts nor references are presented in the FLUENCY evaluation.

Original: We adopt Relative Ranking (RR): raters are presented with outputs from the aforementioned ﬁve systems, which they are asked to evaluate relative to each other, e.g. to determine system A is better than system B (with ties allowed).
Revised: We adopt Relative Ranking (RR): raters are presented with outputs from the aforementioned ﬁve systems, which they are asked to evaluate relative to each other, e.g. to determine system A is better than system B (with ties allowed).

Original: We use source sentences and documents from the BWB test set, but blind their origins by randomizing both the order in which the system outputs are presented.
Revised: We use source sentences and documents from the BWB test set, but blind their origins by randomizing both the order in which the system outputs are presented.

Original: Note that in the DOCUMENT -level evaluation, the same ordering of systems is used within a document.
Revised: Note that in the DOCUMENT -level evaluation, the same ordering of systems is used within a document.

Original: The order of experimental items is also randomised.
Revised: The order of experimental items is also randomised.

Original: Sentences are randomly drawn from these documents, regardless of their position.
Revised: Sentences are randomly drawn from these documents, regardless of their position.

Original: We also use spam items for quality control (Kittur et al., 2008):
Revised: We also use spam items for quality control (Kittur et al., 2008):

Original: In a small fraction of items, we render one of the ﬁve options nonsensical by randomly shufﬂing the order of all translated words, except for 10% at the beginning and end.
Revised: In a small fraction of items, we render one of the ﬁve options nonsensical by randomly shufﬂing the order of all translated words, except for 10% at the beginning and end.

Original: If a rater marks a spam item as better than or equal to an actual translation, this is a strong indication that they did not read both options carefully.
Revised: If a rater marks a spam item as better than or equal to an actual translation, this is a strong indication that they did not read both options carefully.

Original: On documentlevel, we render one of the ﬁve options nonsensical by randomly shufﬂing the order of all translated sentences, except for the ﬁrst and the last sentence.
Revised: On documentlevel, we render one of the ﬁve options nonsensical by randomly shufﬂing the order of all translated sentences, except for the ﬁrst and the last sentence.

Original: We recruit four professional Chinese to English translators and four native English revisers for the adequacy and ﬂuency conditions respectively.
Revised: We recruit four professional Chinese to English translators and four native English revisers for the adequacy and ﬂuency conditions respectively.

Original: Note that the eight translators are different from those professional translators who carry out the human translation PE .
Revised: Note that the eight translators are different from those professional translators who carry out the human translation PE .

Original: We deliberately invite another group of specialists for human evaluation to avoid making unreasonable judgments biased towards PE .
Revised: We deliberately invite another group of specialists for human evaluation to avoid making unreasonable judgments biased towards PE .

Original: In each condition, each raters evaluate 162 documents (plus 18 spam items) and 162 sentences (plus 18 spam
Revised: In each condition, each raters evaluate 162 documents (plus 18 spam items) and 162 sentences (plus 18 spam

Original: [Table] [Table]
Revised: [Table] [Table]

Original: items).
Revised: items).

Original: We use two non-overlapping sets of documents and two non-overlapping sets of sentences, and each is evaluated by two raters.
Revised: We use two non-overlapping sets of documents and two non-overlapping sets of sentences, and each is evaluated by two raters.

Original: Speciﬁcally, we refer the ﬁrst half of the test set as PART 1 and the second half as PART 2.
Revised: Speciﬁcally, we refer the ﬁrst half of the test set as PART 1 and the second half as PART 2.

Original: Note that PART 1 and
Revised: Note that PART 1 and

Original: PART 2 are chosen from different books.
Revised: PART 2 are chosen from different books.

Original: Each rater evaluates both sentences and documents, but never the same text in both conditions so as to avoid repetition priming (Gonzalez et al., 2011): RATERand RATER 2 conduct the DOCUMENT -level ADEQUACY evaluation on 180 documents sampled from
Revised: Each rater evaluates both sentences and documents, but never the same text in both conditions so as to avoid repetition priming (Gonzalez et al., 2011): RATERand RATER 2 conduct the DOCUMENT -level ADEQUACY evaluation on 180 documents sampled from

Original: PART 1 and the SENTENCE -level ADEQUACY evaluation for PART 2; RATER 3 and RATER 4 conduct the
Revised: PART 1 and the SENTENCE -level ADEQUACY evaluation for PART 2; RATER 3 and RATER 4 conduct the

Original: SENTENCE -level FLUENCY evaluation on 180 documents sampled from PART 1 and the DOCUMENTlevel FLUENCY evaluation for PART 2; RATER 5 and
Revised: SENTENCE -level FLUENCY evaluation on 180 documents sampled from PART 1 and the DOCUMENTlevel FLUENCY evaluation for PART 2; RATER 5 and

Original: RATER 6 conduct the DOCUMENT -level FLUENCY evaluation on 180 documents sampled from PARTand the SENTENCE -level FLUENCY evaluation for
Revised: RATER 6 conduct the DOCUMENT -level FLUENCY evaluation on 180 documents sampled from PARTand the SENTENCE -level FLUENCY evaluation for

Original: PART 2; RATER 7 and RATER 8 conduct the SENTENCE -level FLUENCY evaluation on 180 documents sampled from PART 1 and the DOCUMENTlevel FLUENCY evaluation for PART 2.
Revised: PART 2; RATER 7 and RATER 8 conduct the SENTENCE -level FLUENCY evaluation on 180 documents sampled from PART 1 and the DOCUMENTlevel FLUENCY evaluation for PART 2.

Original: E Statistical Analysis of Human Evaluation
Revised: E Statistical Analysis of Human Evaluation

Original: We calculate Cohen’s kappa coefﬁcient:
Revised: We calculate Cohen’s kappa coefﬁcient:

Original: [Equation]
Revised: [Equation]

Original: where P ( A ) is the proportion of times that two raters agree, and P ( E ) is the likelihood of agreement by chance.
Revised: where P ( A ) is the proportion of times that two raters agree, and P ( E ) is the likelihood of agreement by chance.

Original: We report pairwise inter-rater agreement in Table 8.
Revised: We report pairwise inter-rater agreement in Table 8.

Original: F Experiment Settings F. B LON D E
Revised: F Experiment Settings F. B LON D E

Original: We use the named entity recognition module and the POS tagger of spaCy (Honnibal and Montani, 2017) to implement the categorizing function cat for ENTITY and TENSE , respectively.
Revised: We use the named entity recognition module and the POS tagger of spaCy (Honnibal and Montani, 2017) to implement the categorizing function cat for ENTITY and TENSE , respectively.

Original: We use the script provided by Sileo et al.
Revised: We use the script provided by Sileo et al.

Original: as the discourse marker minor.
Revised: as the discourse marker minor.

Original: F. Model Hyperparameters
Revised: F. Model Hyperparameters

Original: We follow the setup of Transformer big model for
Revised: We follow the setup of Transformer big model for

Original: BWB experiments.
Revised: BWB experiments.

Original: More precisely, the parameters in the big encoders and decoders are N =, the number of heads per layer is h
Revised: More precisely, the parameters in the big encoders and decoders are N =, the number of heads per layer is h

Original: = 16 , the dimensionality of input and output is d model = 1024 , and the inner-layer of a feed-forward networks has dimensionality d ff = 4096 .
Revised: = 16 , the dimensionality of input and output is d model = 1024 , and the inner-layer of a feed-forward networks has dimensionality d ff = 4096 .

Original: The dropout rate is ﬁxed as 0.3.
Revised: The dropout rate is ﬁxed as 0.3.

Original: We adopt Adam optimizer with β 1 = 0 .
Revised: We adopt Adam optimizer with β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: 98 , (cid:15) =
Revised: 98 , (cid:15) =

Original: 10 − 9 , and set learning rate 0 .
Revised: 10 − 9 , and set learning rate 0 .

Original: 1 of the same learning rate schedule as
Revised: 1 of the same learning rate schedule as

Original: Transformer.
Revised: Transformer.

Original: We set the batch size as 6,000 and the update frequency as 16 for updating parameters to imitate 128 GPUs on a machine with 8 V100 GPU.
Revised: We set the batch size as 6,000 and the update frequency as 16 for updating parameters to imitate 128 GPUs on a machine with 8 V100 GPU.

Original: The datasets are encoded by BPE with 60K merge operations.
Revised: The datasets are encoded by BPE with 60K merge operations.