BlonDe : An Automatic Evaluation Metric for Document-level Machine Translation (2022)

Abstract: Standard automatic metrics, e.g. BLEU, are not reliable for document-level MT evaluation. They can neither distinguish document-level improvements in translation quality from sentence-level ones, nor identify the discourse phenomena that cause context-agnostic translations. This paper introduces a novel automatic metric BlonDe to widen the scope of automatic MT evaluation from sentence to document level. BlonDe takes discourse coherence into consideration by categorizing discourse-related spans and calculating the similarity-based F1 measure of categorized spans. We conduct extensive comparisons on a newly constructed dataset BWB. The experimental results show that BlonDe possesses better selectivity and interpretability at the document-level, and is more sensitive to document-level nuances. In a large-scale human study, BlonDe also achieves significantly higher Pearson’s r correlation with human judgments compared to previous metrics.

## introduction over the past years neural machine translation (nmt) models have become the models of choice in machine translation (mt; @xcite @xcite . although some recent work @xcite @xcite suggests that nmt has achieved human parity at the sentence level, the reliability of these human-parity claims was quickly contested by @xcite , showing that there is a larger difference between human and machine translation quality when intersentential context is taken into account. therefore, document-level machine translation has received increased attention in the mt community. however, despite various modeling advances, the mt community still lacks an efficient and effective evaluation metric for document-level translation. standard evaluation metrics for mt, e.g., bleu @xcite , ter @xcite and meteor @xcite , focus on the quality of translations at the sentence level and do not consider discourse-level features. thus, test suites that perform context-aware evaluation by targeting characteristic discourse-level phenomena have been proposed @xcite @xcite @xcite @xcite @xcite , inter alia) for document-level mt. however, such test suites need to be re-created for new domains or even language pairs, and their construction can be very labor-intensive. we still lack an easy-to-use automatic metric that can reliably discriminate the quality of document-level translation. in this paper, we curate a large-scale documentlevel parallel corpus (bwb) from heterogeneous data sources, and quantify document-level translation mistakes by performing a large human study. on this dataset, we found that inconsistency, foot_0 ellipsis, and ambiguity were the most noticeable phenomena critical for document-level mt, together amounting to 86.73% of mt mistakes. based on this analysis, we propose blonde, an automatic metric that evaluates translation quality at the document level. at the core of the metric is the similarity-based bijection between subsets of reference and system categories, e.g., pronouns, inflected forms, discourse relations and lexicons, and phrases, e.g., named entities. it computes recall, precision and f1, along with the corresponding measure of n-grams. furthermore, blonde can incorporate human annotation easily by computing scores of human-annotated categories in the same way. we compare blonde with 11 other metrics and demonstrate that blonde is better at distinguishing between context-aware and context-agnostic mt systems. we also observe that the degree to which blonde correlates with sentence-level metrics (e.g., bleu) is lower than the degree to which the sentence-level metrics correlate with each other. this signals that blonde indeed captures additional aspects of translation quality beyond the sentence-level. finally, our human evaluation also reveals significantly higher pearson's r correlation coefficients between blonde and human assessments compared to other metrics. ## bwb: bilingual web book dataset to design a metric that is sensitive to documentlevel phenomena, we first curate a document-level chinese-english parallel corpus, called bwb (bilingual web books). bwb consists of chinese web novels across multiple genres (sci-fi, romance, action, fantasy, comedy, inter alia) and their corresponding english translations crawled from the internet. ## blonde the aforementioned document-level phenomena have little impact on the n-gram statistics of translations. however, as is shown in section 3, they can be key considerations for human readers when evaluating translations at the document level. standard automatic metrics ignore the importance of contextual coherence of translations, which implies that the document-level nuances are not being properly modeled @xcite . in this section, we describe blonde, an automatic metric that explicitly tracks discourse phenomena. ## experiments in this section, we examine the effectiveness of blonde at the document-level mt evaluation through experiments. we answer the following question: do differences in blonde reliably reflect differences in the document-level translation quality of different mt systems? to answer this question, we run several mt baselines and compare their blonde scores to eleven other metrics: standard sentence-level metrics. bleu @xcite , meteor @xcite , ter @xcite , rouge-l @xcite , cider @xcite . ## related work there have been a few studies on automatic evaluation metrics for specific discourse phenomena. pronoun translation. hardmeier and federico (2010) measured the precision and recall of pronouns directly and miculicich werlen and popescu-belis (2017) proposed to estimate the accuracy of pronoun translation (apt) by aligning source and target texts. however, as shown in guillou and hardmeier (2018), apt does not take the antecedents of an anaphoric pronoun into account. they cannot handle the mismatches in the numbers of pronouns either. jwalapuram et al. (2019) also proposed a specialized measure for pronoun evaluation which involves training. in comparison, blonde does not rely on any alignment or training. lexical cohesion. wong and kit (2012) proposed lc and rc. gong et al. (2015) described a cohesion function to measure text cohesion via lexical chain and a gist consistency score based on topic model. however, they fail to distinguish accidental repetition from document-level cohesion. discourse relations. hajlaoui and popescu-belis (2013) proposed to assess the accuracy of connective translation (act). however, such an assessment requires a bilingual dictionary of all possible dm translations, whereas blonde only requires a list of monolingual dms. guzmán et al. (2014) and joty et al. (2014) compute a metric based on the similarity between the discourse trees of reference and system output. those discourserepresentation-based metrics are indirect, and rely on discourse parsing tools, which are much more inaccurate than syntactic and semantic parsing tools used in blonde. unlike previously proposed metrics, blonde does not only focus on one specific discourse phenomenon, and thus has significantly higher pearson correlation coefficients with human assessments. ## conclusion in this paper, we build a large-scale parallel dataset for document-level translation, bwb. we analyze it for common document-level translation errors in practice and propose blonde, an interpretable automatic metric for document-level mt evaluation. we further improve blonde by diagnosing and distilling discourse-related errors in mt outputs and human-annotations to obtain two improved metrics blond-d and blond+. these metrics were shown to have better selectivity than various sentence-level metrics and correlate better with human judgments.