Embarrassingly Easy Document-Level MT Metrics: How to Convert Any Pretrained Metric into a Document-Level Metric (2022)

Abstract: We present a very simple method for extending pretrained machine translation metrics to incorporate document-level context. We apply our method to four popular metrics: BERTScore, Prism, COMET, and the reference-free metric COMET-QE. We evaluate our document-level metrics on the MQM annotations from the WMT 2021 metrics shared task and find that the document-level metrics outperform their sentence-level counterparts in about 85% of the tested conditions, when excluding results on low-quality human references. Additionally, we show that our document-level extension of COMET-QE dramatically improves accuracy on discourse phenomena tasks, supporting our hypothesis that our document-level metrics are resolving ambiguities in the reference sentence by using additional context.

## introduction automatic evaluation is crucial to the machine translation (mt) community for tracking progress, evaluating new ideas and making modeling choices. while human evaluation is the gold standard for mt evaluation, it is very expensive, and thus most research groups must rely on automatic metrics. current state-of-the-art (sota) metrics are pretrained @xcite , leveraging existing language models (lms) or sequence-to-sequence models to judge how well a hypothesis (i.e. mt system output) conveys the same meaning as a human reference translation. sentences are often ambiguous, and many recent works have demonstrated that incorporating inter-sentential (i.e. document-level) context is beneficial in both mt @xcite and human evaluation of mt @xcite @xcite . a human reference translation is (at least ideally) created taking the entire source document into account. however, just as source sentences are often ambiguous, we hypothesize that human reference sentences also contain ambiguities. thus, when a system output deviates from the human reference, we may need to look at additional context to determine if those deviations are acceptable, in the context of the full document translation. in this study, we present a simple procedure for extending pretrained mt metrics to the document level. prior work has used pretrained models models like bert @xcite to embed a single human reference sentence and hypothesis (e.g. an mt output) sentence. we instead argue that a better representation of the reference or hypothesis sentence can be obtained by providing several sentences of context to the pretrained model, allowing the pretrained model to use surrounding context when embedding each sentence of interest. once the embeddings of the reference or hypothesis sentence have been computed (taking into account surrounding sentence context), the metric is computed in the same manner as the sentence-level metric. 1,2 we apply this method to extend four popular pretrained metrics to the document level: 3 we find that our document-level extensions of these four metrics outperform their sentence-level counterparts in 75% of cases considered. excluding @xmath0, where the human reference is of low quality (see § 4.1), we see improvements in 85% of cases. this provides strong evidence that document-level context is useful in the automatic evaluation of mt. we also conduct analysis to better understand the performance improvement that we observe. we demonstrate that our document-level extension of comet-qe significantly improves over its sentence-level counterpart on targeted tasks evaluating discourse phenomena, namely pronoun resolution and word sense disambiguation (wsd). 4 this finding provides further evidence that our document-level metrics are using context to resolve ambiguities in the reference sentence. we also show that using reference context is better than using context from the mt output, likely because the mt output contains more errors than the reference. in summary, our contributions are: 1. we present a simple but effective method to extend pretrained sentence-level metris to the document level, and apply it to four popular metrics. 2. we show that the proposed document-level metrics tend to have better correlation with human judgments than their sentence-level counterparts. 3. we improve over both comet and comet-qe, which appear to be the previous sota automatic metric and reference-free metric, respectively @xcite . 4. we conduct analysis to show that the improvements observed using our approach can be attributed to better context utilization, and also show that using reference context is better than using context from the hypothesis. our work has parallels in human mt evaluation, where document-level judgments are required to distinguish human translation quality from mt system quality @xcite . @xcite showed that many source sentences are ambiguous, but that ambiguities are often resolved using only a few additional sentences of context. this suggests that we do not need to incorporate very many additional sentences of context into a document-level metric in order to see an improvement in quality. pretrained metrics are metrics which leverage large existing pretrained lms or sequence-tosequence models, and include yisi @xcite , comet @xcite , bertscore @xcite , prism @xcite , bleurt @xcite , and others. pretrained metrics have been shown to consistently outperform surface-level metrics such as bleu @xcite , ter @xcite , and chrf @xcite -see @xcite ; @xcite ; @xcite . prior to the rise of pretrained metrics, several works targeted discourse-level phenomena in mt metrics such as pronominal anaphora @xcite @xcite and lexical cohesion @xcite . for a detailed overview of evaluation of discourse-level phenomena, we direct the reader to @xcite . recently, @xcite proposed blonde, a document-level metric that focuses on discourse phenomena in order to score a translated document. however, we find that blonde substantially under-performs modern pretrained metrics, despite taking advantage of document-level context (see § 5.1). take your heavy jacket + the weather is cold today take your heavy jacket + it is freezing today contextual embedding ... pairwise cosine similarity (excludes reference context) reference context reference context maximum similarity reference context reference context ## method at a high level, we propose a very simple procedure for extending pretrained mt metrics to the document level: as in standard sentence-level metrics, we produce a score for a single hypothesis sentence compared to a single human reference translation sentence. however, we use additional context 5,6 from the reference translation when computing the contextual embeddings for both the hypothesis sentence and reference sentence. once the hypothesis and reference sentence have been embedded, we discard the extra context sentences before computing metric scores following the same process as the corresponding sentence-level metric. additional details are provided for each metric below. for the following discussion, let s refer to the source sentence, h refer to the hypothesis (i.e. mt system output) sentence, r refer to the human reference translation sentence, and let c s , c h and c r refer to the source, hypothesis, and reference context, respectively. ## experiments motivated by the finding of @xcite ; @xcite ; @xcite that two previous sentences are sufficient context to correctly resolve ambiguities in the majority of sentences, we use two previous reference sentences as context unless otherwise noted. sentences are separated using the separator token of each model: @xcite for roberta @xmath1. we use reference context c r as reference for the hypothesis, as opposed to hypothesis context c h . this is done in order to avoid propagation of translation errors (see § 6.1 for an ablation using hypothesis context instead of reference context). ## conclusion we proposed a simple and effective approach to generalize pretrained mt metrics to the document level. we apply our approach to bertscore, prism, comet-qe, and comet-qe, and we believe that it could easily be extended to other pretrained sentence-level metrics. to the best of our knowledge, our work is the first example of pretrained document-level mt metrics. we demonstrate that the use of document-level context in pretrained metrics improves correlation with human judgments, and that the improvements are likely due to fact that the document-level metrics can resolving ambiguities in the reference sentence by using additional context. we present results on mt evaluation but our approach may also be beneficial in other natural language generation (nlg) tasks where discourse phenomena are present (e.g paraphrasing, data to text generation, chatbots, etc). in conclusion, we argue that the mt community (and possibly the greater nlg community) should adopt metrics-such as those presented in this work-which take document-level context into account. this would better align automatic metrics with human evaluation, where document-level judgements have been shown to be more discriminative than sentence-level judgements. we also recommend that future research in metrics explore novel ways to incorporate context.