SLIDE : Reference-free Evaluation for Machine Translation using a Sliding Document Window (2024)

Abstract: Reference-based metrics that operate at the sentence-level typically outperform quality estimation metrics, which have access only to the source and system output.This is unsurprising, since references resolve ambiguities that may be present in the source.In this paper, we investigate whether additional source context can effectively substitute for a reference.We present a metric named SLIDE (SLIding Document Evaluator), which operates on blocks of sentences. SLIDE leverages a moving window that slides over each document in the test set, feeding each chunk of sentences into an unmodified, off-the-shelf quality estimation model.We find that SLIDE obtains significantly higher pairwise system accuracy than its sentence-level baseline, in some cases even eliminating the gap with reference-base metrics.This suggests that source context may provide the same information as a human reference in disambiguating source ambiguities. This finding is especially pertinent for reference-free document-level evaluation, wherein SLIDE could provide higher-quality pairwise system assessments while only requiring document boundary annotations.

## introduction the prevailing approach for neural machine translation metrics is to work at the sentence-level, constructing sequences of contextualized encoder states from the source sentence, a reference translation, and a system output. the specific mechanics vary by metric, but a general approach, employed by comet @xcite , is to pool these encodings into separate sentence-level embeddings, concatenate them, and feed them into a regressor, which is trained against human annotations. quality estimation (qe) approaches work similarly, but do not have access to a reference translation. qe metrics typically trail their reference-based counterparts @xcite , for obvious reasons. the default evaluation setting for qe is at the sentence-level. but just as there exist many linguistic phenomena that cannot be translated without context, these same phenomena also cannot be properly evaluated in isolation. as an example, consider the following english sentences with context and their translations into german. where is it? b. ich brauche meinen hut. wo ist er? reference-based evaluation is aided by the fact that the human translation, presumably produced in context, resolves the ambiguity. qe approaches (operating at the sentence-level), on the other hand, cannot correctly score this translation. there are many other document-level phenomena that are also typically captured by references, many of which are subtle and hard to measure @xcite . it therefore stands to reason that providing disambiguating context could be a useful extension to (reference-free) qe metrics. in this work, we are motivated by two related ideas to address this gap: (i) neural metrics often make use of underlying language models trained on wider contexts, which means there is no real impediment to feeding them multiple sentences, and (ii) a sentence's evaluation will differ based on its order in a block of sentences, so it may be helpful to evaluate each sentence in multiple different contexts. we therefore experiment with a strided window approach applied to comet, whose underlying encoder is infoxlm @xcite , trained on wide contexts. we apply a fixedwidth sentence window and slide it across the documents within a test set, accumulating scores of each chunk in normal comet fashion. we experiment with various windows and strides, and find that comet-qe employed in this fashion outperforms its sentence-level reference-based counterparts in many settings. we conclude that this simple extension to qe might be profitably engaged wherever document boundary annotations are available. ## the slide approach our main focus is the system ranking question as posed in @xcite : given two systems, which one is the better one? the task is to score system output over a test set, comprising multiple documents. we define a window size, w, specifying how many sentences to include as input, and a stride, @xmath0, defining how many sentences to advance the window. for a given (w, s) setting, the window is placed at the beginning of the document covering w sentences, and the chunk is sent to comet as a single input. the window is then incremented by s sentences and a new value computed. this proceeds over all documents in a test set. this is depicted in figure 1 . there are a few edge cases that must be considered, both pertaining to situations where we cannot fill a full window: in our initial experiments, we ignore both types of partial documents, i.e., throwing them away, as if they didn't exist in order to evaluate only on chunks with appropriate context available. we will return to this question in section a.1 to consider alternatives to throwing away partial documents. ## related work the vast majority of work in machine translation metrics is focused on sentence-level evaluation. at a high level, it is useful to distinguish two uses of context for machine translation evaluation. in the first setting, contextualized metrics are designed to test document-translation capabilities of a model. chief among these are contrastive test sets, in which a model is tested in its ability to rank good translations from bad ones in a contextualized manner @xcite @xcite . other approaches have employed test-time nlp tool chains to target and reward correct prediction of discourse phenomena @xcite . in the second setting, metrics make use of context to provide or refine their model of what makes a good translation. this is useful even for ranking sentence-based translation systems. the first such system may be doccomet @xcite , which used context to modify the encodings of sentences, but then removed that context before invoking comet's classifier. they looked at both reference-based and qe metrics, and evaluated on pearson's correlation against hu-man scores. context-comet @xcite took an approach similar to that described here, but was not evaluated at all. more recently, @xcite showed that paragraph-level evaluation works just as well for reference-based metrics, even with underlying metrics trained in a sentencelevel fashion. however, they did not experiment with reference-less metrics (the quality estimation task). our approach here fits within this second setting. in contrast to this prior work, our approach requires no changes to the underlying codebase, is evaluated on pairwise system-level accuracy, and focuses on the quality estimation task, where context is the most promising. ## conclusion and future work incorporating context into comet-qe provides critical information that appears to help the metric better adjudicate the difference between systems. even just a single sentence of context drastically improves the ability of the model to discriminate between systems. the method works well even though the scores are accumulated over groups of sentences, which is different from the sentencelevel manner in which comet is trained, and even when the accumulated scores come from overlapping blocks. the results here required no change to the underlying sentence-based evaluators, and in that sense come for free, so long as document boundary annotations are available. we therefore recommend that source context be included with any neural quality estimation metric. an interesting further evaluation would throw some documentcontext translation engines into the mix; we speculate that contextualized quality estimation (like that provided by slide) should help discriminate those systems too. giorgos vernikos, brian thompson, prashant mathur, and marcello federico. 2022. embarrassingly easy document-level mt metrics: how to convert any pretrained metric into a document-level metric. in proceedings of the seventh conference on machine translation (wmt), pages 118-128, abu dhabi, united arab emirates (hybrid). association for computational linguistics. elena voita, rico sennrich, and ivan titov. 2019. when a good translation is wrong in context: context-aware machine translation improves on deixis, ellipsis, and lexical cohesion. in proceedings of the 57th annual meeting of the association for computational linguistics, pages 1198-1212, florence, italy. association for computational linguistics.