Evaluating Metrics for Document-context Evaluation in Machine Translation (2023)

Abstract: We describe our submission of a new metric, SLIDE (Raunak et al., 2023), to the WMT 2023 metrics task. SLIDE is a reference-free quality-estimation metric that works by constructing a fixed sentence-length window over the documents in a test set, concatenating chunks and then sending them for scoring as a single unit by COMET (Rei et al, 2022). We find that SLIDE improves dramatically over its context-less counterpart on the two WMT22 evaluation campaigns (MQM and DA+SQM).

## introduction reference-based metrics such as comet typically perform far above their source-based quality estimation (qe) counterparts. one explanation is that the human reference provides an answer or grounding to many types of translation ambiguities, such as pronoun selection, that may be impossible to predict from just a single input sentence alone. a handful of approaches have looked at extending metrics with source-and target-side context @xcite @xcite in hopes of providing stronger correlation with human judgments. we base our submission on slide @xcite , which explicitly postulates and provides evidence for the claim that source-side context may work to provide the same information as human references. a new value computed. these values are treated independently, summed and averaged over a test set in typical fashion. documents that are shorter than the window size, and the "remainder" portions of documents that cannot be perfectly tiled by the window and stride, are skipped. in practice, we used a (w, s) value of (6, 6) for all languages except en-de and de-en. for those languages, the data was provided at the paragraph level. we therefore simply took the provided segmentations one-by-one, without providing a window or stride. we chose this value because it had some of the best reported results in raunak et al. (2023, figure 1 ). ## results the wmt23 test set @xcite for each language pair comprises a set of documents containing between 1 and 173 lines, with a mean of 9.7 and a median of 7 across 14 language pairs. at the time of publication, official results were not available, so we cannot comment on how well the strong results from @xcite generalized to the new settings in wmt23. we note also that we discovered after the submission that a bug in our code resulted in debugging output appearing in the data to be scored by comet. this unfortunately affects the scores and means that slide's placement in the official rankings are incorrect. ## conclusion in this system description, we presented our submission to the wmt 2023 metrics task. slide is designed as a reference-free quality-estimation metric which leverages the strength of contextual information by constructing a fixed sentence-length window over documents in a test set. the initial findings from @xcite showcased the potential of slide to deliver enhanced performance over context-less metrics, particularly in the wmt22 evaluation campaigns. while we anticipate the official results from the wmt23 metrics task, bug in our code might have affected slide's standing in the rankings. we believe that slide is a step forward in our collective endeavor to create metrics that align more closely with human judgments. future works may explore optimizing window and stride configurations or integrating advanced algorithms to further exploit the potential of context in quality estimation tasks.