On Measuring Context Utilization in Document-Level MT Systems (2024)

Abstract: Document-level translation models are usually evaluated using general metrics such as BLEU, which are not informative about the benefits of context. Current work on context-aware evaluation, such as contrastive methods, only measure translation accuracy on words that need context for disambiguation. Such measures cannot reveal whether the translation model uses the correct supporting context. We propose to complement accuracy-based evaluation with measures of context utilization. We find that perturbation-based analysis (comparing models’ performance when provided with correct versus random context) is an effective measure of overall context utilization. For a finer-grained phenomenon-specific evaluation, we propose to measure how much the supporting context contributes to handling context-dependent discourse phenomena. We show that automatically-annotated supporting context gives similar conclusions to human-annotated context and can be used as alternative for cases where human annotations are not available. Finally, we highlight the importance of using discourse-rich datasets when assessing context utilization.

## introduction documents are one of the primary ways in which we produce and consume text. while for some languages, sentences provide a base unit of meaning, there are many sentences that contain discourse phenomena that are difficult to disambiguate at sentence level (figure 1 ). despite the vital need for document-level translation in order to handle context-dependent phenomena, most of the current works on machine translation focus on sentence-level translation. @xcite listed the problem of evaluation as one of the reasons for the inability to move beyond sentence level. in this work, we focus on this problem of evaluation. in particular, we focus on 1code at https://github.com/wafaa014/context-utilization . evaluating document-level translation models based on how well they utilize inter-sentential information provided when translating at the document level. the research on document-level translation evaluation has progressed significantly. early works used general metrics such as bleu @xcite and ter @xcite which proved to be inadequate for capturing improvements in discourse phenomena. subsequent research introduced phenomena-specific automatic metrics and contrastive test suites. @xcite survey includes a comprehensive list of works in this direction. while these metrics provide an accuracy measure of models' performance on phenomena, they do not account for correct context utilization. unlike prior studies, we adopt an interpretable approach to context utilization evaluation. we evaluate models based on the ability to use the correct context, and not only the ability to generate a correct translation without necessarily utilizing the context. to assess models' correct context utilization, we perform a perturbation-based analysis. previous studies in perturbation analysis, such as the works of @xcite , @xcite , and @xcite , were limited to specific architectures, evaluated on particular metrics, and perturbed only the source context. in a more comprehensive study, we analyze performance across various document-level architectures using multiple metrics: bleu, comet @xcite and cxmi @xcite . additionally, our analysis involves perturbing both source and target contexts to examine the influence of both sides. for more fine-grained analysis at the level of a specific discourse phenomenon, @xcite collected annotations of supporting context words from expert translators for the pronoun resolution phenomenon. they propose using such annotations as supervision to guide models' attention. extending their work, we focus on benchmarking contextaware models' performance on the phenomenon. we evaluate models based on the attribution scores of supporting context. to obtain attribution scores, we use one of the state-of-the-art interpretability methods for transformer models: alti+ @xcite . moreover, we use automatically annotated (using coreference resolution models) supporting context as an alternative to human annotated context and show that it gives similar conclusions. using automatic annotations allowed us to scale to different languages and has the potential to extend to other discourse phenomena. as an accuracy measure on discourse phenomena, @xcite proposed a novel systematic approach to tag words in a corpus with specific discourse phenomena and evaluate models' performance using f1 measure. however, they mention that context-aware models make only marginal improvements over context-agnostic models. our analysis reveals that this depends on the richness of the dataset with phenomena, and that challenge sets curated to target context-dependent discourse phenomena are better in distinguishing the differences between models in handling the phenomena. our contributions are the following: ## method our goal is to build interpretable metrics to measure the extent of context utilization in context-aware mt. to this end, we propose two methods: a perturbation analysis and an attribution analysis. ## related work previous studies on evaluating context influence on mt performance often examined specific contextaware architectures or particular discourse phenomena. @xcite explored context effects on the hierarchical attention context-aware mt model, dataset pronouns cohesion formality verb form no. sent. no. tokens @xmath0.4%) 569 (1.4%) 641 (1.5%) -2,271 40,877 contrapro 14,477 (2.4%) 87 (0.01%) 9,710 (1.6%) -70,718 599,197 @xmath1.2%) 150 (0.6%) 329 (1.3%) 787 (3.1%) 1,210 25,638 contrapro 22,810 (2.6%) 195 (0.02%) 10,505 (1.2%) 16,211 (1.8%) 81,989 865,890 in comparing various document-level models, @xcite found performance variation based on tasks, with no universally superior model. they also highlight back-translation's benefit to document-level systems, noting their robustness against sentence-level noise. unlike their general metric approach, we enhance the analysis using perturbation methods and attribution evaluation. in interpreting context's benefits, @xcite quantified the causes of improvements of context-aware models on general test sets using attention scores. they found that context usually acts as a regularization and is rarely utilized in an interpretable way. our work differs in that we use alti+ attribution scores instead of attention scores to interpret models' behaviors. in a concurrent work, @xcite introduced an end-to-end interpretability pipeline for analyzing context reliance in context-aware models. in contrast, we rely on linguistic rules instead of attention weights or gradient norms to extract contextual cues, which we show to align with human annotated cues. additionally, we use attribution scores to compare different mt models, including single-and multi-encoder ones. ## conclusion in this work, we shed light on multiple angles to look from when evaluating context utilization in document-level mt. we use a perturbation-based analysis to investigate correct context utilization. additionally, for phenomena-specific evaluation, we propose using attribution scores as measure context utilization. we suggest calculating the attributions of only the supporting context that is necessary for handling context-dependent phe-nomena. moreover, we show that automatically annotated supporting context is inline with human annotated supporting context and can be used as an alternative. finally, we highlight the importance of using discourse-rich data in evaluation. based on our proposed analysis and evaluation tools, we argue that the single encoder approaches to document-level mt demonstrate a priori better context use while also scoring high for translation quality, suggesting that multi-encoder models need more careful design or tuning as highlighted by @xcite . for future work, we aim to extend attribution evaluation to other discourse phenomena, by designing rules for automatic annotation of supporting context for the phenomena with the aid of linguistic expertise. we would also like to apply our evaluation tools and setups to different documentlevel architectures to provide a solid benchmark of context utilization by context-aware models.