Exploring Context-Aware Evaluation Metrics for Machine Translation (2023)

Abstract: Previous studies on machine translation evaluation mostly focused on the quality of individual sentences, while overlooking the important role of contextual information. Although WMT Metrics Shared Tasks have introduced context content into the human annotations of translation evaluation since 2019, the relevant metrics and methods still did not take advantage of the corresponding context. In this paper, we propose a context-aware machine translation evaluation metric called Cont-COMET, built upon the effective COMET framework. Our approach simultaneously considers the preceding and subsequent contexts of the sentence to be evaluated and trains our metric to be aligned with the setting during human annotation. We also introduce a content selection method to extract and utilize the most relevant information. The experiments and evaluation of Cont-COMET on the official test framework from WMT show improvements in both system-level and segment-level assessments.

## introduction automatic evaluation metrics play an important role in the field of machine translation (mt) and promote the advancement of mt research. many traditional metrics, such as bleu @xcite and meteor @xcite , were earlier proposed based on lexical-level matching between the human reference and mtgenerated hypothesis. despite excellent efficiency and ease of use, some recent work @xcite has proven that these string-based metrics like bleu perform poorly in terms of consistency with human evaluation. instead, many metric models @xcite @xcite @xcite @xcite using neural networks have emerged recently and achieved great capabilities for mt evaluation with the help of large-scale human annotations. however, these studies focused primarily on the evaluation of the single sentence, while content of the document that contains it should also be considered. the sentence itself may be semantically ambiguous and thus require surrounding sentences for more proper understanding and evaluation. in fact, some work has improved document-level machine translation by introducing contextual information. furthermore, the wmt shared task @xcite , which mainly contributes to the annotation data of mt evaluation, has incorporated contextual information into the process of human annotation since 2019. unfortunately, most existing methods ignored the effect of context and only trained the metrics at the sentence level. @xcite proposed additionally encoding previous sentences during the calculation of evaluation scores. although they achieved improvement, their attempts were not comprehensive enough and performed poorly on segment-level correlation. in this paper, we further explore context-aware methods for mt evaluation and propose our trained metric, cont-comet. to align with the annotation setting of human evaluation, we improve and train our model with additional encoded contextual information based on the comet @xcite framework, which has shown promising performance. in particular, we simultaneously utilize both the preceding and subsequent nearby contexts and attempt different numbers of sentences. furthermore, given that the document is usually too long for the model to encode entirely, we propose a context selection method to extract the most relevant portion of the document as additional context. we evaluate our metric on the official multidimensional quality metrics (mqm) judgments from wmt21 @xcite and wmt22 @xcite . the experimental results demonstrate our cont-comet achieves great improvements in segment-level assessment and better system-level correlation compared with previous work. sentence 1 …… sentence 2 sentence 3 sentence m current sentence context selection document similarity score selected context context concatenation cross-lingual encoder pooling layer regression estimator evaluation score source reference hypothesis in general, our primary contributions are as follows: (1) we observe the gap in context utilization between current neural metrics and human annotation from wmt and improve the training approach with additional context. (2) we explore different incorporations of surrounding sentences and propose a context selection method to extract more important content. (3) we collect and collate the annotated data with document information and accordingly train our metric cont-comet, which achieves overall improvements. 1 ## our methods we first introduce the comet framework, which was proposed for mt evaluation and achieved superior performance in the annual wmt metrics shared task. the basic architecture consists of three main components: the cross-lingual encoder, layer pooling, and estimator module. to leverage the knowledge of pretrained language models, the xlm-roberta-large (conneau et al., 2020) is served as the cross-lingual encoder. it generates the representation e @xmath0, e l 1 , • • • , e l n ) of the input sequence @xmath1, x 1 , • • • , x n ) for each layer l. then these representations are fused into a single embedding sequence through a trained layer-wise attention mechanism. and the average pooling is employed to derive the final sentence embedding for each input. we let s, r, and h refer to the final embeddings of source, reference, and mt-generated hypothesis, respectively. a feed-forward regression estimator is trained according to the combined vector @xmath2; r; h⊙s; h⊙r; |h-s|; |h-r|], where ⊙ represents the element-wise product. more details about comet can be found in @xcite . ## experiments the following experiments and evaluations are based on the annotated dataset provided by wmt metrics shared tasks foot_0 and mtme toolkit foot_1 . our cont-comet metric has been trained upon the comet-21 model proposed by unbabel foot_2 . the detailed hyper-parameters and training settings are described in appendix a. wmt 2022 were not available. therefore, we collect the publicly released da scores from wmt 2020, matching the document information, as our context-aware training set. moreover, we use the multidimensional quality metrics (mqm) data from wmt 2020 as our validation set, and mqm data from wmt 2021 and wmt 2022 for evaluating our trained metric, the same as previous work. more details are stated in appendix b. ## results and discussions we argue the conclusions of @xcite that using two preceding sentences during inference yields the best performance are not adequately substantiated. their work was based on the findings of previous studies @xcite and they did not attempt more sentences or other context settings. therefore, we conducted similar but more in-depth experiments, taking into account the role of both preceding and subsequent context, as well as different numbers of sentences. the results of the original comet-21 model using additional context sentences during inference without extra training are presented in the results demonstrate that the combined effect of preceding and subsequent context yields better performance when just using the same number of previous sentences. furthermore, although this method without additional training improves system-level performance compared to the original comet-21, as more contexts are used, it exhibits significant declines in correlations at the segment level. ## ablation study furthermore, we conduct the ablation study to verify the effectiveness of different training approaches, with corresponding results shown in ## conclusions in this work, we explore context-aware training for mt automatic evaluation by incorporating context content into the encoding of sentences to be evaluated. we believe that our method can better align with the annotation settings of the existing large-scale annotated dataset, which was ignored in previous studies. additionally, we introduce a context selection method to extract the most relevant content from the document to enhance the contextual information, and train our cont-comet metric accordingly. the experiments and ablation studies prove the effectiveness of context utilization and the overall improvement of our metric. in the future, we hope our work will encourage the community to take notice of context-aware evaluation and conduct more in-depth research, and our proposed metric will be applied to document machine translation studies.