R o BLEURT Submission for WMT 2021 Metrics Task (2021)

Abstract: In this paper, we present our submission to Shared Metrics Task: RoBLEURT (Robustly Optimizing the training of BLEURT). After investigating the recent advances of trainable metrics, we conclude several aspects of vital importance to obtain a well-performed metric model by: 1) jointly leveraging the advantages of source-included model and reference-only model, 2) continuously pre-training the model with massive synthetic data pairs, and 3) fine-tuning the model with data denoising strategy. Experimental results show that our model reaching state-of-the-art correlations with the WMT2020 human annotations upon 8 out of 10 to-English language pairs.

## introduction automatically evaluating the adequacy of machine translation (mt) candidates is crucial for judging the quality of mt systems. n-gram-based metrics, such as bleu @xcite , ter @xcite and chrf++ @xcite , have dominated in the topic of mt metric. despite the success, recent studies @xcite also pointed out that, n-grambased metrics often fail to robustly match paraphrases and capture distant dependencies. as mt systems become stronger in recent decades, these metrics show lower correlations with human judgements, leading the derived results unreliable. one arising direction for metric task is using trainable model to evaluate the semantic consistency between candidates and golden references via predicting scores. bertscore @xcite , bleurt @xcite and comet @xcite have shown higher correlations with human judgements than n-gram-based automatic metrics. benefiting from the powerful pre-trained language models (lms), e.g., bert @xcite , those fine-tuned metric models first derive the representation of each input, then introduce an extra linear regression module to give predicted score which describes to what degree the mt system output adequately expresses the semantic of source/reference contents. furthermore, related work @xcite reports that, metrics which additionally introduces source sentences into inputs can further boost the performance of metric model. to push such "model as a metric" approach further, we present robleurt -robustly optimizing the training of bleurt @xcite , to achieve a better consistency between model predictions and human assessments. specifically, for low-resource scenarios, using only hypotheses and references can give more accurate results, alleviating the sparsity of source-side language; for the high-resource language pairs, we format the model input as the combination of source, hypothesis and reference sentences, making model attending to both source input and target reference when evaluating the consistency of semantics. then, we collect massive pseudo data from real mt engines tagged by pseudo scores with strong baselines for supervised model pre-training. as to the fine-tuning phase, we rescore the noisy wmt metric data of previous years with strong metric baselines, which are then utilized to fine-tune our model. experimental results show that, following the setting of wmt2021 metric task, our robleurt model outperforms the reported results of state-of-the-art metrics on multilingual-to-english language pairs. 2 robleurt ## conclusion in this paper, we describe our submission metric -robleurt, from the perspective of combining multilingual and monolingual language model, continuous pre-training with the massive synthetic data pairs, and fine-tuning with data denoising strategy. experimental results confirms the effectiveness of our pipeline, demonstrating state-of-the-art correlations with the wmt2020 human annotations upon 8 out of 10 to-english language pairs.