A libaba-Translate C hina’s Submission for WMT 2022 Metrics Shared Task (2022)

Abstract: In this report, we present our submission to the WMT 2022 Metrics Shared Task. We build our system based on the core idea of UNITE (Unified Translation Evaluation), which unifies source-only, reference-only, and source- reference-combined evaluation scenarios into one single model. Specifically, during the model pre-training phase, we first apply the pseudo-labeled data examples to continuously pre-train UNITE. Notably, to reduce the gap between pre-training and fine-tuning, we use data cropping and a ranking-based score normalization strategy. During the fine-tuning phase, we use both Direct Assessment (DA) and Multidimensional Quality Metrics (MQM) data from past years’ WMT competitions. Specially, we collect the results from models with different pre-trained language model backbones, and use different ensembling strategies for involved translation directions.

## introduction translation metric aims at delivering accurate and convincing predictions to identify the translation quality of outputs with access to one or many goldstandard reference translations @xcite @xcite . as the development of neural machine translation research @xcite , the metric methods should be capable of evaluating the high-quality translations at the level of semantics rather than surfance-level features @xcite @xcite . in this paper, we describe alibaba translate china's submissions to the wmt 2022 metrics shared task to deliver a more adequate evaluation solution at the level of semantics. pre-trained language models (plms) like bert @xcite and xlm-r @xcite have shown promising results in identifying the quality of translation outputs. compared to conventional statistical-(e.g., bleu, @xcite , the model-based approaches (e.g., @xcite @xcite show their strong ability on delivering more accurate quality predictions, especially those approaches which apply source sentences as additional input for the metric model @xcite @xcite . specifically, those metric models are designed as a combination of plm and feedforward network, where the former is in charge of deriving representations on input sequence, and the latter predicts the translation quality based on the representation. the metric model, which is trained on synthetic or human annotations following a regressive objective, learns to mimic human predictions to identify the translation quality of the hypothesis sentence. although those model-based metrics have shown promising results in modern applications and translation quality estimation, they still show their own shortcomings as follows. first, they often handle one specific evaluation scenario, e.g., comet serves source-reference-only evaluation, where the source and reference sentence should be concurrently fed to the model for prediction. for the other evaluation scenarios, they hardly give accurate predictions, showing the straits of metric models due to the disagreement between training and inference. besides, recent studies have investigated the feasibility of unifying those evaluation scenarios into one single model, which can further improve the evaluation correlation with human ratings in any scenario among source-only, reference-only, and source-reference-combined evaluation @xcite . this indicates that, training with multiple input formats than a specific one can deliver more appropriate predictions for translation quality identification. more importantly, unifying all translation evaluation functionalities into one single model can serve as a more convenient toolkit in real-world applications. following the idea of @xcite and the experience in previous competition @xcite , we directly use the pipeline of unite @xcite to build models for this year's metric task. each of our models can integrate the functionalities of source-only, reference-only, and source-reference-combined translation evaluation into itself. when collecting the system outputs for the wmt 2022 metrics shared task, we employ our unite models to predict the translation quality scores following the source-referencecombined setting. compared to the previous version of unite @xcite , we reform the synthetic training set for the continuous pretraining phase, raising the ratio of training examples consisting of high-quality hypothesis sentences. also, during fine-tuning our metric model, we apply available direct assessment (da, @xcite @xcite and multidimensional quality metrics datasets (mqm, @xcite ) from previous wmt competitions to further improve the performance of our model. specifically, for each translation direction among english to german (en-de), english to russian (en-ru), and chinese to english (zh-en) directions, we applied different ensembling strategies to achieve a better correlation with human ratings on mqm 2021 dataset. results on wmt 2021 mqm dataset further demonstrate the effectiveness of our method. ## method as outlined in §1, we apply the unite framework @xcite to obtain metric models. we use three types of input formats (i.e., source-only, reference-only, and source-referencecombined) during training. while during inference, we only use the source-reference-combined paradigm to collect evaluation scores. in this section, we introduce the applied model architecture ( §2.1), synthetic data construction method ( §2.2), and model training strategy ( §2.3) for this year's metric competition. ## results and analysis baselines the experimental results are conducted in unite models as to our unite models, replacing the xlm-r backbone with infoxlm plm for metric models does not deliver consistent improvement on average. specifically, for both news and ted domains, the unite model with infoxlm as the backbone shows a better correlation on en-de direction, whereas worse on en-ru and zh-en than xlm-r. in addition, the comet-da-2021 performs best in en-ru direction, where we think the reason lies in the scarcity of en-ru training examples in mqm. in practice, during collecting the ensembled outputs, we mainly use the unite infoxlm models for en-de, and unite xlm-r for en-ru and zh-en. ## conclusion in this paper, we describe our submission unite for the sentence-level metrics shared task at wmt 2022. we apply unite @xcite as the pipeline of our models. during training, we utilize three input formats to train our models on our synthetic, da, and mqm data sequentially. besides, we ensemble the two models which consist of two different backbones -xlm-r and infoxlm. experiments demonstrate the reliability of our model for identifying the quality of translation outputs, whereas the two models whose backbones xlm-r and infoxlm show different performance for different translation directions. for the future work, we think that exploring the feasibility of model-based evaluation metrics for other natural language processing tasks is interesting. we believe that, building reliable evaluation metrics for translation diversity @xcite , domain-specific translation quality @xcite , and natural language generation @xcite @xcite is also of vital importance for the natural language processing community. besides, we also submit the source-only predictions of our models to this year's wmt quality estimation shared task, achieving 1st place on multilingual and en-ru, and 2nd place on en-de and zh-en sub-tracks. this further demonstrates the effectiveness of our unite approach, that unifying all evaluation scenarios into one single model can enhance the model performances on all evaluation tasks. we believe that, the idea of unifying three kinds of translation evaluation functionalities (i.e., source-only, reference-only, and source-referencecombined) into one single model can deliver strong evaluation models on all scenarios. this research topic is worth further exploration in the future.