A libaba-Translate C hina’s Submission for WMT 2022 Quality Estimation Shared Task (2022)

Abstract: In this paper, we present our submission to the sentence-level MQM benchmark at Quality Estimation Shared Task, named UniTE (Unified Translation Evaluation). Specifically, our systems employ the framework of UniTE, which combined three types of input formats during training with a pre-trained language model. First, we apply the pseudo-labeled data examples for the continuously pre-training phase. Notably, to reduce the gap between pre-training and fine-tuning, we use data cropping and a ranking-based score normalization strategy. For the fine-tuning phase, we use both Direct Assessment (DA) and Multidimensional Quality Metrics (MQM) data from past years’ WMT competitions. Finally, we collect the source-only evaluation results, and ensemble the predictions generated by two UniTE models, whose backbones are XLM-R and infoXLM, respectively. Results show that our models reach 1st overall ranking in the Multilingual and English-Russian settings, and 2nd overall ranking in English-German and Chinese-English settings, showing relatively strong performances in this year’s quality estimation competition.

## introduction quality estimation (qe) aims at evaluating machine translation without access to a gold-standard reference translation @xcite . different from other evaluation tasks (e.g., metric), qe arranges its process of evaluation via only accessing source input. as the performance of modern machine translation approaches increase @xcite @xcite , the qe systems should better quantify the agreement of cross-lingual semantics on source sentence and translation hypothesis. the evaluation paradigm of qe shows its own potential for real-world applications @xcite @xcite . this paper describes alibaba translate china's submission to the sentence-level mqm benchmark at wmt 2022 quality estimation shared task @xcite . in recent years, pre-trained language models (plms) have shown their strong ability on extracting cross-lingual information @xcite . to achieve a higher correlation with human ratings on the quality of translation outputs, plenty of trainable model-based qe approaches appear, e.g., comet-qe @xcite and qemind @xcite . they both first derive the embeddings assigned with source and hypothesis sentence with given plm, then predict the overall score based on their embeddings with a followed feedforward network. those model-based approaches have greatly facilitated the development of the qe community. however, those models can only handle source-only input format, which neglects the other two evaluation scenarios, i.e., reference-only and source-referencecombined evaluation. more importantly, training with multiple input formats can achieve a higher correlation with human assessments than individually training on specific evaluation scenarios @xcite . those findings indicate that, the qe and metric tasks share plenty of knowledge when identifying the quality of translated outputs, and unifying the functionalities of three evaluation scenarios into one model can also enhance the performance of the evaluation model on each scenario. as a consequence, when building a single model for a sentence-level qe task, we use the pipeline of unite @xcite , which integrates source-only, reference-only, and source-referencecombined translation evaluation ability into one single model. when collecting the system outputs for wmt 2022 quality estimation shared task, we employ our unite models to predict the translation quality scores following a sourceonly setting. as for the training data, we collect synthetic data examples as supervision for continuous pre-training and apply a dataset pruning strategy to increase the translation quality of the training set. also, during fine-tuning our qe model, we use all available direct assessment (da, @xcite @xcite and multidimensional quality metrics datasets (mqm, @xcite ) from previous wmt competitions to further improve the performance of our model. besides, regarding the applied plm for unite models, we find that for english-russian (en-ru) and chinese-english (zh-en) directions, plm enhanced with cross-lingual alignments @xcite can deliver better results than conventional ones (xlm-r, @xcite . moreover, for each subtask including english to german (en-de), en-ru, zh-en, and multilingual direction evaluations, we build an ensembled qe system to derive more accurate and convincing results as final predictions. our models show impressive performances in all translation directions. when only considering the primary metric -spearman's correlation, we get 2nd, 3rd, and 3rd place in en-ru, zh-en, and multilingual direction, respectively. more notably, when taking all metrics into account, despite the slight decrease in spearman's correlations, our systems show outstanding overall performance than other systems, achieving 1st place in en-ru and multilingual, and 2nd in en-de and zh-en direction. ## method as outlined in §1, we apply the unite framework @xcite to obtain qe models. we unify three types of input formats (i.e., source-only, reference-only, and source-reference-combined) into one single model during training. while during inference, we only use the source-only paradigm to collect evaluation scores. in this section, we introduce the applied model architecture ( §2.1), synthetic data construction method ( §2.2), and model training strategy ( §2.3). ## analysis in this section, we discuss the effectiveness of all strategies, i.e., data pruning ( §4.1), training data arrangement ( §4.2), backbone selection ( §4.3), and model ensembling methods( §4.4). ## conclusion in this paper, we describe our unite submission for the sentence-level mqm task at wmt 2022. we apply data pruning and a ranking-based scoring strategy to collect massive synthetic data. during training, we utilize three input formats to train our models on our synthetic, da, and mqm data sequentially. besides, we ensemble the two models which consist of two different backbones -xlm-r and infoxlm. experiments show that, our unified training framework can deliver reliable evaluation results on qe tasks, showing the powerful transferability of unite model. for future work, we believe that exploring the domain adaption problem for qe is an essential task. the existing machine translation system has made great progress in the field of domain transferablity @xcite @xcite . nevertheless, the confident evaluation metrics for those translation systems are few to be explored. apart from that, developing a unified framework with high transferability for evaluating translation and other natural language generation tasks @xcite @xcite is quite an interesting direction. notably, we also participated in this year's wmt metrics shared task with the same models. we believe that, the idea of unifying three kinds of translation evaluation functionalities (i.e., source-only, reference-only, and source-reference-combined) into one single model can deliver dominant results on all scenarios. better solutions for achieving this goal are worth to be explored in the future.