BJTU -Toshiba’s Submission to WMT 22 Quality Estimation Shared Task (2022)

Abstract: This paper presents the BJTU-Toshiba joint submission for WMT 2022 quality estimation shared task. We only participate in Task 1 (quality prediction) of the shared task, focusing on the sentence-level MQM prediction. The techniques we experimented with include the integration of monolingual language models and the pre-finetuning of pre-trained representations. We tried two styles of pre-finetuning, namely Translation Language Modeling and Replaced Token Detection. We demonstrate the competitiveness of our system compared to the widely adopted XLM-RoBERTa baseline. Our system is also the top-ranking system on the Sentence-level MQM Prediction for the English-German language pairs.

## introduction machine translation quality estimation (qe) aims to evaluate the quality of machine translation automatically without reference. compared with commonly used machine translation metrics such as bleu @xcite , qe can be applicable to the case where references are unavailable. it has a wide range of applications in post-editing and quality control for machine translation. this paper introduces in detail the joint submission of beijing jiaotong university and toshiba (china) corporation to the quality estimation shared task in the 7th conference on machine translation (wmt22), and we mainly focus on the task 1: quality prediction. this year, the quality prediction task consists of two annotations (da and mqm) and two levels (sentence-level and wordlevel), and we only participate in the sentencelevel mqm prediction, of which the goal is to predict the mqm score @xcite for each source-target sentence pair. three language pairs are involved: english-german, chinese-english 1 our codes are openly available at the public repository https://github.com/huihuichyan/awesomeqe . and english-russian, with roughly 10k-20k training pairs provided for each direction. our system is mainly based on the ensemble of multiple pre-trained models, both monolingual and multilingual. the monolingual models receive only the target sequence to perform regression (only estimating the target fluency). the multilingual models receive both the source and target sequence to perform regression. we also use in-domain parallel data to pre-finetune the pre-trained models, to adapt their representations to the target language and domain. we try two styles of pre-finetuning, namely translation language model (tlm) and replaced token detection (rtd). the translation language model is to predict the random masked tokens based on the concatenation of source-target pairs. the rtd is to first randomly replace some tokens by another generator, then to detect which token is replaced. different models are ensembled to get further improvement. ## conclusion in this paper, we present our wmt22 qe shared task submission to the sentence-level mqm prediction. we perform massive comparison and demonstrate the effectiveness of monolingual language model. we verify that the pre-trained models can be further improved on target language and target domain via pre-finetuning, and we propose different strategies to pre-finetune the model. as the machine translation has been developing rapidly, the translation errors current mt system makes have also become more than shallow disalignment. while mt systems are mostly trained with massive parallel data, using the same amount of parallel data to train another qe model seems inefficient, and the monolingual knowledge contained in monolingual models can be more helpful than we expected. while previous work mainly rely on the semantic alignment to perform qe, we think it might be a better option to rely more on monolingual fluency in real applications.