Tencent submission for WMT 20 Quality Estimation Shared Task (2020)

Abstract: This paper presents Tencent’s submission to the WMT20 Quality Estimation (QE) Shared Task: Sentence-Level Post-editing Effort for English-Chinese in Task 2. Our system ensembles two architectures, XLM-based and Transformer-based Predictor-Estimator models. For the XLM-based Predictor-Estimator architecture, the predictor produces two types of contextualized token representations, i.e., masked XLM and non-masked XLM; the LSTM-estimator and Transformer-estimator employ two effective strategies, top-K and multi-head attention, to enhance the sentence feature representation. For Transformer-based Predictor-Estimator architecture, we improve a top-performing model by conducting three modifications: using multi-decoding in machine translation module, creating a new model by replacing the transformer-based predictor with XLM-based predictor, and finally integrating two models by a weighted average. Our submission achieves a Pearson correlation of 0.664, ranking first (tied) on English-Chinese.

## introduction the development of machine translation (mt) requires efficient quality evaluation of the outputs. the widely used mt metric bleu @xcite satisfies this demand. however, bleu requires human reference translations which costs labor and time to generate. quality estimation (qe) is an alternative to evaluate the quality of mt outputs with no access to reference translations @xcite . we participate in the sentence-level task in task 2 of the wmt20 qe shared task for english-chinese @xcite . the sentence-level task aims to predict the human-targeted translation edit rate (hter) @xcite of the mt output, which reflects the minimal amount of edits that is needed to post-edit the mt output to an acceptable one, thus denotes the overall quality of the mt output. the classical baseline model quest++ @xcite constructed rule-based features and employed machine learning algorithm to predict hter scores. recent neural networks applied the newly-emerged predictor-estimator architecture to qe tasks. @xcite proposed the first predictor-estimator model to extract feature vectors by incorporating large parallel data into a bilingual rnn model, which is subsequently fed into another bidirectional rnn model to predict qe scores. later on, @xcite replaced the rnn-based predictor by a bidirectional transformer and added 4-dimensional mis-matching features; @xcite used a transformer-dlcl based predictor; and @xcite introduced bert and xlm pretrained predictors into their system. besides the improvements on model architectures, choosing the top-performing models using ensemble techniques further improves the qe performance. for example, the submission using ensemble techniques achieved the best result in the sentence-level qe sub-task in both wmt19 @xcite and ccmt19 @xcite . we submit a predictor-estimator based qe system, which extends the open-source openkiwi framework foot_0 @xcite to take advantage of recently proposed pre-trained models by transfer learning technique. our contributions are as follow: we next describe the models, experiments and results in detail. ## models our models employ predictor-estimator architecture and openkiwi framework. overall, we implement two predictor-estimator architectures, namely xlm-based predictor-estimator and transformerbased predictor-estimator, and ensemble multiple systems to boost performance. ## experiments and results we conducted three sets of experiments on the wmt20 qe english-chinese sentence-level task in task 2. ## conclusion we describe tencent's submission to the wmt20 quality estimation sentence-level task in task 2. our systems are based on predictor-estimator architecture and built upon openkiwi framework. we implement two predictor-estimator architectures, xlm-based predictor-estimator and transformerbased predictor-estimator. for xlm-based predictor-estimator, we produces two kinds of contextual token representation, masked and nonmasked representations. both lstm-estimator and transformer-estimator are conducted to predict the mt output scores by using the features produced from the predictors. top-k strategy and multihead attention strategy are employed to enhance the sentence feature representation. for transformerbased predictor-estimator, we integrate one model based on xlm-based predictor to enhance the overall performance. stacking ensemble is also proved to be more effective than simple averaging integra-tion.