QEM ind: A libaba’s Submission to the WMT 21 Quality Estimation Shared Task (2021)

Abstract: Quality Estimation, as a crucial step of quality control for machine translation, has been explored for years. The goal is to to investigate automatic methods for estimating the quality of machine translation results without reference translations. In this year’s WMT QE shared task, we utilize the large-scale XLM-Roberta pre-trained model and additionally propose several useful features to evaluate the uncertainty of the translations to build our QE system, named . The system has been applied to the sentence-level scoring task of Direct Assessment and the binary score prediction task of Critical Error Detection. In this paper, we present our submissions to the WMT 2021 QE shared task and an extensive set of experimental results have shown us that our multilingual systems outperform the best system in the Direct Assessment QE task of WMT 2020.

## introduction quality estimation (qe) aims to predict the quality of a machine translation (mt) system's output without any access to ground-truth translation references or human intervention @xcite @xcite . automatic methods for qe are highly appreciated in mt applications when we expect to efficiently obtain the quality indications for a larget amount of machine translation outputs in a short time, or even at run-time. this paper describes alibaba's submissions to the wmt 2021 quality estimation shared task. we developed a novel qe system, called qemind, that have been applied to two tasks this year, the sentence-level direct assessment (da) and binary score prediction of critical error detection (ced). common approaches in the previous years heavily focus on human-crafted rule-based feature engineering mode such as quest++ @xcite . the features extracted are usually fed into traditional machine learning algorithms such as a support vector regression for the sentence-level scoring or a sequence-labeling model with conditional random fields for the word-level labeling respectively. with the development of neural networks applied in machine translation and other nlp tasks, a neural predictor-estimator framework for qe was proposed and achieved better results in wmt 2017 and wmt 2018 qe shared tasks @xcite . this framework extensively requires a pre-training procedure with a large amount of parallel corpora in the predictor mode and stacks a downstream estimator mode with additional layers for a supervised regression or classification task. since 2019, state-of-the-art (sota) qe systems @xcite have hit record high with transfer learning by leveraging sota pre-trained nlp neural network models, for example, mbert @xcite and xlm-roberta @xcite . till then, only "black-box" qe methods had been mainly used in wmt qe shared tasks. furthermore, with the accessibility to the nmt systems, some "glass-box" qe features have been explored and verified to bring improvements upon "black-box" approaches @xcite . in addition, @xcite have showed that useful information that are extracted from the mt systems performs good correlation with human judgements of quality. inspired by these works, we propose more useful features in this paper, among which, some are derived from the nmt systems and others are created via utilizing the masked language model of xlm. we develop our qe systems by incorporating all the features that can potentially evaluate the uncertainty of the machine translations into a supervised qe model based on the transfer learning from xlm-roberta. we evaluate our method on the direct assessment qe tasks of wmt 2020 and wmt 2021 and our experiment results demonstrate the efficiency and versatility of the features we have proposed on the quality estimation in different language pairs. ## task & data set we participate the sentence-level direct assessment task and critical error detection tasks of this year's qe shared task. (1) for the da task, we merge 7000 and 1000 labeled data in the training and development data sets as our training set and treat the test20 data set as our development set for each of the seven language pairs. however, for the four zero-shot language pairs, we only have the blind test sets. (2) for the ced task, we observed that the distributions of two classes, not and err, are extremely unbalanced for all four language pairs. therefore, we simply up-sample the samples with err labels to get a relatively balanced training set. this strategy of data augmentation has also been empirically verified to be valid. ## methodology in this section, we provide a complete view of our uncertainty feature enhanced approach, including: ## conclusion this paper introduces our machine translation quality estimation model, qemind, for the sentencelevel direct assessment and critical error detection tasks of wmt 2021. we propose novel features to estimate the uncertainty of machine translations and incorporate them into the transfer learning from the large-scale pre-trained model, xlm-roberta. besides, three important strategies are particularly utilized for improving the qe system's performance such as multilingual training, data augmentation and model ensemble. our system has achieved the first ranking in average pearson correlation across all languages, including the zeroshot ones in the multilingual da task of wmt 2021.