HW - TSC ’s Participation at WMT 2020 Quality Estimation Shared Task (2020)

Abstract: This paper presents our work in the WMT 2020 Word and Sentence-Level Post-Editing Quality Estimation (QE) Shared Task. Our system follows standard Predictor-Estimator architecture, with a pre-trained Transformer as the Predictor, and specific classifiers and regressors as Estimators. We integrate Bottleneck Adapter Layers in the Predictor to improve the transfer learning efficiency and prevent from over-fitting. At the same time, we jointly train the word- and sentence-level tasks with a unified model with multitask learning. Pseudo-PE assisted QE (PEAQE) is proposed, resulting in significant improvements on the performance. Our submissions achieve competitive result in word/sentence-level sub-tasks for both of En-De/Zh language pairs.

## introduction quality estimation (qe) assesses the translation quality of machine translation (mt) system output when ground truth reference is not available @xcite . for the word-level qe task, participants are required to tag tokens and gaps of the translation output from an unknown mt system with ok and bad, as well as tokens in the source with the same tags. the result is measured by matthews correlation coefficient (mcc). for the sentence-level task, participants are required to predict the human translation error rate (hter) scores of the machine translation outputs and the result is evaluated in terms of the pearson's correlation coefficient. our team participates in some of the sub-tasks in two language pairs (en-de and en-zh) @xcite . with regard to the en-de track, at word-level, our model achieves the mcc score of 0.5828 on the target side, and 0.5234 on the source side; at sentence-level, our model ranks the first place with a pearson r score of 0.7583. with regard to the en-zh track, we only submit the target side of word-level sub-task, and achieves a mcc score of 0.5872. our system is implemented with fairseq @xcite (for en-de track) and thumt @xcite (for en-zh track). we extend the original transformer @xcite model to fit the qe task, and leverage transfer learning to finetune the model with the task specific dataset @xcite @xcite . the contributions of our work are as follows: the architecture of our model is shown in figure 1 . ## task description a more detailed description of the word-and sentence-level qe tasks is given in this section. ## dataset the dataset contains 7,000 sentences for the training set, 1,000 for the dev and 1,000 for the test. except from tags and hter scores (labels), the dataset also provides post-edit (pe) text, as the reference for generating qe labels. note that this data is also used in the automatic post editing task in wmt 2020. detailed statistics of the dataset is listed in apart from the brief descriptive statistics listed in the table, our in-depth investigation on the provided dataset unveils some interesting findings: unlike a standard translation task, where various data augmentation techniques, such as backtranslation @xcite , are available, qe task can hardly be improved with data augmentation, as it requires unbiased and high-quality pes to generate tags and hter scores. meanwhile, the change of dataset domain makes it impossible to enlarge the dataset by incorporating the dataset of last year. facing this challenging task, we propose the peaqe method, which will be further explained in details in the following section. ## experiment our experiments of all sub-tasks for en-de and part of sub-tasks for en-zh trak are performed on the wmt 2020 dataset. the model without pseudo-pe assistance is considered as the baseline. ## conclusion we present our works for wmt 2020 qe shared task. the experimental results demonstrate that performing transfer learning with a pre-trained nmt model on the qe task is effective. compared to only using src and mt text, we propose peaqe which could significantly improve the performance of the model. but generating reliable pseudo-pes that are compatible with qe tasks remains a problem that would be investigated in our future works.