HW - TSC ’s Participation at WMT 2021 Quality Estimation Shared Task (2021)

Abstract: This paper presents our work in WMT 2021 Quality Estimation (QE) Shared Task. We participated in all of the three sub-tasks, including Sentence-Level Direct Assessment (DA) task, Word and Sentence-Level Post-editing Effort task and Critical Error Detection task, in all language pairs. Our systems employ the framework of Predictor-Estimator, concretely with a pre-trained XLM-Roberta as Predictor and task-specific classifier or regressor as Estimator. For all tasks, we improve our systems by incorporating post-edit sentence or additional high-quality translation sentence in the way of multitask learning or encoding it with predictors directly. Moreover, in zero-shot setting, our data augmentation strategy based on Monte-Carlo Dropout brings up significant improvement on DA sub-task. Notably, our submissions achieve remarkable results over all tasks.

## introduction quality estimation (qe) focuses on estimating the quality of machine translation (mt) system output when no ground truth reference is available @xcite . qe covers wide range of tasks including word-level, sentence-level and document-level. it has wide range of applications in mt quality check and post-editing effort estimation. in wmt2021 quality estimation shared task foot_0 , there are three sub tasks -sentence-level direct assessment task, word and sentence-level postediting effort task and critical error detection task. each sub task involves several language pairs. our team participated in all the above three tasks over all language pairs. we summarized our main contributions as follow: our methods achieve impressive performance on both word and sentence level tasks. specifically, we peak the top-1 on sentence-level da over english-german and english-japanese pairs. for word and sentence-level post-editing effort task, our submissions of the majority language pairs obtain the best pearson's correlation or matthews correlation coefficient. we also win the first place in critical error detection task in english-chinese and english-japanese. we will describe the tasks, datasets, and our methods for da task, post-editing task, and critical error detection task in section 2, section 3, and section 4 respectively. section 5 presents details of our experimental setup and results, with a brief discussion and conclusion in the end. ## conclusion we present our work on wmt 2021 qe shared task in this paper. for all the three tasks to estimate sentence-level da, token and sentence-level post-edit effort and sentence-level critical error, we employ predictor-estimator framework as our baseline. to further boost performance, we investigate the usage of additional high-quality translations. for task1, we mainly focus on introducing postedits with multi-task learning. also, the effect of data augmentation method based on mc dropout is studied here to improve the result of zero-shot pairs. for task 2 and 3, we generate high-quality translations for each observation using multiple welltrained machine translation systems. by directly concatenating amt with the original source and target sentence then encoding it with pre-trained predictor, we achieved remarkable results over all language pairs and tasks. in future, we will continue to invest time and effort on studying the effect of involving additional translations into qe tasks, for example, how the additional translation quality will affect qe performance, what the better ways are to incorporate additional translations in.