Quality Estimation Using Dual Encoders with Transfer Learning (2021)

Abstract: This paper describes POSTECH’s quality estimation systems submitted to Task 2 of the WMT 2021 quality estimation shared task: Word and Sentence-Level Post-editing Effort. We notice that it is possible to improve the stability of the latest quality estimation models that have only one encoder based on the self-attention mechanism to simultaneously process both of the two input data, a source sequence and its machine translation, in that such models have neglected to take advantage of pre-trained monolingual representations, which are generally accepted as reliable representations for various natural language processing tasks. Therefore, our model uses two pre-trained monolingual encoders and then exchanges the information of two encoded representations through two additional cross attention networks. According to the official leaderboard, our systems outperform the baseline systems in terms of the Matthews correlation coefficient for machine translations’ word-level quality estimation and in terms of the Pearson’s correlation coefficient for sentence-level quality estimation by 0.4126 and 0.5497 respectively.

## introduction quality estimation (qe) is the task of estimating the quality of given machine translations without regard to their reference translations @xcite . as reference translations are generally unavailable in real life, qe should help to treat output texts of machine translation (mt) systems. qe can be categorized into several subtasks, and this round of the wmt qe task has three subtasks, yet we focus on task 2: word and sentence-level post-editing effort. in task 2, while sentence-level qe aims to predict the human-targeted translation edit rate (hter, @xcite , which measures the edit distance between an mt output (mt) and its human post-edited text (pe), word-level qe aims to predict ok-bad tags for three sequences of tokens: the sequence of words in a source text (src) depending on whether they are correctly translated referring to mt; the sequence of words in mt depending on their correctness; @xmath0, which each represent the gap between two adjacent words, depending on the existence of any missing words @xcite . as other recent qe models do, our method also applies transfer learning, considering that pretrained language models (lm) have been successfully applied to various natural language processing (nlp) tasks including qe; many previous studies @xcite @xcite @xcite @xcite ) that apply pre-trained lms to qe have adopted multilingual or crosslingual lms such as multilingual-bert @xcite , xlm @xcite , and xlm-r @xcite to process the two input data src and mt. such cross-lingual lms have a single transformer @xcite encoder using only the self-attention mechanism to create vector representations of the input data and predict the labels. however, it appears possible to further improve the stability of those models, considering that they are not propped up by pre-trained lms' monolingual word representations, which are generally accepted as reliable representations for various nlp tasks. with this background, we propose a qe model that has two separate pre-trained encoders that each produce monolingual representations of src and mt, respectively. on top of each encoder, we add a cross attention network for the learning of the cross-lingual context between src and mt; these networks will produce two sets of cross-lingual representations for qe. we conduct simple experiments to compare the performance of our systems and ensembles of them with that of the baseline systems and that of other submitted systems for task 2. experimental results imply that although our systems do not always outperform the baseline systems, they do in terms of the matthews correlation coefficient (mcc) for mt's word-level qe and in terms of the pearson's correlation coefficient (pcc) for sentence-level qe by 0.4126 and 0.5497 respectively. ## related work because our model does not confine its monolingual encoders to specific pre-trained lms, all pretrained lms can be considered relevant. among them, most of the recently proposed pre-trained lms are denoising autoencoders, of which the pretraining task is usually to select about 15% of tokens in unlabeled input sequences and apply the attention mechanism to those tokens @xcite or is to mask certain tokens @xcite and then restore them. however, in our experiments, our systems use electra @xcite . electra introduces "replaced token detection" as an additional pre-training task and let the language model learn to distinguish between real input tokens and specious but artificially generated tokens. in detail, when the generator network predicts the tokens in the masked positions, some of the predicted tokens are corrupted, and then this output sequence is fed into a transformer-based discriminator network, which predicts whether each token in the fed sequence is the same as the original one or is a replaced one (figure 1 ). we suppose that this process and qe are similar to each other in that both of them predict the soundness of the given tokens, so electra would be one of the most appropriate pre-trained lms for our qe model's monolingual encoders, especially for task 2. ## model description our model uses two electras: one electra foot_0 that is pre-trained with english corpora and the other electra foot_1 pre-trained with german corpora. figure 2 depicts the overall structure of our model. ## conclusion we model our systems submitted to task 2 of the wmt 2021 qe shared task on our proposed model, which uses dual pre-trained monolingual encoders and two additional cross attention networks to pro-cess the two input data src and mt more effectively considering that the latest transformer-based qe models are not propped up by pre-trained monolingual word representations. we expect that the cross attention networks enable the two pre-trained monolingual encoders to exchange cross-lingual information without losing their stability and to learn the subtasks of task 2 jointly and also separately. experimental results partially supports this expectation: according to the official leaderboard, our systems outperform the baseline systems in terms of the mt-side words mcc and the sentence-level pcc by 0.4126 and 0.5497 respectively, although they do not in terms of the src-side mcc and the mt-@xmath1. neverhteless, it appears possible to improve the performance of our systems by adopting larger pre-trained lms, and thus, our future work will explore such aspects and other related new methods.