The JHU - M icrosoft Submission for WMT 21 Quality Estimation Shared Task (2021)

Abstract: This paper presents the JHU-Microsoft joint submission for WMT 2021 quality estimation shared task. We only participate in Task 2 (post-editing effort estimation) of the shared task, focusing on the target-side word-level quality estimation. The techniques we experimented with include Levenshtein Transformer training and data augmentation with a combination of forward, backward, round-trip translation, and pseudo post-editing of the MT output. We demonstrate the competitiveness of our system compared to the widely adopted OpenKiwi-XLM baseline. Our system is also the top-ranking system on the MT MCC metric for the English-German language pair.

## introduction in the machine translation (mt) literature, quality estimation (qe) refers to the task of evaluating the translation quality of a system without using a human-generated reference. there are several different granularities as to the way those quality labels or scores are generated. our participation in the wmt21 quality estimation shared task focuses specifically on the word-level quality labels (word-level subtask of task 2), which are helpful for both human @xcite and automatic @xcite post-editing of translation outputs. the task asks the participant to predict one binary quality label (ok/bad) for each target word and each gap between target words, respectively. 1 our approach closely follows our contemporary work @xcite , which focuses on en-de and en-zh language pairs tested in the 2020 version of the shared task. the intuition behind our idea is that translation knowledge is very useful for predicting word-level quality labels of translations. however, usage of machine translation models is limited in the previous work mainly due to (1) the difficulties in using both the left and right context of an mt word to be evaluated; (2) the difficulties in making the word-level reference labels compatible with subword-level models; and (3) the difficulties in enabling translation models to predict gap labels. to resolve these difficulties, we resort to levenshtein transformer (levt, @xcite , a model architecture designed for non-autoregressive neural machine translation (na-nmt). because of its iterative inference procedure, levt is capable of performing post-editing on existing translation output even just trained for translation. to further improve the model performance, we also propose to initialize the encoder and decoder of the levt model with those from a massively pre-trained multilingual nmt model (m2m-100, @xcite . starting from a levt translation model, we then perform a two-stage finetuning process to adapt the model from translation prediction to quality label prediction, using automatically-generated pseudopost-editing triplets and human post-editing triplets respectively. all of our final system submissions are also linear ensembles from several individual models with weights optimized on the development set using the nelder-mead method @xcite . ## method our system building pipeline is consisted of three different stages: ( @xcite to train the levt translation model, except that we initialize the embedding, the encoder, and decoder of levt with those from the small m2m-100-small model (418m parameters, @xcite to take advantage of large-scale pretraining. because of that, we also use the same sentencepiece model and vocabulary as the m2m-100 model. for to-english language pairs, we explored training multi-source levt model. according to the results on devtest data, this is shown to be beneficial for the qe task for ro-en, ru-en and ne-en, but not for other language pairs. stage 2: synthetic finetuning during both finetuning stages, we update the model parameters to minimize the nll loss of word quality labels and gap quality labels, for the deletion and insertion head, respectively. to obtain training targets for finetuning, we need translation triplet data, i.e., the aligned triplet of source, target, and post-edited segments. human post-edited data naturally provides all three fields of the triplet, but only comes in a limited quantity. to further help the model to generalize, we conduct an extra step of finetuning on synthetic translation triplets, similar to some previous work @xcite . we explored five different methods for data synthesis, namely: ## conclusion in this paper, we present our wmt21 word-level qe shared task submission based on levenshtein transformer training and a two-step finetuning process. we also explore various ways to create synthetic data to build more generalizable systems with limited human annotations. we show that our system outperforms the openkiwi+xlm baseline for all language pairs we experimented with. our official results on the blind test set also demonstrate the competitiveness of our system. we hope that our work can inspire other applications of levenshtein transformer beyond the widely studied case of non-autoregressive translation.