The Volctrans GLAT System: Non-autoregressive Translation Meets WMT 21 (2021)

Abstract: This paper describes the Volctrans’ submission to the WMT21 news translation shared task for German->English translation. We build a parallel (i.e., non-autoregressive) translation system using the Glancing Transformer, which enables fast and accurate parallel decoding in contrast to the currently prevailing autoregressive models. To the best of our knowledge, this is the first parallel translation system that can be scaled to such a practical scenario like WMT competition. More importantly, our parallel translation system achieves the best BLEU score (35.0) on German->English translation task, outperforming all strong autoregressive counterparts.

## introduction in recent years' wmt competitions, most teams develop their translation systems based on autoregressive models, such as transformer @xcite . although autoregressive models (at) achieve strong results, it is also worth exploring other alternative machine translation paradigm. therefore, we build our systems with non-autoregressive translation (nat) models @xcite . unlike the left-to-right decoding in the autoregressive models, the nat models employ the more efficient parallel decoding. specifically, our system employs single-pass parallel decoding, which generates all the tokens in parallel at one time, thus can accelerate decoding speed. in this paper, we would like to present the best practice we explored in this year's competition for our parallel translation system, aiming at achieving top results while preserving decoding efficiency. ## backbone model architecture as depicted in figure 1 , our submitted system employs glat @xcite ) as our backbone model architecture, and includes an auxiliary decoder in glat for achieving better translation performance. glat is a method for training nonautoregressive models rather than a model architecture, which adaptively samples target tokens in training. although the target token sampling in glat helps training, it also introduces a gap between training and inference. to close the gap, we introduce the auxiliary decoder that shares the same encoder with the glat decoder, which is only used for training in a multi-tasking fashion. ## data preparation in this section, we will describe our best practice of distilled data construction by employing at models. as illustrated in data preparation in figure 2 , we will first depict the general procedure of data filtering and preprocessing of the provided raw data, followed by the training details of the at models. finally, we will describe how we produced distilled data given the trained at models. the resulting distilled data will be used for training our glat system. ## multi-stage training we train our parallel translation system in a multistage way (see multi-stage training in figure 2 ). in the first stage, the model uses the distilled parallel and source monolingual data for training. in the second stage, we train the model with the target monolingual data (aka. cycle kd data). after training the model on large-scale distilled data until convergence, we finetune the model on small-scale in-domain data. ## experiment for our parallel translation system, we train three glat variants with the distilled data, and get the 5 https://github.com/moses-smt/ mosesdecoder final outputs by reranking candidate hypothesis obtained from multiple glat models. ## conclusion in this paper, we introduced our system submitted to the wmt2021 shared news translation task on @xmath0. we build a parallel translation system based on the glancing transformer @xcite . knowledge distillation, domain adaptation, reranking have proven effective in our system. our constrained parallel translation system gets first place in the @xmath1.0 bleu score.