The R oyal F lush System for the WMT 2022 Efficiency Task (2022)

Abstract: This paper describes the submission of the RoyalFlush neural machine translation system for the WMT 2022 translation efficiency task. Unlike the commonly used autoregressive translation system, we adopted a two-stage translation paradigm called Hybrid Regression Translation (HRT) to combine the advantages of autoregressive and non-autoregressive translation. Specifically, HRT first autoregressively generates a discontinuous sequence (e.g., make a prediction every k tokens, k1) and then fills in all previously skipped tokens at once in a non-autoregressive manner. Thus, we can easily trade off the translation quality and speed by adjusting k. In addition, by integrating other modeling techniques (e.g., sequence-level knowledge distillation and deep-encoder-shallow-decoder layer allocation strategy) and a mass of engineering efforts, HRT improves 80% inference speed and achieves equivalent translation performance with the same-capacity AT counterpart. Our fastest system reaches 6k+ words/second on the GPU latency setting, estimated to be about 3.1x faster than the last year’s winner.

## introduction large-scale transformer models have made impressive progress in past wmt translation tasks, but it is still challenging for practical model deployment due to time-consuming inference speed @xcite . to build a fast and accurate machine translation system, participants in past wmt efficiency tasks developed and validated many efficient techniques, such as knowledge distillation @xcite , light network architecture @xcite , quantization @xcite etc. we noticed that all the above efforts are aimed at autoregressive translation (at) models. in contrast, other translation paradigms, like nonautoregressive translation (nat) @xcite or semi-autoregressive translation (sat) @xcite etc., have not been well studied. in this participation, we restrict ourselves to the gpu latency track and attempt to investigate the potential of non-standard translation paradigms. however, replicating the vanilla non-autoregressive or semi-autoregressive models degrades the translation quality severely in our preliminary experiments. to this end, we explore hybrid-regressive translation (hrt), the two-stage translation prototype, to better combine the advantages of autoregressive and non-autoregressive translation @xcite . specifically, hrt first uses an autoregressive decoder to generate a discontinuous target sequence with the interval k (@xmath0. then, hrt fills the remaining slots at once with a nonautoregressive decoder. the two decoders share the same parameters without adding additional ones. thus, hrt can easily trade-off between translation quality and speed by adjusting k foot_0 . please see in addition to the change of translation paradigm, we have also made a mass of other optimizations. we use the widely used sequence-level knowledge distillation @xcite and deepencoder-shallow-decoder layer allocation strategy @xcite to learn effective compact models. moreover, on the engineering side, we customized an efficient implementation of gpu memory reuse and kernel fusion for hrt following lightseq @xcite . putting all the efforts together, our hrt model achieves almost equivalent bleu scores to the corresponding at counterparts while improving the inference speed by about 80%. our best-bleu [m] ## hybrid-regressive translation one of the most important highlights is the introduction of the newly proposed two-stage translation prototype--hrt. in this section, we will detail the model, training, and decoding of hrt. ## experimental results setup. we mainly compared hrt to the standard autoregressive baselines in other training hyper-parameters are the same as @xcite . we ran all experiments on 8 geforce 3090 gpus. for decoding, the length penalty is 0.6, and the batch size is 1. we report the detokenized sacrebleu score with the same signature as the teacher. besides, we also follow @xcite 's suggestion to provide comet score @xcite for the evaluation of nonautoregressive translation. translation quality. first, we can see that a deeper encoder improves about 0.5 bleu points across the board. when using @xmath4, both 12-layer and 20-layer hrt models have almost equivalent bleu scores to that of at counterparts. our best hrt model hrt_e20d1 with @xmath5.9 bleu points than the teacher using model ensemble. however, in line with @xcite , we find that even when bleu scores are close, hrt's comet scores are significantly lower than those of at, e.g., at_e20d1 vs. hrt_e20d1 (@xmath6. nevertheless, hrt_e20d1 (@xmath7. translation speed. we estimated the inference speed of the fastest system last year according to the data in @xcite . supposing ignoring the difference in test data, our at baselines run about 40%+ faster than it. it indicates that our at engine is a strong baseline. even so, we can see that both 12-layer and 20-layer hrt with @xmath8. moreover, larger k further reduces the autoregressive decoding steps: our fastest model, hrt_e12d1 (@xmath9, decodes 6k+ source words/second, which is 3.1 times faster than the fastest system last year. ## conclusion this paper presented the royalflush system to the gpu latency track of the wmt 2022 translation efficiency task. we proposed hybrid-regressive translation, a novel two-stage prototype to replace conventional autoregressive translation. with a lot of development optimization, we showed that our hrt with a chunk size of 2 achieves equivalent translation performance to the at counterpart while accelerating 80% inference speed. by increasing hrt's chunk size, our system can further speed up 60% to 6k+ words/second, estimated to be about 3.1 times faster than the fastest system in last year's competition.