Hybrid-Regressive Paradigm for Accurate and Speed-Robust Neural Machine Translation (2023)

Abstract: This work empirically confirms that non-autoregressive translation (NAT) is less robust in decoding batch size and hardware settings than autoregressive translation (AT). To address this issue, we demonstrate that prompting a small number of AT predictions can significantly reduce the performance gap between AT and NAT through synthetic experiments. Following this line, we propose hybrid-regressive translation (HRT), a two-stage translation prototype that combines the strengths of AT and NAT. Specifically, HRT first generates discontinuous sequences via autoregression (e.g., make a prediction for every k tokens, k>1 ) and then fills in all previously skipped tokens at once in a non-autoregressive manner. Experiments on five translation tasks show that HRT achieves comparable translation quality with AT while having at least 1.5x faster inference regardless of batch size and device. Additionally, HRT successfully inherits the sound characteristics of AT in the deep-encoder-shallow-decoder architecture, allowing for further speedup without BLEU loss.

## introduction autoregressive translation (at) such as transformer has been the de facto standard for neural machine translation (nmt) @xcite . however, at predicts only one target word at a time, resulting in slow inference speed. to overcome this limitation, non-autoregressive translation (nat) attempts to generate the entire target sequence in one step by assuming conditional independence among target tokens @xcite . while nat offers efficiency, it often suffers from significant degradation in translation quality. achieving a better trade-off between inference speed and translation quality remains an active area of research for nat @xcite @xcite . one of the most successful approaches to this issue is the iterative refinement mechanism (ir-nat) proposed by @xcite , which has been widely adopted by several leading systems @xcite @xcite @xcite . specifically, ir-nat, also known as multi-shot nat, takes the translation hypothesis from the previous iteration as a reference to refine the new translation until it reaches the predefined iteration count i or no translation changes. although a larger i can improve translation accuracy, it may also lead to a speedup degradation @xcite . in this work, we build upon the findings of @xcite and examine the robustness of ir-nat compared to at. our comprehensive examinations confirm that the inference speed of ir-nat is consistently less robust than that of at when involving various decoding batch sizes and computing hardware. for example, when using a gpu, the ten-iteration non-autoregressive model has 1.7/1.2/0.7/0.4 times the inference speed of the at model for decoding batch sizes of 1/8/16/32, respectively. however, when switching to cpu, the relative speed ratio drops to 0.8/0.4/0.3/0.3 times. previous studies have highlighted the complementary nature of at and nat in terms of both translation quality (at being superior) and inference speed (nat being superior) @xcite . our findings, however, suggest that there is also complementary robustness in inference speed (at being superior). taking a further step, we investigate how much target context (i.e., the number of target tokens) is sufficient for one-shot nat to rival multi-shot nat through synthetic experiments. our findings suggest that given a well-trained cmlm model, even if 70% of at translations are masked, the remaining target context can help the cmlm 1 with greedy search compete with the standard cmlm 10 with beam search (see figure 2 ). this could enable us to build the desired target context more cheaply, replacing expensive multiple iterations. to our best knowledge, this is the first study of the masking rate issue in the inference phase of nat. based on the observations from these experiments, we have proposed a novel two-stage translation prototype called hybrid-regressive translation (hrt). this method combines the advantages of autoregressive translation (at) and nonautoregressive translation (nat) by first using an autoregressive decoder to generate a discontinuous target sequence with an interval of k (@xmath0, and then filling the remaining slots in a lightweight non-autoregressive manner. we have also created a multi-task learning framework, enhanced by curriculum learning, for effective and efficient training without adding any model parameters. results on wmt en↔ro, en↔de, and nist @xmath1, and is competitive with state-of-theart ir-nat models. specifically, hrt achieved a bleu score of 28.27 on the wmt @xmath2, and is 1.5x faster than at regardless of batch size and device. additionally, hrt equipped with a deep-encoder-shallow-decoder architecture achieved up to 4x/3x acceleration on gpu/cpu, respectively, without sacrificing bleu. ## background given a source sentence @xmath3, x 2 , . . . , x m } and a target sentence @xmath4, y 2 , . . . , y n }, there are several ways to model p (y|x): autoregressive translation (at) at is the predominant technique in nmt, decomposing p (y|x) using the chain rule: p (y|x) = n @xmath5, @xmath6, where @xmath7. nevertheless, autoregressive models must wait for y t-1 to be generated before predicting y t , thus hindering parallelism over the target sequence. non-autoregressive translation (nat) nat has been proposed to generate target tokens simultaneously @xcite . this approach replaces the traditional autoregressive formulation of @xmath8, resulting in the following formulation: p (y|x) = p (n |x) × n @xmath9, z). various approaches have been proposed for modeling z, such as using source embedding @xcite , reordering the source sentence @xcite , or using a latent variable @xcite . iterative refinement based non-autoregressive translation (ir-nat) ir-nat extends the traditional one-shot nat by introducing an iterative refinement mechanism @xcite . we choose cmlm @xcite as the representative of ir-nat due to its excellent performance and simplification. during training, cmlm randomly masks a fraction of tokens on y as the alternative to z, and is trained as a conditional masked language model @xcite . denoting y m /y r as the masked/residual tokens of y, we have: p (y|x) = |y m | @xmath10, y r ). at inference, cmlm deterministically masks tokens from the hypothesis in the previous iteration ŷ(i-1) according to prediction confidences. this process is iterated until ŷ(i-1) = ŷ(i) or i reaches the maximum iteration count. ## acceleration robustness problem in this section, we comprehensively analyze the inference acceleration robustness problem in ir-nat. without loss of generality, we take cmlm as the agency of ir-nat. 2 problem description the inference overhead of the autoregressive translation model mainly con-centrates on the decoder side @xcite . suppose that the decoder's computational cost is proportional to the size of its input tensor (b, n, h), where b is the batch size, n is the target sequence length, and h is the network dimension. we omit h for convenience due to its invariance in nat and at. thus, the total cost of at model is about c at ∝ n × o(b × 1) foot_1 . likely, the cost of iiteration nat is c nat ∝ i × o(b × n ). given a fixed test set, we can use t d (•) to represent the translation time on computing device d. this allows us to calculate the relative speedup ratio α between i-iteration nat and at as: ## synthetic experiments according to equation 1, reducing the iteration count i helps to increase α. recalling the refinement process of ir-nat, we hypothesize that the essence of multiple iterations is to provide the decoder with a good enough target context (deterministic target tokens). this raises the question of how many target tokens need to be provided to make one-shot nat competitive with ir-nat? to answer it, we conduct synthetic experiments on wmt @xmath11. we then use a pre-trained cmlm model to predict these masks and observe the bleu score curves under different masking rates. ## hybrid-regressive translation we propose a novel two-stage translation paradigm, hybrid-regressive translation (hrt), which imitates the chunk process. in hrt, a discontinuous sequence with a chunk size of k is autoregressively generated in stage i, followed by nonautoregressive filling of the skipped tokens in stage ii. ## analysis impact of chunk size we tested chunk size k on the @xmath12, as shown in which decoding stage is more important? to understand the importance of the two decoding stages of hrt, we exchange the intermediate results of two hrt models (a and b). specifically, we use the skip-at decoder of a to generate its discontinuous target sequence, which is then forced decoded by b's skip-at decoder to obtain corresponding encoding representations and autoregressive model scores. finally, b's skip-cmlm decoder generates the complete translation result based on these. we can reverse the order of a and b as well. we use two models (hrt and hrt 20-6 ) with a large performance gap as a and b, respectively. as shown in ## conclusion we noted that ir-nat has robustness issues with inference acceleration. inspired by our findings in synthetic experiments, we proposed hrt to take advantage of the strengths of both at and nat. our experiments demonstrated that our approach surpasses existing semi-autoregressive and ir-nat methods, providing competitive performance and consistent speedup, making it a viable alternative to autoregressive translation.