H YBRID -R EGRESSIVE N EURAL M ACHINE T RANSLATION
Anonymous authors
Paper under double-blind review
A BSTRACT
In this work, we empirically confirm that non-autoregressive translation with an iterative refinement mechanism (IR-NAT) suffers from poor acceleration robustness because it is more sensitive to decoding batch size and computing device setting than autoregressive translation (AT).
Inspired by it, we attempt to investigate how to combine the strengths of autoregressive and non-autoregressive translation paradigms better.
To this end, we demonstrate through synthetic experiments that prompting a small number of AT’s predictions can promote one-shot nonautoregressive translation to achieve the equivalent performance of IR-NAT.
Following this line, we propose a new two-stage translation prototype called hybridregressive translation (HRT).
Specifically, HRT first generates discontinuous sequences via autoregression (e.g., make a prediction every k tokens, k > 1 ) and then fills in all previously skipped tokens at once in a non-autoregressive manner.
We also propose a bag of techniques to effectively and efficiently train HRT without adding any model parameters.
HRT achieves the state-of-the-art BLEU score of 28.49 on the WMT
En → De task and is at least 1.5x faster than AT, regardless of batch size and device.
In addition, another bonus of HRT is that it successfully inherits the good characteristics of AT in the deep-encoder-shallow-decoder architecture.
Concretely, compared to the vanilla HRT with a 6-layer encoder and 6-layer decoder, the inference speed of HRT with a 12-layer encoder and 1-layer decoder is further doubled on both GPU and CPU without BLEU loss.
I NTRODUCTION
Autoregressive translation (AT) such as Transformer has been the de facto standard for Neural Machine Translation (NMT) Vaswani et al.
However, AT predicts only one target word each time, resulting in a slow inference speed.
To address this problem, non-autoregressive translation (NAT) attempts to generate the entire target sequence in parallel in one shot, assuming that the generation of target tokens is conditional independent Gu et al.
While efficient, one-shot NAT suffers from severe translation quality degradation.
How to achieve a better balance between inference speed and translation quality is still an active field for NAT Wang et al.
(2018a);
Ran et al.
(2020); Qian et al.
; Huang et al.
(2022b).
The iterative refinement mechanism first introduced by Lee et al.
(2018), is one of the most successful approaches to this issue and has been adopted by several leading systems Ghazvininejad et al.
; Kasai et al.
(2020a)
; Guo et al.
; Saharia et al.
; Geng et al.
; Huang et al.
(2022b). Specifically, iterative refinement-based NAT (abbreviated as IR-NAT), also known as multi-shot NAT, “thinks more” than one-shot NAT: IR-NAT takes the translation hypothesis from the previous iteration as a reference and regularly polishes the new translation.
The iteration stops when reaching the predefined iteration count I or no translation changed.
A larger I generally improves translation accuracy while facing the risk of speedup degradation Kasai et al.
(2020b).
In this work, we devote ourselves to understanding the limitations of IR-NAT and attempting to build a new fast-and-accurate translation paradigm beyond it.
Our contributions are the following:
• We comprehensively study the acceleration robustness problem in IR-NAT and extend the finding of Kaiser et al.
(2018) that IR-NAT is more sensitive not only to the decod-
ing batch size but also to the computing device compared with AT.
For example, when the decoding batch size is 1/8/16/32, the ten-iteration non-autoregressive model achieves 1.7x/1.2x/0.7x/0.4x inference speed of the AT model on GPU, respectively.
However, when switching to CPU, the relative speed ratio drops to 0.8x/0.4x/0.3x/0.3x.
These results prove that the two translation paradigms are well complementary to each other.
• We design synthetic experiments to investigate how much target context (i.e., the number of target tokens) is sufficient for one-shot NAT to rival multi-shot NAT.
The value of the answer is that we could build the desired target context more cheaply, replacing expensive multiple iterations.
Specifically, given a well-trained CMLM model, we notice that under the appropriate masking strategy, even if 70% of AT translations are masked, the remaining target context can help the CMLM 1 2 with greedy search compete with the standard CMLM 10 with beam search (see Figure 2).
To our best knowledge, we are the first to study the masking rate issue in the inference phase of NAT.
• Inspired by the observations above, we proposed a novel two-stage translation prototype – hybrid-regressive translation (HRT), to incorporate the advantages of AT and NAT.
Concretely, HRT first uses an autoregressive decoder to generate a discontinuous target sequence with the interval k ( k > 1 ).
Then, HRT fills the remaining slots at once in a lightweight non-autoregressive manner.
We propose to use a multi-task learning framework enhanced by curriculum learning and mixed distillation for effective and efficient training without adding any model parameters.
• Experimental results on WMT En ↔ Ro, En ↔ De, and NIST Zh
→ En show that HRT significantly outperforms prior work combining AT and NAT and has competitive BLEU with state-of-the-art IR-NAT models.
Specifically, HRT achieves a BLEU score of 28.49 on the WMT
En → De task and is robust 1.5x faster than AT regardless of batch size and device.
Moreover, HRT equipped with deep-encoder-shallow-decoder architecture achieves up to 4x/3x acceleration on GPU/CPU, respectively, without BLEU loss.
B ACKGROUND
Given a source sentence x = { x 1 , x 2 , . . .
, x M } and a target sentence y =
{ y 1 , y 2 , . . .
, y N } , there are several ways to model P ( y | x ) :
Autoregressive Translation (AT)
AT is the dominant approach in NMT, which decomposes
P ( y | x ) by chain rules: P ( y | x ) = (cid:81)
Nt =1 P ( y t | x , y <t ) , where y <t denotes the generated prefix translation before time step t .
However, autoregressive models have to wait for the generation of y t − 1 before predicting y t , which hinders the parallelism over the target sequence.
Non-Autoregressive Translation (NAT)
NAT allows generating all target tokens simultaneously
Gu et al.
NAT replaces y <t with target-independent input z and rewrites as: P ( y | x ) =
P ( N | x ) × (cid:81)
Nt =1 P ( y t | x , z ) .
We can model z as source embedding (Gu et al., 2018; Guo et al., 2019a), reordered source sentence (Ran et al., 2019), latent variable (Ma et al., 2019; Shu et al., 2019) etc.
Iterative Refinement based Non-Autoregressive Translation (IR-NAT)
IR-NAT extends the traditional one-shot NAT by introducing an iterative refinement mechanism (Lee et al., 2018).
We choose CMLM as the IR-NAT representative in this work due to its excellent performance and simplification Ghazvininejad et al.
During training, CMLM randomly masks a fraction of tokens on y as the alternative to z and is trained as a conditional masked language model Devlin et al.
Denote y m / y r as the masked/residual tokens of y , then we have:
P ( y | x ) =
(cid:81)
| y m | t =1 P ( y mt | x , y r ) .
At inference, CMLM deterministically masks tokens from the hypothesis in the previous iteration ˆ y ( i − 1) according to prediction confidences.
This process is repeated until ˆ y ( i − 1)
=
ˆ y ( i ) or i reaches the maximum iteration count.
[Figure]
A CCELERATION R OBUSTNESS P ROBLEM
In this section, we attempt to comprehensively understand the inference acceleration robustness problem in IR-NAT.
Without loss of generality, we take CMLM as the agency of IR-NAT.
Problem Description
The inference overhead of the autoregressive translation model mainly concentrates on the decoder sideHu et al.
Suppose that the decoder’s computational cost is proportional to the size of its input tensor ( B, N, H ) , where B is the batch size, N is the target sequence length, and H is the network dimension.
For convenience, we omit H due to its invariance in NAT and AT.
Thus, the total cost of AT model is about C at ∝ N × O
( B × 1) 4 .
Likely, the cost of I -iteration NAT is C nat
∝ I × O
( B × N ) .
Given a fixed test set, We use T D ( · ) to represent the translation time on computing device D .
In this way, we get the relative speedup ratio α between
I -iteration NAT and AT by:
[Equation]
where E ( B, D ) =
T D ( O ( B × 1))
T D ( O ( B × N )) ≤ 1 , denotes the parallel computation efficiency over sequence under batch size B and device D .
When fixing N and I , α is completely determined by E ( B, D ) .
We note that most previous NAT studies only report the inference speed with D =GPU and B =1, without considering cases when B or D change.
Setup To systematically investigate this problem, we compared the actual inference speed of
CMLM 5 and AT under varying environments, including batch size B ∈ { 1 , 8 , 16 , 32 } , device
D ∈ { GPU , CPU } 6 , and the number of iteration I ∈ { 1 , 4 , 10 } .
We use a beam size of 5 in all experiments.
We test inference speed on the widely used WMT En → De newstest2014 test set and report the average results over five runs (see Appendix A for details).
Results
We plot the curve of relative speedup ratio ( α ) in Figure 1, where we can see that:
i. α decreases as decoding batch size increases regardless of the number of iterations, which is in line with Kasai et al.
(2020b).
ii.
α on CPU generally performs worse than that on GPU, except using one iteration.
iii.
The benefit of non-autoregressive decoding is more prone to disappear for larger I .
More concretely, the ten-iteration non-autoregressive model is 70% faster than the autoregressive counterpart when decoding a single sentence on GPU.
In contrast, the IR-NAT model only reaches
[Figure]
30% inference speed of the AT model when switching to batches of 32 on CPU.
These results indicate that the two translation paradigms enjoy different decoding setups and complement each other.
Therefore, combining the advantages of AT and NAT could be an effective way for robust acceleration.
S YNTHETIC E XPERIMENTS
According to Equation 1, we know that reducing the iteration count I helps to increase α .
Recalling the refinement process of IR-NAT, we think the essence of multiple iterations is to provide the decoder with a good enough target context (deterministic target tokens).
Thus, an unexplored question raises that how many target tokens need to be provided to make one-shot NAT compete with IR-NAT ?
In this section, we try to answer this question through synthetic experiments on WMT
En → Ro and
En → De.
Specifically, we control the size of the target context by masking the partial translations generated by a pre-trained AT model.
Then we use a pre-trained CMLM model to predict these masks.
Finally, we observe the BLEU score curves under different masking rates.
Models We use the official CMLM models.
Since the authors did not release the AT baselines, we used the same data to retrain AT models with the standard Transformer-Base configuration (Vaswani et al., 2017) and obtain comparable performance with theirs (see Appendix B for details).
Decoding AT models decode with beam sizes of 5 on both tasks.
Then we replace a certain percentage of AT tokens with [ MASK ] and feed them to CMLM.
The used CMLM model only iterates once with beam size 1.
We substitute all [ MASK ]s with CMLM’s predictions to obtain the final translation.
We report case-sensitive tokenized BLEU scores by multi-bleu.perl .
Mask Strategies We tested four strategies to mask AT results: HEAD , TAIL , RANDOM and
CHUNK .
Given the masking rate p mask and the translation length N , the number of masked tokens is N mask =
max(1, ⌊ N × p mask ⌋ ).
Then HEAD /
TAIL always masks the first/last N mask tokens, while RANDOM masks the translation randomly.
CHUNK is slightly different from the above strategies.
It first divides the target sentence into C chunks, where C =
Ceil ( N/k ) and k is the chunk size.
Then in each chunk, we retain the first token but mask other k − 1 tokens.
Thus, the actual masking rate in CHUNK is 1 − 1 /k instead of p mask .
We ran RANDOM three times with different seeds to exclude randomness and report the average results.
Results The experimental results are illustrated in Figure 2, where we can see that CHUNK is moderately but consistently superior to RANDOM and both of them significantly outperform HEAD and TAIL .
We attribute the success of CHUNK to two aspects: the use of (1) bidirectional context (Devlin et al., 2019) (vs. HEAD and TAIL ); (2) uniformly distributed deterministic tokens (vs.
RANDOM ) 7 .
In addition, when using the CHUNK strategy, we find that exposing 30% AT tokens
[ B ] yyyyyyyy[ E ]
Skip-AT Decoder (a) TASK-AT y
[ P ] [ M ] y
[ M ] yy
[ P ] [ E ] [ P ]
Skip-CMLM Decoder (b) TASK-CMLM
[ B 2 ] yyyy[ E ]
Skip-AT Decoder (c) TASK-SKIP-AT
[ M ] yy
[ P ] [ M ] yy
[ P ] [ E ] [ P ]
Skip-CMLM Decoder (d) TASK-SKIP-CMLM as the input of the decoder is sufficient to make our CMLM 1 (beam=1) compete with the official
CMLM 10 (beam=5), which indicates the importance of a good partial target
H YBRID -R EGRESSIVE T RANSLATION
Inspired by the observations above, we propose a novel two-stage translation paradigm called hybrid-regressive translation (HRT) to imitate the process of CHUNK .
Briefly speaking, HRT autoregressively generates a discontinuous sequence with chunk size k (stage I), and then nonautoregressively fills the skipped tokens (stage II).
5. A RCHITECTURE
Overview HRT consists of three components: encoder, Skip-AT decoder (for stage I), and SkipCMLM decoder (for stage II).
All components adopt the Transformer architecture (Vaswani et al., 2017).
The two decoders have the same network structure, and we share them to make the parameter size of HRT the same as the vanilla Transformer.
The only difference between the two decoders lies in the masking pattern in self-attention: The Skip-AT decoder masks future tokens to guarantee strict left-to-right generation like autoregressive Transformer.
In contrast, the Skip-CMLM decoder eliminates it to leverage the bi-directional context like CMLM Ghazvininejad et al.
No Target Length Predictor Thanks to Skip-AT, we can obtain the translation length as its byproduct: N nat = k
× N at , where N at is the sequence length produced by Skip-AT .
Compared to most NAT models that jointly train the translation length predictor and the translation model, our approach benefits in two aspects: (1) There is no need to carefully tune the weighting coefficient between the sentence-level length prediction loss and the word-level target token prediction loss; (2)
The length predicted by Skip-AT could be more accurate because it can access the already generated sequence information.
5. T RAINING
Next, we elaborate on how to train the HRT model efficiently and effectively.
Please refer to Appendix C for the entire training algorithm.
Multi-Task Framework
We learn HRT through joint training of four tasks, including two primary tasks ( TASK-SKIP-AT , TASK-SKIP-CMLM ) and two auxiliary tasks ( TASK-AT , TASK-CMLM ).
All tasks use cross-entropy as the training objective.
Figure 3 illustrates the differences in training samples among these tasks.
It should be noted that TASK-SKIP-AT shrinks the sequence length from N to N/k compared to TASK-AT , while the token positions follow the original sequence.
For example, in Figure 3 (c), the position of TASK-SKIP-AT input ([ B 2 ], y 2 , y 4 ) is (0, 2, 4) instead of (0, 1, 2).
Involving auxiliary tasks is necessary because the two primary tasks cannot fully leverage all tokens in the sequence due to the fixed k .
For example, in Figure 3 (c) and (d), y
[Table]
[Table]
Curriculum Learning To ensure that the model is not overly biased towards auxiliary tasks, we propose gradually transferring the training tasks from auxiliary tasks to primary tasks through curriculum learning (Bengio et al., 2009).
More concretely, given a batch of original sentence pairs B and let the proportion of primary tasks in B be p k , we start with p k =0 and construct the training samples of TASK-AT and TASK-CMLM for all pairs.
Then we gradually increase p k to introduce more learning signals for TASK-SKIP-AT and TASK-SKIP-CMLM until p k =1.
In implementation, we schedule p k by:
[Equation]
where t and T are the current and total training steps.
λ is a hyperparameter, and we use λ =1 to increase p k linearly for all experiments.
Mixed Distillation Conventional NAT models use smoother distillation data generated by AT models instead of raw data.
However, using only distillation data may lose some important information (e.g., rare words) contained in the original data Ding et al.
Recent WMT competitions have proved that mixing raw data with distillation data is an effective approach to improve the performance of autoregressive translation.
Akhbardeh et al.
; Barrault et al.
To combine the best of both worlds, we propose a simple but effective approach – Mixed Distillation (MixDistill) for non-autoregressive translation.
During training, MixDistill randomly samples the target sentence from the raw version y with probability p raw or its distillation version y ∗ with probability 1 − p raw , where p raw is a hyperparameter 8 .
Compared with recent studies, our method is easier to implement: HRT does not rely on external word alignment Ding et al.
(2020), and also avoids the time-consuming bi-directional distillation process Ding et al.
MixDistill makes the HRT model less prone to overfitting in simple tasks (e.g., WMT’16 En → Ro), but what we emphasize is that the performance of HRT does not depend entirely on it.
See Table 6 for more details.
5. I NFERENCE
HRT adopts two-stage generation strategy.
In the first stage, the Skip-AT decoder starts from [ BOS k ] to autoregressively generate a discontinuous target sequence ˆ y at =
( z 1 , z 2 , . . .
, z m ) with chunk size k until meeting [ EOS ].
Then we construct the input of Skip-CMLM decoder y nat by appending k − 1 [ MASK ]
s before every z i .
The final translation is generated by replacing all [ MASK ]s with the predicted tokens by the Skip-CMLM decoder with one iteration.
If there are multiple [ EOS ]s existing, we truncate to the first [ EOS ].
Note that the beam size b at in Skip-AT can be different from the beam size b nat in Skip-CMLM, as long as b at ≥ b nat : We only feed the top b nat SkipAT hypotheses to Skip-CMLM decoder.
In the implementation, we use standard beam search in
Skip-AT (b at > 1) and greedy search in Skip-CMLM (b nat =1) because b nat =1 is sufficient to obtain competitive BLEU scores, thanks to the good context provided by Skip-AT.
Table 2 gives a detailed discussion about the beam size setting in HRT.
Finally, we choose the translation hypothesis with [Table] [Table] the highest score S(ˆy) by:
[Equation]
, where z i = ˆ y i × k .
5. D ISCUSSION
The basic idea of HRT is to apply AT and NAT in sequence, which has been investigated by Kaiser et al.
; Ran et al.
; Akoury et al.
The main difference between these methods is the content of AT output, such as latent variable (Kaiser et al., 2018), reordered source token (Ran et al., 2019), and syntactic label (Akoury et al., 2019).
In contrast, our approach uses the deterministic target token as Ghazvininejad et al.
Coupling different decoding paradigms in one process is another line to incorporate AT and NAT.
Table 1 shows the differences between
HRT and existing methods, including SAT Wang et al.
(2018a) , RecoverSAT Ran et al.
(2020) and LAT Kong et al.
Although HRT needs more decoding steps than SAT and RecoverSAT, its non-autoregressive process is inexpensive due to the use of greedy search.
In contrast, these three methods require larger beams to explore translations in different lengths.
E XPERIMENTAL R ESULTS
Setup We mainly conduct experiments on four widely used WMT tasks:
WMT’
English ↔ Romanian (En ↔ Ro, 610k) and WMT’
English ↔ German (En ↔ De, 4.5M).
We replicate the same data processing for fair comparisons as Ghazvininejad et al.
To verify the effectiveness in long-distance language pairs, we also test it in the NIST Chinese-English (Zh → En, 1.8M) translation task following the setup of Wang et al.
(2018b).
We use the En ↔ Ro distillation data published by Ghazvininejad et al.
However, for En ↔ De and Zh → En, we retrained the [Table] [Table] [Table]
[Table]
AT teacher models because Ghazvininejad et al.
(2019) did not release the corresponding distillation data.
Specifically, we use the deep PreNorm Transformer-Base with a 20-layer encoder and the standard Transformer-Base as teacher models for En ↔ De and Zh → En, respectively.
We run all experiments on four 2080Ti GPUs.
Unless noted otherwise, we use the chunk size k =2.
We set p raw =0.
for En ↔ Ro and Zh → En, p raw =0.2 for En ↔ De according to validation sets.
We fine-tune HRT models on pre-trained AT models and take 100k/300k/100k training steps for En ↔ Ro/En
↔ De/Zh
→ En, respectively.
Other training hyperparameters are the same as Vaswani et al.
(2017) or Wang et al.
(2019) (deep-encoder).
We report both case-sensitive tokenized BLEU scores and SacreBLEU 9 .
Beam Size on HRT We first verify the influence of two beam sizes of HRT (b at and b nat ) on the
BLEU score and relative acceleration ratio.
We test three different setups, and the results are listed in Table 2.
Consistent with our observations in synthetic experiments, using b nat =1 only slightly reduces BLEU but significantly improves decoding efficiency.
Considering the trade-off between translation quality and speed and fair comparison with baselines 10 , we use b at =5 and b nat =1 unless otherwise stated.
Main Results Table 3 reports the BLEU and SacreBLEU scores on four WMT tasks.
Our HRT outperforms most existing NAT, IR-NAT, and Semi-NAT models and establishes new state-of-the-art results on En → De (See Appendix D for case studies.).
Compared to our re-implemented SAT with the same MixDistill, HRT obtains an improvement of +2.0 BLEU points.
We compared different data strategies in more detail in Section 7.
Besides, in line with Guo et al.
(2020), when using a deeper encoder, HRT-20L can further improve by approximately +0.5 BLEU.
We highlight that
HRT can achieve equivalent or even better performance than the teacher model when having the same model capacity.
In Table 4, we also report the experimental results on the Zh
→ En task.
HRT is once again superior to the original AT and CMLM model, which indicates that the effectiveness of HRT is agnostic to language pairs.
A NALYSIS
Impact of Chunk Size We tested chunk size k on the En → De task as shown in Table 5, where we can see that: (1) A large k has a more significant speedup on the GPU because fewer autoregressive steps are required; (2) As k increases, the performance of HRT drops sharply.
For example, k = is about 1.32 BLEU points lower than k =2 on the test set.
It indicates that the training difficulty of Skip-AT increases as k becomes larger.
We think that skip-generation may require more fancy training algorithms, which is left for our future work.
Data Strategy As shown in Table 6, we compared three data strategies, including raw data ( R ), sequence-level knowledge distillation ( D ), and proposed mixed distillation ( MD ).
Overall, MD is superior to other methods across the board, indicating that training with raw and distillation data is [Table] [Table] complementary.
We emphasize that HRT’s strong performance does not mainly come from MD .
For example, even if using D , HRT can achieve excellent results on the En → De task.
More surprisingly, we found that HRT can be slightly better than AT models in the same network scale trained by D .
We attribute it to two reasons: (1) HRT is fine-tuned on a well-trained AT model; (2) Multi-task learning on autoregressive and non-autoregressive tasks has better regularization than training alone.
Importance of Skip-AT & Skip-CMLM We try to understand the importance of Skip-AT and
Skip-CMLM for HRT.
To this end, we swap the intermediate results of two different HRT models (refer to Model A and B , respectively).
Specifically, we first use the Skip-AT decoder of A to generate its discontinuous target sequence.
Then we let B ’s Skip-AT decoder force decoding this sequence to obtain corresponding encoding representations and autoregressive model scores.
Finally, B uses its Skip-CMLM decoder to generate the complete translation result according to them.
The order of A and B can be exchanged.
In practice, we use two models (HRT and HRT-20L) with large performance gap as A and B respectively.
As shown in Table 7, we find that using the strong model’s
Skip-AT brings more improvement (+0.54 BLEU) than using its Skip-CMLM (+0.09 BLEU).
This result aligns with our claim that a good partial target context is critical.
Deep-encoder-shallow-decoder Architecture Kasai et al.
(2020b) point out that AT with deepencoder-shallow-decoder architecture can substantially speed up without losing translation accuracy.
We also compare HRT and AT in this setting: 12-layer encoder and 1-layer decoder, denoted by
HRT 12 − 1 and AT 12 − 1 , respectively.
AT 12 − 1 and HRT 12 − 1 use the same mixed distillation data for a fair comparison.
As can be seen from Table 8, both AT 12 − 1 and HRT 12 − 1 benefit from the change of layer allocation, gaining comparable BLEU scores and double decoding speed as the vanilla models.
Specifically, HRT 12 − 1 achieves an average acceleration of 4.2x/3.1x than AT baselines.
Kasai et al.
(2020b) report that CMLM does not enjoy this architecture, which indicates the success of HRT 12 −comes from using Skip-AT instead of Skip-CMLM.
Ablation Study In Table 9, we conduct ablation studies on the En → De task, including fine-tuning from pre-trained AT (FT), mixed distillation (MD), training steps (TS), and curriculum learning (CL).
We test two settings about CL: Fixing p k =1 is equivalent to removing auxiliary tasks; Fixing p k =0.5 assigns the same probability to the primary and auxiliary tasks.
All components contribute to the performance, but CL( p k =1) and TS are the most critical.
We also try to exclude them all from the vanilla HRT (-ALL), resulting in a total reduction of 1.53 BLEU points.
C ONCLUSION
We pointed out that IR-NAT suffers from inference acceleration robustness problems.
Inspired by our findings in synthetic experiments, we proposed HRT to combine the advantages of AT and NAT.
Experimental results show that our approach outperforms existing Semi-AT methods and promises to be a good substitute for AT due to competitive performance and stable speedup.
A D ETAILED INFERENCE SPEED
In Table we list the exact decoding time and relative speedup ratio of different models under varying environments on the En → De test set.
When changing the batch size from 1 to 32, the decoding time of AT reduces 20.4x/4.6x on GPU/CPU, respectively, while that of CMLMonly reduces 3.7/0.8x.
In contrast, HRT inherits the good character of AT and achieves 18.7x/3.8x speedup.
On the other hand, HRT has more robust acceleration than multi-shot NAT, such as
CMLM 4 , CMLM 10 .
When using the deep-encoder-shallow-decoder architecture, the performance of HRT 12 − 1 approaches the one-shot NAT (CMLM 1 ) on both GPU and CPU.
Besides, the overall results of HRT-20L is similar to those of HRT because the translation time is mainly consumed in the decoder.
We also report the change of inference speed along with chunk size k
[Table]
We trained all AT models in the synthetic experiment with the standard Transformer-Base configuration: layer=6, dim=512, ffn=2048, head=8. The difference from Ghazvininejad et al. (2019) is that they trained the AT models for 300k steps
[Table]
but we updated 50k/100k steps on En → Ro and En → De, respectively.
Although fewer updates, as shown in Table 11, our AT models have comparable performance with theirs.
C T RAINING ALGORITHM
Algorithm 1 describes the training process of HRT.
The HRT model is pre-initialized by a pretrained AT model (Line 1).
Each training sample B i randomly selects a raw target sentence Y i or its distilled version Y ′
(Line 4-6).
Then according to the schedule strategy p k
= (cid:16) tT (cid:17) λ , we can divide B into two parts: B p for primary tasks and B a for auxiliary tasks, where | B p | / | B | =
p k
(Line 7-8).
Next, we construct four kinds of training samples based on corresponding batches:
B atp ( TASK-SKIP-AT ), B ata ( TASK-AT ), B natp ( TASK-SKIP-CMLM ) and B nata ( TASK-CMLM ). Finally, we collect all training samples together and accumulate their gradients to update the model parameters, which results in the batch size being twice that of standard training. D C ASE STUDY Table 12 shows a translation example from En → De validation set.
Comparing CMLM and HRT, although both have the same masking rate (50%), we can see two main differences: (1) The distribution of masked tokens in CMLM is more discontinuous than in HRT (see blue marks); (2)
The
[Table]
Transformer
[Table]
[Table] [Table] [Table] [Table]
Input: Training data D including distillation targets, pretrained AT model M at , chunk size k , mixed distillation rate p raw , schedule coefficient λ
Output: Hybrid-Regressive Translation model M hrt 1: M hrt ← M at ▷ fine-tune on pre-trained AT 2: for t in 1 , 2 , . .
, T do 3:
X = { x 1 , . . .
, x n } , Y = { y 1 , . .
y n } , Y ′
= { y ′ 1 , . . .
, y ′ n } ← fetch a batch from D 4: for i in 1 , 2 , . . .
, n do 5:
B i =
( X i , Y ∗ i )
← uniformly sampling Y ∗ i ∼ { Y i , Y ′ i } with P ( Y i ) =
p raw ▷ mixed distillation 6: end for 7: p k ← ( tT ) λ
▷ curriculum learning 8:
B p , B a ← B (cid:2) :
⌊ n × p k ⌋ (cid:3) , B (cid:2)
⌊ n × p k ⌋ : (cid:3)
▷ split batch 9:
B atp , B natp ← construct training samples of primary tasks based on B p 10:
B ata , B nata ← construct training samples of auxiliary tasks based on B a 11:
Optimize M hrt using B atp ∪ B ata ∪ B natp ∪ B nata ▷ joint training 12: end for decoder input of HRT contains more correct target tokens than CMLM, thanks to the Skip-AT decoder (see wavy marks).
These two differences make our model easier to generate good translations than CMLM.
It also indicates that our model is capable of generating appropriate discontinuous sequences.
[Table]
Also problematic : civil military jurisdiction will continue to be uph@@ eld .
Reference
Auch problematisch : Die zivile Milit¨ar@@ geri@@ chts@@ barkeit soll weiter aufrechterhalten bleiben .
[Table]
[Table]
Problem@@ atisch : Die zivile milit¨arische Gerichts@@ (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) barkeit wird weiterhin aufrechterhalten .
[ EOS ]
HRT
Auch (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) problematisch : (cid:58)(cid:58)(cid:58) Die zivile (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)
Milit¨ar@@ geri@@ (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) chts@@ barkeit wird weiterhin (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) aufrechterhalten werden (cid:58) .
[ EOS ] [ EOS ]