From: ICLR.cc/2023/Conference/Paper4190/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper4190/Authors
title:  Response to Reviewer #2
comment: Thanks for your insightful reviews, and we appreciate the valuable suggestions! We address your concerns and questions as follows:

### 1. Much hyper-parameter tuning

We thank the reviewer for pointing out this issue. We want to clarify that both k and beam size is constant across all main experiments. For example, we use k=2, $b_{at}$=5, and $b_{nat}$=1 regardless of languages. As for the distillation ratio,  we empirically found that a distillation ratio of 0.5 works better for small/medium-scale datasets (e.g. En-Ro, En-Zh). For the harder En-De task, Table 6 shows that there is a small gap (0.19 BLEU) between mixed distillation (ratio=0.8) and full distillation (ratio=1.0). Therefore, we recommend using a distillation ratio of 0.5/1.0 for good results quickly. 

### 2. Further research direction in this area
   
We fully agree with your suggestion. Recent work [1] shows that many previous studies may exaggerate the advantages of non-autoregressive translation. We believe that there are many complementarities among different translation paradigms. Following this line, our work proposes an efficient and effective method of combining different translation paradigms, achieving high BLEU and robust decoding speed.


### 3. Related work on mixed distillation
Excellent advice. Mixing raw data and distillation data in autoregressive translation is a common data augmentation method in WMT competitions [2][3][4]. We added these related works in the revised version. 


### 4. Ablation on curriculum learning
We conduct this experiment by fixing the distillation ratio as 0.5 ($p_k$=0.5) in the revised version. That is, in the whole training process, half of the samples of a batch are always for the primary tasks, and the other half is for the auxiliary tasks. We list the result below:

| System | BLEU | $\Delta$ |
| ---- | ---- | ---- |
| HRT | 28.49 | ref. |
| -CL($p_k$=1) | 27.72 | -0.77 |
| -CL($p_k$=0.5) | 27.93 | -0.56 | 

We can see that fixing $p_k$=0.5 is slightly better than $p_k$=1, indicating that combining the primary and auxiliary tasks is complementary. At the same time, the BLEU gap between our curriculum learning (0.56) also proves the importance of removing the bias in the auxiliary task.


### 5. Too dense information in Figure 4

The information in Figure 4 and Table 10 is somewhat duplicated. In order to compare different methods more clearly, we deleted Figure 4 and improved Table 10. Please refer to Table 10 in the revised version for details.


[1] Non-Autoregressive Machine Translation: It's Not as Fast as it Seems. NAACL 2022.
[2] Findings of the 2019 Conference on Machine Translation (WMT19). WMT 2019.
[3] Findings of the 2020 Conference on Machine Translation (WMT20). WMT 2020.
[4] Findings of the 2021 Conference on Machine Translation (WMT21). WMT 2021.

From: ICLR.cc/2023/Conference/Paper4190/Authors
title: response for unfair comparisons
comment: 
### 3. Unfair comparisons

- HRT use raw + distilled data
  
Sorry for your confusion. We want to clarify that we have reported the different data strategies (raw, distill, mixdistill) in Table 6. The BLEU score of using full distillation data is 28.30 on the En-De test set, which is only 0.19 lower than using mixdistill. We emphasize that mixdistill is optional for the good performance of HRT. 

- Compare to recent SOTA (Fully-NAT, DA-Transformer etc)

As we respond to reviewer 1, we emphasize that exactly comparing the speed of every listed SOTA model is not trivial due to the differences in hardware and implementation. We use CMLM with different iterations as the speed agency of different NAT models because the number of iterations mainly determines the speed of NAT models. Specifically, the speed of $CMLM_1$ is close to general one-shot NAT models, such as the listed models by the reviewer. As shown in Table 10, when decoding with batch size 32 on GPU, our vanilla HRT has already run faster than $CMLM_1$. Furthermore, the advantage is further expanded when using deep-encoder-shallow-decoder architecture. We want to remind the reviewer that the previous one-shot models (Fully-NAT, GLAT-CTC etc)
will have the same problem and even worse because the CTC-style model requires a longer target sequence than CMLM. 
As for translation quality, we add the result of DA-Transformer (BLEU 27.91 on WMT En-De). We note that HRT obtains SOTA results in widely used WMT En-De task, comparing with existing one-shot NAT models or iterative NAT models (see Table 3, including GLAT, Imputer, Disco etc). We also validate the good translation performance in larger model capacity (HRT-20L: BLEU 29.14) and the high-efficient deep-encoder-shallow-decoder (HRT_E12D1: BLEU 28.44). We hope our clarification can address your concerns.

We hope the above clarification will help you understand our work better and welcome any further comments.

From: ICLR.cc/2023/Conference/Paper4190/Authors
title: Response to Reviewer #4
comment: Thanks for your insightful reviews, and we appreciate the valuable suggestions! We address your concerns and questions as follows:

### 1. Less novelty compared to previous work

Without offense, we disagree with the reviewer's comment on the novelty.
In the original paper, we have clearly pointed out that the combination of AT and NAT is nothing new. Our contribution lies in finding a simple but effective way to combine these two translation paradigms better. We have detailed discussed the differences between existing work in Section 5.4, including SAT, RecoverNAT etc. As suggested by the reviewer, we added the comparison to LAT (<br> denotes parallel generation of "a $\rightarrow$ b $\rightarrow$ c" and "d $\rightarrow$ e $\rightarrow$ f") and updated Table 1 as the following:


| Method | Generation |
| ---- | ---- |
| SAT | a,b $\rightarrow$ c,d $\rightarrow$ e,f  |
| RecoverNAT |  a,c,e $\rightarrow$ b,d,f |
| LAT | a $\rightarrow$ b $\rightarrow$ c <br> d $\rightarrow$ e $\rightarrow$ f | 
| HRT | a $\rightarrow$ c $\rightarrow$ e $\dashrightarrow$ b,d,f |


We emphasize that one of the most important findings of our work is that NAT can obtain very strong translation performance when providing part of good target tokens. We validate the conclusion by synthetic experiment (Section 4) and the extensive experimental results of the proposed HRT (Section 6, 7). 
In contrast, as shown in the above Table, previous methods (SAT, RecoverNAT, LAT) do not guarantee the good quality of the target context before the non-aoturegressive stage. Taking RecoverNAT as an example, before generating "b,d,f", the known context is the non-autoregressive prediction of "a,c,e", which is definitely worse than the autoregressive prediction as HRT. It also explains why HRT performs better than previous similar work. 



Moreover, we also want to kindly remind the reviewer that there are two additional main contributions in this paper:

- As recognized by other reviewers, we comprehensively investigate the speedup robustness problem in NAT models, which was ignored by the community before. To our best knowledge, we are also the first to prove that the combination of AT and NAT can alleviate this problem.

- Another considerable advantage of the proposed HRT is the good character of fitting the deep-encoder-shallow-decoder architecture. Please note that the previous study [1] shows that the conventional NAT models fail to benefit from this efficient architecture. As shown in Table 8, the average speedup ratio of HRT improve from 1.7/1.6 to 4.2/3.1 in GPU and CPU, respectively, with almost no effect on BLEU. We argue that making deep-encoder-shallow-decoder work well in NAT is not easy. To our best knowledge, we are also the first to prove that the combination of AT and NAT can solve this issue.

According to the above clarification, we hope the reviewer can revisit to evaluate the contribution and novelty of our work. Also, we hope our statement can break your concern about why HRT performs better than RecoverSAT. 

### 2. Method is too straightforward

The reviewer points out two possible improvements to the model, including dynamic chunk size and separate model parameters. We agree that these modifications may lead to further performance improvement. However, we kindly remind the reviewer that our purpose is to make our method as simple as possible to solve the observed speedup robustness problem. Specifically, we share AT and NAT model parameters for no increase in parameter size. The simple static chunk size also makes our model keep the same network architecture as the vanilla Transformer. We try our best to make the proposed method simple and reproductive. 
Our intensive experiments also validate the excellent performance in both translation quality (BLEU) and speedup robustness (cf. Table 10). We hope the reviewer can take a new look at the contributions of our simple but effective approach and put aside your preference for complexity models.

From: ICLR.cc/2023/Conference/Paper4190/Authors
title: Response to Reviewer #3
comment: Thanks for your insightful reviews, and we appreciate the valuable suggestions! We address your concerns and questions as follows:

### 1. Deal with BPE

Yes, it is possible to happen that skip-at generates a subword as the start word. We split the original BPE sequence into chunks without any special process. We do not force training skip-at on word sequence, although it makes more sense from the linguistic aspect. Training on BPE sequence is simpler and can guarantee the determinate slots for non-autoregressive generations. 

### 2. Meaning of 'AT + Raw'

'AT + Raw' represents the standard autoregressive model trained by raw training data. There is no distillation data used. More comparisons about different data (Raw, Distill, MixDistill) can be found in Table 6. Another note is that the AT teacher for our WMT En-De task is AT-20L (20-layer encoder, 6-layer decoder), which achieves a BLEU score of 28.79 on the En-De test set. 

### 3. Subscript of CMLM in Figure 1
   
Sorry for the confusion. The subscript represents the number of iteration counts instead of beam size. More iteration count means the model refines the translation more time, which generally results in higher quality and slower inference speed.
 Figure 1 illustrates that non-autoregressive models suffer speedup degradation problems along with the increase of decoding batch size. And the situation is worse if using a larger iteration count (e.g. $CMLM_{10}$). For beam size, we use $b_{at}$ and $b_{nat}$ to denote the corresponding beam size of the autoregressive and non-autoregressive parts of HRT.

### 4. The problem in Table 2
   
It seems that the reviewer has some misunderstanding about Table 2. We kindly remind the reviewer that \alpha(AG) and \alpha(AC) denote the average speedup ratio than the autoregressive baseline under different decoding batch sizes. The larger the value is, the faster the inference speed is. Therefore, Table 2 shows that using smaller beam size on both Skip-AT and Skip-CMLM ($b_{at}$=1, b_{nat}=1) has the fastest speed: \alpha(AG)=2.1, \alpha(AC)=2.8. While increasing $b_{at}$ from 1 to 5, \alpha(AG)=1.7 and \alpha(AC)=1.6, means slower inference speed.

We hope the above clarification can help to understand our work better.

From: ICLR.cc/2023/Conference/Paper4190/Authors
title: Response to Reviewer #1 
comment: 
Thanks for your insightful reviews, and we appreciate the valuable suggestions! We address your concerns and questions as follows:

### 1. Not match the motivation of this paper well

We kindly remind the reviewer that HRT can definitely alleviate the speedup degradation problem by using $b_{nat}$=1 in the non-autoregressive part, as explained in Table 2. Unlike general NAT models, which use a larger beam size (e.g. 5) to explore different length candidates, HRT is insensitive to the beam size in non-autoregressive decoding (please see the table below). Actually, HRT can even obtain a higher speedup ratio if using beam size 1 in both AT and NAT, with only slight performance loss (cf. Table 2). However, for a fair comparison with other systems, we use the setting of $b_{at}$=5 and $b_{nat}$=1. 

|  Model   | BLEU  |
|  ----  | ----  |
| CMLM$_{10}$ (beam=1) | 26.56 |
| CMLM$_{10}$ (beam=5)  | 27.03 |
| GLAT | 25.21 |
| GLAT + NPD (m=7)  | 26.55 |
| Our HRT($b_{at}$=5, $b_{nat}$=1) | 28.49
| Our HRT($b_{at}$=5, $b_{nat}$=5) | 28.65

Another note is that the speed of $CMLM_1$ can be regarded as a typical representative of one-shot NAT, as we argued in footnote 2. Detailed experimental results in Table 10 show that HRT($b_{at}$=5, $b_{nat}$=1) even runs faster than $CMLM_1$ when under batch size 32 on GPU, which indicates that HRT could have a higher speedup ratio than one-shot NAT in specific conditions. Moreover, the advantage of HRT is further expanded when using deep-encoder-shallow-decoder architecture. Therefore, we argue that HRT can effectively alleviate the speedup degradation problem in general NAT models, matching our motivation well.

### 2. Speed-quality trade-off comparison between different NAT models is not convincing

We want to emphasize that exactly comparing the speed of every listed SOTA model is not trivial due to the differences in hardware and implementation. Instead, we use CMLM with different iterations as the speed agency of different NAT models. The reason is that **the speed of NAT models is mainly determined by the number of iterations**. For example, $CMLM_1$ can be regarded as the speed upbound of CTC-style one-shot NAT (e.g. CTC-GLAT) because the CTC model requires generating a longer target sequence (e.g. 3x source sequence length). Likely, we use $CMLM_4$ and $CMLM_{10}$ as the representative of iterative NAT models because the iteration number 4 and 10 is widely used in existing iterative NAT models, such as Imputer (I=8), SMART (I=10), Disco(I$\simeq$4). Therefore, we think the results comparing HRT and CMLM are meaningful for speed comparison. From Table 10, we can clearly see that HRT($b_{at}$=5, $b_{nat}$=1) significantly runs faster than $CMLM_4$ and $CMLM_{10}$ in both GPU and CPU when the batch size increases to 32. Besides, with batch size 32, HRT has a closing inference speed to $CMLM_1$ on GPU, which indicates the efficiency of HRT.
As for translation quality, we add the result of DA-Transformer (BLEU 27.91 on WMT En-De). We note that HRT obtains SOTA results in widely used WMT En-De task, comparing with existing one-shot NAT models or iterative NAT models (see Table 3, including GLAT, Imputer, Disco etc). We also validate the good translation performance in larger model capacity (HRT-20L: BLEU 29.14) and the high-efficient deep-encoder-shallow-decoder (HRT_E12D1: BLEU 28.44). We hope our clarification can break your concern.

### 3. Performance drop when k>2

First, we agree with the review that it is difficult for AT to generate a discontinuous sequence, especially using a larger interval. However, we want to kindly remind the reviewer that the BLEU of HRT(k=3) is 27.98 on En-De test set, which only drops about 0.5 BLEU than that of k=2 but improves about 50% speed than k=2. Comparing existing work in Table 3, we think 27.98 is still a competitive number, and the significant speedup benefit should not be ignored. We think it is interesting to explore how to reduce the cost of AT prompt, for example, using a smaller autoregressive model or lightweight SMT system. We left these in our future work.

### 4. Mixed distillation 

We thank the reviewer for pointing out this issue. We want to clarify that mixed distillation is not necessary for training HRT. In Table 6, we elaborate on the performance with different dataset strategies. When using the standard distillation dataset, HRT obtains a BLEU of 28.30 on the WMT En-De task. These results show that HRT is robust on the training data. As shown in Table 6, mixed distillation works better in small-scale datasets (e.g., En-Ro). 

Thanks again for your insightful reviews and we hope the above clarification will help you understand our work better.

From: ICLR.cc/2023/Conference/Paper4190/Reviewer_RnHw
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper analyzes that NAT suffers from speedup degradation when the decoding batch size increases and proposes a hybrid semi-autoregressive translation model to combine both AT and NAT’s advantages. The authors design a hybrid model (i.e., HRT) which first works in an autoregressive way to generate partial outputs (i.e., first word in each chunk) and then generates remaining outputs fully parallel with only one iteration. Some training strategies (e.g., mixed distillation) are also proposed to train their hybrid model. Extensive experiments demonstrate good translation quality of their model.
strength_and_weaknesses: Strength: 
a) The problem of speedup degradation with large batch size is interesting and ignored by community before. 
b) The proposed method to combine both AT and NAT is novel.
c) Extensive experiments have been conducted.
d) The paper is well-organized and easy to follow.
Weaknesses:
a)	My major concern is that the proposed method does not match motivation of this paper well. The authors argue that NAT suffers from speedup degradation with large batch size. However, the proposed HRT, which is a semi-autoregressive model, does not alleviate this problem. Oppositely, HRT significantly slows down the decoding speed compared with fully-NAT. As shown in Fig.4, the speedup of proposed HRT is less than 2× under all settings of batch size.
b)	The speed-quality trade-off comparison between different NAT models is not convincing. In Table 3, the authors only list the iterations of different NAT models instead of speedup ratios in practical. In Table 4, the authors only compare speedup trade-offs between proposed HRT and two very old baselines (i.e., SAT and CMLM). Considering recent fully-NAT (e.g., CTC+GLAT, DA-Transformer) or iterative models (e.g., Imputer, Disco) achieve much better speedup trade-off than either CMLM or SAT, the comparison in Table 4 &10 is not sufficient. 
c)	In Section 4, the authors claim that CMLM can generate good translation when only 30% AT outputs exposed. However, performance of proposed HRT drops sharply when chunk size is larger than 2. I suspect the reason is that AT is hard to generate a high-quality discontinuous sequence with large interval. It results in that HRT heavily relies on AR decoding to generate large parts of the output, which severely hurts its parallelism.
d)	Mixed distillation is applied to train proposed HRT. This technique is incremental and hurts the fairness of comparison in Table 3.

clarity,_quality,_novelty_and_reproducibility: The paper is clear and the idea is novel. The community could reproduce the results with some difficulty.
summary_of_the_review: The searched problem is interesting and novel. However, the proposed method has some major limitations, making it heavily rely on AR decoding and hurting its speedup ratio. The comparison in experiment part is not convincing. Authors only compare speed-quality trade-off of HRT with very old baselines and recent SOTA fully-NAT models or iterative models are ignored.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper4190/Reviewer_r7mp
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper proposes a new hybrid auto-regressive / non-autoregressive (AT/NAT) translation model. The model has two steps: first an AT model is used to generate an incomplete sequence of words while in a second step a NAT model fills the gaps. The authors show that with careful tuning (training with raw and distilled data, curriculum over the losses, etc.) the model achieves state of the art results (Bleu scores) and is faster than iterative NAT models of similar quality.

The authors provide preliminary experiments to investigate one-step NAT given partial translation output: specifically how much of the translation needs to be present such that a NAT model achieves good results when “filling the gaps” in a single step. They use these to determine both the ratio of present/missing tokens and the optimal type of masking pattern.

strength_and_weaknesses: Strengths:

* Paper: The analysis that shows NAT vs AT decoding speed for various batch sizes is very informative. So are the preliminary experiments to motivate the approach and the comprehensive empirical results. 
* Method: The NAT and AT models share the parameters, both the encoder and the decoder, which brings additional computational advantages to the method. No need to predict the target length. The results are very good.

Weaknesses:

* A lot of tuning was needed to obtain the good quality and quality/speed trade-off (value of k, beam size, distillation ratio). While results in Table 4 on Zh→En indicate the method is likely to be robust, it would have been preferable to tune on a single language (e.g. de or ro) and test on several others.
* Work in this area should move towards a better understanding of translation quality and quality differences between different paradigms. For example, investigations into robustness, potential bias amplification, error type distribution, or even small-scale human-assessed quality are not standard. This hinders our understanding of such work and future efforts to improve MT.

Other comments:

* Mixed distillation: training on a mix of both raw and distilled data is common for AT MT. Please refer to the relevant previous work on this.  
* Ablation on curriculum learning: what are the results when using uniform weights for the four losses? (the curriculum learning ablation in Table 9 removes the two auxiliary tasks).
* Figure 4: too dense in information


clarity,_quality,_novelty_and_reproducibility: The work is sound and clearly written: the authors thoroughly motivate the approach and the modeling choices throughout the paper. While hybrid AT/NAT approaches are not new, the specific model proposed in the paper has significant quality advantages. Latency implications are also thoroughly investigated. 
summary_of_the_review: The paper introduces a new AT/NAT MT model which is a significant improvement in both quality and speed over previous methods. The empirical results are thorough and each step of the work is well motivated.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper4190/Reviewer_PeQ7
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper starts with investigating the decoding speed and Mask strategies on both AT and NAT models. According to their findings, they propose a hybrid-regressive method to take advantages of AT and NAT models into account for keeping the decoding quality and efficiency at once.  This model is trained on the multi-task framework with curriculum learning and mixed distillation technical. In the inference process, this model adopts a two-stage generation strategy. They first autoregressively generate the start words of each chunk, then use a NAT decoder to recover all the masks in each chunk. 

The contributions of this work:
1. Learning the acceleration robustness problem in IR-NAT;
2. Investigating the effectiveness of masking ratio on the inference process;
3. They proposed a hybrid-regressive method to balance the advantages of AT and NAT;
4. The proposed method outperforms the baseline on three language pairs in both speed and quality of generation.
strength_and_weaknesses: Strength:
1. This model is simple and effective, it considers the trade-off between the decoding speed and translation quality.
2. The primary analysis is described clearly and the target problem is well-defined.
3. It is pretty interesting to see a NAT model without a length predictor. This work uses AT model to get the length of the target to cut down the error propagation.

Weakness:
1. It is unclear how this method works with the BPE method.  Will the skip-at generate a subword like '@@ke' as the start word in a chunk? 
2. Unclear description, e.g. (1) In table 3, what is the ' AT + Raw' method? Distill data + Raw data? but AT can get 27.45 BLEU with only raw data.  (2) What is the subscript of CMLM  in figure 1 mean? iteration or beam size? If it is the beam size, this symbol(page 3) appears before its definition (page 5).
3. (minor) Table 2, beam size 1 slower than beam size 5 did not make sense.
clarity,_quality,_novelty_and_reproducibility: This paper is well constructed, but there still are some problems that should be revised (refer to weakness).  The proposed method is simple and the settings are written clearly but there are some random settings that might hurt the reproducibility.
summary_of_the_review: The proposed method is simple and effective. However, there are still many places that need to explore, such as Weakness 1, the learning decay of curriculum learning. Although the writing of this paper is good, there are some places that need to be revised.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper4190/Reviewer_e4XY
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: Kasai et al. (2020) point out that non-autoregressive translation with an iterative refinement mechanism (IR-NAT) suffers from poor acceleration. In response to this problem, this paper builds a hybrid autoregressive model that first generates discontinuous sequences via autoregression, and then fills in all previously skipped tokens in a non-autoregressive manner. It is a promising direction to explore acceleration under the situation of batch size being large enough instead of batch size is 1 as done in the most existing NAT papers. This problem is deserved to be studied and the proposed method can accelerate at least 1.5x. 
strength_and_weaknesses: Pros:
1. The paper is well-written and easy to follow. 
2, The experiments across language pairs are comprehensive, and the analyses are well-designed.

Cons:
1. Novelty: This paper is not the first one that studied the combination of AT and NAT. [1] and [2] both studied this direction and proposed similar solutions. The differences among these papers mainly lie in the part of the generation order of AT and NAT. In my view, this paper does not give a good explanation of why the strategy they adopted is better than others. It seems they lack such component-wise study.

2. The proposed method is too straightforward and not so elegant. The proposed chunk-based autoregressive model is not flexible and has also been studied in [6]. It is not elegant to choose a static chunk $k$, and a dynamic $k$ determined by different source sentences will be a more interesting method. Additionally, the proposed method should maintain both NAT and AT parameters which is infeasible.

3. The comparison is not strictly fair. For the results in Table 3, the authors listed previous NAT models trained with standard distillation datasets, while their model could access both raw data and the kd data by mixdistill. And the mixdistill is not proposed by the authors. It would be better to add one line to show HRL trained with only kd data. Furthermore, for the results in Figure 4 (and also Table 3), the authors ignore new SoTA NAT models like CTC-based models[3][4] and da-transformer[5], which may have a better acceleration against the proposed method with pure NAT model and only one decoding iteration.

[1] Ran et al, Learning to Recover from Multi-Modality Errors for Non-Autoregressive Neural Machine Translation. ACL 
[2] Kong, Xiang, Zhisong Zhang, and Eduard Hovy. "Incorporating a local translation mechanism into non-autoregressive translation." EMNLP
[3]Gu, J., & Kong, X. (2020). Fully non-autoregressive neural machine translation: Tricks of the trade. ACL
[4] Shao, Chenze, Xuanfu Wu, and Yang Feng (2022). "One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation." NAACL
[5] Huang, F., Zhou, H., Liu, Y., Li, H., & Huang, M. (2022). Directed Acyclic Transformer for Non-Autoregressive Machine Translation. ICML
[6] Wang et al, Semi-Autoregressive Neural Machine Translation. EMNLP 
clarity,_quality,_novelty_and_reproducibility: While it is mostly clear and well-written, there is very little in terms of novelty with respect to that previous work. The main contribution of this paper is a combination of existing works like MixDistll and Deep encoder - shallow decoder. The core technical contribution is similar to previous works (e.g. RecoverSAT) and the authors only use empirical results to illustrate why the proposed method is better than RecoverSAT without clear explanation.
summary_of_the_review: While this paper is mostly clear and well-written, there is very little in terms of novelty with respect to that previous work (e.g. RecoverSAT). It is a promising direction to explore the acceleration of large-batch decoding. This paper would be improved significantly if you could propose more innovative methods.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough