Paper:
Non-autoregressive machine translation (NAT) systems predict a sequence of output tokens in parallel, achieving substantial improvements in generation speed compared to autoregressive models. Existing NAT models usually rely on the technique of knowledge distillation, which creates the training data from a pretrained autoregressive model for better performance. Knowledge distillation is empirically useful, leading to large gains in accuracy for NAT models, but the reason for this success has, as of yet, been unclear. In this paper, we first design systematic experiments to investigate why knowledge distillation is crucial to NAT training. We find that knowledge distillation can reduce the complexity of data sets and help NAT to model the variations in the output data. Furthermore, a strong correlation is observed between the capacity of an NAT model and the optimal complexity of the distilled data for the best translation quality. Based on these findings, we further propose several approaches that can alter the complexity of data sets to improve the performance of NAT models. We achieve the state-of-the-art performance for the NAT-based models, and close the gap with the autoregressive baseline on WMT14 En-De benchmark.

Review:
In this paper, the authors investigate non-autoregressive translation (NAT). They specifically look into how using different auto-regressive translation (AT) models for knowledge distillation impacts the quality of NAT models. The paper is well organised and the experiments are sound and interesting, shedding light on an aspect of NAT models that's been rather dismissed as secondary until now: the impact of varying AT knowledge distillation.
First, so as to better please those out there who are more "state-of-the-art" inclined, I suggest the authors to better emphasise their improvements. Results obtained by their analysis can lead to improvements as stated in the last paragraph of Section 4. This could be better stressed in the introduction and it would make the main take-away messages from the paper stronger.
On a more general note, I would like to know how robust these models are. I understand this is a problem NAT and machine translation papers have in general, but I would still like to suggest the authors train each model from scratch multiple times using different random seeds, and report mean and variance of their results (i.e. BLEU). Although I understand that training each AT model 4 times and each NAT model 4x4 times (multiple times for each AT model trained) is unfeasible, you could still report mean and variance separately for AT and NAT, and simply choose one out of the 4 trained AT models per architecture to perform distillation. I recommend training each model at least 3 times and reporting mean BLEU and variance.
I would also recommend using other MT metrics (e.g. chrF, METEOR, BEER, etc.), since BLEU is a comparatively weak metric in terms of correlations with human judgements. For translations into German, look into characTER, chrF and BEER. For more information on the different metrics available, how they correlate with human judgements, and which better apply to different target languages / language pairs, please refer to the WMT evaluation/metrics shared tasks. [1]
I have a few general comments and suggestions:
- In Section 3.1, when introducing the NAT model (simplified version of Gu et al., (2019)), be more specific. I would like to see an actual description of what the modifications consist of in more detail: is there a sentence length prediction step? what happens when source length is different from source length? etc.
- In Section 3.2, right after Equation (3), the authors refer to "x and y" in an inverted manner, please fix it.
- In Section 4.3, you mention that two of the decoding methods used involve "random sampling", which I find misleading. You probably mean sampling according to the model distribution p(y_t | y_{<t}) for all t, which is not random. I suggest you simply remove the word "random" and mention that you use "sampling" and "sampling within the top-10 candidates". Also, when you sampling using the top-10 candidates, do you simply re-normalise the probability mass to include only the top-10 candidate tokens?
- As a suggestion, Tables and Figures could be more self-contained. There is always a compromise between conciseness and clarity, added by the fact that the page limit makes things even harder. However, I would recommend including more information in Table/Figure captions, especially in Figure 3 and Tables 3 and 4. Try to at least mention the training/evaluation data (dev or test?). Ideally one should understand it from reading the abstract and carefully reading the caption.
- In Appendix A, you mention that you select models according to validation loss. Is that really the case? If so, why? I am not sure whether validation loss (i.e. word cross-entropy on the validation set) should correlate so well with translation quality (or sentence-level metrics such as BLEU).
[1] http://www.statmt.org/wmt19/metrics-task.html