Paper: Non-Autoregressive Machine Translation as Constrained HMM (ICLR-2024)

Reviewer comment (Evaluation):
The method does not seem to work for WMT zh-en, and the reasons for this also need explaining.

Author response:
The performance improvements our methods can bring to Directed Acyclic Transformer (DAT) across different translation directions are closely correlated with the performance gap between DAT and the Autoregressive (AR) Transformer.  
As confirmed both theoretically and empirically in [2], Non-Autoregressive (NAR) models are not as expressive as AR models.  
Hence, we can consider the performance of the AR Transformer as a theoretical upper bound for all NAR models.  
We have tabulated the best NAR model performances for all translation directions from the Table 1 of our paper, and compare them with the AR Transformer.  

|             | En-De | De-En | Zh-En | En-Zh |
|-------------|-------|-------| ------ | ----- |
| Transformer | 27.3  | 31.6 | 24.2 | 34.9 |
| DAT         | 26.9 (-0.4)| 30.6 (-1.0)|24.0 (-0.2) | 33.8 (-1.1) |
| AW-HMM/Bi-HMM | 27.2 (-0.1)| 31.4 (-0.2)| 24.1 (-0.1)|34.3 (-0.6)|

As can be seen, the DAT model can perform similarly to AR Transformer on En-De and Zh-En, with only a `0.4` and `0.2` BLEU gap. However, for De-En and En-Zh, the BLEU score differences are `1.0` and `1.1`, respectively.  
Given that our AW-HMM/Bi-HMM are also NAR models and DAT is already pretty close to the upper bound (AR Transformer) on En-De and Zh-En, the performance gains improvements appear to be relatively modest.  

This point has been clarified and added to our revised paper (see `the last paragraph in Section 5.1`). We sincerely thank you for bringing up this important issue.  

We hope that our response has satisfactorily addressed your questions. If there are any additional questions or points of clarification you require, we respectfully invite you to share them.  

**References**  

[1]: Directed Acyclic Transformer for Non-Autoregressive Machine Translation (Huang Fei et al., ICML 2022)  

[2]: An EM Approach to Non-autoregressive Conditional Sequence Generation (Zhiqing Sun, Yiming Yang, ICML 2020)

Reviewer comment (Experiments):
Can the proposed two strategies be applied to the DAT model at the same time?

Author response:
We attempted to integrate two strategies, but we did not see an improvement in performance compared to using each approach separately. We believe this is due to the following reasons:  
- In AW-HMM, as discussed in Section 5.2 (see Figure 4), there exists a delicate balance between mitigating label bias and preserving the expressiveness offered by a larger window size in AW-HMM.   
- In the context of Bi-HMM, L2R and R2L HMMs attempt to counteract the label bias by sharing all the parameters except for the transition matrix (which contains only 0.5M paramters). Consequently, the expressive capabilities of both L2R and R2L HMMs are somewhat restrained.  
- If we impose the window size constraint of AW-HMM on both L2R and R2L HMMs, the expressiveness of both HMMs is further reduced, which consequently did not lead to improved performance.  

We appreciate this insightful question and have incorporated a detailed discussion on this phenomenon in our revised manuscript (see `the first paragraph in Section 6`).

Reviewer comment (Evaluation):
Why does the proposed method behave differently in different translation directions?

Author response:
The performance improvements our methods can bring to Directed Acyclic Transformer (DAT) across different translation directions are closely correlated with the performance gap between DAT and the Autoregressive (AR) Transformer.  
As confirmed both theoretically and empirically in [2], Non-Autoregressive (NAR) models are not as expressive as AR models.  
Hence, we can consider the performance of the AR Transformer as a theoretical upper bound for all NAR models.  
We have tabulated the best NAR model performances for all translation directions from the Table 1 of our paper, and compare them with the AR Transformer.  

|             | En-De | De-En | Zh-En | En-Zh |
|-------------|-------|-------| ------ | ----- |
| Transformer | 27.3  | 31.6 | 24.2 | 34.9 |
| DAT         | 26.9 (-0.4)| 30.6 (-1.0)|24.0 (-0.2) | 33.8 (-1.1) |
| AW-HMM/Bi-HMM | 27.2 (-0.1)| 31.4 (-0.2)| 24.1 (-0.1)|34.3 (-0.6)|

As can be seen, the DAT model can perform similarly to AR Transformer on En-De and Zh-En, with only a `0.4` and `0.2` BLEU gap. However, for De-En and En-Zh, the BLEU score differences are `1.0` and `1.1`, respectively.  
Given that our AW-HMM/Bi-HMM are also NAR models and DAT is already pretty close to the upper bound (AR Transformer) on En-De and Zh-En, the performance gains improvements appear to be relatively modest.  

This point has been clarified and added to our revised paper (see `the last paragraph in Section 5.1`). We sincerely thank you for bringing up this important issue.  

We hope that our response has satisfactorily addressed your questions. If there are any additional questions or points of clarification you require, we respectfully invite you to share them.  

**References**  

[1]: Directed Acyclic Transformer for Non-Autoregressive Machine Translation (Huang Fei et al., ICML 2022)  

[2]: An EM Approach to Non-autoregressive Conditional Sequence Generation (Zhiqing Sun, Yiming Yang, ICML 2020)

Reviewer comment (Experiments):
**[major]**: Despite WMT'14 and WMT'17 being commonly used in the NAT literature, they are now way overhauled in the broader machine translation literature and should be replaced by more recent test sets to put the results into the context of recent research, see **[1]**.

Author response:
Firstly, we would like to clarify that **we aim to improve the best NAR model from a novel theorectical perspective, not to push the translation frontier**.

The primary focus of our study is on the extension of the understanding of the Directed Acyclic Transformer (DAT) as a left-to-right Hidden Markov Model (HMM) and to address the inherent **label bias** problem with our AW-HMM/Bi-HMM approaches. Our objective was to delve deeper into the theoretical aspects of this model and propose improvements in the same, rather than advancing the translation landscape. We believe our findings are original and can add new knowledge to our community.

The concerns you raise about the broader Non-Autoregressive Translation (NAT) community, while relevant to the field as a whole, fall outside the scope of this particular study.
We must highlight that even after the publication of DAT, many research papers continue to use WMT'14, WMT'16, WMT'17 translation tasks as the testbed for NAR models, e.g., `[2],[3],[4],[5],[6],[7],[8],[9],[10]`. Even the authors of `[11]`, whom you cited, used WMT'13, 14, 16 for their evaluations, as their goal was also not to push the translation frontier.

As `Reviewer z6kj` accurately highlighted, the potential limitations existing within the broader NAT community should not overshadow the specific contributions made in individual studies, such as ours, which aims to enhance the state-of-the-art NAR architecture from a novel HMM perspective.

Reviewer comment (Evaluation):
**[major]**: NAT papers should follow the broader machine translation standard to report their evaluation scores using `sacrebleu` and provide the corresponding hash that was used for generating the scores. This will ensure that scores are reproducible and do not vary across papers by up to 1.8 BLEU points due to varying tokenization and normalization, see **[2]**, **[4]**. Mixing `sacrebleu` and tokenized BLEU as done in Table 1 shouldn't be done and needs to be fixed.

Author response:
**About BLEU and SacreBLEU**

We choose to report BLEU for EN-DE and SacreBLEU for ZH-EN for the sake of a consistent and fair comparison with previous work, as it was hard to accurately reproduce all the previous baselines and report their SacreBLEU scores.

Reviewer comment (Experiments):
**[major]**: While it is nice to see that the paper attempts to provide GPU benchmarking numbers, the speed multipliers are heavily inflated since the baseline is a non-optimized autoregressive Transformer. There are many de-facto standard ways in practice to construct are more competitive autoregressive inference speed baseline with negligible translation quality drop using e.g. shallow decoders, shortlisting, or quantization (see **[2]**, **[5]**, **[6]**) which should be adopted here.

Author response:
**About GPU speedup numbers**

Comparing a non-optimized NAR model with an optimized AR model would be unfair. The AR model field has various optimization techniques for acceleration, while the NAR field, being at its early stage, lacks specially-designed speedup techniques.
However, it is crucial to note that the time complexity of NAR/AR model is $O(N)/O(N^2)$ [12], respectively. Theoretically, as indicated in [12], NAR models can surpass AR models in terms of decoding speed under any test conditions, provided that they are properly optimized. We look forward to such advancements in NAR accerleration, but again, it falls outside the scope of this paper.

Reviewer comment (Experiments):
**[major]**: Table 1 doesn't include parameter counts or inference speed numbers which makes it hard to compare the different approaches and understand if the improvement comes from the better algorithm or, simply, the increased parameter count capacity. For example, Bi-HMM uses two different parameter sets to model L2R and R2L and as a result they should have more parameters. Bigger baselines, potentially in parallel branches through e.g. MoE, or scaling up previous approaches might be needed.

Author response:
**About Parameter Count**

We list our parameter count below:
|             | En-De | En-Zh |
|-------------|-------|-------|
| Transformer | 65M   | 86M   |
| DAT         | 67M   | 88M   |
| AW-HMM      | 67M   | 88M   |
| Bi-HMM      | 68M   | 89M   |
|

The table indicates that our AW-HMM/Bi-HMM and DAT models only require a `3%-5%` increase in parameter count compared to the baseline models. This suggests that our performance improvements are due primarily to our algorithm, rather than an increase in parameters.

This information has been added to our revised paper (see Appendix D).

Reviewer comment (Experiments):
**[minor]**: It is unclear how well the proposed approach extends to the multilingual setting.

Author response:
**Response to [minor 1]**

Our models/methods are not language-specific, and should thus be applicable to other language pairs. We chose En-De and En-Zh to maintain consistency with previous work, as in [1].

Reviewer comment (Experiments):
**[minor]**: Figure 4 doesn't show a clear trend in the window size, making it hard to extrapolate the findings to other language pairs or datasets without additional analysis for the dataset at hand. This will require additional grid search tuning trials to adopt and no guidance on how to tune this parameter is given.

Author response:
**Response to [minor 2]**

Regarding the window size trend experiment depicted in Figure 4, we confined our investigation to the WMT'14 En-De dataset. This decision was made due to the significant computational cost associated with examining 8 different window sizes (ranging from 1 to 8) for each of the 4 translation directions (En-De, De-En, Zh-En, En-Zh).

However, please note that we have conducted experiments with window sizes of 7 and 8 for all language pairs, the results of which are documented in Appendix B.

In relation to the tuning of the window size $\beta$ hyperparameter, we followed standard hyperparameter tuning practices. The hyperparameter was tuned on the development set, a process that is mentioned in the caption of Figure 4.

Reviewer comment (Experiments):
How were the hyperparameters tuned for the proposed method and the previous works? If defaults were used for previous methods, the comparison needs to potentially be adjusted to also allow hyper parameter tuning for those methods.

Author response:
**Response to Questions**

Our training hyperparamters follow previous work [1]. For the hyperparmeters in our AW-HMM and Bi-HMM, we tune them on the dev set.