Paper: ResiDual: Transformer with Dual Residual Connections (ICLR-2024)

Reviewer comment (Theory):
Their derivations of theorem 3.1 ignore the impact of non-linearity on the gradient of the transformer MLP layer, instead assuming it to be a single FC.

Author response:
**Q2: The derivations of theorem 3.1 ignore the impact of non-linearity on the gradient of the transformer MLP layer, instead assuming it to be a single FC.**

First, Theorem 3.1 is to illustrate the gradient norm scale across layers in a pre/post-norm transformer, and the MLP with or without ReLU activation in between does not change conclusion at all because of the piece-wise linear property of ReLU and statistical concentration property of randomly initialized weights. Second, the analysis of the forward and backward processes through ReLU activation is straightforward and readily available in many previous literature (Allen-Zhu & Li 2019 “A convergence theory for deep learning via over-parameterization”, Xiong et al.2020 “On layer normalization in the transformer architecture” and many others). Given such circumstances, we choose to present the result in a concise way without too many sub/super scripts, where we use a simple MLP layer. We have revised the paper with these points clearly stated.

Reviewer comment (Theory):
The derivation also ignores back-propagated gradient through the Query - while Keys will not back-propagate any gradient as Query is zero-initialized, Query will have non-zero gradient back-propagating.

Author response:
**Q3: The derivation also ignores back-propagated gradient through the Query.**

In our setup, we initialize $W_Q$ to be zero matrices. When doing back-propagation, there is no backward signal on the $xW_K$ branch because of $W_Q=0$. We can compute the gradient on $W_Q$ matrix but there is no backpropagated signal through the $W_Q$ matrix. Hence, our proof and the claim in Theorem 3.1 are flawless as we denote the block k’s parameters to be $w_k$ which includes the MLP layer weights and query matrix. Nonetheless, we will add a note to avoid future misunderstanding.

Reviewer comment (Evaluation):
As the improvements can sometimes be small (For eg. 27.65 vs 27.3 in Table 3), some measure of statistical significance of the improvements is required.

Author response:
**Q4: Statistical significance test.**

For baseline model (i.e., B2T), we use the checkpoint they provided on the original authors’ GitHub and generated the output for WMT2010 to WMT2016. We then used Moses tool (https://github.com/moses-smt/mosesdecoder/blob/master/scripts/analysis/bootstrap-hypothesis-difference-significance.pl) to compute the significance value (P-value). The output is:
Assuming that essentially the same system generated the two hypothesis translations (null-hypothesis), the probability of actually getting them (p-value) is: 0.001.
Therefore, the null hypothesis, i.e., our system is the same as baseline, should be rejected (p<0.001). In another word, our improvement is statistically significant.

Reviewer comment (Presentation):
Discussion of prior work is somewhat lacking - while the authors reference prior works throughout the text, a dedicated section for more detailed discussion is missing.

Author response:
**Q5: More discussion about previous work is missing.**

We appreciate your feedback on this point. In the first section, we have categorized previous works into three groups: modifying architecture, weights, and initialization. However, if there are additional aspects of previous work that need further discussion, we would be grateful if you could provide more details.