Paper: One Forward is Enough for Neural Network Training via Likelihood Ratio Method (ICLR-2024)

Reviewer comment (Writing):
It is difficult, at least for me, to follow the technical details of the proposed ULR for the following reasons: In Section 3.1, several expectations ($\mathbb{E}$) appear to explain the proposed method. But I could not understand in which variables and distributions these expectations are defined.

Author response:
**AW1:** Thank you for your feedback. We will modify the formulas and definitions for readability in the revision. We provide a step-by-step expansion of the loss expectation, explicitly annotating the random variables, to further illustrate details in the derivation of ULR as follows: $$ \mathbb E_{z^0,\cdots, z^{L-1}}[\mathcal L (x^L)] 
= \mathbb E_{x^l}[\mathbb E_{z^l,\cdots, z^{L-1}}[\mathcal L (x^L)|x^l]] 
= \mathbb E_{x^l}[\mathbb E_{x^{l+1}}[\mathbb E_{z^{l+1},\cdots, z^{L-1}}[\mathcal L (x^L)|x^{l+1},x^l]|x^l]] 
= \mathbb E_{x^l}[\int_{\mathbb R ^{d_{l+1}}} \mathbb E_{z^{l+1},\cdots, z^{L-1}}[\mathcal L (x^L)|\xi,x^l ]g^l(\xi)d\xi],$$ where the first two equalities are justified by the law of iterated expectation, and the third equality comes from the definition. Moreover, the notation $J_{\theta^l} \varphi^l(x^l;\theta^l)$ represents the Jacobian matrix of the original output $\varphi^l(x^l;\theta^l)$ with respect to $\theta^l$, but not that of the perturbed result $x^{l+1}$.

Reviewer comment (Writing):
Conditional expectations such as $\mathbb{E}[\mathcal{L}(x^L)|\xi, x^l]$ are also difficult to understand exactly.

Author response:
**AW1:** Thank you for your feedback. We will modify the formulas and definitions for readability in the revision. We provide a step-by-step expansion of the loss expectation, explicitly annotating the random variables, to further illustrate details in the derivation of ULR as follows: $$ \mathbb E_{z^0,\cdots, z^{L-1}}[\mathcal L (x^L)] 
= \mathbb E_{x^l}[\mathbb E_{z^l,\cdots, z^{L-1}}[\mathcal L (x^L)|x^l]] 
= \mathbb E_{x^l}[\mathbb E_{x^{l+1}}[\mathbb E_{z^{l+1},\cdots, z^{L-1}}[\mathcal L (x^L)|x^{l+1},x^l]|x^l]] 
= \mathbb E_{x^l}[\int_{\mathbb R ^{d_{l+1}}} \mathbb E_{z^{l+1},\cdots, z^{L-1}}[\mathcal L (x^L)|\xi,x^l ]g^l(\xi)d\xi],$$ where the first two equalities are justified by the law of iterated expectation, and the third equality comes from the definition. Moreover, the notation $J_{\theta^l} \varphi^l(x^l;\theta^l)$ represents the Jacobian matrix of the original output $\varphi^l(x^l;\theta^l)$ with respect to $\theta^l$, but not that of the perturbed result $x^{l+1}$.

Reviewer comment (Writing):
What is $J_{\theta^l}$ in Eq.(2), which is not defined in the manuscript? Jacobian of $x^{l+1}$ with respect to $\theta^l$?

Author response:
**AW1:** Thank you for your feedback. We will modify the formulas and definitions for readability in the revision. We provide a step-by-step expansion of the loss expectation, explicitly annotating the random variables, to further illustrate details in the derivation of ULR as follows: $$ \mathbb E_{z^0,\cdots, z^{L-1}}[\mathcal L (x^L)] 
= \mathbb E_{x^l}[\mathbb E_{z^l,\cdots, z^{L-1}}[\mathcal L (x^L)|x^l]] 
= \mathbb E_{x^l}[\mathbb E_{x^{l+1}}[\mathbb E_{z^{l+1},\cdots, z^{L-1}}[\mathcal L (x^L)|x^{l+1},x^l]|x^l]] 
= \mathbb E_{x^l}[\int_{\mathbb R ^{d_{l+1}}} \mathbb E_{z^{l+1},\cdots, z^{L-1}}[\mathcal L (x^L)|\xi,x^l ]g^l(\xi)d\xi],$$ where the first two equalities are justified by the law of iterated expectation, and the third equality comes from the definition. Moreover, the notation $J_{\theta^l} \varphi^l(x^l;\theta^l)$ represents the Jacobian matrix of the original output $\varphi^l(x^l;\theta^l)$ with respect to $\theta^l$, but not that of the perturbed result $x^{l+1}$.

Reviewer comment (Experiments):
In experiments, ULR is only applied to relatively shallow NNs, such as ResNet-5 or VGG-8.

Author response:
**AW2:** Although ULR has better scalability than previous methods and can support training larger models, **the application of pure ULR on large neural networks might not be the most economical,** as stochastic optimization requires multiple evaluations to improve performance. Thus, in our paper, we conduct experiments with pure ULR as well as other baselines on relatively shallow neural networks to verify the effectiveness of ULR training. However, as presented in Section 3.4, **ULR can rearrange the gradient computation graph in deep learning** and work with BP, where ULR is used to estimate the gradient of the last part in each module, and BP passes the information throughout the module. This approach has much lower pressure on memory than pure ULR, as well as improves the parallelism of gradient computation in large and deep models, thereby enhancing efficiency, as verified by two demos in Section 4.2.

Reviewer comment (Evaluation):
As in the title of the paper, it is claimed that ULR requires only one forward computation to estimate the gradient. However, following Algorithm 1 in the paper, it seems to me that ULR requires additional multiple forward computations with respect to $z^l$, whose size is specified in the appendix as ranging from 50 to 800, to compute the loss values. If this is true, I wonder if ULR really reduces the computational effort compared to BP.

Author response:
**AQ1:** Firstly, though only one forward is necessary for ULR, we agree that ULR needs multiple forward evaluations with respect to $z^l$ to enhance the gradient estimation. However, it should be noticed that **forward evaluations with different $z^l$ are independent and thus can be easily parallelized.** We can stack multiple copies of the original data batch into **a large batch for just one forward computation**. The computation with a large batch size can fully utilize the capacity of GPUs. Moreover, we apply multiple tricks to improve estimation efficiency, as shown in Alg. 1, which largely alleviates the issue. There is still significant potential to further reduce consumption, such as by reusing evaluation results through the importance sampling technique.
Secondly, **ULR can work in conjunction with BP/BPTT, balancing training duration and memory occupation to improve training efficiency.** As presented in Section 3.4 and the second demo in Section 4.2, we can use ULR to **cut the original long gradient computation chain into multiple independent short graphs**, which do not require a lot of forward computation but can greatly improve parallelism.
Thirdly, **ULR saves backward computations on those parts that do not need optimization,** thus reducing computational effort compared to BP. When the network embeds pre-trained models or other, even black-box components, ULR can bypass these parts for gradient calculation, as shown in the first demo in Section 4.2.

Reviewer comment (Theory):
In section 3.1, it is suggested to optimize the noise distribution parameter. But it seems a bit strange to me to optimize the noise distribution with respect to the loss value. Instead, I think the noise distribution should be optimized to improve the gradient estimation.

Author response:
**AQ2:** The training process with perturbation-based methods essentially aims at optimizing all parameters in the perturbed model, including those of noise distributions, which is a reasonable proxy of the training without any noise.  Besides, noise injection plays a role in increasing the model capacity and regularization, and appropriate noise parameters can contribute to the performance and robustness.  Thus, we use ULR to calculate the gradient of noise distribution parameters with respect to classification loss and optimize them with gradient descent algorithms.  By viewing the training logs, we surprisingly find that the gradient estimation accuracy is also improved compared to that with the non-optimized noise, which indicates a good byproduct of the improved gradient estimation accuracy by noise optimization in ULR training. We appreciate the idea that noise parameters can be optimized for more accurate gradient estimation and hope this study can motivate the following works to derive new theoretical tools for this purpose.

Reviewer comment (Presentation):
In Figure 4, does ULR-WL for ResNet-5 mean the hybrid model explained in Section 4.3? What about the details of ULR-WL for VGG-8?

Author response:
**AQ3:** Yes, ULR-WL represents the hybrid perturbation manner in which we introduce noise to the weights of the first several layers and to the logits in the subsequent layers for gradient computation. In the case of ULR-WL training on VGG-8, we specifically perturb the weights in the first three layers and the logits in the remaining layers.

Reviewer comment (Writing):
In page 4: $b \in \mathbb{R}^{c_\text{out}}$ -> $b^O \in \mathbb{R}^{c_\text{out}}$

Author response:
**AQ4:** In convolutional neural layers, each element within the same output channel shares a single bias scalar. Therefore, the whole bias term, denoted as $b$, consists of $c_\text{out}$ elements.

Reviewer comment (Theory):
The work has a misleading claim; the authors argue that one forward pass is enough however in practice multiple forward passes are used in order to estimate the gradient (to reduce variance / antithetic variables). I would suggest that the authors rephrase the claim appropriately.

Author response:
**AW1:** Thank you for your suggestion. We wonder whether the reviewer thinks "Only Forward is Necessary for Neural Network Training via the Likelihood Ratio Method" is an appropriate claim/title for this paper. The initial intent of the current title is to highlight the distinction between our ULR algorithm and others like FF and SPSA, which require at least two forward passes, as well as the absence of a need for recursive backward calculations in contrast to forward ones. Although using multiple evaluations is preferable for enhanced training quality, one forward pass is theoretically acceptable. Furthermore, multiple evaluations can be packed into a single batch to be processed in one forward pass.

Reviewer comment (Novelty):
The extension of LR to various network architectures is straightforward and not particularly novel.

Author response:
**AW2:** The purpose of these extensions is to provide readers with specific examples, opening up ideas for the use of ULR. We offer corresponding solutions for issues related to spatio-temporal parameter sharing, unstructuredness, and discontinuity, which cover almost all possible network structures in mainstream deep learning, demonstrating the flexibility of ULR training. This part also inspires our subsequent discussions on computation graph rearrangement and the relationships between different works/paradigms.

Reviewer comment (Experiments):
As far as I understand, the variance reduction techniques discussed at 3.3 seem to not be applied on the baselines, e.g., ES, so it is unclear where to attribute the gap in performance.

Author response:
**AQ3:** We apply our proposed methods for variance reduction, including layer-wise noise injection, antithetic variables, and Quasi-Monte Carlo, to the ES algorithm. We use ES to train ResNet-5 and VGG-8 on the CIFAR-10 dataset. For comparison, we also present the results of ULR in our paper. The results are as follows.
>
>Tab. R3 Classification accuracies (\%) of ResNet-5 and VGG-8 on the CIFAR-10 dataset using the ES and ULR, respectively. 
>
>| Model | ResNet-5 | VGG-8 |
>| -------- | -------- | -------- |
>| ES  without tricks   | 35.1     | 65.8     |
>| ES  with tricks   | 46.9     | 67.2     |
>| ULR     | 64.8     | 77.3     |
>
> As shown in Tab. R3, with the proposed tricks, there is a significant improvement in ES for training neural networks, indicating the effectiveness of our tricks in variance reduction. However, even with the integration of these tricks, ES still struggles to optimize neural networks effectively, demonstrating a lower capacity in gradient estimation compared to ULR. This argument is also supported by our ablation study in Fig. 7(b) of Section 4.3. When perturbing only the weights for gradient estimation, we achieve considerable accuracy in only the first one or two layers but fail to calculate the rest accurately. This further highlights the superiority of ULR over ES in gradient estimation, as shown in Fig. 7\(c\).

Reviewer comment (Experiments):
The architectures and settings are a bit toy; it would be interesting to see how ULR performs on much larger architectures (e.g., ResNet50 and Imagenet or GPT-2 on a reasonably sized text dataset).

Author response:
**AW4:** ULR offers better scalability than previous methods for training larger models, but using it exclusively on large neural networks may not be that cost-effective. Replacing deterministic calculations with stochastic ones may require repeated loss evaluations. Thus, in our paper, we conduct experiments with pure ULR as well as other baselines on relatively shallow neural networks to verify the effectiveness of ULR training. Since ULR is compatible with BP, a viable application prospect in large neural networks is to use ULR to skip the backward computation in non-updating parts of the network or to transform iterative computations into parallel ones, as demonstrated in Section 4.2.

Reviewer comment (Experiments):
Before section 3.2, the authors discuss about adaptive perturbations by optimising the noise, however, as far as I understand, this is not used anywhere in the paper. How is the performance with adaptive noise?

Author response:
**AQ1:** In ULR experiments, we optimize parameters in noise distributions during training for enhanced performance. ULR supports the estimation of gradients for all parameters within the neural network. We calculate the gradients of loss values with respect to the noise generation parameters and apply gradient descent algorithms, such as SGD or Adam, to update these parameters. The estimated gradient formulas for noise distribution parameters are presented in Section 3.1 and Appendices A.1-5. With the adaptive perturbation technique, the performance of neural networks can be further improved. 
> We conduct ablation studies on adaptive perturbation. Following the same setting in our paper, we respectively use ULR with and without adaptive perturbation to train ResNet-5 and VGG-8 on the CIFAR-10 dataset, and RNN, GRU, and LSTM on the Ag-News dataset. The results are reported as follows.
> 
> Tab. R1 Classification accuracies (\%) of ResNet-5 and VGG-8 on the CIFAR-10 dataset. We respectively use ULR with ($\checkmark$) and without ($\times$) adaptive perturbation in neural network training. 
> 
>| Models | Adaptive Perturbation | Accuracy |
>| -------- | -------- | -------- |
>| ResNet-5     | $\times$     |  61.3    |
>| ResNet-5     | $\checkmark$     |  64.8    |
>| VGG-8     | $\times$     |  75.9    |
>|VGG-8     | $\checkmark$     |  77.3    |
>
> Tab. R2 Classification accuracies (\%) of RNN, GRU, LSTM on the Ag-News dataset. We respectively use ULR with ($\checkmark$) and without ($\times$) adaptive perturbation in neural network training. 
> 
>| Models | Adaptive Perturbation | Accuracy |
>| -------- | -------- | -------- |
>| RNN     | $\times$     |  69.2    |
>| RNN     | $\checkmark$     |  70.9    |
>| GRU     | $\times$     |  86.6    |
>| GRU     | $\checkmark$     |  88.9    |
>| LSTM     | $\times$     |  87.8    |
>| LSTM     | $\checkmark$     |  89.4    |
>
>As reported in Tab. R1 and R2, we observe an improvement in neural network training with ULR by adopting adaptive perturbation. This demonstrates the effectiveness of optimizing the noise distribution in boosting the accuracy of gradient estimation for ULR.

Reviewer comment (Experiments):
On the RNN side, it seems that the authors add noise on each step, which can compound over time. Why is this strategy preferred compared to, e.g., just adding noise to the weights and evaluating the entire sequence on these noised weights (e.g., as done for convolutional layers).

Author response:
**AQ2:** Thank you for your insightful suggestion. ULR supports various noise injecting strategies, which hinge on how we define the module $\varphi^l(\cdot;\theta^l)$ in Theorem 1. Injecting noise into the weights can be interpreted as perturbing the logit output of an identity mapping, with the weights as its input. And we agree that there may be better strategies to apply our ULR framework. As mentioned in the CNN part, the variance of the gradient estimation grows as the total dimensionality of injected noises increases. Therefore, the choice between the step-wise and weight perturbation largely depends on $T$ and $d_x$. Furthermore,  if the impact of each step on the final loss is additive, then we can obtain a more refined result for RNNs with step-wise perturbation as shown in Appendix A.7. For scenarios where the effect of a single forward step is transient, truncation tricks can be employed to further reduce the impact of $T$, akin to the introduction of a discount factor in reinforcement learning. Consequently, when $T$ is manageable, and the loss is decomposable, the noise injection strategy outlined in our paper is preferred. In contrast, we may consider the weight perturbation mentioned above.

Reviewer comment (Experiments):
I would suggest the authors to incorporate the variance reduction techniques to ES as well, in order to better highlight what the delta with ULR is.

Author response:
**AQ3:** We apply our proposed methods for variance reduction, including layer-wise noise injection, antithetic variables, and Quasi-Monte Carlo, to the ES algorithm. We use ES to train ResNet-5 and VGG-8 on the CIFAR-10 dataset. For comparison, we also present the results of ULR in our paper. The results are as follows.
>
>Tab. R3 Classification accuracies (\%) of ResNet-5 and VGG-8 on the CIFAR-10 dataset using the ES and ULR, respectively. 
>
>| Model | ResNet-5 | VGG-8 |
>| -------- | -------- | -------- |
>| ES  without tricks   | 35.1     | 65.8     |
>| ES  with tricks   | 46.9     | 67.2     |
>| ULR     | 64.8     | 77.3     |
>
> As shown in Tab. R3, with the proposed tricks, there is a significant improvement in ES for training neural networks, indicating the effectiveness of our tricks in variance reduction. However, even with the integration of these tricks, ES still struggles to optimize neural networks effectively, demonstrating a lower capacity in gradient estimation compared to ULR. This argument is also supported by our ablation study in Fig. 7(b) of Section 4.3. When perturbing only the weights for gradient estimation, we achieve considerable accuracy in only the first one or two layers but fail to calculate the rest accurately. This further highlights the superiority of ULR over ES in gradient estimation, as shown in Fig. 7\(c\).

Reviewer comment (Evaluation):
At Figure 7 what is the reference BP gradient compared against? The true gradient of the noised loss (i.e., the one that includes the noise from ULR) or the gradient of loss without any noise (which is the one we care about)?

Author response:
**AQ5:** In Fig.7, the reference BP gradient is compared with the estimated gradient by ULR. In ULR, the estimated gradient is computed by injecting noise into the loss.  At the same time, the BP gradient is computed on the vanilla network, where no noise is injected into neural activities, thus excluding the effect of perturbation. We use the cosine similarities to evaluate the gradient estimation accuracy.

Reviewer comment (Evaluation):
At the discussion of Figure 5 and Demo 1, the authors argue that the ULR can skip the calculation in late layers, however, this is not entirely true as ULR still needs the forward pass through those layers to compute the loss value.

Author response:
**AQ6:** We agree that the application of ULR cannot avoid the forward pass on non-trainable/frozen layers. The advantage of ULR is to skip the gradient computation of these layers during the backward pass, thus bringing an efficiency improvement as presented in Tab. 1. As pointed out by previous studies [1,2], one backward pass in a neural network theoretically takes about twice as long as the forward pass. Computation hardware, such as GPUs and TPUs, supports the forward pass far better than the backward pass [3]. Thus, by disabling part of the backward pass in gradient computation, ULR can achieve efficiency improvement in practice in our experiments as well as in theoretical analysis. Moreover, ULR can take advantage of the importance sampling technique in the future to update parameters several times with only one forward pass, which further reduces unnecessary calculations.
> 
> [1] Lin, Zhouhan, et al. "Neural networks with few multiplications." arXiv preprint arXiv:1510.03009 (2015).
> [2] Narayanan, Deepak, et al. "PipeDream: Generalized pipeline parallelism for DNN training." Proceedings of the 27th ACM Symposium on Operating Systems Principles. 2019.
> [3] Liu, Xiao-Yang, et al. "High-Performance Tensor Learning Primitives Using GPU Tensor Cores." IEEE Transactions on Computers (2022).

Reviewer comment (Experiments):
Finally, the accuracy of the base BP model on CIFAR 10 is kind of small; do the authors use data augmentations?

Author response:
**AQ7:** We only use one augmentation strategy, namely the random horizontal flip, in our model training on the CIFAR-10 dataset. All the parameters have been fine-tuned for the best performance, and the models are trained sufficiently to converge. However, due to the limited size of the neural networks, i.e., ResNet-5 and VGG-8, they do not achieve high accuracy on the CIFAR-10 dataset, even with BP for gradient computation. Similar results can be found in a previous study [1], which used BP to train a shallow ResNet and achieved around 60% accuracy on the CIFAR-10 dataset. We also compare this work in our appendix of Section B.6. 
>
> [1] Ma, Wan-Duo Kurt, J. P. Lewis, and W. Bastiaan Kleijn. "The HSIC bottleneck: Deep learning without back-propagation." AAAI, 2020.

Reviewer comment (Writing):
Does the star in equation (3) represent a convolution or a term-by-term product?

Author response:
Yes. The star in Eq. (3) represents the convolution of $x^i$ with kernel $\varepsilon^o$. That is the reason why ULR for CNNs can be implemented with existing high-performance convolution algorithms in standard machine learning toolkits, e.g., `torch.nn.functional.conv2d` in PyTorch. More details can be found in lines 58--71 of cnn/module.py, which is provided in the supplementary.

Reviewer comment (Writing):
[Minor comments] "Since these methods change the classical training paradigm or even give up the gradient information. Many algorithmic or hardware technologies developed based on BP by predecessors cannot be fully leveraged, making such approaches computationally unfriendly." this might have been a single sentence.

Author response:
Thanks. We fix this typo in the revision as follows: "Since these methods change the classical training paradigm or even give up the gradient information, many algorithmic or hardware technologies developed based on BP by predecessors cannot be fully leveraged, making such approaches computationally unfriendly."

Reviewer comment (Experiments):
Could the authors provide some hints on the current practical limit of the method with respect to model size and dataset? For example, does this method scale for a resnet18 or resnet50 on CIFAR10 or ImageNet?

Author response:
The current practical limitation of ULR is the large number of data copies required for accurate gradient estimation when training large models on large datasets, which imposes an additional burden on device memory. Training ResNet-50 on ImageNet using pure ULR may be uneconomical due to the expensive memory footprint. 
However, as presented in Sections 3.4 and 4.2, ULR is compatible with BP and can be used to segment the gradient computation graph into multiple subgraphs. In each subgraph, gradient computations from leaf nodes are initiated independently and in parallel using ULR, while BP transmits information throughout the rest. This design doesn’t require as many copies for accurate gradient computation and can make a trade-off between parallelism and repeated forward evaluation, which may enable us to train large models on large datasets with promoted efficiency. Thus, we hope our study can motivate future work on 1) proposing novel techniques to reduce the gradient estimation variance in ULR in a computation- and memory-friendly manner; 2) designing an efficient training framework by rearranging the gradient computational graph with ULR.

Reviewer comment (Writing):
[Minor question] "LR has almost no constraints on the model, including differentiability or traceability, enabling explorations of network architectures." --> Could you explain how it enables explorations of network architectures?

Author response:
ULR offers the flexibility to wrap any structure as a unit for independent gradient computation, which only requires the unit to be differentiable. This feature enables the selection of various structures for the untrainable or non-parametric components within neural networks. The direct impact is that we do not have assumptions of activation functions. In Section 4.1, we effectively train SNNs with ULR, where the non-differentiable spiking activation function hinders the application of vanilla BP. From a broader perspective, ULR enables the integration of black box elements like simulators, pre-trained large models, and commercial optimizers into neural networks as static modules. This significantly facilitates the fusion of domain-specific knowledge with neural network architectures. In Section 4.2, the first demo for domain adaptation provides an example of this purpose.