Paper: Effective Learning by Node Perturbation in Deep Neural Networks (ICLR-2024)

Reviewer comment (Novelty):
**Awareness of prior work**. The contribution of linking perturbations of forward passes to directional derivatives is not really a new contribution, it has been done before in e.g. [1], which I expected to see cited since it is extremely related. The activity-based updates can be thought of as a finite difference implementation of this approach. More generally, linking forward pass to directional derivatives means dealing with forward-mode differentiation, while BP is reverse-mode. Reverse mode is efficient because of the regime many-parameters/scalar loss, and forward mode is all the more inefficient if done exactly, hence the use of noise to probe multiple directions at once and make it more efficient, at the cost of more variance. It would be nice if the authors place their work clearly in this picture, and clarify their contributions after better literature review.

Author response:
In the introduction, a mention has been added of Baydin et al., 2022; Ren et al., 2022, who do link NP to directional derivatives. We also note that our work is quite different from theirs, because in their work noise is used to estimate gradients, but their networks aren’t set up as noisy systems that propagate noise from previous layers to later layers.

Reviewer comment (Evaluation):
**Overfitting**. The decorrelation approach proposed in this work does help for training accuracy, but leads to massive overfitting compared to BP (Fig 3 and 5). I think this is very concerning since achieving small training error is not the hard part of training neural networks, what really matters is the held-out error. It is known e.g. that BP with SGD has an implicit bias that enables the network to generalize [2], and somehow the decorrelation method suppresses this bias, which is interesting. I think this important point should be addressed/solved by this submission. Maybe the authors should try to vary the amount of decorrelation to see whether it affects overfitting. For now, the claim in the abstract that decorrelation makes NP learning competitive with BP is not backed by the data given the overfitting.

Author response:
It is true that decorrelation increases overfitting, which we acknowledge in the text and in the discussion. We would like to point out, however, that despite the overfitting, test performance for NP-D is still much better than for NP.

Reviewer comment (Evaluation):
How do the angles reported in Fig 2 evolve during learning? Do they become more aligned or less aligned?

Author response:
We did not measure the alignment across training, though we would expect that the measured angle ranking would not change, simply their magnitude would be affected by correlations in the network weights.

Reviewer comment (Experiments):
Why use the squared error loss when doing classification? Why not the cross-entropy loss? I can understand why the authors wanted to only use CIFAR-10/100 because these tasks are more difficult than MNIST or Fashion MNIST, but in this case it could be valuable to add experiments on MNIST to see whether the findings are the same regarding overfitting.

Author response:
MNIST was initially used for some experimentation, but we found that, because of its simplicity, accuracy quickly saturated close to 100%, making it hard to distinguish the algorithms in terms of performance, especially visually. CIFAR-10 seems to be hard enough to make low-performing algorithms fail, but still lightweight enough to allow rapid development and iterations and the use of relatively simple networks.

MSE was used instead of CCE mostly because of its simplicity and the stability of it’s loss magnitude, though none of the theory is different for alternative loss formulations.

Reviewer comment (Novelty):
The paper does not cite the state of the art (SOTA) within the NP subfield (Mengye Ren et al., ICLR 2023).

Author response:
We added a brief mention of Baydin et al., 2022; Ren et al., 2022 to the introduction section, as those authors did indeed relate NP to directional derivatives. However, we also note that their forward gradient method is quite different from ours, as the perturbations in their work are only used to estimate gradients and no noise is actually propagated through the network. This would make their method less suitable for noisy systems, where the noise cannot be avoided. Our method focuses more on applications in noisy systems and even works without any clean forward passes, as shown in our “double noise” experiment. In addition, our ANP method requires no access to the noise vector, but simply measures performance differences and relates them to activity differences, helped by decorrelation of said activities. This method would presumably also work in actual noisy systems, where one does not know which part of an activation is signal and which part is noise.

Reviewer comment (Experiments):
The methods that were used in that NP SOTA have not been incorporated in the experiments here, so it is hard to evaluate whether the advantages seen here could be combined with the previously seen progress in NP.

Author response:
We added a brief mention of Baydin et al., 2022; Ren et al., 2022 to the introduction section, as those authors did indeed relate NP to directional derivatives. However, we also note that their forward gradient method is quite different from ours, as the perturbations in their work are only used to estimate gradients and no noise is actually propagated through the network. This would make their method less suitable for noisy systems, where the noise cannot be avoided. Our method focuses more on applications in noisy systems and even works without any clean forward passes, as shown in our “double noise” experiment. In addition, our ANP method requires no access to the noise vector, but simply measures performance differences and relates them to activity differences, helped by decorrelation of said activities. This method would presumably also work in actual noisy systems, where one does not know which part of an activation is signal and which part is noise.

Reviewer comment (Novelty):
The authors do not cite any of the multiple other bio-plausible alternatives to BP than NP. The SOTA in such alternatives actually outperforms NP and is arguably more plausible and suitable for hardware (Journé et al., ICLR 2023).

Author response:
There are indeed many alternative methods of bio plausible learning, like Feedback Alignment, Target Propagation, energy based methods and many others. The decision to focus on NP stems from our interest in noisy systems and their characteristics.

Reviewer comment (Experiments):
The test accuracies reported here are significantly lower than that SOTA, and tests in more advanced datasets than CIFAR 100 have not been performed.

Author response:
The purpose of our study was not to achieve SOTA results but to do a very straightforward comparison between BP and NP, uncomplicated by performance enhancing engineering solutions like dropout, batch norm, momentum optimizers etc, which might confound the results with their many interactions.

Reviewer comment (Evaluation):
The paper's abstract concludes "making it competitive with BP", but this seems heavily exaggerated based on the actual demonstrations.

Author response:
The claims about being competitive with BP have been toned down a bit in the introduction and discussion sections, as our method does suffer from more overfitting.

Reviewer comment (Theory):
Similarly, how could the other aspects of the method be implemented in the brain?

Author response:
In terms of biological plausibility, the ANP method is the most straightforward to envision working in the brain, as it measures only activation differences and how they relate to a global reward signal, meaning no access to a noise vector is needed. Instead of the double forward pass implemented in our code, one could imagine how random fluctuations in neuronal activity are correlated by the brain to a global reward signal in a dynamical way, without clearly distinguishing between “passes”.

Reviewer comment (Theory):
How plausible is it biologically that for each training example, multiple forward iterations are needed before the full update? It seems that hundreds of such iterations were necessary in the simulation.

Author response:
These are fair criticisms, but they only apply to the NP-iterative method. The NP-iterative method was included as a “best case” example for decorrelated NP, but the NP-activity method is the one that is most biologically plausible and potentially cheap in terms of computation.

Reviewer comment (Experiments):
The paper mentions that the convolutional architecture was very shallow. Why couldn't it be scaled up?

Author response:
As CIFAR-100 isn’t a large dataset, we didn’t feel like a much larger convolutional architecture would be required, especially given our aim of making an uncomplicated comparison between decorrelated NP and BP. Scaling the methods described to datasets like ImageNet is intended as the subject of an extension of this work.

Reviewer comment (Experiments):
I could not find an experiment showing the impact of the decorrelation method on the results. Was it implemented in all versions of NP experiments?

Author response:
Initially, we did not include decorrelated BP, but as of the current revision, we have included it as an additional baseline. We used the iterative decorrelation method from Ahmad et. al., 2022 as it’s a fast and provably reduces a specific loss function measuring the level of correlation in each layer. Alternative decorrelation could work but the specific method for decorrelation was not an interest point for this study.

Reviewer comment (Experiments):
Was it (or any other decorrelation method) used in experiments with backprop? Otherwise, could the authors comment on whether comparisons are fair?

Author response:
Initially, we did not include decorrelated BP, but as of the current revision, we have included it as an additional baseline. We used the iterative decorrelation method from Ahmad et. al., 2022 as it’s a fast and provably reduces a specific loss function measuring the level of correlation in each layer. Alternative decorrelation could work but the specific method for decorrelation was not an interest point for this study.

Reviewer comment (Experiments):
The approach to learning rate optimization might introduce inadvertent biases. By optimizing the learning rate specifically for NP and then using this optimized rate for BP without independent optimization, the results might not reflect the accurate comparison with BP. This is further underscored by references like Hiratani et al., which hint at nuances between NP and BP at different learning rates.

Author response:
The reviewer appears to be mistaken - we optimized learning rates separately for every algorithm and experiment to avoid biasing our results. We explain this more clearly in the methods section and appendix in our updated revision.

Reviewer comment (Presentation):
Interpretation of Figure 2 seemed ambiguous to me. With training curves of three methods appearing quite similar without decorrelations, the clarity of conclusions derived becomes muddied.

Author response:
It is true that the ANP method doesn’t outperform NP nearly as much as does INP. This is now more clearly expressed in the text and conclusions. The reason for emphasizing the activity based method is that it does not require access to the noise generation process, making it more bio-plausible than the iterative method, while also making it potentially more suitable for certain kinds of hardware.  

The main message in Figure 2 is that the standard formulation of NP (noise method) can be aligned more closely with BP by the activity based method and even more so with the iterative method, though for the latter method this comes at the expense of additional computation. Based on our theoretical work, the iterative method is expected to align more closely to BP than the activity method in all layers. We suspect the lack of outperformance in the final layer might be a statistical issue, as that layer contains only 10 neurons, making the measurement process very noisy.

Reviewer comment (Experiments):
The research seems to overlook certain types of networks, notably the absence of comparisons for NP iterative in convolutional networks on established benchmarks like CIFAR100.

Author response:
We have now added the DINP method to the CIFAR-100 convnet experiments. We also included 6 and 9 layer networks on CIFAR-10.

Reviewer comment (Reproducibility):
The paper contains formatting discrepancies, notably in equation numbering. Such inconsistencies can be distracting and hinder smooth comprehension. Additionally, the omission of granular details about the implementation, such as the underlying framework, time metrics, and codebase accessibility.

Author response:
Formatting and Implementation Details More implementation details have now been added. The codebase itself will be released upon acceptance.

- Do the authors plan to release the code base? If so, could you please share the link to an anonymized repository?

Yes, the codebase will be released.