Paper: Forward Learning with Top-Down Feedback: Empirical and Analytical Characterization (ICLR-2024)

Reviewer comment (Novelty):
The experimental improvement and evaluation of PEPITA on their own offer only limited novelty. However, together with the analysis connecting PEPITA, FA, and Forward-Forward, this paper amounts to a considerable contribution.

Author response:
We thank the reviewer for appreciating the value of our theoretical results, beyond simple contributions in performance and accuracy.

Reviewer comment (Experiments):
The experimental evaluation uses only small, fully-connected networks, which are not that suited to more complex vision datasets.

Author response:
Our work extends previous work that impeded training deeper networks with the original PEPITA method, and so improves on the complexity of networks that PEPITA can work with. Overall, many local or biologically plausible training algorithms in present-day literature are not yet ready to be tested on challenging large-scale datasets and highly complex network architectures.

Reviewer comment (Experiments):
The experimental evaluation covers only image classification tasks (MNIST, CIFAR10, CIFAR100).

Author response:
It is unfortunately common in the literature to use image classification tasks as the initial benchmark of new training algorithms. We share this choice with Hinton’s FF paper and others in the same area (e.g. [Ororbia and Mali, 2023]). We do think it would be interesting to extend this work to other types of tasks, particularly to time-dependent datasets, where a forward-only algorithm can be especially useful for analyzing incoming data in real-time. However, this is best treated in a separate paper, where we can focus on the specific needs of such a case.

Reviewer comment (Experiments):
There is no experimental comparison to Forward-Forward.

Author response:
At this time, we are not aware of any implementation of Forward-Forward which utilizes feedback connections to generate the negative samples, as proposed by [Hinton, 2022]. Given the significant difference, we decided not to compare PEPITA to Forward Forward with externally generated samples. Despite this, for completeness, we have added the results that FF achieves on MNIST as reported in [Hinton, 2022].

Reviewer comment (Presentation):
Figures can be a bit hard to read.

Author response:
We thank the reviewer for these suggestions, we are currently working to implement them in the paper.

Reviewer comment (Presentation):
In Figure 2 (especially 2b) the overlap of PEPITA and AFA is hard to see without zooming in.

Author response:
We thank the reviewer for these suggestions, we are currently working to implement them in the paper.

Reviewer comment (Presentation):
Figure 3 would benefit from more contrasting colors.

Author response:
We thank the reviewer for these suggestions, we are currently working to implement them in the paper.

Reviewer comment (Presentation):
The references to Figure 2c in the last paragraph of Section 3.2 seem to be mismatched with the actual order of subfigures in Figure 2.

Author response:
We thank the reviewer for noticing this mismatch. We have now correctly referred to Fig.2d at the end of Sec. 3.2

Reviewer comment (Presentation):
In Table 1, line 1 seems to have the wrong citation for FA.

Author response:
We thank the reviewer for noticing this. The table now reports Lillicrap 2016 instead.

Reviewer comment (Writing):
In contribution (vi) there is a spelling error: "analitically" should be corrected.

Author response:
We thank the reviewer for catching the typo. We have corrected this in the revised version of the manuscript.

Reviewer comment (Writing):
In the last sentence of Section 1 the word "gpus" should probably be in uppercase, i.e., "GPUs".

Author response:
We thank the reviewer for the suggestion. In the revised version you will now find “GPUs”.

Reviewer comment (Presentation):
Which approach do the authors consider their primary contribution? In Table 1 the authors refer to PEPITA-TL as "ours" yet this is only discussed briefly in Section 5.1 and left mostly to future work.

Author response:
We thank the reviewer for the fair question. We consider our primary contribution the theoretical analysis of the nonintuitive learning dynamics of PEPITA, the study of training techniques that lead to improved performance, and the connections with FF and DFA. Overall, we believe that these different characterizations help us build a common framework to study forward-learning rules. The reason why we included PEPITA-TL in Table 1 is to have a complete comparison. Figure S1 shows that training with PEPITA-TL is possible, but we leave the study necessary to make its performance competitive for future work. Moreover, we have added PEPITA-Hebbian and PEPITA+WM in Table 1.

Reviewer comment (Experiments):
Why are the combinations "weight-decay + normalization + WM" and "weight-decay + normalization + no WM" missing in Table 2? Can weight decay and activation normalization not be combined?

Author response:
We thank the reviewer for the question. To find the best hyperparameters for training with activation normalization we run a grid search, which includes searching for the optimal value of weight decay. In all cases with normalization, the grid search returns no weight decay, for this reason, we did not include the combination of normalization and weight decay in the table.

Reviewer comment (Writing):
I think this paper requires a clearer definition of the considered “forward-only” algorithms. In the abstract, it is defined as “avoiding a backward pass”. However, PEPITA, different from local FF, indeed requires a direct global error feedback to inputs. If PEPITA is considered as “forward-only”, is DFA also “forward-only”, as it shares the similar form of direct global error feedback without layer-by-layer BP? In the sense of “forward-only” algorithms, I don’t think PEPITA is parallel to FF with local learning.

Author response:
A key difference between DFA and PEPITA/FF is that DFA requires the computation of gradients of activation explicitly. This means that because the feedback is added after the non-linearity, an additional backward computation is needed. In FF and PEPITA, instead, the update only depends on activations computed during the forward pass. We also note that in [Hinton, 2022], the author mentions the possibility of negative samples constructed using top-down connections, which is the case for PEPITA. Moreover, we note that in forward learning algorithms, the error does not travel by itself throughout the network and that the networks continuously respond to inputs (original or modulated/negative), unlike in DFA where a forward pass is followed with a backward pass containing the error info.

Reviewer comment (Evaluation):
In Table 1, why the considered rule has a much worse performance? It is unclear why we should consider it as it does not introduce additional properties over several algorithms while being worse. Additionally, what’s the detailed definition for “local” and “activity freezing”? For PEPITA, it requires global error feedback to inputs, and storage of activations of the first forward propagation until the second forward propagation ends. Why it is considered “local” and solving “activity freezing”? And why PEPITA outperforms DFA in these aspects?

Author response:
We thank the reviewer for the questions. First, we want to clarify the difference between spatial and temporal locality. Spatial locality refers to rules that use only local information (unit activation). In [Nokland, 2016], the author mentions that the update is “almost local”. This is due to the fact that the feedback is added after the nonlinearity, and the update needs to compute the derivative of the activation, causing the update to be non-local (it would be local with linear activation). Instead, PEPITA does not require any computation that does not depend on local signals (unit activation). Temporal locality refers to rules that do not need to store the activations in memory, and are able to compute updates on only temporally-local information. This is the case for PEPITA-TL, which updates the weights on the fly. We note that PEPITA-TL is the only temporally-local rule in Table 1. Second, in DFA - similarly to BP - activity freezing is a consequence of alternating a forward pass (the network responds to an input, and the nodes are “active”) and a backward pass (the error information travels through the network from the top to the bottom layers, but the network does not process any inputs, thus the nodes are “inactive” or “frozen”). In PEPITA and in FF, the error never travels through the network “by itself” to drive the updates. The network always responds to an input in a forward fashion - the input is modulated through the error in PEPITA, the negative samples in FF. Last, we would like to note that this paper is not proposing the PEPITA algorithm, but rather an analytical characterization of the latter. Locality and activity freezing are explained thoroughly in the work by [Dellaferrera & Kreiman, 2022].

Reviewer comment (Experiments):
In Section 4, the networks are still quite shallow and experimental results are quite poor. DFA can scale to large datasets and networks [1], and WM also scales to large-scale scenarios [2].

Author response:
We thank the reviewer for pointing out the work of [Launay et al 2020] and [Aktrout et al 2019], which we now cite. The reviewer raises valid points. In general, it has been hard to get better results than using back-propagation in the field. We usually refer to approximations to back-propagation. As one example of many, DFA [Nokland 2016] also started with relatively shallow networks and worse performance than back-propagation and it took a few years for the community to extend it to deeper networks [Launay et al 2020, as cited by the reviewer], and even in these deeper networks, back-propagation generally outperforms DFA (though DFA is a very reasonable approximation, [Laynay et al 2020]). Such approximations can still be interesting due to their biological plausibility, theoretical understanding of learning, and the possibility of implementing them in neuromorphic hardware.

Reviewer comment (Theory):
Some derivations are not strict and may be contradictory. For example, in Section 3 and Section 5, it is assumed for several times that $Fe$ is much smaller than $x$ so it can be ignored. In Section 5, however, it is also assumed that the “modulated” samples can be used for negative samples in the FF framework. If the modulation is small enough, why can it formulate a valid negative sample? This largely reduces the reliability of the connection between PEPITA and FF.

Author response:
We respectfully disagree with the reviewer’s point that the magnitude of $Fe$ impacts the connection between PEPITA and FF. First, we point out that the small magnitude of $Fe$ is not an assumption, but it is instead a result that we confirmed empirically. Indeed, the approximation of the PEPITA rule (eq. 6) can only hold if $\|Fe\|\ll\| x\|$. The latter is confirmed by Figure 2a and Figure S3: in this figure, we compare models trained with PEPITA versus identically-initialized models trained with AFA (eq.6). The almost identical training curve confirms that the approximation holds, and consequently that $\|Fe\|\ll\| x\|$. Having proved this, we use the same result (not assumption) for Section 5. The fact that despite the small magnitude the feedback can generate valid negative samples is proven by the empirical results we obtain when training networks (Table 2). To answer “why” this is possible, we present an analysis of the learning mechanism of PEPITA (Section 3), we show that our theory matches the empirical results (Figure 2A-B, Appendix D) and we further elucidate the dynamics of learning, and the alignment mechanism (Appendix E).

Reviewer comment (Novelty):
In Section 3.2, what’s the difference between the analysis and previous analyses for DFA?

Author response:
From a technical point of view, the main difference between the analysis of DFA in [Refinetti et al., 2021a] and our derivation in Sec. 3.2 is that our formalism requires an additional ODE for the dynamics of the adaptive feedback. The emergence of adaptive feedback as a natural order parameter governing the dynamics of learning is a theoretical confirmation of the experimental insight in [Dellaferrera & Kreiman, 2022]. Moreover, our results suggest that a perturbative approach (which is not necessary for DFA) is insightful for the analysis of PEPITA.

Reviewer comment (Theory):
In Section 5.1, what’s the connection of the derivation with contrastive Hebbian learning [1] or equilibrium propagation [2], which also takes the form of Hebbian and Anti-Hebbian phases?

Author response:
We thank the reviewer for the interesting question. In both [Xie and Seung, 2003] (equation 2.6) and [Scellier and Bengiom 2017] (Section 2.4) we find very clear similarities with PEPITA-Hebbian in terms of the update rule. Moreover, both algorithms, as well as PEPITA, work on perturbations that are propagated through the network, which treats clean and perturbed signals differently (Hebbian and anti-Hebbian phases). The main difference stems from how the perturbation is propagated through the network: while PEPITA adds the perturbation to the input using a random feedback matrix (thus avoiding the weight transport problem), in both [Xie and Seung, 2003] and [Scellier and Bengiom 2017] the perturbation is fed to the output layer and propagated through weights symmetrical to the forward weights, which is an issue from the biological perspective.

Reviewer comment (Experiments):
Performance on CIFAR10/100 of algorithms is lacking compared to backprop (although surprisingly much less so for CIFAR100). This performance gap would likely increase for larger networks.

Author response:
Generally, achieving superior outcomes compared to back-propagation in the field has proven challenging. Typically, we resort to utilizing approximations to back-propagation. For instance, DFA [Nokland 2016], among various examples, initially employed relatively shallow networks and exhibited poorer performance than back-propagation. It took a few years for the community to enhance it for deeper networks. We believe that our work provides the insights necessary to improve the performance of these types of algorithms.

Reviewer comment (Experiments):
No experiments with convnets, and I haven't found a justification for that. Since cifar10/100 performance gets much better with introducing convolutions (for backprop), it'd be interesting to look at the performance gap there.

Author response:
In [Dellaferrera & Kreiman, 2022], the authors show that the algorithm can work well on CNNs too. In this work, we focused on other aspects, in particular on the theoretical analysis, which can be performed on fully connected networks. We leave the extension of our results to future work.

Reviewer comment (Experiments):
Would performance of both algorithms match if we reduce the learning rate of PEPITA-TL? Intuitively it should, since the anti-Hebbian error terms would match.

Author response:
We have run experiments with PEPITA using the same hyperparameters as the best ones for PEPITA-TL. With this choice of hyperparameters, standard PEPITA also reaches around 40% accuracy, due to the low learning rate. Though these parameters were chosen using a grid search for PEPITA-TL, we conclude that the hyperparameter choice is not sufficient to explain the performance gap.

Reviewer comment (Theory):
Can we consider PEPITA-Hebbian temporally local since it's a long-term plasticity rule? Since the network is trained with two passes over the same input, the PEPITA-TL approximation adds the first Hebbian term immediately after the first forward pass, so 10s/100s of ms. This would be too short for long-term Hebbian changes (as far as I know), so in your model you can treat both parts of the update rule as if they happen for the same network state -- meaning that PEPITA-Hebbian would be temporally local. (I guess you can account for short-term plasticity as a result of the first forward pass (so non-Hebbian changes proportional to input activity only).)

Author response:
We thank the reviewer for the interesting insight. We provide an answer from two different perspectives. From the hardware implementation perspective, we say that a rule is temporally local if the updates are computed without the need to store any information in memory. This is the case for PEPITA-TL, which computes updates on the fly without any need to keep activations in memory. Instead, PEPITA-Hebbian needs to store the activations of the first forward pass and then use them for the update. From the biological perspective, we believe that the reviewer’s insight is very interesting: on the timescale of neural plasticity, we agree that PEPITA-Hebbian might be temporally local, which would also mean that PEPITA is.

Reviewer comment (Experiments):
Tab1 should have PEPITA-Hebbian performance too.

Author response:
We agree with the reviewer, and we added PEPITA-Hebbian and PEPITA+WM in Table 1.

Reviewer comment (Theory):
The theoretical analysis focuses only on shallow 2-layer networks, and I’m uncertain that the Adaptive Feedback rule can be easily extended to analyze deeper networks, as the error signal has to pass through multiple hidden layers either forwards (forward-only) or backwards (feedback alignment). This limits the applicability of the analysis, making it disjoint from the experimental results on multi-layer PEPITA networks.

Author response:
The reviewer is right in observing that our theoretical analysis is limited to one-hidden layer networks. Our goal is to elucidate the alignment phenomenon initially observed in the paper introducing the PEPITA algorithm [Dellaferrera & Kreiman, 2022], which considers one-hidden layer networks. Further work would be required to study the multi-layer case, but our approximation suggests that a perturbative approach could be useful in this direction. We stress that the analytical frameworks to study the dynamics of learning in deep networks are still very limited, and – to the best of our knowledge – tracking analytically the average performance of deep nets with non-linear activations is still an open problem even for networks trained with back-propagation. One possible preliminary step to elucidate the alignment phenomenon in the multi-layer case would be to consider deep linear networks. We added a comment in Sec. 7.

Reviewer comment (Experiments):
I’m a bit confused on how much the PEPITA-Hebbian approximation impacts the accuracy of the model. Figure 3a shows that the impact is very small on CIFAR-10, maintaining over 50% accuracy on CIFAR-10 for 1, 2, and 3 hidden layers, but in Figure S1 the accuracy of a 1-layer PEPITA-TL model drops to around 40%. Could you explain why this is? If this is due to hyperparameter differences, how well would a non-TL PEPITA model compare in performance under the same hypersparameters/conditions as those used for Figure S1?

Author response:
In the work by [Dellaferrera and Kreiman, 2022], the authors show that the presynaptic term ($h_{\ell - 1}^{err}$) can be replaced with the same term computer in the first forward pass ($h_{\ell - 1}$) without changing the performance. For this reason, we expected the Hebbian learning rule in eq. 8 to perform equivalently to the original learning rule, which is empirically the case. In the time local version, the updates are computed separately, meaning that the update computed during the second forward pass is operating on the weights that have already been updated in the first forward pass. We suspect that separating the Hebbian and anti-Hebbian updates changes the loss landscape, although we leave this analysis for further work. With the same choice of hyperparameters, standard PEPITA also reaches around 40% accuracy, due to the low learning rate. Though these parameters were chosen using a grid search for PEPITA-TL, we conclude that the hyperparameter choice is not sufficient to explain the performance gap.