Paper: The Trifecta: Three simple techniques for training deeper Forward-Forward networks (ICLR-2024)

Reviewer comment (Novelty):
NOVELTY: the "Trifecta" (which is a bit excessive as name) is composed of three small modifications, of which one was already published and applied to FF (different loss term), one was published but not applied to FF (Overlapping Local Updates), the final one is using batch normalization instead of layer normalization. These 3 methods are also justified in an informal way or empirically by some visualization (which are mostly copy-paste from a W&B run). Overall, this looks more like a report of someone experimenting on FF than a true scientific publication. This is made worse by the fact that (a) FF is unpublished, and (b) there is no convergence guarantee to begin with on FF optimization.

Author response:
> These 3 methods are also justified in an informal way or empirically by some visualization (which are mostly copy-paste from a W&B run)

We made certain that we referenced the corresponding appendices (A through E) for a more comprehensive evaluation that proposes numerous experiments and plots to back up our statements. As noted in the work, all conclusions drawn from these plots are verified across a multitude of settings that are not all shown. If there are any missing justifications in your opinion, we would be very glad to hear them.

> Overall, this looks more like a report of someone experimenting on FF than a true scientific publication. This is made worse by the fact that (a) FF is unpublished, and (b) there is no convergence guarantee to begin with on FF optimization. 

It is true that the FF paper is unpublished. However, given its author and the recent popularity and scrutiny it has received from the wider ML community, we would argue that it has been verified to a further extent than many publications (for instance, the work was presented at NeurIPS https://neurips.cc/virtual/2022/invited-talk/55869). Further, FF indeed doesn't have a convergence guarantee, its main focus is to be a preliminary investigation into a novel learning algorithm based on the popular contrastive learning method, that has been shown to work in numerous publications. 

> but the paper remains of an incremental value

The Trifecta is composed of three existing techniques, this is correct. However, we argue that our contributions reach further than simply proposing The Trifecta. Primarily, we uncover and study three weaknesses within the vanilla FF algorithm and elucidate how they can be solved by our proposed techniques to achieve very competitive accuracy on several datasets. Additionally, we outline the general characteristics of our solutions that cause the improvement such that future work can build upon our findings. In conclusion, there are several findings that are of significant importance on the road to further our understanding of FF and other local learning algorithms.

Reviewer comment (Novelty):
RELATED WORKS: the paper is very shallow in its analysis of the related works, which should include other backpropagation-free techniques such as zeroth-order gradients (https://arxiv.org/abs/2305.17333), forward gradients (https://openreview.net/forum?id=JxpBP1JM15-), error-driven input modulation (https://proceedings.mlr.press/v162/dellaferrera22a/dellaferrera22a.pdf), etc. Many of these methods are similar to FF.

Author response:
> The paper is very shallow in its analysis of the related works, which should include other backpropagation-free techniques such as zeroth-order gradients

This is a very valid point. Backpropagation-free techniques and local learning have received increased attention in the past years resulting in lots of papers. To our knowledge, all existing items in the literature differ from FF by at least one key aspect (and most times by more). For instance, the reference [https://arxiv.org/abs/2305.17333](https://arxiv.org/abs/2305.17333) is quite similar in approach but is presented as a finetuning technique of LMMs, not a full learning algorithm. Further, [https://openreview.net/forum?id=JxpBP1JM15-](https://openreview.net/forum?id=JxpBP1JM15-) differs by its use of a block-based approach. There are several block-based learning algorithms without any specific approach standing out in similarity to FF or in popularity which led us to the choice to not discuss them. We have included (https://proceedings.mlr.press/v162/dellaferrera22a/dellaferrera22a.pdf) and some additional work in our related work discussion.

Reviewer comment (Experiments):
EXPERIMENTS: connected to 2, it's unclear why the authors are only validating with respect to standard FF and not other alternatives to backpropagation (e.g., direct feedback alignment). Many of these methods are on-par with the results shown here.

Author response:
In our previous revision, we have added comparisons to FA and PEPITA. We also experimented with DFA, however, we found that for the architectures we considered in our experiments, the algorithm was quite unstable and failed to converge. To verify, we cross-referenced these findings with an implementation from https://github.com/lightonai/dfa-scales-to-modern-deep-learning and observed the same results. Therefore we opted to not include them.

Reviewer comment (Novelty):
Work feels largely incremental—authors adopt existing technologies (SymBa loss, Batchnorm, Overlapping Local Updates) to design TFF algorithm.

Author response:
Given the nascent state of the FF algorithm, our goal is to further explore its learning properties and uncover simple changes that improve the scalability and overall accuracy in a wide range of settings. To this end, we uncovered and focussed on three weaknesses of the original algorithm. Following this, we propose three modifications with techniques that are purposefully easily implementable due to their familiarity or simplicity. We argue that the main contributions of our work lie in finding these weaknesses, proposing general solutions to them and subsequently improving the scaling behaviour of the FF algorithm.

Reviewer comment (Novelty):
While the performance of TFF on CIFAR-10 is very good for backpropagation-free training, I found the design of the algorithm to borrow from existing technologies. I wanted to give you the chance to clarify if there were additional contributions in the design of TFF beyond adopting existing methods from the literature.

Author response:
Indeed, the main components of TFF borrow from existing techniques, as motivated above. However, this is not the main contribution of this work. Nevertheless, there are some minor modifications that are not specifically highlighted in the paper. The architecture, or specifically, the layer composition is slightly different from ordinary CNNs. Namely, we use the ordering BN, Conv, ReLU and then Maxpool. We found that performing a ReLU before the maxpool is very beneficial to the overall scaling behaviour of the network. Further, we propose a method to embed the label suitably in a CNN. These modifications are stated in the paper but are also the main reason we openly share our code, such that future experiments of FF can start on a solid foundation.

Reviewer comment (Evaluation):
On p. 8, you say that “Further, on simple datasets, our algorithm is on par with our backpropagation baseline, in the same amount of epochs.” Considering Table 2, this only appears to be the case for MNIST. Am I missing something?

Author response:
This is indeed only the case for MNIST, this is updated to be more clear.

Reviewer comment (Evaluation):
What is the walltime per epoch when using TFF compared to BP? I know this will vary per architecture and dataset so let’s say the largest model on CIFAR-10.

Author response:
We did not include any details on this aspect as this is in line with expectation: FF is twice as slow per epoch due to the inclusion of negative samples. These are the averaged wall-clock times for CIFAR-10 (first) and MNIST (second) when trained for 100 epochs (on an rtx4080). As the models are identical (except the first layer) throughout datasets, this stays largely consistent. Please note that these experiments were not performed in a controlled environment and therefore, suffer from substantial variance.

- BP small: 5m 53s <-> FF small : 10m 36s
- BP large: 7m 54s <-> FF large: 16m 38s
- BP small: 4m 41s <-> FF small: 9m 39s
- BP large: 7m 22s <-> FF large: 15m 03s

Reviewer comment (Theory):
Weight updates as described in section 3, are supposed to be done in 2 forward steps, one for positive and one for negative samples (maybe aiming to be able to completely decouple them further in time eventually (sleep section in Hinton 2022)). However here by removing the threshold it appears one needs the batch to contain both positive and negative examples and resembles a self-supervised contrastive setup.

Author response:
Given the fact that the sleep idea currently is not possible, as stated by Hinton 2022 in the revised version, using a concatenated batch (pos+neg) or two separate batches yields identical results, but the former is slightly faster on most hardware. The FF algorithm differentiates itself from most forms of contrastive learning by its use of goodness. Instead of performing similarity comparisons on entire embeddings, they are done on this goodness value. By changing the loss function to SymBa, these goodness values are indeed compared to each other to calculate the loss, instead of to a set threshold. These would need to be stored if used in a fully decoupled manner.

Reviewer comment (Theory):
Overlapping local updates although not using non-global gradients is a step away from being able to introduce black boxes in the middle of the forward pass.

Author response:
There are several aspects to consider when using semi-local gradients (outlined in Appendix F). 
 - Biological plausibility: Within our setup, all error communication is very local, further, by using slight alterations like random synaptic feedback, biological plausibility can be increased (if desired).
 - Black boxes: OLU doesn't hinder black boxes, it simply improves parts of the network that are uninterrupted. The strength of FF lies in its flexible objective that does not necessitate specialised classification layers that may disturb these black boxes.
 - Non-differentiable operations: There are instances where each layer has a non-differentiable operation and therefore OLU cannot be used. However, these scenarios are quite niche but we would like to highlight that training on MNIST without OLU actually yields slightly better results (~99.7% instead of ~99.6%). On more complex datasets such as Cifar-10, OLU works better as seen in the ablation.

Reviewer comment (Experiments):
I also can't stop to notice that batch normalization as done here also disallows training by seeing one example at a time (though maybe keeping batch statistics would easily address that?).

Author response:
This is exactly true. As our experiments show, BN works well in this scenario as it finds a middle ground between fully renormalising features and no normalisation. We argue any sensible normalisation function with this property would thus perform comparably. We performed the proposed experiment, using only running averages of BN during training, with otherwise identical settings as TFF in Table 1 and achieved 73.3%, which is slightly lower but still very competitive. As this is quite specific these updates will be pushed to the appendix (E), instead of the main text.

Reviewer comment (Experiments):
First of all, while the image classification results are promising when Trifecta is applied to the Forward-Forward algorithm, that does not mean that the only task we need to tackle is the classification task. It is desirable to see how well the Trifecta fares with the vanilla FF and SGD on a wider variety of tasks such as Machine Translation, image generation, or different types of learning mechanisms such as self-supervised learning, reinforcement learning, multimodal learning, etc.

Author response:
This is absolutely correct and is an interesting avenue for future research. However, given the nascent state of FF, we opted to limit the scope of this work to image classification. This has two reasons: the original paper exclusively has results in this modality, to which we wanted to make a direct comparison, and image classification tasks have the most well-known and standardized datasets.

Reviewer comment (Experiments):
In addition, the network used for evaluating this method is relatively small compared to most experiments that are performed in this avenue. It would bring substantial improvement to the paper to use larger networks such as ResNet and attention-based models to evaluate the performance of Trifecta on a given task compared to the vanilla FF and SGD.

Author response:
Again, we certainly agree that a more thorough architecture search is very desirable. In the context of local learning, however, many ubiquitous architectures from BP simply do not work well. For example, a bottleneck architecture will yield terrible accuracy as the error signal is "not strong enough" to learn compressed representations. That said, we have tried adding residual connections to our networks. From our limited experiments, we found that the accuracy increases (slightly less than 1% on the CIFAR-10 model). However, as this is only a very slight improvement, we decided not to include this finding and put more focus on our three main modifications.

Reviewer comment (Experiments):
Finally, even though it is not desired to perform evaluation on large datasets such as ImageNet, I am curious regarding the performance of TFF when it is compared to VFF and BP on image classification task, since most of the methods proposed in this avenue are evaluated on large datasets.

Author response:
In light of this comment, we performed experiments on Cifar-100. The small model is able to achieve 35.8% after 200 epochs and the large model achieves 35.4% after 500 epochs using the same setup as all other datasets. Originally, we decided not to evaluate larger datasets as the main goal was to scale from MNIST to CIFAR-10 and compare the results to the original paper. Most local learning approaches currently are not yet competitive with BP for more complex datasets. We have added these new results to Table 2.