Paper: A Stable, Fast, and Fully Automatic Learning Algorithm for Predictive Coding Networks (ICLR-2024)

Reviewer comment (Theory):
Regarding the following sentence in the page 3: "For a detailed formulation on how this energy function is derived from the variational free energy of Eq. 2, we refer to ..., or to **the supplementary**", which supplementary section includes this derivation? I do not see it.

Author response:
Dear reviewer,

Thank you for your reply. The two statements you pointed out have been removed in our previous update of the manuscript (we had already spotted them, right after submission), as they were residuals of a previous iteration of the manuscript. 

More in detail:


> Is there any possibility of a typographical error in Table 1?

There are no typographical errors in the tables. The now deleted sentence refers to experiments on Alexnet performed with a slightly different setup: the paper now reports the result by using exactly the original implementation, that reshapes the size of the images to 224x224. In a previous iteration, this was not the case. 

> Have I overlooked the supplementary section that elucidates the derivation of the energy function in Eq. (5) from the variational energy in Eq. (2)?

No, you have not: the (now deleted) reference to the supplementary material refers to a section that we decided to remove before submission: the derivation of the energy function we proposed was similar to that of another work [1]. For a matter of correctness, we had decided to remove it, and simply refer the reader to the original work.

[1] Millidge, Beren, Anil Seth, and Christopher L. Buckley. "Predictive coding: a theoretical and experimental review." arXiv preprint arXiv:2107.12979 (2021).

Reviewer comment (Presentation):
The caption of Table 1 should specify whether these are train or test accuracies.

Author response:
We thank the reviewer for the thorough read, and for all the pointers, spotted typos, and suggestions. They have all been addressed in the updated manuscript. We believe that this definitely improves the clarity of the manuscript. More generally, every time that we discuss accuracies, they are test accuracies (now addressed everywhere in the text). 

About the weight transport, we have added the following, where pointed out by the reviewer:

Weight Transport While predictive coding is designed to model information processing in various brain regions, the current formulation still exhibits certain biological implausibilities. A notable example is the weight transport problem, which posits that the synaptic weight responsible for forwarding information is identical to that which conveys error information backward, as highlighted by Lillicrap et al, 2016. This issue has garnered substantial attention, contributing to the evolution of deep learning models that achieve performance levels on par with backpropagation in large-scale image classification tasks, as discussed in Xiao et al. 2019. Within predictive coding research, it has been shown that the removal of these features does not markedly impair classification performance highlighted in Millidge et al.2020. While it is important to emphasize that our proposed algorithm remains functional in the weight transport framework described in the aforementioned study, testing it goes beyond the scope of the paper. To conclude, other works have introduced predictive coding-like algorithms that do not rely on weight transport (Ororbia & Kiefer, 2022).

Questions:

> why PC with smaller T values converges faster (in terms of Energy in Figure 2). Could you give an insight about it, or elaborate it?

We believe that the reason is that out-of-equilibrium updates are be better (and of larger magnitude) than updates performed to models closer to convergence. This has been shown to be consistent in a wide range of experiments. 

> What is the meaning of the notation on page 17?

Thank you again, this was a typo: the correct notation is O(max(n^l,n^{l+1}). This has been updated in the manuscript.

Reviewer comment (Writing):
The sentence preceding the "Comparison with BP" section in page 6 seems redundant (seems to repeat the previous sentence); consider removing it if not necessary.

Author response:
Dear reviewer,

Thank you for your reply. The two statements you pointed out have been removed in our previous update of the manuscript (we had already spotted them, right after submission), as they were residuals of a previous iteration of the manuscript. 

More in detail:


> Is there any possibility of a typographical error in Table 1?

There are no typographical errors in the tables. The now deleted sentence refers to experiments on Alexnet performed with a slightly different setup: the paper now reports the result by using exactly the original implementation, that reshapes the size of the images to 224x224. In a previous iteration, this was not the case. 

> Have I overlooked the supplementary section that elucidates the derivation of the energy function in Eq. (5) from the variational energy in Eq. (2)?

No, you have not: the (now deleted) reference to the supplementary material refers to a section that we decided to remove before submission: the derivation of the energy function we proposed was similar to that of another work [1]. For a matter of correctness, we had decided to remove it, and simply refer the reader to the original work.

[1] Millidge, Beren, Anil Seth, and Christopher L. Buckley. "Predictive coding: a theoretical and experimental review." arXiv preprint arXiv:2107.12979 (2021).

Reviewer comment (Evaluation):
The statement about iPC performing better on standard CNN than on AlexNet contradicts Table 1. According to Table 1, CNN accuracy is around 72\% whereas AlexNet accuracy is around 80\%. Therefore, iPC does not perform better on standard CNN than on AlexNet.

Author response:
Dear reviewer,

Thank you for your reply. The two statements you pointed out have been removed in our previous update of the manuscript (we had already spotted them, right after submission), as they were residuals of a previous iteration of the manuscript. 

More in detail:


> Is there any possibility of a typographical error in Table 1?

There are no typographical errors in the tables. The now deleted sentence refers to experiments on Alexnet performed with a slightly different setup: the paper now reports the result by using exactly the original implementation, that reshapes the size of the images to 224x224. In a previous iteration, this was not the case. 

> Have I overlooked the supplementary section that elucidates the derivation of the energy function in Eq. (5) from the variational energy in Eq. (2)?

No, you have not: the (now deleted) reference to the supplementary material refers to a section that we decided to remove before submission: the derivation of the energy function we proposed was similar to that of another work [1]. For a matter of correctness, we had decided to remove it, and simply refer the reader to the original work.

[1] Millidge, Beren, Anil Seth, and Christopher L. Buckley. "Predictive coding: a theoretical and experimental review." arXiv preprint arXiv:2107.12979 (2021).

Reviewer comment (Evaluation):
It is not really intuitive to me why PC with smaller T values converges faster (in terms of Energy in Figure 2). Could you give an insight about it, or elaborate it?

Author response:
We thank the reviewer for the thorough read, and for all the pointers, spotted typos, and suggestions. They have all been addressed in the updated manuscript. We believe that this definitely improves the clarity of the manuscript. More generally, every time that we discuss accuracies, they are test accuracies (now addressed everywhere in the text). 

About the weight transport, we have added the following, where pointed out by the reviewer:

Weight Transport While predictive coding is designed to model information processing in various brain regions, the current formulation still exhibits certain biological implausibilities. A notable example is the weight transport problem, which posits that the synaptic weight responsible for forwarding information is identical to that which conveys error information backward, as highlighted by Lillicrap et al, 2016. This issue has garnered substantial attention, contributing to the evolution of deep learning models that achieve performance levels on par with backpropagation in large-scale image classification tasks, as discussed in Xiao et al. 2019. Within predictive coding research, it has been shown that the removal of these features does not markedly impair classification performance highlighted in Millidge et al.2020. While it is important to emphasize that our proposed algorithm remains functional in the weight transport framework described in the aforementioned study, testing it goes beyond the scope of the paper. To conclude, other works have introduced predictive coding-like algorithms that do not rely on weight transport (Ororbia & Kiefer, 2022).

Questions:

> why PC with smaller T values converges faster (in terms of Energy in Figure 2). Could you give an insight about it, or elaborate it?

We believe that the reason is that out-of-equilibrium updates are be better (and of larger magnitude) than updates performed to models closer to convergence. This has been shown to be consistent in a wide range of experiments. 

> What is the meaning of the notation on page 17?

Thank you again, this was a typo: the correct notation is O(max(n^l,n^{l+1}). This has been updated in the manuscript.

Reviewer comment (Writing):
What is the meaning of the notation $\mathcal{O}(\text{max} n^l, n^l)$ on page 17?

Author response:
We thank the reviewer for the thorough read, and for all the pointers, spotted typos, and suggestions. They have all been addressed in the updated manuscript. We believe that this definitely improves the clarity of the manuscript. More generally, every time that we discuss accuracies, they are test accuracies (now addressed everywhere in the text). 

About the weight transport, we have added the following, where pointed out by the reviewer:

Weight Transport While predictive coding is designed to model information processing in various brain regions, the current formulation still exhibits certain biological implausibilities. A notable example is the weight transport problem, which posits that the synaptic weight responsible for forwarding information is identical to that which conveys error information backward, as highlighted by Lillicrap et al, 2016. This issue has garnered substantial attention, contributing to the evolution of deep learning models that achieve performance levels on par with backpropagation in large-scale image classification tasks, as discussed in Xiao et al. 2019. Within predictive coding research, it has been shown that the removal of these features does not markedly impair classification performance highlighted in Millidge et al.2020. While it is important to emphasize that our proposed algorithm remains functional in the weight transport framework described in the aforementioned study, testing it goes beyond the scope of the paper. To conclude, other works have introduced predictive coding-like algorithms that do not rely on weight transport (Ororbia & Kiefer, 2022).

Questions:

> why PC with smaller T values converges faster (in terms of Energy in Figure 2). Could you give an insight about it, or elaborate it?

We believe that the reason is that out-of-equilibrium updates are be better (and of larger magnitude) than updates performed to models closer to convergence. This has been shown to be consistent in a wide range of experiments. 

> What is the meaning of the notation on page 17?

Thank you again, this was a typo: the correct notation is O(max(n^l,n^{l+1}). This has been updated in the manuscript.

Reviewer comment (Novelty):
I came away with the impression that the key algorithmic innovation is simply to alternate weight and hidden layer activations every step (without running either to convergence). It's quite hard to believe this hasn't been tried before. Is there something new here that makes this work?

Author response:
This is completely novel and has never been tried before in the field of predictive coding networks (to our knowledge). 
It is important that you mentioned the alternation of the updates of weights and activations, as well as the fact that the algorithm does not run until convergence, as this is key to make iPC work: running iPC until convergence on every batch of data would result in a strong overfitting of the last batch presented, as well as a forgetting of the previously learned information. It is the combination of both that allows the model to generalise well on unseen data (as well as making it much more efficient). 
This is different from other works, which tend to run every iteration for a lot of time steps, or until convergence (for example, in the paper *Predictive coding approximates backprop along arbitrary computation graphs* the authors use T=200 [1]).

Reviewer comment (Theory):
The analysis is limited to Gaussian networks, which limits the range of applicability of the learning rules in (6) and (7).

Author response:
Note that predictive coding is defined on Gaussian networks only: linear prediction errors arise naturally from this formulation (they are the derivatives (or scores) of the variational free energy of a generative model formed by a hierarchy of Gaussians). Having different probability distributions, such as categorical, would result in update rules that do not have a structure of neuron-synapses-neuron, as well as prediction errors.

That being so, this assumption is sometimes relaxed, to allow the algorithm to work on more complex generative models. An example of that is the experiment on Transformer networks that we presented in the last section, where we have added categorical distributions after the attention mechanism, hence showing that iPC naturally generalises to non-Gaussian distributions as well.

Reviewer comment (Evaluation):
While the experiments are good, there is not much insight provided as to *why* this new approach works better.

Author response:
Incremental updates mitigate the risk of overfitting by continuously adjusting the model parameters, preventing it from becoming too fixated on the specificities of the initial data that it was trained on. Standard predictive coding, on the other hand, tends to overfit on single data points. In terms of convergence guarantees, incremental methods are good in escaping local minima and reach more globally optimal solutions, in which non-incremental ones such as standard PC can get stuck. This is due to the continuous update of the parameters when the neural activities have not yet converged, making the dynamics more “noisy” [3].

Reviewer comment (Theory):
The analysis in the paper is focused on feedforward Gaussian networks, but can these ideas be extended to networks with substantial recurrence? In the original PC formulation, it's these recurrent inputs that carry error signals, but I'm curious as to whether the "out of equilibrium" method proposed here would also be able to dynamically balance multiple kinds of feedback.

Author response:
It can naturally be extended to recurrent cases, such as [4]: note that the final update rule is strictly local. Hence, it is not aware of the structure of the model: they will work without any modifications in models with cycles, loop and recursions.

Reviewer comment (Theory):
Put another way: how much biological plausibility is lost if the Gaussian and feedforward assumptions are relaxed to mirror more cortical-like networks.

Author response:
If by cortical-like networks you mean entangled and cyclic structures that resemble brain regions, our method can be applied with no modification (for examples of predictive coding models that use such structures, see [4]).