Paper: Synaptic Weight Distributions Depend on the Geometry of Plasticity (ICLR-2024)

Reviewer comment (Theory):
The article assumes that our brain is a feedforward network with no feedback and lacks any dynamic processes.

Author response:
We don’t assume that the network is feedforward, in fact, in the submission **we present results with recurrent neural networks, namely CORnet (a recurrent architecture)**. We have also added RNN experiments to the revised paper.

Reviewer comment (Theory):
The article assumes that during learning, only the neurons in the last layer of our brain update.

Author response:
**We never state this or make any assumptions this way**. This comment is therefore very wide of the mark and indicates the reviewer has not engaged with the paper. Please see section 2.1 Deep networks onwards.

Reviewer comment (Theory):
The article assumes that our brain should exclusively perform a supervised learning task.

Author response:
**We don’t assume this**. We only assume that the brain is engaged in a task that can be framed as regression, but we would point out that **self-supervised learning and reinforcement learning also often rely on various forms of regression**, thus there is nothing about the assumption of regression that forces us into supervised learning. However, we agree that verifying our theoretical statement that our analysis is loss agnostic is valuable, and **we have included experiments with a self-supervised loss here to demonstrate this point**.

Reviewer comment (Theory):
The article assumes that the optimal values for our brain's neurons for a given problem are unique and deterministic.

Author response:
This is possibly the strangest of the reviewer’s concerns, as we never state this anywhere and actually explicitly assume the opposite – that **the network should be overparameterized**, which would lead to there being many different possible solutions that would be arrived at depending on various stochastic factors, such as initialization and data sampling. Thus, again, the reviewer appears to have not really engaged with the paper, and possibly, not even read it beyond the abstract.

Reviewer comment (Theory):
Each of these five assumptions is overly restrictive, and some are evidently incorrect, conflicting with known experimental evidence.

Author response:
First, the reviewer raises concerns about **five** assumptions we *supposedly* make in this paper, but **only one of them** is made, and then only partially (see item 4, below). It leads us to wonder if it is just a simple misunderstanding, (in which case that may be a result of poor communication on our part), or whether Reviewer mbP3 was simply not engaged enough with the paper to notice that we do not make the assumptions they claim we do. We agree that each of these five assumptions listed here is overly restrictive, and some are evidently incorrect, conflicting with known experimental evidence. But, as noted, we **do not** make these assumptions.

Reviewer comment (Theory):
The article's conclusion appears highly unreasonable, specifically in stating that the distribution of synaptic strength in our brain neurons, as well as the brain's structure and the environmental context (task execution), appear to be unrelated, and that the distribution of synaptic strength is solely related to geometry.

Author response:
Regarding the claim that our conclusion is highly unreasonable, **we’d like to point out that the data collected on synaptic weight distributions in the brain seems to support our conclusions. Notably, the distribution of synaptic spine sizes across brain regions and species [Buzsáki and Mizuseki, 2014] is highly robustly log-normal**. That suggests that regardless of the architecture or function of a neural circuit, the synaptic weight distribution is log-normal, which could easily be explained if the synaptic geometry is the major determining factor, and this was preserved across species and brain regions. Thus, far from being unreasonable, our conclusion is both supported by our theory and experiments, and existing data in the neuroscience literature. We will be sure to make this point in any revisions.

Reviewer comment (Evaluation):
The article's conclusion lacks significance, as it does not address why we would choose this specific geometry, what advantages it offers, and how we arrive at such a choice, even assuming the conclusion is correct.

Author response:
We strongly disagree, and must highlight again that the reviewer is making oddly subjective comments with no argument to back up their claims. Why is this conclusion insignificant? If we are correct, and changes in the weight distribution will be dominated by the synaptic geometry, then that would give us an experimental means of assessing synaptic geometry, which would be a major insight into how the brain works! We also think the reviewer's surprise regarding our conclusions are indicative of actual significance. If science were never surprising, it would be unnecessary. We do agree that future work is required for this theoretical contribution to fully leave its mark on AI and computational neuroscience, but we think this an exciting and novel first step.

Reviewer comment (Experiments):
The main weakness of the paper is that even though the theory is agnostic to the loss function and dataset, it seems that it is not agnostic to the architecture given section 4.3. So it would maybe make sense to also try more plausible architectures than ANNs to test the theory like continuous Hopfield networks or Spiking networks trained with surrogate gradients for instance, or at least vanilla RNNs, since the brain is highly recurrent. This would provide a sense of how important is the architecture vs the learning rule.

Author response:
We agree that architecture choices play a role in our experiments. However, for all architectures, we’ve observed a strong contrast between multiplicative (NE) and additive (2-norm) plasticity rules. The last two architectures, CORnet-S and CORnet-RT, had several recurrent blocks; we included them to test if recurrence can break our conclusions. In the updated pdf (see Appendix B.3), **we included new RNN experiments** on row-wise sequential MNIST; the new results are very consistent with our prediction. (We agree it would be valuable to try e.g. spiking networks, but we would leave it to future work.)

Reviewer comment (Theory):
Can you elaborate on the link between geometry of plasticity and the locality of the learning rule? I would expect non Euclidean geometry to be non local since the metric tensor needs to be inverted, however Eq 7 does look local.

Author response:
If $\nabla\phi(\mathbf{w})_i=f(w_i)$ (so, a function of a single weight and not others), the update becomes local since it doesn’t depend on other weights. This is true for the potentials we discuss. However, for Euclidean geometry, we additionally need $f(w_i) = w_i$. For negative entropy, we get $f(w_i) =1 + \log w_i$, so the primal-dual weight mapping stays local but is no longer Euclidean. We mention this (albeit more briefly) just before Eq. 14, but have now expanded that explanation.

Reviewer comment (Theory):
Isn't the use of optimizers such as Adam a way to have a different geometry of plasticity in ANNs?

Author response:
It would depend on the optimizer. A simple momentum (with scalar, but possible adaptive, parameters), wouldn’t change mirror descent solutions (Gunasekar et al. (2018)). Adam or similar optimizers might change the space of solutions, but they usually try to make gradient updates unit-variance, which intuitively should strengthen Gaussian convergence. See the added discussion after Th. 1 in Sec. 3.

Reviewer comment (Presentation):
Fig 3B: caption is not coherent: top/bottom should be left/right.

Author response:
Thank you, we have corrected them.

Reviewer comment (Presentation):
Fig 4: I see the histograms in pink and not blue as written in the caption.

Author response:
Thank you, we have corrected them.

Reviewer comment (Theory):
The distinction between "loss function" and "potential": The paper claims in section 2 that the mirror descent framework splits a synaptic weight update gradient into two terms: one dependent on "external errors" i.e. the loss function \( l \), and one "intrinsic to the synapse" i.e. the potential \( \phi \). However, I am skeptical that this is generally true: to the best of my understanding, we can say that gradients with respect to the potential \( \phi \), \( \Delta \phi \) are independent of gradients wrt the loss \( \Delta l \). However, given that both the loss function and potential are functions of the synaptic weights, and that the loss function is either chosen by the practitioner or unknown (in the case of the brain), unless we _explicitly_ choose \( l \) to represent _only_ errors extrinsic to the synapse, it is not clear to me how we can guarantee the above statement. Relatedly, it is also not clear to me whether we can always pick a loss function that only captures non-synapse-related changes in weights, and if we can guarantee that we have managed this in every use case.

Author response:
Thank you for your review, the points you have raised have been useful for us. We address weakness 1 and question 1 together below. Regarding weakness 2, please see our response to question 3, and we also understand it to overlap with the applicability of gradient-based learning to neuroscience (shared with reviewer jXnh). Here’s a slightly shortened response:

First, our work uses gradient descent to construct a rigorous theory. However, **our result ultimately relies on the large-sum behavior of update terms**. If we replace the gradient $\nabla l$ wrt the loss in Eq. 5 $\nabla\phi(w^{t+1})=\nabla\phi(w^{t})-\eta\nabla l$, with anything else that drives learning, like a 3-factor Hebbian rule, the intuition would still hold: a large sum of small and mostly independent updates looks Gaussian in the dual space.

More generally, we assume that an ANN trained by gradient-based methods can be used as a working model for the brain, and as a working abstraction can be used to understand aspects of how the brain learns and works. As long as the learning algorithm the brain uses descends a gradient, even if it is never explicitly calculated, then this framework can be applied. However, this does not mean that we are committed to the brain doing gradient descent, per se. But it is a commonly applied framework that allows researchers to make tractable theoretical advances that can generate biological hypotheses - as we do in this manuscript.
> How can we guarantee that the loss function / potential function separation truly represents a separation between extrinsic and intrinsic factors in the weight changes?

The motivation of the paragraph just before Sec. 2.1 was to convey a way to think about credit assignment in terms of the mirror descent framework we present in Sec. 2. **We are not committed to this separation of factors per se in biology. However, we think it can be helpful to think in terms of this separation.** Extrinsic credit assignment factors are those external to the synapse, such as a neuron-wide error signal multiplied by presynaptic activity (which corresponds to the gradient). These external factors are then used to update the synapse (somehow), which is the intrinsic step. These factors are not necessarily separated in real neurons, however, within the mirror descent framework they are, and the intrinsic step is captured by the potential. We hope this clarifies this point and are happy to discuss it further. We have updated our communication of this point in the manuscript.

Reviewer comment (Theory):
The paper claims in section 1 to "make experimental predictions" and provide "new theoretical insights...about learning algorithms in the brain". In section 4.4, while it is clear from the results and the analysis that a non-Gaussian distribution of weights under the linearity assumption might indicate a non-Euclidean synaptic geometry, it is not clear how we can interpret this for insights about how learning happens in the brain, and for potential plasticity mechanisms. It is also not clear how extensible the analogy is from deep recurrent networks to the brain — synaptic weight changes may not be analogous to network weight updates via gradient descent, and therefore any theoretical results based on gradient descent may not port easily to neuroscientific insights.

Author response:
Again, we sincerely thank the reviewer for their helpful feedback. This discussion has been very useful for improving the paper. We also thank the reviewer for their thoughtful consideration in adjusting their score, despite the follow up questions. We hope we can clarify these now:

> Could you elaborate on how you think this separation could be useful, if we are unable to extend this assumption to neurons in every case?
 > ... (a) we have this separation between loss and potential updates, and (b) it allows us (under linearity assumptions in the dual space) to assume a Gaussian distribution for the final synaptic weights. From the experiments in 4.4, we can at least say something about synaptic weights in real neuronal networks, given (b). It is still not clear to me how the same is true of (a).

Thank you for continuing to push us on this! Yes, you are correct that (at least currently) we cannot say anything that links the separation of the loss gradient and potential in mirror descent to a biological separation in real neurons.

Addressing (a), if the underlying biological mechanism violates the mirror descent potential/error signal separation, we can’t apply this framework. This is definitely possible – however, most of the existing theories for error-driven learning (gradient descent, approximations to it, 3-factor Hebbian rules, etc.) do factorize! 

From a neuroscience perspective, in this work we provide evidence against the brain updating synaptic connection strengths via gradient descent because we prove that gradient descent is not consistent with the log-normal weights seen in the brain. While interesting and important (we think), there are also many other reasons why the brain cannot be implementing gradient descent - at least similar to how ANNs do it. Indeed how the brain might approximate gradient descent is active area of research, e.g.  Lillicrap et al. (2016); Liao et al. (2016); Akrout et al. (2019); Podlaski & Machens (2020); Clark et al. (2021). As such, there is ongoing work and many papers exploring how the brain might be calculating error terms and approximating gradients. 

Because of the factorisation, our work does not run counter to this pre-existing body of research. Instead, we are able to leverage these studies for the “extrinsic” error term which is independent of the “intrinsic” potential (resulting in distance) term that we focus on. From our perspective this is very important, because the factorisation can be used to make all of the previous literature on credit assignment (even) more biologically plausible. 

We hope this is clear? In summary we are enthusiastic about the factorisation from a theoretical perspective because it explicitly allows us to not re-evaluate all of the previous work studying how the brain might estimate the extrinsic error term!  
Again thank you for pushing us to be clearer about this point. We have again updated the paragraph in section 2, and will revisit this for a camera ready version.  

> Therefore, while our work leverages gradient-based optimization, the intuition we present applies more broadly." -- it is not clear what this means. Which intuitions apply more broadly? -- that as long as weight updates are approximately independent, we can still assume a final Gaussian distribution? What happens when this assumption breaks too?

Yes, without gradient descent and the application of Theorem 1, as long as weight updates are approximately independent, we can still assume a final Gaussian distribution. This comes from a large sum behavior: $\nabla\phi(w^\infty)=\nabla\phi(w^0) + \sum_t g^t$, so we can expect $\sum_t g^t$ to follow the central limit theorem (CLT; by approximate independence assumption). If we measure the change with a different potential, that sum will be non-linearly transformed, and so CLT would not apply. 
When this independence assumption breaks, then unfortunately we won’t be able to say much. But, this assumption is likely to hold in neural systems since noise is very prevalent in biology . 

Extending the discussion past an answer to your question:
Perhaps it would help to clarify how we use Theorem 1. If we denote $\nabla\phi=f_1$ and another potential $f_2$, we’re saying that $f_1(w^\infty)=f_1(w^0) + \mathcal{N}$. Then if we measure the change with $f_2$, we get $f_2(w^\infty)=f_2(f_1^{-1}(f_1(w^0) + \mathcal{N}))$. If $f_1$ and $f_2$ are similar, then we can linearize $f_2(w^\infty)$ and also see something Gaussian. If not, the change will be non-Gaussian. Therefore we are able to use Gaussianity in the dual space to determine the correct potential. **We have updated the manuscript after the theorem to communicate this point (Section 3).**

Reviewer comment (Theory):
How can we guarantee that the loss function / potential function separation truly represents a separation between extrinsic and intrinsic factors in the weight changes?

Author response:
Thank you for your review, the points you have raised have been useful for us. We address weakness 1 and question 1 together below. Regarding weakness 2, please see our response to question 3, and we also understand it to overlap with the applicability of gradient-based learning to neuroscience (shared with reviewer jXnh). Here’s a slightly shortened response:

First, our work uses gradient descent to construct a rigorous theory. However, **our result ultimately relies on the large-sum behavior of update terms**. If we replace the gradient $\nabla l$ wrt the loss in Eq. 5 $\nabla\phi(w^{t+1})=\nabla\phi(w^{t})-\eta\nabla l$, with anything else that drives learning, like a 3-factor Hebbian rule, the intuition would still hold: a large sum of small and mostly independent updates looks Gaussian in the dual space.

More generally, we assume that an ANN trained by gradient-based methods can be used as a working model for the brain, and as a working abstraction can be used to understand aspects of how the brain learns and works. As long as the learning algorithm the brain uses descends a gradient, even if it is never explicitly calculated, then this framework can be applied. However, this does not mean that we are committed to the brain doing gradient descent, per se. But it is a commonly applied framework that allows researchers to make tractable theoretical advances that can generate biological hypotheses - as we do in this manuscript.
> How can we guarantee that the loss function / potential function separation truly represents a separation between extrinsic and intrinsic factors in the weight changes?

The motivation of the paragraph just before Sec. 2.1 was to convey a way to think about credit assignment in terms of the mirror descent framework we present in Sec. 2. **We are not committed to this separation of factors per se in biology. However, we think it can be helpful to think in terms of this separation.** Extrinsic credit assignment factors are those external to the synapse, such as a neuron-wide error signal multiplied by presynaptic activity (which corresponds to the gradient). These external factors are then used to update the synapse (somehow), which is the intrinsic step. These factors are not necessarily separated in real neurons, however, within the mirror descent framework they are, and the intrinsic step is captured by the potential. We hope this clarifies this point and are happy to discuss it further. We have updated our communication of this point in the manuscript.

Reviewer comment (Evaluation):
How to evaluate whether the linearity / Gaussian assumption has been violated, in cases where the loss and potential function are unknown?

Author response:
> How to evaluate whether the linearity / Gaussian assumption has been violated, in cases where the loss and potential function are unknown?

Sec. 3.1 indirectly addresses this point, in short, whether the loss and/or potential function are unknown does not impact the assessment of linearity /Gaussian assumption violation: 
1. Linearity: the norm of the weight changes should be ~ an order of magnitude smaller than the weights in the dual space. This comes from applying a 1st order Taylor series approximation, and can be verified for a candidate potential.
2. Gaussian assumption: this could be broken by correlations in inputs. However, we think this is unlikely, because this is a common feature in natural sensory data [Ruderman (1994)] and neural activity [Schulz et al. (2015)] (we mention this just before Sec. 3.1). We also empirically investigate the robustness of this violation in the deep network experiments (Figs ).
We will strengthen our communication of these points in the main text.

Reviewer comment (Theory):
The big assumption that the brain is doing gradient descent.

Author response:
Thank you for the review!
> The big assumption that the brain is doing gradient descent…

Our result only partially relies on gradient descent. We need it, along with the result in Eq. 8, to have a rigorous theory for Gaussian behavior during learning. However, as we outline in the second paragraph after Eq. 7, **our result ultimately relies on the large-sum behavior of update terms**, and these updates need not come from gradient descent. So, if we replace the gradient $\nabla l$ wrt the loss in Eq. 5 (the mirror descent update), $\nabla\phi(w^{t+1})=\nabla\phi(w^{t})-\eta\nabla l$, with anything else that governs learning, like a 3-factor Hebbian rule, the intuition would still hold: a large sum of small and mostly independent updates looks Gaussian in the dual space, $\nabla\phi(w^{\infty})=\nabla\phi(w^{0})  + \sum_{t=1}^\infty g^t\approx \nabla\phi(w^{0}) +\mathcal{N}$. But such a general case would make theory much harder (unless we put some trivial assumptions like literal independence of updates). Gradient-based optimization leads to the Bregman divergence result, which in turn leads to a closed-form approximate solution, which we can show to be Gaussian. We have added this explanation to Sec. 3.

More generally, we assume that an ANN trained by gradient-based methods can be used as a working model for the brain, and as a working abstraction can be used to understand aspects of how the brain learns and works. As long as the learning algorithm the brain uses descends a gradient, even if it is never explicitly calculated, then this framework can be applied.  However, this does not mean that we are committed to the brain doing gradient descent, per se. But it is a commonly applied framework that allows researchers to make tractable theoretical advances that can generate biological hypotheses - as we do in this manuscript. 

We also offer another point of view: if backpropagation is taken as the working model of the brain, is it consistent with neural data? Our conclusions are that it’s not in the simplest form (gradient descent) due to the solutions it finds, but other variants such as exponentiated gradient are consistent (with log-normals, Dale’s law, and multiplicative updates). This is important and impactful given the number of computational papers that assume gradient descent as a working model of learning in the brain. 

Lastly, we note that there is a sizable literature exploring learning in the brain under the assumption of gradient-guided plasticity. The question about the validity of this assumption is important – and work like ours help make progress– but this larger question is beyond the scope of our paper.

Reviewer comment (Theory):
“Notably, even if the brain does not estimate gradients directly, as long as synaptic weight updates are relatively small, then the brain’s learning algorithm must be non-orthogonal to some gradient in expectation” What’s the actual citation for this?? (i.e. not a review)

Author response:
> What’s the actual citation for this?? (i.e. not a review)

We referred to this: Richards and Kording. The study of plasticity has always been about gradients. The Journal of Physiology, 2023.

But the basic mathematical idea is that if you’re slowly improving your performance with small weight changes, you must be approximately following the gradient (wrt performance) since it is the mathematical direction for performance improvement. If you were not, then there would be no improvement in loss. As an additional citation (non-review) for this point, see Raman, D. V., Rotondo, A. P., & O’Leary, T. (2019). Fundamental bounds on learning performance in neural circuits. Proceedings of the National Academy of Sciences, 116(21), 10537-10546.

Reviewer comment (Presentation):
Couldn’t parse Fig 3. Needed more help in the caption / annotation.

Author response:
> Couldn’t parse Fig 3

Left plots: with more weights (lazy regime), the weight changes become more Gaussian (CDF difference) and smaller in magnitude. Right plots: if the change is Gaussian for one potential, measuring it with the wrong potential would result in a very non-Gaussian weight change. We hope this is clear?

Reviewer comment (Presentation):
“Nevertheless, with this data we can rule out a Euclidean synaptic geometry, and if we do have access to w0, then our results show that it is indeed possible to experimentally estimate the potential function.” Where do you show this? It’s not in Fig 5.

Author response:
> Where do you show this? It’s not in Fig 5.

Apologies, we should’ve made it more clear. We can rule out Euclidean geometry since it can’t produce Gaussian changes in the log space. We can estimate the geometry by finding the potential with the most Gaussian change (if the weights before and after learning are known).