Paper: Model Based Inference of Synaptic Plasticity Rules (ICLR-2024)

Reviewer comment (Theory):
The statement in the introduction regarding the biological plausibility of backpropagation may be too weak ("While the backpropagation ..., its biological plausibility remains a subject of debate."). It is widely accepted that backpropagation is biologically implausible.

Author response:
**B1.** We agree. We will modify this sentence to be stronger.

Reviewer comment (Writing):
Regarding the following sentence in Section 3 "We further define a readout ... ,i.e., $\mathbf{m}^t = f(y^t).$", did you mean to write $\mathbf{m}^t = f(\mathbf{y}^t)$ ($\mathbf{y}^t$ with boldsymbol)?

Author response:
**B2.** Yes. Thank you for catching this typo. We will correct it.

Reviewer comment (Writing):
On page 4, "setting $\theta_{110} = 1$ and $\theta_{012} = -1$," the second term should be $\theta_{021}$ rather than $\theta_{012}$.

Author response:
**B3.** Yes. Thank you for catching this typo. We will correct it.

Reviewer comment (Experiments):
The paper models synaptic plasticity rules only for feedforward connections. It would be interesting to explore the impact of lateral connections (by adding additional terms in Equation 6). Have you experimented with such a setup?

Author response:
**B5.** We agree with the Reviewer that it would be interesting to extend our method to recurrent neural networks. However, we could perform this numerical experiment in our setup without changing Eq. 6. In particular, x would become the presynaptic neuron’s activity, y would become the postsynaptic neuron’s activity, and w would be the weight between these neurons. The bigger difference is that Eq. 1 would need to be updated to incorporate recurrent network dynamics. We have not experimented with recurrence in our model, and this change would entail significant changes to our code bank. We have therefore decided to postpone these experiments for future work.

Reviewer comment (Experiments):
In page 7, the authors state that "In the case of the MLP, we tested various architectures and highlight results for a 3-10-1 neuron topology." What are the results for other various architectures? Putting them into the paper would also be valuable (as ablation studies).

Author response:
B6. We accepted the Reviewer’s suggestion and summarize additional experiments that varied the architecture of the MLP below. We found that the performance of the MLP architecture could be significantly improved by moderately increasing the size of the hidden layer. As suggested, we will include these simulations in our revised manuscript.

### Simulation information and results
Plasticity MLP architecture: 3 → h_l → 1

Network: 100 → 1000 → 1

Ground truth plasticity rule: x.(R - E[R])

Trained on simulated behavioral data

| Number of hidden units (h_l) | R2 weights | R2 activity | % Deviance Explained |
|------------------------------|------------|-------------|----------------------|
| 10                           | 0.68       | 0.95        | 62.99                |
| 50                           | 0.76       | 0.97        | 66.34                |
| 100                          | 0.62       | 0.93        | 61.32                |

Reviewer comment (Reproducibility):
The hyperparameters for the experiments are missing? What is the learning rate, what is the optimizer, etc.?

Author response:
**B7.** We will revise the manuscript to state these details. For the experiments, hyperparameters were fine tuned by grid search, with performance averaged across three random seeds. Hyperparameters included an L1 sparsity regularizer applied to the plasticity coefficients, with the optimal value found to be 5e-3, and a moving average window for calculating the expected reward, which showed minimal impact and was set at a value of 10. In terms of optimizers, our experiments included AdaBelief, Adam, and AdamW, using their default parameters for each. The results were comparable across these optimizers, and the findings presented in the paper are based on the use of Adam.

Reviewer comment (Novelty):
I do not see that much difference between the experiment presented in Section 4 and the experiment in (Confavreux et al., 2020) (Section 3.1) except the choice of optimization method. In your experimental setup, you also do not model the global reward. Therefore, I think it makes it more similar to the experiment in (Confavreux et al., 2020).

Author response:
**B8.** We agree with the reviewer, but we think that the choice of optimization method is highly significant. As explained in the Introduction and Related Work sections, our focus is on fitting a broad range of arbitrary rules, without being constrained by the specific computation they implement or their optimality. This approach reflects the flexible and expansive nature of our model. The method provided here provides an exciting opportunity to infer biological learning rules from real experimental data sets from biology, where the function of the circuit is a priori unknown. For instance, no one knows if the mushroom body is optimal for some specific task, but we were nevertheless able to estimate its learning rule from behavioral data in Figure 4.

Reviewer comment (Experiments):
Comparison to previous work is missing.

Author response:
**B9.** Can the Reviewer please clarify what comparison they would like to see? We already conceptually compared our work to previous papers in the Related Work section. However, since we’re optimizing different objective functions than almost all of these previous works, apples-to-apples comparisons are usually impossible. In the setting of Figure 4, we could revise the text to compare our model’s performance to the previous model in Rajagopalan et al. Would this help, or does the Reviewer have something else in mind?

Reviewer comment (Novelty):
As a follow-up question related to my comment about the similarity with (Confavreux et al., 2020): (Confavreux et al., 2020) uses Covariance Matrix Adaptation Evolution Strategy (CMA-ES) as they state it has better scalibility with the network size compared to gradient based strategy. Why do you use gradient based strategy? What is the advantage?

Author response:
**B1.** Our method is simpler to implement and appears to work perfectly well for our purposes. We therefore did not explore the CMA-ES approach. We note that Tyulmankov et al. 2022 also successfully used gradient descent to meta-learn plasticity rules, and our implementation more closely follows that paper.

Reviewer comment (Experiments):
In the experimental setup in Section 5, if you do not neglect the hypothetical dependencies on $y_i$, would your framework correctly infer the learning rule, i.e., if you use $$g_\theta = \sum_{\alpha, \beta, \phi, \gamma} \theta_{\alpha, \beta, \phi, \gamma} x_j^\alpha r^\beta y_i^\phi w_{ij}^\gamma$$ would your model learn $\theta_{\alpha, \beta, \phi, \gamma} = 0 \quad \forall \phi \neq 0$?

Author response:
**B2.** This is a good suggestion. We did not explore this learning rule because it involves significantly more coefficients that must be fit, and hypothetical dependencies of the mushroom body’s learning rule on y can be excluded on biological grounds. We will mention this in our revision. We will also try fitting this more complex learning rule in our revision to see whether the coefficients can still be recovered.

Reviewer comment (Experiments):
The paper tries to recover Oja's rule from experimental data. Though it is a good exercise, it would be more interesting if this method could be used to model a more generalized learning form and show the performance of using that learning method in small networks — for example, a Hopfield network with the new plasticity.

Author response:
3. **Generalized learning forms and network performance**: Your suggestion to model a more generalized learning form and benchmark its performance in networks like a Hopfield network is a good idea. However, that is not the topic of this paper, which you already recognize as “really interesting.” Our current focus is on developing and validating a method for inferring learning rules, rather than on enhancing or modifying specific learning methods like Oja's rule. That our paper suggests another interesting computational problem to the reviewer lends further credence to our view that a machine learning conference proceeding fits our work well.

Reviewer comment (Novelty):
The paper could improve on top of Oja's method using the added information and make it more biologically plausible.

Author response:
2. **Choice of Oja's rule**: 
The use of Oja's rule in our study is primarily for validation purposes. It serves as an arbitrary, yet well established, benchmark to demonstrate the efficacy of our methodology. Our choice of Oja's rule was driven by its recognizability and established theoretical foundations, making it a suitable candidate for ground truth comparison. However, it's important to note that our methodology is not restricted to Oja's rule; it can be applied to infer a wide range of synaptic plasticity rules. For example, the alternatives in Table 1 were not derived from any theoretical objective, they were merely many examples of learning rules that could be used to generate neural trajectories, and our method was consistently able to recover them. We do not think that it is necessary to use symbolic regression, or any other method, to find the optimal plasticity rule. Although it is interesting to relate the learning rules of single neurons to the output behavior of the whole system, that is not the topic of our paper. Instead, our focus is on fitting a broad range of arbitrary rules, without being constrained by the specific computation they implement or their optimality. This approach reflects the flexible and expansive nature of our model.

Reviewer comment (Writing):
Can you explain what you mean by "fits neural firing rate trajectories over the course of learning" in Page 2?

Author response:
- Q2. The method minimizes the difference between the measured (experimental or simulated) neural trajectories and those generated by the model network. We will edit the text to make this clearer for future readers.

Reviewer comment (Experiments):
You mentioned "Performance benchmarks reveal that our framework is capable of executing simulations with up to 10^6 synapses and 1000-time step trajectories in less than an hour on an NVIDIA A100 GPU." However, from what I understand, the experiments seem to have been done on a much smaller number of synapses. Can you clarify?

Author response:
- Q3. It’s already possible to record a hundred thousand neurons in larval zebrafish, and, given emerging trends in neuroscience, there is potential for future technologies to enable the recording of millions of neurons. This activity will stem from orders of magnitude more synapses. Although our current datasets do not encompass this scale, our synthetic data experiments show consistent results for both large and small networks. Crucially, our framework is designed to efficiently handle data from millions of neurons, preparing us for when such extensive recordings become available. We therefore think that it is important to quantitatively benchmark the performance of our approach.

Reviewer comment (Writing):
I could not fully understand the results in Section 4.2. It would be great if the authors could add more details on "The remaining (1 − a) fraction of neurons are modeled by incorporating random noise drawn from the same distribution as the observed neurons."

Author response:
- Q4. We recognize the importance of this detail and will include a clarification in our revised paper. In brief, for the unobserved (1 − a) fraction of neurons in our model, we incorporate random noise drawn from the same distribution as the observed neurons (the initial distribution). This is because the exact values of these unobserved neurons are unknown, but they nevertheless have a big impact on the activity level of downstream neurons. We assume knowledge of their overall activity distribution to incorporate this effect.

Reviewer comment (Novelty):
The fact that the method is restricted to rate networks precludes interesting synaptic rules from spiking networks (e.g. spike-timing dependent plasticity) that could explain a wider a range of plasticity mechanisms, and thus limits the set of interesting rules that could be inferred using this method.

Author response:
**Spiking neural network models**

Although our approach uses rate networks as the underlying neural network model, our approach is not limited to rate networks. In the case of stochastic spiking models, we demonstrate an analogous scenario by fitting fly behavioral data with a probabilistic model, maximizing the log-likelihood of the model’s output given the data. This can easily be applied to fitting spiking (e.g. Poisson) neural trajectories rather than stochastic behavioral trajectories. 

Alternatively, we can think of rate-based neural networks as an abstraction over spiking networks. Given spiking data, we can generate a rate code by smoothing e.g. with a Gaussian kernel, which we can fit immediately with a rate model. Furthermore, STDP rules can be mapped onto Hebbian plasticity in rate networks [1], so rate networks do not preclude STDP as a potential plasticity mechanism (although we do concede that the precise timing of the spikes is lost). 

Our approach can also be easily modified to account for spiking data without any preprocessing, using a spiking model. As presented in the current work, the only limitation is the optimization algorithm. If the spiking model is an Hodgkin-Huxley type model with differentiable spike waveforms, we can use our method out-of-the-box – the only limitation is compute power, since these models have extremely detailed dynamics and many CPU cycles are required to simulate a single spike. If the spiking model is comprised of binary neurons with a Heaviside step function nonlinearity, optimization techniques such as surrogate gradients can be used. In the case of non-differentiable models, e.g. integrate-and-fire neurons, we can use evolutionary algorithms.

Reviewer comment (Evaluation):
Section 7 mentions that "[a]nother issue is the model's "sloppiness" in the solution space; it fails to identify a unique sparse solution even with extensive data". If this is the case with the current experiments in sections 4-6, how does this affect the interpretation of the results and how much can we rely on the experimental predictions about plasticity mechanisms? For instance, which particular experiment/solution forms the basis for the conclusions about the decay mechanism in 6.1 and bidirectional plasticity in 6.2?

Author response:
**Sloppiness in fit and interpretability/reliability**

Although the sloppiness of fit is a real problem in our models, we again think that it is a generic problem in statistical model fitting rather than a limitation in our specific approach. Interpretability depends on the functional form of the model (or equivalently, in the case of probabilistic graphical models, its graph structure). We demonstrate our method with an interpretable plasticity model in the form of a Taylor expansion, as well as a less interpretable one in the form of an MLP. The reliability of the method, as with all statistical models, depends on the quality and the quantity of the data. Standard statistical techniques can be used in these cases – confidence intervals, bootstrapping, cross-validation, etc. Some hypotheses, however, by definition cannot be distinguished with a particular dataset. 

We would also like to highlight the fact that the sloppiness occurs in the specific parameterization of our plasticity rule, rather than in a degeneracy of the rule itself (unlike the results in [2], which, again, we will discuss in the final version of our submission). We find a unique plasticity function, i.e., the shape of the input-output curve on a graph, but not a unique parametrization of that function.As a trivial example, the functions $y = x$ and $y = x^2$ are indistinguishable if the only values of $x$ which we sample are 0 and 1, so if the ground truth is $y = x + x^2$, we can fit any function $ax + bx^2$ with $a + b = 1$.

Nevertheless, regardless of the parametrization of the plasticity rule, our method is informative with regard to the inputs that are relevant for a plasticity update, e.g. whether pre- or post-synaptic activity plays a role, or both; or, as we demonstrate in Drosophila, whether a weight decay is present. Following the adage, “All models are wrong but some are useful,” we believe that our model is useful from this perspective.

Reviewer comment (Experiments):
Methodological choices are a main weakness of the work.

Author response:
**Methodological Choices and Theoretical Grounding**

Our decision to include multiple output neurons in our model is primarily to demonstrate scalability. The model effectively functions with a single output neuron, but the use of multiple neurons showcases its applicability to larger, more complex networks. This is not a trivial problem as the loss is computed over the entire trajectory length and parameters are optimized with backpropagation through time. In the context of the behavioral task, the outputs are pooled through a fixed layer followed by a sigmoid function, determining the probability of accept/reject decisions. It's noteworthy that these output neurons typically follow unique trajectories. The convergence to the same outputs in the case of Oja’s rule is a specific instance, not a general trend that holds across learning rules. This aspect also underlines the versatility of our model in fitting diverse rules. 

We will revise the manuscript to mention the link between Oja’s rule and principal component analysis, to state that the theoretical motivation for including the E[R] term is to enable covariance-based learning that leads to operant matching, and to explain that the weight decay term could explain forgetting. However, we do not think that it is necessary to provide a theoretical rationale for the other learning rules that we consider. Instead, our focus is on fitting a broad range of arbitrary rules, without being constrained by the specific computation they implement or their optimality. This approach reflects the flexible and expansive nature of our model. Although we agree that it is interesting to relate the learning rules of single neurons to the output behavior of the whole system, that is not the topic of our paper. 

For the E[R] term, our experiments encompassed various time scales (5, 7, 10, 15, 20) trials, leading to consistent results across these variations. This uniformity in outcomes led us to exclude it as a learnable parameter in our optimization loop. Dynamics within the trial had no bearing on E[R].

Reviewer comment (Theory):
Theoretical grounding of the learning rules is lacking.

Author response:
We will revise the manuscript to mention the link between Oja’s rule and principal component analysis, to state that the theoretical motivation for including the E[R] term is to enable covariance-based learning that leads to operant matching, and to explain that the weight decay term could explain forgetting. However, we do not think that it is necessary to provide a theoretical rationale for the other learning rules that we consider. Instead, our focus is on fitting a broad range of arbitrary rules, without being constrained by the specific computation they implement or their optimality. This approach reflects the flexible and expansive nature of our model. Although we agree that it is interesting to relate the learning rules of single neurons to the output behavior of the whole system, that is not the topic of our paper.

Reviewer comment (Novelty):
There is limited novelty in the trajectory optimization approach.

Author response:
**Limited Novelty of Trajectory Optimization**
We see two possible interpretations of what the reviewer means by endpoint optimization. First, they might be imagining modeling the entire plasticity sequence as one large plasticity update, as done by Lim et al. In this case, the novelty of our approach is highly significant, because many nonlinear weight updates do not sum to be a single nonlinear update of the same mathematical form. Our model is much more aligned to the ground-truth generative process, and that allows us to recover the correct learning rule. We would be willing to add this point to the revision if the reviewer thinks it would help justify the significance of our approach.  Second, end-point optimization might simply suggest that the loss function depends only on the final states, not the full learning trajectories. We could easily do this experiment to compare trajectory optimization with this notion of endpoint optimization. However, we do not think that the outcome of this comparison has any bearing on the significance of this paper, because optimizing either loss function requires backprop through time and the framework that we introduce here to do so. No previous work has optimized a learning rule based on end points in this way. This comparison could turn out to be interesting, but adding it would not significantly improve our paper, and we would prefer to not do so.

Reviewer comment (Experiments):
It is unclear why there are multiple output neurons, how they are pooled, and whether they all learn the same thing. There is a question of whether multiple output neurons are used just to have enough data, making it equivalent to many instantiations of the simulation with single output neurons. Additionally, it is unclear whether neurons using Oja's rule all simply learn the first principal component.

Author response:
The reviewer asked several related questions. Our responses are:
* Output neurons are uniformly summed (weights equal to 1), and a logistic sigmoid function is applied to that sum to generate a probability of choosing the current odor. 
* Our discussion of  “scalability” refers to our implementation’s ability to handle large networks, rather than any comment about data efficiency.
* In the general case, learning rules define nonlinear dynamical systems that can depend sensitively on their initial conditions as well as the particular noise instantiation. Because of this, the output neurons do not necessarily follow identical trajectories – the pooled behavioral readout is then a linear combination of their individual (unique) outputs. As such, additional output neurons do offer additional information about the plasticity rule beyond simple data availability.  However, as with Oja’s rule (which learns weights that align with the first principal component), it may be the case that output neurons behave similarly. Nevertheless, Table 1 shows that the method works for many different learning rules beyond Oja’s rule, and it is inaccurate to say that we’re generalizing because our method “works for a specific case.”

Reviewer comment (Evaluation):
If all neurons in the output layer learn the same thing, the study in Fig. 2 with noise and sparseness might reflect only how many "trials" are needed for a single neuron simulation. In that case, noise would merely reduce the informativeness of the gradients for meta-learning, requiring more trials — a situation that would be true for any learning simulation.

Author response:
The reviewer asked several related questions. Our responses are:
* Output neurons are uniformly summed (weights equal to 1), and a logistic sigmoid function is applied to that sum to generate a probability of choosing the current odor. 
* Our discussion of  “scalability” refers to our implementation’s ability to handle large networks, rather than any comment about data efficiency.
* In the general case, learning rules define nonlinear dynamical systems that can depend sensitively on their initial conditions as well as the particular noise instantiation. Because of this, the output neurons do not necessarily follow identical trajectories – the pooled behavioral readout is then a linear combination of their individual (unique) outputs. As such, additional output neurons do offer additional information about the plasticity rule beyond simple data availability.  However, as with Oja’s rule (which learns weights that align with the first principal component), it may be the case that output neurons behave similarly. Nevertheless, Table 1 shows that the method works for many different learning rules beyond Oja’s rule, and it is inaccurate to say that we’re generalizing because our method “works for a specific case.”

Reviewer comment (Novelty):
One main contribution claimed is optimizing trajectories instead of endpoints, but it is unclear whether this represents a substantial advancement. This point would be strengthened by more results and discussion on the difference between trajectory and endpoint optimization. Questions include: What data types can this model handle that endpoint optimization cannot? Would the results be the same? Is this a matter of data efficiency? How would endpoint optimization perform on the fly dataset? Also, is backpropagation through time used?

Author response:
Our main contribution is indeed optimization over a trajectory with backpropagation through time (BPTT). To be precise, however, we still optimize a loss function, although it is defined as the error between the data and model trajectories, rather than a functional relationship as in previous work [Confavreaux et al. 2021, Tyulmankov et al. 2022]. To optimize this loss function we use BPTT, which was implicit in the definition of the loss function (Eq. 3) and its derivative (Eq. 4) due to the time dependency. We agree that this should be made more clear and explicit, and we will update the text accordingly.

It might be helpful to distinguish between the optimization *problem* and the used optimization *technique* to solve it. Our approach is fundamentally different from alternative setups in terms of the optimization problem. We provide a discussion and comparison in the "Related Work" section. The only other work we are aware of that uses neural trajectories to fit a plasticity rule is [Ramesh et al. 2023], which was only published to *bioRxiv* after our initial submission. We will include a qualitative comparison in the final version of our manuscript nevertheless. In terms of comparison to other optimization techniques, the choice of optimizer is an implementation detail that we could revisit in future work. Indeed, alternatives for computing gradients such as real-time recurrent learning (RTRL) can be used, or approximations to the gradient such as truncated BPTT, or even non-gradient based methods such as evolutionary algorithms.

However, any approximations (whether approximations to the gradient or using only endpoints in the loss) will be strictly less data-efficient (although may be more computationally- and memory-efficient), since we would be ignoring long-term dependencies between the parameters and data. As such, we opted for the most powerful methodology because we wanted it to work for arbitrary rules, some of which might be highly nonlinear and benefit from long time dependencies more than others (see below).

Reviewer comment (Theory):
Another main contribution claimed is fitting behavioral data, but there is a conceptual gap between modeling a synapse and modeling a whole system as if the fly were a single neuron. It is unclear why modeling a single neuron would work in this case, and whether this approach is more appropriate for neural data from relevant neurons in the fly mushroom body.

Author response:
*Modelling Synaptic and System-Level Behavior:*
Recent research, particularly by Aso et al., has demonstrated that activating a specific neuron in the fly mushroom body can influence the fly's upwind movement. This finding is crucial to our experimental setup, where odors flow and the fly's reward-driven upwind movement is a key behavior. This validates our approach of modeling single neurons, as it aligns with the observed behavior in real-world scenarios. Our approach also mimics the setting of a previous biology study by Rajagopalan et al. that we wanted to extend with our methodology. We will modify the text to explain these points and specify the exact neuron referenced in this context for clarity.

Reviewer comment (Theory):
There is a lack of theoretical grounding of the studied learning rules. For example, it is not established that the Oja rule will learn the first principal component, what the optimal learning rules are for the reward-based task, or what the effect of the weight decay term is on the learning rule.

Author response:
We will revise the manuscript to mention the link between Oja’s rule and principal component analysis, to state that the theoretical motivation for including the E[R] term is to enable covariance-based learning that leads to operant matching, and to explain that the weight decay term could explain forgetting. However, we do not think that it is necessary to provide a theoretical rationale for the other learning rules that we consider. Instead, our focus is on fitting a broad range of arbitrary rules, without being constrained by the specific computation they implement or their optimality. This approach reflects the flexible and expansive nature of our model. Although we agree that it is interesting to relate the learning rules of single neurons to the output behavior of the whole system, that is not the topic of our paper.

Reviewer comment (Writing):
The MLP learning rules are included in the work but never discussed. They should be better motivated and explicitly referred to.

Author response:
We parameterized the learning rule with an MLP to see whether we could capture the learning rules with a general expressive model that did not match the ground-truth generative model. However, Figure 3 and Table 1 revealed that the MLP usually underperformed the Taylor series representation. We therefore decided to focus Figure 4 on the Taylor series representation, which is both more interpretable and better aligned to the known biology. We will explain these motivations and findings in the revised manuscript. For similar reasons, we did not attempt to quantify what the MLP rules learned in Figure 3. If the MLP approach had been more empirically successful, then we would have interpreted the learned rules by directly plotting the three-dimensional function of presynaptic activity, reward, and weight that they discovered. This could have easily been compared to the exact function specified by the ground-truth generative model.