Paper: Nature-Inspired Local Propagation (ICLR-2024)

Reviewer comment (Writing):
What is pre-algorithmic view?

Author response:
Many thanks to the Reviewer for his humble approach.  When looking at the
constructive his criticisms, it looks like his/her expertise in
reinforncement learning led to understanding the paper much more than what
mentioned in the "Disclaimer part."
We have tried to use some of the comments exposed in the weakness section
to improve the presentation. In particular we tried to clarify some terms
like `"pre-algorithmic algorithm" and to immediately comment on the role
of loss function of the term $\ell$ right after Eq. (8). We furthermore
changed the notation for the general variable that indicate the $\boldsymbol{w}$
component of the state that was previously denoted as $\boldsymbol{u}$ and
now it is named $\boldsymbol{\omega}$ to further disambiguate it from
the input variable $u$. We would also like to remark that the ${\rm IN}$ function is
defined on right after Eq. (5). In what follows we will answer to the precise
questions that the Reviewer formulated in the corresponding section of
his/her review.

**Q3.** Perhaps we understood the comment, but we invite the Reviewer
to ask again if we didn't grasp the essence of the question. 
Cycles like $A\to B\to A$ amongst non-input units are permitted 
since our assumptions correspond to typical recurrent nets. However, the
consistent computational scheme that we adopt does require that the
processing does not take place "within the same time step"! In general, in
the discrete setting of computation, we need a delay, while in the continuous
framework this correspond to the introduction of temporal derivatives of the
state.

Reviewer comment (Novelty):
**Existing literature**. The paper is motivated by biological learning where the organisms do not have access to the entire data collection ("the agent can only use buffers of limited dimensions for storing the processed information"), however, the paper ignores the existing literature on Online Learning and Online Optimisation, that are highly related to the topic. Putting the work into this context would be highly beneficial for the community.

Author response:
**Q2.** We have added the references and commented how our
approach contributes to the mentioned problem.

Reviewer comment (Writing):
Could you elaborate on the meaning of 'temporal horizon' in Section 2? Imagine we have an RNN which we update using gradient descent. The inputs have temporal aspect (e.g. sequences of real numbers), and the updates have temporal aspect (each weight updates adds one to the time subscript of the weight). Which one is implied in Section 2? Are the two intertwined as mentioned in the second paragraph of Section 1 ("we assume that learning and inference develop jointly")? Could you provide a toy example?

Author response:
**Q1.** Yes, in this work, we are mainly concerned with online
computations where there is a strong connection between the time that
captures the dynamics of the sequence we are processing, expressed in our
notation by the function $t \mapsto u(t)$, and the dynamics of the parameters
of the model defined by the map $t \mapsto \boldsymbol{w}(t)$. Many learning
protocols can be cast in this framework, ranging from reinforcement learning,
lifelong and online continual learning.  The specific strong emphasis in this
paper is that we look for a learning framework which can work without
accessing large data collections, but simply by processing and properly
representing the information as it is comes from sensors over the given
temporal horizon (agent life span).

Reviewer comment (Theory):
Section 2 defines the computational model based on a digraph. Should there also be an acyclicity constraint that does not allow "inputs -> A -> B -> A" within the same time step?

Author response:
**Q3.** Perhaps we understood the comment, but we invite the Reviewer
to ask again if we didn't grasp the essence of the question. 
Cycles like $A\to B\to A$ amongst non-input units are permitted 
since our assumptions correspond to typical recurrent nets. However, the
consistent computational scheme that we adopt does require that the
processing does not take place "within the same time step"! In general, in
the discrete setting of computation, we need a delay, while in the continuous
framework this correspond to the introduction of temporal derivatives of the
state.

Reviewer comment (Theory):
For me, equation 8 comes out of the blue. Section 2 introduces the computational model and the constraints on it. Beginning of Section 3 mentions CAL (Betti et al., 2019) for FNN (feedforward NN?). Then the paper claims that in an online setting the laws of learning for recurrent architectures can also be characterized by minimality of a class of functional. It then considers variational problems for the functional of the given form. Could you explain where equation 8 comes from? Is the \(mc/2 * |w|^2\) the regularisation term? If yes, does the \(l\) in equation 8 differ from the \(l\) in Theorem 2 that already includes the regularisation term?

Author response:
**Q4.** The term $\int_0^T
\ell(\boldsymbol{w}(t),
x(t;\boldsymbol{w}),t))\phi(t)\, dt,$
comes from the ergodic transposition of the
functional risk inspired by classical mechanics as discussed in
(Betti et al., 2019). The basic idea is that when dealing with time and with a
stream of information the expectation of the loss that defines the empirical
risk can be rewritten as a sum over time. In our case however, differently
from previous works in this directions, we are considering recurrent
computations and, therefore, the term $x(t;\boldsymbol{w})$  is a solution of the
recurrent dynamics expressed by Eq. (4). As correctly noticed by the Reviewer,
function $\ell$ can itself contain a regularization term
on the weights (like the classical weight decay). However it is important to
add an additional *temporal regularization term* that links the weights
at different temporal instant of the form $\int_0^T
mc/2|\dot\boldsymbol{w}|^2\, \phi(t) dt$ for at least two reasons:
firstly, it promotes
convergence and the smoothness of the solution and, secondly, it enables
the explicit calculation of the Hamiltonian in as in Eq. (11); 
hence, there is always a classic regularization term on the weights and also 
an appropriate temporal regularization term on the derivative of the weights 
that is consistently considered in Eq. (8) and in Theorem 2.

Reviewer comment (Theory):
The statement "The local structure of equation 10, that comes from the locality of the computational model from Section 2 guarantees the locality of Hamilton's equation 12" — could you, please, explain why this is the case?

Author response:
**Q5.**
This is because the fact that we are considering the particular set
of differential constraints in Eq. (10) allows us (by using the
techniques of optimal control) to describe the stationary point of
(9) in terms again of local differential equations (Hamilton Equation).
In this equation, in addition to the spatial locality, the presence of time
derivatives gives rise to local temporal propagation.

Reviewer comment (Writing):
Beginning of page 5 mentions backprop for FNN leading to equation 15 after introducing normalised costates. Could you explain the jump from considering RNNs to FNNs?

Author response:
**Q6.**
While the reduction for $c\to\infty$ can be formally done in general
as suggested for instance in Corollary 1, it only describes a meaningful and
viable computational scheme only when we have a directed acyclic graph
(which correspond to the FNN case).
So the reduction indeed makes sense only under the additional hypothesis
that are the one that makes possible to apply the standard Backpropagation
algorithm.

Reviewer comment (Theory):
In Corollary 1, "the proof comes immediately from equation 16." Could you add a couple of explicit steps to understand why this is the case?

Author response:
**Q7.** Consider the last equation in the system of Eq. (16) with all
$c_i=c$, and group them as follows
$$ \lambda^i_x -\sigma'(a_i)\sum_{k\in{\rm ch}(i)}\lambda^k_x w_{ki} - L_{\xi_i}(x,t)\sigma'(a_i)= \frac{1}{c} \dot \lambda^i_x -\frac{1}{c}\biggl[-\frac{\dot\phi}{\phi}+\frac{d}{dt}\log(\sigma'(a_i))\biggr] \lambda^i_x,$$
now the rhs of this equation as $c\to\infty$ formally goes to $0$ and
we are left with the desired equation.

Reviewer comment (Theory):
The statement "This is a strong indication that when solving Hamilton's equations with an initial condition on \(p_w\) we will end up with a solution that is far from the minimum" — how does this relate to any of the practical consequences on learning algorithms, e.g. backpropagation?

Author response:
**Q8.** This implies that, in general, it is not possible to simply define
learning/optimization theories that operate in a temporal environment solely
in terms of optimization problems for integral functionals (like the one in
Eq. 8), unless the problem is small enough to be solved in a batch-mode
fashion. This observation is also related to the fact that the solution of
calculus of variations typically describes elliptic problems, and they are
incompatible with hyperbolic problems, which is usually the a class of
problems that are suitable to describe temporal evolution (like learning in
this context).
This is why the discussion proposed in Section 4, addressing boundary
conditions, is instrumental to enable the use of such tools within the
context of learning.

Reviewer comment (Theory):
Can you elaborate more how Corollary 1 can be viewed as backpropagation and Proposition 4 as GD with momentum?

Author response:
**Q1.** Corollary 1 shows that in the limit of infinite speed of
propagation ($c\to+\infty$) the differential equation for the 
rescaled costates, the $\lambda_x^i$, described in Eq. (16) (last equation)
satisfy exactly the classical equation for the delta error terms used in
the Backpropagation algorithm. This reduction, which in Corollary 1
is formally expressed in the general setting, make sense only in the
case when the graph that we consider is acyclic which is an assumption
required by Backprop. In this case, which correspond to the classical
architectures of FNN
we can choose, as it is usually done, the set of output nodes to be the
one at the "end" or at the "top" of the architecture, i.e. the ones
that have no childrens ($i\in O$ implies ${\rm ch}(i)=\emptyset$.
Then if we look at Eq. (17):
$$
\lambda^i_x= \sigma'(a_i)\sum_{k\in{\rm ch}(i)}\lambda^k_x w_{ki} + L_{\xi_i}(x,t)\sigma'(a_i),
$$
we can make the following observations: *i.* if $i$ is an output neurons,
$i\in O$, then the sum $\sum_{k\in{\rm ch}(i)}$ is empty and we get
$$\lambda^i_x = L_{\xi_i}(x,t)\sigma'(a_i)$$
which is exactly the statement that
the delta error on the output neurons is simply the gradient of the loss
with respect to their activation, *ii.* when $i\ne O$ it is
the second term that vanishes since, as we remarked in Theorem 2 $L$
only depends on the value of the neurons in the output and reduces to
$$ \lambda^i_x= \sigma'(a_i)\sum_{k\in{\rm ch}(i)}\lambda^k_x w_{ki},$$
that again is exactly the classical way in which the delta error is computed
in Backprop.

For what concerns Proposition 4 connections with GD with momentum can be
better understood as follows. The second formula in Eq. (20) has the form
$$\ddot w_{ij}+\theta \dot w_{ij} + \gamma \nabla_{u_{ij}} E(w,t)=0$$
where $E$ is a generic compound loss.
This is the case 
since the term $V_{u_{ij}}(\boldsymbol{w},t)$ is an explicit gradient of
a regularization loss with respect to the weight $w_{ij}$ and $\lambda^i_x x_j$
is the Backprop factorization  of the gradient of the loss
that involves the outputs of the NN with respect again to the weight
$w_{ij}$. If you consider an Euler discretization of this ODE
you end up with
$$\frac{1}{\tau}(w_{ij}^{k+1}-2 w_{ij}^k +w_{ij}^{k-1})
+\theta(w_{ij}^{k+1}-w_{ij}^k)+\gamma \nabla_{u_{ij}} E(w^k, t_k)=0$$
that can be rearranged into the GD with momentum update rule
$$w_{ij}^{k+1}=w_{ij}^k -\alpha \nabla_{u_{ij}} E(w^k, t_k) +\beta(w^k_{ij}-w^{k-1}_{ij}).$$
Interestingly this continuous interpretation 
was already introduced in the vary same paper that proposed for the
first time the GD with momentum method, namely in
B. T. Polyak, *U.S.S.R. Comput. Math. Math. Phys.* 4,1–17 (1964).

Reviewer comment (Reproducibility):
Can you recommend an open source to understand how Corollary 1 can be viewed as backpropagation along with the Gori et al. (2023)?

Author response:
**Q2.**
As explained in the response to Q1, any textbook that explains in details the
Backprop algorithm is ok. An excellent, widely available reference is the
classical Rumelhart, David E., James L. McClelland, and PDP Research
Group. "Parallel distributed processing, volume 1: Explorations in the
microstructure of cognition: Foundations." (1986).
For a completely open source material you can look for instance at this
lecture note (section 6) available online 
https://www.cs.cornell.edu/courses/cs5740/2016sp/resources/backprop.pdf
which appears to be ok but of which we cannot guarantee
correctness of all its parts.

Reviewer comment (Theory):
Can a different choice of \phi lead to different learning rules in Proposition 4? Or does it lead only to different schedules for momentum?

Author response:
**Q3.** The structure of the learning rule (not the precise form
however!) that one can achieve with a general $\phi$ is described by
Eq.~(16). A meaningful generalization of Proposition 4 with general $\phi$
can be achieved if we choose $\phi$ such that $\dot\phi/\phi>0$. The effect
on the resulting method is not simply that of having a time dependent
coefficient in front of the momentum term as the choice of $\phi$ also
influences the computation of the delta-term-like variables
$\dot\lambda$. Hence, to some extent, the Reviewer's intuition that a truly
different learning rule might arise makes sense. However, the concrete impact
in the process of learning has still to be explored.

Reviewer comment (Reproducibility):
Can you write algorithms for the proposed learning rules (including sign flipping algorithm)?

Author response:
**Q4.** The algorithmic description of the proposed rules
basically relies on the explicit Euler approximation scheme of an ODE.
[As a reference you can look at the classical book
Richard, L., Faires J Douglas, and M. Annette.
Numerical analysis. sengage, 2016.]
For instance in the case of the ``sign-flipping algorithm'' the 
once you choose a temporal discretization step $\tau>0$ the algorithm looks
like this

1. Randomly initialize the state and costate variables and set the sign
term in Eq. (23) equal to $1$: $s\gets 1$;
2. Compute the update terms following Eq.(14):

    - $\Delta x_i\gets c_i\Bigl(-x_i +\sigma\Bigl(\sum_{j\in{\rm pa}(i)} w_{ij}x_j\Bigr)\Bigr)$

    - $\Delta w_{ij}\gets -p^{ij}_\boldsymbol{w}/(mc)$

    - $\Delta p^i_x\gets c_i p_x^i-\sum_{k\in{\rm ch}(i)} c_k  \sigma'\Bigl(\sum_{j\in{\rm pa}(k)} w_{kj}x_j\Bigr)p_x^k w_{ki} - c L_{\xi_i}(x,t)$

    - $\Delta p_{\boldsymbol{w}}^{ij}(t)\gets - c_i p^i_x \sigma'\Bigl(\sum_{m\in{\rm pa}(i)} w_{im}x_m\Bigr)x_j - c k V_{u_{ij}}(\boldsymbol{w},t)$

3. Update all the variables according to the implicit Euler scheme:

    - $x_i\gets x_i +\tau s \Delta x_i$

    - $w_{ij}\gets w_{ij} +\tau s \Delta w_{ij}$

    - $p^i_x \gets p^i_x +\tau s \Delta p^i_x$

    - $ p_{\boldsymbol{w}}^{ij} \gets p_\boldsymbol{w}^{ij}+\tau s \Delta p^{ij}_\boldsymbol{w}$$

4. Recompute  $s$ according to Eq. (24).
5.  If this is not the last iteration go back to step 2.

Reviewer comment (Theory):
Can you motivate the form of functional in Eq. 8? Specifically, why regularization on weight acceleration is important?

Author response:
**Q5.** The regularization on the velocity of the weights it is
important for at least two reasons: firstly it promotes convergence in the
parameters of the models, as well as the smoothness of the overall learning
process, and secondly enables the explicit calculation of the Hamiltonian in
as in Eq. (11); without this term the Hamiltonian would have been defined in
terms of another implicit minimization problem.

Reviewer comment (Theory):
How q x^2 /2 can be interpreted as the accuracy term in Section 4.2?

Author response:
**Q6.** In the classical LQ problem the objective is to have bring the state variable
as close as zero as possible, while also keeping small values of the control
parameters, hence the more the term $x^2/2$ is weighted (by taking higher value
of $q$) the more the solution is expected to accurately favour the zero,
constant solution. In the experiment done with the sinusoidal signal there
is indeed a misprint in the text the functional that we are considering is
$G(v)=\int_0^T q (x-z)^2/2+rv^2/2 + r_w^2\, dt$ so that $q (x-z)^2/2$
has a direct interpretation as accuracy parameter. This has been now
corrected in the revised version.

Reviewer comment (Writing):
What are z, u and x in Fig. 1-4? z is target, x is a vector predicted with LSTM, and u is input? I failed to see how objectives in section 4.2 encourage x to be close to z. Can you elaborate on this?

Author response:
**Q7.** In this figure $x$ is the value of the output neuron,
$z$ is the target and $u$ is an input as in Eq. (5) (notice that however in
this experiment it is just a null signal). As explained in
Q6, the actual accuracy term that is used in the experiments is $q (x-z)^2/2$
which should clarify the Reviewer's legitimate question.

Reviewer comment (Theory):
You mentioned that the proposed algorithm overcomes the limitation of backpropagation through time, but at the same time the algorithm converges to backpropagation when c → to infinity. Can you elaborate on this?

Author response:
**Q8.** The proposed theory works in general for a directed graph
with the properties described in Section 2, which basically correspond with
recurrent neural network architectures. However as we remarked also in Q1.,
while the reduction for $c\to\infty$ can be formally done in general as
suggested for instance in Corollary 1, it only describes a meaningful and
viable computational scheme only when we have a directed acyclic graph (case
for which Backprop can be used).
So the reduction indeed makes sense only under the additional hypothesis at
the basis of standard Backpropagation algorithm. In all other general cases,
that are the one of interest for this study, the expression we found in the
limit are just formal expressions. However, what is really important is that
the proposed theory clearly leads to a spatiotemporal local propagation
scheme which, to the best of our knowledge, is an open problem for recurrent
neural nets.