Paper: RNNS with gracefully degrading continuous attractors (ICLR-2024)

Reviewer comment (Theory):
The theoretical framework proposed, while enlightening the UBLA and BLA models, does not lead to a concrete prescription as to how to build RNNs that are robust to weight perturbations.

Author response:
There is some literature on the necessary conditions for attractive and bounded solutions to exist, see for example Yi (2001). See conditions for a line attractors in ReLU RNNs in Yu (2009), one just needs to rotate these line attractors so that they become bounded. Furthermore, conditions of the equilibria in more generality (line and plane attractors) of the 3D RELU network are presented in Xiang (2021) and parameters for bounded plane attractors for 4D tanh networks are discussed in Yu (2023). These construction should lead to RNNs with continuous attractors that are robust to weight perturbations given our theory.

Reviewer comment (Experiments):
The models considered are two-dimensional, how do the findings extend to larger models?

Author response:
The theory tells us that regardless of the dimensionality of the RNN, as long as they have bounded normally hyperbolic continuous attractors the only way to create exploding gradients is if there is a chaotic system in the neighborhood of the original.
Currently, all systems considered have a slow flow after perturbation, which can be maintained with the correct learning rate. If one would take non normally hyperbolic attractors, the exploding gradients would form an even bigger problem. For RNNs with a ReLU  activation function, all continuous attractors are normally hyperbolic however, whether they are bounded or not.

Reviewer comment (Novelty):
If I understood the paper correctly, the sole takeaways of the study to have robust RNNs are to pick ones which have bounded and smooth stable manifolds and to use error gradients to counter act noise (however not so efficiently per the results).

Author response:
This indeed underlies the generality of the theory. 
However, it is not required that the invariant manifold be smooth.
Furthermore, using backpropagated gradients would be one way to counteract noise, which is especially problematic for non-normally hyperbolic invariant manifolds.

Reviewer comment (Experiments):
The experimental setup may not be relevant to larger models, given that the RNN robustness is studied when the model is *already initialized at its optimum*, which may not be the case for a realistic task.

Author response:
We agree that our theory only covers cases where there is a continuous attractor implemented in an RNN. However, our theory does apply to RNNs of all sizes, hence also for larger models. 
these theoretical results nullify the concern of the fine tuning problem in neurosciences
We agree with the reviewer that this is too strong of a claim given our current understanding of continuous attractors and we have softened our language in the manuscript accordingly.
However, we would still like to note that the BLA has the broadest range of learning rates that are optimal and far away from really badly exploding gradients.
Furthermore, the somewhat negative results are specific for the backpropagated gradient that we consider here. It is not a general principle that homeostatic restoration of continuous attractors is challenging, but only for the gradient signal considered in the paper. In fact, a control mechanism could be theoretically devised that relies on finite control input for all bounded continuous attractors, while this is not guaranteed for unbounded continuous attractors.

Reviewer comment (Experiments):
The way the generality of the results are phrased in the introduction ("these theoretical results **nullify** the concern of the fine tuning problem in neurosciences and suggest **general principles** for evaluating and designing new architectures and initialization strategies for RNNs in ML") starkly contrast with the experimental setup considered (namely 2-dimensional RNNs on toy problems) and the conclusion which tends to be negative. Also, I do not see in this paper any proposal of "initialization strategies for RNNs in ML".

Author response:
We agree that our theory only covers cases where there is a continuous attractor implemented in an RNN. However, our theory does apply to RNNs of all sizes, hence also for larger models. 
these theoretical results nullify the concern of the fine tuning problem in neurosciences
We agree with the reviewer that this is too strong of a claim given our current understanding of continuous attractors and we have softened our language in the manuscript accordingly.
However, we would still like to note that the BLA has the broadest range of learning rates that are optimal and far away from really badly exploding gradients.
Furthermore, the somewhat negative results are specific for the backpropagated gradient that we consider here. It is not a general principle that homeostatic restoration of continuous attractors is challenging, but only for the gradient signal considered in the paper. In fact, a control mechanism could be theoretically devised that relies on finite control input for all bounded continuous attractors, while this is not guaranteed for unbounded continuous attractors.

Further research needed to test efficiency of BLA as an initialization strategy. Our theory suggests that networks with bounded continuous attractors) will not exhibit exploding gradients under noisy conditions, while RNNs initialized with the identity matrix will.

Reviewer comment (Theory):
Lack of mathematically precise explanations and lack of guidance on how to interpret the 2D results for RNN training in general.

Author response:
> i. the lack of mathematically precise explanations

We have adjusted the language in Theorem 1 (Fenichel's Invariant Manifold Theorem) to make the statement mathematically mode precise. To comment on the complications of determining the extent of noise that a given system can handle, a lot more research is needed to develop a full account for what the maximal $\epsilon$ (the timescale parameter) value is for which persistence holds. But in principle it can be calculated, see Chapter 6.5 in Wiggins (1994).
 If there are any particular parts of the manuscript that the reviewer  found imprecise, we would be happy to know so that we can improve the overall mathematical precision.

> ii. the lack of guidance as to how to interpret the 2D results for RNN training in general

Normal hyperbolic attractors are persistent in arbitrary dimensions. This is demonstrated for a higher dimensional example of the head direction network implementing a ring attractor. All perturbations in the vicinity of the original system lead to a bounded invariant manifold, for which we describe the Morse decomposition in terms of the fixed points and their connecting orbits. Saddle nodes are always connected to stable fixed points in the direction of their unstable manifold.

Reviewer comment (Theory):
There is a brief remark about bifurcations into chaos also causing exploding gradients; it is unclear whether this means that the attractors of the hidden states are never chaotic attractors.

Author response:
> Does this mean that the attractors of the hidden states are never chaotic attractors?

For 1 and 2 dimensional continuous attractors, small enough perturbations never lead to chaotic attractors, because the perturbed invariant manifold needs to be homeomorphic to the original continuous attractor and there can be no chaos in dimensions 1 or 2 (see for example the Poincaré–Bendixson theorem). For higher dimensional continuous attractors this is not guaranteed, however, the chaotic flow will be slow according to the theory.

Reviewer comment (Experiments):
While the authors mention that these theoretical insights can guide RNN training dynamics away from problematic bifurcations, it is unclear how that can be accomplished in practice, particularly in high-dimensional cases.

Author response:
> While the authors mention that these theoretical insights can be used to guide RNN training dynamics away from problematic bifurcations, it is unclear how that can be accomplished in practice, in high-dimensions.

Normal hyperbolic attractors will be persistent in arbitrary dimensions, even in PDEs. For ReLU RNNs, there is some literature on the necessary conditions for attractive and bounded solutions to exist, see for example Yi (2001). See conditions for a line attractors in linear threshold RNNs in Yu (2009), one just needs to rotate these line attractors so that they become bounded. Furthermore, conditions of the equilibria in more generality (line and plane attractors) of the 3D RELU network are presented in Xiang (2021) and parameters for bounded plane attractors for 4D tanh networks are discussed in Yu (2023). 

We will include these in the manuscript. However, for RNNs with arbitrary activation functions, a general theory is still lacking.

Reviewer comment (Theory):
It is unclear when RNN dynamics can be decomposed into slow-fast systems for which Fenichel duality can be applied, and when the input dynamics (discrete-time map) induce Fenichel-type attractors in the dynamics of the hidden state.

Author response:
> When can RNN dynamics be decomposed into slow-fast systems for which we can apply Fenichel duality? 

For continuous attractors this decomposition is always possible. We would kindly bring to the attention of the reviewer that Fenichel’s Invariant Manifold Theorem should not to be confused with Fenchel’s duality theorem (https://en.wikipedia.org/wiki/Fenchel%27s_duality_theorem).
For other systems, decomposability into fast and slow subsystems needs to be investigated on a case-by-case basis.

Reviewer comment (Theory):
The authors could expand on their observations about the loss landscape near the normally hyperbolic fixed point.

Author response:
> When does the input dynamics (discrete-time map) induce Fenichel-type attractors in the dynamics of the hidden state (may be the authors can expand a bit on their observations about the loss landscape near the normally hyperbolic fixed point).

We show some results of RNN trained on an integration task that show that bounded normally hyperbolic invariant sets (is this what the reviewer refers to as Fenichel-type attractors?) develop during learning and that even with (constant) input the system exhibits Fenichel-type attractor behavior, typically in the form of limit cycles and fixed points. These results are included in Supp.Sec. 12 (Supp.Fig. 20.

Reviewer comment (Theory):
First off, it's important to note that, as demonstrated in the study by Mikhaeil et al. (2022), when dealing with chaotic or unstable dynamics, gradients will always explode and **this issue cannot be solved by RNN architectural design or regularization and needs to be addressed in the training process**. So, the authors should consider this more precisely.

Author response:
In this paper we focus on the properties of continuous attractors that might be necessary for some specific tasks, and are not concerned with learning chaotic dynamics.
	Furthermore, unstable dynamics won’t be present in our proposed RNNs as we are concerned with bounded continuous attractors that will have a persistent attractive invariant manifold. We clarified this in the paper on page 3 at the beginning of Section 2.

Reviewer comment (Experiments):
The paper primarily focuses on a specific RNN variant, employing the ReLU activation function, 2-dimensional RNNs with specific parameter values outlined in equations (9) and (10). This specificity may limit the generalizability of its findings to higher dimensions, other architectural designs, and parameter configurations. In one particular parameter setting, the RNN exhibits 'well-behaved' dynamics, converging to a fixed point with bounded loss gradients. In such cases, established solutions (e.g., Hochreiter & Schmidhuber 1997; Schmidt et al., 2021) can be employed to prevent gradient vanishing. While this result is interesting, it's essential to recognize that it represents only one specific scenario and may not cover the broader spectrum of possibilities.

Author response:
We agree that the paper only covers one specific scenario (that of bounded invariant manifolds in RNNs) and may not cover the broader spectrum of possibilities. However, note that our theory is applicable to all RNNs and biological recurrent networks that can realize such invariant manifold structures. We believe that it is this generality of the theory that makes it relevant to the broader theoretical neuroscience and machine learning communities.

Reviewer comment (Theory):
Utilizing Fenichel’s invariant manifold theorem, the authors established a sufficient condition to ensure that RNNs implementing continuous attractors remain immune to exploding gradients. However, pleae note that a ReLU RNN, discrete- or continuous-time, is a piesewise linear (PWL) dynamical system, a subclass of piesewise smooth (PWS) dynamical systems, that is smooth everywhere except on some boundaries separating regions of smooth behavior. These boundaries, called switching manifolds or borders, divide the phase space into countably many regions. Many existing theories of smooth DS do not extend to PWS systems. In PWS systems the interaction of invariant sets with switching manifolds often gives rise to bifurcations, known as discontinuity-induced bifurcations, which are not observed in smooth systems. Moreover, in the PWS setting, for bifurcations containing non-hyperbolic fixed points, similar slow (center) manifold approaches can be applied, but only to part of phase space. However, bifurcations involving switching manifolds cannot be examined in the same manner, as there is no slow (center) manifold. Consequently, the applicability of Fenichel’s invariant manifold theorem to different parameter settings of the considered system (eq. (7)) may be limited or pose considerable challenges.

Author response:
Smoothness of the system under consideration is indeed a valid worry in general. However, for continuous piecewise linear (PWL) dynamical systems for our purposes of stability of the persistent invariant manifold, the theory is applied (ReLU RNNs are continuous). If the critical manifold is globally stable then the system is forward invariant in a neighbourhood of the critical manifold, and Fenichel’s invariant manifold theorem is applicable (Simons, 2018). We believe that the theory can be applied to PWL continuous systems with continuous attractors in high generality because of their global attraction property, see for example the analysis of the head direction model in Section 3.2. 
For remarks on the application of the fast-slow theory to PWLs, see Theorem 19.3.3 in Kuehn (2015). And in practice we see that the continuous attractor persists as an invariant manifold, see the bifurcation analysis both on the bounded line attractor network and the head direction representation network. For Filippov systems the situation is different because they are not continuous. For fast-slow analysis of Filippov systems, see Cardin (2013), however, note that these are outside the scope of our investigation. We included a remark on the applicability of Fenichel’s invariant manifold theorem to continuous piecewise linear (PWL) dynamical systems on page 4 in the second paragraph.
This means that even for PWL continuous dynamical systems, the normally hyperbolic invariant manifold that is the continuous attractor is going to persist. It will not maintain smoothness because $r<1$, but we don’t think that causes an issue for the (biological) systems that implement continuous attractors. If it is important that the invariant manifold persists in a smooth way, namely that the perturbed invariant manifold is $C^r$ then other activation functions need to be used.

Reviewer comment (Novelty):
Regarding the link between bifurcations and EVGP during training, it is essential to consider additional relevant literature. For instance, as demonstrated in (https://arxiv.org/pdf/2310.17561.pdf) specific bifurcations in ReLU-based RNNs are always associated with EVGP during training.

Author response:
We thank the reviewer for this suggestion and will incorporate it in the text as one way of finding maximal epsilon bifurcation parameters around continuous attractors. 
Our theory shows that for 1 and 2 dimensional continuous attractors, there are no exploding gradients for small enough perturbations, even though bifurcations do occur. The only possibility is the creation of chaotic orbits, which could still cause a problem. This paper however is not considering bifurcations involving chaotic behavior, so we don’t see the relevance of considering this part of the literature.  Note finally that while we acknowledge the relevance of this recent paper to our research, it was published after the submission of our manuscript.