Paper: Sufficient conditions for offline reactivation in recurrent neural networks (ICLR-2024)

Reviewer comment (Experiments):
It is unclear what is meant with tuning of the value of $\tau$. Is this for a fixed noise level? Or is it chosen as the optimal for all used noise levels for training?

Author response:
**Response:** We choose $\tau$ for the RNN based on the task to ensure it performs optimally, and it remains the same for all noise levels. The intention is to ensure that the network is faster than the inputs or else it does not react quick enough to new inputs, causing the activity to remain stuck and the performance to be poor. In practice, we do not have to tune $\tau$ very carefully, we set it to a reasonable value that ensures that the network's dynamics are fast enough to process the task's inputs and perform optimally. We clarify this point in the revised manuscript.

Reviewer comment (Experiments):
The demonstration contribution of this framework could benefit from some additional experiments.

Author response:
**Response:**  We apologize if this was unclear from the text. We have compared to training without noise and with added noise in quiescent state in Suppl. Fig. A.2e-f. We have shown in this experiment that these trajectories are erratic, and do not properly correct for lack of exploration arising from training without noise. In an additional experiment, we have shown that the point-to-point distances are much higher for these erratic trajectories than even waking behavior, indicating their implausibility.

It is true that we don't compare to more traditional ring attractor models, with which we have some key differences. The critical distinction is that our model recapitulates statistics of waking navigation, whereas networks with hand-tuned weights, for instance, would only be able to produce a uniform distribution over the attractor manifold in question.

Reviewer comment (Experiments):
Comparison to training without noise and added noise in quiescent state is missing.

Author response:
**Response:**  We apologize if this was unclear from the text. We have compared to training without noise and with added noise in quiescent state in Suppl. Fig. A.2e-f. We have shown in this experiment that these trajectories are erratic, and do not properly correct for lack of exploration arising from training without noise. In an additional experiment, we have shown that the point-to-point distances are much higher for these erratic trajectories than even waking behavior, indicating their implausibility.

It is true that we don't compare to more traditional ring attractor models, with which we have some key differences. The critical distinction is that our model recapitulates statistics of waking navigation, whereas networks with hand-tuned weights, for instance, would only be able to produce a uniform distribution over the attractor manifold in question.

Reviewer comment (Experiments):
Comparison to other ring attractor/head direction models is missing.

Author response:
**Response:**  We apologize if this was unclear from the text. We have compared to training without noise and with added noise in quiescent state in Suppl. Fig. A.2e-f. We have shown in this experiment that these trajectories are erratic, and do not properly correct for lack of exploration arising from training without noise. In an additional experiment, we have shown that the point-to-point distances are much higher for these erratic trajectories than even waking behavior, indicating their implausibility.

It is true that we don't compare to more traditional ring attractor models, with which we have some key differences. The critical distinction is that our model recapitulates statistics of waking navigation, whereas networks with hand-tuned weights, for instance, would only be able to produce a uniform distribution over the attractor manifold in question.

Reviewer comment (Theory):
Some of the implications of the proposed framework require more justification.

Author response:
**Response:** We apologize if our claims were not clear. In our mathematical results, we are only claiming that the steady-state distribution of quiescent activity matches that of waking activity. Specifically, we do not make any comment on the sequential dynamics of quiescent trajectories themselves. Thus, for reverse replay, the only point we make is that for true diffusive rehearsal, reversed trajectories are as probable as forward trajectories. In additional experiments with our RNNs on the head direction task (Suppl. Fig. A.5), we biased the waking moment-to-moment transition structure to be counter-clockwise. We found that quiescent trajectories partially recapitulated the transition structure of active phase trajectories, and maintained their bearing for longer periods, resembling real neural activity more closely. The biased velocities were also reflected during quiescence, although less clearly than waking. We do not yet have theoretical justification for this. However, we note that reversals in the trajectories still remain possible and do occur (Suppl. Fig. A.5).

Reviewer comment (Evaluation):
The conclusion that reverse replay could be explained by diffusive reactivations seems like a stretch and should be substantiated better.

Author response:
**Response:** We apologize if our claims were not clear. In our mathematical results, we are only claiming that the steady-state distribution of quiescent activity matches that of waking activity. Specifically, we do not make any comment on the sequential dynamics of quiescent trajectories themselves. Thus, for reverse replay, the only point we make is that for true diffusive rehearsal, reversed trajectories are as probable as forward trajectories. In additional experiments with our RNNs on the head direction task (Suppl. Fig. A.5), we biased the waking moment-to-moment transition structure to be counter-clockwise. We found that quiescent trajectories partially recapitulated the transition structure of active phase trajectories, and maintained their bearing for longer periods, resembling real neural activity more closely. The biased velocities were also reflected during quiescence, although less clearly than waking. We do not yet have theoretical justification for this. However, we note that reversals in the trajectories still remain possible and do occur (Suppl. Fig. A.5).

Reviewer comment (Presentation):
Reactivations such as replay are not the type of reactivations that are found in the networks in the paper (this is admitted in the paper) even though the introduction is discussing those for a large part.

Author response:
**Response:** We apologize if our claims were not clear. In our mathematical results, we are only claiming that the steady-state distribution of quiescent activity matches that of waking activity. Specifically, we do not make any comment on the sequential dynamics of quiescent trajectories themselves. Thus, for reverse replay, the only point we make is that for true diffusive rehearsal, reversed trajectories are as probable as forward trajectories. In additional experiments with our RNNs on the head direction task (Suppl. Fig. A.5), we biased the waking moment-to-moment transition structure to be counter-clockwise. We found that quiescent trajectories partially recapitulated the transition structure of active phase trajectories, and maintained their bearing for longer periods, resembling real neural activity more closely. The biased velocities were also reflected during quiescence, although less clearly than waking. We do not yet have theoretical justification for this. However, we note that reversals in the trajectories still remain possible and do occur (Suppl. Fig. A.5).

Reviewer comment (Evaluation):
The fact that training without noise resulted in erratic output trajectories might be explained by the statistics of the input.

Author response:
**Response:** We thank the reviewer for this question. Our data generation protocol is identical to influential prior work (Sorscher et al., 2019; Banino et al., 2018) which generate rat motion trajectories based on the model described in yet another influential work (Erdem & Hasselmo, 2012). As our work is consistent with these prior works and uses the same data generation as Sorscher et al., 2019, we believe it reflects natural animal behavior.

We would like to point out that the trajectories arising from noisy reactivations after noiseless training are not 'exploring' more of the state space, they are truly discontinuous and erratic, and violate the actual map that was formed. Thus, it is hard to see how these could be useful. This is evident from Suppl. Fig. A.3, where we see that the distances between consecutive points in the noisy quiescent trajectories with noiseless training is much higher than those for all other conditions including active behavior.

We also carried out an experiment with the head direction task to comment more on the trajectory-level statistics. We trained our networks solely on counter-clockwise trajectories for the head direction task (Suppl. Fig. A.5) and found that the reactivations recapitulated the transition structure of active phase trajectories. They also maintained their bearing for longer periods, resembling real neural activity more closely. The biased velocities were reflected in quiescent trajectories, but less clearly than during waking. This shows that trajectory level features of active behavior are reflected during quiescence.

Sorscher et al. A unified theory for the origin of grid cells through the lens of pattern formation. Neural Information Processing Systems (2019).

Banino et al. Vector-based navigation using grid-like representations in artificial agents. Nature 557 (2018): 429 - 433.

Erdem & Hasselmo. A goal-directed spatial navigation model using forward trajectory planning based on grid cells. European Journal of Neuroscience, 35: 916-931. 2012.

Reviewer comment (Evaluation):
It would be good to compare the statistics of state sequences and reactivation sequence on the level of the sequences themselves.

Author response:
**Response:** We thank the reviewer for this question. Our data generation protocol is identical to influential prior work (Sorscher et al., 2019; Banino et al., 2018) which generate rat motion trajectories based on the model described in yet another influential work (Erdem & Hasselmo, 2012). As our work is consistent with these prior works and uses the same data generation as Sorscher et al., 2019, we believe it reflects natural animal behavior.

We would like to point out that the trajectories arising from noisy reactivations after noiseless training are not 'exploring' more of the state space, they are truly discontinuous and erratic, and violate the actual map that was formed. Thus, it is hard to see how these could be useful. This is evident from Suppl. Fig. A.3, where we see that the distances between consecutive points in the noisy quiescent trajectories with noiseless training is much higher than those for all other conditions including active behavior.

We also carried out an experiment with the head direction task to comment more on the trajectory-level statistics. We trained our networks solely on counter-clockwise trajectories for the head direction task (Suppl. Fig. A.5) and found that the reactivations recapitulated the transition structure of active phase trajectories. They also maintained their bearing for longer periods, resembling real neural activity more closely. The biased velocities were reflected in quiescent trajectories, but less clearly than during waking. This shows that trajectory level features of active behavior are reflected during quiescence.

Sorscher et al. A unified theory for the origin of grid cells through the lens of pattern formation. Neural Information Processing Systems (2019).

Banino et al. Vector-based navigation using grid-like representations in artificial agents. Nature 557 (2018): 429 - 433.

Erdem & Hasselmo. A goal-directed spatial navigation model using forward trajectory planning based on grid cells. European Journal of Neuroscience, 35: 916-931. 2012.

Reviewer comment (Evaluation):
In terms of optimality of exploration erratic trajectories might be more optimal to fully explore, see for example McNamee (2021). But see also Supplementary Figure A.1 (a) vs (e) that seems to show that a network that has been trained without noise (e) explores a bigger part of the state space than a network that has been trained in the presence of noise (a).

Author response:
**Response:** We thank the reviewer for this question. Our data generation protocol is identical to influential prior work (Sorscher et al., 2019; Banino et al., 2018) which generate rat motion trajectories based on the model described in yet another influential work (Erdem & Hasselmo, 2012). As our work is consistent with these prior works and uses the same data generation as Sorscher et al., 2019, we believe it reflects natural animal behavior.

We would like to point out that the trajectories arising from noisy reactivations after noiseless training are not 'exploring' more of the state space, they are truly discontinuous and erratic, and violate the actual map that was formed. Thus, it is hard to see how these could be useful. This is evident from Suppl. Fig. A.3, where we see that the distances between consecutive points in the noisy quiescent trajectories with noiseless training is much higher than those for all other conditions including active behavior.

We also carried out an experiment with the head direction task to comment more on the trajectory-level statistics. We trained our networks solely on counter-clockwise trajectories for the head direction task (Suppl. Fig. A.5) and found that the reactivations recapitulated the transition structure of active phase trajectories. They also maintained their bearing for longer periods, resembling real neural activity more closely. The biased velocities were reflected in quiescent trajectories, but less clearly than during waking. This shows that trajectory level features of active behavior are reflected during quiescence.

Sorscher et al. A unified theory for the origin of grid cells through the lens of pattern formation. Neural Information Processing Systems (2019).

Banino et al. Vector-based navigation using grid-like representations in artificial agents. Nature 557 (2018): 429 - 433.

Erdem & Hasselmo. A goal-directed spatial navigation model using forward trajectory planning based on grid cells. European Journal of Neuroscience, 35: 916-931. 2012.

Reviewer comment (Evaluation):
The claim that even with the addition of noise the failure of exploration could not be corrected (page 8, middle) should be better substantiated with a comparison based on distributions rather than just example trajectories (currently Suppl.Fi. A.1e-f is used as justification).

Author response:
**Response:** We thank the reviewer for this useful suggestion. We have added Suppl. Fig. A.3 which shows exactly this by comparing the distribution of distances between successive points in a trajectory across conditions. We find that this distance is much higher for quiescent trajectories with added noise without noise in training, than even waking behavior. This indicates simply the implausibility of these trajectories.

Reviewer comment (Evaluation):
How is the bias exactly defined for figure 2?

Author response:
**Response:**

1. The bias is defined by adding a drift term towards an anchor point, such as the center of the environment. When we generate a random trajectory to train on, we generate a series of random turns and movements for each timestep. The bias/drift is defined as the displacement vector between the current location and the anchor point, scaled down by a constant. Thus, for each timestep, we have:
    ```
    new_position = old_position + drift_constant * (anchor_point - old_position) + random_movement
    ```
    This ensures that the positions occupied are values close to the anchor point.

Reviewer comment (Presentation):
In Figure 2, what does it mean that a random network is trained and tested in the presence of noise (as would be implied in the column $\text{R}^\sigma$)?

Author response:
**Response:**

3. Random networks are untrained, and testing in the presence of noise means that noise is added to the activity during the quiescent phase. We thank the reviewer for pointing us to the incorrect framing of the caption. We have modified the caption to avoid misunderstandings in the updated draft.

Reviewer comment (Evaluation):
How do different trained networks compare to each other in terms of the reactivation statistics?

Author response:
**Response:** Different networks would have different individual neurons, so it is not possible to directly compare the activity of different trained networks. However, different networks trained on the same task statistics (i.e., unbiased or biased) have similar attractor manifolds, and show similar behavioral outputs during quiescence. For example, two different RNNs trained on the biased navigation task will have similar steady-state distributions of output trajectories, since the task statistics are identical, with the only differences arising from samples of randomly generated training data (the individual samples may differ but the distribution is identical) and the values of noise sampled.

Reviewer comment (Theory):
Is it truly a ring attractor that solves the angular integration task? On the given time scale it could be the case that the solution is provided by a line attractor that gets mapped to the ring by $D$.

Author response:
**Response:** We indeed believe that a ring attractor is required to solve the task. Our network learned a ring attractor without issue, and without having to enforce this attractor structure in its dynamics. While a line attractor could provide a local approximation of the ring attractor, it would be unable to match the nonlinear topology of the full ring---the output has periodic boundary conditions, and a linear decoder could not warp a line attractor such that it has periodic boundary conditions. We initialized randomly our initial head bearing, so the local approximation would have to be accurate across the whole manifold, irrespective of the time scale.

Reviewer comment (Theory):
In Equation (17) is the optimum unique?

Author response:
**Response:** This is the minimum norm solution, but in general, $\mathbf{D}$ is mapping a higher-dimensional activity space to a lower-dimensional output space, which means that there are extra degrees of freedom in the RNN dynamics that are not constrained by the task. So this optimum is not unique.

We believe we have addressed all of the reviewer's concerns, and would be grateful if the reviewer would consider raising their score. We would be happy to discuss further if the reviewer has any additional concerns. We eagerly await the reviewer's response.

Reviewer comment (Theory):
The paper is potentially confusing to some readers in that the presentation (and the numerical experiments if I have followed) regard training a network with a fixed nonlinearity and trainable weight parameters (i.e. a typical RNN) but the theoretical analysis considers not a fixed nonlinearity, but rather the space of all possible dynamics that could govern the system. I suspect this may misdirect some readers regarding the basic argument (possibly me as well).

Author response:
**Response:** We apologize if this was not clear. The idea is that the RNNs trained optimally are approximating the ground-truth optimal solution, given by the greedily optimal dynamics (Section 2.3). It is indeed reasonable for an RNN to implement Eq. 18—according to the literature on RNNs being universal function approximators (Funahashi & Nakamura, 1993). We are thus assuming that our trained RNNs are near the optimum, and our empirical results indicate that they are. Our theoretical results do not make any assumptions about the architecture of the RNN (in fact we have added simulations with GRUs, Suppl. Fig. A.4), only that it is trained to be optimal. Indeed we see the Langevin sampling during quiescence leading to the distributional match. It is certainly possible for suboptimal (or very small) RNNs to fail to align with the theory, the theory only applies for very well-trained networks that are performing near-optimally such as our networks.

Funahashi & Nakamura. Approximation of dynamical systems by continuous time recurrent neural networks. Neural Networks 6 (1993): 801-806.

Reviewer comment (Theory):
Training a regular RNN on the cost function, L, does not guarantee anything about the cost function L_upper, so one must make additional assumptions about the structure of the network dynamics, namely something akin to demanding that the space of permissible dynamics admits the derived minimum of L_upper. In the case of the numerical experiments, the authors use ReLU and decompose the input to the nonlinearity into a recurrent part and a signal integration part, which should approximately allow for the decomposition that leads to L_upper. One needs to add an argument that this decomposition is reasonable for certain classes of RNNs. For example, if the network is in the active regime, the ReLU doesn't contribute much and then the composition amounts to considering the activity updates from the recurrent part and the input part separately. One could make a similar argument about sigmoids. In short, the authors should be direct about the assumptions on the RNN necessary for their theory.

Author response:
**Response:** We apologize if this was not clear. We would like to state that our mathematical analysis is a theory for high capacity networks capable of legitimately approximating the optimal solution. RNNs are capable of doing this, as they are universal function approximators (Funahashi & Nakamura, 1993) as long as the activation function is nonlinear. We would like to draw attention to a new experiment we have included with GRUs thanks to the reviewer's suggestion, to highlight the universality of our results. We found that our results do hold even with continuous-time GRUs (Suppl. Fig. A.4), showing that these reactivation phenomena are not unique to a particular network architecture or activation function. We add a discussion about this important point in the revised manuscript.

Funahashi & Nakamura. Approximation of dynamical systems by continuous time recurrent neural networks. Neural Networks 6 (1993): 801-806.

Reviewer comment (Theory):
Regarding the quiescent state, the authors are perhaps too cavalier with the fact that, strictly speaking, the noise term must change in order to guarantee replay. As presented, it feels like a weakness that is not adequately addressed. The authors should either supply 1) a reason to believe this sort of change happens in real networks or 2) a demonstration that the deviations are not sufficient to qualitatively change the main result. There is also the possibility that this represents an opportunity for the authors, since deviations from perfect replay if the noise does not change amount to a prediction, and one which may have computational advantages (e.g. facilitating exploration in terms of a planning algorithm).

Author response:
**Response:** We apologize if there has been a misunderstanding, but the increase in noise variance is **not required** to see reactivation. It is only required for an **exact** distributional match for the Langevin sampling. We demonstrate in Suppl. Fig. A.1 (in the updated manuscript) that reactivation occurs and matches the waking behaviour even when the noise variance remains the same during quiescence. The noise variance merely behaves like a temperature parameter for the sampling, reactivation is not precluded if it stays at the same level. As the reviewer points out, there may indeed be related computational advantages which future work could explore. We thank the reviewer for raising this point and the suggestion for the additional demonstration, which we believe improves our work.

Reviewer comment (Writing):
Last sentence of the introduction: the term "ecologically" should be "ethnologically".

Author response:
**Response:** We thank the reviewer for the suggestion, we have modified this sentence to use "ethologically" instead of "ecologically" in the updated draft.

Reviewer comment (Experiments):
Does this work for more general RNNs (more general nonlinearities, interactions between recurrence and signal integration, etc.)?

Author response:
**Response:** We do not believe that there is any principled reason why our analyses wouldn't extend to other RNNs. We thank the reviewer for this suggestion, and we decided to check this experimentally with continuous-time GRUs. We found that our results do hold (Suppl. Fig. A.4), showing that these reactivation phenomena are not unique to a particular network architecture or activation function. While our theory does not preclude them (Funahashi & Nakamura, 1993; Maheswaranathan et al., 2019), in practice, GRUs were more sensitive to increases in quiescent noise, and other activation functions would require more hyperparameter tuning for the network's optimal performance.

Funahashi & Nakamura. Approximation of dynamical systems by continuous time recurrent neural networks. Neural Networks 6 (1993): 801-806.

Maheswaranathan et al. Universality and individuality in neural dynamics across large populations of recurrent networks. Advances in neural information processing systems 2019 (2019): 15629-15641.

Reviewer comment (Theory):
Is there reason to believe the noise distribution changes in the quiescent state?

Author response:
**Response:** To our knowledge, there is no evidence that this must occur. However, we would like to reiterate that our main results hold even without the change in noise variance, as stated previously (Suppl. Fig. A.4). Noise only acts like a temperature parameter on the distribution.

Reviewer comment (Theory):
perhaps this is not in the scope of the paper, but the authors do not provide any theoretical/numerical support for the functional benefit of offline reactivation. The authors mention other works which demonstrate that it may aid the formation of long-term memories, schema, planning. Do the theoretical/numerical results in this paper support these possibilities, or any one in particular?

Author response:
**Response:** This is indeed an interesting research question. However, as the reviewer themselves point out, we believe this is out of scope for our current work---our scope here is to explain theoretically the emergence of reactivation and justify this with experiments, not to identify its functional benefits. It is nevertheless an important direction for future work. Many papers have talked about the functional benefits of offline reactivation in other contexts (Mnih et al. 2015; Hayes et al. 2019; Hayes & Kanan 2022), but the reviewer is correct in noting that we would have to concretely demonstrate the benefits of this particular form of reactivation. We acknowledge that this is an immediate direction to be explored and are revising our discussion to discuss the implications of our results in this context.

Mnih et al. Human-level control through deep reinforcement learning. Nature 518, 529–533, 2015.

Hayes et al. Memory efficient experience replay for streaming learning. International Conference on Robotics and Automation (ICRA), pp. 9769–9776. IEEE, 2019.

Hayes & Kanan. Online continual learning for embedded devices. arXiv preprint arXiv:2203.10681, 2022.

Reviewer comment (Theory):
Relatedly, a key feature of this study is that the RNNs considered are noisy during awake states. It's unclear to me exactly why this is necessary. The authors "found that even trajectories generated by networks that were not trained in the presence of noise, and also were not driven by noise in the quiescent phase, still generated quiescent activity distributions that corresponded well to the active phase distributions"; is the point that there's more space exploration with noise, giving rise to a functional benefit? Or that somehow the introduction of noise better captures observed neural data?

Author response:
**Response:** Indeed, as the reviewer points out, the noise leads to more exploration. Noise has been shown to improve RNN training (Qin & Vučinić, 2018). Furthermore, neural activity in the brain is noisy (Destexhe & Rudolph-Lilith, 2012), for example, in the probabilistic nature of synaptic transmission, and in other ways.

It is critical to note that the match in distributions is across multiple trajectories (multiple initializations), and so does not entirely capture the properties of individual trajectories, such as the amount of exploration.

We thus carried out an experiment to show the amount of exploration of these noiseless trajectories. Indeed we found that individual trajectories do not explore the space well, and get stuck (Suppl. Fig. A.2c-d). Adding noise during quiescence alone does not fix this problem---it makes the trajectories erratic and implausible (Suppl. Fig. A.2e-f), with much higher point-to-point distances than even waking behavior (Suppl. Fig. A.3). So individual trajectories from noiseless networks do a worse job relative to the noisy trained networks. Noisy training stabilizes noisy quiescent activity, which in turn explores more of the task manifold than noiseless quiescent activity. We clarify this point in the revised manuscript.

Qin & Vučinić. Training Recurrent Neural Networks against Noisy Computations during Inference. 2018 52nd Asilomar Conference on Signals, Systems, and Computers (2018): 71-75.

Destexhe & Rudolph-Lilith. Neuronal Noise. Springer, 2012.

Reviewer comment (Theory):
I found the link between the theoretically optimal solution for update dynamics and the actual application of RNNs somewhat unclear. Are we to take that (presumably backprop-trained) RNNs employ update dynamics similarly to the theoretical solution; is it biologically reasonable for e.g. equation 18 to be implement by an RNN? If so, is it possible to show that they do; e.g. by comparing these recurrent inputs to the optimal solution? If not, are the experimental results relevant to the theory?

Author response:
**Response:** We apologize if this was not clear. The idea is that the RNNs trained optimally (following any optimization scheme) are approximating the ground-truth optimal solution, given by the greedily optimal dynamics (Section 2.3). It is indeed reasonable for an RNN to implement Eq. 18---according to the literature on RNNs being universal function approximators (Funahashi & Nakamura, 1993). We did not show that the network dynamics explicitly align with the optimal solution, though our empirical results indicate that they do. Indeed we see the Langevin sampling during quiescence leading to the distributional match. It is certainly possible for suboptimal (or very small) RNNs to fail to align with the theory, the theory only applies for very well-trained networks that are performing near-optimally such as our networks. We clarify these points in the revised manuscript.

Funahashi & Nakamura. Approximation of dynamical systems by continuous time recurrent neural networks. Neural Networks 6 (1993): 801-806.

Reviewer comment (Novelty):
Relatedly, perhaps this is overly harsh, but I also do not find the result that RNNs without stimuli visit similar states as with stimuli very surprising.

Author response:
**Response:** We believe that there may have been a misunderstanding---our claim is much stronger than merely saying that similar states are visited in the absence of stimuli. Instead, we prove and demonstrate that the networks visit the same states with the same, i.e., **identical steady-state distribution** during quiescence as active behavior. During our biased and unbiased tasks, the same path integrating continuous attractor could theoretically solve both tasks, but then rehearsal would be the same in both conditions. We see that the networks learn different solutions for the biased and unbiased tasks, and that these different solutions produce rehearsals that reflect the task statistics. This is non-trivial, and demonstrating this is, we believe, our main novel contribution. We further clarify this point in the revised manuscript.

Reviewer comment (Theory):
The authors suggest an interesting discrepancy in experimental prediction for their noise-based theory of offline reactivation and generative modeling; specifically, that "while generative models necessarily recapitulate the moment-to-moment transition statistics of sensory data, our approach only predicts that the stationary distribution will be identical". I am confused by this statement and not sure if it is true. Is that to say that the proposed model would not encode sensory transitions but would simply replicate the overall probability distribution of sensory states? I would find this surprising for an RNN whose recurrent weights are trained on sensory data; that is, I would suppose that the RNN weights would themselves somehow capture transition statistics of the task variables (e.g. maybe this is valuable as a kind of denoising effect on noisy observations). Perhaps I am wrong though.

Author response:
**Response:** The reviewer is indeed correct that it is **possible** for our networks to capture the transition dynamics (and this is what happens in practice), it is just that our theory does not comment on the dynamics of reactivations. In a new experiment with training solely on counter-clockwise trajectories for the head direction task (Suppl. Fig. A.5), we found that the reactivations recapitulated the transition structure of active phase trajectories, and maintained their bearing for longer periods, resembling real neural activity more closely. This would indeed appear to be due to the RNN weights capturing the transition structure, but is not a prediction of our theory. The biased velocities were reflected during quiescence, but less clearly than during waking. However, reversals in the trajectories still occurred (Suppl. Fig. A.5b).

Reviewer comment (Writing):
In section 2.1 it may be relatively obvious that p denotes probability, but I would still clarify this.

Author response:
**Response:** We thank the reviewer for the suggestion, we have updated this in the draft.

Reviewer comment (Writing):
In equation 15 it's a bit odd to provide a new definition for L_{noise} having just defined it previousply with the D term involved.

Author response:
**Response:** We have used the `\equiv` ($\equiv$) symbol to denote an equivalence between the location of minima for $\mathcal{L}_{noise}$ (which was defined previously) and the RHS of Eq. 15. We apologize if this was not clear. We have updated the text above to clearly state the equivalence, and modified the equation to be more explicit about the equivalence in location of minima.

Reviewer comment (Writing):
the delta t is given as 0.02. What's the unit, seconds?

Author response:
**Response:** Yes, the unit is indeed seconds. We have updated the draft to add the missing unit, and thank the reviewer for the observation.

Reviewer comment (Novelty):
A number of relevant studies were not cited and discussed. This includes work using RNNs and attractor dynamics to model the replay and theta sequences in the hippocampus, e.g., Hopfield, John J. "Neurodynamics of mental exploration." Proceedings of the National Academy of Sciences 107.4 (2010): 1648-1653; Kang, Louis, and Michael R. DeWeese. "Replay as wavefronts and theta sequences as bump oscillations in a grid cell attractor network." Elife 8 (2019): e46351; Chu, Tianhao, et al. "Firing rate adaptation affords place cell theta sweeps, phase precession and procession." bioRxiv (2022): 2022-11. The paper also misses several pieces of work in training RNNs to study the grid cells systems and HD systems, e.g., Cueva, C. J., & Wei, X. X. (2018). Emergence of grid-like representations by training recurrent neural networks to perform spatial localization. ICLR; Uria, B., Ibarz, B., Banino, A., Zambaldi, V., Kumaran, D., Hassabis, D., Barry, C. and Blundell, C., (2020). The spatial memory pipeline: a model of egocentric to allocentric understanding in mammalian brains. BioRxiv, pp.2020-11; Cueva, C.J., et al (2020). Emergence of functional and structural properties of the head direction system by optimization of recurrent neural networks. ICLR.

Author response:
**Response:** We thank the reviewer for pointing us to these works. We have cited these in our updated draft. We would also like to stress that the focus of our work is to demonstrate that reactivations are statistical reproductions of waking activity (i.e. the steady-state distribution of reactivations is **identical** to that of waking activity), and not just observed as a result of our networks learning continuous attractors. We formalize and prove this mathematically, and also show this with experiments. We apologize if this was not clear. We hope that our comments here and in the updated draft improve on this.

Reviewer comment (Novelty):
The main results are not surprising after considering what we know so far on this topic. Prior work has shown that training RNNs to perform path integration or angular integration tasks leads to networks that exhibit attractor dynamics similar to continuous attractor models. Furthermore, it is well known that attractor dynamics can perform denoising. Thus, it is not surprising that RNNs trained on these previously studied tasks can perform denoising. It should also be noted that prior work has further characterized the rate of diffusion along the low-dimensional manifold, e.g., see Fig. 6 of the paper: Burak, Y. and Fiete, I.R., 2009. Accurate path integration in continuous attractor network models of grid cells. PLoS computational biology, 5(2), p.e1000291.

Author response:
**Response:** We apologize if the main message appeared to be the related to the RNNs learning continuous attractors and performing denoising, this is not our intention. In fact, our very starting point is the recognition that continuous attractor networks emerge with training, it is not one of our major claims. The central finding and novel contribution of our manuscript is to explore the conditions under which reactivation, driven by noise, matches the statistics of waking activity.

With respect to denoising, we believe that one of our novel contributions is to show theoretically how denoising comes to be, and gives rise to statistically faithful reactivations as described below, providing a principled theoretical understanding. We would like to reiterate that our focus is not to show that denoising occurs, although we build on works that explore this idea (Section 2.2--2.3).

More precisely, our novel contribution is to prove and demonstrate that the steady-state distribution of reactivations is **identical** to that of waking activity, under a well-defined and minimal set of conditions. While this mechanism is related to denoising, which will occur along attractor manifolds, our contribution is to demonstrate that optimal denoising under different task statistics will show rehearsal patterns that reflect those different task statistics. This is the key novel contribution, which we believe is non-trivial in and of itself. We make sure to clearly communicate our findings in the revised manuscript (Section 2.4).

Reviewer comment (Evaluation):
As described in the paper, the trained network follows diffusion dynamics when the stimulus is turned off. This is naturally expected because of the diffusion of network state on a low-dimensional manifold, which would result in Brownian-motion-like trajectories. Furthermore, the replay trajectories observed in the hippocampus are typically described by a systematic drift towards one direction (e.g., in linear or circular tracks), not diffusive dynamics. Can the authors be more explicit about the experimental data they were modeling?

Author response:
**Response:** We apologize if this was not clear, but as stated previously, the focus of our mathematical analysis is on the resulting steady-state distributions of quiescent activity, rather than the transient moment-to-moment transition statistics. While diffusion-like motion on the attractor manifold has been described previously, to our knowledge it has not been formally or theoretically explored how this gives rise to statistically faithful reactivations. Moreover, it is not clear whether this reactivation phenomenon can be well-described by a formal diffusion process, and what are the conditions under which autonomous network dynamics would lead to such activity.

Although experts might find the presence of reactivation intuitive, our work formalizes this phenomenon and provides critical details about the statistics of reactivation distributions. We kindly note that our work gains importance specifically by formalizing an observation that seems intuitive, as it now provides a theoretical grounding to study reactivation.

While we have not modeled any specific experimental data for circular and linear tracks, our results do not state anything specific about the sequential dynamics on the manifold, only that the steady-state distribution is matched to waking statistics. However, in a new experiment with the head direction task (Suppl. Fig. A.5), we biased the waking moment-to-moment transition structure to be counter-clockwise. We found that quiescent trajectories partially recapitulated the transition structure of active phase trajectories, and maintained their bearing for longer periods, resembling real neural activity more closely. The biased velocities were also reflected during quiescence, although less clearly than waking. However, reversals in the trajectories still occurred.

Reviewer comment (Theory):
The authors stated “we found that even trajectories generated by networks that were not trained in the presence of noise, and also were not driven by noise in the quiescent phase, still generated quiescent activity distributions that corresponded well to the active phase distributions.” This seems to argue against the denoising dynamics and makes the usefulness of the mathematical analysis questionable.

Author response:
**Response:** While we do observe this, it is critical to note that the match in distributions is across multiple trajectories (multiple initializations), and so does not entirely capture the properties of individual trajectories, such as the amount of exploration.

We thus carried out an experiment to show the amount of exploration of these noiseless trajectories. Indeed we found that individual trajectories do not explore the space well, and get stuck (Suppl. Fig. A.2c-d). Adding noise during quiescence alone does not fix this problem---it makes the trajectories erratic and implausible (Suppl. Fig. A.2e-f), with much higher point-to-point distances than even waking behavior (Suppl. Fig. A.3). So individual trajectories from noiseless networks do a worse job relative to the noisy trained networks. Noisy training stabilizes noisy quiescent activity, which in turn explores more of the task manifold than noiseless quiescent activity.

Reviewer comment (Theory):
The mathematical analysis relied on a set of approximations and assumptions, which were not well justified.

Author response:
**Response:** We thank the reviewer for the suggestion, and we have added a section to the Supplementary Material to clarify the key assumptions for our theory. We provide the assumptions and explanations here:
1. We consider discrete-time approximations of noisy continuous-time RNNs. Using continuous-time RNNs is a common practice in the literature, and we use its discrete-time approximation to be in line with our implementation.
2. The network must be performing some variant of path integration, by integrating change-based information about environmental state variables to some function of these variables. This condition is often met in natural settings, such as spatial navigation.
3. We assume that the inputs to the network are drawn from a stationary distribution, which would translate to assuming that the effects of initial conditions on state occupancy statistics are ignored, and also assuming that the behavioral policy remains constant throughout time.
4. We consider greedy optimization of the loss at every timestep. Greedy optimization is a sensible way of partitioning effort across time in this task: the agent does the best that it can on each timestep, assuming that at each previous timestep the best possible job has been done. In the absence of noise the greedily optimal solution is equivalent to path integration, which is also a **globally** optimal solution.
5. We assume that the network is performing optimally in the presence of noise. In practice, we train our networks until their loss reaches very low, near-zero values.
6. We assume that our network dynamics can be decomposed into two terms with different functional dependencies. The first term depends only on the activity while the second term depends on all original dependencies—the activity, the state and the change-based inputs.
7. For the quiescent state, we assume that the change-based sensory inputs are zero. This is reasonable because for tasks like those we have considered, which involve integrating self-motion cues, $\frac{\mathrm{d}s(t)}{\mathrm{d}t}$ must be zero during periods of quiescence like sleep, where the animal is not moving and hence doesn't receive sensory inputs associated with self-motion.
8. While we assume that the noise variance is doubled during quiescence to show exact equivalence with Langevin sampling, this is by no means necessary to witness reactivation (Suppl. Fig. A.4). This noise variance is equivalent to a temperature parameter for the sampling.

RNNs trained optimally (following any optimization scheme) are approximating the ground-truth optimal solution, given by the greedily optimal dynamics (Section 2.3). It is indeed reasonable for an RNN to implement Eq. 18---according to the literature on RNNs being universal function approximators (Funahashi & Nakamura, 1993). Our empirical results indicate that the network's dynamics align with this solution. Indeed we see the Langevin sampling during quiescence leading to the distributional match. It is certainly possible for suboptimal (or very small) RNNs to fail to align with the theory, the theory only applies for very well-trained networks that are performing near-optimally such as our networks. Overall, we believe our assumptions are well-justified, and the conclusions of the theory are indeed applicable to our numerical experiments as evidenced by our results.

Funahashi & Nakamura. Approximation of dynamical systems by continuous time recurrent neural networks. Neural Networks 6 (1993): 801-806.

We hope that our response and updates to the draft have addressed all of the reviewer's concerns, and would be grateful if the reviewer would consider raising their score. If the reviewer has any additional questions, we would be happy to discuss further. We eagerly await the reviewer's response.

Reviewer comment (Evaluation):
Would it be possible to discuss more explicitly how their results are connected or supported by the empirical data? In particular, how is the “diffusive reactivation” related to neural data?

Author response:
We thank the reviewer for going through our responses and revised manuscript. We appreciate the reviewer's detailed and helpful suggestions that have helped us considerably improve our work. We apologize that our discussions of some relevant work and connections to neuroscience data were incomplete. Thanks to the reviewer's suggestions, we have uploaded a new revision with some key changes:

> while the relevant references were added to the revised version, generally there is a lack of appropriate discussion of these studies in the context.

We have discussed the relevant works suggested by the reviewer in greater detail, in the context of our work. Here are specific changes we have made to improve our discussion (italicized portions indicate key changes):

* **Introduction, paragraph 3: discussing previous 'emergent' modeling approaches**: [...] local synaptic plasticity mechanism (Litwin-Kumar & Doiron, 2014; Theodoni et al., 2018; Haga & Fukai, 2018; Asabuki & Fukai, 2023; _Hopfield, 2010). Other approaches have modeled replay and theta sequences in the hippocampus as emergent consequences of firing rate adaptation (Chu et al., 2023) or input modulation (Kang & DeWeese, 2019) in continuous attractor network models._ [...]

* **Introduction, paragraph 4: discussing works training RNNs to study grid/HD cells**: We trained continuous-time recurrent neural networks (RNNs) to optimally integrate and track perceptual variables based on sensations of change (angular velocity, motion through space, etc.), in the context of two ethologically relevant tasks: spatial navigation and head direction integration. _Critically, training in this manner has been shown to produce grid cell (Cueva & Wei, 2018; Sorscher et al., 2019) and head direction cell representations (Cueva et al., 2020; Uria et al., 2020), which correspond to neural systems in which reactivation phenomena have been observed (Gardner et al., 2022; Peyrache et al., 2015)._ We see that these networks exhibit reactivation during quiescent states [...]

* **Discussion, paragraph 2:** Previous models have been able to reproduce these reactivation dynamics, _by embedding a smooth attractor in the network’s recurrent connectivity along which activity may diffuse during quiescence (Burak & Fiete, 2009; Khona & Fiete, 2022)._ Similarly, we identified attractors in our trained networks’ latent activation space [...] beyond previous studies, we were able to show that the statistics of reactivation in our trained networks mimicked the statistics of activity during waking behavior, and that manipulation of waking behavioral statistics was directly reflected in offline reactivation dynamics (Fig. 2). Thus, our work complements these previous studies by providing a mathematical justification for the emergence of reactivation dynamics in terms of optimal task performance.

> the connection to empirical research/ neuroscience data remains to be strengthened. I don’t feel this point was appropriately addressed in the revised version.

We thank the reviewer for this suggestion, and we have updated our manuscript to discuss this in greater detail. In particular, we draw attention to the following:

* **Numerical Experiments, Head Direction Estimation (last paragraph): discussing connections to experimental data:** To explore whether reactivation dynamics also recapitulate the moment-to-moment transition structure of waking activity [...] quiescent trajectories partially recapitulated the transition structure of active phase trajectories, and maintained their bearing for longer periods, _resembling real neural activity more closely. The biased velocities were reflected during quiescence, but less clearly than during waking. However, reversals in the trajectories still occurred. These results demonstrate that the type of ‘diffusive’ rehearsal dynamics explored by our theory are still able to produce the temporally correlated, sequential reactivation dynamics observed in the head direction system (Peyrache et al., 2015)._

* In the **discussion**, we have discussed our work in the context of:
  1. **2nd paragraph:** explaining reactivations in brain regions that perform path integration for navigation and head direction estimation
  2. **4th paragraph:** explaining reactivation in various other systems with continuous attractor dynamics which have been observed, in a diverse set of experimental studies.

We hope that our updated manuscript adequately addresses the reviewer's concerns, and we would be grateful if the reviewer could consider raising their score. We would like to thank the reviewer again for their thorough review and helpful suggestions to improve our work. We eagerly look forward to the reviewer's response.

Reviewer comment (Theory):
The mathematical analysis relies on a number of assumptions, to the extent that it is difficult to judge whether the conclusion would actually be applicable to the numerical experiments. Can these assumptions be justified or better motivated (beyond the sake of mathematical convenience)?

Author response:
**Response:** We thank the reviewer for the suggestion, and we have added a section to the Supplementary Material to clarify the key assumptions for our theory. We provide the assumptions and explanations here:
1. We consider discrete-time approximations of noisy continuous-time RNNs. Using continuous-time RNNs is a common practice in the literature, and we use its discrete-time approximation to be in line with our implementation.
2. The network must be performing some variant of path integration, by integrating change-based information about environmental state variables to some function of these variables. This condition is often met in natural settings, such as spatial navigation.
3. We assume that the inputs to the network are drawn from a stationary distribution, which would translate to assuming that the effects of initial conditions on state occupancy statistics are ignored, and also assuming that the behavioral policy remains constant throughout time.
4. We consider greedy optimization of the loss at every timestep. Greedy optimization is a sensible way of partitioning effort across time in this task: the agent does the best that it can on each timestep, assuming that at each previous timestep the best possible job has been done. In the absence of noise the greedily optimal solution is equivalent to path integration, which is also a **globally** optimal solution.
5. We assume that the network is performing optimally in the presence of noise. In practice, we train our networks until their loss reaches very low, near-zero values.
6. We assume that our network dynamics can be decomposed into two terms with different functional dependencies. The first term depends only on the activity while the second term depends on all original dependencies—the activity, the state and the change-based inputs.
7. For the quiescent state, we assume that the change-based sensory inputs are zero. This is reasonable because for tasks like those we have considered, which involve integrating self-motion cues, $\frac{\mathrm{d}s(t)}{\mathrm{d}t}$ must be zero during periods of quiescence like sleep, where the animal is not moving and hence doesn't receive sensory inputs associated with self-motion.
8. While we assume that the noise variance is doubled during quiescence to show exact equivalence with Langevin sampling, this is by no means necessary to witness reactivation (Suppl. Fig. A.4). This noise variance is equivalent to a temperature parameter for the sampling.

RNNs trained optimally (following any optimization scheme) are approximating the ground-truth optimal solution, given by the greedily optimal dynamics (Section 2.3). It is indeed reasonable for an RNN to implement Eq. 18---according to the literature on RNNs being universal function approximators (Funahashi & Nakamura, 1993). Our empirical results indicate that the network's dynamics align with this solution. Indeed we see the Langevin sampling during quiescence leading to the distributional match. It is certainly possible for suboptimal (or very small) RNNs to fail to align with the theory, the theory only applies for very well-trained networks that are performing near-optimally such as our networks. Overall, we believe our assumptions are well-justified, and the conclusions of the theory are indeed applicable to our numerical experiments as evidenced by our results.

Funahashi & Nakamura. Approximation of dynamical systems by continuous time recurrent neural networks. Neural Networks 6 (1993): 801-806.

We hope that our response and updates to the draft have addressed all of the reviewer's concerns, and would be grateful if the reviewer would consider raising their score. If the reviewer has any additional questions, we would be happy to discuss further. We eagerly await the reviewer's response.