Paper: Traveling Waves Encode The Recent Past and Enhance Sequence Learning (ICLR-2024)

Reviewer comment (Theory):
Since $\mathbf{u}$ has dimensions $c \times c \times f$, this means that there are convolutional kernels between rings, but the intuition provided does not seem to acknowledge this.

Author response:
Indeed the convolutional kernels are fully parameterized and therefore support waves between channels. We believe the intuition provided in our introduction still supports this. Specifically, since the mapping between channels and the cortical surface is not firmly fixed, it would fit within our model that channels may be laid out parallel on the cortical surface, and such cross-channel communication could then again simply be implemented by local lateral connectivity. We appreciate the reviewer bringing this up, and will strive to make the multi-channel nature of the convolution more clear.

Reviewer comment (Experiments):
The usage of identity-initialized RNNs was also not clear. From the parameter count of such models, it seems like every element of the weight matrix is being optimized. That is not mentioned anywhere in the text (or if it is, it should be highlighted more clearly). Indeed, that is the case for Le et al 2015, but this would detract from the bump model which the authors originially intended to be the main baseline model to compare with wave-RNNs.

Author response:
In both settings all parameters are being optimized as this is necessary for both models to achieve optimal performance. We appreciate the reviewer highlighting that this is not clear, and we have modified the text to reflect this. As we found and demonstrated empirically in Figure 2, despite this full parameterization and optimization, the iRNN still learns a solution which resembles a static ‘bump’ solution, and therefore we believe it to still fit within our motivation.

Reviewer comment (Writing):
The way wave-RNNs are named is confusing and inefficient. At some times, it is labeled as $n = 100, c=6$ to represent $n$ as the total number of neurons and $c$ as the number of channels. This means that there are 16 neurons per channel, and where does the last 4 neurons go (how does the floor function in page 3 work)? At other times it is called $16c$ which I assume refers to the same thing. In both ways of naming, there is no mention about the dimensionality of the kernel -- I suspect it may not be constant because of the remaining 4 neurons, but there has to be a better way to label everything.

Author response:
We thank the reviewer for highlighting potential confusion with respect to the total number of neurons and our notation. In our plots, $n$ refers to the number of neurons per channel, so the total number of neurons is $n \times c$. We have modified the text to make this more clear.

Reviewer comment (Presentation):
In Figure 2, neurons in the iRNN are sorted according by time of maximum activation. How are the neurons in the wRNN sorted and where is the channel separation?

Author response:
There is no sorting of the wRNN neurons, they inherently have spatial locality due to the convolution operation. We therefore simply plot the hidden state vector in the order of the array over which the convolution is applied.

Reviewer comment (Experiments):
The training curves in Figures 3,5,6 are extremely transient and subject to random "resets" in performance. This means that in a single training session, the efficiency of training depends (by luck) on number of resets that happen, which is very apparent in Figure 3. This makes any conclusions drawn about training efficiency unconvincing. More models should be trained and the loss curves averaged.

Author response:
While we agree that the dramatic increases in training loss are less than ideal, we would like to make two notes which may at least temporarily ease the reviewers concerns. First, we note that the y-axis is on a logarithmic scale, therefore although it may look like the loss is 'resetting’, in practice these changes in performance are not as significant as they may first appear. Secondly, we note that all of the believably harmful 'resets’ which happen occur for the wRNN model, which still learns orders of magnitude faster than the iRNN.  

Although we unfortunately do not have time to rerun all of the loss curves and average them for this rebuttal we would certainly be happy to do so for the camera ready if accepted, and we can assure the reviewer that for the reruns we did during development, we saw the same results consistently — the wRNN trained orders of magnitude faster than the iRNN.

Reviewer comment (Writing):
Did the authors really initialize $\mathbf{V}$ to be zero (as claimed on page 3)? I think it is a typo and they actually meant initializing $\mathbf{b}$ to be zero instead?

Author response:
We thank the reviewer for highlighting their confusion. As noted on page 3, we initalize V to be zero *except for a single row which is the identiy*. This means that it the input is effectively driving each single channel independantly from a single source location. In Appendix B The reviewer can see the exact code which is used to initalize the matrices.

Reviewer comment (Evaluation):
There seems to be no additional point in training wRNN + MLP? It provides a small improvement and a single sentence explaining that linear decoding is a bottleneck (which is in fact an important bottleneck to prevent overfitting in neural data) -- am I missing something?

Author response:
We believe the linear decoding bottleneck is indeed highly important to emphasize since this is a major distinction between the wRNN and prior work with which our model is compared (e.g. the best performing models in Tables 2 & 3 are highly non-linear). Specifically, due to the linear encoders in our model, there is effectively very little depth to our model, making comparison with the very deep counterparts listed less fair. To partially account for this we have added the MLP experiment, but ideally would add a multi-layer wRNN experiment in the future. This serves as an intermediate step to show that indeed the linear decoder is a bottleneck. In practice we do not find the model to be overfitting in any significant way yet and therefore politely disagree with the reviewers assertion that a linear decoder is necessary in this setting.

Reviewer comment (Experiments):
Although the concept of this study is novel, I feel the present paper can be strengthened by conducting a deeper analysis to show why internally generated traveling waves in RNNs can improve those computational tasks. For example, since the network model is small (I see the limitation discussed in the end), the authors could perform a dimensionality analysis to demonstrate the network's evolution in the low-dimensional manifold. I do see the author providing an intuitive explanation of the benefit of traveling waves in the introduction, and showing neurons' response in Fig. 2, but they are not enough from my point of view.

Author response:
We appreciate the reviewer noting how the clarity of our presentation could be improved. In response to this, we have updated the intuition provided in our introduction to give a better understanding of the working mechanism of our model which we believe may partially address the reviewer's concern regarding a lack of analysis. Specifically, as pointed our by reviewer h1zv, it is potentially easiest to understand the success of the wRNN model as a having a hidden state in the form of a 'register' or 'stack' where information is sequentially written or pushed. The propagation of waves then serves to prevent overwriting of past input, allowing for superior invertible memory storage.

Pursuant to this, we greatly appreciate the reviewer's suggestion to perform low dimensional analysis of the hidden state dynamics, however we believe that with an intuitive understanding of the dynamics as described above this may not be necessary. Specifically, since we can interpret the hidden state as a set of different `stacks' (one for each channel) where information largely flows within each individual channel separately, the low-dimensional manifold over which the hidden state will evolve is explicitly biased towards these 1-D circular topologies that we have plotted in Figure 2, and in the appendix. We encourage the reviewer to look at Figure 9 of the updated appendix which includes visualization of all channels, as this reinforces the understanding that in general inter-channel communication is relatively minor, and the subspaces of interest for hidden state evolution are exaclty those plotted, as this is the intended design of the network.

Reviewer comment (Theory):
The illustrative example in Fig. 1 (middle) is probably not a perfectly matching example in explaining the benefit of traveling waves. The Fig. 1 illustrates an example of a two-way wave equation, however, the traveling wave in the RNN is only a one-way wave equation that the network state moves in a single direction. I am not clear about why the traveling wave could maintain the spatiotemporal information.

Author response:
We thank the reviewer for highlighting the potential confusion caused by this figure. Indeed in response to these concerns and those of Reviewer h1zv, we have updated Figure 1 to better match the 1-dimensional structure of the wRNN hidden state. Additionally, we believe this new figure better reflects the new intuition provided in the introduction, suggesting to interpret the hidden state as a 'register' or 'stack' where information is written/pushed. We believe this may help to better convey the understanding of the model that the reviewer felt was lacking and thereby additionally contribute to addressing the reviewer's first major point.

Reviewer comment (Theory):
Eq. 2: it seems that the matrix $\Sigma$ misses the time step $\Delta t$ in discretizing Eq. 1. Otherwise the speed $v$ cannot exceed 1 (the term $1-v$ in $\Sigma$). I mean you can absorb the $Delta t$ into a new parameter in Eq. 2 without explicitly expressing it, but in this case the $v$ in Eq. 2 is not the same $v$ in Eq. 1.

Author response:
We thank the reviewer for highlighting the skipped steps in our derivation of the wRNN. Indeed we had assumed that the timestep was absorbed into $\nu$ in equation 2. The full derivation is as follows:
$$ h_{t+1}(x)  = h_t(x) + \Delta t \left[\nu \frac{\partial h(x, t)}{\partial x}\right] $$
$$  h_{t+1}(x) \approx h_t(x) + \Delta t \nu \left[h_t(x+1) - h_t(x)\right] $$
$$  h_{t+1}(x) = (1 - \Delta t \nu) h_t(x) + \Delta t \nu  h_t(x+1) $$
In the updated submission we have changed the velocity in equation 2 to $\nu'$ and noted that this incorporates the timestep.

Reviewer comment (Writing):
Below Eq. 2: I am confused about what the input channel means.

Author response:
We use the term 'input channels' in the standard sense employed when describing convolutional neural network layers. The only difference is that in our setting the convolution operates on an RNN hidden state vector, rather than an input image or feature map. By channels of the hidden state, we then refer to subsets of the hidden state obtained by reshaping the traditional $n$ dimensions into $c$ separate $n' = \lfloor\frac{n}{c}\rfloor$ dimensional subsets (as described after Equation 3). In traditional convolutional neural networks, these subsets of the input signal are typically called the input channels, and the resulting subsets of the output vector (feature maps) are often called the output channels. This naming extends to the convolutional kernel as well, which has weights for each input-output pair of channels respectively. This then extends into the reviewer's minor point 3 below.

Reviewer comment (Writing):
Below Eq. 3: it is not clear how the convolution between $u$ with dimension [c,c,f] and $h$ with dimension [c,n'] is calculated.

Author response:
The convolution between the kernel $\mathbf{u}$ with dimensionality [$c_{out} \times c_{in} \times f$] and the input  with dimensionality [$c_{in} \times n'$], can be understood as computing $c_{out}$ separate convolutions of shape [$c_{in} \times f$] with signal  [$c_{in} \times n'$], where each convolution has a different set of weights -- this is identical to convolution in a standard convolutional neural network layer. In our setting, since we are employing this in a recurrent neural network, the input and output dimensionality must be the same and thus $c_{out} = c_{in} = c$.

Reviewer comment (Experiments):
Do you need to retrain the RNN to copy sequences with different lengths, or produce waves with different speeds?

Author response:
Since the speed of each wave is determined by the parameters of the convolution kernel (which are optimized through training), in order to get waves with different speeds, the network will need to be retrained. In future work, one could imagine alternative 'fast-weight' methods for determining the convolutional kernel parameters (such as with a hypernetwork) where retraining would not be necessary and the speed of waves would be a function of the input itself. This would certainly be a very interesting extension to our model. With respect to generalization to different sequence lengths, we have not performed these experiments and therefore cannot comment with certainty, however we believe there could be a possibility that the network would generalize to longer time delays or sequence lengths without retraining. We believe this would be an extremely interesting subject for future work and believe these types of questions serve to demonstrate the potential value of our submission as a seed for the development of future models in this direction.

Reviewer comment (Theory):
The connection between this computational model and traveling waves in the cortex is extremely tenuous. The problem is that the goal of this model is to allow information to be remembered for a long time, whereas the traveling waves over the cortex (and hippocampus and striatum) are much faster. For instance, the Siapas & Lubenov paper shows that theta travels the length of the hippocampus in on the order of 200 ms.

Author response:
In response to the reviewer’s concern about the biological relevance of our model, and the uncertainty regarding the reality of waves as a memory mechanism in the brain, we would like to highlight two potential answers. 

The first is that there is indeed preliminary evidence that some traveling waves do exist over significantly longer timescales than the millisecond scale referenced by the reviewer, and that these do indeed have a relation to memory. Specifically (as noted in our introduction), "King & Wyart (2021) provide evidence that traveling waves propagating from posterior to anterior cortex serve to encode sequential stimuli in an overlapping `multiplexed' manner". In detail, they use a task of carefully constructed sequential stimuli presented over 2 seconds and show that such stimuli are decodable from EEG measurements at sequentially propagated regions of the cortex, implying a wave of activity encoding this information. Others such as Zabeh et al. (2023) also show traveling waves in frontal and parietal lobes encode memory of recent rewards. Therefore, while we certainly agree with the reviewer that certain waves (such as those which exist in the millisecond timescale of the hippocampus) are inappropriate for encoding longer term memories, we believe that there are likely a diversity of types of traveling waves in the brain and therefore this mechanism should not be discounted so quickly. 

The second potential answer to the reviewers concern is that the current studies of traveling wave dynamics in the brain are inherently very limited. It is simply very challenging to measure traveling waves in the brain due to the necessity for both high spatial and temporal resolution simultaneously. Therefore, while we agree that there is currently limited evidence that traveling waves may support the type of memory implied by our model, we again do not believe that this should be a reason to discount such a potential mechanism. Specifically, we believe that our model may indeed encourage further targeted study of waves. Furthermore, we note that we are not the first to suggest such a relation between waves and memory as outlined in our introduction and in the review paper (Muller et al. 2018).

Reviewer comment (Evaluation):
There is some evidence for very slow oscillations in MEC, but there is no evidence these are traveling waves.  
https://doi.org/10.1101/2022.05.02.490273  
https://doi.org/10.1016/j.celrep.2023.113271

Author response:
In response to the reviewer’s concern about the biological relevance of our model, and the uncertainty regarding the reality of waves as a memory mechanism in the brain, we would like to highlight two potential answers. 

The first is that there is indeed preliminary evidence that some traveling waves do exist over significantly longer timescales than the millisecond scale referenced by the reviewer, and that these do indeed have a relation to memory. Specifically (as noted in our introduction), "King & Wyart (2021) provide evidence that traveling waves propagating from posterior to anterior cortex serve to encode sequential stimuli in an overlapping `multiplexed' manner". In detail, they use a task of carefully constructed sequential stimuli presented over 2 seconds and show that such stimuli are decodable from EEG measurements at sequentially propagated regions of the cortex, implying a wave of activity encoding this information. Others such as Zabeh et al. (2023) also show traveling waves in frontal and parietal lobes encode memory of recent rewards. Therefore, while we certainly agree with the reviewer that certain waves (such as those which exist in the millisecond timescale of the hippocampus) are inappropriate for encoding longer term memories, we believe that there are likely a diversity of types of traveling waves in the brain and therefore this mechanism should not be discounted so quickly. 

The second potential answer to the reviewers concern is that the current studies of traveling wave dynamics in the brain are inherently very limited. It is simply very challenging to measure traveling waves in the brain due to the necessity for both high spatial and temporal resolution simultaneously. Therefore, while we agree that there is currently limited evidence that traveling waves may support the type of memory implied by our model, we again do not believe that this should be a reason to discount such a potential mechanism. Specifically, we believe that our model may indeed encourage further targeted study of waves. Furthermore, we note that we are not the first to suggest such a relation between waves and memory as outlined in our introduction and in the review paper (Muller et al. 2018).

Reviewer comment (Evaluation):
There is a critical difference between the sequences observed with time cells and the traveling waves in this model. In particular, sequences in the brain slow down as they unfold, as if the wave was traveling through a fluid whose properties change systematically from one end of a channel to the other. This does not seem to be a property of this model. Although Figure 9 in Appendix C shows waves that travel at different speeds in different channels, the waves within each channel proceed at a constant velocity (the right panel is more complicated but certainly doesn't slow down the way the neural data do).

Author response:
The reviewer's second primary concern is how our model might relate to the well known time cells of the cortex. Specifically we would like to propose an alternative complimentary viewpoint to the reviewer's comment that  ‘the sequences in the brain slow down as they unfold’. In our understanding, from (Jacques et al. 2021 <https://arxiv.org/abs/2104.04646> & Cao et al. 2022 <https://doi.org/10.7554%2FeLife.75353>), an alternative equally valid description of the activity of time cells is as a set of basis functions which are logarithmically spaced in time. In other words, the frequency (velocity) of each neuron (wave) does not change over time, but rather different sets of neurons represent the same input over different (logarithmically spaced) time scales. This phenomena can be seen in Figure 3E of the reviewer's reference (https://doi.org/10.1016/j.cub.2021.01.032). We believe that the experiments in Figure 10 of Appendix C precisely demonstrate such a phenomenon — different speed waves effectively encode the input over different time scales. We did additionally experiment with logarithmically spaced velocities but found the presented randomly initialized velocities to perform better. We believe that the connection between time cells and the waves in our model is indeed relevant and interesting, and we thus appreciate the reviewer bringing this point to our attention. We have included a preliminary discussion of time cells in our related work, and we have updated our discussion in the appendix to reference time cells more prominently. Ultimately we believe that rather than an alternative to ideas of time cells our model may provide a complimentary view by which further discoveries may be made.

Reviewer comment (Theory):
Suppose that the velocity v in Eq. 2 changed systematically as a function of the row of the matrix. In particular, what if v went down like 1/x? How would that wave behave? How would this change how the longest time horizon of the memory scales with number of weights/units? Would this model behave better or worse on the problems in this paper than wRNN does? Presumably worse because the temporal resolution at long delays would be poor and problems like psMNIST require precise timing information. Could those problems be mitigated with a deep network?

Author response:
We thank the reviewer for these questions and suggestions for how the wave velocity might me modulated during wave propagation. We appreciate this highly original suggestion, it is not something we had considered before, and therefore decided to perform this additional experiment. In the updated appendix section titled 'Waves with variable velocity' we include visualizations of the hidden state of a wRNN with a hidden-to-hidden connectivity matrix initialized exactly as described by the reviewer (v scaled as 1/x where x is the row index). Indeed, when trained on sMNIST we see that the model performs roughly the same as the wRNN presented in the main text (test accuracy %97.76), and that waves appear to slow in velocity as they propagate. We agree that this might improve the longest time-horizon the model is able to handle at a sacrifice of temporal resolution, but leave precise quantification of this to future work. Ultimately we present this experiment to demonstrate that such modifications are possible in our framework, thereby increasing the benefit of the present submission to the community.

Reviewer comment (Writing):
The wave-field comparisons, claims, and references seem strained and unnecessary. The term “wave-field” is not commonly used in neuroscience or ML, and if it is kept, a brief definition should be provided. However, it may not be necessary or helpful.

Author response:
The reviewer is correct in understanding the name 'wave-field' is simply a reference to the hidden state of the network which supports and encourages wave solutions. This is a term derived from physics to describe "the extended area or space taken up by a wave" (wiktionary). We decided to use this name in order to distinguish this type of hidden state from the baseline RNN hidden state which has no notion of spatial organization and no bias towards traveling wave dynamics. Our initial motivation for appealing to physical wave dynamics was in reference to the review paper (Muller et al. 2018) in which they make an analogy with the pilot-wave work of (Perrard et al. 2016). We include a quote below from that review  which we believe may directly address some of the reviewer's concerns related to its inclusion:
> A recent study examined the dynamics of a silicone droplet bouncing on a vibrating oil bath (Perrard et al. 2016). The vibrations of this medium propel the droplet upward and cause it to bounce continually, creating a pattern of successive impacts across the surface (Fig. 6a). With each bounce, the droplet creates a set of waves in the oil bath. In our analogy, these droplet waves may be similar to those created in the sensory cortex by the impact of afferent, stimulus-driven action potentials. The waves evoked by each impact of the droplet then create a pattern across the oil bath that leaves a trace of the path of the droplet. With sufficient knowledge of the dynamic properties of the medium, one could decode the history of the path of the droplet simply from the observed spatial pattern at each point in time. Furthermore, temporal reversibility is preserved for waves even in chaotic systems; this property, along with a clever manipulation of the experimental apparatus, can be used to cause the droplet to exactly reverse its trajectory (Fig. 6a). It is argued that this last manipulation shows that the global wave field stores information about the trajectory of the droplet with sufficient precision that it can be read back out. While the authors did not have neuroscience in mind and, indeed, note that they have not yet considered practical implementations, they posit that this system implements the basic elements for general computations that could find application in future work.

We appreciate that the reviewer has highlighted that our initial emphasis on this analogy may have been distracting to readers. We have therefore followed the reviewer's suggestion and rephrased our introduction to highlight this is an imperfect analogy while including the more intuitive `register' description earlier on. We have additionally updated Figure 1 to match the wRNN dynamics more closely and resemble physical wave dynamics less.

Reviewer comment (Novelty):
That the brain supports wavelike activity is already well-established, and many computational neuroscience models (e.g., Wilson-Cowan neural fields equations) support waves in ways that have been mathematically analyzed. The authors’ discretization methodology suggests a connection to such analyses.

Author response:
The reviewer is correct in understanding the name 'wave-field' is simply a reference to the hidden state of the network which supports and encourages wave solutions. This is a term derived from physics to describe "the extended area or space taken up by a wave" (wiktionary). We decided to use this name in order to distinguish this type of hidden state from the baseline RNN hidden state which has no notion of spatial organization and no bias towards traveling wave dynamics. Our initial motivation for appealing to physical wave dynamics was in reference to the review paper (Muller et al. 2018) in which they make an analogy with the pilot-wave work of (Perrard et al. 2016). We include a quote below from that review  which we believe may directly address some of the reviewer's concerns related to its inclusion:
> A recent study examined the dynamics of a silicone droplet bouncing on a vibrating oil bath (Perrard et al. 2016). The vibrations of this medium propel the droplet upward and cause it to bounce continually, creating a pattern of successive impacts across the surface (Fig. 6a). With each bounce, the droplet creates a set of waves in the oil bath. In our analogy, these droplet waves may be similar to those created in the sensory cortex by the impact of afferent, stimulus-driven action potentials. The waves evoked by each impact of the droplet then create a pattern across the oil bath that leaves a trace of the path of the droplet. With sufficient knowledge of the dynamic properties of the medium, one could decode the history of the path of the droplet simply from the observed spatial pattern at each point in time. Furthermore, temporal reversibility is preserved for waves even in chaotic systems; this property, along with a clever manipulation of the experimental apparatus, can be used to cause the droplet to exactly reverse its trajectory (Fig. 6a). It is argued that this last manipulation shows that the global wave field stores information about the trajectory of the droplet with sufficient precision that it can be read back out. While the authors did not have neuroscience in mind and, indeed, note that they have not yet considered practical implementations, they posit that this system implements the basic elements for general computations that could find application in future work.

We appreciate that the reviewer has highlighted that our initial emphasis on this analogy may have been distracting to readers. We have therefore followed the reviewer's suggestion and rephrased our introduction to highlight this is an imperfect analogy while including the more intuitive `register' description earlier on. We have additionally updated Figure 1 to match the wRNN dynamics more closely and resemble physical wave dynamics less.

Reviewer comment (Theory):
Appealing to “physical wave fields” to relate waves and memory is overly speculative and unnecessary for the simple system under study. The brain is a dissipative rather than a conservative system, so many aspects of physical wave fields may not apply. The single reference provided (Perrard et al. 2016) does not clearly apply to either the brain or the wave-RNN.

Author response:
The reviewer is correct in understanding the name 'wave-field' is simply a reference to the hidden state of the network which supports and encourages wave solutions. This is a term derived from physics to describe "the extended area or space taken up by a wave" (wiktionary). We decided to use this name in order to distinguish this type of hidden state from the baseline RNN hidden state which has no notion of spatial organization and no bias towards traveling wave dynamics. Our initial motivation for appealing to physical wave dynamics was in reference to the review paper (Muller et al. 2018) in which they make an analogy with the pilot-wave work of (Perrard et al. 2016). We include a quote below from that review  which we believe may directly address some of the reviewer's concerns related to its inclusion:
> A recent study examined the dynamics of a silicone droplet bouncing on a vibrating oil bath (Perrard et al. 2016). The vibrations of this medium propel the droplet upward and cause it to bounce continually, creating a pattern of successive impacts across the surface (Fig. 6a). With each bounce, the droplet creates a set of waves in the oil bath. In our analogy, these droplet waves may be similar to those created in the sensory cortex by the impact of afferent, stimulus-driven action potentials. The waves evoked by each impact of the droplet then create a pattern across the oil bath that leaves a trace of the path of the droplet. With sufficient knowledge of the dynamic properties of the medium, one could decode the history of the path of the droplet simply from the observed spatial pattern at each point in time. Furthermore, temporal reversibility is preserved for waves even in chaotic systems; this property, along with a clever manipulation of the experimental apparatus, can be used to cause the droplet to exactly reverse its trajectory (Fig. 6a). It is argued that this last manipulation shows that the global wave field stores information about the trajectory of the droplet with sufficient precision that it can be read back out. While the authors did not have neuroscience in mind and, indeed, note that they have not yet considered practical implementations, they posit that this system implements the basic elements for general computations that could find application in future work.

We appreciate that the reviewer has highlighted that our initial emphasis on this analogy may have been distracting to readers. We have therefore followed the reviewer's suggestion and rephrased our introduction to highlight this is an imperfect analogy while including the more intuitive `register' description earlier on. We have additionally updated Figure 1 to match the wRNN dynamics more closely and resemble physical wave dynamics less.

Reviewer comment (Presentation):
The manuscript would improve by removing the discussion of wave fields and instead providing or moving the intuitive explanations (e.g., the “register or ‘tape’” description on p. 20) into the Introduction.

Author response:
The reviewer is correct in understanding the name 'wave-field' is simply a reference to the hidden state of the network which supports and encourages wave solutions. This is a term derived from physics to describe "the extended area or space taken up by a wave" (wiktionary). We decided to use this name in order to distinguish this type of hidden state from the baseline RNN hidden state which has no notion of spatial organization and no bias towards traveling wave dynamics. Our initial motivation for appealing to physical wave dynamics was in reference to the review paper (Muller et al. 2018) in which they make an analogy with the pilot-wave work of (Perrard et al. 2016). We include a quote below from that review  which we believe may directly address some of the reviewer's concerns related to its inclusion:
> A recent study examined the dynamics of a silicone droplet bouncing on a vibrating oil bath (Perrard et al. 2016). The vibrations of this medium propel the droplet upward and cause it to bounce continually, creating a pattern of successive impacts across the surface (Fig. 6a). With each bounce, the droplet creates a set of waves in the oil bath. In our analogy, these droplet waves may be similar to those created in the sensory cortex by the impact of afferent, stimulus-driven action potentials. The waves evoked by each impact of the droplet then create a pattern across the oil bath that leaves a trace of the path of the droplet. With sufficient knowledge of the dynamic properties of the medium, one could decode the history of the path of the droplet simply from the observed spatial pattern at each point in time. Furthermore, temporal reversibility is preserved for waves even in chaotic systems; this property, along with a clever manipulation of the experimental apparatus, can be used to cause the droplet to exactly reverse its trajectory (Fig. 6a). It is argued that this last manipulation shows that the global wave field stores information about the trajectory of the droplet with sufficient precision that it can be read back out. While the authors did not have neuroscience in mind and, indeed, note that they have not yet considered practical implementations, they posit that this system implements the basic elements for general computations that could find application in future work.

We appreciate that the reviewer has highlighted that our initial emphasis on this analogy may have been distracting to readers. We have therefore followed the reviewer's suggestion and rephrased our introduction to highlight this is an imperfect analogy while including the more intuitive `register' description earlier on. We have additionally updated Figure 1 to match the wRNN dynamics more closely and resemble physical wave dynamics less.

Reviewer comment (Evaluation):
The wave and Fourier analysis appear inconsistent and potentially problematic. While wRNNs clearly display waves in spatial-temporal plots, the mapping and analysis of the Fourier domain (FD) does not always match patterns in regular spatiotemporal plots (RSP), and it is unclear how much these analyses add.

Author response:
We thank the reviewer for highlighting the inconsistency of wave velocity estimates from our Fourier analysis and regular spatio-temporal analysis. Thanks to the reviewer's comments, we have indeed located a miscalibration of our calculation of the axes labels in the original submission. Specifically, in the prior submission, we did not consider the sampling rate, and therefore did not have correct bounds on our axes. In response we have updated all Fourier analysis plots in our submission, and encourage the reviewer to re-validate their consistency with the regular spatio-temporal velocity estimates. To facilitate this validation, we have additionally included a new set of calibration experiments in the appendix (Figure 12), where we have computed the 2D Fourier transform of synthetic data with known velocities and observed slopes of peak magnitude in the Fourier domain which match the known velocities. 

In response to the reviewer's concern that the Fourier analysis may be superfluous, while we agree this may be the case for the example in Figure 2 where waves are very clear in the regular domain, we found the Fourier domain plots to be highly beneficial in developement of the model and in studying baselines where the existence of waves is much less clear cut. This is exemplified by the prior work on traveling waves (Davis et al., 2021) in which the authors strongly rely on such analysis to validate which aspects of model architecture yield waves. Ultimately we believe this analysis is helpful both as a second point of validation for the existence/lack of traveling waves in the wRNN/iRNN, as well as a demonstration of a tool for researchers who may be interested in continuing this line of work. We further believe with the corrected calibration there is little negative side effect to including this extra analysis in the paper, and therefore have left it in our updated submission.

Reviewer comment (Evaluation):
Constant-velocity, 1-D waves do not need Fourier transformation to infer their speeds, as slopes in the RSP correspond to speeds. For example, in Fig. 2 (top left), there is a large discrepancy between the speed calculated from the RSP (~1.7 units/step) and the speed indicated by the FD (~0.3 units/step).

Author response:
We thank the reviewer for highlighting the inconsistency of wave velocity estimates from our Fourier analysis and regular spatio-temporal analysis. Thanks to the reviewer's comments, we have indeed located a miscalibration of our calculation of the axes labels in the original submission. Specifically, in the prior submission, we did not consider the sampling rate, and therefore did not have correct bounds on our axes. In response we have updated all Fourier analysis plots in our submission, and encourage the reviewer to re-validate their consistency with the regular spatio-temporal velocity estimates. To facilitate this validation, we have additionally included a new set of calibration experiments in the appendix (Figure 12), where we have computed the 2D Fourier transform of synthetic data with known velocities and observed slopes of peak magnitude in the Fourier domain which match the known velocities.

Reviewer comment (Evaluation):
Similar wave speed mismatches exist in the Appendix (e.g., Fig. 9, 2nd column, top), where slopes in the RSP suggest speeds of 0.35-0.42 units/step, but the FD suggests ~0.3 and ~0.9 units/step. The authors should clarify possible causes and verify results.

Author response:
We thank the reviewer for highlighting the inconsistency of wave velocity estimates from our Fourier analysis and regular spatio-temporal analysis. Thanks to the reviewer's comments, we have indeed located a miscalibration of our calculation of the axes labels in the original submission. Specifically, in the prior submission, we did not consider the sampling rate, and therefore did not have correct bounds on our axes. In response we have updated all Fourier analysis plots in our submission, and encourage the reviewer to re-validate their consistency with the regular spatio-temporal velocity estimates. To facilitate this validation, we have additionally included a new set of calibration experiments in the appendix (Figure 12), where we have computed the 2D Fourier transform of synthetic data with known velocities and observed slopes of peak magnitude in the Fourier domain which match the known velocities.

Reviewer comment (Evaluation):
Some fast pulse bursts are visible in the RSP for iRNN in Fig. 2 but do not show in the FD. This includes both fast and slower pulses observable directly in the RSP. The absence of these in the FD should be explained, especially if they are deemed aberrant.

Author response:
Indeed, due to this sorting we do see the short wave-bursts the reviewer has identified, however they simply do not contain enough power to be visible in the fourier domain, and additionally do not persist long enough to be useful as a form of memory.

Reviewer comment (Theory):
There may be an error in the initial calibration steps (steps 1 and 2) of the original algorithm from the “Recognition and Velocity Computation of Large Moving Objects in Images” (RVC) paper that could explain speed disparities between RSPs and FDs.

Author response:
We thank the reviewer for highlighting the inconsistency of wave velocity estimates from our Fourier analysis and regular spatio-temporal analysis. Thanks to the reviewer's comments, we have indeed located a miscalibration of our calculation of the axes labels in the original submission. Specifically, in the prior submission, we did not consider the sampling rate, and therefore did not have correct bounds on our axes. In response we have updated all Fourier analysis plots in our submission, and encourage the reviewer to re-validate their consistency with the regular spatio-temporal velocity estimates.

Reviewer comment (Evaluation):
There appear to be different wave speeds in some cases (e.g., Fig. 9 shows narrow excitatory waves and broad inhibitory waves). It is unclear why a single channel would support multiple wave speeds, and this should be explored in greater depth.

Author response:
Yes, the convolutional kernel $\mathbf{u}$ is fully parametrized and thus allows for the potential for activity to propagate between channels if weights are optimized that way. In the updated submission, we have included a new figure (Figure 9) which visualizes activity for all channels of the wRNN. However, since there is no implicit ordering of the channels, it is similar to the situation with the iRNN where it will be difficult to directly identify cross-channel waves since they may move between arbitrary channel indexes at each time-point (see comment below). In practice however, we agree with the reviewer that this is likely the reason multiple wave speeds can exist in a single channel, since each channel does have its own velocity parameter which only supports a single velocity independently.

Reviewer comment (Evaluation):
It is unclear whether there is cross-talk across channels, and whether multiple wave speeds per channel could be explained by cross-channel waves. The authors should examine hidden unit images across channels for evidence of this.

Author response:
Yes, the convolutional kernel $\mathbf{u}$ is fully parametrized and thus allows for the potential for activity to propagate between channels if weights are optimized that way. In the updated submission, we have included a new figure (Figure 9) which visualizes activity for all channels of the wRNN. However, since there is no implicit ordering of the channels, it is similar to the situation with the iRNN where it will be difficult to directly identify cross-channel waves since they may move between arbitrary channel indexes at each time-point (see comment below). In practice however, we agree with the reviewer that this is likely the reason multiple wave speeds can exist in a single channel, since each channel does have its own velocity parameter which only supports a single velocity independently.

Reviewer comment (Experiments):
It remains unclear what Fourier transforms add to detecting 1-D waves, which should be directly observable in the RSPs. The FD might be more useful for detecting cross-channel waves or multiple wave speeds, if verified. The authors are encouraged to present corresponding images and use marginalization decomposition as described in the RVC algorithm.

Author response:
In response to the reviewer's concern that the Fourier analysis may be superfluous, while we agree this may be the case for the example in Figure 2 where waves are very clear in the regular domain, we found the Fourier domain plots to be highly beneficial in developement of the model and in studying baselines where the existence of waves is much less clear cut. This is exemplified by the prior work on traveling waves (Davis et al., 2021) in which the authors strongly rely on such analysis to validate which aspects of model architecture yield waves. Ultimately we believe this analysis is helpful both as a second point of validation for the existence/lack of traveling waves in the wRNN/iRNN, as well as a demonstration of a tool for researchers who may be interested in continuing this line of work.

Reviewer comment (Evaluation):
The authors mention re-sorting iRNN units to reveal hidden waves, which is problematic. Re-sorting can destroy genuine waves or artificially make random activity appear wavelike.

Author response:
We would like to clarify what appears to be a potential confusion by the reviewer in this comment. In the iRNN there is no notion of spatial locality, so it does not make sense to speak of waves in the network hidden state without any sorting. As another way to understand this, in a network without spatial biases (such as the iRNN) ‘waves’, if they do exist, would be equally likely to propagate between any two neurons in the hidden state over time (not just those which are arbitrarily adjacent to one another in our hidden state vector). Therefore, in order to identify if there is any smooth continuous flow of activity between any neurons (mathematically equivalent to a wave), it makes sense to sort activity according to maximal ‘onset time’. Indeed, due to this sorting we do see the short wave-bursts the reviewer has identified, however they simply do not contain enough power to be visible in the fourier domain, and additionally do not persist long enough to be useful as a form of memory. We do appreciate that this sorting may raise concern in readers and thank the reviewer for bringing it up. To address this, in the updated paper appendix (Figure 11) we have included the unsorted iRNN activations where significantly less structure is present, as is expected.

Reviewer comment (Evaluation):
There is too little analysis of mechanisms underlying performance improvements. Although the ablation study is helpful, more analysis is needed to characterize the relationship between wavelike activity and network performance.

Author response:
We appreciate the reviewer’s concern that the overall mechanism could be better explained. In the updated version of the paper we have updated our introduction and initial figure to describe the intuition of this mechanism more clearly. We hope this partially alleviates the reviewers concerns, however we certainly agree that more could be done to understand the 'impressive' performance of the wave-RNN. Overall, we believe mechanistic understanding is often proceeded by empirical findings which we believe contain significant value to the community on their own. We therefore present this work as a working model of wave-based memory, opening the door to future work which may improve mechanistic understanding.

Reviewer comment (Presentation):
Figure 2: Both plots on the right have the leftmost y-axis digits obscured.

Author response:
We thank the reviewer for highlighting the graphical issues with some of our plots. We have fixed these in the updated revision.

Reviewer comment (Presentation):
Figure 9 (top) plots appear to have their x- and y-axis labels transposed (or else the FD plots in Fig. 2 are transposed).

Author response:
We thank the reviewer for highlighting the graphical issues with some of our plots. We have fixed these in the updated revision.

Reviewer comment (Presentation):
Figure 15 needs axis labels.

Author response:
We thank the reviewer for highlighting the graphical issues with some of our plots. We have fixed these in the updated revision.