Paper: Resonator-Gated RNNs (ICLR-2024)

Reviewer comment (Writing):
What does the OTDB acronym mean?

Author response:
Please apologize that we did not further elaborate this acronym. QTDB stands for QT Database where QT is the waveform interval in ECGs. We added a remark in the manuscript.

Reviewer comment (Reproducibility):
Which seed values have been used? Without the seeds, it won't be possible to reproduce the paper's experimental results exactly.

Author response:
We will provide respective seeds with publication of the code. But we do not agree with this argument: if the method is sufficiently stable, randomized trials (which can be replicated) that deliver reasonable performance are more convincing than choosing some well working seeds. Moreover, in contrast to the LSTM, the standard deviation for all random runs of the RG-RNN is extremely narrow.

Reviewer comment (Novelty):
Some authors suspect extra weights improve convergence (https://arxiv.org/abs/1803.03635).

Author response:
We agree that - even if unlikely - a few additional weights could in principle make a difference. While the number of parameters was higher for the RG-LSTM the difference is less pronounced than comparing the LSTM with an LSTM that has just a single additional hidden unit. To clarify we will report the number of parameters for LSTM and RG-LSTM. In addition we can retrain an LSTM with an additional hidden unit e.g. 129 for SMNIST. The performance impact of this will be negligible based on first evaluations.

Reviewer comment (Experiments):
If the number of trainable weights was not the same, is the comparison fair?

Author response:
We agree that - even if unlikely - a few additional weights could in principle make a difference. While the number of parameters was higher for the RG-LSTM the difference is less pronounced than comparing the LSTM with an LSTM that has just a single additional hidden unit. To clarify we will report the number of parameters for LSTM and RG-LSTM. In addition we can retrain an LSTM with an additional hidden unit e.g. 129 for SMNIST. The performance impact of this will be negligible based on first evaluations.

Reviewer comment (Experiments):
While experimental section overall is sound and (seemingly) reproducible, it can be greatly improved with experiments on time-series data from other data domains (ex. sensor data, from UCI Data-repo).

Author response:
We see the demand for more results on another dataset and hence added an analysis of gait trials from sensor data provided on UCI. We agree that adding another data domain can emphasise the strengths as well as give clearer indication of weaknesses of the method.

Reviewer comment (Experiments):
While Figure~1 demonstrates how the discrete time resonator dynamics behaves, it would be interesting to see this also as part of the LSTM architecture. For example, working with a 1D synthetic sequence prediction task (say on some periodic signal), is the difference in performance between an LSTM and RG-LSTM evident?

Author response:
To properly answer the reviewers first question we would need some more information of the type of experiment that was in mind. It would be possible to perform the experiment conducted for Figure 1, but it would be difficult to make any statements about the output because it is randomly initialized and untrained. It would in fact be interesting to evaluate the behavior of the RG-RNN in regression tasks (such as time-series forecasting), which is another line of research for us. Our intuition is, however, that the resonator functions primarily as a detector and does not necessarily facilitate the creation of oscillations.

Reviewer comment (Experiments):
Can the modified gating mechanism be applied to GRU as well? How does the performance compare?

Author response:
The resonator-based gating mechanism can be applied to GRUs as well. We think that it would best be applied to the update gate since it regulates how the new inputs are combined with the current hidden state. This is definitely an interesting question and something we will look into - we made a respective comment in the paper. We would assume a similar effect to the network's convergence. But as there is no fundamental and consistent advantage of the GRU over LSTM (there is actually a tendency of the latter working slightly better), we expect the same results when comparing RG-GRUs and RG-LSTMs.

Reviewer comment (Writing):
'prioritize retaining static relevant information': Could you elaborate on `static relevant information`?

Author response:
We see that the term 'static relevant information' is not accurate enough. The type of information that is retained by RNNs is more clearly described as ``transient'', it arises and/or disappears during the sequence.