Paper: Inverse Approximation Theory for Nonlinear Recurrent Neural Networks (ICLR-2024)

Reviewer comment (Theory):
The definition of memory for nonlinear RNNs is consistent with that for linear RNNs, but the rationality of the proposed definition needs more explanations. The authors review the definition of memory for linear RNNs and observe that if the memory decays fast, then the target has short memory, and vice versa. The observation can be obtained directly from the definition and satisfies the intuition of memory. But for the memory for nonlinear RNNs, authors motivate it from a derivative form. This motivation is too mathematical and lacks an intuitive explanation. What is the relationship between the decaying speed of memory and the dependence of the target function on early inputs?

Author response:
We thank the reviewer for their thoughtful comments and constructive feedback.

Weaknesses:

1. `The definition of memory for nonlinear RNNs is consistent with that for linear RNNs, but the rationality of the proposed definition needs more explanations.`

(**Rationality of the proposed definition**)
People use the outcomes of long-term memory tasks to demonstrate the long-term memory of models. 
Although the performance in common experiments, such as addition problems, copying tasks, and associative recalls is indicative of long-term memory attributes, it is important to note that these experiments are heuristic in nature.
We propose the memory function definitions for a precise, task-independent characterization of the model memories. (Definition 3.1 Page 5)

`The authors review the definition of memory for linear RNNs and observe that if the memory decays fast, then the target has short memory, and vice versa. The observation can be obtained directly from the definition and satisfies the intuition of memory. But for the memory for nonlinear RNNs, authors motivate it from a derivative form. This motivation is too mathematical and lacks an intuitive explanation. What is the relationship between the decaying speed of memory and the dependence of the target function on early inputs?`

(**Relation between memory decaying speed and target dependence on early inputs**)
An intuitive explanation is that current memory functions characterize how fast the memory about the sudden input change at $t=0$ decays as $t$ grows. 
Inputs such as Heaviside inputs, reversed inputs, and impulse inputs all have this feature. 
Therefore the memory function evaluated over these inputs can be interpreted as **a measure for "dependence of the target function on early inputs".** 
$$\rho_T := \sup_{x \neq 0} \frac{|y_T(\mathbf{i}^x)|}{|x|}  = \sup_{x \neq 0} \frac{H_T((x, 0, ...))}{|x|}. \quad \textrm{Impulse-inputs-based memory}$$
We show the "equivalence" of different memory functions in linear functional sense in Appendix F and G. 
Since memory functions over Heaviside inputs are easier to evaluate numerically (as shown in Appendix H), we adopt the corresponding derivative form (as shown in Equation 5) to define the memory function. 

(**Why special inputs**)
A natural question is why we need a particular set of inputs rather than use any continuous input. 
The identity functional $H_t(\mathbf{x}) = x_t$ takes the input sequence $x_t = \frac{1}{t}, t > 0$ to a polynomial decaying output $y_t = \frac{1}{t}$. However, this does not mean the functional has a polynomial-decaying memory. In fact, this identity functional has no memory of past inputs in the sense that changing $x_0$ does not influence the output of $y_t$ with $t > 0$. 
The above example indicates the selection of inputs to extract the memory pattern needs to be chosen properly. 
This is also one of the reasons we select the Heaviside inputs as the test inputs to construct the derivative form.

Reviewer comment (Presentation):
I have no strong reservations about the paper. I do have a question about Figure 3, which I did not understand. I am willing to raise the rating should this interrogation be answered. [Update on Nov. 18: the authors clarified this point in their comments below, and I updated my rating accordingly].

Author response:
Yes, the takeaway for the filtering experiment is "stable approximation and decaying memory implies exponential decay". 
This paper primarily focuses on the inverse-type approximation theorem.
The assertion that "Exponential decay of the target implies stable approximation" is a forward-type (Jackson-type) result that has not been established as of now.

In our theory, we show that nonlinear RNNs with decaying memory are decaying exponentially. 
Randomly initialized nonlinear RNNs might not have decaying memory. 
In Appendix I.1 (Page 30) and Figure 11 (Page 31), we present examples of nonlinear RNNs having non-decaying memory.
In particular, in Figure 11 (b), the trajectory of hidden state derivatives indicate the existence of periodic memory function from the 2D tanh RNN. 
Therefore, we construct several targets in the experiment for Figure 3 and use decaying memory and existence of stable approximation by smaller RNNs as the criterions to filter the targets teacher models. 

The reason we pick larger RNNs as teacher models is that nonlinear RNNs are universal approximators. 
Therefore from the density perspective, larger models with randomly initialized weights should have more diverse memory pattern than the smaller models.
We also investigate the memory patterns of transformers. 
It can be seen in Appendix I.2 (Page 31 and 32) that transformers do not have a decaying memory so the stable approximation cannot be achieved by nonlinear RNNs: We failed to achieve a smaller validation loss with larger models ($m=64$) over 1000 epochs of training.

Reviewer comment (Experiments):
I do not understand the filtering part of the experiment. Since we know from the theoretical results that RNNs can only represent exponentially decaying functions, the teacher models should all be exponentially decaying. So why is any filtering required?

Author response:
Yes, the takeaway for the filtering experiment is "stable approximation and decaying memory implies exponential decay". 
This paper primarily focuses on the inverse-type approximation theorem.
The assertion that "Exponential decay of the target implies stable approximation" is a forward-type (Jackson-type) result that has not been established as of now.

In our theory, we show that nonlinear RNNs with decaying memory are decaying exponentially. 
Randomly initialized nonlinear RNNs might not have decaying memory. 
In Appendix I.1 (Page 30) and Figure 11 (Page 31), we present examples of nonlinear RNNs having non-decaying memory.
In particular, in Figure 11 (b), the trajectory of hidden state derivatives indicate the existence of periodic memory function from the 2D tanh RNN. 
Therefore, we construct several targets in the experiment for Figure 3 and use decaying memory and existence of stable approximation by smaller RNNs as the criterions to filter the targets teacher models. 

The reason we pick larger RNNs as teacher models is that nonlinear RNNs are universal approximators. 
Therefore from the density perspective, larger models with randomly initialized weights should have more diverse memory pattern than the smaller models.
We also investigate the memory patterns of transformers. 
It can be seen in Appendix I.2 (Page 31 and 32) that transformers do not have a decaying memory so the stable approximation cannot be achieved by nonlinear RNNs: We failed to achieve a smaller validation loss with larger models ($m=64$) over 1000 epochs of training.

Reviewer comment (Experiments):
If filtering is indeed required, is it also performed before plotting the left-side plot? Otherwise we should be seeing some non-converging data points.

Author response:
Yes, the takeaway for the filtering experiment is "stable approximation and decaying memory implies exponential decay". 
This paper primarily focuses on the inverse-type approximation theorem.
The assertion that "Exponential decay of the target implies stable approximation" is a forward-type (Jackson-type) result that has not been established as of now.

In our theory, we show that nonlinear RNNs with decaying memory are decaying exponentially. 
Randomly initialized nonlinear RNNs might not have decaying memory. 
In Appendix I.1 (Page 30) and Figure 11 (Page 31), we present examples of nonlinear RNNs having non-decaying memory.
In particular, in Figure 11 (b), the trajectory of hidden state derivatives indicate the existence of periodic memory function from the 2D tanh RNN. 
Therefore, we construct several targets in the experiment for Figure 3 and use decaying memory and existence of stable approximation by smaller RNNs as the criterions to filter the targets teacher models. 

The reason we pick larger RNNs as teacher models is that nonlinear RNNs are universal approximators. 
Therefore from the density perspective, larger models with randomly initialized weights should have more diverse memory pattern than the smaller models.
We also investigate the memory patterns of transformers. 
It can be seen in Appendix I.2 (Page 31 and 32) that transformers do not have a decaying memory so the stable approximation cannot be achieved by nonlinear RNNs: We failed to achieve a smaller validation loss with larger models ($m=64$) over 1000 epochs of training.

Reviewer comment (Evaluation):
As a consequence, I am wondering if the takeaway of the experiment should be “stable approximation implies exponential decay” or rather “exponential decay implies stable approximation”.

Author response:
Yes, the takeaway for the filtering experiment is "stable approximation and decaying memory implies exponential decay". 
This paper primarily focuses on the inverse-type approximation theorem.
The assertion that "Exponential decay of the target implies stable approximation" is a forward-type (Jackson-type) result that has not been established as of now.

In our theory, we show that nonlinear RNNs with decaying memory are decaying exponentially. 
Randomly initialized nonlinear RNNs might not have decaying memory. 
In Appendix I.1 (Page 30) and Figure 11 (Page 31), we present examples of nonlinear RNNs having non-decaying memory.
In particular, in Figure 11 (b), the trajectory of hidden state derivatives indicate the existence of periodic memory function from the 2D tanh RNN. 
Therefore, we construct several targets in the experiment for Figure 3 and use decaying memory and existence of stable approximation by smaller RNNs as the criterions to filter the targets teacher models. 

The reason we pick larger RNNs as teacher models is that nonlinear RNNs are universal approximators. 
Therefore from the density perspective, larger models with randomly initialized weights should have more diverse memory pattern than the smaller models.
We also investigate the memory patterns of transformers. 
It can be seen in Appendix I.2 (Page 31 and 32) that transformers do not have a decaying memory so the stable approximation cannot be achieved by nonlinear RNNs: We failed to achieve a smaller validation loss with larger models ($m=64$) over 1000 epochs of training.

Reviewer comment (Presentation):
Page 5: I found the paragraph above Definition 3.2 hard to follow. I am not sure to understand the reason for introducing the terminology “queried memory”. The experiments of Appendix B are important in my opinion because they suggest that RNNs may share the exponential decay property with LSTMs, but not Transformer. But I do not understand why they are introduced at this point in the paper, and not later on, e.g. in the conclusion.

Author response:
1. Page 5: The queried memory is proposed as an empirical evaluation of the target memory. 
    As our memory function is defined by a derivative form, the queried memory is the memory evaluated in practice to approximate the memory function. 
    In Appendix F and G (Page 28), we discuss the memory function queried by other test inputs and the equivalence over linear functional are discussed. 
    Numerical equivalence is demonstrated with the memory function of the randomly-initialized models in Appendix H (Page 29).

Reviewer comment (Presentation):
Page 7: the proof sketch is difficult to follow, and I am not sure it brings significant added value. If you need additional space in the main paper, I’d suggest putting the proof sketch in the Appendix just before the proof.

Author response:
2. Page 7: We move the proof sketch to the Appendix and replace it with the discussion on the extension to GRU/LSTM.

Reviewer comment (Writing):
Page 7, line -2: “In order” -> “in order”.

Author response:
`Minor remarks`: Thank you for the detailed writing suggestions and pointing out the typos, we have corrected them in the paper revision (marked in blue).

Reviewer comment (Writing):
Page 8: “approaching 0” -> “approaching 0 on the negative side”?

Author response:
`Minor remarks`: Thank you for the detailed writing suggestions and pointing out the typos, we have corrected them in the paper revision (marked in blue).

Reviewer comment (Writing):
Page 8: “Take exponential…” -> “Taking exponential”.

Author response:
`Minor remarks`: Thank you for the detailed writing suggestions and pointing out the typos, we have corrected them in the paper revision (marked in blue).

Reviewer comment (Writing):
Page 16: “acivations” -> “activations”.

Author response:
`Minor remarks`: Thank you for the detailed writing suggestions and pointing out the typos, we have corrected them in the paper revision (marked in blue).