Paper: Gated recurrent neural networks discover attention (ICLR-2024)

Reviewer comment (Experiments):
More discussion and analysis around some of the results would strengthen the paper. In particular, the compression result from Figure 3.B where the gated RNNs can solve the linear regression task with a size smaller than the theoretical construction size. Are there other tasks where this is not the case? E.g. perhaps the associative recall task from the point above? More analysis and experimentation around this point would strengthen the paper.

Author response:
-   For the first point, we refer the reviewer to our global response. Regarding the second point, we agree with the reviewer that this is a very intriguing result. We have been investigating the reasons behind this performance but so far remained unsuccessful. We believe that this is an interesting direction for future work.

Reviewer comment (Experiments):
My enthusiasm is tempered by the rather extreme limitations placed on both the Transformers and the GLRUs in this paper.

Author response:
We appreciate the reviewer’s valuable time in reviewing our paper and would like to nuance their point of view on the limitations imposed on the two architectures we consider.

- As argued in the global response, there is a growing literature showing how extensions of linear self-attention layers can reach, if not surpass, Transformers-level performance. The interest of linear self-attention thus goes beyond its amenability to theoretical analysis.
- While we provide a theoretical construction for the linear case in the main text, we show how this still holds when the gating is nonlinear in Appendix A.1. We agree with the reviewer that it makes learning more difficult, as shown in Figure 4.A. However, we think that it still helps understand the inductive bias behind architectures such as LRUs, by showing that they are more akin to reproducing linear self-attention like behaviors than other types of RNNs.

We made clearer the reasons why we think our paper can have a great impact in the global response. We encourage the reviewer to consider those arguments and hope they may convince them to change their score.

Reviewer comment (Theory):
Linear self-attention is far less powerful than softmax self-attention, and as demonstrated in this paper, linear gated recurrence is less powerful than nonlinear gated recurrence.

Author response:
We appreciate the reviewer’s valuable time in reviewing our paper and would like to nuance their point of view on the limitations imposed on the two architectures we consider.

- As argued in the global response, there is a growing literature showing how extensions of linear self-attention layers can reach, if not surpass, Transformers-level performance. The interest of linear self-attention thus goes beyond its amenability to theoretical analysis.
- While we provide a theoretical construction for the linear case in the main text, we show how this still holds when the gating is nonlinear in Appendix A.1. We agree with the reviewer that it makes learning more difficult, as shown in Figure 4.A. However, we think that it still helps understand the inductive bias behind architectures such as LRUs, by showing that they are more akin to reproducing linear self-attention like behaviors than other types of RNNs.

We made clearer the reasons why we think our paper can have a great impact in the global response. We encourage the reviewer to consider those arguments and hope they may convince them to change their score.

Reviewer comment (Theory):
Is there any reason to believe that the demonstrated equivalence would continue to hold for neural nets that include nonlinearities?

Author response:
While we provide a theoretical construction for the linear case in the main text, we show how this still holds when the gating is nonlinear in Appendix A.1. We agree with the reviewer that it makes learning more difficult, as shown in Figure 4.A. However, we think that it still helps understand the inductive bias behind architectures such as LRUs, by showing that they are more akin to reproducing linear self-attention like behaviors than other types of RNNs.

Reviewer comment (Presentation):
In Figure 2, the statement "The block structure almost perfectly matches the one of our construction" is unclear as the reviewer does not understand the block structure being referred to.

Author response:
2. In our construction the input and output gating matrices have a characteristic 2x2 block structure. For example, if we look at the input gating, one of the matrix has top right and bottom left blocks that are equal to 0. We recover the same structure in $W_x^\mathrm{in}$ in Figure 2.A. The same thing holds for the 4 other weights of the gating matrix.

Reviewer comment (Presentation):
In Table 2, the meaning of the terms $x_i^j y _1$ for different $i, j$ is unclear; notations would help readers parse the results of the probing experiments.

Author response:
4. Indeed, the caption was not sufficiently explaining the table. The input at time $t$ consists in the concatenated vectors $x_t=(x_{t,1},x_{t,2},x_{t,3})^\top$ and $y_t=(y_{t,1},y_{t,2},y_{t,3})^\top$. The instantaneous function for each output neuron can implement a polynomial of degree 4 in these terms. The table shows the coefficient associated to the terms of the polynomial implemented by the first neuron after training. Interestingly, the only terms without a negligible coefficients are $x_{t,1}^2y_{t,1}, x_{t,2}^2y_{t,1}$ and $x_{t,3}^2y_{t,1}$, where $x^2$ refer to $xx$. This is virtually identical to the polynomial implemented by a single step gradient descent algorithm (GD), as shown in the table.

Reviewer comment (Experiments):
The experiments conducted in sections 2 and 5 are with a fixed dimension. The reviewer asks how the loss behavior changes with different parameter counts at different dimensions, and suggests that such a plot can give an empirical dependence on the order of parameters necessary with dimension.

Author response:
(b) In all our experiments, we vary the number of neurons and as, a consequence, the number of parameters. The reviewer might be interested in Figure 5 in the appendix, in which we look at how the loss behaves as a function of the number of parameters of the different models. We are happy to provide more information if needed.

Reviewer comment (Experiments):
The linear regression experiments show that with sparsity in the key-value matrix, the gated RNN models can simulate more efficiently than the theoretical construction. The reviewer suggests it would be interesting to conduct similar experiments in section 2, where the authors impose low-rank/sparse constraints on the key-value matrix product and observe the empirical behavior of loss with different parameter counts.

Author response:
(c) We thank the reviewer for this great suggestion. We report the result of this experiment in the global answer to all reviewers.

Reviewer comment (Theory):
Section 3.2 discusses the invertibility of the value matrix per the number of hidden neurons the RNN needs to store KVs. Under which conditions is this matrix invertible?

Author response:
**Invertibility of the value matrix.** We require the matrix $W_V$ to be invertible to decrease the number of neurons required by our construction, which of course depends on the self-attention layer we want to replicate. See the global discussion for a more comprehensive discussion on this topic.

Reviewer comment (Theory):
In Section 4.2, the authors describe overparameterization insofar as twice as many neurons are needed to replicate the behavior of self-attention with the RNN construction. What might the effect of regularization be here, implicit or otherwise?

Author response:
**Section 4.2.** As highlighted in the previous point, not all recurrent neurons end up being used at the end of learning. However, having those extra neurons is important for learning as some neurons will get “killed” during learning. This is why overparametrization (in the sense of more recurrent neurons than the construction) is needed and we empirically observe that having twice as many neurons as in the construction is enough to reach 0 training loss. Yet, it turns out that only the neurons from the construction are actually used.