Paper: Exploring the Promise and Limits of Real-Time Recurrent Learning (ICLR-2024)

Reviewer comment (Experiments):
One concern I have is that the authors do not convincingly demonstrate the benefit of their model compared to the status quo, which is a fully connected RNN + truncated BPTT. In terms of complexity, the authors do reduce the memory/computational cost on pointwise RNNs to order N^2 which is certainly better than RTRL on standard RNNs, but this is still as expensive as BPTT on a fully connected RNN. In fact, for a pointwise RNN I wonder if BPTT is in fact of order N (as the forward pass?) - so still cheaper than RTRL - but I may be incorrect there. In terms of performance, the authors show that eLSTM + RTRL outperforms truncated BPTT on all RNN architectures for one task (watermaze) which is impressive, but I would have liked to see this for more tasks; e.g. can RTRL outperform fully connected RNN + BPTT on any of the 5 Atari tasks in Fig 2?

Author response:
> *One concern I have is that the authors do not convincingly demonstrate the benefit of their model compared to the status quo, which is a fully connected RNN + truncated BPTT. In terms of complexity, the authors do reduce the memory/computational cost on pointwise RNNs to order N^2 which is certainly better than RTRL on standard RNNs, but this is still as expensive as BPTT on a fully connected RNN. In fact, for a pointwise RNN I wonder if BPTT is in fact of order N (as the forward pass?) - so still cheaper than RTRL - but I may be incorrect there.*

No, that is not correct: BPTT for eLSTM remains the same as the standard RNN. We had also provided a review/background material on BPTT in Appendix A.3 in case this might be helpful.

> *In terms of performance, the authors show that eLSTM + RTRL outperforms truncated BPTT on all RNN arhitectures for one task (watermaze) which is impressive, but I would have liked to seen this for more tasks; e.g. can RTRL outperform fully connected RNN + BPTT on any of the 5 Atari tasks in Fig 2?*

Yes, eLSTM + RTRL outperforms the LSTM + TBPTT baseline on 3 / 5 tasks (Gravitar, Q*Bert, Seaquest) and perform similarly on the two others. The scores (corresponding to Fig 2) were provided in Table 5 in the appendix. That said, the most important result is that of Watermaze which is the hardest task we have in terms of memory.

Reviewer comment (Experiments):
Related to the above, it is not clear whether online learning has any intrinsic benefits for performance. The authors note that online learning "allows for updating weights immediately after consuming very new input", but in practice the weights are only updated at the same rate as truncated BPTT (at rate M) to avoid 'sensitivity matrix staling'. I appreciate this is a difficult question to answer, and I believe this shortcoming is mentioned as the last presented limitation, but I think this deserves more direct clarification/exploration given the subject of this work.

Author response:
> *Related to the above, it is not clear whether online learning has any intrinsic benefits for performance. The authors note that online learning "allows for updating weights immediately after consuming very new input", but in practice the weights are only updated at the same rate as truncated BPTT (at rate M) to avoid 'sensitivity matrix staling'.*

First of all, we’d like to clarify that as we wrote in Sec 3.2 (last sentence): *“The main focus of this work is to evaluate learning with untruncated gradients, rather than the potential for online learning.“* That said, a potential benefit of *more online* learning is to be more sample efficient. At least on DMLab Select-Non-Matching-Object (Figure 1 (a) vs (d)), we do observe some sample efficiency benefits of RTRL with a small M: at x = 2.5M steps, RTRL with M=10 achieves a score over 50, while the one with M=100 is below 40 at the same number of steps.
However, if the goal is to obtain the best performing model, our conclusion is that a too small M is indeed not a good idea (more online learning deteriorates the performance, likely due to sensitivity staling).

Reviewer comment (Experiments):
All main results presented are with respect to performance in reinforcement learning (RL) tasks, but what about supervised learning tasks? e.g. sequence modeling in language. The authors do use the 'copy task', which is a supervised learning task, as a diagnostic task, but I would have liked to see comparisons with BPTT for hard supervised learning tasks. Or at least a rationale for only considering RL tasks.

Author response:
> *All main results presented are with respect to performance in reinforcement learning (RL) tasks, but what about supervised learning tasks? e.g. sequence modeling in language. The authors do use the 'copy task', which is a supervised learning task, as a diagnostic task, but I would have liked to see comparisons with BPTT for hard supervised learning tasks. Or at least a rational for only considering RL tasks*

We agree we all want to see RTRL applied to hard supervised learning tasks or any tasks in general. But that is the very challenge of the RTRL research. This is exactly what we discuss in the 3rd and 4th paragraph of Appendix B1 starting from *“Remarks on language modelling and supervised learning”*. There, we essentially explain that (1) with the one-layer constraint, it is hard to obtain competitive models on meaningful supervised learning tasks (and alike), e.g., language modelling, and (2) TBPTT implementations used for supervised sequence learning tasks are highly optimised; we argue that, to be competitive with them, it requires both engineering (custom CUDA implementation) and algorithmic (referring to hybrid RTRL-BPTT algorithms we discuss in Appendix A.4) efforts.
With RL tasks, (1) is not really an issue as we can still obtain good/competitive systems even with one-layer models (compared to the well-known IMPALA and R2D2 baselines).
The second point (2) is not just a limitation of our work but an open challenge for RTRL itself, which we honestly discussed in the paper (reflecting the title of our paper). This is also our unique contribution, which can not be found in any prior work on RTRL. We really hope this perspective improves the reviewer’s view on the contribution of our work, currently rated as low as 2.

Reviewer comment (Theory):
The connection to biology and the 'memory' trace (e.g. Eqs 12-14) is interesting but presently quite vague/weak. Some references to biological evidence for such traces (e.g. eligibility traces) would be advised.

Author response:
> *The connection to biology and the 'memory' trace (e.g. Eqs 12-14) is interesting but not presently quite vague/weak.*

Yes, we’ll add some references there for the final version. If the reviewer has any specific suggestions, that’ll be also welcome.

Reviewer comment (Evaluation):
Given this is a paper which explores the practical/empirical outcomes of RTRL, I would advise a more thorough analysis of the ideal conditions for RTRL. For example, why do the authors believe RTRL is only valuable in some of the 5 Atari environments in Fig 2? For what types of task would RTRL be ill-advised? The space for technical description and equations in the main text is significant (e.g. equations 8-14) and in general I believe this should be in part sacrificed (to the Appendix) for more interpretability of the results.

Author response:
> *Given this is a paper which explores the practical/emperical outcomes of RTRL, I would advise a more thorough analysis of the ideal conditions for RTRL. For example, why do the authors believe RTRL is only valuable in some of the 5 Atari environments in Fig 2? For what types of task would RTRL be ill-advised?*

The answer is actually very simple: in any tasks that benefit from a recurrent policy, there is no reason not to use RTRL (if it is tractable as in our case). The benefit is expected to be larger if a task requires longer time-span credit assignments. Strictly speaking, there is no case where RTRL is *“ill-advised,”* but obviously, if a task is purely reactive, RTRL, and also any recurrent policy anyway, is unnecessary. Empirically, in all our experiments, we never observed any critically bad performance using RTRL.

Reviewer comment (Theory):
The authors state that, "in practice, it is known that M > 1 is crucial (for TD learning of the critic) for optimal performance". Is this true? My belief was that one-step TD learning was frequently employed. Is there a relevant citation?

Author response:
> *The authors state that, "in practice, it is known that M > 1 is crucial (for TD learning of the critic) for optimal performance". Is this true? My belief was that one-step TD learning was frequently employed. Is there a relevant citation?*

For example, you could check the A3C paper/Sec 3. (Mnih et al. 2016 “Asynchronous Methods for Deep Reinforcement Learning” https://arxiv.org/abs/1602.01783), and maybe also simply Sutton and Barto’s textbook/Chapter 7 in general, or any other recent papers using actor-critic methods (e.g., the baseline IMPALA paper we cite).

Reviewer comment (Writing):
Typos: Mozer (1989; 1991) already explore an RNN -> Mozer (1989; 1991) already explored/s an RNN; several modern RNN architectures such Quasi-RNN -> several modern RNN architectures such as Quasi-RNN

Author response:
> *Typos ...*

Thank you very much! We’ll fix them.

Reviewer comment (Experiments):
Most of the experimental results (all except baseline comparison) seem to some extent obvious to me: the only difference between TBPTT to RTRL is an extended context length. It is thus not surprising that RTRL > TBPTT in all experiments. The context length for TBPTT is relatively short (it seems far smaller than the one modern GPUs would offer) so those experiments do not seem to answer the question:  
> In what scenarios would RTRL be able to replace BPTT in today’s deep learning?  
asked by the authors. Adding a baseline that uses as much context length as possible would help better contextualize the findings of the paper. To me, this is the main weakness of the paper, and fixing it may require substantial changes.

Author response:
We thank the reviewer for valuable time reviewing our work.
While we also thank the reviewer for acknowledging certain strengths of our work, we also would like to draw the reviewer’s attention to other important aspects of our work which seem to be currently overlooked.

> *Most of the experimental results (all except baseline comparison) seem to some extent obvious to me: the only difference between TBPTT to RTRL is an extended context length. It is thus not surprising that RTRL > TBPTT in all experiments.*

We’d like to clarify that the goal of research on RTRL is not to *“surprise”* the readers; we all know that RTRL should be “better” but the whole challenge of any practical RTRL research is to build an actual system trained by RTRL that exhibits practical improvements. 

>  *The context length for TBPTT is relatively short (it seems far smaller than the one modern GPUs would offer) so those experiments do not seem to answer the question: In what scenarios would RTRL be able to replace BPTT in today’s deep learning? asked by the authors. Adding a baseline that uses as much context length as possible would help better contextualize the findings of the paper. To me, this is the main weakness of the paper, and fixing it may require substantial changes.*

We’d like to clarify two important points here.

First of all, our experiments on DMLab memory tasks (Table 1) already show two contrastive cases. On Select-Non-Matching-Object (where the mean episode length is short; about 100 steps) TBPTT with a large M is nearly equal to full BPTT: consequently we only observe very limited improvements with RTRL. On the other hand on Watermaze (where the mean episode length is about 1000 steps; in Appendix B.2 DMLAB, paragraph *“Practical Computational Costs”*, we discuss the corresponding memory requirements), clear benefits of RTRL are observed (Table 1). Overall, RTRL is effectively promising for tasks with long-span dependencies, and rather irrelevant otherwise.

Now regarding the question “In what scenarios would RTRL be able to replace BPTT in today’s deep learning?”, the reviewer is missing one very important perspective of our work.
We formulated this as a fundamental question that anybody interested in RTRL should ask.
We do not intend to answer this question only through these experiments (not because something is missing in our experiments; but because the scope of this question is much broader). Rather, our goal is to encourage more broad thinking about RTRL in the scope of today’s (and maybe future) machine learning, going beyond the current focus on approximation methods. 
It is absolutely true that for many tasks (like Select-Non-Matching-Object here) full backpropagation spans are short enough to be fitted to current GPUs. This will be the case for even more tasks in the future with advances in the hardware. This is exactly what we discuss in Sec 5 paragraph *“Principled vs. practical solution”*. In a similar spirit, we also discuss the limitations and challenges of RTRL in supervised learning (3rd and 4th paragraphs in Appendix B1 starting from *“Remarks on language modelling and supervised learning”*).

Overall, all these honest questions and discussions bring new perspectives on RTRL research which are completely ignored in prior work on RTRL. “Promise” and “limits” in our title reflect this.
We argue that this is an important and novel contribution, which, we believe, merits more than a contribution score of 2.

Reviewer comment (Novelty):
The paper is missing a few relevant references:  
- Bellec et al. 2017: RTRL from a computational neuroscience perspective. One of the main contributions of the paper is to discuss how RTRL can be implemented biologically through eligibility traces.  
- Gupta et al. 2022: show that it is possible to parametrize some classes of linear RNNs (more precisely linear SSMs) diagonally, without losing too much performance. Orvieto et al. 2023, which is cited in this paper, build on this result.  
- Zucchet et al. 2023: show that spatial backpropagation combined with element-wise complex-valued recurrence enables learning multilayer networks with a performance close to the one of BPTT.

Author response:
> *The paper is missing a few relevant references ...*

Thank you very much for pointing out these papers. We'll cite them in our revision (except Javed et al. 2023 which is already cited), and add comments on existing efforts to deal with the multi-layer case.

Reviewer comment (Novelty):
The paper mentions the multilayer case as an important limitation but does not discuss any possible or existing solutions (layer-wise training of Javed et al. 2023 / spatial backpropagation of errors of Zucchet et al. 2023).

Author response:
> *The paper focuses on one layer, but mentions that multiple layers are key in deep learning. Would the experimental results presented here would benefit from RNNs with multiple layers?*

Yes, we observed slight improvements with deeper models (see the next paragraph) but here, we intentionally focused on RL tasks where we can still achieve good performance (compared to well-known IMPALA and R2D2 baselines) even with one layer. This does not contradict our statement on the general importance of deep models though. Also generally, a learning algorithm that is only exact and efficient for one-layer models is too limited.

In our preliminary studies, we evaluated a baseline LSTM (trained with TBPTT) on DMLab-30 Watermaze with 1 or 2 layers: we found the 2-layer model (score 39.7 +- 4.6) to slightly outperform the 1-layer model (37.6 +- 5.3) while both of them underperform our 1-layer eLSTM (45.6 +- 4.7).

Reviewer comment (Experiments):
Only one element-wise RNN instantiation, namely the eLSTM was provided for comparison. It would be better to show the generality of the method on some other element-wise RNN instantiations.

Author response:
We had to make a decision given our limited compute resource. As we expressed in the main text Sec 3.1 (*“While one could further discuss myriads of architectural details (Greff et al., 2016), most of them are irrelevant to our discussion on the complexity reduction in RTRL; the only essential property here is that “recurrence” is element-wise.”*), all these existing element-wise RNN architectures are similar up to certain details. Therefore, instead of evaluating various architectures with small differences, we opted for conducting experiments across many tasks. We hope this clarifies the logic of our decision.

Reviewer comment (Experiments):
The experiments are conducted on reinforcement learning tasks, which is fine as it is stated clearly as the study goal. However, it would be much more convincing if it is shown with certain supervised learning tasks.

Author response:
This is a natural remark which we actually discuss in the paper (3rd and 4th paragraphs in Appendix B1 starting from *“Remarks on language modelling and supervised learning”*). There, we essentially comment that (1) with the one-layer constraint, it is hard to obtain competitive models on meaningful supervised learning tasks (and alike), e.g. language modelling, and (2) TBPTT implementations typically used for supervised sequence learning tasks are highly optimised; we argue that, to be competitive with them, it requires both engineering (custom CUDA implementation) and algorithmic (referring to hybrid RTRL-BPTT algorithms we discuss in Appendix A.4) efforts. This is not just a limitation of our work but an open challenge for RTRL itself, which we honestly discuss in the paper, reflecting the title of our paper. Such discussions are also our unique contributions, which can not be found in any prior work on RTRL. We really hope this perspective improves the reviewer’s view on the contribution of our work, currently rated as low as 2.

Reviewer comment (Evaluation):
The variances shown in Figure 2 across different Atari environments are not consistent. For example, in Gravitar the variance for RTRL is larger than the other two and it’s the opposite in Seaquest. But overall it seems RTRL has a larger variance. Could the authors provide some explanation or intuition for that?

Author response:
Right, these variances do not have to be consistent. Our intuition is as follows. There are two key phenomena: discovering a key credit assignment within a trajectory, and encountering “good” trajectories containing such credit assignment.  RTRL should facilitate the former (thanks to untruncated gradients) as long as the latter is successful. The latter, experiencing good “trajectories” containing the relevant credit assignment paths, depends on the nature of the environment in particular. We speculate that it happens to be easy to encounter good trajectories in Seaquest, allowing RTRL to consistently find good strategies (low variance), while this does not seem to be the case for Gravitar where RTRL can yield good performance only when “good” trajectories are found (high variance). This may be a possible explanation of what we observe there.

Reviewer comment (Novelty):
The elementwise recurrence that the authors propose for eLSTM is very similar to that used in IndyLSTM [1], but a comparative discussion and citation is missing.

Author response:
Yes, the reviewer is right. We listed/cited many of them in Sec 3.1., including “IndLSTM” which even has almost the same name, but this one was indeed missing. Thank you for pointing this out; we’ll cite it.
We emphasise that they are all “similar” but not identical, and eLSTM has a minimalistic design.

Reviewer comment (Experiments):
Since exact RTRL computes exactly the same gradient as full BPTT, a specific analysis of why TBPTT rather than BPTT is used in each of the demonstrated tasks would be important to provide context (for e.g. in regards to memory requirements).

Author response:
We primarily use TBPTT as it is the standard algorithm used to train recurrent policies in general. As we also wrote in the main text (paragraph “Principled vs. practical solution” in Sec. 5), in certain tasks such as DMLab Select-Non-Matching-Object, TBPTT with a large M is nearly full BPTT. Regarding the memory benefits of RTRL, we discuss its details in Appendix “B.2 DMLAB” paragraph “Practical Computational Costs“.

Reviewer comment (Theory):
A discussion/analysis of the computational expressivity of having restricted recurrent and its implications is missing.

Author response:
We actually discuss this in Appendix “B.1 DIAGNOSTIC TASKS” (Page 22), paragraph “Remarks on computational capabilities of eLSTM.” In particular, we refer to Merrill et al. ACL 2020’s theoretical work on quasi-RNNs’ expressivity (which is a representative element-wise RNN). That said, we admit that this paragraph should be hard to find currently. We will add a sentence in the main text pointing to this appendix section so that people can find this (and we likely should make this an independent subsection instead of a paragraph).

Reviewer comment (Theory):
A discussion (or even speculation) of why eLSTM works as well as it does given its limited recurrence would be useful.

Author response:
One speculation is that element-wise recurrence is easier to learn as it does not involve any full matrix-vector multiplication for temporal/recurrent dependencies, making it more sample efficient. But this remains a tricky question.

Reviewer comment (Experiments):
Certain combinations of baselines are missing: eLSTM with SnAP-1, full BPTT.

Author response:
There should be some confusion here: “SnAP-1” and “full BPTT” are both learning algorithms which can only be used exclusively. Also, the combination “eLSTM with SnAP-1” does not exist: SnAP-1 is an approximation method for the fully recurrent models, which does not apply to eLSTM whose recurrence is element-wise. As far as we can tell, our current Table 2 is complete: it already covers all interesting/relevant cases.

Reviewer comment (Evaluation):
Why does IMPALA with TBPTT+eLSTM do better than standard IMPALA (with LSTM+TBPTT presumably?)

Author response:
One speculation is that element-wise recurrence is easier to learn as it does not involve any full matrix-vector multiplication for temporal/recurrent dependencies, making it more sample efficient. But this remains a tricky question.

Reviewer comment (Experiments):
Is having a small value of M beneficial for RTRL in any other way? Not clear what the motivation for studying the effect of M is.

Author response:
A potential benefit of small M is *more online* learning which could be more sample efficient. At least on DMLab Select-Non-Matching-Object (Figure 1 (a) vs (d)), we do observe some sample efficiency benefits of RTRL with a small M: at x = 2.5M steps, RTRL with M=10 achieves a score over 50, while the one with M=100 is below 40 at the same number of steps.
However, if the goal is to obtain the best performing model, our conclusion is that a too small M is indeed not a good idea (more online learning deteriorates the performance, likely due to sensitivity staling).
Overall, studying/evaluating the effect of M here is natural. We also note that it is rare that we can conduct such experiments using the *exact* RTRL without worrying about the approximation quality, which is a clear benefit of our framework.

Reviewer comment (Experiments):
The specific tasks the authors test on is not described anywhere in the paper (the Appendix points to a webpage). Having this would help a lot with understanding the tasks used in the paper.

Author response:
We originally skipped these descriptions as they will be very long given the number of tasks we cover. But if the reviewer thinks they are really necessary, we’ll be happy to add the corresponding task descriptions to the appendix. 
Regarding the full details about the external baseline models, we refer to the corresponding papers.

Reviewer comment (Reproducibility):
The differences between the baseline methods of various variants of IMPALA and R2D2 are not sufficiently described.

Author response:
We originally skipped these descriptions as they will be very long given the number of tasks we cover. But if the reviewer thinks they are really necessary, we’ll be happy to add the corresponding task descriptions to the appendix. 
Regarding the full details about the external baseline models, we refer to the corresponding papers.