Paper: Learning to Optimize for Reinforcement Learning (ICLR-2024)

Reviewer comment (Experiments):
The inability of LinearOptim to learn anything is very surprising and should be investigated further to ensure it is not erroneous.

Author response:
- **The inability of LinearOptim to learn anything is very surprising and should be investigated further to ensure it is not erroneous.** We checked the implementation again and ensured it was not erroneous.

Reviewer comment (Novelty):
There is a broad base of related work on meta-learned RL objective functions, which is not discussed or compared against. While these are a different class of inductive bias, they are solving the same problem as the class of meta-optimizers discussed here. Notably, the evaluation procedure and environments are from Oh et al. (2020), a learned objective algorithm, but it is not compared against. While it is understandable to not compare Optim4RL against all of these methods, they should at least be discussed as alternative approaches to the same problem in the related work. Namely, EPG (Houthooft et al., 2018), LPG (Oh et al., 2020), MetaGenRL (Kirsch et al., 2020), ML^3 (Bechtle et al., 2021), SymLA (Kirsch et al., 2022), DPO (Lu et al., 2022), GROOVE (Jackson et al., 2023).

Author response:
- **Add more citations, modify misleading claims, and correct typos.** We will add those citations and fix the typos and misleading claims.  
R3: Dear Reviewer,

We did not find the mentioned related work GROOVE (Jackson et al., 2023). Could you please provide more details about this work?

Thanks.

Reviewer comment (Novelty):
Many of the claims are misleading or ambiguous. In conclusion, the claim that Optim4RL is "the first learned optimizer that can be meta-learned to optimize RL tasks entirely from scratch" is confusing, since all existing learned optimizers can be and are applied to RL in this paper. If this is intended to claim this is the first meta-learned optimizer designed for RL, then the omission meta-learned objective function literature becomes even more apparent, since there is extensive work solving the same problem for RL.

Author response:
- **Add more citations, modify misleading claims, and correct typos.** We will add those citations and fix the typos and misleading claims.

Reviewer comment (Presentation):
Transformation of the gradient output is a major component of gradient processing, but only the input transformations are discussed in the main body. An expanded form of the output could be presented in the main body.

Author response:
- **Transformation of the gradient output is a major component of gradient processing. An expanded form of the output should be presented.** We will add more explanations for this part.

Reviewer comment (Experiments):
The reviewer notes that in Figure 4 (b), STAR appears to be starting to learn, and asks whether with more hyper-parameter tuning it could match other optimizers in performance.

Author response:
**In Figure 4 (b) it seems like STAR is starting to learn a bit. Is it possible that with a bit more hyper-parameter tuning it would match the other optimizers in performance?** We tuned STAR with more hyper-parameter options and improved its performance (average return) to 10; however, this is still much worse than the performance of our method (average return>20).

Reviewer comment (Experiments):
Resetting provides the optimizer data at different training stages. Have the authors analyzed how different values of m and n impact this? Do we require them to be equal all the time?

Author response:
m and n are not required to be equal all the time. In our experiments, n is the number of training environments that depends on the task itself; m is affected by the training steps of each task and it has a similar magnitude as n. We leave the study of the performance impact of m and n as future work.

Reviewer comment (Writing):
(Minor) Section 4.1: SeeAppendix B <= missing a space.

Author response:
We will add those citations as well as fixing the typos.