Paper: Discovering Minimal Reinforcement Learning Environments (ICLR-2024)

Reviewer comment (Experiments):
The work is often not clear enough in which parts are coming from the framework proposed by Ferreira et al. It took time to understand that the only change to the framework is the idea of learning an initial state distribution (with fixed dynamics) rather than learning the dynamics. This lack of clarity is particularly highlighted in the experiments where “the plain T setup” is declared as similar to Ferreira et al. but it is not elaborated on what the exact differences are.

Author response:
>The work is often not clear enough in which parts are coming from the framework proposed by Ferreira et al. It is not elaborated on what the exact differences [between "the plain T setup" and Ferreira et al.] are. 

We have made the distinction clearer throughout the paper.

In our experiments, its necessary to recreate Ferreira at al’s method within our framework, since we need to leverage our highly parallelizable implementation of RL algorithms to enable experiments on Brax. This means the “plain T setup” differs in the implementation of RL algorithms. We added clarification to the corresponding parts of the text.

Reviewer comment (Experiments):
In the experiments there is no clear comparison to the prior work and one is left to guess the actual performance differences.

Author response:
>Further, in the experiments there is no clear comparison to the prior work and one is left to guess on the actual performance differences.

To our best knowledge, there is no work on meta-learning synthetic environments besides Fereirra et al., who only present results for CartPole and Acrobot. We benchmark against our implementation of their method in figure 3.

Reviewer comment (Presentation):
Figure 3 most often does not show the learning curve for the proposed training setup. Only for hopper and Mountain car does it show the actual learning curve of the proposed IC training setup, but for all other ablations only a dashed line of the final performance is shown. This omission of the learning curves seems rather strange and requires an explanation.

Author response:
>Figure 3 most often not show the learning curve for the proposed training setup.

With the IC learning curve in each plot, they easily become cluttered. However, we acknowledge that this might make the plot less convincing, and created a new version including the curves with improved clarity. In Pendulum, we could not think of a sensible curriculum to apply, and therefore did not use one. We added a clarifying remark in the figure description.

Reviewer comment (Presentation):
In Figure 3 the 95% confidence intervals seem wholly missing for some or even all methods.

Author response:
>[In figure 3] often the 95% confidence intervals seem wholly missing for some or even all methods. 

Because of the high computational cost, we only have one run per ablation in hopper, so no confidence intervals can be drawn.

Note that while meta-evolution may only use a single seed, evaluation during meta-training uses 64 seeds.

Reviewer comment (Experiments):
In the ablation shown in Figure 3, it seems strange to not include “no curriculum” as an option when showing the impact of different curricula. This is an important baseline to show. Although the difference between using curricula and no curricula is shown in the first row, it is not easy to visually compare the ablated element in this way.

Author response:
>Lastly, in this ablation it seems rather strange to not include “no curriculum” as an option when showing the impact of different curricula. 

We have included your suggestion about including “no curriculum” in the last row of figure 3.

Reviewer comment (Writing):
The writing could be improved as statements are often not easy to follow. For example, early in Section 3 it is stated that learning the transition dynamics is not as useful as only learning an initial state distribution, but no reasons are given as to why. At least a reference to later experiments would have shown that this statement is supported.

Author response:
>The writing could be improved as often statements are made which are not easy to follow. 

We have improved the clarity of writing in a lot of places, thank you for your suggestions.

Reviewer comment (Experiments):
The reviewer asks how much tuning is necessary to achieve good results.

Author response:
>How much tuning is necessary to achieve good results?

Our approach is relatively robust to outer loop hyperparameters. For example, we use the same hyperparameters for all Gym and small Brax environments. The exception is Pendulum, which needs a more stable fitness evaluation due to the very random nature of returns induced by favorable or unfavorable initial states. This leads to an increased number of rollouts per population member, and thereby a decreased population size due to memory constraints.

Reviewer comment (Theory):
A contextual bandit (CB) can be converted to a Markov decision process (MDP), but not vice versa, because CB is stateless. This means that the synthetic contextual bandit (SCB) model may not be able to accurately capture the dynamics of more complex environments, such as Go, where state is essential for planning and decision-making.

Author response:
While environments with temporal dynamics are better modeled by MDPs, training a agent on a CB can imprint the policy of the optimal agent on the MDP. This can be done by setting the reward function to the value function of the optimal agent, and sampling the observation space sufficiently. By maximizing the reward in the CB, the agent will therefore maximize the value in the real environment. We recognize that this point was not addressed clearly enough in the paper, but the method remains general to MDPs, including Go. We added an explanation of this relation in our methods section (see "synthetic environment setup").

Reviewer comment (Theory):
It is unclear how a synthetic contextual bandit can work for partial observation environments, where the agent does not have access to all of the state information.

Author response:
Partial observations can be handled in the same way, since the CB can present partial observations.

Reviewer comment (Experiments):
It is unclear how the SCB model would perform in partial observation environments.

Author response:
Partial observations can be handled in the same way, since the CB can present partial observations.

Reviewer comment (Experiments):
There is no comparison with the state-of-the-art results on the environments/tasks in this paper. It is understandable if the proposed method does not outperform the state of the art, as the inner policy optimization can be different. However, it would be interesting to see how the proposed method compares to other methods on the same tasks.

Author response:
We are using common SotA RL algorithms in the inner loop, and compare to the current SotA work on meta-learning synthetic environments by Ferreira et al. For a comparison to training directly on the evaluation environment please refer to figure 1.

Reviewer comment (Novelty):
The motivation for using a synthetic environment is not entirely clear. In the first paragraph of the introduction, the authors mention the biological fact that organisms can learn from artificial stimuli. However, it is not clear how this relates to the use of synthetic environments in reinforcement learning. Additionally, the authors do not explain why learning or meta-learning RL policies is not sufficient.

Author response:
In the biological analogy, artificial stimuli correspond to the synthetic environment, while the real world corresponds to the evaluation environment. Before birth, the agents are pretrained in a way that transfers well to the real world, similar to our synthetic training, which transfers well to the evaluation environment.

Besides this biological analogy, we list several motivations for training a synthetic environment, including computationally efficient simulation, faster RL training, no need for hyperparameter tuning, and the application to several downstream tasks.

Learning RL policies on an evaluation environment directly is sufficient for achieving a well-performing agent. However, training synthetic environments has several additional benefits, such as enabling the downstream tasks we present in section 6.

Reviewer comment (Presentation):
The colors in Figure 5 should be revised so that synthetic curves have the same color, and the same for the real curves. This would make the figure easier to read and interpret.

Author response:
We have changed the colors of figure 5 to be consistent.

Reviewer comment (Evaluation):
Important claims made at the beginning of the paper, such as improved wall-clock time, are not substantiated.

Author response:
> Important claims made at the beginning of the paper, such as improved wall-clock time, are not substantiated.

We show a significant increase in simulation speed and an example of wall clock training curves for halfcheetah in figure 1. We added more information on these experiments to the caption.

Reviewer comment (Writing):
Section 2 (Curricula): you state that you learn a curriculum in this section, but in contribution 1 it seems that you in-fact design a curriculum. These two claims seem contradictory.

Author response:
> Section 2 (Curricula): you state that you learn a curriculum in this section, but in contribution 1 it seems that you in-fact design a curriculum.

In section 2 we state that "we use meta-optimization to discover state reset and state-action-dependent reward functions maximizing downstream learning progress". The reset and reward functions are not inner loop curricula. The curriculum mentioned in our contributions refers to a curriculum on the outer loop. We changed the paragraph in the background section such that this becomes clear.

Reviewer comment (Novelty):
Section 2 (Synthetic Data): I understand that there is some prior work that attempts to train reinforcement learning algorithms from synthetic data, but I do not think this is well-motivated. Why not just use the real data? You need it for evaluation anyway to learn the synthetic enviornment, so you may as well use it during meta-learning as well.

Author response:
>Section 2 (Synthetic Data): Why not just use the real data [instead of synthetic data]?

In many cases synthetic data is used because it is cheaper to generate. After meta-training, this is also the case for our SCBs (see figure 1). We added this motivating example to the relevant paragraph in section 2.

Reviewer comment (Experiments):
In your experiments, is the meta-learned synthetic environment tuned to a specific real environment?

Author response:
>In your experiments, is the meta-learned synthetic environment tuned to a specific real environment?

We are unsure what the reviewer means by this question. In our experiments, we train a separate synthetic environment per target environment.

Reviewer comment (Evaluation):
Results, figure 3: why is there a discrepancy between the evaluation of the ablation and the results in Figure 2. Specifically for hopper? I understand ablations can be expensive, and may necessitate a smaller set of results, but this ablation does not explain the performance differences with respect to the original results.

Author response:
>Results, figure 3: why is there a discrepancy between the evaluation of the ablation and the results in Figure 2. Specifically for hopper?

Figure 2 shows the results of meta-training with sampled inner loop algorithms, while Figure 3 shows a fixed inner loop algorithm (see column title). We added this information to the caption to clear up any confusion.

Reviewer comment (Evaluation):
I do not see how synthetic environments provide any interpretability beyond what a learned value function can provide. It would be interesting to see whether the reward function meta-learned for synthetic environments differs in some qualitative way from that of a learned value function. But otherwise, this analysis does provide me with any additional insight.

Author response:
>I do not see how synthetic environments provide any interpretability beyond what a learned value function can provide.

When given a value function with the same parameterization as the synthetic reward, it can be analyzed in the same way. However, the two functions differ (see above).