Paper: Policy Rehearsing: Training Generalizable Policies for Reinforcement Learning (ICLR-2024)

Reviewer comment (Theory):
Based on the previous points, one of my major concerns is why the proposed methods can help optimize the gap calculated in Thm 3.3. The authors have summarized the three errors in Thm 3.3 as diversity and eligibility, which indeed provides insights for analyzing this problem. But I think a more direct connection, like whether the objective in Sec 3.3 can be proven to directly control the three errors in Thm 3.3, will make the analyses more solid.

Author response:
**A2**: The error term \$\\epsilon_e\$ requires the candidate dynamics to contain paths toward high rewards. We formulate this as a reinforcement learning task, by treating the dynamics as an agent and optimizing for the reward achieved by a fixed planner. Regarding \$\\epsilon_m\$, after building the connection between controlling this term and enhancing the diversity, we adopted an adversarial methodology which is proven to generate distinguished candidate models (Lemma 3.4). we will continue to explore how to optimize these errors more directly in future work.

Reviewer comment (Experiments):
In experiments, providing the results directly trained in the target environments as the reference will better show the results.

Author response:
**A3**: We trained expert policies directly in the three environments using SAC, and the relative performances (the performance of SAC minus that of the random policy) are listed in the following table. Although expert policies achieve higher scores, we find that ReDM still manages to obtain about 30% and 70% of the expert’s performances in MountainCar and Acrobot respectively. We think this is notable considering that ReDM requires no interaction with the environment. More details about this can be found in Appendix F. 

| Environment | grav=0.5 | grav=0.8 | grav=1.0 | grav=1.2 | grav=1.5 |
| -- | -- | -- | -- | -- | -- |
| InvertedPendulum | 993.5 | 991.8 | 994.8 | 992.5 | 993.8 |

| Environment | angle=2.0 | angle=2.5 | angle=3.0 | angle=3.5 | grav=4.0 |
| -- | -- | -- | -- | -- | -- |
| MountainCarContinuous | 128.7 | 128.4 | 128.5 | 128.4 | 106.8 |

| Environment | dt=0.10 | dt=0.15 | dt=0.20 | dt=0.25 | dt=0.30 |
| -- | -- | -- | -- | -- | -- |
| Acrobot | 375.2 | 414.7 | 428.6 | 435.9 | 445.6 |

Reviewer comment (Novelty):
Lack of some related works, like utilizing model-based methods for improving generalization [1-3], and finding diverse skills for unsupervised RL [4-6] as this work hopes to find diverse models.

Author response:
**A4**: Thanks for pointing out this. We updated Section 2 in the main text to discuss about papers that are most related to ReDM. It's worth noting that a detailed discussion about related works is deferred to Appendix G due to space limits. We also updated this appendix section to include papers about generalization and unsupervised RL based on your recommendation.

Reviewer comment (Writing):
In my opinion, the considered setting is that the agent can only get the reward function of the target task but has no knowledge about the dynamic of the target task. Is it right? Given the offline data, it is understandable that the agent can learn the dynamic to some degree. But without an offline dataset, it seems that there is no idea for the agent to learn the dynamic of the target task.

Author response:
**A1**: We would like to clarify that ReDM does NOT learn the target dynamics. ReDM generates diversified and eligible candidate models. ReDM then trains a meta-policy in these models, so that the meta-policy is able to generalize over different dynamics. Without any interaction data, the generated candidate models are expected to represent the whole model space. In such cases, the meta-policy can adapt to any environment including the real environment.

Reviewer comment (Experiments):
Based on the previous question, I'm confused about the setting of Experiment 4.1 "ReDM With no Interaction Data". As there are no data about the environment and the agent can not interact with the environment, how does the agent to learn about the environment?

Author response:
**A1**: We would like to clarify that ReDM does NOT learn the target dynamics. ReDM generates diversified and eligible candidate models. ReDM then trains a meta-policy in these models, so that the meta-policy is able to generalize over different dynamics. Without any interaction data, the generated candidate models are expected to represent the whole model space. In such cases, the meta-policy can adapt to any environment including the real environment.

Reviewer comment (Experiments):
As Unsupervised RL considers training an agent in the environment without reward, in my opinion, the setting in this work is like training an agent and models in the environment with reward but without dynamic. As the dynamic of the target environment will vary a lot, whether finetuning the agent (as well as the model) in the target environment with few steps will be more reasonable?

Author response:
**A2**: It is quite novel and interesting to interpret rehearsal as a kind of unsupervised RL without dynamics, and we have added a discussion about the relation in Appendix G. As for fine-tuning, it is worth noting that in Section 4.1 Figure 2, we conduct a fine-tuning experiment on InvertedPendulum, where we fine-tuned the learned meta-policy with online trajectories. Our meta-policy can achieve fast adaptation compared to directly learning from online trajectories.

Reviewer comment (Experiments):
About $r_e$ for Eligibility. The proposed method is to randomly sample N trajectories and estimate the biggest return. Is this inefficient as the state space and action space are continuous in experiments? Also, what is the choice of N in experiments?

Author response:
**A3**: We agree that a random shooting planner can be less efficient and there are more advanced methods that can be applied instead. However, for the purpose of concept-proof and simplicity, we show that with such a simple method ReDM can also yield good results. N is set to \$50\$. We have updated the hyper-parameter table in Appendix C.2.

Reviewer comment (Experiments):
I'm curious about the performance of ReDM in the D4RL setting (Sec. 4.3) but without any Interaction Data.

Author response:
**A4**: As the state dimensionality increases, ReDM requires much more computation power to generate sufficient many models for training an effective meta-policy. We will report the results on D4RL benchmarks. In these cases, we simultaneously explore ways to incorporate more constraints to reduce the model space, such as ReDM-o that can incorporates mismatching data.

Reviewer comment (Presentation):
The paper writing is not attractive. In my perspective, the main paper contains too much tedious content regarding the theoretical analysis and lacks an explanation for the rehearsal framework. My suggestion would be to move some theoretical content to the appendix and include at least one figure to explain the procedures of this new rehearsal framework and what it can achieve or why we need it. People don't care about the theoretical stuff until they are attracted by the idea and want to dive into it. Thus I suggest making some figures to explain the idea or the method.

Author response:
- **Using Figure 15 in Appendix H to explain the idea of policy rehearsing.** We do not use the figure to replace the theory part in our main paper currently since there are concerns about the theories from other reviewers. However, we are willing to reorganize the paper structure once these concerns are addressed.

Reviewer comment (Evaluation):
No standard deviation is included for experiments in Table 1. Also, there is no error bar in Figure 7.

Author response:
- **Adding standard deviation for Table 1.** However, we did not add an error bar to Figure 7 since the results are aggregated over different datasets and gravity levels. Instead, we provide the mean and standard deviation of the performance for each task in Appendix E. Furthermore, we have revised our paper here to enhance clarity in our expression.

Reviewer comment (Writing):
What is the $D_{TV}$ should be explained in the main paper. It is strongly related to your main theorem but without definition.

Author response:
- **Explaining \$D_\{TV\}\$ in Theorem 3.3.**

Reviewer comment (Evaluation):
What is relative performance? Is it calculated through minus the baseline performance?

Author response:
- **Explaining the meaning of relative performance.** Yes, it is calculated by minus the baseline performance.

Reviewer comment (Presentation):
The axis *Number of models* in Figure 3 should be [0, 10, 20, 30, 40], right?

Author response:
- **Changing the ticks of the x-axis in Figure 3.**

Reviewer comment (Experiments):
How about replacing the random model for calculating the eligible reward with a human-crafted planner? It is supposed to be helpful for improving the performance as well. I guess this can be a good direction for exploration and to make this method more practical. A simple rule-based planner is also as easily accessible as a reward function in most practical settings like robotics.

Author response:
**Q1**: How about replacing the random shooting planner with a human-crafted planner?  

**A1**: Thanks for your suggestions. We agree that a random shooting planner can be less efficient and there are more advanced methods that can be applied instead. However, for the purpose of concept-proof and simplicity, we show that with such a simple method ReDM can also yield good results.

Reviewer comment (Experiments):
In the zero interaction data setting, the method indeed works well in three simple gym environments. I wonder if the method still works well in the more complex Mujoco environment without any pre-collected interaction data. I am curious about its performance on high-dimensional control tasks.

Author response:
**Q2**: ReDM's performance in complex MuJoCo environments without offline data.  

**A2**: As the state dimensionality increases, ReDM requires much more computation power to generate sufficient many models for training an effective meta-policy. We will report the results on D4RL benchmarks. In these cases, we simultaneously explore ways to incorporate more constraints to reduce the model space, such as ReDM-o that can incorporates mismatching data.

Reviewer comment (Theory):
The main concern is the limited applicability of this method beyond low-dimensional benchmark tasks due to significant assumptions. The method assumes access to a query-able reward/termination function and the initial state distribution. More importantly, it assumes that the dynamics can be easily parameterized and optimized over with RL.

Author response:
**Comment1**: Significant assumptions that may limit the applicability of ReDM: (1) random plans to achieve non-zero data; (2) query-able reward, initial state distribution and parameterized dynamics.

**A1**: **About planners**: We agree that a random shooting planner can be less efficient and there are more advanced methods that can be applied instead. However, for the purpose of concept-proof and simplicity, we show that with such a simple method ReDM can also yield good results.

**About reward, termination functions, and initial state distribution.** In many real-world applications of reinforcement learning, human experts are aware of the basic properties of the task, including the reward function, termination function, and initial states. These properties can often be cheaper to obtain than interaction data. Therefore, assuming known reward and termination functions are acceptable in these situations. Meanwhile, we will continue to explore more methods without these assumptions in the future.

**About parameterization of dynamics models.** We employ neural networks to represent the dynamics, which can have strong expressive power and have been extensively applied in previous model-based RL studies (e.g., MBPO, MOPO, MuZero, etc). Thus we don't think this is an overly strong assumption.

Reviewer comment (Theory):
The method assumes that random plans through the candidate dynamics models will achieve some non-zero reward (to optimize the dynamics models for the eligibility reward). These assumptions make the method difficult to apply (if not impossible) in sparse-reward or high-dimensional (e.g., image-based) environments.

Author response:
**Comment1**: Significant assumptions that may limit the applicability of ReDM: (1) random plans to achieve non-zero data; (2) query-able reward, initial state distribution and parameterized dynamics.

**A1**: **About planners**: We agree that a random shooting planner can be less efficient and there are more advanced methods that can be applied instead. However, for the purpose of concept-proof and simplicity, we show that with such a simple method ReDM can also yield good results.

**About reward, termination functions, and initial state distribution.** In many real-world applications of reinforcement learning, human experts are aware of the basic properties of the task, including the reward function, termination function, and initial states. These properties can often be cheaper to obtain than interaction data. Therefore, assuming known reward and termination functions are acceptable in these situations. Meanwhile, we will continue to explore more methods without these assumptions in the future.

**About parameterization of dynamics models.** We employ neural networks to represent the dynamics, which can have strong expressive power and have been extensively applied in previous model-based RL studies (e.g., MBPO, MOPO, MuZero, etc). Thus we don't think this is an overly strong assumption.

Reviewer comment (Experiments):
The experiments with interaction data only test random interaction data and relatively small amounts of data (200 and 5000 transitions). While this is understandable as the setting where the proposed method would excel, it would be good to also show comparison to prior work with interaction data of varying optimality and amounts (including the full D4RL datasets).

Author response:
**Comment2**: Another concern is the limited scope of the experiments relative to prior work. It would be good to additionally include at least Walker2d. Additionally, it would be good to also show comparisons to prior work with interaction data of varying optimality and amounts.

**A2**: We actually did the comparison between ReDM-o with other baseline offline RL methods using full D4RL datasets including Walker2D, and the aggregated results are presented in Section 4.3 Figure 7 in the manuscript. Detailed results are listed in Table 5 in Appendix E, where *Gravity-1.0* corresponds to standard offline RL tasks, and others mean the dataset and the target environment are mismatched. We also updated our paper and included some D4RL Adroit tasks in the following table, which also shows the effectiveness of our method.

| Environment | Type | ReDM-o | MAPLE | MOPO | CQL | TD3BC |
| --- | --- | --- | --- | --- | --- | --- |
| Pen | Cloned | 57.3 $\\pm$ 16.5 | 45.7 | 54.6 | 27.2 | -2.1 |
| Pen | Human | 35.7 $\\pm$ 17.2 | 27.5 | 10.7 | 35.2 | -1.0 |
| Hammer | Cloned | 1.5 $\\pm$ 0.5 | 0.9 | 0.5 | 1.4 | -0.1 |
| Hammer | Human | 0.3 $\\pm$ 0.1 | 0.2 | 0.3 | 0.6 | 0.2 |

Reviewer comment (Novelty):
There is no discussion of MAPLE in the related work section. MAPLE is very related (just a different model generation process) so the similarities and differences should be addressed.

Author response:
**Comment3**: MAPLE is very related and deserves discussion in the related work.

**A3**: Thank you for your suggestions. We discussed with MAPLE in Appendix G, and we have revised to move the discussion into Related Work. We would like to clarify that MAPLE does not focus on model generation, and its models are learned from data with some randomness. ReDM studies how to generate models for training generalizable policies, and shows to work even with zero data or with mismatching data.

Reviewer comment (Writing):
The explanation of the optimal policy gap is confusing. Specifically, the sentence: "This discrepancy highlights the candidate model set’s capability to derive a proficient policy in the model itself." It is unclear whether "model" here means the true dynamics.

Author response:
**Q1**: The explanation of the optimal policy gap is confusing. Specifically this sentence: "This discrepancy highlights the candidate model set’s capability to derive a proficient policy in the model itself." Does "model" here mean the true dynamics?

**A1**: Sorry for the confusing expression. The "model" refers to the learned candidate model for policy optimization instead of the real dynamics. The *optimal policy gap* refers to the difference in performance between the optimal policy in the true dynamics and the best policy found in each candidate model. A large optimal policy gap indicates that the learned model has a large difference from the real dynamics.

Reviewer comment (Theory):
The statement "we conjecture that a diversified dynamics model set will correspond to a smaller ϵa since recognizing the dynamics is much easier" is unclear. It is not clear why a more diverse candidate model would lower the adaptation cost.

Author response:
**Q2**: It's not clear why a more diverse candidate model would lower the adaptation cost.

**A2**: This statement is only an intuition without proof. We will revise and remove it.

Reviewer comment (Presentation):
In Figure 6, it is unclear what the shown performance is relative to. Specifically, is this the performance of the policy at each iteration in the model at that iteration minus the performance of the policy at that iteration in the ground truth model?

Author response:
**Q3**: In Figure 6, what is the shown performance relative to? Is this the performance of the policy at each iteration in the model at that iteration minus the performance of the policy at that iteration in the ground truth model?

**A3**: The relative performance at iteration $i$ is calculated as (return of the final policy in the $i$-th model) - (return of the final policy in the ground truth model). We will revise to make this clear.

Reviewer comment (Evaluation):
For Figure 7, it is unclear whether these results are averaged over Hopper and HalfCheetah and averaged over each gravity level.

Author response:
**Q4**: Figure 7: Are these results averaged over Hopper and HalfCheetah and averaged over each gravity level?

**A4**: Yes, the results are averaged over Halfcheetah, Hopper and **Walker2D** and over all gravity levels and datasets in D4RL MuJoCo. The detailed result for each domain, dataset and gravity level is listed in Table 5 in Appendix E.

Reviewer comment (Theory):
Considering the proposed method needs to train the new dynamics models and meta-policy simultaneously, the complexity of this method and the training stability/convergence are encouraged to be clarified and analyzed.

Author response:
We would like to clarify that the dynamics models and meta-policy are trained alternately. Such alternating training has been widely employed in solving minimax problems. We will revise to discuss that when there are sufficiently many dynamics models generated, the models can cover the model space, and thus the performance of the meta-policy should converge.

Reviewer comment (Theory):
The assumed accessibility to the task reward function and initial state distribution is often unrealistic in the real applications.

Author response:
**About reward, termination functions, and initial state distribution.** In many real-world applications of reinforcement learning, human experts are aware of the basic properties of the task, including the reward function, termination function, and initial states. These properties can often be cheaper to obtain than interaction data. Therefore, assuming known reward and termination functions are acceptable in these situations. Meanwhile, we will continue to explore more methods without these assumptions in the future.

**About parameterization of dynamics models.** We employ neural networks to represent the dynamics, which can have strong expressive power and have been extensively applied in previous model-based RL studies (e.g., MBPO, MOPO, MuZero, etc). Thus we don't think this is an overly strong assumption.

Reviewer comment (Evaluation):
I am curious if totally no interaction data, how can the generated dynamics model approximates the real dynamics in the target environment. It seems there lacks enough grounding points to support this potential. Does there exist the probability that the generated dynamics models are far from the dynamics in the target environment? I hope to see more analysis on this during the rebuttal.

Author response:
ReDM does not approximate the real dynamics as we explained in **Common Question 1**. ReDM generates diverse dynamics models and thus dynamics models far from the real dynamics are often generated. These diverse dynamics models help train strongly generalizable meta-policies. We also present ReDM-o for when some interaction data is available. ReDM-o can generate dynamics models around the real dynamics.

Reviewer comment (Experiments):
The D4RL benchmark in your experiments is all Mujoco tasks with low input dimensions. Could you please consider incorporating some more high-dimensional task, in which the hypothesis space is too large to narrow down?

Author response:
Please refer to our response to **Comment 1**.

Thank you for your suggestions. We have conducted experiments on 6 environments and will include more environments in the revised version. For example, we are conducting more experiments on D4RL Adroit tasks. The results on 4 of the environments are listed below, showing that the proposed method is still effective.

| Environment | Type | ReDM-o | MAPLE | MOPO | CQL | TD3BC |
| --- | --- | --- | --- | --- | --- | --- |
| Pen | Cloned | 57.3 $\\pm$ 16.5 | 45.7 | 54.6 | 27.2 | -2.1 |
| Pen | Human | 35.7 $\\pm$ 17.2 | 27.5 | 10.7 | 35.2 | -1.0 |
| Hammer | Cloned | 1.5 $\\pm$ 0.5 | 0.9 | 0.5 | 1.4 | -0.1 |
| Hammer | Human | 0.3 $\\pm$ 0.1 | 0.2 | 0.3 | 0.6 | 0.2 |

Reviewer comment (Theory):
Considering your method encourages the diversity in the model learning part, some learned dynamics models may be unreasonable though the meta policy can still achieve high returns via planning in such models, like violating the physics laws or economics laws. And I can hardly expect the *eligibility* part in your method can help alleviate this 'short-path' issue. More explanations and discussions are encouraged during the rebuttal phase.

Author response:
We would like to clarify that the key to the success of ReDM is to avoid generating too many unreasonable dynamics models. ReDM has employed eligibility to prune some unreasonable dynamics models, which has already led to good performance in cases. If more unreasonable dynamics models can be pruned, ReDM can surely improve its efficiency. We will also explore other principles to prune more unreasonable dynamics models.