Paper: COPlanner: Plan to Roll Out Conservatively but to Explore Optimistically for Model-Based RL (ICLR-2024)

Reviewer comment (Experiments):
Generally, the paper would benefit from a stronger selection of established baselines. E.g., for proprioceptive DMC model-free MPO/DMPO/D4PG or model-based DreamerV3, for visual DMC DrQ-v2 as a model-free baseline.

Author response:
Thank you for your suggestion. Following your advice, we add a comparison with D4PG in the proprioceptive control task in Figure 4, and a comparison with DrQv2 in the visual control task in Figure 5. 

Additionally, due to space limitations, we include a comparison with DreamerV3 in six medium-difficulty proprioceptive control tasks in Appendix D.1. For specific experimental details and results, please refer to General Response and the updated paper.

Reviewer comment (Experiments):
The visual control experiments should be run for more than 1M steps as agents have not converged on the Acrobot/Finger/Quadruped tasks. Comparing with Dreamer-v1 Figure 10, agents would be expected to converge and solve the tasks at approximately 2M steps.

Author response:
We would like to offer our apologies here. Due to differences in the counting methods for Counter and Environment Step in the JAX code of DreamerV3, there is a mistake in the previous version of our paper. The original version of Figure 5 represents the performance of all methods at 500k environment steps. We have run these methods up to 1M steps and have reported the new results in both Figure 5 and Table 5 in the revised manuscript. The new results show that nearly all methods have converged at 1M steps, and DreamerV3 significantly outperforms the results reported in the DreamerV1 paper. We apologize again for this mistake.

Reviewer comment (Reproducibility):
The Dreamer-v2 scores in Figure 9 appear to be significantly lower than the ones provided by the original authors on GitHub – which Dreamer implementation was used to obtain these scores?

Author response:
We use one of the most popular PyTorch implementations of DreamerV2 available on GitHub, which has the second-highest number of stars: https://github.com/jsikyoon/dreamer-torch. Since it is implemented in PyTorch, there may be differences in the results compared to the TensorFlow version of the original source code. Therefore, we only include these experiments in the Appendix as a reference and do not feature them in the main paper. In the main paper, for the DreamerV3 experiments, we use the original JAX code released with the DreamerV3 paper.

Reviewer comment (Experiments):
The hyperparameter study in Appendix D.4 is interesting, but should be extended to more environments to observe clearer trends.

Author response:
In the updated version of our paper, we add two sets of hyperparameter studies, one using COPlanner-MBPO on the Cheetah-run task and the other using COPlanner-DreamerV3 on the Hopper-hop task. The trends we observed are largely consistent with our previous experimental results. For specific experimental results, please refer to **Appendix D.4, Figure 11** in the updated paper.

Reviewer comment (Novelty):
Some highly related work is missing. E.g., the optimistic exploration of uncertain future returns for visual control has previously been investigated in [1], also building on the Dreamer-v2 agent, while optimistic finite-horizon exploration under nominal reward functions was studied in [2]. Additionally, work such as POLO [3] would be relevant.

Author response:
Thank you for your reminder. We have added references to these papers in the related works section of our updated paper.

Reviewer comment (Theory):
Action selection by optimizing over a 5-step rollout without terminal value function is surprising. Do you have intuition for why this short-term reasoning is sufficient?

Author response:
This is indeed an interesting point we've noted as well. We experimented with incorporating a terminal value function into the MPC process, but found that it did not offer significant improvements over our current method (for both proprioceptive and visual control). Additionally, since we perform planning to estimate long-term uncertainty, introducing a terminal value function would also require estimating the uncertainty of the terminal value, which would further increase computational costs. For these reasons, we did not include a terminal value function.

As for why incorporating a terminal value function didn't work, we suspect it's because model compounding errors lead to significant inaccuracies in the state predicted by the model after several rollout steps. Consequently, the value function may not provide an accurate value estimation. Furthermore, the scale of state value is much larger than the rewards and estimated uncertainty, which might result in the state value solely dictating the MPC's action choices, rendering our method ineffective. Therefore, we believe short-term reasoning is a better solution. In future work, we plan to further explore how to integrate long-term reasoning into our framework.

Reviewer comment (Experiments):
The original Dreamer-v3 paper also evaluated on proprioceptive DMC environments - why not use Dreamer-v3 as a proprioceptive baseline as well?

Author response:
We choose to use MBPO because we want to demonstrate that our method can be combined with any dyna-style MBRL method, not just Dreamer. MBPO is one of the most representative MBRL methods in proprioceptive control. As a supplement, we add a comparison with DreamerV3 in proprioceptive control tasks in **Appendix D.1**. The experimental results show that our method still significantly improves both performance and sample efficiency compared to DreamerV3 in proprioceptive control tasks.

Reviewer comment (Experiments):
There are several task-specific parameter choices for proprioceptive control. How impactful are these? Ideally a single set of hyperparameters would be used throughout.

Author response:
Due to the time constraints of the submission and the longer experimental duration required for MBPO-based methods, we did not unify the parameters for GYM in the initial paper. However, **in the updated paper, we have unified the experimental hyperparameters, and now for all proprioceptive control experiments, the hyperparameters are the same (optimistic rate=1, conservative rate=2, action candidate number=5, planning horizon=5)**. Please refer to Figure 4 for the updated experimental results. We find that after unifying the hyperparameters, the performance of COPlanner in GYM's Hopper, Walker, and Ant environments has improved compared to the previous version. The performance in Humanoid remains largely unchanged. This is consistent with the conclusions of our hyperparameter studies, as for Humanoid, the only change was in the conservative rate from 1 to 2, and this parameter's setting between 1 and 2 has little difference in performance across different environments.

Reviewer comment (Experiments):
It's still not quite clearly to me how Conservative rate and Optimistic rate are selected. What's the intuition here about why they are always different than each other in every experiment settings? Are they selected according to Action candidate number, and Planning horizon as well?

Author response:
As discussed in Appendix D.4 of our paper, we conduct a detailed hyperparameter study to discuss how parameters should be chosen and then selected a set of parameters that perform relatively well across all tasks. Regarding why these two parameters are different, we believe that these two are essentially completely unrelated parameters, one controlling exploration and the other controlling exploitation (reducing model error). In proprioceptive control tasks, since the model directly predicts the next state, it is more prone to model compounding errors, thus requiring a larger Conservative rate. In contrast, in visual control tasks, the dynamics model predicts in the latent space, which to some extent mitigates the impact of model compounding errors. Therefore, the Conservative rate can be set relatively lower, allowing the model rollout to more likely choose high-reward, model-generated samples for training the policy. The choice of these two parameters is independent of Action candidate number and Planning horizon.