From: ICLR.cc/2023/Conference/Paper3346/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper3346/Reviewer_Dajh
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: Typically modern model-based RL (MBRL) algorithms keep a buffer of real environment observations, and sample the initial states for model rollouts uniformly from this buffer. The sampled states are used both for model learning and for policy learning. However, there is a mismatch to the current policy of the agent, as the data includes historical data from previous policies. It may be better to focus the training on recent data as they are more relevant to the current policy.

The paper proposes a method to prioritize the sampling of the data based on the KL-divergence of the current policy to previous policies, and combine this together with the MBRL algorithm MBPO to propose PDML. This requires storing the previous policies, but it is explained that the additional memory requirements are small compared to the data buffer, and the computation time change to the original MBPO was also negligible.
The authors argued in favor of their approach theoretically by deriving a bound for the model prediction error that depends on the data sampling distribution.

The experimental results included comparisons on 4 MuJoCo domains against MBRL methods: MBPO, AMPO, VaGram and model-free methods: SAC and REDQ. And the proposed method appeared better in all scenarios. The work also included comparisons with previous prioritized experience replay methods in the model-free RL literature PER, RECALL and MaPER, when these methods were adapted for prioritizing data in MBPO, and the results showed that the newly proposed method is better.

They also looked at the model prediction errors, and showed that the error is smaller for PDML on the data from the current policy compared to MBPO.
strength_and_weaknesses: **Strengths**
_______________
- They compared with various previous model-based and model-free algorithms, and the performance seemed good.
- There were comparisons with prioritization methods previously used in model-free RL.
- The work established that sampling uniformly in model-based RL is a problem, and showed that the issue can be alleviated.

**Weaknesses**
______________
- There was no experimental study on what the prioritized sampling distribution looked like in practice. It would be good to visualize a few histograms during training to show that the distribution was non-trivial.
- There could have been comparisons with simple prioritization methods, e.g. exponentially decay the priority or linearly decay the priority to the past. If the new method were not able to beat such a simple baseline it would undermine the relevance of the method.
- I was not convinced by the relevance of the theoretical analysis. The difference in accuracy on the recent data compared to past data may be due to the model not having converged on the recent data yet (as there have not been that many gradient updates with the recent data compared to past data). However, the theoretical analysis does not look at convergence speed. Intuitively it makes sense to prioritize the more recent data so that the model could learn the new relevant data quickly rather than rehashing old data, but I think the theoretical analysis did not capture such points. It could be experimentally tested whether the convergence speed or the distributional mismatch is the bottleneck by training the model until convergence before gathering new data, and checking the model prediction error.
- The amount of environments, 4, is a bit low. Running experiments with 2-4 more environments would strengthen the results.
- The computational speed is slow. This is not an issue of the new method, but a problem of the original MBPO. For example, Walker or Cheetah take ~60h to train. In comparison Dreamer can learn these tasks in 10h computation time when using pixel observations, and it will be even faster when using proprioceptive observations. I don't think we can call MBPO with the current settings SOTA. 
- When prioritizing the sampling, it not only adjusts the distribution for the model learning, but also for the policy learning. It remained unclear whether the improvement in performance was due to faster policy learning or due to better model learning. To test these points it could have been possible to sample the states for the model learning and policy learning separately to explain which component was more important. For example, if the same improvements could be seen by sampling the data for the policy from the prioritized distribution, but sampling the data for the model from the previous uniform distribution, it would undermine the story of the paper.
clarity,_quality,_novelty_and_reproducibility: **Clarity**
______________
I think the clarity was good.
The style of writing could be improved a bit. I think there was too much repetition of "SOTA".


**Quality**
______________
The method seemed reasonable, and the main experimental results with the learning curves in Figure 2 seemed convincing. In particular, the comparisons with REDQ and AMPO indicated that the performance of the method is good.
However, there were also several loose ends, and I was not entirely convinced by all of the claims and experimental results as detailed in the weaknesses section.

**Novelty**
____________
There are several prioritized replay methods in model-free RL, but I believe this is the first one in model-based RL.

**Reproduciblity**
____________
I believe sufficient details were provided to reproduce the work.
summary_of_the_review: The method is reasonable, and while I am not completely convinced by all of the claims, the empirical results indicate that the method performs better than other recent efficient methods such as REDQ or AMPA. Moreover, it was made clear from the work that the performance of MBRL methods can be improved by pursuing a prioritized sampling approach. Therefore, I am leaning towards recommending to accept. The work could be further improved by examining the unclear areas I pointed out in more detail.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3346/Reviewer_bWBH
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: - This paper makes the following observation in the context of model-based RL: learning a dynamics model using a replay buffer of old policies isn’t optimal to improve the current policy.
- The intuition is that the dynamics model has limited capacity, which can be better allocated to model the state transitions that are more likely under the current policy, rather than transitions collected from old policies.
- The authors empirically validate this phenomenon on Mujoco continuous control tasks, where the dynamics model has significantly higher error for the current policy transitions than for the remaining of the replay buffer.
- They also derive an upper-bound for this error gap, which motivates the intuition that transitions most likely under the current policy should be prioritized.
- Finally, they propose Policy-adapted Dynamics Model Learning (PDML), a method that builds on the above insights; they validate PDML on standard Mujoco testbeds, where it outperforms competitive model-based methods.
strength_and_weaknesses: - Strengths:
    - The underlying hypothesis of the paper is sound and worth investigating. At first sight, learning a dynamics model is “just” a supervised learning problem but, in the context of using this model to learn a policy, we’d like this model to be especially good on the transitions induced by the policy.
    - To the best of my knowledge, the bound derived by the authors is novel (caveat: I did not carefully check the proof), and so is the proposed PDML. Some prior work discusses re-weighting transitions but are motivated from a different perspective and result in different methods.
    - PDML is a generic method, which can be combined with any model-based method. It also seems to be effective, as MBPO+PDML obtains good empirical results on Mujoco tasks and significantly improves the performance MBPO.
- Weaknesses:
    - Fig. 1 is problematic: for a) and b), the results are invalid because T-SNE actively tries to minimize the variance within a group and maximize the variance across groups. In fact, I’d argue that those plots show very impure clusters when compared to other results obtained from T-SNE — thus contradicting the claim that old and current state visitations are different.
        
        For c) and d): is the x-axis the steps for RL training (ie, the replay changes as steps increase) or the supervised steps to learn a model with a fixed replay? If the former, wouldn’t it be expected that the fresh samples from the current policy incur higher loss since they haven’t been optimized as much as older samples? Note that this has nothing to do with the gap the authors trying to show — namely, that learning on the old and current transitions is especially detrimental to modelling the transitions induced by the current policy.
        
    - I’d like to see some simple baselines to validate the effectiveness of PDML. First, how effective is it to linearly (or exponentially) decay the weight of transitions based on how long they’ve been in the replay? Second, I’d like to see the performance of a crude importance sampling estimator with weights given by the ratio of the density of observing a transition according to $p_\text{new}(s, a, s’)$ and $p_\text{old}(s, a, s’)$. Concretely, I think this would simply boil down to multiplying the each error of the model prediction by $\pi_\text{new}(a \mid s) / \pi_\text{old}(a \mid s)$, where $\pi_\text{old}(a \mid s)$ needs to be stored together with each transition in the replay.
    - I wish the following trade-off were discussed: modelling the current transitions might be optimal for the updating the current policy, but maybe not when using the model in the context of exploration, or to approximate the optimal policy. For example, one could imagine learning a value function by sampling values through Monte-Carlo rollouts of the model (ie, approximating $\pi^\star$ with the model). In this case, we care about all transitions not just the ones most likely under the current policy.
    - A minor remark: Mujoco benchmarks (from joint observations, as in this paper) are not great benchmarks anymore for RL because modern methods all obtain excellent policies which take advantage of quirks in the simulator. So observed improvements might not be due to a particular method working better but to implementation details, which really matter. For example, note that the SAC and TD3 implementations in OpenAI’s spinning-up outperform or closely match the results in Fig. 2.
clarity,_quality,_novelty_and_reproducibility: - Clarity: Mostly clear.
- Quality: Overall ok, but I have raised some concerns based regarding the correctness of the motivational figure and the lack of simpler baselines.
- Novelty: Good, as far as I know.
- Reproducibility: The results seem reproducible, but having open-source code would help (I don’t see a supp. mat. on openreview).
summary_of_the_review: - Strengths:
    - An interesting hypothesis.
    - A novel bound and method, PDML.
    - PDML works well empirically, and is compatible with many model-based RL methods.
- Weaknesses:
    - Some (potential) errors in the figure which motivates the work.
    - Some missing simple baselines built on the same motivation.
    - Some more nuanced discussion around the proposed method, and a remark on the choice of testbeds.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3346/Reviewer_tneA
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper studies the data distribution used to train the dynamics model in model-based RL. They propose to draw samples not uniformly at random as is usually done but according to probabilities that give data similar to the data produced by the current policy a higher weight. The method is applied to MBPO and achieve increased results in the considered environments. 
strength_and_weaknesses: Strength:
+ The paper is written well and easy to understand
+ learning a better dynamics model is an important topic
+ the method shows improved performance for the evaluated environments


Weaknesses:
- It is strange to me that for the evaluation the typical mujoco gym environments are used except for Ant. That leaves the uneasy feeling the proposed method might not perform well on Ant, which would be important to know
- as also mentioned in the Discussion section, the proposed method naturally leads to overfitting on the data similar to the data generated by the current policy and hence to worse generalization. That might increase performance for a single task but make the algorithm more fragile with respect to small changes in the environment and hurt exploration. It would be valuable to have another experiment in the paper that evaluates the method in a harder exploration task.
- The method is motivated by some theoretical analysis. However, the final method is derived from the theory with a lot of approximations such that it is not clear to me how useful the theoretical part actually is.


minor points:
- p.9 "After combining with the state-of-the-art modelbased method, .."  -> "After combining with the state-of-the-art modelbased method MBPO, .."
- the plots in Figure 1 are very small and there is a lot of space between the single plots. I suggest the authors reduce the space between the Figures to increase the size of the single plots.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly and well written. The proposed method is novel and tackles an important problem. 
summary_of_the_review: The paper is interesting and interesting to read. The proposed method seems promising. However, the evaluation does not convince me currently which puts the paper for me marginally below the acceptance threshold. I am inclined to raise my score if the mentioned experiments are added for the revised version.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3346/Reviewer_bmEZ
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper studies the problem of model learning in model-based reinforcement learning. Specifically, the authors find that uniform training on all of the data collected by historical policies yields poor model predictions for the current policy, due to limited model capacity. Instead, they propose re-weigh samples based on the distribution shift between the current policy and the historical policies used to collect them. With this re-weighting scheme, their method Policy-adapted Dynamics Model Learning (PDML), built on top of MBPO, achieves better sample complexity compared to other RL algorithms and model-free prioritized experience replay algorithms.
strength_and_weaknesses: Strengths:
- I appreciate how the authors presented evidence for the gap between the current error and the overall error and the performance gap analysis.
- There is an array of comparisons in the experimental evaluation, including model-based and model-free RL algorithms and prioritized experience replay-based algorithms. Compared to these methods, the proposed PDML is shown to be more sample-efficient on four different MuJoCo tasks.
- While the proposed method requires storing every historical policy, the authors include a discussion on the associated storage and computation cost.

Weaknesses:
- The objective mismatch problem has been previously identified and analyzed in other papers such as by Lambert et al. These papers also propose to re-weight dynamics model training with varying definitions for and ways to compute the weights [1,2,3]. Comparisons to these existing methods would be useful to understand the significance of this work.
- The current design for the weights seems very complicated, especially the choice of the weight for the current policy’s samples. How sensitive is performance to the choice of \alpha?
- Similarly, it’s stated that “if we only use a small number of samples from just one policy, it is difficult to learn accurate transition dynamics for the current policy.” It would be useful to verify this empirically by choosing a larger value for \alpha.
- Could alternative, simpler weighting schemes work, such as weights that exponentially decaying with time (i.e., more recently collected samples receive higher weights while older samples are receive lower weights)?

[1] Lambert et al. Objective Mismatch in Model-based Reinforcement Learning. 2nd Annual Conference on Learning for Dynamics and Control.

[2] Eysenbach et al. Mismatched No More: Joint Model-Policy Optimization for Model-Based RL.

[3] Rajeswaren et al. A Game Theoretic Framework for Model Based Reinforcement Learning.
clarity,_quality,_novelty_and_reproducibility: Clarity/Quality: The paper is written well overall. I also like the flow of the paper, which starts with an empirical analysis of model error, transitions into a more formal analysis, and proposes a solution based on this.

For completeness, could the one-step error plot for Humanoid be plotted as well?

In Figure 4, are the reported errors on the training samples or samples not seen by the model? If it’s the former, could we also evaluate the model on new samples (still collected by the same policy though)?

Novelty: The problem of objective mismatch and the solution of changing the model learning objective is not novel, thus it would be good to include comparisons to them.

Reproducibility: The paper includes pseudo-code of their algorithm, open source code the authors built off of, and all of the hyperparameter values used. Together, I believe a reader could re-implement the proposed algorithm.

summary_of_the_review: While the experiments have demonstrated the sample efficiency of PDML, I think there are important MBRL algorithms modifying model training that should be compared to. Without additional analysis of PDML, such as on the sensitivity of alpha, it's also difficult to understand how robust the method is.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold