Paper: Diffusion World Models (ICLR-2024)

Reviewer comment (Evaluation):
While video metrics are good compared to others, these metrics are agnostic to actions and hence of limited value in evaluating a world model; a useful metric should include the faithfulness to some action and compare against baselines.

Author response:
As the reviewer notes, a perfect action-conditioned video metric is hard to find. However, the LPIPS metric is, to the best of our knowledge, the current favored choice. LPIPS compares frames in a pairwise fashion (real to generated), and in order to do well in this metric, the effect of an action _must_ be modeled correctly. Note however that it does not account for an environment’s stochasticity, hence being imperfect. On the other hand, the FVD metric measures action faithfulness less directly, but _does_ account for stochasticity in the environment being modeled. Hence when combined, the two metrics are able to measure action faithfulness and stochasticity.

Reviewer comment (Evaluation):
It is unclear how well the learned model generalizes; more material similar to Figs. 7 and 8 should be provided in the appendix to clarify this.

Author response:
Thank you for highlighting the value of Figure 7 & 8. [Here is a link](https://drive.google.com/drive/folders/1aaCx2TljjKh4nefccktxk286npeRFf7g?usp=sharing) to several videos showing generated trajectories, beginning from the same initial state and applying various constant actions, for CS:GO and motorway driving. Since the world model is trained on offline datasets from expert-level policies, some of the trajectories quickly become out of distribution when certain constant action sequences are input. For example, the driving dataset does not contain trajectories where a steering is applied to take the car off the road. Hence, whilst the world model tends to do well at modeling immediate frames, they sometimes do not generalize well when pushed beyond this. We note this is a symptom of the training data, rather than anything specific to our diffusion world model.

Reviewer comment (Evaluation):
Video metrics, although nice, are not as relevant as faithfulness to the input action; a “conditional video metric” that evaluates such faithfulness should be introduced and used to evaluate both the proposed model and baselines.

Author response:
As the reviewer notes, a perfect action-conditioned video metric is hard to find. However, the LPIPS metric is, to the best of our knowledge, the current favored choice. LPIPS compares frames in a pairwise fashion (real to generated), and in order to do well in this metric, the effect of an action _must_ be modeled correctly. Note however that it does not account for an environment’s stochasticity, hence being imperfect. On the other hand, the FVD metric measures action faithfulness less directly, but _does_ account for stochasticity in the environment being modeled. Hence when combined, the two metrics are able to measure action faithfulness and stochasticity.

Reviewer comment (Evaluation):
Uncertainties in Table 2 should be added, as interpreting the results without them is not possible.

Author response:
We are in the process of rerunning this benchmark, but have not been able to complete this during the rebuttal period. We will share a fully updated table in due course.

Reviewer comment (Evaluation):
A quantitative result should be added to demonstrate that the world model indeed behaves like a world model, relating to the earlier point about faithfulness to input actions.

Author response:
Thank you for this question, which we believe we were not clear enough about in the main paper. The purpose of dividing the experiments into two sections was to show: 1) DWM can be effective as part of a typical dyna-style algorithm to train an RL agent in imagination, which is a holistic test of DWM’s abilities. 2) DWM also scales to visually complex environments. We chose to demonstrate this in real-world motorway driving and a modern video game. Due to the complexity of these environments, however, we were unable to also test dyna-style agents in these settings, and hence report the best available offline metrics.