Paper: Advantage-Conditioned Diffusion: Offline RL via Generalization (ICLR-2024)

Reviewer comment (Experiments):
Limited ablation studies. Most design choices in this paper are not well elaborated.

Author response:
We thank Reviewer CsHK for their effort in reading our paper and providing detailed feedback that we will use to turn it into a better shape.

Q1. The ablation studies in this paper are limited.

A1. We tried to provide the most important design insights in the paper, which included the choice of architecture (MLP vs DDPM) and different encoding schemes for advantage value. Can the reviewer point out which crucial design choices were not ablated?

Reviewer comment (Experiments):
The authors have said that " poor results for ACG with scalar advantages can be improved by changing the optimizer from Adam to AdamW ". Is there any ablation study about the performance of Adam and AdamW?

Author response:
A2. We have not. We only saw the performance improvement in the experiment that we mentioned. We did not run additional ablation to avoid introducing additional components of complexity to our analysis, especially one that is quite orthogonal to our main contribution. Since the majority of algorithms use Adam, we kept it for fair comparison.

Reviewer comment (Writing):
As the title of this work includes ''Offline RL via Generalization", this work seems to mainly tackle the single task, what is the meaning of "generalization" in this work?

Author response:
A3. It stands for the ability of neural networks to generalize beyond seen examples. That is, our policy relies on the diffusion model’s understanding of actions with zero advantage, even though individual states may not be paired with such actions in the dataset.

Reviewer comment (Novelty):
Could the authors relate the work to other approaches to diffusion for offline RL that directly try to do return conditioned generative modeling such as Decision Diffuser?

Author response:
Q2. How is this algorithm from Decision Diffuser?

A2. Decision Diffuser can be viewed as one of the RvS methods, since it conditions the policy on the trajectory return, and as such is captured by prior work that we described. Our method leverages the temporal compositionality of the MDP by employing the advantage function which (approximately) indicates an action’s optimality if it equals zero. On the engineering side of things, Decision Diffuser models the future states. It concatenates K future states and diffuses over the resulting matrix like if it was an image. Instead, we model (diffuse over) the action variable directly.