Paper: BDQL: Offline RL via Behavior Diffusion Q-learning without Policy Constraint (ICLR-2024)

Reviewer comment (Experiments):
The performance of BDQL-SWA on MuJoCo tasks still trails behind modern offline RL methods.

Author response:
**A1**: Although the results of BDQL are not outstanding, the experiments is achieved under the condition of "**without constraint to overcome OOD**". Combined with our newly added theoretical analysis, we comprehensively proves that BDQL is an algorithm capable of solving offline RL without the need for constraints.

Reviewer comment (Theory):
Why 'The output of the diffusion policy is the deterministic action rather than the distribution of action'? If I understand correctly, the diffusion policy models a stochastic state-conditioned action distribution as Equation (3) rather than a single deterministic action. So Equation (8) should be $\mathbb{E}_{s_t \sim \mathcal{D}, a \sim \pi_\theta(s_t)} [Q_\phi(s_t, a)]$? Does this one still follow the deterministic policy gradient (DPG) theorem?

Author response:
**A4**: From the details of the diffusion model, this action is sampled from a distribution. However, from an external perspective, the diffusion model takes in a state and outputs an action. This aligns with the definition of a deterministic policy and can be updated using DPG. Similar approaches are also adopted by Diffusion Q-learning [1].

[1] Wang, Z., Hunt, J. J., & Zhou, M. (2022). Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06193.

Reviewer comment (Experiments):
First, from the experimental studies, the results of BDQL is not convincing. The performance of BDQL is close to other competitors and BDQL-SWA is even inferior to other competitors.

Author response:
We address these two questions together. BDQL and its variants achieve results comparable to the strongest baseline (ensemble-based methods). While this may not be a  breakthrough,  it is important to note that this result is achieved **without introducing constraints and with OOD preserved**. This experiments, along with our newly added theoretical insights, demonstrate that offline RL may not necessarily require constraints to solve.

Reviewer comment (Evaluation):
From the authors' explanation, the Theoretical performance refers to the on-policy scenario, and the Practical performance refers to the offline scenarios. In offline scenario, the performance of BDQL-SWA is still not good enough.

Author response:
We address these two questions together. BDQL and its variants achieve results comparable to the strongest baseline (ensemble-based methods). While this may not be a  breakthrough,  it is important to note that this result is achieved **without introducing constraints and with OOD preserved**. This experiments, along with our newly added theoretical insights, demonstrate that offline RL may not necessarily require constraints to solve.

Reviewer comment (Writing):
In SWA, the author mentions ``SWA averages the multiple checkpoints during the optimization''. Do you mean averaging the parameters of the models at different training time? So it is similar to the Target Actor/Critic network trick in most offline RL methods?

Author response:
Indeed, SWA averages different checkpoints during training, but it differs from the Target Actor/Critic network trick. The key distinction is that the Target Actor/Critic network actively participates in training, whereas SWA does not. SWA averages different checkpoints during training to stabilize fluctuations and avoid collapse.

Reviewer comment (Experiments):
Some offline-RL baselines, such as BEAR are missed in experiment, and the recent popular SPOT [1] method is not considered in experiments as well. It is suggested to consider the baselines in offline-RL methods.

Author response:
The descriptions of the support constraint value function in BEAR and SPOT inspired our proof, and we have cited both of these papers. The instability of BEAR's performance is a major reason why we did not consider it in our baseline. The strength of the SPOT algorithm lies in its offline-to-online fine-tuning while the performance is comparable to TD3+BC in addressing offline RL (SPOT: 773/9 = 85.89, TD3+BC: 84.83). Meanwhile, our baseline also includes ensemble-based methods, which are among the strongest algorithms.

Reviewer comment (Theory):
The author claims that the BDQL has sufficiently solves the offline RL problem (OOD issue), and the OOD issue only causes fluctuation in training. The authors tries to illustrate this point with some ablation studies. However, this claim seems weak. It is suggested to add some theoretical analysis supporting this claim.

Author response:
In the new version of the paper, we analyze the optimality of BDQL and its tenable conditions (please see Section 4). We find that when the estimated behavioral policy is close enough to the true behavioral policy, BDQL can converge to the optimal policy. Inspired by this conclusion, we divide the training process into two stages based on the distance between the current training policy and the true behavior policy. This leads to improved analytical results and better demonstrationl in our experiments (Figure 4).

Reviewer comment (Experiments):
Actually, the diffusion model is time-costly in model inference. Will this issue also occur in BDQL? Some ablation studies on computation issues are suggested.

Author response:
we compare the training time between BDQL and BQL. The training time of BDQL (1492 min) is approximately 2.5 times that of BQL (608 min). In future work, we will consider further optimizations in terms of algorithm and implementation to reduce the time overhead.

Reviewer comment (Presentation):
The significance of section 2.2 is unclear, since both methods are not used in the current paper.

Author response:
In Section 2.2, we introduce on-policy and off-policy learning. The former is highly relevant to our algorithm since BDQL can be viewed as a form of on-policy learning in the context of offline RL. When introducing off-policy learning, we also highlight the challenges in offline RL. This section serves as essential groundwork for the subsequent discussions.

Reviewer comment (Theory):
The SARSA update requires the assumption that the offline data is coming from trajectories, and the data collecting policy is a single stationary policy.

Author response:
SARSA can also be updated from transitions, not only trajectories. Offline datasets can be assumed to be collected by the behavior policy, which allows update by SARSA. This assumption has been adopted by on-policy style offline algorithms like Onestep RL [1] and BPPO [2].

[1] Brandfonbrener, D., Whitney, W., Ranganath, R., & Bruna, J. (2021). Offline rl without off-policy evaluation. Advances in neural information processing systems, 34, 4933-4946.

[2] Zhuang, Z., Kun, L. E. I., Liu, J., Wang, D., & Guo, Y. (2022, September). Behavior Proximal Policy Optimization. In The Eleventh International Conference on Learning Representations.

Reviewer comment (Theory):
The argument on regularization for ood from the current algorithm is not very convincing. It seems like an alternative way of regulizing with behavior cloning with a diminishing regularization coefficient. Using this perspective, one can also suspect if this varying objective is causing the instability in the practical performance.

Author response:
We may not fully understand this question and are unsure if the reviewer is asking if SWA is an alternative OOD regularization method. If so, our response is as follows; SWA does not participate in training and does not alter the original optimization objectives. This fundamentally distinguishes the role of SWA from other constraints. The training of the original BDQL is unstable, and SWA is used to eliminate this instability

Reviewer comment (Reproducibility):
No concrete algorithm box is provided.

Author response:
We have added the algorithm box in appendix and the Figure 1 can also clearly express the algorithm logic.

Reviewer comment (Writing):
The usage of the term "theoretical performance" in the experiment section is confusing.

Author response:
We have replaced the “theoretical performance” with “best result”, representing the best result obtained from online evaluation.