Paper: Adjustable Quantile-Guided Diffusion Policy for Diverse Behavior Generation in Offline RL (ICLR-2024)

Reviewer comment (Evaluation):
**(Unclear empirical benefit)** Whilst the paper shows a strong improvement on speed, the empirical benefit of the approach is unclear, as there is only a slight gain over the baseline DQL. This is compounded by the fact that the main empirical evaluation in Section 6.1 maximizes over an extremely large 70 different hyperparameter configurations, which is extremely unrealistic for offline reinforcement learning. Coupled with the sweep in Figure 4, the main empirical evaluation is likely maximizing over statistical noise. The results over a smaller hyperparameter space in Section 6.3 do not show a clear improvement.

Author response:
Thank you for your suggestions.
IDQL's experimental results are quoted from the result of IDQL-A. "-A" is refered as reported results that allow any amount of tuning and "-1" as results that only allow one hyperparameter to be tuned between domains[1]. And IDQL-1's results are only 1150.9. 
So, the performance of qGDP, 1220.6, in a smaller hyperparameter space is sufficiently good.

[1] Hansen-Estruch P, Kostrikov I, Janner M, et al. Idql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.10573, 2023.

Reviewer comment (Evaluation):
**(Quantitative metric of success in toy experiments)** Whilst the authors helpfully compare the effect of different guidance and sampling schemes on their toy dataset in Figure 2-3, it is unclear what the optimal or desired behavior is. It would be helpful to include some indication of what the optimal behavior should be and some kind of metric to assess this (e.g. conforming to some desired distribution).

Author response:
Thank you for your suggestions.
The results in Fig. 2 are generated based on the toy datasets, as illustrated in Fig. 1. This experiment is primarily conducted for a qualitative analysis of the impact of hyperparameters on the results produced by different methods. The goal is to assess whether these methods exhibit generalization capabilities and to examine the effects of hyperparameters on the generated results, so there is no specific desired distribution.

Reviewer comment (Evaluation):
**(Incomplete speed discussion)** Whilst Table 4 shows an improvement over DQL, it would be also valuable to compare runtime to related approaches, IDQL and SfBC. Furthermore, details are missing on which environment exactly is being timed in the table.

Author response:
Thank you for your suggestions.We test the release codes of DQL, IDQL, and SfB with the halfcheetah-medium-v2 task on a single RXT 2080Ti graphics card, and their runtimes are compared below:

method  | critic                             | actor | sampling | total 

DQL        | 8.6ms$\\times$2e6     | 12.4ms$\\times$2e6 | 51.2s$\\times$40 | 12.9h

IDQL      |               3.3ms$\\times$1.5e6 (critic and actor) | 70.0$\\times$6 | 1.5h

SfBC      | 68.4ms$\\times$4.9e4 | 39.1ms$\\times$5.9e5 | 50.6s$\\times$20 | 7.6h

qGDP     | 9.3ms$\\times$4e5      | 5.9ms$\\times$4e5 | 10.0s$\\times$70 | 2.0h

where, 'critic,' 'actor,' and 'sampling' respectively refer to the time spent on value function training, policy function training, and interacting with the environment of these algorithms. The recorded form of these terms is the time required for a single update of the value function multiplied by the number of updates, the time required for a single update of the policy function multiplied by the number of updates, and the time required for model evaluation (10 interactions with the environment) multiplied by the number of evaluations. 

It should be noted that IDQL is implemented based on JAX, resulting in shorter runtimes. In comparison, both qGDP and DQL are implemented in PyTorch. Specifically, in the actor update phase (related to training the diffusion model), qGDP exhibits significantly shorter runtime compared to DQL. During the model evaluation phase, qGDP reduces time consumption through parallelization.

Most importantly, the presented runtimes for DQL, IDQL, and SfBC are for a single run. If hyperparameter tuning is required, they would incur several times more runtime. In contrast, qGDP has completed hyperparameter tuning during the 70 model evaluation runs.

Reviewer comment (Theory):
The quantile labels rely on the action values Q learned by IQL, which may have overestimation bias. Errors in Q would propagate to incorrect quantile labels, which could be amplified for higher quantiles. This could negatively impact the quality of the diffusion model training.

Author response:
Thank you for your question. 
In our code implementation, we have utilized the Clipped Double Q-learning technique provided by TD3[1] to mitigate Overestimation Bias. 
[1] Fujimoto S, Hoof H, Meger D. Addressing function approximation error in actor-critic methods[C]//International conference on machine learning. PMLR, 2018: 1587-1596.

Reviewer comment (Novelty):
While the quantile input is motivated by IDQL, the paper does not sufficiently differentiate the advantages of the quantile mechanism compared to IDQL itself. The paper does not sufficiently differentiate the advantages of the quantile input compared to just using weighted actions like in IDQL. The results between qGDP and IDQL in Table 1 are fairly close. More analysis is needed to clearly explain the relationship between the two methods and highlight the unique benefits of the quantile inputs.

Author response:
Thank you for your suggestions. 
qGDP has two main advantages over IDQL: Firstly, IDQL increases the probability of the diffusion model generating high-value actions by reweighting samples. So it cannot generate samples with higher values outside the dataset. In contrast, qGDP learns the probability distribution of actions conditioned on different quantiles. Therefore, when the true action value function changes smoothly, qGDP can output action distributions corresponding to higher action values (associated with quantile inputs greater than 1). Secondly, adjusting the generation preference in IDQL requires retraining the model, whereas qGDP only requires changing the model's input. This significantly reduces the computational cost required for parameter tuning.
On the locomotion-v2 task set and the tasks of antmaze-umaze-v0 and antmaze-umaze-diverse-v0, our method significantly outperforms IDQL. In the remaining tasks, the reason for qGDP being weaker than IDQL may be attributed to the excessively large state space. The probability of each state being labeled with different labels significantly decreases, leading to insufficient training for quantile-guided diffusion.

Reviewer comment (Evaluation):
The paper over-emphasizes the impact of tuning the guidance scale, even though adjusting the scale for diversity is intrinsic to diffusion models themselves. However, the main innovation of the paper is using the quantile network to label samples and train the diffusion policy conditioned on quantiles. More analysis is needed on how the quantile inputs specifically affect diversity, rather than just tuning the guidance scale which is a general feature of diffusion models.

Author response:
Thank you for your question. 
The quantile input can take values larger than 1. In Figure 3, we visualize the generation results of qGQP under different guidance scales and quantile inputs. In each subfigure, points of different colors represent generated samples for quantile inputs y=0.5, 0.7, 1.0, and 1.3. The figure is evident that qGQP exhibits strong generalization capabilities, generating samples not present in the dataset and consistent with the underlying trend.
Furthermore, on the D4RL dataset, we experimented with different quantile inputs. The results indicate that both quantile input and guidance scale significantly impact the algorithm's performance, as shown in Figure 4.

Reviewer comment (Experiments):
In Table 4, the runtime of the proposed offline RL algorithm qGDP is compared to the older online RL algorithm DQL? I cannot get the meaning of this comparison. For a fair runtime comparison, it would make more sense to compare against other recent offline RL algorithms such as IDQL.

Author response:
Thank you for your suggestions. 
We test the release codes of DQL, IDQL, and SfB with the halfcheetah-medium-v2 task on a single RXT 2080Ti graphics card, and their runtimes are compared below:

method  | critic                             | actor | sampling | total 

DQL        | 8.6ms$\\times$2e6     | 12.4ms$\\times$2e6 | 51.2s$\\times$40 | 12.9h

IDQL      |               3.3ms$\\times$1.5e6 (critic and actor) | 70.0$\\times$6 | 1.5h

SfBC      | 68.4ms$\\times$4.9e4 | 39.1ms$\\times$5.9e5 | 50.6s$\\times$20 | 7.6h

qGDP     | 9.3ms$\\times$4e5      | 5.9ms$\\times$4e5 | 10.0s$\\times$70 | 2.0h

where, 'critic,' 'actor,' and 'sampling' respectively refer to the time spent on value function training, policy function training, and interacting with the environment of these algorithms. The recorded form of these terms is the time required for a single update of the value function multiplied by the number of updates, the time required for a single update of the policy function multiplied by the number of updates, and the time required for model evaluation (10 interactions with the environment) multiplied by the number of evaluations. 

It should be noted that IDQL is implemented based on JAX, resulting in shorter runtimes. In comparison, both qGDP and DQL are implemented in PyTorch. Specifically, in the actor update phase (related to training the diffusion model), qGDP exhibits significantly shorter runtime compared to DQL. During the model evaluation phase, qGDP reduces time consumption through parallelization.

Most importantly, the presented runtimes for DQL, IDQL, and SfBC are for a single run. If hyperparameter tuning is required, they would incur several times more runtime. In contrast, qGDP has completed hyperparameter tuning during the 70 model evaluation runs.

Reviewer comment (Writing):
The paper lacks an introductory explanation of quantile networks and their relevance to the proposed method, potentially hindering comprehension for readers less versed in the domain. The authors are encouraged to elaborate on the concept and role of quantile networks within qGDP to provide readers with a foundational understanding of the methodology. An illustration to overview the entire work would be helpful.

Author response:
Thank you for your suggestions. The relevant explanations are provided below, and we will incorporate these details into the main text in the subsequent revisions：
The quantile network, proposed by Dabney et al.[1], is designed to describe the action value distribution for each state-action pair (i.e., estimating the values corresponding to different quantiles under this distribution). It constructs the corresponding Distributional Bellman Operator to improve the performance of DQN. IQL employs a similar approach to characterize the action value distribution under the behavioral policy, and uses the values corresponding to high quantile points as an approximate estimate for the optimal action. This helps in avoiding querying out-of-sample actions in the TD loss.
In this work, we utilize the quantile network to describe the distribution of action values under the behavioral policy, assign different labels to samples based on the value according to the quantiles, and finally use these labeled samples to train a diffusion model. This trained diffusion model is able to generate appropriate actions based on the input quantiles and guidance scales and exhibits different decision preferences.
[1] Dabney W, Rowland M, Bellemare M, et al. Distributional reinforcement learning with quantile regression, Proceedings of the AAAI Conference on Artificial Intelligence. 2018, 32(1).

Reviewer comment (Novelty):
The core innovation, applying quantile labels to guide the diffusion process, appears somewhat incremental, casting doubt on the method's novelty. Clarification on the specific novelties and contributions of qGDP would be beneficial.

Author response:
Thank you for your suggestions. The main contributions of qGPD are as follows:
Hyperparameter tuning is the big problems that still lack a satisfying solution.[2] Most offline RL methods select the optimal hyperparameters by evaluating the performances of models trained with different hyperparameters by interacting with the environment[3]. This implies that these methods require repeating the training process much times. Our approach achieves a behavior-controllable policy with just one training iteration. By adjusting the input quantile and guidance scale, this policy can be biased towards generating actions with higher estimated values or towards actions that closely resemble the behavioral policy. 
[2] Levine S, Kumar A, Tucker G, et al. Offline reinforcement learning: Tutorial, review, and perspectives on open problems[J]. arXiv preprint arXiv:2005.01643, 2020.
[3] Prudencio R F, Maximo M R O A, Colombini E L. A survey on offline reinforcement learning: Taxonomy, review, and open problems[J]. IEEE Transactions on Neural Networks and Learning Systems, 2023.

Reviewer comment (Writing):
The introduction does not adequately articulate how qGDP surmounts the limitations of prior diffusion-based offline RL methods. It would be advantageous to outline explicitly how qGDP addresses the deficits of previous diffusion-based methods in the introduction.

Author response:
Thank you for your suggestions. The limitations of prior diffusion-based offline RL methods and the advantages of qGDP are listed below:
a. IDQL increases the probability of the diffusion model generating high-value actions by reweighting samples, so it cannot generate samples with higher values outside the dataset. In contrast, qGDP learns the probability distribution of actions conditioned on different quantiles, thus qGDP can output action distributions corresponding to higher action values (associated with quantile inputs greater than 1).  
b. DQL uses the Q-network to guide the training of the diffusion model, so it involves computing gradients for a relatively deep neural network. qGDP is designed to train the diffusion model to generate samples at different quantiles, so the computational cost of qGDP is comparable to that of a regular Diffusion model.
c. Adjusting the generation preference in other diffusion-based offline RL methods requires retraining the model, whereas qGDP only requires changing the model's input. This significantly reduces the computational cost required for parameter tuning.