Paper: Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning (ICLR-2024)

Reviewer comment (Novelty):
(major) The novelty of this paper is limited. The authors simply borrow an existing method from the vision community and apply it to the RL tasks, with some minor modifications. I do not see much novelty in doing so. This paper seems more like a technical report or experimental report, in that the authors conduct several experiments and summarize the conclusions. One serious flaw of this paper is that the author merely reports some experimental phenomenons while the corresponding explanations and discussion are unfortunately missing. From an ICLR paper of this kind, I would expect to understand why such a phenomenon occurs. This paper leaves me with more questions than answers. For example, why on many offline and offline2online tasks, the consistency model underperform the diffusion model, while on some online hard tasks, the consistency model seems to be better?

Author response:
We appreciate the insightful comments and suggestions from the reviewer. We will try our best to answer those questions raised by the reviewer to remove his/her concerns.

The novelty concern is replied in the above general response.

We appreciate the reviewer for raising the question regarding why consistency models are able to achieve higher online performances than diffusion models. Our conjecture is that the expressiveness of a model is more essential in offline setting than online setting, especially when multi-modal distributions appear in offline dataset and the algorithm is of the behavior-cloning (BC) type. Our results of BC in offline setting as Tab.1 show that the expressive policies can significantly improve the BC performances, which can be helpful for policies to quickly converge in offline RL.  However for online RL in Markov decision process, the theoretically optimal policy can always be deterministic, which indicates that the expressive models will shrink back to unimodal to approach optimality, thus expressiveness may only help with initial exploration instead of final convergence of the policy. This conjecture can be hard to verify consistently due to the heterogeneous distributions of different tasks. However, this is an interesting question and we will explain this in a modified version of the paper.

Reviewer comment (Evaluation):
(major) This paper does not consider statistical significance. Written statements and the presentation of the results as tables (often without standard deviations) obscure this flaw. In fact, ALL tables in this paper do not include any signal of statistical significance for baseline methods, e.g., std, IQM [1]. We have reached a point of maturity in the field where claims need to be made in reference to actual statistical evidence, which seems to be lacking in the current presentation.  
[1] Deep reinforcement learning at the edge of the statistical precipice. NeurIPS

Author response:
**Statistical significance**. 
For all the results derived by ourselves, we reported the standard deviations to show its statistical significance, for both algorithm performances as Tab.1,2,4 and Fig. 3 and time consumption as Tab. 3, also the learning curves in Fig. 4 are shaded by 95% confidence intervals. Other results without standard deviations are adapted from previous papers, where most of the values are reported without statistical significance. The results of BC, BEAR, BRAC, AWR, BCQ, SAC and CQL can be found in D4RL paper and CQL paper, all without standard deviations reported. The results of DT, TD3+BC, Onestep RL, AWAC, IQL can be found in IQL paper, Diffuser and IDQL can be found in IDQL paper, results of $\mathcal{X}$-QL and MoRel can be found in their own papers, these are all reported without standard deviations as well. For Tab. 1 and 2, we manage to find the standard deviations reported for ARQ and already fill the values in the modified draft. The results of SAC, AWAC, ACA in Tab. 4 are also reported without standard deviations in ACA paper. It turns out our paper is the very few papers reporting the statistical significance of our own algorithms in the domain. We appreciate the attention of the reviewer for the statistical significance of the results and we have tried our best to satisfy this requirement.

Reviewer comment (Evaluation):
(major) The proposed consistency-BC or consistency-AC do not show much improvement over the diffusion-based counterparts. If one looks at the experiments in this paper, it is clear that the consistency-BC and consistency-AC cannot beat diffusion-BC or diffusion-QL on most of the tasks. The authors say that consistency-AC or consistency-BC can achieve less training time. Well, I do not see this as an appealing advantage over the diffusion-based methods, since training cost is somewhat unimportant, while the most critical part, from my perspective, is the inference speed. Let us then take a look at the inference cost. Based on Table 3 in the main text, it is clear that the inference speed of diffusion-QL is quite similar to that of the consistency-AC. For instance, when setting $N=5$, the inference speed of diffusion-QL gives 3.76ms and consistency-AC gives 3.39ms, while their performance differs (diffusion-QL has an average score of 108.2, while consistency-AC only has 101.4). I actually do not see many advantages of utilizing consistency-AC or consistency-BC in practice.

Author response:
**Performances**.
We admit the current consistency model does not yield better performances than diffusion models for offline RL settings, and we do not expect it since otherwise the consistency model will not just be an acceleration method of the diffusion model but a total replacement of it. There is always the trade-off of computational efficiency and the model accuracy. We agree that the inference speed is more critical for RL. However, the Tab. 3 may be interpreted in another way by the reviewer as we intended. As also in above general response, in Tab. 3, our major claim is that Consistency-AC can achieve a similar performance as Diffusion-QL with a much smaller $N$, instead of comparing with the same $N$. Consistency-AC saturates its performances with $N=2$ for most of the tasks in our experiments rather than just this one. Specifically, if we look at $N=2$ for Consistency-AC while $N=5$ for Diffusion-QL, the scores are similar but Consistency-AC saves half inference time (1.84/3.76) and training time (31.94/57.06). By default in our experiments, Consistency-AC and Consistency-BC use $N=2$ while Diffusion-BC and Diffusion-QL use $N=5$ for results reported in paper.

Reviewer comment (Experiments):
(major) Even worse, it seems that the performance of consistency-AC and consistency-BC are acquired by carefully tuning the hyperparameters. As a piece of evidence, one can see the hyperparameter setup in Appendix C (Table 5). This indicates that the generality and effectiveness of the proposed method are limited.

Author response:
**Hyperparameter tuning**. We may disagree with the reviewer that hyperparameter tuning is making the proposed method less convincing. Actually, the hyperparameters in Tab. 5 are all inherited from Diffusion-QL paper [3] except for the Q norm, some are not reported in the previous paper but can be found in their experiments. Most of the present state-of-the-art algorithms for specific domains require hyperparameter tuning, which is quite common in literature, with evidence in Tab. 3 of paper [3] and Tab. 4 of paper [4]. They may not report full hyperparameters in the algorithms, and some other papers do not even report their hyperparameters, but this does not indicate that there is no hyperparameter tuning in their reported results. We don’t see any reason that our reported results with good hyperparameter tuning indicates the generality of the method is bad. We can understand the concern if the reviewer is asking how robust the proposed method is against each hyperparameter. This can be computationally heavy to evaluate and we do not see a lot of previous work reporting it. This is further answered in later question regarding the choice of gradient norm.

Reviewer comment (Writing):
(minor) On page 3, Eq 3 is the objective using clipped double Q-learning instead of vanilla double Q-learning. The authors ought to cite the TD3 paper here instead of the double Q-learning paper.

Author response:
We thank the reviewer for pointing out the mis-reference of Eq. 3, and will modify it in the modified version.

Reviewer comment (Reproducibility):
(minor) No codes are provided in this paper, and the authors do not include a reproducibility statement section in the main text.

Author response:
We plan to open-source the code after the review process and the clean-up of the code. We have added the statement in the modified version, thanks for pointing out.

Reviewer comment (Experiments):
Why do you use different baselines on different domains? It is somewhat confusing that you use some quite weak baselines on domains like antmaze, adroit, and kitchen. As an example, why do you use AWR, BRAC, and REM on adroit tasks and kitchen tasks in Table 1? It seems to me that the advantages of consistency-BC and consistency-AC are illustrated by carefully picking the baselines.

Author response:
For the questions of different baseline on different domains, those baseline results are adapted from previous Diffusion-QL paper [3] but not cherry-picked by us. We guess one reason that those baselines are filled on *Adroit* and *Kitchen* tasks is because some strong baselines (like MoRel, DT, Diffuser, etc) on Gym tasks do not report their values on *Adroit* and *Kitchen* tasks while these 'weak' baselines are reported. Our paper just tries to report thoroughly about previous baselines but not take any cherry-picking.

Reviewer comment (Experiments):
How important is the gradient norm to the consistency-BC and consistency-AC algorithms?

Author response:
For the question regarding the gradient norm, most of the gradient norm values are directly inherited from Diffusion-QL paper [3] given the usage of the same neural network architectures, which means we do not thoroughly search over the values. Exception exists for *Kitchen* tasks where we reduce the gradient norm values to stabilize training. However, this is still a rough tuning instead of a thorough searching over the values. A thorough search over hyperparameters can be computationally very expensive. We found the Q value normalization and coefficient $\eta$ to be more influential on performances than gradient norm.

Reviewer comment (Evaluation):
The results in Tab. 1 and 2 can not support the conclusion in a way, which shows there are some significant drops in some tasks (such as halfcheetah-me, kitchen-xxx) between the Diffusion-BC and Consistency-BC or Diffusion-QL and Consistency-AC.

Author response:
Less comparable performance for consistency models on *halfcheetah-me*, *kitchen-xxx* in Tab. 1 and 2. We admit that there are some tasks showing consistency policies are less performant than diffusion policies, and the computational efficiency is not obtained for free compared with diffusion models at present, as it is an acceleration method without full capability of replacing the diffusion models with a much smaller $N$ yet. However, there are still quite a few tasks showing at least equivalent performances of consistency policies over diffusion policies, like *hopper-medium*, *walker2d-medium*, *hopper-medium-replay*, *walker2d-medium-replay*, *Antmaze* tasks on BC setting, and *halfcheetah-medium*, *halfcheetah-medium-replay*, *walker2d-medium-expert*, *antmaze-umaze-diverse* for actor-critic RL setting.

Reviewer comment (Experiments):
The authors claim that "By behavior cloning alone (without any RL component), using an expressive policy representation with multi-modality like the consistency or diffusion model achieves performances comparable to many existing popular offline RL methods.", but it is unclear where the multi-modality is. Is the consistency policy trained for all tasks across all suites? Or does the offline dataset have different successful behavior policies? The author should make this more clear.

Author response:
For the first question regarding the multi-modality of the data, the model is not trained across tasks but within each task. Even so, the offline dataset for each task is usually multi-modal since it can be collected by a mixing of behavior policies. For example, the 'medium-replay' dataset in Gym tasks is the replay buffer of a RL policy to reach a medium level performance, which contains samples collected by an evolving policy. In our paper Fig. 5 in Appendix A shows the diversity of the data distribution, although it is not the action distribution given a certain state, the shapes also indicate the complexity of the sample distributions. More details about these tasks are provided in the original D4RL paper. We thank the reviewer for pointing out this problem and will add explanations in the modified version.

Reviewer comment (Experiments):
In the specific tasks in RL, the scenarios are not abundant and the trajectories are quite similar. If more expressive policy representation can result in better performances, what if using some large pre-trained representation models, can this problem be solved? (E.g. R3M)

Author response:
For the second question regarding whether large pre-trained representation model will be an alternative of expressive policy,  we believe pre-trained visual representation can definitely lead to improved performance for certain domains of RL, e.g., robotic manipulation, but the improvement can source from different aspects compared with the expressiveness of the policies. Pre-trained visual representation is more helpful for providing image information abstraction with semantic understanding. R3M is encoding images as state representation, while consistency/diffusion model represents the conditional distribution of actions given states, so the representation spaces are not the same. If R3M is modified to encode both images and robot actions, while maintaining a multi-modal model, it might be close to the expressive policies. But for current R3M in imitation learning, if the behavior dataset is multi-modal, and the downstream policy of R3M is unimodal (like Gaussian), we may not expect it to capture the multi-modality.

Reviewer comment (Experiments):
On the harder tasks like Kitchen and Adroit, there is a significant gap with Diffusion-BC baseline.

Author response:
For the performance gap on Kitchen and Adroit tasks, we show that the computational efficiency is not obtained for free compared with diffusion models. The dataset of the two domains has poor coverage (Adroit) or long-term trajectories (Kitchen), which makes them harder to learn. How to improve the performance of consistency models on these tasks requires further investigation.