Paper: Projected Off-Policy Q-Learning (POP-QL) for Stabilizing Offline Reinforcement Learning (ICLR-2024)

Reviewer comment (Novelty):
The framework is described as an incremental advancement on Kolter (2011). An incremental approach might not provide significant improvements or breakthroughs over existing methodologies.

Author response:
Regarding the “incremental advancement” point, we strongly disagree with this point. We do share the motivation and use of the contraction mapping condition with Kolter (2011), but our approach differs greatly after that. Firstly, the approach in Kolter (2011) only works for very small MRPs. Our approach uses a completely new technique for satisfying the contraction mapping condition and includes a policy projection step that is crucial for policy optimization. We will make sure to emphasize this further.

Reviewer comment (Evaluation):
While the paper presents Projected Off-Policy Q-Learning (POP-QL) as a novel method with certain improvements over the previous works, the evaluation results indicate that it struggles to match or surpass state-of-the-art performance in scenarios with near-expert data-collection policies. This raises questions about its practical utility and whether the theoretical improvements translate to real-world benefits.

Author response:
Regarding the poor performance results, what is unique about our method is that, unlike the state-of-the-art offline RL techniques, our method does not regularize the policy towards the data policy. For that reason, we expect conservative methods and explicit policy regularization methods to out-perform our method on near-optimal datasets. In our (updated) small-scale results and the “random” D4RL results we show that as the dataset becomes less optimal, our method outperforms other methods. We will try to make this point clearer.

Reviewer comment (Experiments):
How generalizable is the contraction theory discussed in this paper to other offline algorithms that use a structure of Soft Actor-Critic (SAC) like Conservative Q-Learning (CQL)? Could the author provide an ablation study to test the effect of different components (including the specific design in Section 4.3 and the policy optimization) on the performance of the algorithm?

Author response:
Regarding the ablation study question, this is a good suggestion. From our informal tests, without the policy projection, POP-QL diverges and when combining CQL with POP-QL we get some slight improvements, but we will conduct these experiments formally and add them to the results.

Reviewer comment (Experiments):
Will there be some other datasets that the algorithm can perform better than other SOTA methods? The authors mentioned that this algorithm performs better when the datasets are far from the optimal policy distribution, specifically the “random” datasets. However, this method still performs worse than other method in walker2d-random. Will the author generate some new datasets using some random policies in the mujoco environment and test the algorithm's performance?

Author response:
Your questions regarding additional datasets is a great point. We tried to stick with common benchmark datasets because we didn’t want the datasets to seem contrived. But adding additional datasets might further illustrate that point.

Reviewer comment (Experiments):
The baselines compared do not fully reproduce what has been reported in the literature.

Author response:
Your point about the baselines is a good point and something we struggled with. The reason is around the “Lagrange threshold” parameter discussed in the Appendix F of Kumar et al. (2020). The goal of this parameter is to automatically tune the conservatism parameter, \alpha, to the specific task (similar to the automatic entropy tuning of SAC). However, when using the library provided by the authors (https://github.com/young-geng/JaxCQL), using the automatic \alpha tuning significantly reduces performance for some of the tasks compared to a fixed \alpha. Reading through the paper you sent, it looks like Li et al. (2023) used a fixed \alpha parameter, which is different from the parameters reported in Kumar et al. (2020). For our next revision, we will re-run the CQL experiments using a fixed \alpha parameter and report both numbers.

Reviewer comment (Theory):
The derivation of Lemma 1 is incomplete as it uses Theorem 1, which is not given.

Author response:
Good catch on the proof of Lemma 1! This is meant to refer to Theorem 2 from Kolter (2011). In a previous draft of the paper, we restated this theorem, which is why there was a missing reference.

Reviewer comment (Theory):
In (9), it shows \(q^*\) is proportional to things on the right, not equal; the missing normalization factor, while not a function of (s, a), is a function Z. Therefore, (10) is not exactly equal to (8). The same issue occurs for (18) later on.

Author response:
Regarding the derivation bug, we do in fact keep track of the normalization term, even though we don’t explicitly state it in the paper. In fact, ignoring the normalization term leads to a very different result. Below is our step-by-step derivation. We’ll try to make this clearer.  
R2: Our analytical solution (equation 9) can be written as:

$q^{\star}(s) = \frac{1}{C} \ \mu(s) \exp \left(\operatorname{tr} Z^T F(s)\right)$

With $C = \sum_s \mu(s) \exp \left(\operatorname{tr} Z^T F(s)\right)$. 
Plugging this into equation 8 yields.

$\max_{Z \succeq 0} \mathrm{KL}(q^* \| \mu)-\operatorname{tr} Z^T \mathbf{E}_{q^*}[F(s)]$

$\max_{Z \succeq 0} \ \mathbf{E}_{q^*} [ \log \left(\frac{1}{C} \frac{q^*(s)}{\mu(s)} \right) ] - \operatorname{tr} Z^T \mathbf{E}{q^*} \left[ F(s) \right]$

$\max_{Z \succeq 0} \ \mathbf{E}_{q^*} \left[- \log(C) + \log \left( \exp \left(\operatorname{tr} Z^T F(s)\right) \right) \right] -\operatorname{tr} Z^T \mathbf{E}{q^*}[F(s)]$

$\max_{Z \succeq 0} \ - \log(C)$

$\max_{Z \succeq 0} \ - \log\left(\mathbf{E}_{\mu} \exp \left(\operatorname{tr} Z^T F(s)\right)\right)$

Reviewer comment (Theory):
In (24), how is \(\nabla_\pi E_\mu[ Q^\pi(s,a)]\) computed? How do you get \(Q^\pi\) (which cannot be estimated in general for offline RL) and how is its gradient with respect to \(\pi\) computed?

Author response:
Regarding your question on equation 24, this is the gradient with respect to the Q-network not the true Q-function. We tried to make the equation less cluttered by removing the parameter subscript, but that clearly made it more confusing. We will make that clearer.

Reviewer comment (Reproducibility):
The baselines' performance does not reproduce known results from the literature; while exact replication is not required, the current results are far from the typical ballpark. For example, CQL has been reproduced by many papers with roughly 10%-20% differences from the original paper's results (see Li et al. 2023 as a recent example). Failure to reproduce existing results makes the experimental results questionable.

Author response:
Your point about the baselines is a good point and something we struggled with. The reason is around the “Lagrange threshold” parameter discussed in the Appendix F of Kumar et al. (2020). The goal of this parameter is to automatically tune the conservatism parameter, \alpha, to the specific task (similar to the automatic entropy tuning of SAC). However, when using the library provided by the authors (https://github.com/young-geng/JaxCQL), using the automatic \alpha tuning significantly reduces performance for some of the tasks compared to a fixed \alpha. Reading through the paper you sent, it looks like Li et al. (2023) used a fixed \alpha parameter, which is different from the parameters reported in Kumar et al. (2020). For our next revision, we will re-run the CQL experiments using a fixed \alpha parameter and report both numbers.

Reviewer comment (Writing):
Minor: In (7) \(D_{KL}\) = KL?

Author response:
Thanks for the note on using two different notations for KL. We fixed that in the newest version.

Reviewer comment (Evaluation):
Although this paper tackles an important problem in RL, the experimental results fail to demonstrate any advantages gained from employing this method in offline RL. Moreover, this method introduces more complexities compared to other offline RL methods and has more moving parts which further diminishes its appeal in light of the performance of other offline methods which outperform this method.

Author response:
The reason we chose to evaluate the fully offline setting is because the challenges of distribution shift are more pronounced. Unlike our method, all the best performing offline RL methods have some regularization back to the data policy. For that reason, of course our method will not be able to compete with policy-regularization or conservative methods on near-optimal datasets. However, our (updated) small scale results and the “random” D4RL results illustrate that this method can outperform policy-regularization and conservative methods when the datasets are severely sub-optimal. We will try to further highlight this.

Reviewer comment (Novelty):
The paper fails to adequately discuss relevant off-policy methods that deal with distribution mismatch such as [1,2] among many others. In particular, this paper should have discussed and compared with existing off-policy RL methods.

Author response:
Thank you for pointing out the missing literature. We agree we should spend more time discussing other off-policy methods and will make sure to include those papers.

Reviewer comment (Theory):
Authors mentioned in page 3 that "One critical challenge with IS methods is that they do not address support mismatch and, thus, tend to perform poorly on larger scale problems." That is not entirely correct as the main objective of importance sampling is to alleviate distribution mismatch by learning unbiased estimators, albeit at the expense of increased variance.

Author response:
Regarding your point about IS methods, our understanding is most IS methods (like many other off-policy methods) assume the support of the data and candidate policies match, otherwise there would be division-by-zero issues. However, in the offline RL case, this assumption is often violated. Could you elaborate on what part you disagree with?