From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: The paper provides an application of Taylor expansion based objective + behavior regularization in the offline RL setting, and has shown empirical improvements over baseline algorithms. This might motivate carrying out analytic updates (based on Taylor expansion derived objective) in future investigation on offline RL.

Strengths: The idea of having a closed-form solution for the policy improvement step is exciting and novel.
The idea is easy to implement and can be used as a standalone method and in combination with offline reinforcement learning methods such as OnestepRL and IQL, which do not require fitting a policy for value learning.
The method improves over the baselines on a standard benchmark for offline reinforcement learning.

Weaknesses: Using Taylor expansion is not novel in the RL community in general. As the authors have noted, PDL itself arises from the application of Taylor expansion to the value function objective. Renowned algorithms such as TRPO and PPO have applied the results to trust region setting, where there is a regularization from the behavior policy. Lately, Tang et al. has developed second order Taylor expansion approach to a similar setup in online RL.

While the theory does require adaptations from the original PDL, the adaptation is pretty straightforward.

The connection between theory and practice is weak.

Overall there was not enough enthusiasm among the reviewers to warrant publication at ICLR.
justification_for_why_not_higher_score: See the weaknesses above.  Overall the level of enthusiasm among the reviewers is too low to warrant acceptance --- many papers were more highly rated.
justification_for_why_not_lower_score: This is the lowest score.

From: ICLR.cc/2023/Conference/Paper1188/Authors
title: Evaluation results of our iterative algorithm on AntMaze!
comment: Dear Reviewer CVbg,

Thank you again for your valuable feedback! As the rebuttal period is coming to an end, we would like to gently remind you that we have added the experimental results to demonstrate the effectiveness of combining our CFPI operators with iterative policy evaluation. On the challenging AntMaze domain, our Iterative $\mathcal{I}\_{\text{MG}}$ outperforms all baselines on 5 out of 6 tasks and obtains the best overall performance.

We hope that the new results address your concerns. If you have any further feedback or concerns, please do not hesitate to let us know. We would be happy to address them.

Best regards,

The authors

From: ICLR.cc/2023/Conference/Paper1188/Authors
title: Looking forward to further discussion
comment: Dear Reviewer m3D5,

Thank you again for your time and efforts in reviewing our work. We have revised Theorem 3.1 and the manuscript based on your valuable feedback. Moreover, we have provided additional experiments to demonstrate the effectiveness of combining our CFPI operators with iterative policy evaluation. If our responses addressed your concerns, we would be grateful if you could re-evaluate our work.

If you have further questions or comments, we would be happy to discuss them with you!

Best regards,

The authors

From: ICLR.cc/2023/Conference/Paper1188/Authors
title: Looking forward to further discussion
comment: Dear Reviewer ypTb,

Thank you again for your time and efforts in reviewing our work. We have provided additional experiments and revised the manuscript based on your valuable feedback. If our responses addressed your concerns, we would be grateful if you could re-evaluate our work.

If you have further questions or comments, we would be happy to discuss them with you!

Best regards,

The authors

From: ICLR.cc/2023/Conference/Paper1188/Authors
title: Looking forward to further discussions
comment: Dear Reviewer CVbg,

Thank you again for your time and efforts in reviewing our work. We have provided additional experiments and revised the manuscript based on your valuable feedback. If you have further questions or comments, we would be happy to discuss them with you!

Best regards,

The authors

From: ICLR.cc/2023/Conference/Paper1188/Authors
title: (2/2) Response to Reviewer m3D5 
comment: >The theory front of the paper is weak in that much of the results come from technical derivation

Could you please elaborate on your question so that we can respond better? We believe that technical derivation will not weaken our theoretical results.


>Thm 3.1 is a fairly straightforward application of PDL (admittedly, certain adaptations are needed since in this case the action is being expanded in the first order, but not the policy as in the original PDL case). If we obtain the finite sample bound version of Thm 3.1, does it improve over the bound derived in CQL? How do they compare?

Our theorem is based on PDL for deriving policy improvement, but technicality subtleties make our analysis non-trivial. We mention two parts. First, our result does not rely on the assumption that the estimated Bellman operator is close to the true one. Unlike CQL, where the safe policy improvement is based on **assuming** $|\hat{\mathcal{B}}-\mathcal{B}|(s,a)\leq C/\sqrt{D(s,a)}$, we can **prove** $|Q^\pi -\hat{Q}^\pi|\leq C/\sqrt{D(s,a)}$ by using concentration like Bernstein inequality (this is summarized in "On the learning coefficient of SARSA" discussion in Appendix A.4). This makes our theorem widely applicable since we rely on very weak assumptions. Second, we include the multi-step result as well, which might yield further improvement in addition to the original one-step improvement. In contrast, the theoretical analysis of CQL is designed for analyzing the optimal policy by its definition, which cannot be directly transferred to study the one-step update. Therefore, our theorem is more flexible.

>**Taylor expansion based approach to RL** 

We thank the reviewer for raising these meaningful questions and providing an excellent starting point for us to compare with the prior works. In Appendix H, we included an additional discussion of the relation between our work and prior works [1, 2, 3, 4] that leveraged the Taylor expansion approach. In summary, we argue that 
1. Our motivation to use Taylor expansion is fundamentally different from the previous works, although both our works and previous methods propose to create a surrogate of the original policy objective by leveraging the Taylor expansion approach.
2. Our closed-form policy update is made possible by directly optimizing the parameter of the policy distribution (Dirac distribution in our case).

> why the new algo performs better than the previous SOTA, e.g., IQL in M and ME dataset, but not as much in the MR dataset (in Table 1).

In this particular case, we believe the broader state-action coverage of the MR datasets contributes to the decent performance of both IQL and our methods. Therefore, the performance differences are not significant.

>...I think adding a "total" row in the table is not very informative...

We add the "total" row in our tables by following the convention in offline RL literature.

From: ICLR.cc/2023/Conference/Paper1188/Authors
title: (1/2) Response to Reviewer m3D5
comment: Thank you for the valuable feedback and comments. We provide our detailed responses below

>The theoretical and conceptual contributions of the paper are a bit weak.

While we would like to emphasize the main contributions of our paper come from the empirical perspective, we respectfully disagree with your evaluation of our theoretical and conceptual contribution. Our methods are derived rigorously by leveraging the first-order Taylor approximations to the policy objective. However, deriving a closed-form solution even from the linear objective is non-trivial when modeling the behavior policy as a Gaussian Mixture, as it induces a non-convex constraint, as shown in Equation (10). To combat the optimization difficulties, we leverage LogSumExp's lower bound and Jensen's Inequality, finally giving rise to a closed-form policy improvement operator that can work with the Gaussian Mixture behavior policy.

Also, in the updated version of Theorem 3.1, we have updated the one-step policy improvement version to the multi-step version. Our result does not assume the estimated Bellman operator accurately approximates the true Bellman operator (like CQL did), but rather directly shows SARSA or OPE estimator can provide an accurate estimator of $Q^\pi$. We believe it makes our result stronger. See our response that compares our method to CQL and PDL for a more detailed discussion.

Finally, we would be extremely grateful if you might consider evaluating our paper based on its core empirical contributions. Our empirical results show that our CFPI operators can be used to construct successful offline RL algorithms in both one-step and iterative manners, demonstrating superior performance over SOTA methods on the standard D4RL benchmarks. In Section 5.1 and Appendix C.1, our newly added Iterative $\mathcal{I}\_{\text{MG}}$ achieves impressive results by outperforming SOTA baseline methods on 5 out of the 6 tasks from the challenging AntMaze domain.

>It is not clear why carrying out such updates should be useful in the offline RL setting in the first place. Is this due to statistical efficiency? **Why closed form update is useful in offline RL case**?

We argue that having a closed-form policy improvement operator in the offline RL setting is quite attractive. Both our CFPI operator and Rev. KL Reg. require learning a Q-function $\hat{Q}$ and behavior policy $\hat{\pi}\_{\beta}$. While Rev. KL Reg. requires training for 100K gradient steps to perform one-step policy improvement, our $\mathcal{I}\_{\text{MG}}$ directly returns the improved policy as $\mathcal{I}_{\text{MG}}(\hat{\pi}\_{\beta}, \hat{Q})$, avoiding this extra computation. Moreover, empirical results in Table 4 and Fig. 2 show that the policy returned by our $\mathcal{I}\_{\text{MG}}$ outperforms the policy learned by Rev. KL Reg by a clear margin.
   
Our CFPI operators solve the policy improvement step in closed form. Such merit becomes even more attractive when combined with iterative offline RL algorithms. Given the non-linear policy objective, existing iterative actor-critic methods (e.g., CQL, TD3+BC) optimize the policy towards its training objective for only one gradient step every policy update iteration. In contrast, our CFPI operators enable the exact solution to the policy improvement step given a learned Q-function and behavior policy, contributing to the superior performance of our Iterative $\mathcal{I}\_{\text{MG}}$ on the AntMaze domain as is shown in Section 5.1 and Appendix C.1.

>Does it help stabilize the update? 

Results in Section 5.2 clearly expose the problem of training instability of IQL's policy improvement operator, which optimizes towards the policy objectives via SGD. As is shown in Table 3, the final 1M-step policy $\pi_{\text{IQL}}$ (1M) cannot match the running average of the evaluation results during training $\pi_{\text{IQL}}$ (train), especially on the `antmaze-m-p` and `antmaze-m-d` tasks. This instability is also well documented in the previous offline RL literature [1]. In contrast, our CFPI operators do not suffer from the instability issue thanks to the closed-form solution. 

Moreover, results in Table 4 show that when optimizing towards the same Q-function, the performance obtained by the learning-based policy improvement operator Rev. KL Reg suffers from a significant amount of variance, echoing the instability issue we propose to resolve. In contrast, learning-free methods EBCQ and our CFPI operators achieve robust performance with a much lower variance.

Finally, we do not claim that any iterative algorithm equipped with our CFPI operators will be absolved from the problem of training instability, as the policy evaluation steps may still introduce a non-trivial amount of instability due to stochastic updates, which calls for future work to address.

[1] Fujimoto, Scott, and Shixiang Shane Gu. "A minimalist approach to offline reinforcement learning." Advances in neural information processing systems 34 (2021).

From: ICLR.cc/2023/Conference/Paper1188/Authors
title: (2/2) Response to Reviewer ypTb 
comment: >What will happen if the value estimation suffers from the over-estimation issue on out-of-distribution data points?

As we use the SARSA-style update to learn the value function in our one-step algorithm, there is no over-estimation issue during the policy evaluation step. As for our iterative algorithm, we leverage the Clipped Double Q-learning technique to avoid overestimation, as is shown in Appendix C.1.

From: ICLR.cc/2023/Conference/Paper1188/Authors
title: (1/2) Response to Reviewer ypTb
comment: >Could you please show the performance when the operator is combined with other offline RL algorithms, such as CQL and TD3-BC?

As you mentioned in the review, our CFPI operator is compatible with the general actor-critic offline RL algorithm. In Section 5.1 and Appendix C.1, we instantiate an iterative algorithm with our CFPI operator $\mathcal{I}\_{\text{MG}}$, which is equivalent to running TD3+BC with the original policy improvement step replaced with our $\mathcal{I}\_{\text{MG}}$. Note that we also leverage the IQN architecture to model the Q function. Evaluation results demonstrate the superior effectiveness of our Iterative $\mathcal{I}\_{\text{MG}}$, which outperforms SOTA methods on the challenging AntMaze domain.

In the case you are asking if our method can be leveraged to improve the policy learned by a general offline RL algorithm, we additionally show that our method can improve the policy learned by CQL to some extent in Appendix G.7.

>Could you provide a specific example where existing offline RL algorithms to learn value function and policy via SGD really suffers from the instability issue

We believe the results in Table 3 clearly exposes the problem of training instability of IQL, as the final 1M-step policy $\pi_{\text{IQL}}$ (1M) does not match the running average of the evaluation results during training, especially on the `antmaze-m-p` and `antmaze-m-d` tasks. We also note that such instability is well-documented in previous offline RL literature [1].

Moreover, results in Table 4 show that when optimizing towards the same Q-function, the performance obtained by the learning-based policy improvement operator Rev. KL Reg suffers from a significant amount of variance, echoing the instability issue we propose to resolve. In contrast, learning-free methods EBCQ and our CFPI operators achieve robust performance with a much lower variance.

[1] Fujimoto, Scott, and Shixiang Shane Gu. "A minimalist approach to offline reinforcement learning." Advances in neural information processing systems 34 (2021).

> Although there are experimental results showing the proposed method is better than existing methods, it is unclear whether the advantage really comes from better training stability.

While our methods are partly motivated by the learning instability issue caused by SGD, another critical aspect is that the requirement of a limited distributional shift in the offline RL setting naturally motivates our use of the first-order Taylor approximation to the policy objective, enabling a closed form solution to policy improvement. Moreover, we do not claim that resolving the training instability results in all of our improvement over the baseline methods. Instead, results in Table 4 (Section 5.3) show that modeling the behavior policy as a Gaussian Mixture helps us better capture the underlying distribution, which is a significant contributor to the performance improvement.

Additionally, our Iterative $\mathcal{I}\_{\text{MG}}$ solves the policy improvement step in closed form in every training iteration. In contrast, existing iterative actor-critic methods (e.g., CQL, TD3+BC) optimize the policy towards its training objective for only one gradient step every policy update iteration. We believe such an algorithmic difference contribute to the superior performance of our Iterative $\mathcal{I}\_{\text{MG}}$.

>Does it mean that the policy evaluation step in Algorithm 1 should be very accurate? 

As is shown in our newly added results in Section 5.1 and Appendix C.1, the policy evaluation step in Algorithm 1 does not necessarily need to be very accurate. Even though the policy evaluation step of our Iterative $\mathcal{I}\_{\text{MG}}$ only updates the critics for one gradient step every iteration, it can still achieve superior performance.

>Does it affect the choice of $\mathcal{E}$ in Algorithm 1? 

Current literature shows that different policy evaluation operators $\mathcal{E}$, e.g., Monte-Carlo evaluation or temporal-difference learning, can result in different generalization abilities of the learned value function [2, 3]. In our paper, the $\mathcal{E}$ learns the value function via TD(0) [4].  Specifically, our one-step algorithm uses a SARSA-style update to learn the value function with the next action $a'$ sampled from the replay buffer. On the other hand, our iterative algorithm employs a Q-learning-style update with the next action $a'$ generated by the policy returned by our CFPI operator $\mathcal{I}\_{\text{MG}}$. In both cases, our methods can obtain great performance.

[2] Kumar, Aviral, et al. "Dr3: Value-based deep reinforcement learning requires explicit regularization." arXiv preprint arXiv:2112.04716 (2021).

[3] Lyle, Clare, et al. "Learning Dynamics and Generalization in Reinforcement Learning." arXiv preprint arXiv:2206.02126 (2022).

[4] Sutton, Richard S., and Andrew G. Barto. Reinforcement learning: An introduction. MIT press, 2018.

From: ICLR.cc/2023/Conference/Paper1188/Authors
title: Response to reviewer CVbg
comment: Thank you for the valuable feedback and the suggestions to evaluate our iterative algorithms on the AntMaze domain! We provide our detailed response below.

> ...The experimental evaluation needs to be extended to demonstrate how the method combined with the iterative policy evaluation performs on antmaze tasks.

In our revised manuscript, we instantiate an iterative algorithm with our CPFI operator in Section 5.1 and Appendix C.1. On the challenging AntMaze domain, our Iterative $\mathcal{I}_{\text{MG}}$ outperforms all baselines on 5 out of 6 tasks and obtains the best overall performance. We note that we use the same set of HP for all tasks. Please refer to Appendix C.1 for additional details with the respective pseudo-codes and training curves. 


> I will increase my score if the authors also add the results on antmaze for the method.

We have included the results for our $\mathcal{I}_{\text{MG}}$ in Section 5.1 and Appendix C.1. We believe that the results are quite promising, and are happy to follow up the discussion if you have any further concerns!

From: ICLR.cc/2023/Conference/Paper1188/Authors
title: Common responses to reviewers: New results with our iterative algorithms on the AntMaze Domain
comment: We thank all reviewers for their time and efforts in providing valuable feedback on our work! We have updated our manuscript and responded to individual reviews below. Here, we would like to bring your attention to our new results that may interest all of you.

1. We instantiate an iterative algorithm called Iterative $\mathcal{I}\_{\text{MG}}$. In Section 5.1, we show that Iterative $\mathcal{I}\_{\text{MG}}$ outperforms baseline methods on 5 out of 6 tasks and obtains the best overall performance, demonstrating the effectiveness of combining our CFPI operators with iterative policy evaluation. We also include detailed training procedures, HP settings, and pseudo-codes in Appendix C.1.
2. We revise Theorem 3.1 and extend it to address the multi-step update. Our result does not assume the estimated Bellman operator accurately approximates the true Bellman operator (like CQL did), but rather directly shows SARSA or OPE estimator can provide an accurate estimator of $Q^\pi$. We believe it makes our result stronger.

3. We revise the introduction to emphasize that we only approach the training instability from the policy learning perspective in this work.

4. We cite the paper showing that deep RL algorithms' performance can exhibit significant variance across different random seeds.

From: ICLR.cc/2023/Conference/Paper1188/Reviewer_CVbg
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper introduces a method for constraining a policy for offline reinforcement learning. In particular, based on the first order Taylor approximation of the objective, the authors introduce closer form policy improvement operators.
strength_and_weaknesses: # Strength
* The idea of having a closed-form solution for the policy improvement step is exciting and novel.
* The idea is easy to implement and can be used as a standalone method and in combination with offline reinforcement learning methods such as OnestepRL and IQL, which do not require fitting a policy for value learning.
* The method improves over the baselines on a standard benchmark for offline reinforcement learning. 

# Weaknesses
* The method outperforms the policy extraction scheme used in the original implementation of IQL on the antmaze tasks. However, the experimental evaluation needs to be extended to demonstrate how the method combined with the iterative policy evaluation performs on antmaze tasks.
clarity,_quality,_novelty_and_reproducibility: In overall, the paper is easy to follow. The method is novel.
summary_of_the_review: The paper is interesting and introduces a simple method that outperforms the baselines. Therefore, I recommend this paper for acceptance. I will increase my score if the authors also add the results on antmaze for the method.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1188/Reviewer_ypTb
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper focuses on behavior-constrained policy optimization in offline RL problems. The authors use first-order Taylor approximation to get a closed-form solution for the policy update objective with behavior constraint. The behavior policy can be modeled as a single Gaussian or Gaussian mixture model and this paper provides a closed-form policy improvement operator in both cases. 

The authors described a detailed theoretical derivation of this closed-form operator to map the behavior policy to a higher-valued policy. The policy-improved operator can be incorporated into general actor-critic offline RL algorithms to replace the policy update step. The experimental results empirically demonstrate that the proposed method is better than the existing SOTA methods on D4RL.
strength_and_weaknesses: Strengths:
The authors provide both theoretical foundation and empirical results to demonstrate the efficacy of the proposed method.
The evaluation shows convincing metrics in Figure 2.
The ablative study is extensive.
The proposed method is novel because all prior works conduct the policy update step in offline RL via SGD, but this paper introduces a closed-form solution.

Weaknesses:

As shown in Algorithm 1, the closed-form policy improvement operator should be compatible with the general actor-critic offline RL algorithm. Could you please show the performance when the operator is combined with other offline RL algorithms, such as CQL and TD3-BC? Does it always bring improvement no matter what's the base offline RL algorithm?

One strong motivation for this work is that "training stability poses a major challenge in offline RL". Could you provide a specific example where existing offline RL algorithms to learn value function and policy via SGD really suffers from the instability issue, and the proposed method directly solved this? Although there are experimental results showing the proposed method is better than existing methods, it is unclear whether the advantage really comes from better training stability. So the claim in the introduction about "mitigate the issue of learning instability by leveraging optimization techniques" is not fully supported.
clarity,_quality,_novelty_and_reproducibility: Clarity:
This paper is generally well-written at a high level. I did not check the derivation in Sec. 3 line-by-line, so no comments about the clarity in details.

Quality:
Overall the quality is great, especially the evaluation and analysis part. 

One concern about the limitation. As mentioned in Sec. 6, the proposed method requires learning a good Q function. Does it mean that the policy evaluation step in Algorithm 1 should be very accurate? Does it affect the choice of \Epsilon in Algorithm 1? What will happen if the value estimation suffers from the over-estimation issue on out-of-distribution data points?

Novelty:
The proposed closed-form policy improvement operator is novel. In offline RL, it is the first work to replace SGD optimization for policy updates with a closed-form solution.

Reproducibility:
The code is provided in supplementary materials.




summary_of_the_review: A novel optimization approach for policy update with behavior constraints in offline RL, well supported by theoretical derivation and extensive evaluation.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1188/Reviewer_m3D5
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper proposes a closed form update for offline RL. A few major assumptions and steps of derivations apply: (1) policy is Gaussian or mixture of Gaussians; (2) apply Taylor expansion to the optimization objective. The paper shows some theoretical properties of the proposed method, and empirical improvements over prior work baselines.
strength_and_weaknesses: === Strength ===

The paper provides an application of Taylor expansion based objective + behavior regularization in the offline RL setting, and has shown empirical improvements over baseline algorithms. This might motivate carrying out analytic updates (based on Taylor expansion derived objective) in future investigation on offline RL.

=== Weakness ===

The theoretical and conceptual contributions of the paper are a bit weak. Though the contribution of deriving analytic updates based on Taylor expansion + Gaussian policy assumption is technically solid, it is not clear why carrying out such updates should be useful in the offline RL setting in the first place. Is this due to statistical efficiency? Does it help stabilize the update? The theory front of the paper is weak in that much of the results come from technical derivation, or Thm 3.1 is a fairly straightforward application of PDL (admittedly, certain adaptations are needed since in this case the action is being expanded in the first order, but not the policy as in the original PDL case). 
clarity,_quality,_novelty_and_reproducibility: === Clarity ===

The paper is written pretty clearly.

=== Quality ===

The contribution of the paper is not strong enough, both in terms of novelty and theoretical statements. Otherwise, the paper provides some empirical evidence of the merits of the new method.

=== Novelty ===

The application of Taylor expansion based approach to offline RL is novel, but the method itself is not very novel in the general RL setting.

=== Reproduce ===

The results should arguably be reproducible because the source code is provided.
summary_of_the_review: A few main questions.

=== **Taylor expansion based approach to RL** ===

Using Taylor expansion is not novel in the RL community in general. As the authors may have noted, PDL [1] itself arises from the application of Taylor expansion to the value function objective. Renowned algorithms such as TRPO and PPO have applied the results to trust region setting, where there is a regularization from the behavior policy (or any regularized policy chosen by the algo designers) [2]. Lately, Tang et al. [3] has developed second order Taylor expansion approach to a similar setup in online RL.

Admittedly, there is some difference between the setup in the current paper and aforementioned work, most notably, here the Taylor expansion is applied to the action directly (mainly because the policy is deterministic) whereas prior work mostly handles stochastic policy. In prior work, the expansion takes place in the policy distribution space; here, the expansion takes place in the action space. Such technical differences aside, I think it is worth discussing current work's relation with such prior work.

[1] Approximately optimal approximate RL, Kakade et al, 2007
[2] Proximal policy optimization, Schulman et al, 2015
[3] Taylor expansion policy optimization, Tang et al, 2020

=== **Why closed form update is useful in offline RL case** ===

Though experiments have showed performance gains, I think it helps the paper can be greatly improved to clarify more on the motivational question: why is using closed form update useful in the offline RL case. Intuitively, I think this might make sense because compared to usual SGD based updates, closed form update may reduce the stochastic jittering a bit and hence make the update more stable. Can we characterize the effect in theory? Or can we have some clean simple setup to illustrate this point? This question I think is quite fundamentally interesting overall but is a bit lacking in the current paper.

=== **Theory is a bit weak** ===

The flagship theoretical result in the paper is Thm 1, where they show an one-step improvement bound on the objective. On the technical side, I think showing the theory does require adaptations from the original PDL, due to the difference between expanding on the action space or policy space, but overall the adaptation is pretty straightforward. 

The main issue I have is that the connection between theory and practice is very weak. The efficiency of any offline RL algorithms over baseline methods, should be due to its statistical efficiency, i.e., given finite amount of offline RL data, can the algorithm achieve good performance. This setup underlies the challenge of offline RL where no new data can be collected. However, Thm 3.1 deals with the "expectation" behavior of the algorithm, i.e., how the algorithm behaves when there is infinite amount of data such that finite sample bounds turn into expected improvements. This is not very insightful because by studying the expected behavior of the algorithm, we bypass much of the challenge that underlies offline RL itself.

There should arguably, be comparison to sample complexity bound mentioned in [4]. If we obtain the finite sample bound version of Thm 3.1, does it improve over the bound derived in CQL? How do they compare? They have different dependencies on diff hyper-parameters on the env, what do they mean in practice?

[4] Conservative Q-learning for offline RL, Kumar et al, 2020

=== **Experiment** ===

The experiments show that the new algo obtains benefits in certain envs over the SOTA baselines, but understandably the improvements are not uniform and not always. This is expected because different algorithms behave differently under the same env, and we may not expect the same algo to outperform all other SOTA at the same time. I think what's valuable is to elucidate why the new algo performs better than the previous SOTA, e.g., IQL in M and ME dataset, but not as much in the MR dataset (in Table 1). Such a comparison would be valuable to understand relative strengths of diff methods.

As a side note, I think adding a "total" row in the table is not very informative, because aggregating statistics across all games does not help understand the performance much in this specific case.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
details_of_ethics_concerns: No concerns.
recommendation: 5: marginally below the acceptance threshold