From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: This paper establishes a connection between one-step regularization method and the multi-step critic regularization in off-line RL problems. The connection is very natural if the regularizations of both one-step and multi-step methods encourage the policy to stay close to the behavioral policy. This paper shows this connection by constructing a specific one-step algorithm and a specific multi-step method with a very specific way of making POPULATION updates (not practical stochastic updates). 

According to the reviewers, the main issue of the paper is that the result of the paper is too limited.  Indeed, the equivalence only happens for specific regularizations and specific population update (not practical stochastic update) rules that are not commonly used in current off-line RL problems. For general one-step and multi-step algorithms, the equivalence no longer exists and what the paper provides is no more than some raw intuitions.  
justification_for_why_not_higher_score: The result is too limited. The equivalence does not hold for the practically used algorithms and also does not hold for the practically used stochastic updates. 
justification_for_why_not_lower_score: N/A
summary_of_AC-reviewer_meeting: N/A

From: ICLR.cc/2023/Conference/Paper4281/Reviewer_3nPe
title: Updating my score
comment: Dear readers,

I have read the other reviews and the discussion with the area chair. Unfortunately, even after fixing the impreciseness in the Lemma, I believe the problems pointed out during the discussion with the other reviewers and the area chair changed my opinion on the contributions and overall technical soundness of this paper.

Therefore, I tend to recommend rejection.

From: ICLR.cc/2023/Conference/Paper4281/Reviewer_hMZj
title: After Response
comment: Thank the authors for their detailed response.

While the response addressed my concerns, I noticed that reviewer PLRB pointed out a problem in the paper that may affect its correctness. After reading the discussion and also the meta reviewer's comment I agree with them that Lemma 4.1 is not completely solid. Therefore I have lowered my score.

From: ICLR.cc/2023/Conference/Paper4281/Authors
title: Response to pLRb
comment: Dear reviewer,

We hope that you've had a chance to read our response. **Does this clarify any of the concerns?** If not, please let us know and we can further revise the paper or clarify any concerns.

Kind regards,

The authors.

From: ICLR.cc/2023/Conference/Paper4281/Authors
title: Response to hMZj
comment: Dear reviewer,

We hope that you've had a chance to read our response. **Does this clarify any of the concerns?** If not, please let us know and we can further revise the paper or clarify any concerns.

Kind regards,

The authors.

From: ICLR.cc/2023/Conference/Paper4281/Authors
title: Author response
comment: Dear AC,

We believe that the proposed approach is guaranteed to converge to the optimal solution (in tabular settings), but our original statement of Lemma 4.1 was ambiguous: we have revised the Lemma so that it now refers to a a _sequence_ of incremental updates, exactly analogous to how Q-learning corresponds to a sequence of incremental updates.  **Does this clarification, together with the discussion below, address the AC's concerns regarding Lemma 4.1?**

In the same way that the Q-learning updates can be written as sequence of optimization problems (minimizing the Bellman error), the critic updates can be written as a _sequence_ of optimization problems (minimizing Eq. 4). The critic is not learned by minimizing a single fixed objective, but rather is solving a minimizing a different cross-entropy loss at each update.

We might emphasize that Lemma 4.1 is a small extension of the standard Q-learning convergence proof [Agarwal 2019]. Namely, the proposed ''classifier actor critic'' is equivalent to performing Q-learning on transformed Q-values ($\frac{Q}{Q+1}$). Because Q-learning provably converges (in tabular settings [Agarwal 2019]), it makes sense that Q-learning with a change of variables would also converge. We have added a ''proof sketch'' before the proof of Lemma 4.1 to clarify the proof strategy.


We ran a simple experiment [code below] on the proposed 1-arm bandit, and find that the algorithm does converge to the correct true Q-function ($Q = 10$), within numerical precision.

---------------------------
```
import jax
import jax.numpy as jnp
import tqdm

def loss(Q, y):
  loss = -1.0 * y * jnp.log(Q / (Q + 1)) - jnp.log(1 / (Q + 1))
  return loss[0]

r = 1.0
gamma = 0.9
grad_fn = jax.grad(loss)
@jax.jit
def step(Q):
  y = r + gamma * Q
  grad = grad_fn(Q, y)
  return Q - grad

Q = jnp.array([1.0])
for t in tqdm.trange(10_000):
  Q = step(Q)
  if t % 1000 == 0:
    print('(t = %d) Q = %.3f' % (t, Q))
```

From: ICLR.cc/2023/Conference/Paper4281/Authors
title: Author response 2/2
comment: 
**2. Other questions/suggestions**

> Introduction … starting with a clear definition of what “you” mean by regularization

We have revised the first paragraph of the introduction to provide a definition of what we mean by regularization.

> Clearly convey the contributions in short and informative statements.

As suggested by the reviewer, we have revised the contributions paragraph of the introduction to state the contributions in short, informative statements.

> Second paragraph of related work: Not sure how imitation learning (IL) falls into these categories

This paragraph is about outcome-conditioned IL, which does implicitly perform policy improvement [2]. We have revised this paragraph to note that IL (without any outcome conditioning) is different and not the focus of this paper.

> Your Figure 1 seems to have not been referred to at all in the text.

We have added a reference in the text.

>  If that’s the case (which seems to be from the following part), make it explicit that the MDP is not episodic and there is no termination.

We have revised the first paragraph of the notation section to explicitly state that episodes have infinite length, and that there are no terminal states.

> your definition of reward is indeed expected reward (over next state)

We have added a footnote to clarify what happens if the reward function also depends on the next state, $r(s, a, s')$. Please let us know if we have misunderstood the suggestion here.

> your definition of Qπ is undiscounted

This was a typo, which has been fixed.

> "we would apply a stop-gradient operator” -> what do you mean? So, you do not try to solve second line of equation (1)?

We mean that we are applying the standard TD backups, like the vast majority of Q-learning papers (e.g., DQN, TD3, DDPG). We have revised the notation and text to clarify this.

> this update is SARSA, not Q-learning

We have fixed this typo.

>  As an alternative to the cross-entropy, one may directly learn the transformed Q values.

Thanks for sharing this. We will investigate whether this allows us to extend our results to other loss functions, besides the cross entropy loss.

---------------------
[1] Agarwal et al, "Reinforcement learning: Theory and algorithms", 2019.

[2] Eysenbach et al, "Imitating Past Successes can be Very Suboptimal", 2022.

From: ICLR.cc/2023/Conference/Paper4281/Authors
title: Author response 1/2
comment: Dear Reviewer,

We thank the reviewer for the thoughtful review and detailed suggestions for improving the paper. It seems like the reviewer's primary concern is about the correctness of Lemma 4.1, which claims that the algorithm converges: we believe that the whole algorithm does converge, but the wording of Lemma 4.1 was ambiguous, and we have revised this. A secondary concern seems to be the clarity of the writing: we have incorporated the all reviewer's suggestions throughout the paper, and believe that these changes have significantly improved the clarity. **Do these revisions and the answers below address all the reviewer's concerns?** We welcome additional suggestions for improving the clarity of the analysis or writing, and look forward to continuing the discussion.

**1. Questions about the theoretical results**

> Proof of Lemma 4.1 … You need to show that under incremental change, the whole algorithm converges.

We believe that the whole algorithm does converge, but the wording of Lemma 4.1 was ambiguous. We have revised the statement of the Lemma to clarify that we are interested in what this _sequence_ of Q functions converges to. Precisely, the critic is learned via a _sequence_ of incremental updates, exactly analogous to how Q-learning corresponds to a sequence of incremental updates. In the same way that the Q-learning updates can be written as sequence of optimization problems (minimizing the Bellman error), the critic updates can be written as a _sequence_ of optimization problems (minimizing Eq. 4). We have revised the formal theoretical claims and discussion to clarify this point. 

We might emphasize that Lemma 4.1 is a small extension of the standard Q-learning convergence proof [1]. Namely, the proposed ''classifier actor critic'' is equivalent to performing Q-learning on transformed Q-values ($\frac{Q}{Q+1}$). Because Q-learning provably converges (in tabular settings [1]), it makes sense that Q-learning with a change of variables would also converge. We have added a ``proof sketch'' before the proof of Lemma 4.1 to clarify the proof strategy.

Also, in case it wasn't clear, this proof used the following reasoning:
$$\frac{Q}{Q+1} = \frac{y}{y+1} \iff Qy + Q = yQ + y \iff Q = y$$

> what do you mean by “in tabular settings”?

We mean that the TD target can be computed exactly, without sampling (e.g., with a known model). We have revised the paragraph after Eq. 4 to clarify this. We have also clarified this in the statement of Lemma 4.1. We emphasize that our experimental results show that our analysis still makes accurate predictions even in settings where these assumptions are violated.

> The proof of Lemma 4.1 does not look right since you have a nonlinear transformation which does not commute with the expectation and Jensen’s inequality applies.

The confusion here might be in that we were using ''tabular'' to refer to the setting where the TD targets can be computed exactly (e.g., with a known model). Under this assumption, we believe that the proof of Lemma 4.1 is correct. We have revised the statement of Lemma 4.1 to clarify these assumptions. 

When this assumption does not hold, we agree that the nonlinear transformation does not commute with the expectation. We have already included a discussion of how sampling introduces Jensen's inequality in Appendix A.1. Our experiments demonstrate that our theoretical results hold approximately even in settings where the TD update cannot be computed exactly (see, e.g., Fig. 6).



> Lemma 4.1 and 4.2 -> You need to clarify the assumptions under which these results hold.

We have revised the first sentence of these Lemma statements to clarify that we assume that the states/actions are discrete and finite, that the rewards are positive, and that the TD targets can be computed exactly, without sampling. We welcome additional suggestions for clarifying these theoretical results.

From: ICLR.cc/2023/Conference/Paper4281/Authors
title: Author response
comment: Dear Reviewer,

We thank the reviewer for the detailed review, and for the suggestions for improving the paper. It seems like the main concern is about what the theoretical results actually show, and a second concern is about the writing clarity. Below, we discuss these two concerns in detail. We have revised the paper in many places to incorporate the reviewer's suggestions, and added a new experiment to address the question about the regularization coefficient. **Have these responses sufficiently addressed the concerns? Are there additional questions or suggestions for improving the paper?** We look forward to continuing the discussion!

> since the main results ignore the difference in optimization methods, these algorithms actually have big difference

As suggested by the reviewer, we have revised the discussion of Theorem 4.3 to emphasize that our theoretical results only show the connection between _objective functions_, and to emphasize that different offline RL methods use different _procedures_ to optimize these objective functions. Thus, they may obtain different estimates for $Q^\beta(s, a)$, violating our assumption in Theorem 4.3 that the Q-values are estimated exactly. Nonetheless, we believe that comparing the overall policy-learning objectives of different offline RL algorithms is useful because it reveals connections about offline RL methods. For examples, our analysis suggests (and experiments confirm) that one-step RL methods should underestimate Q-values.

> for eq.(3) , although the range of the normalized Q function and y function lies in (0,1), their integration is not one and hence not probability function. So what's the justification to compare them with cross entropy?

Eq. 3 does represent the cross entropy distribution using a proper probability: it uses a Bernoulli distribution with parameter $\frac{Q}{Q+1}$ (Note that $\frac{Q}{Q+1}+ \frac{1}{Q+1} = 1$). Our original motivation for using the cross-entropy loss was that we were looking at goal-conditioned RL problems, where one of the state-of-the-art methods uses this sort of cross-entropy loss (see Appendix C). We have added additional discussion and motivation for the cross entropy loss in Appendix A.3.

> About the coefficient λ, the theoretical analysis suggests that λ=1 can lead to the same convergence for both methods, while the experiments show both methods have the most similar results (Fig.5) if λ=10. Maybe more analysis on various coefficients should be carried out.

As suggested by the reviewer, we ran an additional experiment to analyze various coefficients in a setting where our assumptions hold exactly. Precisely, we took the task from Fig 2 (left) and measured the similarity between one-step RL and critic-regularized classifier actor critic, for varying values of the critic regularization parameter. We measured the similarity using the $R^2$ score. A new figure in the Appendix, Figure 8, confirms our theoretical predictions: these methods are most similar for $\lambda = 1$.

Figure 5 shows that practical offline RL methods, which violate the assumptions of our theoretical analysis, nonetheless behave similarly. Note that it is not surprising that they are most similar for a slightly different value of $\lambda$, because these methods violate the assumptions of our analysis, .

> About the lower bounds on Q-values(Figure.6), the original paper proves that the complete CQL objective can lead to the underestimation on V value while not Q value, so I'm not sure it's appropriate to compare Q values.

These Q-values are equivalent to the value function of the current policy: $V^\pi(s) = E_{\pi(\cdot \mid s)}[Q^\pi(s, a)]$. We have revised the caption of Figure 6 to clarify this.

> Some of the experimental results require more detailed explanation

What details are missing from the experimental results? We would gladly include any missing details to improve the paper.

> It's better to provide the performance of both methods for more sufficient comparison.

Could the reviewer kindly clarify this suggestion? We are happy to run additional experiments to provide a more thorough comparison.

From: ICLR.cc/2023/Conference/Paper4281/Authors
title: Author response
comment: Dear reviewer,

Thank you for the detailed review! As suggested, we have revised the paper to explain in more detail why our analysis requires the cross entropy loss (new Appendix A.3). **Does the reviewer have any additional concerns or questions about the paper?**

From: ICLR.cc/2023/Conference/Paper4281/Authors
title: Author response
comment: Dear reviewer,

We thank the reviewer for the detailed review, and for the suggestions for improving the paper. Below, we answer the reviewer's questions about accounting for errors, sample efficiency, and policy gradients. We have also revised the paper to mention these answers. **Do the answers to the questions below address all the reviewer's concerns?**

>  For Theorem 4.3, since in practice there are stochastic errors and function approximation errors, showing that the ideal update equations are equivalent is not entirely satisfactory. It will be interesting to see how these errors propagate in these two types of algorithms.

Generally speaking, accounting for the propagation of errors in TD learning with function approximation remains an open problem, largely orthogonal to the contributions of this paper. Indeed, many [1, 2, 3] (though not all [4]) prior methods do not account for stochastic errors. Nonetheless, we can provide some preliminary analysis. First, Appendix A.1 shows that using stochastic updates for the TD targets results in underestimating the Q-values, because of the nonlinear transformation of the Q-values. While such underestimation means that classifier actor-critic will not learn the true Q-values in non-tabular settings, it may be beneficial to combat value overestimation in the offline setting. Second, if we assume that the TD updates are deterministic (e.g., in the tabular setting with a known model), then the updates for classifier actor-critic are identical to those of Q-learning (Lemma 4.1). Thus, it immediately inherits any theoretical results regarding the propagation of errors for Q-learning. We have added this discussion to Appendix A.2.

> Since multi-step critic regularization and one-step RL are equivalent, but multi-step critic regularization seems to be less sample efficient to implement, does this suggest that one should use one-step RL in practice?

Our analysis shows that the two methods have the same fixed points, but does not say how many transitions or gradient updates are required to reach those fixed points. We have added this clarification to Appendix A.2. We agree that one-step RL is simpler than multi-step critic regularization, so practitioners might prefer to use it for that reason.

> In both cases, the authors focus on policy iteration type of policy update. What about using policy gradient or natural policy gradient type of policy update?

Our analysis fundamentally requires using TD learning: the key step is that doing TD backups using one policy is equivalent to doing (modified) TD backups with a different policy. However, the actor updates for both methods could be implemented using a policy gradient or natural gradient, rather than a straight-through gradient estimator. Indeed, much of the work on one-step RL methods [5, 6] uses an actor update that resembles a policy gradient or natural policy gradient (see, e.g., the reverse KL version of [7]). We have added this discussion to Appendix A.5.

---------------------
[1] An et al, 2021. Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble

[2] Fujimoto et al, 2021. A Minimalist Approach to Offline Reinforcement Learning

[3] Nachum et al, 2019. AlgaeDICE: Policy Gradient from Arbitrary Experience

[4] Vieillard et al, 2020. Leverage the Average: an Analysis of KL Regularization in RL

[5] Peng et al, 2019. Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning

[6] Siegel et al, 2020. Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning

[7] Brandfonbrener et al, 2021. Offline RL without off-policy evaluation.

From: ICLR.cc/2023/Conference/Paper4281/Area_Chair_drRi
title: About Lemma 4.1
comment: I think your comment about Lemma 4.1 is correct. Indeed, this lemma is completely wrong, and the proof logic is weird. The lemma itself is talking about the relationships of OPTIMAL solution. However, the proposed approach does not converge to the optimal solution. 

In the proof, the author seems to fix the Q in y(Q) and minimize over the rest part of Q in each iteration. This is by no means a correct algorithm for finding minimum. A simple counterexample is $\min f(y(Q),Q) = (y(Q)-1)^2 + (Q-2)^2$. In this example if one fix the Q in y(Q) and update the other Q, he will get stuck at Q=2. However, the optimal solution will not equal to 2 for general Y(Q), e.g. y(Q)=Q.

Even if we restrict ourselves to the MDP objective. The counterexample is still easy to obtain. Let us consider the simplest 1-arm bandit with deterministic reward r=1. Let gamma = 0.9. Then the only possible policy is pi = 1, the only possible data distribtion is p=1, the Q function is 1+0.9+0.9^2+... = 10.  Now, with y = r + gamma*Q = 1 + 0.9Q, the proposed objective function becomes L_critic(Q,pi) = -(1+0.9Q)log(Q/(Q+1)) - log(1/(Q+1)). In this case, argmin L_critic(Q,pi) < 1, one can visualize the plot easily. This is not the true Q function.

From: ICLR.cc/2023/Conference/Paper4281/Reviewer_hMZj
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper makes a connection between regularizing the critic and regularizing the actor in the actor-critic framework. Specifically, the authors show that these two are equivalent with a specific choice of the regularizing parameter.  Extensive numerical simulations are provided to justify the theoretical findings. 
strength_and_weaknesses: This paper is easy to follow and the result seems interesting. 

(1) For Theorem 4.3, since in practice there are stochastic errors and function approximation errors, showing that the ideal update equations are equivalent is not entirely satisfactory. It will be interesting to see how these errors propagate in these two types of algorithms.

(2) Since multi-step critic regularization and one-step RL are equivalent, but multi-step critic regularization seems to be less sample efficient to implement, does this suggest that one should use one-step RL in practice?

(3) In both cases, the authors focus on policy iteration type of policy update. What about using policy gradient or natural policy gradient type of policy update?



clarity,_quality,_novelty_and_reproducibility: The paper is well written. While the authors do not propose new algorithms, the insight from analyzing existing algorithms seems interesting.
summary_of_the_review: Overall I think this paper provides an interesting viewpoint on the connection between critic regularization and actor regularization, and the numerical simulations are convincing. 
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper4281/Reviewer_3nPe
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper presents a theoretical and empirical analysis with the main goal showing the similarity between two main categories of algorithms, one-step methods and critic regularization-based methods. Specifically, the paper shows that, under some conditions, the two types of methods are theoretically equivalent; it also shows that, in relevant practical cases, they also behave similarly in practice.
strength_and_weaknesses: Strengths:
- The writing is very clear and the presentation of the content is easy to follow;
- The theoretical results are compact and clear, and also well explained in the context of the full paper;
- The experimental analysis is remarkably clear as well (I really like visualization in Figure 4 and Figure 5);
- The extended analysis for other settings (e.g., goal-conditioned RL) reported in Appendix adds interesting material to the paper.

Weaknesses:
- The theoretical analysis a bit limited by the "classifier actor-critic" setting; it would be nice to explain in more detail why an analysis in the regression case is especially difficult to help the reader understand this choice.
clarity,_quality,_novelty_and_reproducibility: The clarity of the paper is exceptional.

The quality of the research is good.

To the best of my knowledge, this equivalence was never explicitly investigated, and the work thus looks novel.
summary_of_the_review: In summary, I found this to be a good paper that can be of general interest for people working with offline RL algorithms of various kinds, and I thus recommend acceptance.

-----
_After rebuttal phase_: after reading the discussion with the area chair and the other reviewers, I now share their concerns on the significance and technical soundness of the results and I have thus lowered my score.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper4281/Reviewer_pLRb
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper aims to draw a connection between the 'one-step' offline RL algorithms and the critic regularization-based ones. With some modification in actor or critic objective, the authors prove the same convergence results for the both methods in tabular settings. And the numerical simulations and empirical results on offline RL benchmark show the close connection between the both algorithms.
strength_and_weaknesses: Strength:

1. this paper tries to reveal the relationship between 'one-step'-based and critic regularization-based offline RL methods, and provide a new perspective on the understanding of various offline RL algorithms.


Weaknesses:
1. Though this paper proves that both the offline methods can have the same convergence results, some preconditions should be satisfied and some key changes seems strange.
2. The main results, i.e., thm 4.3 says that "the critic regularization results in the same policy as one-step RL" , which is doubtful. Actually, this theorem only shows that their objective is equivalent, not necessarily the algorithm. We know that different algorithms will generally results in quite different policy, which is essentially the key motivation behind one-step RL: all the value-based methods are optimizing the Bellman equation in some sense, but multiple step or iterative type algorithms are not neccearily better than the single step ones.  In particular, for critic regularization algorithm such as CQL, it is their optimizing method that defines what the algorithm really is - in this case it is just an iterative type value-based algorithm, and hence may suffer from the over use of the estimated behavior Q function, no matter its objective is equivalent to that of one-step RL or not.  Due to the above reason, CQL may not results in the same policy as one-step RL.

3. Some of the experimental results require more detailed explanation, (e.g. numerical simulation results in Figure.2 make readers confusing). And there is a minor disagreement on the coefficient choice for both theoretical and empiciral results.
3. It's better to provide the performance of both methods for more sufficient comparison.

4. for eq.(3) , although the range of the normalized Q function and y function lies in (0,1), their integration is not one and hence not probability function. So what's the justification to compare them with cross entropy?

5. About the coefficient $\lambda$, the theoretical analysis suggests that $\lambda=1$ can lead to the same convergence for both methods, while the experiments show both methods have the most similar results (Fig.5) if $\lambda=10$. Maybe more analysis on various coefficients should be carried out.

6. About the lower bounds on Q-values(Figure.6), the original paper proves that the complete CQL objective can lead to the underestimation on V value while not Q value, so I'm not sure it's appropriate to compare Q values.

clarity,_quality,_novelty_and_reproducibility: fair.
summary_of_the_review: This paper shows that the objectives of one step RL and CQL are equivalent. However, since the main results ignore the difference in optimization methods,  these algorithms actually have big difference. I think that the paper should mention this.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper4281/Reviewer_TugW
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper presents a theoretical analysis, hoping to show that one-step RL is equivalent to a certain type of critic regularization. 
strength_and_weaknesses: Strength:

- The general idea of discovering equivalence between methods, which look like arbitrary choices, is quite interesting (and can be useful as it clarifies false expectations for performance improvement if different variations are indeed equivalent).

Weaknesses:

- Poor presentation: it is not an easy paper to follow, especially definitions and assumptions are often unclear (aside from notational and conceptual mistakes here and there). 

- Some of the results are in question, particularly, switching regression to classification has been done through a nonlinear transformation. It is unclear if this transformation still guarantees convergence to the desired values.
clarity,_quality,_novelty_and_reproducibility: Clarity: poor

Quality: average

Novelty: average

Reproducibility: good -> Authors stated to release the code (beginning of section 5), also some implementation details are provided. 
summary_of_the_review: - The introduction is hard to follow. Particularly, your definition of regularization is different from what a normal RL reader may expect. Most of the methods cited in the introduction to combat over-estimation of offline RL are not normally known as *regularization* techniques. I would strongly suggest starting with a clear definition of what “you” mean by regularization before even start citing other papers and compare various method.

- Last paragraph of the introduction is very unclear, likewise the contributions part. Clearly convey the contributions in short and informative statements. 

- Second paragraph of related work: Not sure how imitation learning (IL) falls into these categories. IL normally tries to generalize the seen actions at seen states, with no regard to maximizing expected return. That is, IL tries to clone the behavioural policy. In contrast, one step methods still have a policy improvement step, meaning that their resultant policy is likely to be better than the behavioural policy and it does not necessarily clone that. 

- Your Figure 1 seems to have not been referred to at all in the text. 

- Section 3.1: “… we assume rewards are positive, adding a positive constant to all rewards …” this is only correct if there is no terminal state, otherwise the values of terminal states should also be shift from (by definition being) zero. If that’s the case (which seems to be from the following part), make it explicit that the MDP is not episodic and there is no termination. 

- Section 3.1: your definition of reward is indeed expected reward (over next state). Paying attention to small details help the reader to follow your main points better. 

- Section 3.1: your definition of $Q^\pi$ is undiscounted, yet your reward is strictly positive. The summation simply diverges. 

- Page 4, first paragraph: “… TD targets $y(s, a)$ are not considered learnable (i.e., we would apply a stop-gradient operator)” -> what do you mean? So, you do not try to solve second line of equation (1)?

- Page 4, last line -> this update is SARSA, not Q-learning. 

- Paragraph after equation 4: what do you mean by “in tabular settings”? Do you mean the standard SARSA? Same issue in Lemma 4.1. The proof of Lemma 4.1 does not look right since you have a nonlinear transformation which does not commute with the expectation and Jensen’s inequality applies. 

- Again, proof of Lemma 4.1: you said the update can be written as $\frac{Q}{Q+1}\leftarrow \frac{y}{1+y}$. This is only the loss function (and not the update) which then *incrementally* changes the value of $Q$. You need to show that under incremental change, the whole algorithm converges. (See for example textbooks on stochastic approximation, e.g., chapter 4 of Neuro-Dynamic Programming, Bertsekas and Tsitsiklis, 1996)

- Page 5, paragraph after equation 5 -> this also rings a bell in my mind. As an alternative to the cross-entropy, one may directly learn the transformed Q values. See this reference: https://arxiv.org/abs/2203.07171 

- Lemma 4.1 and 4.2 -> You need to clarify the assumptions under which these results hold. 

correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
details_of_ethics_concerns: N/A
recommendation: 3: reject, not good enough