From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Accept: notable-top-5%
metareview:_summary,_strengths_and_weaknesses: (a) Claims: The paper introduces an algorithm that learns a confidence-conditioned lower bound for $Q$ values from an offline dataset.  This allows the algorithm to adaptively change its degree of conservatism online/at test time based, rather than having to choose a conservatism hyperparameter in advance.  One reviewer described this as allowing for simultaneously learning with many confidence levels at once.

(b) Strengths: The main strength of the paper is the novelty of learning an approximation that allows for online tuning based on how different the target distribution of trajectories generated by the algorithm appears to be from the distribution of trajectories in the offline dataset.  The empirical evaluations give evidence that this approach is fruitful.

(c) Weaknesses: Most reviewers noted some clarity issues, that seem to have largely been fixed in an intermediate draft.  One reviewer had serious concerns about the empirical evaluation, especially the dynamics of the gridworld environment.
summary_of_AC-reviewer_meeting: Reviewer DkfW, Reviewer gE4Y, Reviewer Fh1m, and I met on November 30.  Reviewer pdN5 did not respond to requests to schedule a time.

Reviewer Fh1m advocated strongly for the paper.  He considers  the ability to adaptively change the confidence level used by the algorithm at test time to be very novel.  He notes that without this ability, the performance of the algorithm is very close to that of the baselines.

Reviewer DkfW's original rating for the paper was marginal, but he found Reviewer Fh1m's points compelling.

All reviewers agreed that the main weakness of the paper was the experimental design and discussion.

Reviewer gE4Y in particular considered the empirical evaluation to be fatally flawed.  His view was that it was inconsistent with the theoretical discussion in the paper to a degree that rendered it uninformative.  During the discussion he was open to arguments about novelty, but wanted to read the paper again before he could make a call on whether that compensated for the experimental evaluation; upon consideration, he later decided that he could not support a positive recommendation for the paper.

Reviewer DkfW described some related work that is concurrently under submission at ICLR-2023 that uses a similar idea, but all reviewers agreed that novelty should only be evaluated with respect to published work and/or publicly-available preprints.

In the end, I have decided to recommend acceptance.  I weighed the novelty of the contribution more heavily than the reviewers' concerns about the empirical evaluation.
justification_for_why_not_higher_score: n/a
justification_for_why_not_lower_score: I agree with the reviewers that this paper contributes novel ideas in the important area of offline reinforcement learning.  Reviewer gE4Y's concerns about the empirical setup are well-taken, but I don't consider them to be a fatal flaw.
note_from_PC: if the above contains the word "oral" or "spotlight" please see: "oral" presentation means -> notable-top-5% and "spotlight" means -> notable-top-25%. As stated in our emails, we are disassociating presentation type from AC recommendations

From: ICLR.cc/2023/Conference/Paper5450/Authors
title: Remaining concerns
comment: Dear Reviewer gE4Y,

We saw that you updated your score to a 5. In the remaining time we would like to address the remaining concerns that you have that prevent you from accepting the paper. We would appreciate it if you could engage with us on the remaining concerns, as we want to address them.

Thanks!

From: ICLR.cc/2023/Conference/Paper5450/Authors
title: Let us know if you have further questions!
comment: Thank you for considering our response, and we look forward to your updating the score. Let us know if you have any further clarifications!

From: ICLR.cc/2023/Conference/Paper5450/Reviewer_pdN5
title: Response
comment: Thanks for the clarification. It makes more sense to me now and I am raising my score.

From: ICLR.cc/2023/Conference/Paper5450/Reviewer_gE4Y
title: Thank you for your response
comment: Thank you for the update and I'll adjust the score accordingly.

From: ICLR.cc/2023/Conference/Paper5450/Authors
title: Response to Reviewer pdN5
comment: Thank you for your review. You raised some good questions that we answer below. We have also updated our paper to clarify the points that you raised. Let us know if our response clarifies the unclear points for you!

**Q1:**
It is true that $Q^*(s, a)$ is deterministic, but the $Q(s, a, \delta)$ that we estimate are not due to randomness in the dataset. This means that $Q(s, a, \delta)$ can only probabilistically lower-bound $Q^*(s, a)$ due to potential tail events in dataset sampling. We have clarified this in our paper.

___

**Q2:** 
We use (5) in our empirical comparisons in Section 7.2 as it performs much better in non-tabular environments. We only use (9) (additionally learning upper-bounds) in Section 7.3 when performing experiments with online fine-tuning. We find that (9) does not provide a significant performance benefit in pure offline learning. We briefly mentioned this in Section 7.2, 7.3 respectively, but moved these details to the beginning of Section 7 so that they are more clear. 

It is true that sampling $\delta_1, \delta_2$ introduces bias in the objective given by (4). Theoretically, this does not affect that our learned $Q(s, a, \delta)$ are high-probability lower-bounds, however, but only affects the tightness of these lower-bounds. This is because the lower-bound holds with high-probability for any $\delta_1, \delta_2$. Empirically, we also do not believe this to be an issue as approximation due to finite sampling appears in many algorithms; for example, algorithms as QR-DQN and IQN also require sampling $\delta$’s uniformly (though their $\delta$ has a different meaning than ours). 

___

**Q3:**
Regarding the inner maximization over  $\delta_1, \delta_2$ in (4), the benefit is in potentially learning a tighter lower-bound. This is because traditional pessimistic algorithms could be interpreted as learning a lower-bound where $\delta_1 = \delta_2 = \delta$ in (4). This means that the policies learned by CCVL can be less conservative than competing pessimistic algorithms.

Regarding potential theoretical benefits of our proposed CCVL over CQL, we believe we can show that CCVL can improve upon the behavior policy more, due to the additional weighting by state visitation in (5) over (2). We added this discussion to our paper, but provide a summary below.  We know that the policy improvement of the return of the learned policy $J(\hat\pi)$ over the behavior policy $J(\pi_\beta)$ is bounded as

$
J(\hat{\pi}) - J(\pi_\beta) 
\leq  \frac{1}{(1- \gamma)^2} \mathbb{E}_{s \sim d^{\hat\pi}(s)}[D(\hat\pi, \pi_\beta)(s)]
$

where $D(\hat\pi, \pi_\beta)(s)$ scales with the difference between the learned and behavior policies at state s. Under CQL, the learned policy solves (Theorem 3.5)

$
\hat\pi= \arg\max_\pi \hat{J}(\pi) - \frac{\alpha}{1 - \gamma} \mathbb{E}_{s \sim d^{\pi}(s)}[D(\pi, \pi_\beta)(s)]
$

Under CCVL, we can show that it is (due to the state weighting in (5))

$
\hat\pi = \arg\max_\pi \hat{J}(\pi) - \frac{\alpha}{1 - \gamma} \mathbb{E}_{s \sim d^{\pi}(s)}[n(s)^{-1/2} D(\pi, \pi_\beta)(s)]
$

In CCVL, due to the state-visitations weighting, the learned policy $\hat\pi$ can diverge more from the behavior policy $\pi_\beta$ under frequently visited states in the dataset. Formally, this means that if everything else was equal (dataset, hyperparameter settings, etc.), then CCVL can learn a policy that improves more upon the behavior policy than CQL.

___


**Q4:**
The optimal trajectory does in fact appear in the dataset, but with much lower density than the suboptimal one. We agree that it is hard to see that trajectory in Figure 1 due to the color scale that we use. We have modified Figure 1 so that both trajectories are now visible.

From: ICLR.cc/2023/Conference/Paper5450/Authors
title: Response to Reviewer Fh1m
comment: Thank you for your review! We are happy that you view our algorithmic novelties positively, and agree with many of the improvements that you suggested, and address how we will resolve them below. 

**Additional ablation study and more description of methods in existing ablation studies**

We found that explicit bonus-based objectives such as (4) that involve subtracting a bonus from the target Bellman update do not work well with neural networks. The issue is not with the statement of the objective, but rather that optimizing such an objective using neural networks is often difficult. Some existing works have gotten bonuses to work (namely [1]), but use sophisticated estimators such as CVAEs to compute the bonuses. This is why we propose and use (5) in our experiments as an alternative to using (4). 

We agree with your suggestion of adding in more details about the baselines we compared against. We added a section in the Appendix that does this. 

[1] https://arxiv.org/pdf/2106.06431.pdf

**The proposed algorithm requires additional sampling or computation**

You are correct that our algorithm incurs a computational cost due to sampling multiple $\delta$. That said, empirically, we find that a small number of samples of $\delta$ is sufficient for good performance (we use 8 in our experiments) and this does not incur significant computational costs. Many other existing algorithms also rely on sampling i.e. QR-DQN, IQN, so we do not view this additional cost as a large problem. Per your suggestion, we will report runtimes in the appendix of our algorithm against baselines to show that the additional cost is not significant.

**Clarification on n(s) computation**

You are correct. The reason why we simply go with $\phi(s)^T\phi(s)$ is just because we have that 
$\phi(s)^T\Lambda\phi(s) = (\Lambda^{½}\phi(s))^T (\Lambda^{½}\phi(s)) \leq ||\Lambda^{½}||_2 \phi(s)^T\phi(s)$.
Since our hyper-parameter $\alpha$ already scales the objective, we found it natural to drop the additional scaling by $||\Lambda^{½}||_2$. However, we understand that this may cause confusion, and have revised our work to use the exact computation of n(s).

From: ICLR.cc/2023/Conference/Paper5450/Authors
title: Response to Reviewer DkfW
comment: Thank you for your review. You raised some important points in our paper that require clarification or improvement. We will address them below, and have updated our paper to reflect any changes that we proposed. Let us know if this improves your perspective of our work!

**Clarification on proofs**

For your first question, when $\delta_1, \delta_2 \leq \delta’$, that means that $\delta_1, \delta_2$ are potential solutions to the inner maximization for solving the target value of $Q(s, a, \delta’)$ in (4). Because of the maximization, this means that $Q(s ,a, \delta’)$ will be at least as large as $Q(s, a, \delta)$.
 
For your second question, the reduction from (4) to (1) is under a different choice of $\alpha$, so the $log(1/\delta)$ would be folded into the hyper-parameter. We clarified this in a revised version of our paper. 

**Hyperparameter $\alpha$**

In CQL, we believe that $\alpha$ is depends on two properties:

(1) First, in order to be a proper lower-bound, $\alpha$ must be chosen large enough depending on constants that capture the complexity of the environment (see Theorem 3.2 of [1]).

(2) In addition, $\alpha$ also needs to be scaled depending on the desired confidence-level.

In CCVL, hyper-parameter $\alpha$ still considers (1) but avoids (2), which we instead handle during online adaptation over confidence-level $\delta$. Because of this, we believe that $\alpha$ in our work is easier to tune than in comparable works on conservative offline RL algorithms. We add an experiment that shows this in a tabular environment (see next section).

[1] https://arxiv.org/pdf/2106.06431.pdf

**Simpler/tabular experiments**

We agree that this is an important consideration during empirical evaluation. To remedy this, we do a more proper evaluation of CCVL against baselines in the tabular Gridworld domain. We report the returns of CCVL against baselines for different choices of $\alpha$, to show that CCVL achieves more stable performance across choices of $\alpha$ than the comparable CQL algorithm.

From: ICLR.cc/2023/Conference/Paper5450/Authors
title: Response to Reviewer gE4Y (2/2)
comment: Finally, we would like to provide answers to the questions that you raised. 
1. Regarding a lower-bound over states rather than state-action pairs, we do not view this as a disadvantage. Rather, by only being a lower-bound in expectation over actions, our learned Q-values are tighter bounds than if they were a lower-bound across all actions. Further note that this is also the case with prior lower bounds in offline RL: CQL only lower bounds the average Q-value at every state, Bellman-consistent pessimism [4] lower bounds only the average Q-value at the initial state, yet these algorithms attain good performance guarantees.  

[4] https://arxiv.org/pdf/2106.06926.pdf

2. Thank you for the suggestion. We have included more details on the derivation of Theorem 6.2 in Appendix A.2.

From: ICLR.cc/2023/Conference/Paper5450/Authors
title: Response to Reviewer gE4Y (1/2)
comment: Thank you for your review. 

**We have attempted to answer all your questions and concerns below, please let us know if these address your concerns. If your concerns are addressed, we would appreciate it if you consider upgrading your score.** We are happy to answer any more questions that you might have. 

First, we would like to point out that our submission does not appear in NeurIPS 2022, but only in workshops. These workshops are non-archival, and hence there are no issues with plagiarism, dual submission, etc. 

Next, we would like to address the problems that you raised (in order):

1. **“Epistemic and aleatoric uncertainty”** 

We have added explanations of epistemic and aleatoric to Section 1 of our paper. We do not aim to claim that modeling epistemic uncertainty is better than aleatoric, but rather that it is important to consider both, particularly in offline RL. We have rephrased that paragraph to make this more clear. In Section 2, we compare our approach to various ensemble-based approaches, which aim to estimate uncertainty. We have also added a discussion comparing general uncertainty-based approaches to ours.  

2. **“Markovian policy”**

The reference you listed is indeed a very relevant work. The primary difference is that the algorithm proposed operates in the online RL setting. By prior methods only learning Markovian policies, we were referring to policies learned via pessimistic or conservative offline RL methods, where we cannot collect more data to perform exploration over different confidence levels. Certainly works in online RL have attempted to use adaptive policies, including the one you pointed out (where adaptivity is a result of utilizing the bandit distribution). We have added a discussion of this work in Section 2. 

3. **“Gridworld example”**

The reward and transition dynamics of our Gridworld environment are taken directly from prior works [1, 2] that were published in ICML and ICLR conferences, and appear in open-source codebases (https://github.com/justinjfu/diagnosing_qlearning). We chose this example as it was directly available, with starter code already containing Q-learning implementations for us to build upon. 

We feel that imposing a large negative penalty decreases the diversity of trajectories taken, which is detrimental to our illustrative example. However, we are open to changing the example if the environment feels very unintuitive or unnatural. 

You are correct in pointing out that the shorter path is not always better. In our environment, the shorter path does indeed result in higher return. To make this clear, we have included a comparison of the return of the two methods (see response to 4 for details). 

[1] https://arxiv.org/pdf/1902.10250.pdf

[2] https://arxiv.org/pdf/2204.05618.pdf

4. **“Missing study on weight alpha”**

We agree that $\alpha$ is still an important hyper-parameter. We have now added the following empirical analysis. In the Gridworld experiments, we evaluate both CCVL and CQL on different choices of $\alpha$. We show that CCVL achieves more stable performance as a function of $\alpha$ than CQL does, because it does not require committing to a single degree of conservatism. 

To provide more intuition, the reason why we claim that $\alpha$ is easier to tune in CCVL than CQL is that in CQL, choosing $\alpha$ doesn’t provide flexibility to choose a confidence level. This means that $\alpha$ corresponding to an incorrect level of confidence will cause CQL to fail, whereas CCVL will not due to its ability to adapt over confidence levels. Thus, $\alpha$ must satisfy lower fidelity requirements in CCVL and because of this, we believe that tuning $\alpha$ in CCVL is easier. 
 
[3] https://arxiv.org/pdf/2006.04779.pdf

5. **“Variance of CCVL is way larger”**

We understand that the performance improvement of CCVL is moderate. We believe that the high variance of CCVL is due to the additional noise due to online adaptation, which is not present in existing baselines. However, in addition to the improved performance, CCVL offers the advantage of not having to commit to a degree of conservatism during hyper-parameter tuning, which (as explained in the answer to 4) makes the algorithm easier to tune over standard conservative algorithms. In addition, we show that the improvements are more significant outside of the standard offline RL setting such as generalizing to new environments (Section 7.1), or online fine-tuning (Section 7.3). We believe that these empirical findings already demonstrate that CCVL has noticeable advantages over standard conservative algorithms.

From: ICLR.cc/2023/Conference/Paper5450/Reviewer_gE4Y
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: To solve the distribution shift problem in offline reinforcement learning of conservative methods, the paper let value functions learn on different degrees of conservatism (optimism/pessimism) instead of a fixed degree of conservatism. The paper proposes an algorithm, called CCVL, to achieve its goal in discrete-action environments.

The major contribution of the paper shows the shift from fixed conservatism to flexible conservatism improves performance on some discrete control domains.

strength_and_weaknesses: This paper has the idea to learn confidence-conditioned value fundction for adaptive policy optimization. The idea is the strength of the paper. However, there are flaws in the experimental design and experiment discussion, which make it impossible to thoroughly access the new idea.

Problems (ordered by appearance in the paper)
1. Two important terms, epistemic and aleatoric uncertainty, are unexplained in Section 2. It would be better for the paper to include the definition especially when the paper tries to emphasize its work on epistemic uncertainty.

    1.1 The last sentence of Section 2 gives a false impression that studying epistemic uncertainty is better than aleatoric uncertainty. Focusing purely on epistemic is no better or worse than focusing purely on aleatoric uncertainty. These are two different areas. The paper should address the difference of epistemic (parametric) vs. aleatoric (intrinsic) uncertainties.

    1.2 Could the paper also include a comparison between uncertainty-based work and confidence-conditioned work (this paper)?

2. Last sentence ("existing Markovian policies that can only act according to a fixed level of pessimism") of Section 4.2 is a false claim, which shows the paper does not cover a thorough literature review. One NeurIPS paper (Tactical Optimism and Pessimism for Deep Reinforcement Learning, https://arxiv.org/abs/2102.03765) uses the idea of dynamically apply the level of conservatism (optimism/pessimism).

3. The Gridworld example in Section 7.1 is unconvincing. The idea of the setup is similar to the Cliff Walking example in Rich Sutton's RL book, but the reward dynamics is weird: entering a lava state (a dangerous state) only receives 0 reward for the remaining trajectory and without further penalty.

    3.1 Should the paper use a large negative reward for dangerous states instead of 0?

    3.2 The paper only compares the optimal path between CQL and CCVL. However, a shorter path does not always mean a good path (like the SARSA vs. Q-learning performance in the Cliff Walking experiment). It would be more convincing if the paper includes a comparison plot of CQL and CCVL on cumulative reward.

4. Missing study on weight alpha. The paper changes from tuning conservatism "delta" to tuning weight "alpha" from my understanding. I believe the alpha is important in CCVL. As in Section 4.1, why tuning hyperparameter \alpha in CCVL is easier (or better in any aspects) than tuning an "opaque" hyperparameter (degree of conservatism)? Could the paper clearly explain or elaborate its statements?

    4.1 Also missing alpha in Table 3: Hyperparameters setup

5. As in Table 4 - 6 in Appendix B., the CCVL shows a level of minor to moderate improvement in the majority of the selected Atari games. However, the variance of CCVL is way larger than CQL in all games in Table 4 settings, and way larger than CQL in some games in Table 5-6 settings. Given the level of improvement of CCVL, I believe the introduction of large variance is unacceptable.

    5.1 The paper should include an explanation of the large variance.

Minors:
1. repeated article "a" in the first paragraph of Section 1
2. "delta" -> "\delta" in the first paragraph of Section 4 
3. mixed use of symbols of "Pr" vs. "\mathbb{P}" in Section 4
4. References order is confusing, making it hard to follow. It's neither alphabetically ordered nor first-appearance ordered. May I ask what is the reference setting?

Questions:
1. For Theorem 6.1, what is the drawback of only showing lower-bound on states than state-action pairs?
2. In Appendix A.2, could the paper include more detailed steps of solving inner-minimization over Q while proofing Theorem 6.2?



clarity,_quality,_novelty_and_reproducibility: Quality: The idea is interesting. However, it currently feels like a stretch to call the paper complete due to the aforementioned problems.

Clarity: I like that all notations are clearly defined. The paper reads clearly overall. The flow of content is reasonable as well.

Novelty: The idea of flexible conservatism in conservative methods under offline RL is novel.

Reproducibility: The pseudocode is clear, but it might still take some effort to reproduce the results as the equation 4/5 is somewhat complicated.
summary_of_the_review: I really like the idea of using flexible or dynamic level of conservatism/optimism/pessimism in RL. I feel there is an ongoing trend in this direction, in both online and offline RL. However, because of the flaws in the empirical study of the paper, I recommend the paper to be rejected at this point.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
details_of_ethics_concerns: Dual submission for NeurIPS 2022
- link: https://openreview.net/forum?id=PjBBFo8X2D
- link: https://openreview.net/forum?id=PjBBFo8X2D&referrer=%5Bthe%20profile%20of%20Sergey%20Levine%5D(%2Fprofile%3Fid%3D~Sergey_Levine1)
- link: https://neurips.cc/Conferences/2022/ScheduleMultitrack?event=60659
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper5450/Reviewer_pdN5
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The authors propose a novel approach to estimating lower bounds of $Q$-functions. Based on the approach, the authors propose CCVL and empirically justify that CCVL outperforms several offline RL SOTAs.
strength_and_weaknesses: $\textbf{Strength:}$

The topic of offline RL is essential to the field. The quantile estimation and confidence adaptation proposed in this paper are novel to me and perform well empirically.


$\textbf{Weaknesses and Questions:}$

There are a couple of unclear points to me. 
$\textbf{Q1:}$
The authors may want to specify the randomness source in equation (3), as $Q^*(s, a)$ is itself deterministic, so the probability that $Q^*(s, a)>q$ is either $0$ or $1$. I assume that the probability bound means the bound on some estimated $Q^*$ given the dataset, or is there a typo in (3)?

$\textbf{Q2:}$
Which objective ((4) or (5)) does the authors adopt in the empirical comparisons? Is the policy in (9) adopted in empirical comparisons? Does the uniformly sampled $\delta$, $\delta_1$, and $\delta_2$ affect the lower bound estimation in Algorithm 1, as the right-hand side does not align with (4)?

$\textbf{Q3:}$
In addition, the theorem feels insufficient to me. What is the benefit of utilizing different confidence levels in the target estimation of (4)? Can the authors justify their benefit compared with standard pessimistic methods in (1)? 

$\textbf{Q4:}$
In the grid world experiments, how does CCVL learns the optimal trajectory given that there seems to be no coverage of such trajectory in the dataset (illustrated in the leftmost figure of Figure 1)? Is the performance of CCVL desirable, as offline RL is supposed to rely more on confident regions from the dataset?

clarity,_quality,_novelty_and_reproducibility: See questions above. I might have missed it but I did not find the code for experiments.
summary_of_the_review: The paper seems interesting but there are a few unclear points to me. I will raise my score if the authors can address my questions.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper5450/Reviewer_Fh1m
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper proposed a new offline RL algorithm, by learning the value function conditioned on the confidence level. The proposed algorithm learns a Q function parameterized by the confidence level of future backup and executes an adaptive policy during test time. The paper shows the learned Q function as a high-confidence bound of the optimal Q function. Empirical results in Atari games show the benefit of the proposed algorithm over CQL and REM.
strength_and_weaknesses: Strength:
1. The idea of confidence-conditioned value function learning is a very novel way of encoding confidence levels into offline reinforcement learning algorithms. This is different from all previous papers using explicit or implicit confidence-based penalty in Q learning, as CCVL realize a more ambitious learning goal of learning Q at all confidence level.
2. At the test time, CCVL executes a non-Markovian policy which adaptively changes the confidence level \delta. This gives an additional capacity for test time learning/tuning for the policy. It is also a novel contribution of this paper.

Weakness:
1. Additional ablation study and more description of methods in existing ablation studies.
 - The implementation use Eq (5) instead of Eq (4), what will be the performance of CCVL with the update rule from Eq (4)?
 - A more detailed description of how AEVL and Fixed-CCVL are implemented should be included in the appendix.
2. The proposed algorithm requires additional sampling or computation. It would be better to conduct an analysis of how much these parts cost additional computation compared with baselines.
 - Eq (5) need to hold for all \delta. So the CCVL needs to perform the learning over all \delta or sample \delta enough times (as in Alg 1) for convergence. 
 - Confidence adaptive policy requires more computation at the test time as well.


Minor comments:
1. In section 5, it is unclear why n(s) is approximated by phi(s)^T phi(s). According to the argument in O’Donoghue et al. (2018), ignoring the difference between actions, it should still be phi(s)^T \Lambda^{-1} phi(s), where \Lambda is the empirical feature covariance matrix.
2. Eq (4) and Eq(5) have ambiguity around the operator max_{\delta_1, \delta_2}. There should be a parenthesis to clarify the scope.
3. \hat{Q} is not well defined without showing the limit exists or contraction, as the Bellman operator is modified.
clarity,_quality,_novelty_and_reproducibility: Clarity: There are some clarity issues with the paper. See minor comments in the previous section.

Quality: This paper gives theoretical guarantees for finite state-action MDP and empirical study in Atari games. This is solid work, though the empirical study can be improved by more benchmarks and baselines. 

Novelty: The proposed algorithm has two novel ideas: confidence-conditioned value learning and adaptive policy. 

Reproducibility: No code provided. Hyperparameters and implementation details are provided.

summary_of_the_review: A good paper with several novel ideas. Some clarity issues and additional studies need to be settled before acceptance.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper5450/Reviewer_DkfW
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper proposes CCVL (confidence conditioned value learning), an offline RL method that learns value functions parameterized by the "confidence level". The main algorithm is adapted from CQL and anti-exploration bonus, with explicit modifications (supported by theory) to guarantee that the learned Q-values are lower (or upper) bounds with the probability at least being the selected "confidence level". Experiments were conducted in a toy grid-world problem as well as Atari benchmark and shows favorable performance compared to CQL and other baselines. 
strength_and_weaknesses: **Strengths:**
- Problem is well-motivated, mathematical descriptions and derivations are detailed. 
- I really like the toy gridworld problem, which is illustrative of the expected effect of using the proposed algorithm. 

**Weaknesses:**
- Some theoretical details need clarification:
  - Could you clarify the proof for Lemma 6.1, I don't think I follow why $\delta_1,\delta_2 \le \delta'$ "implies $Q(s, a, δ) ≤ Q(s, a, δ' )$, as desired". 
  - How does setting $\delta_1 = \delta$ in Eqn (4) reduce it to Eqn (1), the numerator of the fraction under the sqrt is $\log(1/\delta)$ instead of 1
- Is $\alpha$ still a hyperparameter, how should it be chosen, and what is its interpretation now that $\delta$ "confidence" is introduced. Since in CQL $\alpha$ is used to implicitly control the confidence level in the pessimistic Q-values. 
- It's best to make clearer how much of the observed performance gains in experiments is from the proposed learning confidence-conditioned values, vs from some of the empirical/engineering decisions in Sec 5 practical algorithms (IQN, approx inverse visitation). Perhaps if there's a simpler (maybe tabular?) problem that doesn't have these deep RL complexities that would strengthen the results and help the reader understand the contribution better - and help to show what the learned Q-values are actually doing. 
- There are some typos throughout, "hyperparamters" and "suprenum"
clarity,_quality,_novelty_and_reproducibility: **Clarity:** The paper is largely well-written, with mathematical details and experiment descriptions. 

**Quality:** Experiments adopted best practices (IQM) in reporting RL benchmark results. 

**Novelty:** As far as this reviewer is aware, the proposed method is new and not seen in previous work. 

**Reproducibility:** Implementation is currently not provided with the submission. Since RL results tend to be highly variable depending on the implementation, and there are a lot of moving pieces in the proposed approach (confidence sampling, hyperparameters, quantile nets, inverse visitation), I highly encourage the authors to share code after publication. 
summary_of_the_review: Overall the paper is well-written with good methodological contribution and theoretical analysis, supported by benchmarking results. Some theoretical details can be clarified. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold