From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: This paper extends existing benchmarks for offline policy evaluation and studies the problem of offline policy comparison with confidence (OPCC). It creates benchmarks, introduces multiple metrics and compares a set of baselines using model-based policy evaluation with ensemble-based uncertainty quantification.

Strengths:
- All reviewers agree that offline policy evaluation with uncertainty quantification is an important and realistic problem setup.
- It introduces additional useful metrics for comparison baselines.
- Some reviewers consider the benchmarks include sufficient number of tasks, and policy comparison queries.

Weaknesses:
- Incremental contribution and novelty based on existing OPE benchmarks
- Lack of diversity in the environments and datasets
- The baseline comparison on the proposed benchmarks do not provide clear conclusions about the relative advantage of each method.
- Clarify should be improved even after rebuttal.

The main concerns from reviewers are not addressed after rebuttal.

justification_for_why_not_higher_score: The authors' rebuttal reinforced the concerns pointed out by the reviewers.
justification_for_why_not_lower_score: N/A

From: ICLR.cc/2023/Conference/Paper3003/Reviewer_SRAe
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper proposes a new problem in reinforcement learning namely offline policy comparison with confidence (OPCC) and proposes a set of evaluation benchmarks and baseline methods. The problem is similar to the offline policy selection problem but requires the algorithm to output a confidence level as well. The authors propose different evaluation metrics such as Area under the risk-coverage curve (AURCC), Reverse Pair Proportion (RPP), and Coverage Resolution (CR) to evaluate the policy selection and confidence output results.
strength_and_weaknesses: Strength: 
1. The problem setup proposed in this paper is very meaningful and realistic. The policy comparison problem is necessary for different offline RL parts: hyper-parameter selection, cross-validation, safe policy improvement, etc. This part does not have enough discussion in previous offline RL literature and a rigorous methodology and benchmark can help the field.
2. Using different evaluation metrics and the discussion risk versus coverage trade-off is helpful to illustrate some properties of the OPCC problem. 

Weakness:
As a general benchmark for the OPCC problem, I think the proposed evaluation tasks/datasets are in a very limited scope. 
1. It lacks diversity in the types of environment: 3 of the 7 tasks are control problems in Mujoco simulation. These problems share many common properties such as the magnitude of the horizon, types of state features, action dimensions, and reward distribution, and not all RL application domains share these. The other 4 are the same type of visual maze problem with different maps. The lack of more types of problems could be concerning for a benchmark that could guide future algorithm design.
2. The evaluation benchmark uses D4RL datasets in the 3 Mujoco tasks, which are known to consist of a "significant fraction of near-optimal trajectories." (Kostrikov et al., 2021, Offline Reinforcement Learning with Implicit Q-Learning). Some other datasets such as AntMaze in D4RL with more different patterns are not included. it is also concerning whether this special structure will prefer a particular type of algorithm.
3. It lacks domains with discrete action space, either simple control tasks or Atari games.
clarity,_quality,_novelty_and_reproducibility: Clarity: This paper is written clearly. It can be easily followed and the main contribution is easy to understand. The definition of the OPCC problem part is written especially clearly.

Quality: See the Strength And Weaknesses section. 

Novelty: Due to the similarity, it is better to extend the discussion of prior offline policy selection work (including the evaluation metrics there), the connection with OPCC, and the motivation of using confidence and query per state.

Reproducibility: the author released the datasets and code for baselines.
summary_of_the_review: This paper proposes several benchmarks and baseline methods for the problem named offline policy comparison with confidence. The benchmarks are very limited and may not be representative enough.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper3003/Reviewer_ErjK
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: First, this paper formalized the problem of "offline policy comparison with confidence" (OPCC), where the goal is to answer "policy comparison queries" (PCQs) the state-values $V$ of possibly different policies in possibly different states. The answer to PCQs are expected to include both a binary output of whether "$π_1, s_1$'s value is less than $π_2, s_2$'s", and a confidence level $c$. It then outlines three evaluation metrics for OPCCs with respect to a space of PCQs, including the AURCC, Reverse Pair Proportion, and Coverage Resolution. 

The second contribution is benchmark construction from two sets of existing RL benchmarks (namely Maze2d and mujoco), describing how the space of PCQs are constructed. 

The third contribution is a pilot experiments applying existing ensemble methods as baselines, and shows there signification future is needed to achieve good performance of OPCC. 
strength_and_weaknesses: **Strengths**
- The paper is overall well written, and the problem studied is well motivated and interesting. 

**Weaknesses**
- The main contribution is the new evaluation metrics and the new benchmark construction strategies. I think it needs to be make clearer how it differentiates from past benchmarks like DOPE. 
- I think there needs to be an expanded discussion on the selected metrics, especially since the Spearman rank correlation was not used, which was used in past works including the cited DOPE benchmark by Fu et al., Paine et al. [1] and Tang [2]. It seems to be related to the proposed RPP so it would be good to provide a clarification. 
clarity,_quality,_novelty_and_reproducibility: Clarity: overall this paper is clear. 

Quality: I have no serious concerns about the technical quality of the paper. 

Novelty: some work utilizing similar ideas have been proposed in the past, but were not cited in the paper nor compared to. e.g. [3].

Reproducibility: anonymized github code is provided and looks comprehensive and well documented.

References:
[1] Paine et al. Hyperparameter selection for offline reinforcement learning. https://arxiv.org/abs/2007.09055
[2] Tang & Wiens. Model Selection for Offline Reinforcement Learning: Practical Considerations in Healthcare. MLHC 2021. https://proceedings.mlr.press/v149/tang21a.html
[3] Irpan et al, Off-Policy Evaluation via Off-Policy Classification, NeurIPS 2019. https://proceedings.neurips.cc/paper/2019/hash/b5b03f06271f8917685d14cea7c6c50a-Abstract.html
summary_of_the_review: This paper is generally clear, but the main contribution needs to be more differentiated from the existing literature, and there appear to be some missing related works. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3003/Reviewer_GVaQ
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper proposes a benchmark for offline policy comparison with confidence (OPCC). The benchmark builds on top of datasets from D4RL by specifying both sets of policy comparison queries (PCQs) that compare a variety of policies at a variety of states and a set of metrics to evaluate the performance of different evaluation algorithms. The paper also provides a set of baseline experiments using model-based policy evaluation with ensemble-based uncertainty quantification.
strength_and_weaknesses: ### Strengths

1. The introduction of PCQs that evaluate candidate policies at a variety of initial states is a nice improvement over prior work like DOPE that focuses on evaluation at initial states which may not capture the full effect of distribution shift. This seems relevant for practical applications, as described in the paper.

2. The metrics take into account explicit uncertainty estimates provided by the evaluation algorithm. This is a nice improvement over prior work that evaluates point estimates of policy performance. This also seems like a practically important consideration.

3. The benchmark and evaluation seem to have been rigorously tested with sufficient numbers of tasks, PCQs, seed, etc. 

4. The provided code seems to be fairly clean and well-documented which is important for a benchmark paper.

### Weaknesses

1. The main weakness in my mind is that it is unclear whether the benchmark will be able to meaningfully and clearly separate different approaches. This is seen in the experimental results where there are essentially no clear takeaways despite the rigorous experimentation. The paper argues that this is evidence of room for improvement, but I think it may, on the contrary, be evidence that the benchmark data and evaluation sets are constructed such that all reasonable algorithms perform about the same. I understand that it may just be the case that most of these algorithmic decisions do not matter very much, but it is important for a benchmark to provide clear outputs on what works where. I can see two potential solutions that the authors could pursue to partially address this issue: (1) demonstrate that there are naive (but still reasonable) baselines that perform substantially worse than the algorithms that are evaluated in the current version, and/or (2) create some more easily interpretable summary metrics that argue that the current results are actually showing significant differences between algorithms.

2. I also have some worries about the low-level details of the collection of the datasets. First, I agree that it makes sense to filter out the ambiguous policy comparisons (step 4 of section A.2). However, the proposed method of just filtering out queries with difference less than 10 seems quite arbitrary given that the tasks have very different reward scales. In particular, this can be seen in Figure 4, where this filtering does not seem to have much effect in the gym tasks. It seems that it would make more sense to filter based on some more adaptive criteria, perhaps related to the variance of the distribution of returns. Second, there is not sufficient detail in appendix A.2 to understand how the policies and initial states are selected. What reward functions were the policies trained on and for how long? How were the candidate states selected from the rollouts? Third, I am not sure of the reasoning behind the choices of horizons between 10 and 50. This doesn't seem like a totally unreasonable choice, but there needs to be some rationale for using horizons much shorter than the length of the task being considered (1000 for the gym tasks).  
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to follow.

The quality of benchmark construction and baselines is solid, up to the issues mentioned above.

The proposed benchmark and metrics are novel in subtle, but important ways as listed in the strengths section.

The provided code should make the results reproducible, but I did not try to run it myself.
summary_of_the_review: Overall, I think that the paper takes a well-considered approach to benchmarking offline policy evaluation algorithms. I am still worried about the ability of the benchmark to clearly separate different approaches and about some minor details, so I will rate the paper as a weak accept for now. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3003/Reviewer_J8DJ
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper is concerned with the confidence of offline policy evaluation. The contribution is on the one hand that existing benchmarks for offline policy evaluation from (Fu et al., 2021) are extended, and on the other hand that a series of model-ensemble based offline RL methods are evaluated in experiments using the modified benchmark.
strength_and_weaknesses: **Strengths**
* The paper addresses an important topic.
* Further development of benchmarks can stimulate research progress.
* The paper provides a systematic investigation of different approaches to derive uncertainty from model ensembles.

**Weaknesses**
* The contribution is incremental. It remains unclear whether it contains sufficient novelty.
* Despite the extensive appendix, some aspects remain unclear.
clarity,_quality,_novelty_and_reproducibility: **Clarity** 

Despite the extensive appendix, some points are unclear to me.

* The term "sequential-action policies" seems unusual to me. What is the reason for using it? What is meant by it?
 * "baselines" is used in the sense of "baseline methods". I would have found it easier to get into the paper if "baseline methods" had been used. "baselines" has other meanings as well. E.g., Laroche et al, Safe Policy Improvement with Baseline Bootstrapping, baseline policiy, baseline performance.

* The term "offline policy comparison" has not been defined and is not distinguished from "offline policy evaluation."

* It is said that (Fu et al., 2021). do "not propose evaluation metrics and protocols for measuring
uncertainty quantification over policy rankings." What is meant by "uncertainty quantification over policy rankings"? This term is not used in a second place in the paper and accordingly it is not shown that this paper (in contrast to (Fu et al., 2021)) provides "evaluation metrics and protocols for measuring
uncertainty quantification over policy rankings."


**Quality** 
High

**Novelty** Contains novelty, but incremental and somewhat unclear if this is sufficient.

**Reproducibility** Very good (github repository).

**Further remarks**

Please note that in Hans et al., Agent self-assessment: Determining policy quality without execution (2011) a method for offline policy comparision with confidence has been presented, but only for discrete MDPs and with limited success. This should be mentioned,

I like very much that in Table 1-3 the 95% confidence interval is given as uncertainty (after the $\pm$).
However, I do not understand why in Table 5 the standard deviation is given and I do not agree that mean $\pm$ standard deviation is written. The $\pm$ sign is used to specify the uncertainty of the measured value (here the mean). The standard deviation is not suitable as a measure of uncertainty. Confidence intervals or standard errors are suitable.


**Typos**

"a MDP" -> "an MDP"

"bayesian"

"markov"

"Monte carlo"

Blanks are missing in many places. In some places there are extra blanks.
summary_of_the_review: Although the novelty is only incremental, I still consider the contribution important because it aims at systematic investigation of an important topic. The other points of criticism can be remedied with reasonable effort.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold