Paper: Towards Assessing and Benchmarking Risk-Return Tradeoff of Off-Policy Evaluation (ICLR-2024)

Reviewer comment (Evaluation):
The benchmark results show that SharpeRatio@k can sometime diverge significantly from the conventional metrics in terms of estimator selection. Additional discussion should be added on how practitioners may consolidate the insights given by the different metrics evaluated under certain scenarios.

Author response:
Thank you for the valuable feedback. The estimator selection results under SharpeRatio@k sometimes diverge from others because SharpeRatio takes the risk into account when evaluating OPE estimators, while other metrics ignore the risk factor. In general, it would be ideal to be able to develop an estimator that performs best in terms of both existing metrics and SharpeRatio. If it is not possible, one should prioritize SharpeRatio@k more when they do not want to take any risk of deploying detrimental policies in the A/B test. When an estimator is preferred by only existing metrics, the estimator is likely to be high-return but also high-risk, so it should be avoided in high-stakes scenarios such as medical applications. We will provide an in-depth discussion about this point in the revision.

Reviewer comment (Evaluation):
The classic sharpe ratio uses the mean return of the portfolio as the numerator instead of the best policy's return. Is there a particular reason that the best return is used in the proposed metric?

Author response:
This is a great point. We focus on the best policy’s performance in the numerator because only the best policy in the policy portfolio will be finally chosen via A/B testing as a production policy, as illustrated in Figure 1. To define the numerator, we also subtract the behavior policy’s performance ($J(\pi_b)$). By doing so, we can avoid an undesired edge case where an estimator that chooses only policies that are worse than the behavior policy but accidentally has extremely small std (denominator) is evaluated as the best method under our metric. We will clarify this in Section 3.

Reviewer comment (Novelty):
My biggest concern is about the technical contribution of the work. While the idea seems novel it also sound quite natural and simple and bears concern about the actual interest that would spark in the community.

Author response:
Thank you for the great comment. First, we think that the simplicity of our evaluation metric is actually one of its important advantages. It means that it can be used fairly easily in future research and practice. Reviewer n49S acknowledges that being natural and easy-to-use is one of the main advantages of our metric. Second, a method’s simplicity does not necessarily mean that it is not novel and valuable. The concept of evaluating OPE estimators’ risk-return tradeoff and the use of SharpeRatio as an evaluator of OPE estimators are the ideas that we have never seen in the relevant literature. Besides, the similar types of contributions have been evaluated highly by the relevant community. We can actually list several published papers that share similar types of contributions to ours, i.e., (1) highlighting the problem of existing evaluation protocols and (2) resolving the issues with simple and easy-to-use metrics.

- **(Fu et al., ICLR2021)** (1) This paper identified that an OPE estimator that has a lower estimation error (MSE) does not always align candidate policies better than those with a higher estimation error. (2) To address this gap in evaluating policy alignment in OPE, the paper proposed simple rank correlation and regret to see if OPE estimators align candidate policies well. These metrics are now recognized as important metrics of OPE and are used in e.g., (Tang and Wiens, 2021), (Qin et al, 2022), and (Chang et al., 2022).

- **(Agarwal et al., NeurIPS2021)** (1) This paper identified that taking the naive average of the performance of online RL can be sensitive to variance, particularly when only a handful of trials are allowed due to high computational costs. (2) To enable a more statistically robust evaluation of online RL in such situations, the paper proposed to use the simple average of the middle 50% of trials. This paper was awarded the *Outstanding Paper Award for NeurIPS 2021*.

- **(Chan et al., ICLR2020)** This paper (1) identified that the naive average of the performance of online RL may overlook a critical safety concern. (2) To see if the RL policy is risk-averse or not, the paper proposed to report the simple average of the worst $\alpha$% of trials (referred to as conditional value at risk; CVaR). CVaR is widely used in safe-RL. 

**It would thus be great if the reviewer could evaluate our contributions based on their quality, significance, soundness, impact, rather than what the proposed method looks like.** It would also be great if the reviewer could pay attention to another key contribution we have provided, i.e., the development of the open-source software. 

====

(Fu et al., ICLR2021) Justin Fu, Mohammad Norouzi, Ofir Nachum, George Tucker, Ziyu Wang, Alexander Novikov, Mengjiao Yang, Michael R. Zhang, Yutian Chen, Aviral Kumar, Cosmin Paduraru, Sergey Levine, Tom Le Paine. “Benchmarks for Deep Off-Policy Evaluation.” 

(Agarwal et al., NeurIPS2021) Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, Marc G. Bellemare. “Deep Reinforcement Learning at the Edge of the Statistical Precipice.”

(Chan et al., ICLR2020) Stephanie C.Y. Chan, Samuel Fishman, John Canny, Anoop Korattikara, Sergio Guadarrama. “Measuring the Reliability of Reinforcement Learning Algorithms.”

(Tang and Weins, 2021) Shengpu Tang, Jenna Wiens. "Model selection for offline reinforcement learning: Practical considerations for healthcare settings." Machine Learning for Healthcare Conference, 2021.

(Qin et al., 2022) Rongjun Qin, Songyi Gao, Xingyuan Zhang, Zhen Xu, Shengkai Huang, Zewen Li, Weinan Zhang, and Yang Yu. “NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning.” NeurIPS Datasets & Benchmarks, 2022.

(Chang et al., 2022) Jonathan D. Chang, Kaiwen Wang, Nathan Kallus, Wen Sun. “NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning” ICML, 2022.

Reviewer comment (Novelty):
The introduction of Sharpe ratio like measure in OPE seems poorly motivated by the authors who should try to sell better the motivations for the idea.

Author response:
Let us begin with clarifying that SharpeRatio is not a new estimator in OPE, but is a new metric to evaluate the risk-return tradeoff of OPE estimators (in case the reviewer misunderstands what it is). It is novel because no existing metrics consider the risk aspect of OPE estimators. It is practically important to measure the risk in OPE because, given the two-stage policy evaluation procedure in Figure 1, one would often like to avoid deploying detrimental policies in the A/B testing phase. We have provided two example situations in Section 3 where every existing metric (MSE, Regret, and RankCorr) cannot evaluate the risk in OPE since they focus merely on the accuracy of OPE (MSE), the resulting policy alignment (RankCorr), or top-1 policy selection (Regret). Reviewers S1vJ and n49S acknowledge that the toy examples we provided are helpful for understanding the importance of considering the risk-return tradeoff in the evaluation of OPE estimators and advantages of the proposed metric in their review.

Reviewer comment (Theory):
Is there any provable theoretical advantages of using your method compared to others?

Author response:
There are no particular theoretical advantages about our metric. We do not think that such a theoretical proof is necessary or possible to compare evaluation metrics that are defined under completely different motivations. For example, how would we be able to prove that the AUC score is better than the F1 score (and vice versa) for the typical classification problem? An important point here is that there is no metric that can evaluate the risk-return tradeoff of OPE estimators in previous research, and our formulation and metric are the first to enable it. We have demonstrated that existing metrics (MSE, Regret, and RankCorr) cannot evaluate the risk aspect of OPE estimators in Section 3 through toy examples. We (and Reviewers S1vJ and n49S) think this is sufficiently useful to motivate our metric and demonstrate its value. We would thus argue that there is no particular need to perform theoretical analysis in our paper.

We would appreciate it if the reviewer could clarify what kind of theoretical analysis about the comparison of different evaluation metrics is desired (or even possible) if our response is not satisfactory.

Reviewer comment (Writing):
Seems like $\hat J(\pi; D)$ was never defined explicitly.

Author response:
Thank you for catching. It is an arbitrary OPE estimator. There is no single definition, but examples include (but not limited to) DM, PDIS, DR, and MIS as we described in Section 4.2. We provided rigorous definitions of these estimators using math notations in Appendix A.1. We will clarify this in the revision.

Reviewer comment (Presentation):
I felt the figures and toy examples could use more explanation. For example, it wasn’t clear in Fig 2 what the red dots are, black dots are and how to interpret the axes.

Author response:
Thank you for catching the ambiguity. The red dots indicate the policies whose values are estimated correctly by the estimator, while the black dots indicate the policies whose values are not correctly estimated.

Your understanding about estimator X (“my understanding of .. next best policies”) is correct indeed, and the three points in the shaded region are chosen as the top-$k$ policies under estimator X (these three policies have higher estimated policy value (y-axis) than others).

On the other hand, estimator Y substantially overestimates the value of black policies. As a result, one red policy (the rightmost one) and two black policies are chosen as the top-$k$ policies (policy portfolio) under this estimator, which is considered risky because the true values of these black policies are quite low.

Reviewer comment (Novelty):
Related work: In the section on “Risk Measures and risk-Return Tradeoff in Statistics and Finance”, the paper discusses the Sharpe,1998 paper in length (relevant for the Sharpe ratio used in current paper). However, there are only two other related works identified in this entire section. I’m not an expert in this domain but I suspect there has to be more prior research done in this space – and if that is true, the existing related work section seems a little thin.

Author response:
Thank you for the valuable comment. Indeed, there are several variants of risk measures used in finance such as the information ratio, value at risk (VaR), marginal VaR, and entropic VaR (some of them have been used in bandits and RL). We will add comments about these relevant metrics in the section and keep improving the writing of the paper.

Reviewer comment (Writing):
In description of Fig 2, I found the sentence “X underestimates the performance… Y overestimates” confusing – I was wondering if it should be the other way around. If the current sentence has to be true, then my understanding of the interpretation is that black policies suffer an underestimation (i.e. even though x-axis value is high, y-axis value is lower than ground truth) and so the top-k ends up picking next best policies. Is this correct or am I missing something?

Author response:
Thank you for catching the ambiguity. The red dots indicate the policies whose values are estimated correctly by the estimator, while the black dots indicate the policies whose values are not correctly estimated.

Your understanding about estimator X (“my understanding of .. next best policies”) is correct indeed, and the three points in the shaded region are chosen as the top-$k$ policies under estimator X (these three policies have higher estimated policy value (y-axis) than others).

On the other hand, estimator Y substantially overestimates the value of black policies. As a result, one red policy (the rightmost one) and two black policies are chosen as the top-$k$ policies (policy portfolio) under this estimator, which is considered risky because the true values of these black policies are quite low.

Reviewer comment (Novelty):
Top-k policies are considered in several resource allocation setups where there are limited resources and some index is computed so that resource can be allocated to the top k indexes (For e.g. [1] “Restless Bandits: Activity Allocation in Changing World”, P Whittle; [2] https://arxiv.org/abs/2110.02128). Can the method be used for designing robust top-k selection/resource allocation policies in this context? I also wonder how it compares to existing robustness in bandits work such as [3] https://arxiv.org/abs/2107.01689 and whether those ideas can be applied to the OPE evaluation setting discussed in the current paper.

Author response:
We agree that the high-level concept of the risk-return tradeoff can be applied to the problem of resource allocation and restless bandit as suggested by the reviewer. A notable difference is that we used the concept to define a new metric to evaluate OPE estimators while the same concept can be useful to define a new performance measure of a bandit policy in the field of resource allocation and restless bandit. We can actually find some similar attempts such as the risk-aversion bandit (Sani et al. 2012). Even though this is an independent interest, we thank the reviewer for sharing this interesting thought.

(Sani et al. 2012) Amir Sani, Alessandro Lazaric, and Rémi Munos. Risk-Aversion in Multi-armed Bandits. NuerIPS2012.