Paper: Kernel Metric Learning for In-Sample Off-Policy Evaluation of Deterministic RL Policies (ICLR-2024)

Reviewer comment (Novelty):
I would appreciate it if the authors could explain more on the significance of the research question. The paper indicates the method is designed for continuous control tasks. In continuous control tasks, it is common to see an agent learn a stochastic policy instead of a deterministic policy, as many of the reinforcement learning algorithms suitable for continuous control tasks can work with a stochastic policy (BCQ, IQL, InAC, IAC, etc.), and it is also a simple and straightforward solution to avoid the zero importance sampling ratio. I think the importance of solving the issue in deterministic policy evaluation can be clearer if the paper can include an explanation of when people suffer from this issue in practice.

Author response:
**Q1. The practicality of evaluating a deterministic policy and its example**

Please note that our algorithm is not an offline RL algorithm that can choose the form of the target policy (stochastic/deterministic) but an algorithm for evaluating a given deterministic policy.

We focused on evaluating deterministic target policies since there are many cases where the evaluation of deterministic policies is needed, e.g., safety-critical systems such as industrial robot control and drug prescription, where the sequences of actions need to be precisely and consistently controlled without introducing variability [Silver et al., 2014][Kallus and Zhou, 2018]. In these situations, one may want to evaluate the performance of the deterministic policies based on offline data when the robot control or drug prescription policy is updated before deployment.

We included an explanation of the practicality of our problem setting and an example in the second paragraph of the introduction section.


$ $

[Silver et al., 2014] Silver et al. “Deterministic Policy Gradient Algorithms” ICML 2014.

[Kallus and Zhou, 2018] Kallus and Zhou. "Policy Evaluation and Optimization with Continuous Treatments.." AISTATS 2018.

Reviewer comment (Experiments):
Another question is about the experiment setting. When proposing the new method, the paper indicates that the method is for in-sample learning. However, in the experiment section, the target policy is trained by TD3. The behavior policy is different from the target policy and does not guarantee to be in-sample (the target policy is a stochastic policy generated from another TD3 learned policy, whose performance is 70%~80% of the target policy, according to the paper). It is not clear to me how the in-sample condition is ensured in the experiment.

Author response:
**Q2. How in-sample learning is conducted in the experiments?**

Please note that in-sample learning does not mean on-policy learning. In-sample learning denotes the method that learns the value function by only using actions in the (off-policy) dataset [Xu et al., 2023][Kostrikov et al., 2022]. Even though the stochastic behavior policy and the deterministic target policy differ, it is in-sample learning if the actions used in the Q updates are in the dataset. Our method updates the Q-function with the update vector estimated only using the samples in the data following Eq.(6). Detailed procedure of our algorithm used in the experiments is presented in Algorithm 1 in Appendix B. Our algorithm works when the support of a deterministic target policy relaxed by a kernel is in the support of a stochastic behavior policy, and the experiment setting is made to satisfy the condition.

To clear up how our algorithm does in-sample learning, we explicitly state in the main text that the pseudo-code of our algorithm is in Algorithm 1 in Appendix B. Furthermore, we modified Algorithm 1 to explain how the samples in the data are used to learn Q-function.

$ $

[Xu et al., 2023] Xu et al., "Offline RL with no OOD actions: In-sample learning via implicit value regularization." ICLR 2023.

[Kostrikov et al., 2022] Kostrikov et al., "Offline reinforcement learning with implicit q-learning." ICLR 2022.

Reviewer comment (Writing):
The paper is begging for an example of when you might want to evaluate a deterministic policy using data generated by a stochastic policy. From my experience, in reinforcement learning we often have the inverse problem. For instance, if using expert data generated through a PID controller or other classic control algorithm we will encounter deterministic behavior policies. Other than as baselines to compare RL algorithms, I’m struggling to understand when we might encounter the need to evaluate a deterministic policy. If this is the only example, that isn’t a deal breaker but an example would make the paper much better.

Author response:
**W1. Practicality of evaluating a deterministic policy using data generated by a stochastic policy and its example**

There are many cases where the evaluation of deterministic policies is needed, e.g., safety-critical systems such as industrial robot control and drug prescription, where the sequences of actions need to be precisely and consistently controlled without introducing variability [Silver et al., 2014][Kallus and Zhou, 2018]. In these situations, one may want to evaluate the performance of the deterministic policies based on offline data when the robot control or drug prescription policy is updated before deployment. For the offline data, if the data is collected from the policies used in the past, the data can be viewed as sampled from a single stochastic behavior policy.

We included an explanation of the practicality of our problem setting and an example in the second paragraph of the introduction section.

$ $

[Silver et al., 2014] Silver et al. “Deterministic Policy Gradient Algorithms” ICML 2014.

[Kallus and Zhou, 2018] Kallus and Zhou. "Policy Evaluation and Optimization with Continuous Treatments.." AISTATS 2018.

Reviewer comment (Reproducibility):
There should be a statement on how the hyperparameters are chosen for all the methods. While this is in the appendix, there should be space made for this in the main paper. One notable missing part of this was a discussion on how to set the bandwidth. Is this given or determined by the data (through equation 10)? If it is given, are there any rules of thumb that would help practitioners? This is discussed briefly in terms of how this determines the bias-variance trade-off, but I believe this is lacking further exploration for practical use-cases.

Author response:
**W3. Hyperparameter selection**

For the hyperparameters in our method, bandwidth $h$ in our method is learned by Eq. (10).  As for the IS ratio clipping range, we selected the range by grid search on Hopper-v2 domain from the cartesian product of minimum IS ratio clip values  {1e-5, 1e-3, 1e-1} and maximum IS ratio clip values  {1, 2, 10} and applied same hyperparameters on the other environments. For the hyperparameters in the baselines, we mostly followed the settings in SR-DICE as they experimented on the same or similar environments.

We added a brief explanation of the hyperparameter settings in the experiment section and details in Appendix C.1, “Hyperparameters” section.

Reviewer comment (Writing):
Section 4/4.1 is written in an extremely dense manner. This is partially necessary due to the complexity of the math behind the kernel relaxation for the target policy, but I believe the authors could have done more to explain the theorems and propositions in more intuitive forms. I also believe separating section 4.2 into two sections could be beneficial (Optimal bandwidth/metric in one and bounding the contraction of the Bellman operators).

Author response:
**S1. Intuitive explanation of the theorems and propositions**

Thank you for your suggestion. We added intuitive explanations of the theorems and propositions in the main text. For Theorem1, we included an intuitive analysis of the leading-order bias in the paragraph after Theorem 1. For Proposition 3, we included an intuitive explanation of how the metric measures similarity between actions and how it is reflected in the importance resampling in the second paragraph after Proposition 3.

$ $

**S2. Separation of section 4.2 into Optimal bandith/metric section and error bound analysis section**

Thank you for your suggestion. The contents related to the error bound are put in a separate subsection “4.3 Error Bound Analysis”