Paper: Universal Off-Policy Selection for Human-Centric Systems via Participant Sub-grouping (ICLR-2024)

Reviewer comment (Experiments):
At the outset, the idea of selecting a policy out of many for each sub-group sounds like designing a new policy. I have difficulty understanding whether, during the partitioning step, any information unavailable to the policy is being used or not. In fact, if there is some information, like patient characteristics, which are used to cluster participants but were not used in training the policy, why not incorporate them in the first place to train the policy?

Author response:
This is a great question. Unfortunately **we were not able to integrate the sub-group partitioning (or patient characteristic) information into the offline training stage due to the regulatory limitations in HCSs**. Please refer to our response to Txkj's Q2 for more details on this. Moreover, such a practical limitation further highlights the needs of off-policy evaluation and selection prior to online deployment in HCSs, for safety verification as well as improving the efficiency of online testing, which is the motivation of this work. Please also see our response to Q5 below as it is related to this question.

Reviewer comment (Writing):
The writing could be improved as there are many typos in the text. For example: "it it" in the abstract, "a initial" in Assumption 2, "an classic" on page 6.

Author response:
Thank you for pointing out this. We have fixed these typos in the most recent revision uploaded.

Reviewer comment (Experiments):
I recommend applying your technique to real offline data of sepsis treatment. For instance, I chose Raghu et al. from your paper and checked its 2023 citations and it seems there are many real evaluations of the policy values including new reliable methods.

Author response:
Thank you for this suggestion. However, as pointed out in [1] that healthcare databases, such as MIMIC, are commonly used for retrospective studies, making it intractable to obtain the actual returns the policies would lead to (as they cannot be deployed). Moreover, as suggested by [2], simulated environments are helpful for "shedding light on certain operating characteristics and best practices". As a result, we chose to rather follow the counterfactual environment introduced in [1], as it would greatly help **isolate the source of improvement regardless of the evaluation approaches used**. Note that this environment is used by many other existing works as cited in the paper, i.e., Hao et al., 2021; Nie et al., 2022; Tang & Wiens, 2021; Lorberbom et al., 2021; Gao et al., 2023; Namkoong et al., 2020 -- which serve as concrete examples to justify the effectiveness of the proposed methods, since ground-truth performance can be accessed (reported in Table 1). Moreover, **we had validated our method over a real-world intelligent education experiment as a result of testing efforts over years, with data collected over 1,000+ students**, besides the sepsis environment.

Reviewer comment (Experiments):
Please clarify what information may be available at the time of partitioning and if any why they cannot be used during training.

Author response:
Besides the regulatory concern following from our response to Q1 above, sub-grouping human participants is considered a non-trivial problem in general. Simply populating the features used for sub-grouping into the input of learning/decision models (but without actually considering the sub-groups as a conditional variable of the models) may fall short in HCSs, as many types of unobserved confounders can affect the outcome; one the other hand, it is challenging to capture these underlying factors explicitly as well [3, 4, 5]. Given both reasons, our work follows the setup that is practically feasible for most HCSs, followed by the approach addressing the problems of sub-grouping (Section 2.2) and policy selection for each sub-group (Section 2.4).

Reviewer comment (Writing):
In Problem 1, you say the initial state is given but in calculating $V^\pi$ take the expected value wrt $s_0$. What does it mean?

Author response:
Thank you for this question that helped us to improve the clarity of the problem statement. In Problem 1 we meant the initial states are **observable**, which are **variables** sampled from corresponding $\mathcal{S}_0(i')$ -- the term 'known' we originally used may lead to the impression that they can be reduced to constants, which are not our intentions. We have now updated Problem 1 -- please see the texts highlighted in blue there.

Reviewer comment (Theory):
I thought Assumptions 1 and 2 imply a one-to-one correspondence between participants and the distribution of initial state. However, a stronger assumption seems to be made on page 3 last paragraph.

Author response:
This is due to the setup we considered under HCSs where **one can only sample the initial state $s_0$ once from corresponding $\mathcal{S}_0$** -- the offline data available to use would only contain one trajectory collected from each participant (i.e., repeated sampling of $s_0$'s would be intractable in practice); this fact had been illustrated in detail in the paragraph under assumption 2. Combining it with assumptions 1 and 2, then comes the setup stated on page 3 last paragraph.

Reviewer comment (Theory):
Regarding objective (1), isn't the sum over the second term just the value for behavioral policy?

Author response:
The definition of $V^\beta_{K_m=k\big(s_0^{(i)}\big)}$ still follows from definition 1, but by replacing the target policy $\pi$ with the behavioral policy $\beta$. Conceptually, it's the value of the behavioral policy specific to sub-group $K_m$. Please note that in definition 1 we refer to $\pi$ as 'a policy', not specific to target policies. This is also aligned with our general setup -- the same policy may have different effects toward multiple sub-groups.

Reviewer comment (Novelty):
The motivation for the method seems weak. For example, one proposed problem is the time and cost to collect data, however the proposed method still requires trajectories to be collected a priori thus the time and cost of data collection is not removed.

Author response:
In both abstract and introduction we emphasized that **online testing** of policies obtained from offline RL training are costly. For example, it would be intractable to deploy all the policies trained with different hyper-parameter sets directly to participants in an intelligent tutoring experiment, since only a limited number of policies can be deployed in a semester. In contrast, **with OPS/OPE one could first estimate how the policy candidates would perform and only deploy the promising ones to online testing**, saving many iterations that would have been used on deploying the policies with sub-par performance. If our writing led to such a confusion, we would be more than happy to further clarify these parts in the manuscript.

Reviewer comment (Writing):
The terminology of "policy selection" instead of what the more common "policy evaluation" is a little confusing. There does not seem to be any reason why this method couldn't be referred to as an improved policy evaluation technique.

Author response:
In lieu of solely estimating (or ranking) the performance of the policy candidates $\pi\in\Pi$ (i.e., what typical OPE does), our work rather considers **how to assign the policy candidates that could potentially work best for each new participant who joins the cohort over time, with only the initial states being observable, as formulated in Problem 1 -- it is a practical problem currently preventing RL from facilitating real-world applications in human-centric systems (HCSs)**; this is illustrated by the intelligent tutoring experiment in Section 3.2. Consequently, we tend to believe that the term "off-policy selection (OPS)" fits better for the topic considered in our work. Moreover, as pointed out in the recent benchmark paper [2], both OPE and OPS are important toward closing the gap between offline RL training and evaluation.

Reviewer comment (Novelty):
How does this work relate to Konyushova, Ksenia, et al. "Active offline policy selection." Advances in Neural Information Processing Systems 34 (2021): 24631-24644.

Author response:
Our response here would be somewhat convoluted with the one above. Konyushova's work followed the typical OPS/OPE setup which targeted to choose the best possible policy using limited historical data, while the same agent is used during collection of historical experience and upon deployment, e.g., the offline dataset is collected from exactly the same type of robot to which the policy chosen by OPS will be deployed. In contrast, our work focuses on resolving the practical issue often encountered in HCSs -- each participant (e.g., student or patient) has a rather distinct background to some extent, so different criteria would be needed to evaluate how well the candidate policies would perform across participants, e.g., some students are visual/auditory learners while others may benefit more from hands-on experiences. This also serves as the motivation for sub-grouping the human participants, as its effectiveness has been demonstrated by other applications in healthcare [3], education [4, 5], etc.

Reviewer comment (Theory):
Why do the authors believe clustering improves performance? If the initial state is unique to participants wouldn't it be possible to learn a single policy that performs well across all states? Why do they think this doesn't happen? Is the policy class being trained on offline data not rich enough?

Author response:
These are very insightful questions highly related to the philosophy behind the motivation of this work as well as the design of the methodology. Besides the citations above showed that sub-grouping participants would be in general beneficial for HCSs, our experimental analyses (centered around Figure 2) did show that different types of students would benefit from specific types of candidate policies. Moreover, in reality it would be very challenging to collect sufficient data over all different types of participants, due to the nature of HCSs that each one has unique past records/backgrounds. For example, if we denote $\mathcal{S_{K_m}}$ the state space specific to the group $K_m$ (i.e., think of it as a specific type of agent in Mujoco, say HalfCheetah), then the overall state space of the HCS is as large as $(\mathcal{S_{K_m}})^K$, with $K$ being the total number of sub-groups (e.g., the total number of environments Mujoco has). Consequently, the complexity of obtaining a one-size-fit-all policy in HCSs would be analogous to training a single policy that can control any type of robot provided in Mujoco, Adroit, etc. However, we appreciate the reviewer's constructive thoughts to this end and we believe that this would be a very promising avenue for future works to follow up.

Reviewer comment (Novelty):
While the paper is quite convincing in its results on the education dataset, I'm not sure that ICLR is the best venue for these results. The methodology presented is less novel/interesting than the education dataset and results. This leads me to think that this work could be better suited for an education-based venue.

Author response:
Thank you for recognizing the impact of the real-world education experiment as a result of years of deployment, data collection, and analyses efforts. We would emphasize that the general goal of OPS/OPE is to bridge the gap between offline RL policy optimization and online testing in many **real-world** domains (as pointed out in [1, 2, 3, 4] etc.), due to safety and cost concerns, e.g., the IRB committee may require the policies to demonstrate sufficient performance before they can be deployed to students/patients. There **do exist a number of recent works published in relative venues, which also focus on empirically developing/improving intelligent education systems with OPS and RL techniques** that are published in relative venues, e.g., NeurIPS publications [5, 6, 7] from Brunskill's and Finn's groups, another recent NeurIPS paper [8], as well as AAMAS publication [9] from Levine and Brunskill. Consequently, we would tend to believe that ICLR would be a good fit for this empirical/application work, as it also has exceptional records of being inclusive for diversified research topics.

Reviewer comment (Experiments):
Consider a simple approach to the same problem: Cluster the students using some basic technique, then run any out-of-the-box RL method on each group independently. How would this compare to your results? It seems that there is a decently large sample size and not a crazy high number of subgroups.

Author response:
Thank you for sharing this very interesting idea that points out one of the future avenues for our work. Please note that our IRB, as in many experiments with human participants, requires comprehensive scrutiny over policies' theoretical ground as well as empirical performance before deployment, so we only have very limited options toward what types of policies can be deployed -- as a result, we were only able to use DQN due to it's straightforward methodology, and there exist many works that have shown it's efficacy in human-centered experiments [10,11,12] etc. This also serves as the main obstacle following your idea above, as directly deploying custom RL policies to students would be intractable. Moreover, please note that sub-grouping human participants is considered a non-trivial problem, with a number of works concurrently exploring this topic concurrently through different approaches over various applications [13, 14, 15] etc., and a main part of our work (Section 2.2) attempted to address this problem in the context of intelligent education. Again, we appreciate the reviewer's thoughtful comment and hope this idea can be further explored in the future.