Paper: Deep Reinforcement Learning for Efficient and Fair Allocation of Health Care Resources (ICLR-2024)

Reviewer comment (Theory):
It's not clear what value this Q function is estimating. Further elaboration is required here.

Author response:
The Q function estimates the expected cumulative reward (Eq. (5)) when allocating ventilators under the constraint of the total available ventilators, taking into account the clinical conditions of the patients. This Q function estimates a trade-off between the survival rate of all patients and the fairness of allocation across different ethnoracial groups. To address computational challenges, we decomposed the overall $Q(s,a)$ into the sum of individual $Q_\theta(s)_{i,a_i}$ and used an attention mechanism to consider interactions among patients. We have included a clarification of the Q function in Section 4 of the revision.

Reviewer comment (Writing):
Minor typos: Eq. 3: $x'$ to $x'_i$.  Page 3: in 3 to in Fig. 3.

Author response:
We appreciate the reviewer's efforts in helping us with improving the manuscript and providing an opportunity to make clarifications. We have corrected the minor typo as suggested. Please find our point-by-point response below.

Reviewer comment (Presentation):
It would have been nice to see the limitations in the main text rather than the appendix. However, it is understandable due to the page limit.

Author response:
We thank the reviewers for their dedicated effort and valuable time spent on reviewing our manuscript, which has greatly contributed to enhancing the quality of our study. In the upcoming camera-ready version, we will incorporate a summary of the study's limitations to the main text. We will share the de-identified dataset upon publication of the paper.

Reviewer comment (Writing):
The notations are confusing in Section 3.3 and 3.4. Based on the context, are $x_i$ and $s_i$ the same (the medical condition of patient i), and $P^\text{on}$ and $P^\text{vent}$ the same (the ventilation transition)? If so, please ensure that the notations are consistent. Also, I think the action $a$ is N-dimensional and the i-th coordinate $a_i \in \{0, 1\}$ is the action applied on patient i, if this is the case, does $I'_i = a$ mean $I'_i = a_i$ in 1? Also, what is the dimension of the overall transition matrix $P$?

Author response:
We thank the reviewer for pointing out the notation inconsistency. We have addressed them in Section 3 of the revision. Specifically, the notation $x_i$ and $s_i$ are the same, we updated all notation of $s_i$ to $x_i$; The notation $P^\text{vent}$ and $P^\text{on}$ are different, meaning the transition of patient ventilation status and medical status, respectively; Typo of $a_i$ is corrected; The dimension of the transition matrix $P$ is $(|S|\times |A|) \times |S|$, where $|S| = Nk+N$ denotes the dimension of the state space, and $|A| = 2^N$ denotes the dimension of the action space. When considering fairness, $|S|$ expands to $Nk + N + 8$ by adding 8 dimensions to account for cumulative total and ventilated patients across four ethnoracial groups. We updated Eq. (5) for the correct formula of $P$.

Reviewer comment (Theory):
Currently, the state and action are the concatenations of the medical/ventilator states and the ventilator assignments of individual patients. Will there be scalability issue when the number of patients $N$ is really large and if there is, how to handle that with the current framework?

Author response:
We thank the reviewer for asking this important question. Naively concatenating all patients' state vectors together does indeed pose computational challenges as the size of the input is exponentially large. To overcome this computational intractability, we therefore proposed the use of a transformer structure that treats patients' states as 'tokens' and inputs them into the model rather than simply concatenating them. By employing this parametrization, we were able to decompose the transitions of patients' states, improving scalability while maintaining interactions among patients. Our approach has already substantially reduced the input complexity from $O(N \times 2^N)$ to $O(N)$. In our implementation, we utilized only one layer of the transformer structure with an embedding dimension of 1024 and 16 attention heads, resulting in just 5.3 million trainable parameters. Our model can recommend resource allocation for over 10,000 beds with a single 32 GB Tesla V100 GPU card. As a reference, the state of Illinois has 3,535 critical care beds [1].  

[1] UIC School of Public Health. (2023). Number of hospital beds in Illinois. Retrieved November 21, 2023.

Reviewer comment (Reproducibility):
Some details in the transition model are missing. How is $P^\text{vent} (x'_i | x_i)$ determined? From the description of the simulator in Section 5.2, it seems that you consider the factual data, e.g. $P(x'_i | x_i) = 1$ if $x'_i$ is the actual next state for patient i and $P(x'_i | x_i) = 0$ otherwise? Did you consider any counterfactuals? Will the results be negatively impacted by any potential selection bias in the data if only factual data is used? Also, how are $q_i(s)$ (bed assignment distribution) and $\xi$ (initial medical condition distribution) determined?

Author response:
We thank the reviewer for suggesting the need for clarification. In our project, we built a simulator to model individual patient's state path and the multi-patient ventilator allocation process. The simulator is built on real-world patients data. All patients in our real-world datasets were ventilated. Our simulator reflected scenarios where some patients cannot receive ventilators due to resource shortages. We assumed, following Bhavani et al. [1], that patients not allocated a ventilator do not survive (In our sensitivity analysis, see Appendix A.4, we also conducted additional experiments, exploring the possibility that patients not ventilated may continue to survive along their own trajectory with various probabilities). Thus we believe the simulator can provide a reasonable state-transition model for a random patient. Please note that our TDDQN method can work with any simulator of this kind, and building better simulators to approximate real-world hospital operation will be long-term efforts.  

We totally agree that having an accurate state-transition simulation model is crucial for success of deploying ML model in practice, and in particular counterfactuals could be a big concern. In this initial study, we tried our best to collect available data and present a proof-of-concept simulator+RL approach. In the longer term, we do envision this initial efforts would lead to transformative health care resource allocation in real healthcare systems. Thus, beyond the scope of this ICLR submission, we are actively collaborating with physicians and three other academic health systems, each with multiple hospitals to expand the efforts of equitable and effective health care resource allocation.

We clarified these two distributions $q_i(s)$  and  $\xi$  in the revised version of our manuscript, as follows. For the distribution of bed assignments, $q_i(s)$ depends on how new patients are allocated to empty beds. It satisfies $q_i(s) = 0$ if bed $i$ is already occupied, and $\sum_{i=1}^N q_i(s) =  E \sim Pois(\Lambda)$, assuming the number of incoming patients $E$ follows a Poisson distribution with parameter $\Lambda$.  In our implementation, we randomly sampled $E$ patients from the training cohort without replacement and randomly assigned them to the empty beds, where the number of patients $E$ is drawn from a Poisson distribution with $\Lambda=12$, as estimated from our training data. The initial medical condition distribution $\xi$ represents the clinical condition of patients on their first day in the critical care units.  

[1] Bhavani et al. "Simulation of ventilator allocation in critically ill patients with COVID-19." American journal of respiratory and critical care medicine 204.10 (2021): 1224-1227.

Reviewer comment (Experiments):
Some details in the Method and Experiments are missing. The learning objective is missing. From the context, I am assuming that you are using Q-learning with the proposed reward and then used the greedy policy with the learned Q function, but it's not clear from what's written now in Section 4. Also, the choice of the ventilator cost $c_1$ and the $\lambda$ to trade-off the fairness reward term are missing in the main paper (found them in appendix). Is there any justification on how they are determined? I am also wondering how the results will change if $c_1$ and $\lambda$ are chosen differently.

Author response:
We used the Q-learning objective with the proposed reward. We incorporated an action constraint to model the limited number of ventilators, when selecting the best action via the greedy policy in Eq.(10) and Eq.(11). We have clarified these details in Section 4 of the revision.  In practice, these parameters should be tuned and determined by doctors. In our experiment, we tested various candidate values of $\lambda$ to identify the best-performing one based on both effectiveness and fairness on the validation set. The $\lambda$ value has an impact on the balance between survival and fairness. In Appendix 2, we present a Pareto frontier obtained by varying the value of $\lambda$. We observe that the increase of $\lambda$ does not compromise the survival rate but enhances fairness until reaching a threshold. When $\lambda$ is set to a very large number, e.g. 1e6, the trained protocol tends towards a random policy, which means equal allocation but lowers the overall survival rate.

Reviewer comment (Reproducibility):
Will the dataset be made public if the paper is published? So that the results can be reproduced.

Author response:
We will share the de-identified dataset upon publication of the paper.

Reviewer comment (Writing):
There are some confusions in the writing, e.g.  
- Page 3, "... proposed model in 1" -> "... proposed model in Figure 1"?  
- Page 6: "... the action set $\mathcal{A}$ are defined in Eq. equation 2" -> "... the action set $\mathcal{A}$ are defined in Eq. 2"

Author response:
Thank you for pointing those confusions out. We have corrected these typos in the revised manuscript.