Paper: A Multi-Agent Reinforcement Learning Framework for Evaluating the U.S. ‘Ending the HIV Epidemic’ initiative (ICLR-2024)

Reviewer comment (Writing):
In general, the paper's current writing has many public health terms. Thus, without much prior knowledge of HIV and public health terms, the paper is difficult to read.

Author response:
Thank you very much for raising this concern. We understand that the paper might be difficult to comprehend for the readers outside of the HIV and public health domain. We have added a terminology definition in the Appendix of the paper.

Reviewer comment (Experiments):
It is unclear whether a comparable single-agent baseline is used. According to Section 3.3.2, SARL formulation applies $j=1$, so SARL outputs one action every timestep whereas MARL outputs $N$ actions every timestep. If this is true, then MARL outperforming the SARL baseline could be a straightforward result (due to outputting more actions). Instead, a more competitive baseline could be a centralized agent that outputs $N$ actions instead of one action.

Author response:
The reviewer is correct that the main difference between MARL and SARL is that MARL identifies an optimal policy tailored to a jurisdiction while SARL identifies an aggregated national optimal policy. We now refer to SARL as aggregated-SARL ot A-SARL. Kindly note that we compared MARL with two different models (now referred to as A-SARL and Independent-SARL or I-SARL), each representing the two broad categories in current HIV modeling literature. To note, current models in the literature are either national i.e., evaluate one aggregated policy for the nation, or independent jurisdictional, i.e., do not model the mixing across jurisdictions but estimate actions specific to that jurisdiction. We formulated A-SARL to represent the former. To represent models of the latter type, we modeled I-SARL, where we evaluated one jurisdiction independently, i.e., evaluated optimal policy under assumption of no cross-jurisdictional interactions (no-partnership-mixing). Figure A.1 (a) compares results between MARL and I-SARL. Results suggest that discarding the effect of mixing leads to different policies with different impacts. Without loss of generality, we can conclude that if we do this analyses for each of the N jurisdictions, i.e., estimate optimal policy for each of the N independent jurisdictions, the resulting optimal policy will differ from that obtained from MARL. We believe the reviewer’s suggestion on having a centralized agent that output N actions can be considered representative of this type of analyses. 
We have rewritten the introduction and modified several parts of the paper to more clearly discuss this.

Reviewer comment (Experiments):
SOTA MARL applies centralized training and decentralized critic with a centralized critic. However, Section 3.3 applies multiple single-agent algorithms (i.e., multiple PPO) without the use of centralized critics.

Author response:
This is a great suggestion. We are trying to use a similar method for the extension of our work over all the jurisdiction in the US. In this paper, our aim was to evaluate the advantages of multi-agent approach compared to single agent, and thus, we used a simple case of multi-agent reinforcement learning.

Reviewer comment (Evaluation):
Results in Figures 2 and 3 need multiple seeds for statistical significance.

Author response:
We used multiple seeds in our training to show the convergence of the algorithm. Due to page limitations, we did not include those results from various random seeds. We will add a short description of this in the Appendix.

Reviewer comment (Theory):
Because multiple agents are interacting in the environment, MDP (Section 2) may not be the correct term, and a Markov game (Littman 1994) would be a more correct term to use.

Author response:
Though there are interactions across agents we assume a non-competitive and cooperative environment, as is the case in reality. Though Markov games is a broader term that can include these variations, we use the term MDP to avoid implications of competition between jurisdictions.

Reviewer comment (Experiments):
No validation is provided, which is important since this can affect real-world decision making.

Author response:
We agree that it is extremely important to have validation steps. Thus far we have taken precautions to extensively validate the environment (in prior work Tatapudi et. al., 2022]) and in data inputs. The data inputs are from well-established CDC’s survey and surveillance systems, with detailed cost functions informed through the literature. However, the data for the cost functions are limited when related to outreach and support programs, or as new technologies evolve (e.g., higher efficacy in HIV self-tests, which may have more uptake than clinic-based tests). Therefore, in future work, when applying this model to inform policy, we plan to do extensive sensitivity analyses. We also additionally plan to compare our model results with results from status-quo policy, under the range of uncertainty assumptions for all the sensitive input values, and under varying cost functions. This will help to evaluate if our model’s optimal policy would outperform the status-quo policy in all input assumptions or only in certain cases, which will help make informed decisions.

Reviewer comment (Novelty):
Is the difference between the single agent model and the multi-agent model only one of centralized versus decentralized decision making?

Author response:
Yes, that is correct. In the single-agent (now aggregated-SARL or A-SARL), one aggregated national-level action is selected and applied to all jurisdictions. In the multi-agent model (MARL), each agent is evaluating an action specific to its own jurisdiction, which makes it decentralized. We had a third scenario that was presented in the appendix, which we now refer to as independent-SARL or I-SARL, which represents evaluating each jurisdiction independently, i.e., ignoring the epidemic interactions across jurisdictions. A-SARL and I-SARL are representative of models that are used in current HIV literature.  
We have rewritten the introduction and modified several parts of the paper to more clearly discuss this.

Reviewer comment (Theory):
Are there situations where the rewards of one agent (jurisdiction) depends on the actions of other jurisdictions?

Author response:
Yes. The agents are acting in an environment with epidemic interactions across jurisdictions, i.e., sexual partnerships can form between persons of different jurisdictions. Thus, the actions taken in one jurisdiction will affect the infection rate of its neighboring jurisdictions, i.e., the state transitions and rewards of each jurisdiction are influenced by the actions taken by all jurisdictions. We made modifications to the methods to clarify these points.

Reviewer comment (Theory):
Is this multi-agent environment a fully cooperative one, in which case, this can be modelled as a Dec-POMDP.

Author response:
We assume it is fully cooperative as  in reality. In the model, while the current state is known to all agents and agents share a common objective, as each agent evaluates its own action and all agents simultaneously evaluate their actions, the evaluated state-transition will be different than actual state transition. Therefore, from an agent’s perspective, the environment presents properties of non-stationarity, making the formulation equivalent to Dec-POMDP in model-based methods. For purposes of scalability or in the model-free context as ours, MARL methods have been evaluated as approximate solution algorithms for Dec-POMDP [Hyun-Rok et. al. Operations Research, 2021; Shayegan, et al., PMLR, 2017].  
In this work, we focused on evaluating the significance of MARL instead of SARL as in current HV modeling literature. In future work, in application of the model to the U.S, we plan on evaluating multiple solution algorithms suitable for this context.

Reviewer comment (Theory):
Since the algorithm is run only for $T$ timesteps, shouldn't the author use a finite horizon MDP framework, where the timestep should also be added to the state?

Author response:
That is a great point that, generally, this application would require a non-stationary policy if modeling epidemic prevalence as the state, and including time-step ‘t’ into the state space can do that. In our case, though we did not prove this and is a limitation, the chance that the same state occurs at two different time-points is low, because of a few points related to our formulation and the application features as follows. Our state is formulated as a conjunction of prevalence (proportion infected) and care metrics (proportion aware, proportion VLS, and proportion PrEP). We set constraints on care metrics to be non-decreasing over-time. HIV has no cure, and therefore unless deaths exceed incidence, prevalence is non-decreasing. In the case that deaths are lower than incidence, i.e., prevalence is non-decreasing over time: If care metrics increases at a slower rate, prevalence increases at a faster rate, and thus, having the same care metrics at two time-points would have two difference prevalence, and similarly, having same prevalence at two time-points can only be generated from two different care metrics. In the case that deaths are lower than incidence, prevalence can decrease over time. However, as people with HIV can live almost normal lives, deaths can exceed incidence only when incidence is very low, which can happen only if care metrics are very high. Therefore, as before, the same prevalence cannot be generated by two different care metrics and vice-versa. However, without a formal proof, there is a chance that we may overlook certain scenarios that may lead to same state at two different time-points  (a limitation of our work), but we believe the chance of those scenarios are small. Adding a ‘t’ would increase complexity of training, so our current formulation is more desirable. Though adding care metrics into the state formulation also increased the complexity, it has several other useful properties (as shown in previous work [1]). However, with the availability of more time, we hope to conduct numerical analyses that include ‘t’ in the state-space to evaluate its significance in our current application.

Reviewer comment (Theory):
Are there any resource constraints across all jurisdictions that need to be modeled?

Author response:
Although every jurisdiction has different sources of funding, public and private, in economic modeling terms, we use a ‘societal perspective’, as is the typical case when evaluating national policies. Specifically, the national policy sets the guidelines, then the necessary resources are allocated through multiple funding sources.  
Here we assumed a fixed national budget, and evaluated how to optimally allocate it across jurisdictions. In application of the model to inform the EHE (Ending the HIV Epidemic), we will evaluate optimal policies under different budgets, to collective identify resource needs and optimal policy.

Reviewer comment (Evaluation):
Is the difference in the performance of the SARL and MARL policies a result of sub-optimality in learning or is it because of some other structural difference between the single and multi-agent environment?

Author response:
The environments are similar in both SARL (now  A-SARL) and MARL. The difference is that MARL identifies an optimal policy for ‘each’ jurisdiction while A-SARL identifies an ‘aggregated’ national optimal policy. Kindly note that we compared MARL with two different models (now A-SARL and I-SARL), each representing the two broad categories in current HIV modeling literature. Kindly note that the scope of this paper was to demonstrate the use of MARL to overcome current gaps in the HIV modeling literature.  
We have rewritten the introduction and modified parts of the paper to clarify this.

Reviewer comment (Experiments):
In such an environment, how does a baseline policy perform? Baseline policy could be the bestb policy from literature or if nothing is available for this environment, then a random policy could be used.

Author response:
The status quo policy acts as the baseline policy in our problem.

Reviewer comment (Theory):
In section 3.2.1 the proportion of PWH unaware of the infection is used as part of the state; is it possible in real life to know this proportion if people are not aware that they are infected? Does it affect the model if this number is unknown?

Author response:
That is correct that for infectious diseases, typically in real-life, proportion unaware are not always know. However, for HIV, the surveillance methods are well advanced that there exists good molecular methods that have been shown to provide accurate estimates. They are routinely estimated as part of CDC’s NHSS (National HIV/AIDS Surveillance Systems) and made publicly available annually. 
Previous work [Khatami, 2021] showed that this representation of state with the action representation of directly using changes in proportion unaware (instead of testing frequency, which is more implementable) has very good mathematical properties suitable for convergence. First, it removes the infeasible solutions, and second it makes the feasible space convex, thus assisting in faster convergence. Further, once we determine the optimal action (i.e., proportion change in unaware) they can be converted into testing frequency, i.e., given a state, a function mapping the action to testing frequency is a bijection function (one-to-one mapping).

Reviewer comment (Experiments):
According to Algorithm 1, each episode is T years long. This means that each time step in an episode is one year; this makes me question whether this is a reasonable choice, since it means that we take only one action per year. Is that enough? During a year many changes might happen and several governmental measures might be needed.

Author response:
That is correct that some infections, such as the COVID-19 pandemic, required constant policy changes to quickly adapt to the changing pandemic given: a) COVID-19 is a fast spreading disease (transmission rate per contact per day ~14%), b) it was a newly emerging infectious disease with no prior knowledge for setting appropriate intervention guidelines, and thus, frequent changes were needed, and c) we believe, it was feasible to implement frequent changes under the declaration of a global emergency. 
On the contrary, HIV is a slow spreading disease (transmission rate of 4% per year), and thus, the impact of frequent changes would not be feasible to evaluate as part of routine monitoring. The national strategic plans and guidelines are modified less frequently, e.g., the most recent changes in HIV testing guidelines were in 2017 and only for one population (MSM) [2]. Further, we believe, frequent changes in public health guidelines may not be feasible in the case of non-emergency situations given the time it takes to implement changes, such as US congressional appropriations, preparation of healthcare professionals and infrastructure, and changes in insurance policy. 
However, we chose a shorter timeline of 1 year to provide the flexibility to balance feasibility with need . For example, PrEP guidelines were updated in 2022 because a new drug was FDA approved in 2021. Estimating optimal annual rates provides the flexibility to set appropriate guidelines. For example, with recent advances in HIV take-home self-test kits, there is interest in suitable guidelines for combinations of clinic-based tests v. self-tests (weighted by effectiveness, adherence, etc.) to reach diagnostic rate targets to achieve the EHE (Ending the HIV Epidemic) goal. Our approach of identifying optimal annual diagnostic rate targets fits well in this context.

Reviewer comment (Theory):
It is unclear to me how MARL is being used here when compared to the SARL approach. From my understanding, in the SARL approach, there is a state (as per eq (1)) that describes the conditions of all the jurisdictions within a state (using the average, as stated in section 3.3.2), whereas in the MARL approach we have the same state but each jurisdiction has its own independent values; then, in SARL the actions performed influence the values in the state for all jurisdictions and then in MARL each jurisdiction performs an action that influences only its own state conditions. How are the multi-agent interactions between different jurisdictions integrated here? In order to say that this is following a MARL approach there needs to be some sort of interaction among the agents that will make them cooperate or work together towards some common goal. In the presented approach, I do not see that being done. Even the rewards are said to be given individually and its components only correspond to jurisdiction $j$ (as per eq (3)). This means that each agent is maximising an independent objective on its own, without any effect on the other agents. This sounds like multiple single-agent problems happening in parallel, without any interaction.

Author response:
That is correct that in the case of SARL, a single agent learns the actions for all the jurisdictions in it. But in the case of MARL, each jurisdiction (agent) learns its own actions. The interactions across jurisdictions in both MARL and SARL are in the epidemic environment, modeled as sexual partnership mixing across jurisdictions. 
In the setup of our MARL model, each agent is determining an optimal action based on the current state of the system (current state of each jurisdiction is known to all jurisdictions). However, as state transitions depend on the actions implemented in all jurisdictions (because of sexual partnership mixing), and as all agents (jurisdictions) are simultaneously evaluating this decision, it is infeasible to know the actual state transition. This introduces properties of non-stationarity into state-transitions, making this a suitable application for MARL.
We also evaluated a SARL (now called aggregated-SARL or A-SARL) formulation as a representation of current models in HIV literature. Specifically, the models in the HIV literature are typically ‘aggregated’ national or ‘independent’ jurisdictional (no epidemic mixing across jurisdictions). We use A-SARL as a representation of the former type. To represent models of the latter type, we evaluated one jurisdiction independently, i.e., ignored the mixing (presented in Appendix Figure A1). We now refer to this as independent-SARL or I-SARL. The corresponding results differed from that of MARL highlighting that ignoring mixing can lead to non-optimal decisions. 
The scope of this work was to present the use of MARL to overcome the gaps in current HIV modeling literature. We believe our contribution is developing a comprehensive modeling framework and simulation environment that is suitable for analyses of the U.S. ‘Ending the HIV Epidemic’ national strategic plan. 
We have rewritten the introduction and modified several parts of the paper to more clearly discuss this.