Original: Approaches to Optimizing Medical Treatment Policy using Temporal Causal Model-Based Simulation.
Revised: Approaches to Optimizing Medical Treatment Policy using Temporal Causal Model-Based Simulation

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: It is notoriously difficult to draw conclusions about the effects of medical interventions from observational data, where statistical confounding is rampant.
Revised: It is notoriously difficult to draw conclusions about the effects of medical interventions from observational data, where statistical confounding is rampant.

Original: Animportant example is "confounding by severity"Salas et al. [1999] in which sickerpatients receive more aggressive intervention, leading to a misleading positivecorrelation between stronger intervention and worsening outcome.
Revised: An important example is "confounding by severity" in which sicker patients receive more aggressive intervention, leading to a misleading positive correlation between stronger intervention and worsening outcome.

Original: This scenario isquite generally applicable because it represents negative feedback control, wheresome control mechanism responds to a change by affecting the change in theopposite direction.
Revised: This scenario is quite generally applicable because it represents negative feedback control, where some control mechanism responds to a change by affecting the change in the opposite direction.

Original: This leads to a causal loop: the change affects the feedbackand the feedback affects the change.
Revised: This leads to a causal loop: the change affects the feedback and the feedback affects the change.

Original: We employ the classic approach to breakingsuch loops by unrolling them in time, so that the disease severity before treatment is a separate node from the severity after treatment.
Revised: We employ the classic approach to breaking such loops by unrolling them in time, so that the disease severity before treatment is separated from the severity after treatment.

Original: Unrolling produces a datasetwhere the information about a patient is no longer contained on a single row ofa dataframe, but is spread over a set of rows representing timeslices.
Revised: Unrolling produces a dataset where the information about a patient is no longer contained on a single row of a dataframe, but is spread over a set of rows representing timeslices.

Original: We wantto base treatment decisions on the final outcome, which is only found at the endof this set of rows.
Revised: We want to base treatment decisions on the final outcome, which is only found at the end of this set of rows.

Original: Since we are interested in outcomes that occur at a futuretimeslice, we borrow a term from reinforcement learning and describe our type ofintervention as a "policy".
Revised: Since we are interested in outcomes that occur at a future timeslice, we borrow a term from reinforcement learning and describe our type of intervention as a "policy".

Original: Our challenge is to properly integrate temporal modelingwith causal modeling on observational data so that we can deconstruct these causalloops and reach useful analytical conclusions.
Revised: Our challenge is to properly integrate temporal modeling with causal modeling on observational data so that we can deconstruct these causal loops and reach useful analytical conclusions.

Original: Here we demonstrate a suitableanalytical approach with a simple toy problem, a drug dosing policy to treat thedisorder arising from infection with the fictitious pathogen Bogovirus .
Revised: Here we demonstrate a suitable analytical approach with a simple toy problem, a drug dosing policy to treat the disorder arising from infection with the fictitious pathogen Bogovirus .

Original: We beginby writing a simple bespoke simulation program to match a given causal graph;this generates a simulated dataset where we know the ground-truth about causalinteractions.
Revised: We begin by writing a simple bespoke simulation program to match a given causal graph; this generates a simulated dataset where we know the ground-truth about causal interactions.

Original: Using the known correct influence graph, together with other aspectsof "domain knowledge", we build causal model-based simulations of the simulateddata ("simsim" models) that let us estimate the expected effects of various treatmentpolicies on ultimate outcomes.
Revised: Using the known correct influence graph, together with other aspects of "domain knowledge", we build causal model-based simulations of the simulated data ("simsim" models) that let us estimate the expected effects of various treatment policies on ultimate outcomes.

Original: We compare this approach to the closely-relate fieldof reinforcement learning, and show how they are complementary
Revised: We compare this approach to the closely-relate field of reinforcement learning, and show how they are complementary

Original:  Introduction
Revised:  Introduction

Original: 1.1 Why apply causal models to synthetic data?

Original: This paper demonstrates how temporal causal models can be used as the basis of simulations thatmake it possible to optimize treatment policies in virtual experiments.
Revised: This paper demonstrates how temporal causal models can be used as the basis of simulations that make it possible to optimize treatment policies in virtual experiments.

Original: We demonstrate the approachon data that has itself been generated by a simulation, so that we know the ’ground truth’ causalrelationships.
Revised: We show the approach on data that has itself been generated by a simulation, so that we know the ’ground truth’ causal relationships.

Original: Our ultimate goal is to develop better treatments by applying this approach to historicalpatient data. 
Revised: Our ultimate goal is to optimize treatment protocols by applying this approach to patient data. infection drug cum [Figure] outcome severity

Original: BogoVirus: A Simulated Dataset2.1 Components of a medical simulation
Revised: [Figure] toxicity efficacy infection [Figure] prev drug cum

Revised: [Figure]

Revised: outcome severity toxicity efficacy infection drug cum_drug outcome severity_next toxicity efficacy

Revised: [Figure]

Revised:  BogoVirus: A Simulated Dataset

Original: A common problem in causal modeling of medical disorders is the presence of feedback loops.
Revised: A common problem in causal modeling of medical disorders is the presence of feedback loops.

Original: Adrug is given to manage a disorder, and the disorder responds to the drug.
Revised: A drug is given to manage a disorder, and the disorder responds to the drug; this circular feedback leads to "confounding by indication" ( Salas et al. [1999]), or its continuous-domain analog, ’confounding by severity".

Original: The traditional way tobreak such causal influence cycles is to unroll them in time; the severity of the disorder in the currenttimestep is the basis for administering the drug, and the response to the drug is reflected in the severityof the disorder in the next timestep.
Revised: The traditional way to break such causal influence cycles is to unroll them in time; the severity of the disorder in the current timestep is the basis for administering the drug, and the response to the drug is reflected in the severity of the disorder in the next timestep.

Revised: 2.1 Primary simulation

Original: Our model includes an underlying condition (infection with the imaginary organism Bogovirus ); thiscondition will resolve over time as long as the patient does not die.
Revised: Our model includes an underlying condition (infection with the imaginary organism Bogovirus ); this condition will resolve over time as long as the patient does not die.

Original: Infection leads to a disorder(it could be something like a problem with blood clotting, breathing, or circulation) that must bemanaged because if it gets too severe it can lead to death.
Revised: Infection leads to a disorder (it could be something like a problem with blood clotting, breathing, or circulation) that must be managed because if it gets too severe it can lead to death.

Original: The disorder is managed by a drug whichhas a direct effect on the severity of the disorder, but becomes less effective as the level of the drugaccumulates in the body.
Revised: The disorder is managed by a drug which has a direct effect on the severity of the disorder, but becomes less effective as the level of the drug accumulates in the body.

Original: High cumulative levels of the drug can lead to fatal toxicity.
Revised: High cumulative levels of the drug can lead to fatal toxicity.

Original: The ultimateoutcome is either recovery or death, but on most days neither of these things happens, and the patientjust stays in the hospital for another day.
Revised: The ultimate outcome is either recovery or death, but on most days neither of these things happens, and the patient just stays in the hospital for another day.

Revised: We use simulation to generate data for which we know the true causal graph.

Revised: This is done by writing a simple Python program that implements the causal graph shown in Figure 1; each node is represented by a function whose inputs are dictated by the edges leading into the node, and this set of functions are called in the order shown in the DAG to compute a row of variable values for each day of the simulated patient’s illness.

Revised: For each patient, this process continues until the infection process completes (leading to a "recover" outcome), or the patient succumbs to either the disorder or the drug toxicity ("die" outcome).

Revised: Through trial and error, we adjusted the simulation parameters so that a simple scan of dose (see

Revised: Figure 3) will show an optimum, but the peak performance is less than 100% survival, so it leaves some room for improvement.

Original: These are the variables in the simulation:  infection : records how far the patient has passed throughthe course of the infection (this is basically a counter for percent progress; once it passes 100 and youhave not died, you recover).
Revised: These are the variables in the simulation: • infection : records how far the patient has passed through the course of the infection (this is basically a counter for percent progress; once it passes 100 and you have not died, you recover).

Original:  drug : a dose of the treatment.
Revised: • drug : a dose of the treatment.

Original: This is an adjustable quantity.
Revised: This is an adjustable quantity.

Original:  cum_drug : The accumulated dose of the treatment drug.
Revised: • cum_drug : The accumulated dose of the treatment drug.

Original: This is an exponential  average, and is subject to decayover time if treatment doses are not administered.
Revised: This is an exponential moving average, and is subject to decay over time if treatment doses are not administered.

Revised: • efficacy : The extent to which the drug dose affects severity

Revised: ; the drug becomes less effective at higher cumulative doses.

Original:  severity : Quantified level of severity of the disordercaused by the infection.
Revised: • severity : Quantified level of severity of the disorder caused by the infection.

Original: The worse this gets, the more likely the patient is to die.
Revised: The worse this gets, the more likely the patient is to die.

Original:  outcome : if severitygets high enough, the patient’s chances of death increase.
Revised: • outcome : if severity gets high enough, the patient’s chances of death increase.

Original: If the infection runs its course and thepatient does not die, they recover.
Revised: If the infection runs its course and the patient does not die, they recover.

Original:  toxicity : a function of cum_drug that is much more pronounced athigh cumulative dose.
Revised: • toxicity : a function of cum_drug that is much more pronounced at high cumulative dose.

Original: High toxicity leads to increased probability of death.
Revised: High toxicity leads to increased probability of death.

Original: Variables with suffixes’_prev’ or ’_next’ hold lagging or leading values from adjacent timeslices.
Revised: Variables with suffixes ’ _prev ’ or ’ _next ’ hold lagging or leading values from adjacent timeslices.

Original: Through trial and error, we adjusted the simulation parameters so that a simple scan of dose (see

Original: Figure 3) will show an optimum, but the peak performance is less than 100% survival, so it stillleaves some room for improvement.

Original: 2.2 The simulated dataset
Revised: 2.3 Simulated datasets

Revised: To simulate a randomized controlled trial (RCT) of drug dose, patients were assigned to cohorts, and each cohort received a fixed dose of the drug on each day of treatment.

Original: Figure 2 shows simulated data for a single patient episode.
Revised: Figure 2 shows simulated data for a single patient episode.

Original: There is one column for each node in themodel, plus housekeeping attributes patient_id , and day_number (how many days that patient hasbeen in the hospital).
Revised: There is one column for each node in the model, plus housekeeping attributes patient_id , and day_number (how many days that patient has been in the hospital).

Original: The cohort column indicates the group into which the patient was randomizedin the simulated RCT; this determines the dose of drug the patient receives.
Revised: The cohort column indicates the group into which the patient was randomized in the simulated RCT; this determines the dose of drug the patient receives.

Revised: To simulate observational data, we use the primary simulator to generate another dataset in which a random dose of drug is given to each patient each day.

Revised: This randomized policy dataset is the input to the subsequent modelling tasks.

Original: 2.3 Finding the Optimal Dose
Revised: 2.4 Finding the optimal dose

Original:  Figure 3 shows a scan of different doses of drug run in the primary simulation.
Revised: The solid red curve in Figure 3 shows survival at different doses of drug from the RCT dataset.

Original: This is a simulatedexperiment, where patients are assigned to cohorts, and each cohort receives a fixed dose of the drug.

Original: The y-axis shows the fraction of each cohort surviving the Bogovirus infection.

Original: The optimal dose(0.7 units) gives us a standard to compare with results we obtain from analytical simulations.
Revised: The optimal dose (0.7 units) gives us a standard to compare with results we obtain from analytical simulations.

Original: To simulate observational data, we use the primary simulator to generate another dataset (not thecohort-dose dataset used in Figure 3), in which a random dose of  drug is given to each patient eachday.
Revised: Our goal in subsequent sections is to use analytical ’simsim’ simulations to approximate this solid red curve without requiring an RCT. described obtained trained almost uni formly fatal is fatal the giving This CPT is a three-dimensional table where each cell holds a probability, and the coordinates of the cells are the categorical values of the inputs ( drug and cum_drug ) and the output ( efficacy ). Note that the extreme ranges of cum_drug are not represented in the observational data, and the model assumes a uniform prior.

Original: This randomized policy dataset is input to the subsequent modelling tasks.
Revised: These areas require editing to incorporate domain expertise for extrapolation.

Original:  "SimSim-BN": An Analytical Simulation Built on a Causal BayesNetModel
Revised:  "SimSim-BN": An Analytical Simulation Built on a Causal BayesNet Model

Original: A Bayes network whose parameters are learned from the randomized dosage dataset simulatesbuilding a model from observational data. This uses a different simulated dataset from that used togenerate the optimal policy inFigure 3: For that run—shown by the solid line—a fixed dose wasassigned to each patient, as in a randomized controlled trial. We  use this  to find the dosage policythat gives the best results by which we can compare the Bayes network model simulation, along withthe “SimSim-DECI” model and the Offline Reinforcement model, as shown in Figure
Revised: Rather than testing a treatment policy in the primary simulation (analogous to an experiment), we can use a causal model to build a simulation (since this is a simulation of a simulation we call this a ’simsim’ model, of which we present three versions). We then use this causal simulation to perform the optimization scan.

Original: This CPT is a three-dimensional table where each cell holds a probability, and the coordinates of the cells are the categorical values of theinputs ( drug and cum_drug ) and the output ( efficacy ). This node is not shown in Figure 1; it was added to the causal graph to make it easier to fit the model, as explained in the text. 3.1 Modifying the graph to facilitate training
Revised: The simsim model is trained on the simulated "observational" dataset where each patient was given a random dose of the drug each day (as opposed to the dose-optimizing simulation where patients were assigned to cohorts that each received the same dose each day). This simulates the process of building a causal model from non-experimental data and using it to estimate an optimal treatment policy without actually running an experiment.

Original: In construction of the Bayes network, multiple high-cardinality inputs to a node lead to a curse ofdimensionality problem. In some cases we can apply domain knowledge to break these complicatednodes into a set of smaller, more easily learned nodes.
Revised: To simulate building a model from observational data, we used the known causal graph from Figureto build a Bayes network whose parameters were learned from the randomized dosage dataset.

Original: The astute reader may notice that the influence diagram in Figure 1 does not contain a node forefficacy ; it has four values, drug , cum_drug , infection and severity flowing directly intoseverity_next . Though we were able implement the primary simulation using all four inputs in asingle node, having this many inputs into a single node in the Bayes Net led to sparsity problems. Ifeach variable is discretized into N levels, 4 inputs require N 4 cells in the CPT. So we once again drewupon our simulated domain knowledge; because drug and cum_drug interact with each other, butnot directly with severity or infection , we were able to add a node to capture this interaction.
Revised: 3.1 Extrapolating to areas not covered by the training data

Original: Figure 4 shows the CPT for efficacy for three selected discrete values of drug .
Revised: Figure 4 shows the CPT for efficacy for three selected discrete values of drug .

Original: We can see thatthese tables have captured the relationship that the efficacy of a given dose of the drug decreases athigher levels of cum_drug .
Revised: We can see that these tables have captured the relationship that the efficacy of a given dose of the drug decreases at higher levels of cum_drug .

Original: However, the very high or very low values of cum_drug (the extremes ofthe x -axis in each panel) show a uniform probability across the levels of efficacy .
Revised: However, the very high or very low values of cum_drug (the extremes of the x -axis in each panel) show a uniform probability across the levels of efficacy .

Original: This is becausethese values of cum_drug are not represented in the primary simulated data from which this Bayes
Revised: This is because these values of cum_drug are not represented in the random-dose dataset from which this Bayes

Original: Net was trained, and it defaults to using a uniform prior probability in these ranges.
Revised: Net was trained (that would require a consistently high or consistently low dose, which is unlikely in the random-dose dataset), and it defaults to using a uniform prior probability in these ranges.

Original: We therefore toadvantage of our domain knowledge to modify the CPTs to extrapolate better to extreme values.
Revised: We therefore took advantage of our domain knowledge to modify the CPTs to extrapolate better to extreme values.

Original: Thiswas done by simply filling out the extreme columns  with copies of the closest non-uniform column.
Revised: This was done by simply filling out the uniform-probability columns at the extremes with copies of the closest non-uniform column.

Original: We also simplified the input infection , which is basically a counter that keeps track of how far thepatient has progressed through the course of infection, into the binary variable infection_overthat is true if infection reaches 100 or more, and false otherwise.
Revised: We also simplified the input infection , which is basically a counter that keeps track of how far the patient has progressed through the course of infection, into the binary variable infection_over that is true if infection reaches 100 or more, and false otherwise.

Original: With these modifications, wefeed only three inputs ( severity , efficacy , and a binary variable for infection_over ) intoseverity_next .

Original: This greatly simplifies the probability table.
Revised: This greatly simplifies the probability table for severity_next .

Original: 3.2 Using the causal model to re-simulate the data

Original: Rather than testing a treatment policy in the simulation (analogous to an experiment), we can use acausal model to build a simulation (since this is a simulation of a simulation we call this a ’simsim’model, of which we present three versions).

Original: We then use this causal simulation to perform theoptimization scan.

Original: The simsim model is trained on a simulated dataset where each patient was givena random dose of the drug each day (as opposed to the dose-optimizing simulation where patientswere assigned to cohorts that each received the same dose each day).

Original: This simulates the process ofbuilding a causal model from non-experimental data and using it to estimate an optimal treatmentpolicy without actually running an experiment.

Original:  "SimSim-DECI": Expert-in-the-loop Causal Inference for Intervention 
Revised:  "SimSim-DECI": Expert-in-the-loop Causal Inference for Intervention Optimization

Original: Optimization

Original: Deep End-to-end Causal Inference (DECI) Geffner et al.
Revised: Deep End-to-end Causal Inference ("DECI", Geffner et al.

Original: [2022] is an autoregressive-flow basednon-linear additive noise structural equation model (SEM), which is designed to perform both causaldiscovery and inference, including average treatment effect (ATE) and conditional average treatmenteffect (CATE) estimation.
Revised: [2022]) 1 is an autoregressive-flow based non-linear additive noise structural equation model (SEM), which is designed to perform both causal discovery and inference, including average treatment effect (ATE) and conditional average treatment effect (CATE) estimation.

Original: DECI discovery takes prior knowledge of the graph structure as input,defined by a constraint matrix, and uses the data-driven causal calculation to estimate the graphstructure, and ATE between nodes.
Revised: DECI discovery takes prior knowledge of the graph structure as input, defined by a constraint matrix, and uses the data-driven causal calculation to estimate the graph structure and ATE between nodes.

Original: 4.1  DECI for Timeseries
Revised: 4.1 Using DECI for Timeseries: A Featurization Method

Original: SimSim-DECI uses the DECI model to simulate temporal records for the patients under variousdrug-dose policies and finds the policy for the optimum dosage level.
Revised: SimSim-DECI uses the DECI model to simulate temporal records for the patients under various drug-dose policies and finds the policy for the optimum dosage level.

Original: As DECI is primarily designedto digest generic types of tabular Dataset.
Revised: DECI is primarily designed to digest generic types of tabular Dataset.

Original: In order to capture the temporal dynamics of our data, wedesigned our framework as follows:
Revised: To capture the temporal dynamics of our data, we designed our framework as follows:

Original: • Temporal Sliding-Window featurization : each row contains the information of the current time-step and the previous one as in ( X t − 1 , X t ) .
Revised: • Temporal Sliding-Window featurization : each row contains the information of the current time-step and the previous one as in ( X t − 1 , X t ) .

Original: Figure 5 shows what
Revised: Figure 5 shows what

Original: re-framed dataset looks like.
Revised: re-framed dataset looks like.

Original: • Temporal constraints : Features in the current steps cannot cause features in the previous step.
Revised: • Temporal constraints : Features in the current steps cannot cause features in the previous step.

Original: • Temporal training : The DECI model is trained on each row as an independent sample.
Revised: • Temporal training : The DECI model is trained on each row as an independent sample.

Original: Sinceeach row holds the past and current features, the connection between features of the same day is learned as well as features of two consecutive days.
Revised: Since each row holds the past and current features, the connection between features of the same day is learned as well as features of two consecutive days.

Original: • Temporal simulation : We simulate the information of each step by conditioning the simulation on the information from the previous step.
Revised: • Temporal simulation : We simulate the information of each step by conditioning the simulation on the information from the previous step.

Original: That means for data in Figure 5, we take thevalues of the seven right columns as the condition for the intervention and predict the next step by sampling the trained model. 
Revised: That means for data in Figure 5, we take the values of the seven right columns as the condition for the intervention and predict the next step by sampling the trained model. [Figure]

Original: 4.2 Causal Discovery
Revised: 4.2 Causal Discovery

Original: DECI recovers the structure of the underlying causal graph that we can compare with the trueunderlying graph.
Revised: DECI recovers the structure of the underlying causal graph that we can compare with the true underlying graph.

Original: Thus, we can be the expert-in-the-loop to modify the data-driven findings toimprove its correspondence to the truth.
Revised: Thus, we can be the expert-in-the-loop to modify the data-driven findings to improve its correspondence to the truth.

Original: We do this by re-training with additional constraints.
Revised: We do this by re-training with additional constraints.

Original: Figure6 shows the discovered unconstrained graph compared to the final graph with expert-in-the-loopconstraints applied.
Revised: Figure 6 shows the discovered unconstrained graph compared to the final graph with expert-in-the-loop constraints applied.

Original: Note that the domain expert prevented confounding by indication Kyriacou and
Revised: Note that the domain expert prevented confounding by indication (Kyriacou and

Original: Lewis [2016] by reversing the direction of drug-to-severity causality.
Revised: Lewis [2016]) by reversing the direction of drug-to-severity causality.

Original: 4.3 Simulation-based Policy Optimization
Revised: 4.3 Simulation-based Policy Optimization

Original: The trained model is a structural causal model (SCM) that captures the functional relations and errordistributions.
Revised: The trained model is a structural causal model (SCM) that captures the functional relations and error distributions.

Original: Hence, it is capable of estimating the expected updates following an intervention.
Revised: Hence, it is capable of estimating the expected updates following an intervention.

Original: Weuse this capability to synthesize the current day, having information about the previous day.
Revised: We use this capability to synthesize the current day, having information about the previous day.

Original: Weadd the desired drug dose based on the candidate policies, explained in previous sections, to theinterventions.
Revised: We add the desired drug dose based on the candidate policies, explained in previous sections, to the interventions.

Original: Tomake reduce estimation error, we take  average  50 samples for the continuousvariables and take mod for the binary ones.
Revised: To reduce estimation error, we take the average of 50 samples for the continuous variables and take the mode for the binary ones.

Original: The simulation for each patient will stop if “die” or“recove” returns True.
Revised: The simulation for each patient will stop if “die” or “recover” returns True.

Original: Figure 3 demonstrates the results of different dose policies.
Revised: Figure 3 demonstrates the results of different dose policies.

Original: The model capturesthe optimum drug dose correctly, although is less accurate at extremes where the observational datais sparse. [Figure]
Revised: The model captures the optimum drug dose correctly, although is less accurate at extremes where the observational data is sparse (see the "simsim-DECI" curve in Figure 3).

Original:  Offline Reinforcement Learning: The Third Simulation Reconstruction
Revised:  Offline Reinforcement Learning: The Third Simulation Reconstruction

Original: For purposes of comparison we took the same randomized policy simulated observation datasetand used a comparable offline reinforcement learning (RL) method to derive a constant-dose policycomparable to the previous simulations.
Revised: For purposes of comparison we took the same randomized policy simulated observational dataset and used a comparable offline reinforcement learning (RL) method to derive a constant-dose policy comparable to the previous simulations.

Original: Since we used a constant policy, the method is degenerate,since the policy function does not depend on the current state, and might equivalently be consideredjust a simple Markov process.
Revised: Because we used a constant policy the method is degenerate, since the policy function does not depend on the current state, and might equivalently be considered just a simple Markov process.

Original: For the RL environment, we wrapped the simulation dataset with OpenAI Gym [Brockman et al.[2016]], to simulate the stage transition probability distribution.
Revised: For the RL environment, we wrapped the simulation dataset with OpenAI Gym (Brockman et al. [2016]), to simulate the stage transition probability distribution.

Original: Unlike the previous “sim-sim”reconstructions, state variables are continuous-valued, so to generate a random state value given thecurrent state, we used a k-nearest neighbor regressor built on the faiss [Johnson et al.
Revised: Unlike the previous “simsim” reconstructions, state variables are continuous-valued, so to generate a random state value given the current state, we used a k-nearest neighbor regressor built on the faiss (Johnson et al.

Original: [2019]] libraryfor similarity search.
Revised: [2019]) library for similarity search.

Original: Since, even with the simulation dataset of 1 M rows, there is no likelihood that alookup will find a record within some epsilon of the current state, so instead we average the predictednext-stage values of the k = 3 nearest neighbours.
Revised: Since, even with the simulation dataset of 1 M rows, there is no likelihood that a lookup will find a record within some epsilon of the current state, so instead we average the predicted next-stage values of the k = 3 nearest neighbours.

Original: Equivalently the prediction function could havebeen implemented as a neural-network trained on the simulation dataset.
Revised: Equivalently the prediction function could have been implemented as a neural-network trained on the simulation dataset.

Original: Either way, predictionsare confined to the space that has been explored in the simulation data, with all the limitations thisimplies for an offline-algorithm.
Revised: Either way, predictions are confined to the space that has been explored in the simulation data, with all the limitations this implies for an offline algorithm.

Original: We ran a constant policy in dose increments of 0 .
Revised: We ran a constant policy in dose increments of 0 .

Original: 1 from 0 to 1 . 4 , for 100 episodes, which in almostall cases terminated in less than 20 stages, then averaged the termination counts to estimate survivalrates, as shown in Figure 3.
Revised: 1 from 0 to 1 . 4 , for 100 episodes, which in almost all cases terminated in less than 20 stages, then averaged the termination counts to estimate survival rates, as shown in Figure 3.

Original: Admittedly using a constant policy is a trivial algorithm, and cannotdo better than an adaptive policy.
Revised: Admittedly using a constant policy is a trivial algorithm, and cannot do better than an adaptive policy.

Original: But for purposes of comparison with the previous "sim-sims", itillustrates the consequences of the lack  causal analysis.
Revised: But for purposes of comparison with the previous "sim-sims", it illustrates the consequences of the lack of causal analysis.

Original: As the figure shows, the offline-RL finds thesame optimum dose, but over-estimates the survival rates at extremes.
Revised: As the figure shows, the offline-RL finds the same optimum dose, but over-estimates the survival rates at extremes.

Original: Most notably it believes highdoses that should accumulate toxic cumulative effects are not deleterious.
Revised: Most notably it believes high doses that should accumulate toxic cumulative effects are not deleterious.

Original: Inspection of predictionsas revealed in the episode traces show that, at extreme values, the offline predictor is wildly off withpredicting accumulation of the drug over time.
Revised: Inspection of predictions as revealed in the episode traces show that, at extreme values, the offline predictor is wildly off with predicting accumulation of the drug over time.

Original: This highlights a known limitation of offline policyoptimization, where even in modest dimension state-spaces the ability to explore new policy areas isseverely limited by the existing data. Levine et al.
Revised: This highlights a known limitation of offline policy optimization, where even in modest dimension state-spaces the ability to explore new policy areas is severely limited by the existing data (Levine et al.

Original:  Discussion: Where do we go from here?
Revised:  Discussion: Where do we go from here?

Original: We plan to collaborate with medical researchers in our next iteration of this simulation and modelingprocess to focus on an actual disease and therapies, to reproduce the analytical logic in a more realisticsetting.
Revised: We plan to collaborate with medical researchers in our next iteration of this simulation and modeling process to focus on an actual disease and therapies, to reproduce the analytical logic in a more realistic setting.

Original: We have experience implementing a similar forward-inference Bayes Net in the Syntheaelectronic medical record simulator [HHS2021]; porting our next version to that platform would letus develop shareable feature engineering exercises to conduct the kind of simulation-based analysisdescribed here on data in a realistic schema.
Revised: We have experience implementing a similar forward-inference Bayes Net in the Synthea electronic medical record simulator (Walonoski et al. [2017]) 2 ; porting our next version to that platform would let us develop shareable feature engineering exercises to conduct the kind of simulation-based analysis described here on data in a realistic schema.

Revised: The DECI team is extending their work to support temporal use cases(Gong et al.

Revised: and we plan to evaluate our simulations in that tool.

Original: Causal graphs provide an opportunity to capture domain expertise; this is a modeling process that, aswe have shown, blends gracefully into simulation modeling.
Revised: Causal graphs provide an opportunity to capture domain expertise; this is a modeling process that, as we have shown, blends gracefully into simulation modeling.

Revised: The code for this project is in Github 3 .