Original: Approaches to Optimizing Medical Treatment Policy using Temporal Causal Model-Based Simulation
Revised: Approaches to Optimizing Medical Treatment Policy using Temporal Causal Model-Based Simulation

Original: Abstract
Revised: Abstract

Original: It is notoriously difficult to draw conclusions about the effects of medical interventions from observational data, where statistical confounding is rampant.
Revised: It is notoriously difficult to draw conclusions about the effects of medical interventions from observational data, where statistical confounding is rampant.

Original: An important example is "confounding by severity" in which sicker patients receive more aggressive intervention, leading to a misleading positive correlation between stronger intervention and worsening outcome.
Revised: An important example is "confounding by severity" in which sicker patients receive more aggressive intervention, leading to a misleading positive correlation between stronger intervention and worsening outcome.

Original: This scenario is quite generally applicable because it represents negative feedback control, where some control mechanism responds to a change by affecting the change in the opposite direction.
Revised: This scenario is quite generally applicable because it represents negative feedback control, where some control mechanism responds to a change by affecting the change in the opposite direction.

Original: This leads to a causal loop: the change affects the feedback and the feedback affects the change.
Revised: This leads to a causal loop: the change affects the feedback and the feedback affects the change.

Original: We employ the classic approach to breaking such loops by unrolling them in time, so that the disease severity before treatment is separated from the severity after treatment.
Revised: We employ the classic approach to breaking such loops by unrolling them in time, so that the disease severity before treatment is separated from the severity after treatment.

Original: Unrolling produces a dataset where the information about a patient is no longer contained on a single row of a dataframe, but is spread over a set of rows representing timeslices.
Revised: Unrolling produces a dataset where the information about a patient is no longer contained on a single row of a dataframe, but is spread over a set of rows representing timeslices.

Original: We want to base treatment decisions on the final outcome, which is only found at the end of this set of rows.
Revised: We want to base treatment decisions on the final outcome, which is only found at the end of this set of rows.

Original: Since we are interested in outcomes that occur at a future timeslice, we borrow a term from reinforcement learning and describe our type of intervention as a "policy".
Revised: Since we are interested in outcomes that occur at a future timeslice, we borrow a term from reinforcement learning and describe our type of intervention as a "policy".

Original: Our challenge is to properly integrate temporal modeling with causal modeling on observational data so that we can deconstruct these causal loops and reach useful analytical conclusions.
Revised: Our challenge is to properly integrate temporal modeling with causal modeling on observational data so that we can deconstruct these causal loops and reach useful analytical conclusions.

Original: Here we demonstrate a suitable analytical approach with a simple toy problem, a drug dosing policy to treat the disorder arising from infection with the fictitious pathogen Bogovirus .
Revised: Here we demonstrate a suitable analytical approach with a simple toy problem, a drug dosing policy to treat the disorder arising from infection with the fictitious pathogen Bogovirus .

Original: We begin by writing a simple bespoke simulation program to match a given causal graph; this generates a simulated dataset where we know the ground-truth about causal interactions.
Revised: We begin by writing a simple bespoke simulation program to match a given causal graph; this generates a simulated dataset where we know the ground-truth about causal interactions.

Original: Using the known correct influence graph, together with other aspects of "domain knowledge", we build causal model-based simulations of the simulated data ("simsim" models) that let us estimate the expected effects of various treatment policies on ultimate outcomes.
Revised: Using the known correct influence graph, together with other aspects of "domain knowledge", we build causal model-based simulations of the simulated data ("simsim" models) that let us estimate the expected effects of various treatment policies on ultimate outcomes.

Original: We compare this approach to the closely-relate field of reinforcement learning, and show how they are complementary
Revised: We compare this approach to the closely-relate field of reinforcement learning, and show how they are complementary

Original:  Introduction
Revised:  Introduction

Original: This paper demonstrates how temporal causal models can be used as the basis of simulations that make it possible to optimize treatment policies in virtual experiments.
Revised: This paper demonstrates how temporal causal models can be used as the basis of simulations that make it possible to optimize treatment policies in virtual experiments.

Original: We show the approach on data that has itself been generated by a simulation, so that we know the ’ground truth’ causal relationships.
Revised: We show the approach on data that has itself been generated by a simulation, so that we know the ’ground truth’ causal relationships.

Original: Our ultimate goal is to optimize treatment protocols by applying this approach to patient data.

Original: infection drug cum

Original: [Figure]
Revised: [Figure]

Original: outcome severity
Revised: efficacy

Original: [Figure]
Revised: [Figure]

Original: toxicity efficacy infection
Revised: efficacy infection

Original: [Figure]
Revised: [Figure]

Original: prev drug cum
Revised: outcome severity

Original: [Figure]
Revised: [Figure]

Original: outcome severity toxicity efficacy infection drug cum_drug outcome severity_next toxicity efficacy
Revised: toxicity efficacy

Original: [Figure]
Revised: [Figure]

Revised: Our ultimate goal is to optimize patient treatment protocols using simulations based on causal models derived from a combination of learning from patient data and incorporating domain knowledge.

Original:  BogoVirus: A Simulated Dataset
Revised:  BogoVirus: A Simulated Dataset

Original: A common problem in causal modeling of medical disorders is the presence of feedback loops.
Revised: A common problem in causal modeling of medical disorders is the presence of feedback loops.

Original: A drug is given to manage a disorder, and the disorder responds to the drug; this circular feedback leads to "confounding by indication" ( Salas et al.
Revised: A drug is given to manage a disorder, and the disorder responds to the drug; this circular feedback leads to "confounding by indication" ( Salas et al.

Original: [1999]), or its continuous-domain analog, ’confounding by severity".
Revised: [1999]), or its continuous-domain analog, "confounding by severity".

Original: The traditional way to break such causal influence cycles is to unroll them in time; the severity of the disorder in the current timestep is the basis for administering the drug, and the response to the drug is reflected in the severity of the disorder in the next timestep.
Revised: One way to break such causal influence cycles is to unroll them in time to create a “Dynamic Bayesian Network” (Dean and Wellman [1991]); one observes the severity of the disorder in the current timestep when administering the drug, and the response to the drug is reflected in the severity of the disorder in the next timestep.

Original: 2.1 Primary simulation
Revised: 2.1 Primary simulation

Original: Our model includes an underlying condition (infection with the imaginary organism Bogovirus ); this condition will resolve over time as long as the patient does not die.
Revised: Our model includes an underlying condition (infection with the imaginary organism Bogovirus ); this condition will resolve over time as long as the patient does not die.

Original: Infection leads to a disorder (it could be something like a problem with blood clotting, breathing, or circulation) that must be managed because if it gets too severe it can lead to death.
Revised: Infection leads to a disorder (it could be something like a problem with blood clotting, breathing, or circulation) that must be managed because if it gets too severe it can lead to death.

Original: The disorder is managed by a drug which has a direct effect on the severity of the disorder, but becomes less effective as the level of the drug accumulates in the body.
Revised: The disorder is managed by a drug which has a direct effect on the severity of the disorder, but becomes less effective as the level of the drug accumulates in the body.

Original: High cumulative levels of the drug can lead to fatal toxicity.
Revised: High cumulative levels of the drug can lead to fatal toxicity.

Original: The ultimate outcome is either recovery or death, but on most days neither of these things happens, and the patient just stays in the hospital for another day.
Revised: The ultimate outcome is either recovery or death, but on most days neither of these things happens, and the patient just stays in the hospital for another day.

Original: We use simulation to generate data for which we know the true causal graph.
Revised: We use simulation to generate data for which we know the true causal graph.

Original: This is done by writing a simple Python program that implements the causal graph shown in Figure 1; each node is represented by a  function whose inputs are dictated by the edges leading into the node, and this set of functions are called in the order shown in the DAG to compute a row of variable values for each day of the simulated patient’s illness.
Revised: This is done by writing a simple Python program that implements the causal graph shown in Figure 1; each node is represented by a random function whose inputs are dictated by the edges leading into the node, and this set of functions are called in the order shown in the DAG to compute a row of variable values for each day of the simulated patient’s illness.

Original: For each patient, this process continues until the infection process completes (leading to a "recover" outcome), or the patient succumbs to either the disorder or the drug toxicity ("die" outcome).
Revised: For each patient, this process continues until the infection process completes (leading to a "recover" outcome), or the patient succumbs to either the disorder or the drug toxicity ("die" outcome).

Revised: We replicate this to simulate a population of patients.

Original: Through trial and error, we adjusted the simulation parameters so that a simple scan of  dose (see Figure 3) will show an optimum, but the peak performance is less than 100% survival, so it leaves some room for improvement.
Revised: Through trial and error, we adjusted the simulation parameters so that a simple scan of a constant daily dose (see Figure 3) will show an optimum, but the peak performance is less than 100% survival, so it leaves some room for improvement.

Original: 2.2 Components of a medical simulation
Revised: 2.2 Components of a medical simulation

Original: These are the variables in the simulation:
Revised: These are the variables in the simulation:

Original: • infection : records how far the patient has passed through the course of the infection (this is basically a counter for percent progress; once it passes 100 and you have not died, you recover).
Revised: • infection : records how far the patient has passed through the course of the infection (this is basically a counter for percent progress; once it passes 100 and you have not died, you recover).

Original: • drug : a dose of the treatment.
Revised: • drug : a dose of the treatment.

Original: This is an adjustable quantity.
Revised: This is an adjustable quantity.

Original: • cum_drug : The accumulated dose of the treatment drug.
Revised: • cum_drug : The accumulated dose of the treatment drug.

Original: This is an exponential moving average, and is subject to decay over time if treatment doses are not administered.
Revised: This is an exponential moving average, and is subject to decay over time if treatment doses are not administered.

Original: • efficacy : The extent to which the drug dose affects severity
Revised: • efficacy : The extent to which the drug dose affects severity

Original: ; the drug becomes less effective at higher cumulative doses.
Revised: ; the drug becomes less effective at higher cumulative doses.

Original: • severity : Quantified level of severity of the disorder caused by the infection.
Revised: • severity : Quantified level of severity of the disorder caused by the infection.

Original: The worse this gets, the more likely the patient is to die.
Revised: The worse this gets, the more likely the patient is to die.

Original: • outcome : if severity gets high enough, the patient’s chances of death increase.
Revised: • outcome : if severity gets high enough, the patient’s chances of death increase.

Original: If the infection runs its course and the patient does not die, they recover.
Revised: If the infection runs its course and the patient does not die, they recover.

Original: • toxicity : a function of cum_drug that is much more pronounced at high cumulative dose.
Revised: • toxicity : a function of cum_drug that is much more pronounced at high cumulative dose.

Original: High toxicity leads to increased probability of death.
Revised: High toxicity leads to increased probability of death.

Original: Variables with suffixes ’ _prev ’ or ’ _next ’ hold lagging or leading values from adjacent timeslices.
Revised: Variables with suffixes ’ _prev ’ or ’ _next ’ hold lagging or leading values from adjacent timeslices.

Original: 2.3 Simulated datasets
Revised: 2.3 Simulated datasets

Original: To simulate a randomized controlled trial (RCT) of drug dose, patients were assigned to cohorts, and each cohort received a fixed dose of the drug on each day of treatment.
Revised: To simulate a randomized controlled trial (RCT) of drug dose, patients were assigned to cohorts, and each cohort received a fixed dose of the drug on each day of treatment.

Original: Figure 2 shows simulated data for a single patient episode.
Revised: Figure 2 shows simulated data for a single patient episode.

Original: There is one column for each node in the model, plus housekeeping attributes patient_id , and day_number (how many days that patient has been in the hospital).
Revised: There is one column for each node in the model, plus housekeeping attributes patient_id , and day_number (how many days that patient has been in the hospital).

Original: The cohort column indicates the group into which the patient was randomized in the simulated RCT; this determines the dose of drug the patient receives.
Revised: The cohort column indicates the group into which the patient was randomized in the simulated RCT; this determines the dose of drug the patient receives.

Original: To simulate observational data, we use the primary simulator to generate another dataset in which a random dose of drug is given to each patient each day.
Revised: To simulate observational data, we use the primary simulator to generate another dataset in which a random dose of drug is given to each patient each day.

Original: This randomized policy dataset is the input to the subsequent modelling tasks.
Revised: This randomized policy dataset is the input to the subsequent modelling tasks.

Original: 2.4 Finding the optimal dose
Revised: 2.4 Finding the optimal dose

Original: The solid red curve in Figure 3 shows survival at different doses of drug from the RCT dataset.
Revised: The solid red curve in Figure 3 shows survival at different doses of drug from the RCT dataset.

Original: The optimal dose (0.7 units) gives us a standard to compare with results we obtain from analytical simulations.
Revised: The optimal dose (0.7 units) gives us a standard to compare with results we obtain from analytical simulations.

Original: Our goal in subsequent sections is to use analytical ’simsim’ simulations to approximate this solid red curve without requiring an RCT.
Revised: Our goal in subsequent sections is to use analytical ’simsim’ simulations to approximate this solid red curve without requiring an RCT.

Original: described obtained trained almost uni formly fatal is fatal the giving

Original: This CPT is a three-dimensional table where each cell holds a probability, and the coordinates of the cells are the categorical values of the inputs ( drug and cum_drug ) and the output ( efficacy ).

Original: Note that the extreme ranges of cum_drug are not represented in the observational data, and the model assumes a uniform prior.

Original: These areas require editing to incorporate domain expertise for extrapolation.

Original:  "SimSim-BN": An Analytical Simulation Built on a Causal BayesNet Model
Revised:  "SimSim-BN": An Analytical Simulation Built on a Causal BayesNet Model

Original: Rather than testing a treatment policy in the primary simulation (analogous to an experiment), we can use a causal model to build a simulation (since this is a simulation of a simulation we call this a ’simsim’ model, of which we present three versions).
Revised: Rather than testing a treatment policy in the primary simulation (analogous to an experiment) we can use a causal model to build a simulation (since this is a simulation of a simulation we call this a ’simsim’ model of which we present three versions).

Original: We then use this causal simulation to perform the optimization scan.
Revised: We then use this causal simulation to perform the optimization scan.

Original: The simsim model is trained on the simulated "observational" dataset where each patient was given a random dose of the drug each day (as opposed to the dose-optimizing simulation where patients were assigned to cohorts that each received the same dose each day).
Revised: The simsim model is trained on the simulated "observational" dataset where each patient was given a random dose of the drug each day (as opposed to the dose-optimizing simulation where patients were assigned to cohorts that each received the same dose each day).

Original: This simulates the process of building a causal model from non-experimental data and using it to estimate an optimal treatment policy without actually running an experiment.
Revised: This simulates the process of building a causal model from non-experimental data and using it to estimate an optimal treatment policy without actually running an experiment.

Original: To simulate building a model from observational data, we used the known causal graph from Figureto build a Bayes network whose parameters were learned from the randomizeddosage dataset.
Revised: To simulate building a model from observational data we used the known causal graph from Figureto build a Bayes network whose parameters were learned from the randomized

Original: 3.1 Extrapolating to areas not covered by the training data
Revised: 3.1 Extrapolating to areas not covered by the training data

Original: Figure 4 shows the CPT for efficacy for three selected discrete values of drug .
Revised: Figure 4 shows the CPT for efficacy for three selected discrete values of drug .

Original: We can see that these tables have captured the relationship that the efficacy of a given dose of the drug decreases at higher levels of cum_drug .
Revised: We can see that these tables have captured the relationship that the efficacy of a given dose of the drug decreases at higher levels of cum_drug .

Original: However, the very high or very low values of cum_drug (the extremes of the x -axis in each panel) show a uniform probability across the levels of efficacy .
Revised: However, the very high or very low values of cum_drug (the extremes of the x -axis in each panel) show a uniform probability across the levels of efficacy .

Original: This is because these values of cum_drug are not represented in the random-dose dataset from which this Bayes
Revised: This is because these values of cum_drug are not represented in the random-dose dataset from which this Bayes

Original: Net was trained (that would require a consistently high or consistently low dose, which is unlikely in the random-dose dataset), and it defaults to using a uniform prior probability in these ranges.
Revised: Net was trained (that would require a consistently high or consistently low dose, which is unlikely in the random-dose dataset), and it defaults to using a uniform prior probability in these ranges.

Original: We therefore took advantage of our domain knowledge to modify the CPTs to extrapolate better to extreme values.
Revised: We therefore took advantage of our domain knowledge to modify the CPTs to extrapolate better to extreme values.

Original: This was done by simply filling out the uniform-probability columns at the extremes with copies of the closest non-uniform column.
Revised: This was done by simply filling out the uniform-probability columns at the extremes with copies of the closest non-uniform column.

Original: We also simplified the input infection , which is basically a counter that keeps track of how far the patient has progressed through the course of infection, into the binary variable infection_over that is true if infection reaches 100 or more, and false otherwise.
Revised: We also simplified the input infection , which is basically a counter that keeps track of how far the patient has progressed through the course of infection, into the binary variable infection_over that is true if infection reaches 100 or more, and false otherwise.

Original: This greatly simplifies the probability table for severity_next .
Revised: This greatly simplifies the probability table for severity_next .

Original:  "SimSim-DECI": Expert-in-the-loop Causal Inference for Intervention Optimization
Revised:  "SimSim-DECI": Expert-in-the-loop Causal Inference for Intervention Optimization

Original: Deep End-to-end Causal Inference ("DECI", Geffner et al.
Revised: Deep End-to-end Causal Inference ("DECI", Geffner et al.

Original: [2022]) 1 is an autoregressive-flow based non-linear additive noise structural equation model (SEM), which is designed to perform both causal discovery and inference, including average treatment effect (ATE) and conditional average treatment effect (CATE) estimation.
Revised: [2022]) 1 is an autoregressive-flow based non-linear additive noise structural equation model (SEM), which is designed to perform both causal discovery and inference, including average treatment effect (ATE) and conditional average treatment effect (CATE) estimation.

Original: DECI discovery takes prior knowledge of the graph structure as input, defined by a constraint  matrix, and uses the data-driven causal calculation to estimate the graph structure and ATE between nodes.
Revised: DECI discovery takes prior knowledge of the graph structure as input, defined by a constraint adjacency matrix, and uses the data-driven causal calculation to estimate the graph structure and ATE between nodes.

Original: 4.1 Using DECI for Timeseries: A Featurization Method
Revised: 4.1 Using DECI for Timeseries: A Featurization Method

Original: SimSim-DECI uses the DECI model to simulate temporal records for the patients under various drug-dose policies and finds the policy for the optimum dosage level.
Revised: SimSim-DECI uses the DECI model to simulate temporal records for the patients under various drug-dose policies and finds the policy for the optimum dosage level.

Original: DECI is primarily designed to digest generic types of tabular Dataset.
Revised: DECI is primarily designed to digest generic types of tabular datasets.

Original: To capture the temporal dynamics of our data, we designed our framework as follows:
Revised: To capture the temporal dynamics of our data, we designed our framework as follows:

Original: • Temporal Sliding-Window featurization : each row contains the information of the current time-step and the previous one as in ( X t − 1 , X t ) .
Revised: • Temporal Sliding-Window featurization : each row contains the information of the current time-step and the previous one as in ( X t − 1 , X t ) .

Original: Figure 5 shows what re-framed dataset looks like.
Revised: Figure 5 shows what re-framed dataset looks like.

Original: • Temporal constraints : Features in the current steps cannot cause features in the previous step.
Revised: • Temporal constraints : Features in the current steps cannot cause features in the previous step.

Original: • Temporal training : The DECI model is trained on each row as an independent sample.
Revised: • Temporal training : The DECI model is trained on each row as an independent sample.

Original: Since each row holds the past and current features, the connection between features of the same day is learned as well as features of two consecutive days.
Revised: Since each row holds the past and current features, the connection between features of the same day is learned as well as features of two consecutive days.

Original: • Temporal simulation : We simulate the information of each step by conditioning the simulation on the information from the previous step.
Revised: • Temporal simulation : We simulate the information of each step by conditioning the simulation on the information from the previous step.

Original: That means for data in Figure 5, we take the values of the seven right columns as the condition for the intervention and predict the next step by sampling the trained model.
Revised: That means for data in Figure 5, we take the values of the seven right columns as the condition for the intervention and predict the next step by sampling the trained model.

Revised: a model, which misses 4 arcs, but recovers 14 arcs.

Original: [Figure]
Revised: It also, suggests 11 extra arcs, which are not accurate based on the original model. Note, this 2-day graph is the equivalent to the original 3-day one on Figure 1. This is possible because each feature only depends on features not more than one day away. Also, we do not need efficacy node as DECI can handle more complex inputs than the table-based Bayes net model used above.

Original: 4.2 Causal Discovery
Revised: 4.2 Causal Discovery

Original: DECI recovers the structure of the underlying causal graph that we can compare with the true underlying graph.
Revised: DECI recovers the structure of the underlying causal graph that we can compare with the true underlying graph.

Original: Thus, we can be the expert-in-the-loop to modify the data-driven findings to improve its correspondence to the truth.
Revised: Thus, we can be the expert-in-the-loop to modify the data-driven findings to improve its correspondence to the truth.

Original: We do this by re-training with additional constraints.
Revised: We do this by re-training with additional constraints.

Original: Figure 6 shows thediscovered unconstrained graph compared to the final graph with expert-in-the-loop constraints applied.
Revised: Figure 6 shows the final graph with expert-in-the-loop constraints applied.

Original: Note that the domain expert prevented confounding by indication (Kyriacou and Lewis [2016]) by reversing the direction of drug-to-severity causality.
Revised: Note that the domain expert prevented confounding by indication (Kyriacou and Lewis [2016]) by reversing the direction of drug-to-severity causality.

Original: 4.3 Simulation-based Policy Optimization
Revised: 4.3 Simulation-based Policy Optimization

Original: The trained model is a structural causal model (SCM) that captures the functional relations and error distributions.
Revised: The trained model is a structural causal model (SCM) that captures the functional relations and error distributions.

Original: Hence, it is capable of estimating the expected updates following an intervention.
Revised: Hence, it is capable of estimating the expected updates following an intervention.

Original: We use this capability to synthesize the current day, having information about the previous day.
Revised: We use this capability to synthesize the current day, having information about the previous day.

Original: We add the desired drug dose based on the candidate policies, explained in previous sections, to the interventions.
Revised: We add the desired drug dose based on the candidate policies, explained in previous sections, to the interventions.

Original: To reduce estimation error, we take the average of 50 samples for the continuous variables and take the mode for the binary ones.
Revised: To reduce estimation error, we take the average of 50 samples for the continuous variables and take the mode for the binary ones.

Original: The simulation for each patient will stop if “die” or “recover” returns True.
Revised: The simulation for each patient will stop if “die” or “recover” returns True.

Original: Figure 3 demonstrates the results of different dose policies.
Revised: Figure 3 demonstrates the results of different dose policies.

Original: The model captures the optimum drug dose correctly, although is less accurate at extremes where the observational data is sparse (see the "simsim-DECI" curve in Figure 3).
Revised: The model captures the optimum drug dose correctly, although is less accurate at extremes where the observational data is sparse (see the "simsim-DECI" curve in Figure 3).

Revised: Note that this model is too optimistic about survival rates for intermediate doses; this may be because the learned graph did not find the edge from ‘cum_drug’ to ’efficacy’, which reduces the benefit provided by the drug at moderate doses.

Original:  Offline Reinforcement Learning: The Third Simulation Reconstruction
Revised:  Offline Reinforcement Learning: The Third Simulation Reconstruction

Revised: Without the causal knowledge conveyed by the network, optimization of the dose does poorly.

Original: For purposes of comparison we took the same randomized policy simulated observational dataset and used a comparable offline reinforcement learning (RL) method to derive a constant-dose policy comparable to the previous simulations.
Revised: We show this for purposes of comparison by taking the same randomized policy simulated observational dataset and using a comparable offline reinforcement learning (RL) method to derive a constant-dose policy comparable to the previous simulations, but ignorant of the causal structure underlying the data.

Original: Because we used a constant policy the method is degenerate, since the policy function does not depend on the current state, and might equivalently be considered just a simple Markov process.
Revised: Because we used a constant policy the method is degenerate, since the policy function does not depend on the current state, and might equivalently be considered just a simple Markov process.

Original: For the RL environment, we wrapped the simulation dataset with OpenAI Gym (Brockman et al.
Revised: For the RL environment, we wrapped the simulation dataset with OpenAI Gym (Brockman et al.

Original: [2016]), to simulate the stage transition probability distribution.
Revised: [2016]), to simulate the stage transition probability distribution.

Original: Unlike the previous “simsim” reconstructions, state variables are continuous-valued, so to generate a random state value given the current state, we used a k-nearest neighbor regressor built on the faiss (Johnson et al.
Revised: Unlike the previous “simsim” reconstructions, state variables are continuous-valued, so to generate a random state value given the current state, we used a k-nearest neighbor regressor built on the faiss (Johnson et al.

Original: [2019]) library for similarity search.
Revised: [2019]) library for similarity search.

Original: Since, even with the simulation dataset of 1 M rows, there is no likelihood that a lookup will find a record within some epsilon of the current state, so instead we average the predicted next-stage values of the k = 3 nearest neighbours.
Revised: Since, even with the simulation dataset of 1 M rows, there is no likelihood that a lookup will find a record within some epsilon of the current state, so instead we average the predicted next-stage values of the k = 3 nearest neighbours.

Original: Equivalently the prediction function could have been implemented as a neural-network trained on the simulation dataset.
Revised: Equivalently the prediction function could have been implemented as a neural-network trained on the simulation dataset.

Original: Either way, predictions are confined to the space that has been explored in the simulation data, with all the limitations this implies for an offline algorithm.
Revised: Either way, predictions are confined to the space that has been explored in the simulation data, with all the limitations this implies for an offline algorithm.

Original: We ran a constant policy in dose increments of 0 .
Revised: We ran a constant policy in dose increments of 0 .

Original: 1 from 0 to 1 . 4 , for 100 episodes, which in almost all cases terminated in less than 20 stages, then averaged the termination counts to estimate survival rates, as shown in Figure 3.
Revised: 1 from 0 to 1 . 4 , for 100 episodes, which in almost all cases terminated in less than 20 stages, then averaged the termination counts to estimate survival rates, as shown in Figure 3.

Original: Admittedly using a constant policy is a trivial algorithm, and cannot do better than an adaptive policy.
Revised: Admittedly using a constant policy is a trivial algorithm, and cannot do better than an adaptive policy.

Original: But for purposes of comparison with the previous "sim-sims", it illustrates the consequences of the lack of causal analysis.
Revised: But for purposes of comparison with the previous "sim-sims", it illustrates the consequences of the lack of causal analysis.

Original: As the figure shows, the offline-RL finds the same optimum dose, but over-estimates the survival rates at extremes.
Revised: As the figure shows, the offline-RL finds the same optimum dose, but over-estimates the survival rates at extremes.

Original: Most notably it believes high doses thatshould accumulate toxic cumulative effects are not deleterious.
Revised: Most notably it believes high doses that accumulate toxic cumulative effects are not deleterious.

Original: Inspection of predictions as revealed in the episode traces show that, at extreme values, the offline predictor is wildly off with predicting accumulation of the drug over time.
Revised: Inspection of predictions as revealed in the episode traces show that, at extreme values, the offline predictor is wildly off with predicting accumulation of the drug over time.

Original: This highlights a known limitation of offline policy optimization, where even in modest dimension state-spaces the ability to explore new policy areas is severely limited by the existing data (Levine et al.
Revised: This highlights a known limitation of offline policy optimization, where even in modest dimension state-spaces the ability to explore new policy areas is severely limited by the existing data (Levine et al.

Revised: Introducing causal domain knowledge as we propose, is one remedy.

Revised: It is similar in spirit to the notion of exposing a tunable user parameter to moderate the extremes of offline-generated policies, as proposed by (Swazinna et al.

Original:  Discussion: Where do we go from here?
Revised:  Discussion: Where do we go from here?

Original: We plan to collaborate with medical researchers in our next iteration of this simulation and modeling process to focus on an actual disease and therapies, to reproduce the analytical logic in a more realistic setting.
Revised: We plan to collaborate with medical researchers in our next iteration of this simulation and modeling process to focus on an actual disease and therapies, to reproduce the analytical logic in a more realistic setting.

Original: We have experience implementing a similar forward-inference Bayes Net in the Synthea electronic medical record simulator (Walonoski et al.
Revised: We have experience implementing a similar forward-inference Bayes Network in the Synthea electronic medical record simulator (Walonoski et al.

Original: [2017]) 2 ; porting our next version to that platform would let us develop shareable feature engineering exercises to conduct the kind of simulation-based analysis described here on data in a realistic schema.
Revised: [2017]) 2 ; porting our next version to that platform would let us develop shareable feature engineering exercises to conduct the kind of simulation-based analysis described here on data in a realistic schema.

Original: The DECI team is extending their work tosupport temporal use cases(Gong et al. 
Revised: For our next iteration we plan to use Rhino (Gong et al. [2022]), the vector auto-regressive extension of DECI, instead of bespoke temporal featurization.

Original:  and we plan to evaluate our simulations in that tool.
Revised: Rhino can learn history-dependent noise, and we look forward to exploring circumstances where this leads to better simulation performance

Original: Causal graphs provide an opportunity to capture domain expertise; this is a modeling process that, as we have shown, blends gracefully into simulation modeling.
Revised: Causal graphs provide an opportunity to capture domain expertise; this is a modeling process that, as we have shown, blends gracefully into simulation modeling.

Revised: The advantage of adding the causal aspect to the modeling process is that we have an explanation of the model’s function in terms comparable to current clinical understanding, and therefore improves credibility and the extensibility of the model’s results.

Revised: We imagine adding a causal modelling step to extend offline simulation for an envisioned causal reinforcement learning model.

Original: The code for this project is in Github 3 .
Revised: The code for this project is in Github 3 .