When to intervene: learning optimal intervention policies for critical events
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Providing timely interventions before the onset of a critical event, such as a systemfailure is an important problem in many industrial settings.
Before the onset of thecritical event, usually, systems exhibit behavioral changes which often manifest asstochastic co-variate observations which may be leveraged to trigger intervention.
In this paper, for the ﬁrst time, we formulate the problem of ﬁnding an optimallytimed intervention (OTI) policy as minimizing the expected residual time to event,subject to a constraint on the probability of missing the event.
Existing machinelearning approaches to intervention on critical events focus on predicting eventoccurrence within a pre-deﬁned window (a classiﬁcation problem) or predictingtime-to-event (a regression problem).
Interventions are then triggered by settingmodel thresholds.
These are heuristic-driven, lacking guarantees regarding optimality.
To model the evolution of system behavior, we introduce the concept ofa hazard rate process.
We show that the OTI problem is equivalent to an optimalstopping problem on the associated hazard rate process.
This key link has not beenexplored in literature.
Under Markovian assumptions on the hazard rate process,we show that an OTI policy at any time is completely determined by the conditional hazard rate function at that time.
We proceed to analytically characterizethe optimal stopping policy, enabling practical survival-model-based critical eventintervention algorithms.
Further, we show that our framework includes, as a specialcase, the important class of neural hazard rate processes generated by recurrentneural networks (RNNs).
To model neural hazard rate processes we propose adynamic deep recurrent survival analysis (DDRSA) architecture, introducing an
RNN encoder into the static DRSA setting.
Finally, we demonstrate RNN-based
OTI policies with practical experiments and show that they outperform popularintervention methods.
Introduction
We consider a problem that involves the monitoring of observations from a discrete-time stochasticprocess with the goal of intervening to prevent the occurrence of a critical event that will happenin the future.
A critical event is missed (loosely termed a false negative) when intervention is nottriggered before the critical event occurs.
The general objective is the design of algorithms that can beused to trigger a timely intervention (the objective may be to maximize useful life, minimize overalldomain speciﬁc costs e.t.c.), subject to false negative constraints.
This problem has signiﬁcant practical signiﬁcance in a number of ﬁelds.
Critical events could be thefailure of a machine or key component, customer churn, loan default, or even death.
Intervening onsuch events ahead of time, in an optimal manner is of great signiﬁcance to businesses and society,due to the potential of mitigating the impact of the occurrences of these events.
1.1 Interventions on critical events
The monitored stochastic process is represented by a discrete, temporal sequence of random variableswith the random variable X j representing a d-dimensional real-valued observation at time step j .
Letthe set
X j = { X s : 1 ≤ s ≤ j } represent the observation history until time step j .
The critical eventis assumed to occur at a random time-step L < L max 1 .
The sequence of observations terminatesafter the occurrence of the critical event.
[-In this case we can assume a ﬁnal observation X L +1 = 
-]
An intervention at time step j is considered early if the intervention occurs at a time-step j < L .
Weconsider present and future interventions (at j = L and beyond) too late by deﬁnition.
We consider a data-driven sequential decision rules (intervention policies) that are deﬁned as follows:
Deﬁnition 1.1 (Intervention policies) .
An intervention policy is a sequence of functions ϕ =( ϕ 1 , ϕ 2 , · · · ϕ j , · · · ) , where ϕ j : R d × j (cid:55)→ { 0 , 1 } maps the history of observations at a time step j toa decision of whether to intervene ( ϕ j ( X j ) = 1 ) or not ( ϕ j ( X j ) = 0 ).
The intervention policies we consider in this paper are terminal, i.e: once a decision to intervene istriggered, the policy stops.
In effect, this means that intervention triggers after the ﬁrst trigger areignored for a particular realization of the process.
Note that we can also allow for policy reactivationafter an intervention or critical event.
The sequence of observations starting at policy reactivationsare simply treated as independent realizations of the process.
So if we recover from the occurenceof a ﬁrst critical event and re-activate the policy to continue to monitor for a second second criticalevent, or re-activate the policy to continue monitoring after an intervention, both these cases will beconsidered as independent trials.
Deﬁnition 1.2 (policy stopping time) .
Intervention on a monitored process happens at a random timestep τ := min { j ≥ 1 : ϕ j ( X j ) = 1 } .
We call this time the stopping time 2 of the policy.
1.2 Optimally
timed intervention ( OTI on critical
events
We now formally state the optimally timed intervention (OTI) problem for the ﬁrst time.
We seek toﬁnd intervention policy (or equivalently, the stopping time) that minimizes expected time-to-event 61 E
[( L − τ ) + ] subject to the constraint that the probability of missing the critical event is at mostα ∈ (0 , 1] : [Equation]
[Equation]
The expected time-to-event, also called expected residual life or mean residual life (MRL)) is aquantity that is well studied in literature with application in multiple domains [1, 14].
Note that there exists a multiplier C α , such that the OTI problem may be expressed in the followingequivalent, unconstrained form:
[Equation]
The objective simply encodes the tradeoff between expected residual life and the probability ofmissing the critical event.
C α can directly be interpreted as the risk (cost) of missing the criticalevent.
Changing C α allows one to vary the risk tradeoff between the miss-rate and expected timeto-event.
Note that when we miss the critical event i.e: τ ≥ L , expected residual life is zero, so 71 E
[( L − τ ) + ] = 0 .
A related objective also appears in optimal stopping formulations for Bayesian quickest change pointdetection problem [37], also called the Shirayaev problem [29], where the objective is to monitor astochastic process for the quickest possible detection of a change point or stochastic disorder.
Herethe tradeoff is between stopping too early (proabability of a false alarm) and the expected time delayin detecting the change point, after it has occured.
1.3 Related work
Window based critical event prediction: Intervention is a binary decision.
At each time-step adecision must be made to intervene or not.
This is often modeled directly as a binary classiﬁcationproblem where we look for event occurrence within a speciﬁed future window.
In window basedintervention (WBI), time-steps where an event is present within a lookahead period are labeled aspositive class examples and the rest are labeled as negative examples.
A classiﬁer may then be learntto predict the probability of the positive class.
An optimal threshold on this probability is then tunedover a validation set that measures costs of triggered interventions directly to choose an optimal triggerthreshold.
WBI policies are standard practice in predictive maintenance[33, 2, 22, 2, 38, 40, 13].
Time-to-Event prediction (Survival analysis): Classical survival analysis deals with predicting thetime to an event (ex: death, machine failure) given a vector of observations (co-variates) obtainedfrom a subject in the presence of incomplete (censored) observations [23].
The goal is to estimatefunctions such as the hazard rate (rate of death as a function of time) or survival function (probabilityof survival past a given time) and understand the impact of co-variates on these functions.
Classicalparametric model-based approaches [10, 41] have been extended in recent work using machinelearning approaches [42, 18, 5, 30, 20].
In some cases, the impact of a co-variate on the survivalfunction may vary over time (for instance, the impact of smoking may increase with age), resulting inapproaches that extend survival analysis to model the time-varying effects of covariates [35, 3].
Thesehave deep learning counterparts [31, 12] that use recurrent neural networks to model the survivalfunction.
Recent work has applied modern deep learning approaches to longitudinal data which are sequentialobservations taken over time for each subject.
Corresponding work also exists in thearea of machine failure prediction [26].
The relevant papers to our work are the recurrent neuralnetwork approaches [19, 26] that operate on longitudinal data to predict the probability distributionof failure time given previous history of co-variates for any subject, at each observation time.
Whilethese methods are mostly complementary to the primary goal of this paper where we focus onobtaining optimal intervention policies, we do propose improvements in to these methods in Section4.1 to overcome the distributional assumptions in [26] and difﬁculty with long sequences [19].
Inpractice, intervention policies based on survival analysis are typically just thresholds on the predictedtime-to-event.
Optimal stopping problems: Optimal stopping problems [39, 9, 4] are a subclass of stochasticcontrol problems with only two possible actions:stop and continue.
The goal is to stop takingobservations when the current reward (or cost) exceeds (or falls below) expected future reward(or cost).
Optimal stopping techniques have been applied to several problems such as sequentialhypothesis testing [39, 11, 7], early change point detection [29, 37, 36, 32] and early classiﬁcation[25].
Optimal stopping policies for machine maintenance have been studied in reliability theory[6, 24], where the goal is to ﬁnd the best stopping time to replace/maintain a machine based oncondition.
However, the approaches in such cases are limited to simple probabilistic models ofmachine degradation with a single or few co-variates.
General model-free reinforcement learningapproaches to the optimal stopping problem are generally feasible in situations where we have a lotof data or in a model-based setting where the underlying randomness can efﬁciently be simulatedand the model can be used to draw Monte-Carlo samples [8].
This is not an option in most criticalevent prediction settings when the critical event data is often scarce and no simple model exists forthe evolution of the co-variates.
1.4 Outline of contributions
The OTI problem is hard since system evolution towards a critical event is complex with typicallyscarce occurences of critical events.
Current modeling approaches are challenged either because theymake signiﬁcant overly simplifying assumptions on the underlying process or because scarce eventsprecludes powerful model-free approaches.
To our knowledge, we are the ﬁrst to frame the OTI problem and link it to both survival analysis andoptimal stopping theory.
This allows, for the ﬁrst time, the ability to model complex system evolutiontowards scarce critical events with the ability to intervene optimally.
• A process under observation may exhibit degradation/disorder as it evolves towards a critical event.
We capture this feature by introducing the concept of a hazard rate process, which isa stochastic process modeling the evolution of conditional hazard rates as new observationsare taken.
A key contribution of our work is to formalize the OTI problem in equation (OTIproblem) and show that the OTI problem is equivalent to an optimal stopping problem on theunderlying hazard rate process (Proposition 2.1).
This allows us to tap into optimal stoppingtheory [39, 9, 4] to unlock the structure of OTI policies and also leverage computationalmethods of estimating/learning hazard rate functions.
These are key to our development of a general machine learning approach to the OTI problem.
• Under Markovian assumptions on the hazard rate process, we show that the optimal inter- vention policy decision rule at each time step can be analytically determined by the hazard rate function at that time step (Theorem 3.1).
Further we show that the optimal policy is reduces to a variable threshold on the expected time-to-event(Corollary 3.1.1), showing thatstatic threshold policies are sub-optimal in general.
• We show that our theory applies to an important class of hazard rate processes generatedby recurrent neural networks (Section 4).
We term these processes Neural hazard rate processes.
This allows us to apply survival analysis methods to learn hazard rate processes and analytically compute optimal intervention policies for critical events.
We adapt the stateof the art deep dynamic survival analysis (DRSA) work [31] to our setting.
Unlike DRSA, our dynamic DRSA (DDRSA) setting requires modeling dynamic hazard rate functions.
We
do this by adapting the DRSA architecture to our dynamic covariate evolution setting using an encoder-decoder seq2seq architecture.
OTI as an optimal stopping problem on a hazard rate process
In this section we establish a link between the OTI problem, survival analysis and optimal stoppingtheory.
We model the degradation of the monitored stochastic process towards the critical event bythe corresponding evolution of underlying hazard rate functions conditioned on observation history.
To this end, we will use the following deﬁnitions of a conditional hazard rate function h j ( · ) , andsurvival function S j ( · ) :
[Equation]
Note that we have omitted the implied conditioning on the covariates, so that h j ( k |X j ) is writtenas h j ( k ) for convenience.
Also, we have h j ( − 1) :
= 0 and S j ( − 1) :
= 1 reﬂecting the fact that thecritical event has not occured until time step j .
We now deﬁne the concept of a hazard rate processes:
Deﬁnition 2.1 (Hazard Rate Process) .
Consider the random vector constructed from the conditionalhazard rate function h j ( · ) given by H j =
[ h j (0) , h j (1) · · · h j ( L max − 1)] T .
We call the sequenceof tuples { ( X j , H j ) } a hazard rate process.
Thus, hazard rate processes model the relation between the evolution of co-variates and onset of thecritical event.
2.1 Optimal intervention on a hazard rate process
Proposition 2.1 (Optimal intervention on a hazard rate process) .
Given a hazard rate process{ ( X j , H j ) } , the intervention risk at a stopping time τ is given by:
[Equation]
In this setting, the optimal critical event intervention problem of equation (OTI problem) reduces tothe following optimal stopping problem 3 on the corresponding hazard rate processes:
[Equation]
a stochastic process
Proof.
We only give a proof sketch here.
The proof of the theorem is given in Appendix A. Startingwith terms in the OTI problem: for expected residual life we may write ( L − τ )
+ = (cid:80) k ≥ 0
( L >τ + k ) , and then use the law of iterated expectations, E [( L − τ ) + ] =
E [ (cid:80) k ≥
E [ 1 ( L > τ + k ) |X τ ]] .
We see that this expression is E
[ (cid:80) k ≥
S τ ( k )] and convert to hazard rates.
Similarly, probability ofmissing the event P ( L ≤ τ ) =
E [ E [ 1 ( L ≤ τ ) |X τ ]] , reduces to E [ h τ (0)] .
Note that the formulation of equations (3) and (4) exposes the dependence of the OTI objective onthe co-variates, since h τ ( . ) depends on the covariates.
This was implicit in the original formulation.
Analytic solution to the OTI problem for Markov hazard rate processes
In order to solve the OTI optimal stopping problem we make the assumption that the hazard rateprocesses we deal with satisfy the Markov property:
Deﬁnition 3.1 (Markov hazard rate process) .
Consider the hazard rate process { ( X j , H j ) } .
We callit a Markov hazard rate process, if we have:
[Equation]
With this assumption in place we can state the central result of this paper:
Theorem 3.1 (OTI policy for a Markov hazard rate process) .
Given a Markov hazard rate process{ ( X j , H j ) } and associated derived risk process { R H ( H j ) } , the optimal intervention policy thatsolves the problem posed in Proposition 2.1 is a policy ϕ ∗ that calls for intervention at stopping timeτ ∗ such that:
[Equation]
) ≤ V j ( H j )) for value functions
V j ( H j ) given by:
[Equation]
Proof.
We only give a proof sketch here.
The complete proof of the theorem is given in Appendix
A.
The key is to notice that under the Markov hazard rate process assumption, and the form of ofequations (3) and (4), that risk R j depends only on X j
so we have an optimal stopping problemthat has a so-called Markov Representation [29].
The optimal policy has a standard form givenby ϕ ∗ j ( R j )
= 1 ( R j ≤ V j
( X j ) with V j ( X j ) =
inf l ≥
E [ R j + l |X j ] .
So we need only compute 193 E [ R j + l |X j ] , where the expectated future risk over the possible occurences of the critical event for arule that stops at j + l .
Reasoning, using the hazard rates h j ( k ) , we calculate E [ R j + l |X j ] analyticallygiving rise to the ﬁnal form for V j .
This theorem tells us that in the case of a hazard rate processes, the optimal policy may be constructedexplicitly with knowledge of only the conditional hazard rate functions at that time.
The valuefunctions V j ( · ) gives the minimum expected cost of continuing to take observations, having obtaineda conditional hazard rate function ( H j ) at time j .
It is clear that the the optimal policy is a timevarying threshold on the risk process in general.
Also note that putting l = 0 in the value functionrecovers the instantaneous risk at time step j as the ﬁrst term over which the minimum is computed inthe value function.
The value function gives the best stopping risk over all stopping decisions in thepresent and future.
The following Corollary allows a further important simpliﬁcation of the optimalpolicy:
Corollary 3.1.1 (OTI policy as a time varying threshold on time-to-event) .
Let T j be the expectedtime to event at time j .
The optimal stopping time and optimal OTI policy of Theorem 3.1 may besimpliﬁed to the following form:
[Equation]
where
[Equation]
with T j given by:
[Equation]
The proof is given in Appendix A.
This Corollary clearly shows that static threshold TTE policiesare sub-optimal in general.
The value function contain two terms.
The ﬁrst term, which is the riskof stopping early before the critical event which intuitively gets smaller as j progresses toward thecritical event.
The second term is the risk of missing the critical event that depends on the cost ofmissing the event.
It is high where the onset of the critical event is expected.
Neural hazard rate processes
While it is true that one may adapt any method (in theory) that is used for Survival Analysis toestimate the condtitional hazard rate functions used in our work, RNNs are a good ﬁt for being ableto model complex patterns in multi-variate time-series data and summarizing observation history.
The previous theory applied to Markov hazard rate processes.
We show that this is not a signiﬁcantrestriction.
In fact, the class of hazard rate processes that can be modeled by recurrent neural networks(RNNs) also satisfy our Markovian assumptions.
To see this, note that RNNs may be viewed as afeature transformation, mapping X j to features Z j = { Z s : 1 ≤ s ≤ j } where Z j is the hidden statevector of the RNN at time step j .
If we output hazard rate vectors as functions of the hidden state wegenerate a neural hazard rate process given { ( Z j , H j ) } .
ue to the recurrent dynamics of the RNN,the ﬁnal state encode all history, therefore P ( Z j +1 |Z j ) =
P ( Z j +1 | Z j ) and the process is Markovian.
4.1 Dynamic deep recurrent survival analysis (DDRSA)
Since the OTI theory developed in the previous section directly applies to neural neural hazard rateprocesses, we may ﬁt a neural hazard rate process model to data and then apply the optimal OTIpolicy to intervene.
Thus, our approach requires estimates of conditional hazard rates, that evolveover time (hazard rate processes).
Ren et.
al.
[31] developed Deep Recurrent Survival Analysis (DRSA), a deep recurrent neuralarchitecture that uses and RNN to map a static covariate vector to a single conditonal hazard ratefunction (so it is not a hazard rate process that evolves over time).
Unlike DRSA, our Dynamic
DRSA (DDRSA) setting requires modeling dynamic survival and hazard rate functions.
We dothis by adapting the DRSA architecture to our dynamic covariate evolution setting.
While there areother attempts at modeling the dynamic evolution of survival functions [19, 26] they either makedistributional assumptions [26] or have difﬁculty with long sequences due to large softmax layer [19].
Figure 1 shows our DDRSA architecture which is a classic encoder-decoder architecture.
DDRSAuses an encoder RNN that maps co-variate history X j to an single encoder hidden state vector Q j .
This vector is then passed into a decoder which has a standard DRSA architecture, producing decoderhidden states Z jk and corresponding conditional hazard rates h j ( k )
=
σ ( D ( Z jk )) .
DDRSA can beseen as a generative model for a Markov hazard rate process.
During training, each training sample, randomly selected from one of the time-series at time stepj includes a lookback window of co-variates X j − K · · · X j that are fed into the encoder input.
Theencoder’s ﬁnal hidden state is replicated and provided to the decoder as input to each time step alongwith a linear time-step encoding feature.
We leverage the DRSA loss of Ren et.
al.
[31] which iscomposed of the following loss components for each sample:
[Equation]
LSTM LSTM
LSTM LSTM
LSTM LSTM
X j −
X j
[Equation]
X j ( Q j , 0) ( Q j , 1) ( Q j , L max − 1) ( Q j , c j )
FC σ
FC σ
FC σ ˆ h j (0)
ˆ h j (1)
ˆ h j
( L max − 1) l z is the likelihood of the critical event occurance at at time j + l , assuming that the event is notcensored.
l u is the likelihood of event occurance (a.k.a. event rate) at any future time in the decoderoutput, given that the sample is not censored.
l c gives the likelihood of a censored sample, when theevent does not occur at a future time over the decoder outputs.
The ﬁnal loss for a sample is given by :
[Equation]
where λ is a tradeoff fraction and c is the censoring indicator, with c = 1 for a censored sample andc = 0 for uncensored.
Losses averaged over a mini-batch of samples are used to learn the survivalmodel.
The sequential DRSA loss allows effective supervision at each time-step allowing it to beapplied to model long survival functions without resorting to restrictive parametric distributions.
During inference, we estimate, at any time-index j h j (0) , h j (1) , · · · h j ( L max − 1) as decoderoutputs by passing values of co-variates X j − K · · · X j into the DDRSA architecture encoder.
Then,we calculate expected time to event using equation (8) and the value function using equation (7).
Intervention is triggered at time j if expected time to event falls below the value function.
Note that the model architecture is such that we can generate a long sequence of hazard rates.
Duringtraining time we may restrict the decoder time-steps to a ﬁxed value ( ≤ L max ) and consider eventsoutside this as censored.
Such handling of long sequences is a key practical advantage of using thesurvival formulation.
Experiments
The goal of this section is twofold:
To complement the theory developed in the earlier sections with examples to validate and
build intuition into survival and hazard rate processes how the optimal policy works to
produce interventions.
Section 5.
To demonstrate that the theory does indeed translate into an effective practical methodologyembodied by the DDRSA-RNN that can model hazard rate processes and generate interventions using the OTI policy.
We call this combined approach the OTI-DDRSA-RNN.
Weshow that OTI-DDRSA-RNN outperforms static threshold policies that are either windowbased (WBI-RNN) or time-to-event based (TTE-RNN)(Section 5.2).
Experiments are performed on two real-world datasets to validate our goals:
Turbofan Engine Failure Data [28, 34]: This dataset from NASA provides train and test data topredict Turbofan engine failures.
The training set has simulated multivariate time-series observationsfrom 21 sensors on a ﬂeet of 218 engines.
Each engine operates normally at the start and develops afault at some point resulting in eventual failure.
The test set consists of 218 partial engine time-serieswith remaining useful life (in engine cycles) indicated in each case.
We could not use the test dataas provided since it is missing part of every time series close to failure.
We standardized the sensorobservations to prepare them for input into our models.
Azure Predictive Maintenance Guide Data[27]: This is a dataset from a guide provided by Microsoft as a case study in failure prediction.
The data comes from multiple sources including
[Figure]
time-series of voltage, rotation, pressure and vibration measurements collected from 100 machinesin real time averaged over every hour, error logs, machine information (type, age etc.) and 720failure records.
We adopt the same pre-processing, feature generation as given in the guide.
Also, wedownsample the data to take one sample every 15 hours.
For both datasets, we train on 70% of randomly selected co-variate time-series sequences and holdout 30% of the sequences for testing.
From the training set a further 30% of the sequences are setaside as validation data to tune model parameters and policy thresholds.
This process is repeated toproduce 5 random splits.
Model selection details and parameter settings are discussed in Appendix B.
5.1 Discussion of OTI interventions on real data
We ﬁt the DDRSA-RNN on the covariate time-series leading upto failure and predict on a single testset sequence.
This produces, at each time step j of the sequence, an estimated conditional hazard ratevector H j (representing the conditional hazard rates into the future), which gives us a probabilisticestimate of the onset of the critical event at each time step, taking into account co-variate history uptothat time step.
Figure 2(a) shows the evolution of estimated conditional hazard rate functions.
We seethat the onset of the critical event is clearly reﬂected in the progression of the estimated vectors H j .
Next, we demonstrate the working of the OTI-policy, which was shown in Section 3 to be an adaptivethreshold on the estimated time to event.
Figure 2(b) plots the estimated time-to-event at each timestep, estimated from H j using equation (8).
We also plot the adaptive threshold V (cid:48)
j ( H j ) , calculatedusing equation (7) of Corollary 3.1.1.
An intervention is triggered when our time-to-event estimatefalls on or below the adaptive threshold.
Consider what happens as the cost of missing the criticalevent is varied.
First, note that since time-to-event is independent of C , only the threshold changes.
We see that increasing the cost of missing the critical event increases the threshold causing earlierintervention as we would expect.
5.2 Performance Evaluation
In this section we will evaluate the performance of the proposed OTI-DDRSA-RNN vs. two staticthreshold policy baselines:
A window based intervention RNN (WBI-RNN) and a time-to-eventbased intervention RNN (TTE-RNN).
All RNNs share the same encoder architecture which is an
LSTM with 128 step lookback and hidden state dimension of 16 units.
WBI-RNN architecture :
The WBI-RNN has the same encoder as the DDRSA-RNN with the ﬁnalencoder output Q j in Figure 1 passed to a dense (fully connected) layer to produce a single outputprobability per training sample.
We train the model with a binary cross-entropy loss between the
[Figure]
rate     a v e r a g e t i m e t
o e v e n t
OTI TTE
WBI (a) NASA turbofan data: policy tradeoff plot 0.
0.5 event miss rate
a v e r a g e t
i m e t
o e v e n t
OTI TTE
WBI (b) Azure PM guide data: policy tradeoff plot probability output and the binary state of event in a future window W of time steps.
A threshold isused on this probability to trigger interventions once the probability is greater than the threshold.
Thewindow size and threshold are selected by cross-validation (see Apendix B).
TTE-RNN architecture :
The TTE-RNN uses the DDRSA architecture shown in Figure 1 and leartwith the DRSA loss as discussed above.
However, interventions are trigerred with a static thresholdthat is set via cross-validation similar to the WBI-RNN.
WBI and TTE thresholds are tuned (individually, for 125 different settings of C α ∈ { 8 , 10 , · · · 256 } )on each validation set using an empirical intervention policy risk:
[Equation]
(11) where ( L = L i , τ = τ i ) are realized outcomes for the i th series.
All methods were trained, validatedand tested using 10 random train-validation-test splits.
Model parameter initialization seeds arerandomized across trials.
Results reported are medians over the 10 test splits.
Computational setup and inference time: All experiments were run on a Tensorbook Laptop with32GB RAM and having a single NVIDIA GeForce GTX 1070 with Max-Q GPU.
Training on a singledataset for 10 folds and 125 cost points took around 2 hours.
DDRSA inference time per time-steptime was around 12 milliseconds.
The corresponding OTI policy evaluation time per time-step wasaround 0.5 milliseconds.
Figure 3 summarizes the evaluation results in the form of policy tradeoff curves.
Each point representsa median over trials of the average time to event and event miss-rate, calculated for a particular settingof C α (the corresponding standard error reported in Appendix B).
Regarding the tradeoff plots, alower frontier is preferred, since it implies a lower average time-to-event for a given miss-rate and alower miss-rate for a given average time-to-event.
We see that the OTI-DDRSA-RNN outperformsthe static threshold policies on both datasets over a wide range of miss-rates.
Limitations: The quality of the OTI policy depends on the accuracy of the underlying survivalanalysis.
Note that our approach requires ﬁtting a survival model for h j ( k ) at each time step overseveral values of k .
It is possible that some of these are better estimated than others, while in contrasta threshold algorithm can be tuned for a speciﬁc C α on a validation set.
What we expect with OTI,however is a broad range of miss-rates at which it will outperform.
Being able to weight the survivalestimation for enhanced policy performance for a speciﬁc C α is an interesting next step.
Also, thereis no direct way of setting C α to guaruntee a particular α .
We are forced to try a range of values for
C α and pick the values with most acceptable tradeoffs.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[TODO]
YES
(b) Did you describe the limitations of your work?
[TODO]
YES
(c) Did you discuss any potential negative societal impacts of your work?
[TODO]
Noobvious negative societal impact
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[TODO]
YES
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[TODO] YES
(b) Did you include complete proofs of all theoretical results?
[TODO] YES, in supple-
mental material
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[TODO] YES
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[TODO] YES
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[TODO] YES, included in supplemental material
(d) Did you include the total amount of compute and the type of resources used (e
g., type of GPUs, internal cluster, or cloud provider)?
[TODO] YES, included in supplemental
material
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[TODO] YES
(b) Did you mention the license of the assets?
[TODO] YES, links to data provided in the
paper have license information
(c) Did you include any new assets either in the supplemental material or as a URL?
[TODO]
NO
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[TODO] Not applicable.
Only free license resources used
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[TODO] Not applicable
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[TODO] Not applicable
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[TODO] Not applicable
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[TODO] Not applicable