When to intervene: learning optimal intervention policies for critical events
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Providing timely interventions before the onset of a critical event, such as a systemfailure is an important problem in many industrial settings.
Before the onset of thecritical event, usually, systems exhibit behavioral changes which often manifest asstochastic co-variate observations which may be leveraged to trigger intervention.
In this paper, for the ﬁrst time, we formulate the problem of ﬁnding an optimallytimed intervention (OTI) policy as minimizing the expected residual time to event,subject to a constraint on the probability of missing the event.
Existing machinelearning approaches to intervention on critical events focus on predicting eventoccurrence within a pre-deﬁned window (a classiﬁcation problem) or predictingtime-to-event (a regression problem).
Interventions are then triggered by settingmodel thresholds.
These are heuristic-driven, lacking guarantees regarding optimality.
To model the evolution of system behavior, we introduce the conceptof a survival process.
We show that the OTI problem is equivalent to an optimalstopping problem on the associated survival process.
This key link has not beenexplored in literature.
Under Markovian assumptions on the survival process, weshow that an OTI policy at any time is completely determined by the conditionalsurvival function at that time.
We proceed to analytically characterize the optimalstopping policy, enabling practical survival-model-based critical event interventionalgorithms.
Further, we show that our framework includes, as a special case, theimportant class of neural survival processes generated by recurrent neural networks(RNNs).
To model neural survival processes we propose a dynamic deeprecurrent survival analysis (DDRSA) architecture, introducing an RNN encoderinto the static DRSA setting.
Finally, we demonstrate RNN-based OTI policieswith practical experiments and show that they outperform popular interventionmethods.
Introduction
We consider a problem that involves the monitoring of observations from a discrete-time stochasticprocess with the goal of intervening to prevent the occurrence of a critical event that will happenin the future.
A critical event is missed (loosely termed a false negative) when intervention is nottriggered before the critical event occurs.
The general objective is the design of algorithms that can beused to trigger a timely intervention (the objective may be to maximize useful life, minimize overalldomain speciﬁc costs e.t.c.), subject to false negative constraints.
This problem has signiﬁcant practical signiﬁcance in a number of ﬁelds.
Critical events could be thefailure of a machine or key component, customer churn, loan default, or even death.
Intervening onsuch events ahead of time, in an optimal manner is of great signiﬁcance to businesses and society,due to the potential of mitigating the impact of the occurrences of these events.
1.1 Interventions on critical events
The monitored stochastic process is represented by a discrete, temporal sequence of random variables,where X j ∈ R d represents an observation at time-step j .
Let X j =
X s : 1 ≤ s ≤ j } represent theobservation history until time step j .
The critical event occurs at a random time-step L in the series.
The sequence of observations terminates after the occurrence of the critical event.
An interventionat time step j is considered early if the intervention occurs at a time-step j < L .
An interventionis considered late (the critical event is effectively missed) if no intervention occurs at any time-stepj < L .
We consider a data-driven sequential decision rules (intervention policies) that are deﬁned as follows:
Deﬁnition 1.1 (Intervention policies) .
An intervention policy is a sequence of functions ϕ =( ϕ 1 , ϕ 2 , · · · ϕ j , · · · ) , where ϕ j : X j (cid:55)→ { 0 , 1 } maps the history of observations at a time step jto a decision of whether to intervene ( ϕ j ( X j ) = 1 ) or not ( ϕ j ( X j ) = 0 ).
The intervention policies we consider in this paper are terminal, i.e: once a decision to intervene istriggered, the policy stops.
In effect, this means that intervention triggers after the ﬁrst trigger areignored for a particular realization of the process 1 .
Deﬁnition 1.2 (policy stopping time) .
Intervention on a monitored process happens at a random timestep τ = τ ( X j ) : = min { j ≥ 1 : ϕ j ( X j ) = 1 } .
We call this time the stopping 2 time of the policy.
Note that the stopping time random variable may be understood as a general mapping of observationhistory to an intervention time, while an intervention policy maps observation history to a decision tointervene.
The two concepts are equivalent.
Indeed, we have ϕ j ( X j )
= 1 ( τ ( X j ) = j ) .
1.2 Optimally timed intervention (OTI) on critical events
We seek to ﬁnd intervention policy (or equivalently, the stopping time) that minimizes expectedtime-to-event subject to the constraint that the probability of missing the critical event is at mostα ∈ (0 , 1] .
We now formally state the optimally timed intervention (OTI) problem for the ﬁrst time as:
[Equation]
] where: M α = { 1 ≤ τ < L max : P [Equation] L ≤ τ ) ≤ α
[Equation]
Optimal interventions are efﬁcient in the sense that they optimize the tradeoff between expected timeto-event and the probability of missing the event (the miss-rate). For a given expected time-to-event,the miss-rate is minimized and for a given miss-rate the expected time-to-event is minimized.
Note that there exists a multiplier C α , such that the OTI problem may be expressed in the followingequivalent, unconstrained form:
[Equation]
C α can be directly be interpreted as the risk (cost) of missing the critical event.
Changing C α allowsone to vary the risk tradeoff between the miss-rate and expected time-to-event.
1.3 Related work
Window based critical event prediction: Intervention is a binary decision.
At each time-step adecision must be made to intervene or not.
This is often modeled directly as a binary classiﬁcationproblem where we look for event occurrence within a speciﬁed future window. In window basedintervention (WBI), time-steps where an event is present within a lookahead period are labeled aspositive class examples and the rest are labeled as negative examples.
A classiﬁer may then be learntto predict the probability of the positive class.
An optimal threshold on this probability is then tuned function of X n over a validation set that measures costs of triggered interventions directly to choose an optimal triggerthreshold.
WBI policies are standard practice in predictive maintenance[31, 1, 20, 1, 36, 38, 12].
Time-to-Event prediction (Survival analysis): Classical survival analysis deals with predicting thetime to an event (ex: death, machine failure) given a vector of observations (co-variates) obtainedfrom a subject in the presence of incomplete (censored) observations [21].
The goal is to estimatefunctions such as the hazard rate (rate of death as a function of time) or survival function (probabilityof survival past a given time) and understand the impact of co-variates on these functions.
Classicalmodel-based approaches for modeling co-variate impact on survival such as the Cox proportionalhazards model [9] and the accelerated failure time model [39] have been extended in recent workusing machine learning approaches [40, 16, 4, 28, 18].
In some cases, the impact of a co-variate onthe survival function may vary over time (for instance, the impact of smoking may increase with age),resulting in approaches that extend survival analysis to model the time-varying effects of covariates[33, 2].
These have deep learning counterparts [29, 11] that use recurrent neural networks to modelthe survival function.
Note that even in this case co-variates for a subject themselves are static anddoe not evolve over time.
It is the impact of the co-variates that is time-varying.
Recent work insurvival analysis considers longitudinal data which are sequential observations taken over time foreach subject.
The regression approaches in this case are based on the concept that degradation ordeterioration of a subject’s health follows a stochastic process and failure occurs when the processﬁrst reaches a failure state or threshold (a ﬁrst hitting time).
This requires underlying parametricstochastic process models for the joint evolution of the time varying observations and the time to event[13, 19].
More recent work has applied modern deep learning approaches to this problem [14, 17, 15].
Corresponding work also exists in the area of machine failure prediction [24].
The relevant papers toour work are the recurrent neural network approaches [17, 24] that operate on longitudinal data topredict the probability distribution of failure time given previous history of co-variates for any subject,at each observation time.
While these methods are mostly complementary to the primary goal of thispaper where we focus on obtaining optimal intervention policies, we do propose improvements in tothese methods in Section 4.1 to overcome the distributional assumptions in [24] and difﬁculty withlong sequences [17].
In practice, intervention policies based on survival analysis are typically justthresholds on the predicted time-to-event.
Optimal stopping problems: Optimal stopping problems [37, 8, 3] are a subclass of stochasticcontrol problems with only two possible actions:stop and continue.
The goal is to stop takingobservations when the current reward (or cost) exceeds (or falls below) expected future reward(or cost).
Optimal stopping techniques have been applied to several problems such as sequentialhypothesis testing [37, 10, 6], early change point detection [27, 35, 34, 30] and early classiﬁcation[23].
Optimal stopping policies for machine maintenance have been studied in reliability theory[5, 22], where the goal is to ﬁnd the best stopping time to replace/maintain a machine based oncondition.
However, the approaches in such cases are limited to simple probabilistic models ofmachine degradation with a single or few co-variates.
General model-free reinforcement learningapproaches to the optimal stopping problem are generally feasible in situations where we have a lotof data or in a model-based setting where the underlying randomness can efﬁciently be simulatedand the model can be used to draw Monte-Carlo samples [7].
This is not an option in most criticalevent prediction settings when the critical event data is often scarce and no simple model exists forthe evolution of the co-variates.
1.4 Outline of contributions
The OTI problem is hard since system evolution towards a critical event is a non-stationary in generalwith typically scarce occurences of critical events.
Current modeling approaches are challenged eitherbecause they make signiﬁcant overly simplifying assumptions on the underlying process or becausescarce events precludes powerful model-free approaches.
To our knowledge, we are the ﬁrst to frame the OTI problem and link it to both survival analysis andoptimal stopping theory.
This allows, for the ﬁrst time, the ability to model complex non-stationarysystem evolution towards scarce critical events with the ability to intervene optimally.
• A process under observation may exhibit degradation/disorder as it evolves towards a
critical event.
We capture this feature by introducing the concept of a survival (or hazard
rate) process, which is a stochastic process modeling the evolution of conditional survival functions as new observations are taken.
A key contribution of our work is to formalize the OTI problem in equation (OTI problem) and show that the OTI problem is equivalent to anoptimal stopping problem on the underlying survival (or hazard rate) process (Proposition2.1).
This allows us to tap into optimal stopping theory [37, 8, 3] to unlock the structure ofOTI policies and also leverage computational methods of estimating/learning survival/hazardrate functions.
These are key to our development of a general machine learning approach tothe OTI problem.• Under Markovian assumptions on the survival process, we show that the optimal interventionpolicy decision rule at each time step can be analytically determined by the conditionalsurvival (or equivalently, hazard rate) function at that time step (Theorem 3.1).
Further weshow that the optimal policy is non-stationary in general and reduces to a variable thresholdon the expected time-to-event(Corollary 3.1.2), showing that static threshold policies aresub-optimal in general.•
We show that our theory applies to an important class of survival and hazard rate processes
generated by recurrent neural networks (Section 4).
We term these processes Neural survivaland hazard rate processes respectively.
This allows us to apply survival analysis methods to
learn survival processes and analytically compute optimal intervention policies for criticalevents.
We adapt the state of the art deep dynamic survival analysis (DRSA) work [29]to our setting.
Unlike DRSA, our dynamic DRSA (DDRSA) setting requires modeling
dynamic survival and hazard rate functions.
We do this by adapting the DRSA architecture to our dynamic covariate evolution setting using an encoder-decoder seq2seq architecture.
OTI as an optimal stopping problem on a survival or hazard rate process
In this section we establish a link between the OTI problem, survival analysis and optimal stoppingtheory.
We require a few deﬁnitions. 2.1 Preliminaries We model the degradation of the monitored stochastic process towards the critical event by thecorresponding evolution of underlying survival functions or hazard rate functions conditioned onobservation history.
To this end, we will use the following deﬁnitions of a conditional hazard ratefunction h j ( · ) , and survival function S j (
[Equation]
[Equation] Note that we have omitted the implied conditioning on the the hazard and survival functions, sothat h j ( k |X j ) is written as h j ( k ) for convenience.
Also, we have h j ( − 1) :
= 0 and S j ( − 1) : =
1reﬂecting the fact that the critical event has not occured until time step j .
We now deﬁne the conceptof survival and hazard rate processes:
Deﬁnition 2.1 (Survival Process) .
Consider the random vector constructed from the survival function
S j ( · ) given by S j = [ S j (0) , S j (1) · · · S j ( L max − 1)] T .
We call the sequence of tuples { ( X j , S j ) }a survival process.
Deﬁnition 2.2 (Hazard Rate Process) .
Consider the random vector constructed from the conditionalhazard rate function h j ( · ) given by H j =
[ h j (0) , h j (1) · · · h j ( L max − 1)] T .
We call the sequenceof tuples { ( X j , H j ) } a hazard rate process.
Thus, survival and hazard rate processes model the relation between the evolution of co-variates andonset of the critical event.
2.2 Optimal intervention on a survival or hazard rate process
Proposition 2.1 (Optimal intervention on a survival or hazard rate process) .
Given a survival process{ ( X j , S j ) } or a hazard rate process { ( X j , H j ) } , the intervention risk at a stopping time τ is givenequivalently by:
[Equation]
In this setting, the optimal critical event intervention problem of equation (OTI problem) reduces tothe following optimal stopping problems 3 on the corresponding survival or hazard rate processes:
[Equation]
The proof is established in Appendix A.[-Startingwith terms in the OTI problem: for expected residual life we may write ( L − τ ) + = (cid:80) k ≥ 0 
-]
[-E [ (cid:80) k ≥ 
-]
[-[ (cid:80) k ≥ 
-]
The ﬁrst term in these risks may be viewed as the cost ofintervention when the critical event does not occur at time τ , while the second term is the cost ofintervention when the critical event does occur at time τ .
Analytic solution to the OTI problem for Markov survival processes
In order to solve the OTI optimal stopping problem we make the assumption that the survival andhazard rate processes we deal with satisfy the Markov property:
Deﬁnition 3.1 (Markov survival or hazard rate process) .
Consider the survival process { ( X j , S j ) }or the hazard rate process { ( X j , H j ) } .
These are said to be Markov survival and Markov hazard rateprocesses respectively, if we have:
[Equation]
Further, the process is termed homogeneous if P ( X j +1 | X j ) does not depend on j .
With this assumption in place we can state the central result of this paper:
Theorem 3.1 (OTI policy for a Markov hazard rate process) .
Given a Markov hazard rate process{ ( X j , H j ) } and associated derived risk process risk process { R Hj ( H j ) } , the optimal interventionpolicy that solves the problem posed in Proposition 2.1 is a decision rule that calls for intervention atstopping time τ ∗ such that:
[Equation]
6) for a sequence of value functions
V j ( H j ) given by:
[Equation]
Further if the Markov hazard rate process is homogeneous the optimal policy is stationary, with valuefunction V ( H j ) that does not depend on j .
The proof of this theorem is given in Appendix A.
The following corollary establishing a similarresult for Markov survival processes follows: Corollary 3.1.1 (OTI policy for a Markov survival process) .
Given a Markov survival process{ ( X j , S j ) } and associated derived risk process risk process { R Sj ( S j ) } , the optimal interventionpolicy that solves the problem posed in Proposition 2.1 is a decision rule that calls for intervention atstopping time τ ∗ such that:
[Equation]
for a sequence of value functions V j ( S j ) given by: [Equation]
Further if the Markov survival process is homogeneous the optimal policy is stationary, with valuefunction V ( S j ) that does not depend on j .
This theorem tells us that in the case of a Markov survival or hazard rate processes, the optimal policymay be constructed explicitly with knowledge of only the conditional survival or hazard rate functionsat that time.
The proof of the above theorem leverages fundamental results in optimal stopping theory[8, 3] and is given in Appendix A.
The value function V j ( · ) gives the minimum expected cost of continuing to take observations, having obtained a conditional survival ( S j ) or hazard rate function( H j ) at time j .
It is clear that the the optimal policy is non-stationary in general.
Also note thatputting l = 0 in the value function recovers the instantaneous risk at time step j as the ﬁrst term overwhich the minimum is computed in the value function.
The value function gives the best stoppingrisk over all stopping decisions in the present and future.
The following Corollary allows a furthersimpliﬁcation of the optimal policy:
Corollary 3.1.2 (OTI policy as a time varying threshold on time-to-event) .
Let T j be the expectedtime to event at time j .
The OTI policy of Theorem 3.1 may be simpliﬁed to the following form:
[Equation]
10) where
[Equation]
with T j given by:
[Equation]
The proof is given in Appendix A.
This Corollary clearly shows that static threshold TTE policies aresub-optimal in general.
The value functions in these results contain two terms.
The ﬁrst term, whichis the risk of stopping early before the critical event which intuitively gets smaller as j progressestoward the critical event.
The second term is the risk of missing the critical event that depends on thecost of missing the event.
It is high where the onset of the critical event is expected.
Neural survival processes
The previous theory applied to Markov survival processes. We show that this is not a signiﬁcantrestriction.
In fact, the class of survival and hazard rate processes that can be modeled by recurrentneural networks (RNNs) also satisfy our Markovian assumptions. To see this, note that RNNs maybe viewed as a feature transformation, mapping X j to features Z j = { Z s : 1 ≤ s ≤ j } where Z jis the hidden state vector of the RNN at time step j .
If we output survival or hazard rate vectors asfunctions of the hidden state we generate neural survival and hazard rate process given by { ( Z j , S j ) }or { ( Z j , H j ) } respectively.
Due to the recurrent dynamics of the RNN, the ﬁnal state encode allhistory, therefore P ( Z j +1 |Z j ) =
P ( Z j +1 | Z j ) and the processes are Markovian and stationary (sincethe processes depend entirely on the hidden state evolution and not time-step).
However, RNN modelscan be made non-stationary by encoding time into the inputs.
4.1 Dynamic deep recurrent survival analysis (DDRSA)
Since the OTI theory developed in the previous section directly applies to neural survival processes,we may ﬁt a neural survival process model to data and then apply the optimal OTI policy to intervene.
In the static co-variate setting, Ren et.
al.
[29] developed a deep recurrent neural architecturefor survival analysis. In this work the authors use an RNN to map a static feature vector X 0 toa conditional hazard rate vector H 0 ) by using an RNN that generates a sequence of hidden states{ Z 0 k } .
The k th hidden state vector Z 0 k is mapped through a dense layer D with a sigmoid activationto produce conditional hazard rate h 0 ( i ) = σ ( D ( Z 0 k )) . The model is ﬁt using a novel sequentialloss, producing state of the art survival analysis results without requiring assumptions on distributionof time-to-event probabilities [29].
Unlike DRSA, our dynamic DRSA (DDRSA) setting requires modeling dynamic survival and hazardrate functions.
We do this by adapting the DRSA architecture to our dynamic covariate evolutionsetting.
While there are other attempts at modeling the dynamic evolution of survival functions[17, 24] they either make distributional assumptions [24] or have difﬁculty with long sequences dueto large softmax layer [17].
Our adaptation of DRSA uses an encoder RNN that maps co-variatehistory X j to an single encoder hidden state vector Q j .
This vector is then passed into a decoderwhich has a standard DRSA architecture, producing decoder hidden states Z jk and correspondingconditional hazard rates h j ( k )
=
σ ( D ( Z jk )) .
The model is then learnt end-to-end using the DRSAloss.
DDRSA can be seen as a generative model for a Markov hazard rate process.
We use the DDRSA architecture in all our experiments. [Figure] [Figure]
[Figure]
h a z a r d r a t e j=L- [Figure]
j=
Lj=L-
[Figure]
j=Lj=-L- (a) Progression of survival functions       350 time steps from j 0.
s u r v
i v a l p r o b a b ili t y j=
L-200 j =
Lj=L-50 j=Lj=L-
[Figure]
hazard rates
Experiments
The goal of this section is twofold:1.
To complement the theory developed in the earlier sections with examples to validate andbuild intuition into survival and hazard rate processes how the optimal policy works toproduce interventions.
(Section 5.1)2.
To demonstrate that the theory does indeed translate into an effective practical methodologyembodied by the DDRSA-RNN that can model survival processes and generate interventionsusing the OTI policy.
We call this combined approach the OTI-DDRSA-RNN.
We showthat OTI-DDRSA-RNN outperforms static threshold policies that are either window based(WBI-RNN) or time-to-event based (TTE-RNN)(Section 5.2).
Experiments are performed on two real-world datasets to validate our goals:
Turbofan Engine Failure Data [26, 32]: This dataset from NASA provides train and test data topredict Turbofan engine failures.
The training set has simulated multivariate time-series observationsfrom 21 sensors on a ﬂeet of 218 engines.
Each engine operates normally at the start and develops afault at some point resulting in eventual failure.
The test set consists of 218 partial engine time-serieswith remaining useful life (in engine cycles) indicated in each case.
We could not use the test dataas provided since it is missing part of every time series close to failure.
We standardized the sensorobservations to prepare them for input into our models.
Azure Predictive Maintenance Guide Data[25]: This is a dataset from a guide provided by Microsoft as a case study in failure prediction.
The data comes from multiple sources includingtime-series of voltage, rotation, pressure and vibration measurements collected from 100 machinesin real time averaged over every hour, error logs, machine information (type, age etc.) and failurerecords indicating four different component failures.
We adopt the same pre-processing, featuregeneration as given in the guide.
Also, we downsample the data to take one sample every 15 hours.
For both datasets, we train on 70% of randomly selected co-variate time-series sequences and holdout 30% of the sequences for testing.
From the training set a further 30% of the sequences are setaside as validation data to tune model parameters and policy thresholds.
This process is repeated toproduce 5 random splits.
Model selection details and parameter settings are discussed in Appendix B.
5.1 Discussion of OTI interventions on real data
We ﬁt the DDRSA-RNN on the covariate time-series leading upto failure and predict on a single testset sequence.
This produces, at each time step j of the sequence, an estimated conditional hazard ratevector H j (representing the conditional future hazard rates), which gives us a probabilistic estimateof the onset of the critical event at each time step, taking into account co-variate history upto that [Figure] time step.
The conditional hazard rates may then be converted into a corresponding survival function
S j at each time step.
Figure 1 shows the evolution of estimated conditional hazard rate functions andsurvival functions.
We see that the onset of the critical event is clearly reﬂected in the progression ofthe estimated vectors H j and S j .
Next, we demonstrate the working of the OTI-policy, which was shown in Section 3 to be an adaptivethreshold on the estimated time to event.
Figure 2 plots the estimated time-to-event (estimated from H j or S j on two sequences as we progress toward the critical event.
We also plot the adaptivethreshold given by the OTI policy, calculated using Corrolary 3.1.2.
An intervention is triggeredwhen our time-to-event estimate falls on or below the adaptive threshold.
Consider what happens as the cost of missing the critical event is varied.
First, note that sincetime-to-event is independent of C , only the threshold changes.
On both sequences we see thatincreasing the cost of missing the critical event increases the threshold causing earlier intervention aswe would expect.
Also, note how the threshold adapts to the sequence.
For example at the end ofsequence B the threshold (which signiﬁes continuation risk) rises rapidly while this feature is notobserved on sequence A which is subject to different co-variate observations.
5.2 Performance Evaluation
In this section we will evaluate the performance of the proposed OTI-DDRSA-RNN vs. two staticthreshold policy baselines:
A window based intervention RNN (WBI-RNN) and a time-to-eventbased intervention RNN (TTE-RNN).
All RNNs share the same encoder architecture which is an
LSTM with 128 step lookback and hidden state dimension of 16 units.
WBI-RNN: The WBI-RNN uses a dense layer with a sigmoid activation to convert co-variate historyencoded in the encoder ﬁnal hidden-state into a probability that the critical event appears within awindow of size W .
A threshold is used on this probability to trigger interventions once the probabilityis greater than the threshold.
The window size and threshold are selected by cross-validation (see Apendix B).
TTE-RNN: The TTE-RNN uses the same DDRSA-RNN architecture, sharing both encoder anddecoder.
However, interventions are trigerred with a static threshold that is set via cross-validationsimilar to the WBI-RNN.
All methods are evaluated using an empirical intervention policy risk (of policy ϕ ) deﬁned over arange of costs:
[Equation]
[Figure]
where ( L = L i , τ = τ i ) are realized outcomes for the i th series.
All methods were trained on training splits and tuned on validation splits.
Policy thresholds for the
WBI-RNN and TTE-RNN that were determined on the different validation splits are averaged to get aﬁnal threshold setting. Allmodels are then re-trained on the entire training data (including validationsplits) and tested on the test splits.
Model parameter initialization seeds were also randomizedbetween splits. Results reported are averaged over the 5 test splits.
Figure 3 summarizes the evaluation results.
We see that the OTI-DDRSA-RNN outperforms the staticthreshold policies on both datasets having lower empirical intervention risk over a range of costs(Figure 3(a) and Figure 3(c)) reﬂected in a better tradeoff between time-to-event and critical eventmiss-rate.
Regarding the tradeoff plots, a lower frontier is preferred, since it imples a lower time-toevent for a given miss-rate and a lower miss-rate for a given time-to-event.
On the NASA dataset the
TTE-RNN outperformed WBI-RNN while on the Azure PM data the WBI policy performed better.
Limitations Since both TTE and OTI policies are model based, the quality of the policies depend onthe accuracy of the underlying survival analysis.
A key challenge in this domain is the scarcity ofpublicly available datasets.
Supplemental Material The Appendix included with this submission as supplemental materialincludes complete proofs of all theorems, and further details on the experiments including modelselection.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[TODO]
YES
(b) Did you describe the limitations of your work?
[TODO]
YES
(c) Did you discuss any potential negative societal impacts of your work?
[TODO]
Noobvious negative societal impact
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[TODO]
YES
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[TODO] YES
(b) Did you include complete proofs of all theoretical results?
[TODO] YES, in supple-
mental material
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[TODO] YES
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[TODO] YES
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[TODO] YES, included in supplemental material
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[TODO] YES, included in supplemental
material
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[TODO] YES
(b) Did you mention the license of the assets?
[TODO] YES, links to data provided in the
paper have license information
(c) Did you include any new assets either in the supplemental material or as a URL?
[TODO]
NO
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[TODO] Not applicable.
Only free license resources used
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[TODO] Not applicable
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[TODO] Not applicable
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[TODO] Not applicable
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[TODO] Not applicable