Original: N EURAL D ISTRIBUTION L EARNING FOR G ENERALIZED T IME - TO -E VENT P REDICTION
Revised: N EURAL D ISTRIBUTION L EARNING FOR G ENERALIZED T IME - TO -E VENT P REDICTION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Predicting the time to the next event is an important task in various domains.
Revised: Predicting the time to the next event is an important task in various domains.

Original: However, due to censoring and irregularly sampled sequences, time-to-event prediction has resulted in limited success only for particular tasks, architectures and data.
Revised: However, due to censoring and irregularly sampled sequences, time-to-event prediction has resulted in limited success only for particular tasks, architectures and data.

Original: Using recent advances in probabilistic programming and density networks, we make the case for a generalized parametric survival approach, sequentially predicting a distribution over the time to the next event.
Revised: Using recent advances in probabilistic programming and density networks, we make the case for a generalized parametric survival approach, sequentially predicting a distribution over the time to the next event.

Original: Unlike previous work, the proposed method can use asynchronously sampled features for censored, discrete, and multivariate data.
Revised: Unlike previous work, the proposed method can use asynchronously sampled features for censored, discrete, and multivariate data.

Original: Furthermore, it achieves good performance and near perfect calibration for probabilistic predictions without using rigid network-architectures, multitask approaches, complex learning schemes or non-trivial adaptations of coxmodels.
Revised: Furthermore, it achieves good performance and near perfect calibration for probabilistic predictions without using rigid network-architectures, multitask approaches, complex learning schemes or non-trivial adaptations of coxmodels.

Original: We ﬁrmly establish that this can be achieved in the standard neural network framework by simply switching out the output layer and loss function.
Revised: We ﬁrmly establish that this can be achieved in the standard neural network framework by simply switching out the output layer and loss function.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Many real-world tasks can be formulated as time-to-event (TTE) prediction problems, forecasting the time taken until an event-of-interest happens in the future.
Revised: Many real-world tasks can be formulated as time-to-event (TTE) prediction problems, forecasting the time taken until an event-of-interest happens in the future.

Original: Examples are predicting time to the onset of a particular disease (Price et al., 2017), predicting when a machine fails (Salfner et al., 2010) or predicting future user logins (Sobaszek & Gola, 2016).
Revised: Examples are predicting time to the onset of a particular disease (Price et al., 2017), predicting when a machine fails (Salfner et al., 2010) or predicting future user logins (Sobaszek & Gola, 2016).

Original: Unfortunately, the data often consists of temporal features and recurrent events which are sparsely and irregularly sampled over time.
Revised: Unfortunately, the data often consists of temporal features and recurrent events which are sparsely and irregularly sampled over time.

Original: This makes the data challenging to work with.
Revised: This makes the data challenging to work with.

Original: Furthermore, the main challenge is that if we do not observe a terminating event for a sequence, the time to event becomes right censored (Klein &
Revised: Furthermore, the main challenge is that if we do not observe a terminating event for a sequence, the time to event becomes right censored (Klein &

Original: Moeschberger, 2005).
Revised: Moeschberger, 2005).

Original: This essentially means that we only know that the event will take place later than when it was known to not occur.
Revised: This essentially means that we only know that the event will take place later than when it was known to not occur.

Original: Proposed solutions taking account for censoring includes predicting TTE binned across ﬁxed time windows (classiﬁcation approaches (Harutyunyan et al., 2017; Lee et al., 2018)), predicting TTE point-wise (regressive approaches (Ishwaran et al., 2008)), ranking the risk of subjects (semiparametric (Katzman et al., 2016; Luck et al., 2017) and non-parametric (Kalderstam, 2015; Chen et al., 2013)), or estimating the target and the feature distributions jointly (variational, stochastic formulations (Soleimani et al., 2017; Ranganath et al., 2016; Mei & Eisner, 2017; Xu et al., 2017)).
Revised: Proposed solutions taking account for censoring includes predicting TTE binned across ﬁxed time windows (classiﬁcation approaches (Harutyunyan et al., 2017; Lee et al., 2018)), predicting TTE point-wise (regressive approaches (Ishwaran et al., 2008)), ranking the risk of subjects (semiparametric (Katzman et al., 2016; Luck et al., 2017) and non-parametric (Kalderstam, 2015; Chen et al., 2013)), or estimating the target and the feature distributions jointly (variational, stochastic formulations (Soleimani et al., 2017; Ranganath et al., 2016; Mei & Eisner, 2017; Xu et al., 2017)).

Original: Numerous works have used combinations of these approaches (Du et al., 2016; Li, 2017; Xiao et al., 2017b;a; Harutyunyan et al., 2017; Lee et al., 2018).
Revised: Numerous works have used combinations of these approaches (Du et al., 2016; Li, 2017; Xiao et al., 2017b;a; Harutyunyan et al., 2017; Lee et al., 2018).

Original: However, most existing solutions have problems with tight couplings to particular types of queries, restrictive and data-dependent loss formulations, complex model architectures, or improper handling of censored data.
Revised: However, most existing solutions have problems with tight couplings to particular types of queries, restrictive and data-dependent loss formulations, complex model architectures, or improper handling of censored data.

Original: While there has been no lack of novelty in the proposed solutions, we ﬁnd that the seemingly most straightforward generalized parametric survival approach has lacked a thorough treatment.
Revised: While there has been no lack of novelty in the proposed solutions, we ﬁnd that the seemingly most straightforward generalized parametric survival approach has lacked a thorough treatment.

Original: In this paper, we formulate the TTE prediction task as, in each timestep, predicting all parameters of some discretized probability distribution over the current time to the next event.
Revised: In this paper, we formulate the TTE prediction task as, in each timestep, predicting all parameters of some discretized probability distribution over the current time to the next event.

Original: We show the beneﬁts of deﬁning distributions in terms of cumulative hazard functions (Sec. 3.1).
Revised: We show the beneﬁts of deﬁning distributions in terms of cumulative hazard functions (Sec. 3.1).

Original: Finally, we show that the regular negative log-likelihood for right-censored data, censored log-likelihood (known at least since de Moivre (1731)) as the loss function is sufﬁcient to yield near perfect calibration.
Revised: Finally, we show that the regular negative log-likelihood for right-censored data, censored log-likelihood (known at least since de Moivre (1731)) as the loss function is sufﬁcient to yield near perfect calibration.

Original: All together, this leads to an obvious yet novel framework for general-purpose TTE prediction.
Revised: All together, this leads to an obvious yet novel framework for general-purpose TTE prediction.

Original: We will refer to this as HazardNet .
Revised: We will refer to this as HazardNet .

Original: While generalizing previous work, it is easy to implement and evaluate while being capable of yielding real-time predictions on various inference queries, such as predicting the probability of a user returning in 30 days, the expected time to failure or median time of survival.
Revised: While generalizing previous work, it is easy to implement and evaluate while being capable of yielding real-time predictions on various inference queries, such as predicting the probability of a user returning in 30 days, the expected time to failure or median time of survival.

Original: In section 4 we show that the model makes unbiased, calibrated probabilistic predictions and efﬁciently utilizes training data.
Revised: In section 4 we show that the model makes unbiased, calibrated probabilistic predictions and efﬁciently utilizes training data.

Original: This is shown by evaluating on three publicly available datasets comparing multiple TTE-distributions and neural network architectures.
Revised: This is shown by evaluating on three publicly available datasets comparing multiple TTE-distributions and neural network architectures.

Original: For the purpose of establishing a strong baseline, we propose a novel evaluation scheme, designed for real-life applications.
Revised: For the purpose of establishing a strong baseline, we propose a novel evaluation scheme, designed for real-life applications.

Original: In addition, in section 5 we reformulate the binary task of musical onset detection into a multivariate distribution prediction task, achieving state of the art results.
Revised: In addition, in section 5 we reformulate the binary task of musical onset detection into a multivariate distribution prediction task, achieving state of the art results.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: There is a diverse array of problem formulations for TTE-prediction (Wang et al., 2017).
Revised: There is a diverse array of problem formulations for TTE-prediction (Wang et al., 2017).

Original: The methods for censored data are mainly based on classical semi-parametric Cox-models for continuous target values (Katzman et al., 2016; Luck et al., 2017; Joshi & Reeves, 2006).
Revised: The methods for censored data are mainly based on classical semi-parametric Cox-models for continuous target values (Katzman et al., 2016; Luck et al., 2017; Joshi & Reeves, 2006).

Original: Others formulate it as classiﬁcation problems (Harutyunyan et al., 2017; Lee et al., 2018) or multi-task learning (Luck et al., 2017; Lee et al., 2018) by predicting the event probability in each step for a ﬁxed number of steps τ ahead together with a ranking loss.
Revised: Others formulate it as classiﬁcation problems (Harutyunyan et al., 2017; Lee et al., 2018) or multi-task learning (Luck et al., 2017; Lee et al., 2018) by predicting the event probability in each step for a ﬁxed number of steps τ ahead together with a ranking loss.

Original: Non-parametric learning-to-rank methods have been proposed (Kalderstam, 2015) but they suffer from scalability problems.
Revised: Non-parametric learning-to-rank methods have been proposed (Kalderstam, 2015) but they suffer from scalability problems.

Original: Some studies (Du et al., 2016; Li, 2017; Xiao et al., 2017b;a) predict both TTE and classiﬁcation of next event jointly using composition of loss functions.
Revised: Some studies (Du et al., 2016; Li, 2017; Xiao et al., 2017b;a) predict both TTE and classiﬁcation of next event jointly using composition of loss functions.

Original: Others have learned from censored data by predicting features and target jointly, either using Gaussian processes (Soleimani et al., 2017) or deep exponential families (Ranganath et al., 2016).
Revised: Others have learned from censored data by predicting features and target jointly, either using Gaussian processes (Soleimani et al., 2017) or deep exponential families (Ranganath et al., 2016).

Original: Recent extensions of this work (Miscouridou et al.) includes an interesting discussion on methods for approximating arbitrary distributions, discretization, evaluation and missing feature data.
Revised: Recent extensions of this work (Miscouridou et al.) includes an interesting discussion on methods for approximating arbitrary distributions, discretization, evaluation and missing feature data.

Original: Sequential prediction is often formulated as asynchronously predicting at the same time as the event (Du et al., 2016; Li, 2017; Xiao et al., 2017b;a; Lee et al., 2018;
Revised: Sequential prediction is often formulated as asynchronously predicting at the same time as the event (Du et al., 2016; Li, 2017; Xiao et al., 2017b;a; Lee et al., 2018;

Original: Avati et al., 2018).
Revised: Avati et al., 2018).

Original: However, this makes it unclear how to make predictions between events.
Revised: However, this makes it unclear how to make predictions between events.

Original: For features arriving at different times, it has been proposed to use two recurrent neural networks (RNNs) acting on different timescales (Xiao et al., 2017b), but it did not deal with censored data.
Revised: For features arriving at different times, it has been proposed to use two recurrent neural networks (RNNs) acting on different timescales (Xiao et al., 2017b), but it did not deal with censored data.

Original: For the general problem of unevenly spaced sequences, there has been other notable successful neural network approaches such as Phased LSTM (Neil et al., 2016) or Time-LSTM (Zhu et al.).
Revised: For the general problem of unevenly spaced sequences, there has been other notable successful neural network approaches such as Phased LSTM (Neil et al., 2016) or Time-LSTM (Zhu et al.).

Original: Limitations: Most solutions are designed speciﬁcally for each task, such as rank prediction (Katzman et al., 2016; Luck et al., 2017; Kalderstam, 2015; Chen et al., 2013), classiﬁcation (Harutyunyan et al., 2017; Lee et al., 2018), and more.
Revised: Limitations: Most solutions are designed speciﬁcally for each task, such as rank prediction (Katzman et al., 2016; Luck et al., 2017; Kalderstam, 2015; Chen et al., 2013), classiﬁcation (Harutyunyan et al., 2017; Lee et al., 2018), and more.

Original: When using stochastic formulations (Soleimani et al., 2017;
Revised: When using stochastic formulations (Soleimani et al., 2017;

Original: Ranganath et al., 2016; Mei & Eisner, 2017; Xu et al., 2017), inference requires extra steps and are arguably made for other tasks (such as missingness of data or understanding feature importance).
Revised: Ranganath et al., 2016; Mei & Eisner, 2017; Xu et al., 2017), inference requires extra steps and are arguably made for other tasks (such as missingness of data or understanding feature importance).

Original: Without taking into account censored data (Du et al., 2016; Li, 2017; Xiao et al., 2017b;a) probabilistic predictions might have little meaning.
Revised: Without taking into account censored data (Du et al., 2016; Li, 2017; Xiao et al., 2017b;a) probabilistic predictions might have little meaning.

Original: Most models are based on strong assumptions about data or underlying distributions, and are difﬁcult to adapt to new problems.
Revised: Most models are based on strong assumptions about data or underlying distributions, and are difﬁcult to adapt to new problems.

Original: The pure parametric survival approaches based on density networks either focused on speciﬁc distributions (Martinsson, 2016; Avati et al., 2018) or made cases against the use of regular log-likelihood (Avati et al.,  2018).
Revised: The pure parametric survival approaches based on density networks either focused on speciﬁc distributions (Martinsson, 2016; Avati et al., 2018) or made cases against the use of regular log-likelihood (Avati et al., 2018; Ren et al., 2018).

Original: While there has been recent interest in the importance of calibration (Luck et al., 2017; Avati et al., 2018), we argue for a different method of evaluating it in a real-world setting, and most importantly we want to establish a baseline to compare against whether the regular log-likelihood loss needs improvement at all.
Revised: While there has been recent interest in the importance of calibration (Luck et al., 2017; Avati et al., 2018), we argue for a different method of evaluating it in a real-world setting, and most importantly we want to establish a baseline to compare against whether the regular log-likelihood loss needs improvement at all.

Original: Our contribution: Building on the work of Martinsson (2016), we present a ﬂexible and generalizable framework for time to event prediction that can work with multiple model architectures and distributions, supporting various probabilistic inference queries and handle asynchronously arriving features.
Revised: Our contribution: Building on the work of Martinsson (2016), we present a ﬂexible and generalizable framework for time to event prediction that can work with multiple model architectures and distributions, supporting various probabilistic inference queries and handle asynchronously arriving features.

Original: We found no in-depth experiments or convincing results on how to evaluate performance, especially calibration for probabilistic predictions.
Revised: We found no in-depth experiments or convincing results on how to evaluate performance, especially calibration for probabilistic predictions.

Original: To this end, we propose an evaluation strategy in Section 4.
Revised: To this end, we propose an evaluation strategy in Section 4.

Original: We found no general discussions on learning parametric survival distributions in this setting nor any in depth examples using the Weibull , Pareto , LogLogistic ,  or other distributions as discretized forms.
Revised: We found no general discussions on learning parametric survival distributions in this setting nor any in depth examples using the Weibull , Pareto , LogLogistic , their mixtures, or other distributions as discretized forms.

Original: By thoroughly investigating what ought to be a standard approach and suggesting evaluation methods for it, we hope to establish a strong baseline for time to event prediction.
Revised: By thoroughly investigating what ought to be a standard approach and suggesting evaluation methods for it, we hope to establish a strong baseline for time to event prediction.

Original: The implementation will be released online.
Revised: The implementation will be released online.

Original:  H AZARD N ET
Revised:  H AZARD N ET

Original: While many previous approaches have focused on speciﬁc data distributions or tasks, our goal is to provide a general framework by focusing on the problem formulation.
Revised: While many previous approaches have focused on speciﬁc data distributions or tasks, our goal is to provide a general framework by focusing on the problem formulation for TTE-prediction itself.

Revised: We frame it as a Distribution Learning problem.

Original: The basic idea is a density network coupled with a parametric survival approach.
Revised: The basic idea is a density network coupled with a parametric survival approach.

Original: Let a neural network predict parameters of some TTE-distribution and train it using log-likelihood for censored data.
Revised: Let a neural network predict parameters of some TTE-distribution and train it using log-likelihood for censored data.

Original: As shown in Figure 1, in every step t we let the predictor map features x t to predicted parameters θ t , governing the shape of a distribution.
Revised: As shown in Figure 1, in every step t we let the predictor map features x t to predicted parameters θ t , governing the shape of a distribution.

Original: The distribution is speciﬁed by ﬁxing a functional form of a cumulative hazard function Λ .
Revised: The distribution is speciﬁed by ﬁxing a functional form of a cumulative hazard function Λ .

Original: In the following sections we will show how to use discrete target values and the possibility of learning to approximate arbitrarily complex distributions.
Revised: In the following sections we will show how to use discrete target values and the possibility of learning to approximate arbitrarily complex distributions.

Original: 3. E FFICIENTLY W ORKING WITH D ISTRIBUTIONS USING C UMULATIVE H AZARD F UNCTIONS
Revised: 3. E FFICIENTLY W ORKING WITH D ISTRIBUTIONS USING C UMULATIVE H AZARD F UNCTIONS

Original: We will employ the notational convenience of using cumulative hazard functions (CHF) to deﬁne probability distributions.
Revised: We will employ the notational convenience of using cumulative hazard functions (CHF) to deﬁne probability distributions.

Original: In the survival analysis context, it is common to focus on hazard functions , but we found it more efﬁcient to focus on its integral, deﬁned in the following.
Revised: In the survival analysis context, it is common to focus on hazard functions , but we found it more efﬁcient to focus on its integral, deﬁned in the following.

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: A cumulative hazard function Λ : R + → R + is a monotonically increasing positive function such that for all x ≥ 0 , (cid:15) ≥ 0 ,
Revised: A cumulative hazard function Λ : R + → R + is a monotonically increasing positive function such that for all x ≥ 0 , (cid:15) ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original: where the hazard function λ ( x )
Revised: where the hazard function λ ( x )

Original: = ∂∂x Λ( x ) is its positive derivative if it exists.
Revised: = ∂∂x Λ [Equation] is its positive derivative if it exists.

Original: A cumulative hazard function Λ( x ) is a straightforward way of representing a positive distribution.
Revised: A cumulative hazard function Λ( x ) is a straightforward way of representing a positive distribution.

Original: If X is a random variable, we can write its corresponding cumulative density function as F ( x ) =
Revised: If X is a random variable, we can write its corresponding cumulative density function as F ( x ) =

Original: Pr ( X < x ) = 1 − e − Λ( x ) or conversely, Λ ≡ − log(1 − F ) .
Revised: Pr ( X < x ) = 1 − e − Λ( x ) or conversely, Λ ≡ − log(1 − F ) .

Original: Its probability density function can be written f ( x ) =
Revised: Its probability density function can be written f ( x ) =

Original: ∂∂x F ( x ) =
Revised: ∂∂x F ( x ) =

Original: λ ( x ) e − Λ( x ) .
Revised: λ ( x ) e − Λ( x ) .

Original: If we let the random variable X represent a life span, the survival function S ( x ) =
Revised: If we let the random variable X represent a life span, the survival function S ( x ) =

Original: Pr( X > x ) = 1 − F ( x ) =
Revised: Pr( X > x ) = 1 − F ( x ) =

Original: e − Λ( x ) is the probability of surviving until time x .
Revised: e − Λ( x ) is the probability of surviving until time x .

Original: Cumulative Hazard functions makes a good abstraction for implementation since the conditions of
Revised: Cumulative Hazard functions makes a good abstraction for implementation since the conditions of

Original: Equation 1 are easily veriﬁable.
Revised: Equation 1 are easily veriﬁable.

Original: It also simpliﬁes transition from continuous to discrete distributions, an important consideration since real-world data tend to be discrete.
Revised: It also simpliﬁes transition from continuous to discrete distributions, an important consideration since real-world data tend to be discrete.

Original: Given Λ( x ) , we can easily form a discrete distribution by deﬁning a probability mass function p ( t ) =
Revised: Given Λ( x ) , we can easily form a discrete distribution by deﬁning a probability mass function p ( t ) =

Original: F ( t ) − F ( t + 1) =
Revised: S ( t ) − S ( t + 1) =

Original: [Table]
Revised: [Table]

Original: e − Λ( t ) − e − Λ( t +1) for t = 1 , 2 , . . .
Revised: e − Λ( t ) − e − Λ( t +1) for t = 1 , 2 , . . .

Original: , ∞ .
Revised: , ∞ .

Original: This also makes it easy to use different levels of discretization (i.e., days to weeks).
Revised: This also makes it easy to use different levels of discretization (i.e., days to weeks).

Original: In doing so, instead of using a continuous loss function after discretization, we can learn the discretized distribution directly, which in turn can be used to approximate a continuous distribution if needed.
Revised: In doing so, instead of using a continuous loss function after discretization, we can learn the discretized distribution directly, which in turn can be used to approximate a continuous distribution if needed.

Original: Example The exponential distribution f ( x ) = 1 α e − xα can be deﬁned with the CHF Λ Exponential ( x ) = xα .
Revised: Example The exponential distribution f ( x ) = 1 α e − xα can be deﬁned with the CHF Λ Exponential ( x ) = xα .

Original: Discretization yields the geometric distribution, with probability mass function p ( x ) =
Revised: Discretization yields the geometric distribution, with probability mass function p ( x ) =

Original: e − xα − e − x +1 α .
Revised: e − xα − e − x +1 α .

Original: 3. C OMPOSING M IXTURES OF D ISTRIBUTIONS
Revised: 3. C OMPOSING M IXTURES OF D ISTRIBUTIONS

Revised: Another clear beneﬁt of the Cumulative Hazards-perspective of distributions is that it simpliﬁes the creation of new distributions.

Original: Recent developments on simplifying probabilistic programming have made it possible to effectively compose and work with distributions and their mixtures (Siddharth et al., 2017; Dillon et al., 2017; Tran et al., 2016).
Revised: Recent developments on simplifying probabilistic programming have made it possible to effectively compose and work with distributions and their mixtures (Siddharth et al., 2017; Dillon et al., 2017; Tran et al., 2016).

Original: One of the most well-known form of distribution composition is the mixturedensity network (MDN) (Bishop, 1994), where the probability density f = (cid:80)
Revised: One of the most well-known form of distribution composition is the mixture-density network (MDN) (Bishop, 1994), where the probability density f = (cid:80)

Original: k w k f k of the target data is a linear combination of more simple basis distributions.
Revised: k w k f k of the target data is a linear combination of more simple basis distributions.

Original: From the perspective of CHFs, we propose that it is simple to create other powerful compositions that can easily be learned from censored data.
Revised: From the perspective of CHFs, we propose that it is simple to create other powerful compositions that can easily be learned from censored data.

Original: It is easy to show that the space of CHFs are closed under strictly increasing mappings ( Λ 2 a ), composition ( Λ a (Λ b ) ), multiplication ( Λ a · Λ b ), addition Λ a + Λ b , and multiplication with positive scalars.
Revised: It is easy to show that the space of CHFs are closed under strictly increasing mappings ( Λ 2 a ), composition ( Λ a (Λ b ) ), multiplication ( Λ a · Λ b ), addition Λ a +Λ b , and multiplication with positive scalars.

Original: As a simple example, Λ LogLogistic = Λ Pareto (Λ Weibull ) .
Revised: As a simple example, Λ LogLogistic = Λ Pareto (Λ Weibull ) .

Original: Similar to MDNs, we can create complicated distributions by composing simpler CHFs, which can be easily learned with censored data and be discretized to any time resolution.
Revised: Similar to MDNs, we can create expressive possibly multimodal distributions by composing simpler CHFs, which can be easily learned with censored data and be discretized to any time resolution.

Original: In section 4, apart from using the commonly used Weibull-, Pareto 1 - and LogLogistic distributions we try their more expressive additive compositions respectively (named MixedHazards ).
Revised: In section 4, apart from using the commonly used Weibull-, Pareto 1 - and LogLogistic distributions we try their more expressive additive compositions respectively (named MixedHazards ).

Revised: We do this in order to show the generality of our approach and how it works with arbitrary distributions.

Original: 3. C ENSORED L OG - LIKELIHOOD
Revised: 3. C ENSORED L OG - LIKELIHOOD

Original: Under mild assumptions discussed here, we can accurately do maximum likelihood estimation using censored data utilizing censored log-likelihood .
Revised: Under mild assumptions discussed here, we can accurately do maximum likelihood estimation using censored data utilizing censored log-likelihood .

Original: With Y a random variable of interest parametrized by θ , c a constant or random variable of censoring time s.t Y ⊥ c | θ , X = min ( Y, c ) the censored (truncated) random variable of interest, and U =
Revised: With Y a random variable of interest parametrized by θ , c a constant or random variable of censoring time s.t Y ⊥ c | θ , X = min ( Y, c ) the censored (truncated) random variable of interest, and U =

Original: [ Y ≤ c ] is the non-censoring indicator, (with u =indicating not censored ).
Revised: [ Y ≤ c ] is the non-censoring indicator, (with u =indicating not censored ).

Original: Under these conditions it is well known that the likelihood of a censored (Klein & Moeschberger, 2005) random variable X
Revised: Under these conditions it is well known that the likelihood of a censored (Klein & Moeschberger, 2005) random variable X

Original: =
Revised: =

Original: x
Revised: x

Original: with an observed non-censoring indicator
Revised: with an observed non-censoring indicator

Original: U = u can be factored as
Revised: U = u can be factored as

Original: [Equation]
Revised: [Equation]

Original: censored
Revised: uncensored

Original: [Equation]
Revised: [Equation]

Original:  Which also holds in discretized case, where the above are discretized as ( Y d , c d , X d )
Revised: (right censored) [Equation] impossible query) Which also holds in discretized case, where the above are discretized as ( Y d , c d , X d )

Original: = ( (cid:98) Y (cid:99) , (cid:98) c (cid:99) , (cid:98) X (cid:99) ) and U d =
Revised: = ( (cid:98) Y (cid:99) , (cid:98) c (cid:99) , (cid:98) X (cid:99) ) and U d =

Original: [ Y d < c d ] is the non-censoring indicator (Martinsson, 2016).
Revised: [ Y d < c d ] is the non-censoring indicator (Martinsson, 2016).

Original: The practical implications of this assumption is that c should not be predictable by the features.
Revised: The practical implications of this assumption is that c should not be predictable by the features.

Original: Our experiments (Sec. 4) show that we learn proper calibration, hence this assumption can be made to hold in practice.
Revised: Our experiments (Sec. 4) show that we learn proper calibration, hence this assumption can be made to hold in practice.

Original: The log-likelihood for continuous (Eq. 2) and discrete (Eq. 3) observations ( x, u ) , can be simpliﬁed as below. 
Revised: The log-likelihood for continuous (Eq. 2) and discrete (Eq. 3) observations ( x, u ) , can be simpliﬁed as below. [Figure]

Original: [Equation]
Revised: [Equation]

Original: In this work we focus on the discrete log-likelihood as loss function, using the CHFs in Table 1.[Figure]
Revised: In this work we focus on the discrete log-likelihood as loss function, using the CHFs in Table 1.

Original:  L OSS FUNCTION FOR S EQUENTIAL P REDICTION
Revised: 3. L OSS FUNCTION FOR S EQUENTIAL P REDICTION

Original: Consider a sequence indexed by t = 1 , . . .
Revised: Consider a sequence indexed by t = 1 , . . .

Original: T .
Revised: T .

Original: We model the possibly censored observation ( x t , c t ) of time to event and censoring time at timestep t as a realizations of some random variable Y t , censored using a known censoring time c t s.t X t =
Revised: We model the possibly censored observation ( x t , c t ) of time to event and censoring time at timestep t as a realizations of some random variable Y t , censored using a known censoring time c t s.t X t =

Original: min
Revised: min

Original: ( Y t , c t ) .
Revised: ( Y t , c t ) .

Original: The optimization task is to predict parameters θ t of a target distribution, given by a ﬁxed form of Λ , which maximizes the log-likelihood of the observation.
Revised: The optimization task is to predict parameters θ t of a target distribution, given by a ﬁxed form of Λ , which maximizes the log-likelihood of the observation.

Original: The proposed (discrete) loss for one sequence is thus (cid:80)
Revised: The proposed (discrete) loss for one sequence is thus (cid:80)

Original: Tt − L d ( θ t , x t , c t ) .
Revised: Tt − L d ( θ t , x t , c t ) .

Original: While this might seem like an obvious formulation, most prior work has chosen different paths.
Revised: While this might seem like an obvious formulation, most prior work has chosen different paths.

Original: Most work has focused on speciﬁcs of certain continuous distributions.
Revised: Most work has focused on speciﬁcs of certain continuous distributions.

Original: While widely known, we hope to reafﬁrm that the principles of parametric survival analysis extends to all positive and discrete distributions.
Revised: While widely known, we hope to reafﬁrm that the principles of parametric survival analysis extends to all positive and discrete distributions.

Original: In terms of data shape, the dominant theme is to asynchronously predicting continuous inter-arrival times (Du et al.
Revised: In terms of data shape, the dominant theme is to asynchronously predicting continuous inter-arrival times (Du et al.

Original: ; Li (2017); Xiao et al.
Revised: ; Li (2017); Xiao et al.

Original: (2017b;a); Lee et al.
Revised: (2017b;a); Lee et al.

Original: ; Avati et al.
Revised: ; Avati et al.

Original: (2018) etc) assuming the time of prediction to coincide with the event times.
Revised: (2018) etc) assuming the time of prediction to coincide with the event times.

Original: We focus on the discrete form of TTE, predicted from arbitrary but equally-spaced timesteps.
Revised: We focus on the discrete form of TTE, predicted from arbitrary but equally-spaced timesteps.

Original: Equal-spacing and discrete TTE addresses two problems.
Revised: Equal-spacing and discrete TTE addresses two problems.

Original: First, to use sequence models we need a ﬁxed order and want to reduce the maximum sequence length.
Revised: First, to use sequence models we need a ﬁxed order and want to reduce the maximum sequence length.

Original: With continuous time, the number of events can be high, and the order of events may be unclear due to tied event times.
Revised: With continuous time, the number of events can be high, and the order of events may be unclear due to tied event times.

Original: Both of these issues are solved through binning and discretization.
Revised: Both of these issues are solved through binning and discretization.

Original: A second problem is that with unevenlyspaced predictions the loss function involves unevenly spaced terms over time.
Revised: A second problem is that with unevenlyspaced predictions the loss function involves unevenly spaced terms over time.

Original: The adequacy of this scheme when using arbitrary distributions is unclear.
Revised: The adequacy of this scheme when using arbitrary distributions is unclear.

Original: A subsequent question becomes how to predict or incorporate features arriving between events.
Revised: A subsequent question becomes how to predict or incorporate features arriving between events.

Original: By predicting at ﬁxed intervals these are both solved since we decoupled the prediction and the event times.
Revised: By predicting at ﬁxed intervals these are both solved since we decoupled the prediction and the event times.

Original: We have found no work addressing all these questions for censored time to event.
Revised: We have found no work addressing all these questions for censored time to event.

Original: Apparent drawback with our method is that if we want to train on a duration of 1000 time-units of data it yields a possibly sparse sequence of length 1000 (as in
Revised: Apparent drawback with our method is that if we want to train on a duration of 1000 time-units of data it yields a possibly sparse sequence of length 1000 (as in

Original: Figure 2).
Revised: Figure 2).

Original: It also limits the frequency of inference to the hop size (i.e. daily predictions).
Revised: It also limits the frequency of inference to the hop size (i.e. daily predictions).

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: We compare a diverse number of time to event distributions (Table 1) learned with different network architectures (Figure 3) on three different datasets (Table. 2) against a binary baseline.
Revised: We compare a diverse number of time to event distributions (Table 1) learned with different network architectures (Figure 3) on three different datasets (Table. 2) against a binary baseline.

Original: 4. D ATA D ESCRIPTION
Revised: 4. D ATA D ESCRIPTION

Original: [Table]

Original: We used three publicly available event-log datasets close to real-world applications.
Revised: We used three publicly available event-log datasets close to real-world applications.

Original: LastFM-1k (Celma, 2010) contains complete listening histories of about 1000 users.
Revised: LastFM-1k (Celma, 2010) contains complete listening histories of about 1000 users.

Original: BPI (Dees & Van Dongen, 2016) includes one year of click stream data from 26k users who logged in an Employee Insurance 
Revised: BPI (Dees & Van Dongen, 2016) includes one year of click stream data from 26k users who logged in an Employee Insurance [Table]

Original: Agency ofﬁce website.
Revised: Agency ofﬁce website.

Original: We created the Linux commit log dataset, which contains all commits from 13k active contributors since 2011.
Revised: We created the Linux commit log dataset, which contains all commits from 13k active contributors since 2011.

Original: For each dataset, we discretized the resolution from millisecond to daily measurements.
Revised: For each dataset, we discretized the resolution from millisecond to daily measurements.

Original: In doing so, a day with multiple events is considered a day with event hence its discrete time to event is zero.
Revised: In doing so, a day with multiple events is considered a day with event hence its discrete time to event is zero.

Original: Features were aggregated by summation at each day per sequence.
Revised: Features were aggregated by summation at each day per sequence and logarithmed if positive.

Original: The  number of events for a sequence in a day becomes one of the feature inputs to the day after.
Revised: The log of the number of events for a sequence in a day becomes one of the feature inputs to the day after.

Original: Furthermore, we deﬁne a new feature by aggregating events by date, marking the fraction of daily active users the day prior.
Revised: Furthermore, we deﬁne a new feature by aggregating events by date, marking the fraction of daily active users the day prior.

Original: This is a dense feature while other features are sparse.
Revised: This is a dense feature while other features are sparse.

Original: If a user was observed for 300 days, it yieldstime steps of data, even if they were only seen once.
Revised: If a user was observed for 300 days, it yields 300 time steps of data, even if they were only seen once.

Original: For every dataset, we use 80% of the dates for training and 20% for evaluation.
Revised: For every dataset, we use 80% of the dates for training and 20% for evaluation.

Original: The training target
Revised: The training target

Original: TTE is calculated from the training set keeping training and test temporally separated but sharing sequences.
Revised: TTE is calculated from the training set keeping training and test temporally separated but sharing sequences.

Original: This can be visualized as in Figure 4.
Revised: This can be visualized as in Figure 4.

Original: We only evaluate on the date immediately after the testing set ends, yielding one prediction per active sequence.
Revised: We only evaluate on the date immediately after the testing set ends, yielding one prediction per active sequence.

Original: This is the same as training a model on currently available data and doing a follow-up study to compare if events occurred as predicted.
Revised: This is the same as training a model on currently available data and doing a follow-up study to compare if events occurred as predicted.

Original: This differs from prior work as many split into train-test by sequence.
Revised: This differs from prior work as many split into train-test by sequence.

Original: Note that temporal models such as CNN and RNN will use past history as input.
Revised: Note that temporal models such as CNN and RNN will use past history as input.

Original: 4. B ASELINE : B INARY W INDOW M ODEL
Revised: 4. B ASELINE : B INARY W INDOW M ODEL

Original: In order to show the reliability of the model we need to show proper calibration of predicted probabilities and good discriminatory performance.
Revised: In order to show the reliability of the model we need to show proper calibration of predicted probabilities and good discriminatory performance.

Original: We did this by training individual binary models[Table] predicting probability of event within a threshold τ = 10 , 30 , 90 , 300 timesteps ahead.
Revised: We did this by training individual binary models predicting probability of event within a threshold τ = 10 , 30 , 90 , 300 timesteps ahead.

Original: We compared this against HazardNet by querying its predicted distribution on θ τt =
Revised: We compared this against HazardNet by querying its predicted distribution on θ τt =

Original: Pr ( Y t < τ ) .
Revised: Pr ( Y t < τ ) .

Original: While a binary model is explicitly trained for one threshold, HazardNet needs to learn them all.
Revised: While a binary model is explicitly trained for one threshold, HazardNet needs to learn them all.

Original: For this reason one could view the binary models’ performance as an upper bound.
Revised: For this reason one could view the binary models’ performance as an upper bound.

Original: While technically not a time to event model, in practice the binary window model is the dominant modeling method for time to event problems.
Revised: While technically not a time to event model, in practice the binary window model is the dominant modeling method for time to event problems.

Original: For experimental purposes it is illuminating.
Revised: For experimental purposes it is illuminating.

Original: No other relevant baseline could be justiﬁable compared for discrete TTE over different network architectures and different probabilistic queries.
Revised: No other relevant baseline could be justiﬁable compared for discrete TTE over different network architectures and different probabilistic queries.

Original: The only confounding factor is the treatment of censoring in the training set.
Revised: The only confounding factor is the treatment of censoring in the training set.

Original: HazardNet treats it explicitly while for the binary model there is two choices.
Revised: HazardNet treats it explicitly while for the binary model there is two choices.

Original: One can keep the last τ timesteps of data (which doesn’t have unbiased ground truth about no event ) or remove it.
Revised: One can keep the last τ timesteps of data (which doesn’t have unbiased ground truth about no event ) or remove it.

Original: We used both methods, denoted as Bin and Bin*.
Revised: We used both methods, denoted as Bin and Bin*.

Original: We use standard evaluation metrics for binary classiﬁcation to evaluate the predicted probabilities.
Revised: We use standard evaluation metrics for binary classiﬁcation to evaluate the predicted probabilities.

Original: Calibration is evaluated measuring Expected Calibration error (ECE) (Naeini et al., 2015), discriminatory performance using Area Under the Curve (AUC) and overall performance using Binary
Revised: Calibration is evaluated measuring Expected Calibration error (ECE) (Naeini et al., 2015), dis- [Table] criminatory performance using Area Under the Curve (AUC) and overall performance using Binary

Original: Cross Entropy (BCE) 3 , the loss that the binary model is optimizing for.
Revised: Cross Entropy (BCE) 3 , the loss that the binary model is optimizing for.

Original: We ran an excess of 452 individual training runs.
Revised: We ran an excess of 452 individual training runs.

Original: HazardNet was trained repetedly for every architecture and distribution 4 .
Revised: HazardNet was trained repetedly for every architecture and distribution 4 .

Original: Binary models were trained for every architecture and threshold, with and without the last τ timesteps in the training set.
Revised: Binary models were trained for every architecture and threshold, with and without the last τ timesteps in the training set.

Original: More details can be found in Appendix.
Revised: More details can be found in Appendix.

Original: 4. R ESULTS
Revised: 4. R ESULTS

Original: HazardNet beats or has identical performance to the binary model for almost every dataset, architecture and distribution.
Revised: HazardNet beats or has identical performance to the binary model for almost every dataset, architecture and distribution.

Original: While HazardNet is signiﬁcantly better on most metrics, the difference is very small.
Revised: While HazardNet is signiﬁcantly better on most metrics, the difference is very small.

Original: In Table 3 we report the best AUC achieved per experiment.
Revised: In Table 3 we report the best AUC achieved per experiment.

Original: Additional ﬁgures and tables can be found in Appendix.
Revised: Additional ﬁgures and tables can be found in Appendix.

Original: We summarize the results here.
Revised: We summarize the results here.

Original: Different datasets have different optimal TTE-distributions: The 1-parameter exponential distribution was consistently worse than the others.
Revised: Different datasets have different optimal TTE-distributions: The 1-parameter exponential distribution was consistently worse than the others.

Original: Weibull and WeibullMixedHazards was the least numerically stable, often failing mid training.
Revised: Weibull and WeibullMixedHazards was the least numerically stable, often failing mid training.

Original: LogLogistic seems like a generally good choice.
Revised: LogLogistic seems like a generally good choice.

Original: Weibull was optimal for LastFm-1k, differing from other datasets with it’s very high event density.
Revised: Weibull was optimal for LastFm-1k, differing from other datasets with it’s very high event density.

Original: Weibull is the only distribution that can model increasing hazard, suitable when lack of events implies event is getting closer.
Revised: Weibull is the only distribution that can model increasing hazard, suitable when lack of events implies event is getting closer.

Original: The highly seasonal BPI-dataset was dominated by LogLogistic which can model hazards that peak at some time, possibly suitable to model users certain to return at end of month.
Revised: The highly seasonal BPI-dataset was dominated by LogLogistic which can model hazards that peak at some time, possibly suitable to model users certain to return at end of month.

Original: The highly unpredictable Linux-dataset, characterized by high dropoff was dominated by
Revised: The highly unpredictable Linux-dataset, characterized by high dropoff was dominated by

Original: Pareto which a strictly decreasing hazard.
Revised: Pareto which a strictly decreasing hazard.

Original: The binary model should not train on the last timesteps: The temporal CNN and RNN learned that event-probability is artiﬁcially low at the end of the training set since we lack negative ground truth samples.
Revised: The binary model should not train on the last timesteps: The temporal CNN and RNN learned that event-probability is artiﬁcially low at the end of the training set since we lack negative ground truth samples.

Original: This makes the Binary model that were allowed to train on these timesteps poorly calibrated (high ECE and BCE).
Revised: This makes the Binary model that were allowed to train on these timesteps poorly calibrated (high ECE and BCE).

Original: On the calibration independent metric AUC, the models were indistinguishable except on the seasonal BPI-dataset where we assume the artifactc were more easily overﬁtted on.
Revised: On the calibration independent metric AUC, the models were indistinguishable except on the seasonal BPI-dataset where we assume the artifactc were more easily overﬁtted on.

Original: This conﬁrms that the binary approach is either biased or misses training data.
Revised: This conﬁrms that the binary approach is either biased or misses training data.

Original: HazardNet beats the baseline overall: HazardNet was better calibrated (lower minimum ECE) and had better discriminative performance (higher calibration independent AUC).
Revised: HazardNet beats the baseline overall: HazardNet was better calibrated (lower minimum ECE) and had better discriminative performance (higher calibration independent AUC).

Original: The results are most pronounced for the stronger models (CNN, RNN) implying that our model is most of the time better or equal and less prone to overﬁt.
Revised: The results are most pronounced for the stronger models (CNN, RNN) implying that our model is most of the time better or equal and less prone to overﬁt.

Original: Comparing the metric the binary model was optimized for (BCE), the results are less convincing but very close.
Revised: Comparing the metric the binary model was optimized for (BCE), the results are less convincing but very close.

Original: Over all metrics, HazardNet is signiﬁcantly better, but the difference is very small.
Revised: Over all metrics, HazardNet is signiﬁcantly better, but the difference is very small.

Original:  A PPLICATION : M USICAL O NSET D ETECTION
Revised:  A PPLICATION : M USICAL O NSET D ETECTION

Original: [Table]
Revised: [Table]

Original: Aiming to show the versatility of the proposed model [Table] we applied it for musical onset detection.
Revised: Aiming to show the versatility of the proposed model, we applied it for musical onset detection.

Original: The task is to detect relevant musical events in audio signals which is broadly used in applications such as automatic piano transcription (Hawthorne et al., 2017).
Revised: The task is to detect relevant musical events in audio signals which is broadly used in applications such as automatic piano transcription (Hawthorne et al., 2017).

Original: The state of the art poses it as a binary prediction task.
Revised: The state of the art poses it as a binary prediction task.

Original: To counter class imbalance and sparseness they apply various smoothing techniques and weights on predicted probabilities (Schluter & B¨ock, 2014).
Revised: To counter class imbalance and sparseness they apply various smoothing techniques and weights on predicted probabilities (Schluter & B ¨ ock, 2014).

Original: We propose to soften this problem instead by formulating it as a multivariate time to event problem, jointly predicting the distribution of time to onset and time since onset (TSE).
Revised: We propose to soften this problem instead by formulating it as a multivariate time to event problem, jointly predicting the distribution of time to onset and time since onset (TSE).

Original: In this way the onset probability in each timestep can be predicted as in Eq. 5:
Revised: In this way the onset probability in each timestep can be predicted as in Eq. 5:

Original: [Equation]
Revised: [Equation]

Original: Experiments were performed on the B ¨ ock dataset, a dataset for evaluating onset detection that is used in several papers (B ¨ ock et al., 2012; Schluter & B ¨ ock, 2014)
Revised: Experiments were performed on the B¨ock dataset, a dataset for evaluating onset detection that is used in several papers (B¨ock et al., 2012; Schluter & B¨ock, 2014)

Original: The dataset contains 321 audio clips taken from various sources, including piano, violin, percussions, and more (B¨ock et al., 2012).
Revised: The dataset contains 321 audio clips taken from various sources, including piano, violin, percussions, and more (B¨ock et al., 2012).

Original: As feature input we use log scale mel-spectrograms (80 bins) computed with Librosa (McFee et al., 2015) with frequency ranges between 27.Hz to 16kHz.
Revised: As feature input we use log scale mel-spectrograms (80 bins) computed with Librosa (McFee et al., 2015) with frequency ranges between 27.Hz to 16kHz.

Original: For each audio clip, three mel-spectrograms with different STFT window sizes(23ms, 46ms, 93ms) but same hop size(10ms) were computed and concatenated channel-wise.
Revised: For each audio clip, three mel-spectrograms with different STFT window sizes(23ms, 46ms, 93ms) but same hop size(10ms) were computed and concatenated channel-wise.

Original: A single network input is 15 frames of a precomputed mel-spectrogram, where we predict on the center frame.
Revised: A single network input is 15 frames of a precomputed mel-spectrogram, where we predict on the center frame.

Original: The network architecture is shown in Figure 3 (d).
Revised: The network architecture is shown in Figure 3 (d).

Original: To compare with previous work we used the CNN architecture of Schl¨uter (Schluter & B¨ock, 2014).
Revised: To compare with previous work we used the CNN architecture of Schl ¨ uter (Schluter & B ¨ ock, 2014).

Original: Our model had two extra layers for the parametric output so a comparable baseline model was made to match the number of parameters (Schluter & B¨ock, 2014).
Revised: Our model had two extra layers for the parametric output so a comparable baseline model was made to match the number of parameters (Schluter & B¨ock, 2014).

Original: Target values for training were naturally censored at start and end of song and artiﬁcially censored s.t y <
Revised: Target values for training were naturally censored at start and end of song and artiﬁcially censored s.t y <

Original: c with y the TSE and TTE and c = 10 , 20 .
Revised: c with y the TSE and TTE and c = 10 , 20 .

Original: All experiments are trained on 300 epochs, using SGD with a learning rate of 0.001 and momentum linearly increasing at 10 to 20 epochs from 0.45 to 0.9.
Revised: All experiments are trained on 300 epochs, using SGD with a learning rate of 0.001 and momentum linearly increasing at 10 to 20 epochs from 0.45 to 0.9.

Original: To choose predicted onset labels from the predicted probabilities we applied the peak-picking method from Librosa (McFee et al., 2015), using parameters recommended by B¨ock for ofﬂine detection (B¨ock et al., 2012).
Revised: To choose predicted onset labels from the predicted probabilities we applied the peak-picking method from Librosa (McFee et al., 2015), using parameters recommended by B¨ock for ofﬂine detection (B¨ock et al., 2012).

Original: For evaluation, we use the onset evaluation metric from mir-eval (Raffel et al., 2014) with 50ms tolerance.
Revised: For evaluation, we use the onset evaluation metric from mir-eval (Raffel et al., 2014) with 50ms tolerance.

Original: We achieved best results using the LogLogistic distribution and c ≤ 10 .
Revised: We achieved best results using the LogLogistic distribution and c ≤ 10 .

Original: Results as average optimal F1-scores from 8-fold cross validation are reported in Table 4.
Revised: Results as average optimal F1-scores from 8-fold cross validation are reported in Table 4.

Original: We ﬁnd that despite only differing by the loss function, our method achieves state of the art results.
Revised: We ﬁnd that despite only differing by the loss function, our method achieves state of the art results.

Original:  S UMMARY & C ONCLUSION
Revised:  S UMMARY & C ONCLUSION

Original: In this paper we presented a simple generalized parametric survival approach for using neural networks to sequentially predict probability distributions over time to the next event.
Revised: In this paper we presented a simple generalized parametric survival approach for using neural networks to sequentially predict probability distributions over time to the next event.

Original: In the experiments we trained the proposed model (HazardNet) and queryed it on  subtasks for which baseline binary models were explicitly trained for.
Revised: In the experiments we trained the proposed model (HazardNet) and queryed it on classiﬁcation subtasks for which baseline binary models were explicitly trained for.

Original: We found that we often achieve both higher performance and better calibrated probabilistic predictions.
Revised: We found that we often achieve both higher performance and better calibrated probabilistic predictions.

Original: Since the model predicts a distribution one can readily calculate other meaningful quantities from it such as predicting quantiles, expected value or sample random TTEs.
Revised: Since the model predicts a distribution one can readily calculate other meaningful quantities from it such as predicting quantiles, expected value or sample random TTEs.

Original: While much prior research has focused on speciﬁcs of certain probability distributions, we unsurprisingly ﬁnd that the optimal choice depends on the dataset.
Revised: While much prior research has focused on speciﬁcs of certain probability distributions, we unsurprisingly ﬁnd that the optimal choice depends on the dataset.

Original: We think this emphasizes the importance of a broader discussion on generalized solutions and good abstractions, making it easy to experiment and create new distributions.
Revised: We think this emphasizes the importance of a broader discussion on generalized solutions and good abstractions, making it easy to experiment and create new distributions.

Original: To this goal, we showed how to train with censored data and how this extends to various continuous, discrete and multivariate distributions with minimal effort.
Revised: To this goal, we showed how to train with censored data and how this extends to various continuous, discrete and multivariate distributions with minimal effort.

Original: We introduced and tested a way to compose distributions using their cumulative hazard functions while being able to use censored data ( MixedHazards -distributions).
Revised: We introduced and tested a way to compose distributions using their cumulative hazard functions while being able to use censored data ( MixedHazards -distributions).

Original: Despite being much more expressive, we found no proof that they performed better (or worse) than simple distributions.
Revised: Despite being much more expressive, we found no proof that they performed better (or worse) than simple distributions.

Original: It is tempting to work on trying to encode more information into ever more expressive predicted distributions or ﬁne-grained predicted hazard rates.
Revised: It is tempting to work on trying to encode more information into ever more expressive predicted distributions or ﬁne-grained predicted hazard rates.

Original: For the real-life event sequence data we used, we ﬁnd little to indicate that this is a fruitful path of research.
Revised: For the real-life event sequence data we used, we ﬁnd little to indicate that this is a fruitful path of research.

Original: The main intention of the experiments was to verify that our model makes unbiased, calibrated probabilistic predictions and efﬁciently utilizes training data.
Revised: The main intention of the experiments was to verify that our model makes unbiased, calibrated probabilistic predictions and efﬁciently utilizes training data.

Original: Our criteria for relevant baselines was to be able to train with discrete censored TTE, use asynchronously arriving feature inputs while predicting at each timestep and work with arbitrary neural network-architectures.
Revised: Our criteria for relevant baselines was to be able to train with discrete censored TTE, use asynchronously arriving feature inputs while predicting at each timestep and work with arbitrary neural network-architectures.

Original: It should also be able to predict queries such as Pr ( Y < 10) , . . .
Revised: It should also be able to predict queries such as Pr ( Y < 10) , . . .

Original: Pr ( Y < 300) .
Revised: Pr ( Y < 300) .

Original: To the best of our knowledge, no existing work satisﬁed all of these demands.
Revised: To the best of our knowledge, no existing work satisﬁed all of these demands.

Original: What was left for fair comparison was the binary window prediction model.
Revised: What was left for fair comparison was the binary window prediction model.

Revised: A surprising result was that this baseline makes biased predictions unless one removes the last time-window from the training data.

Revised: This could have implications for all approaches relying on ﬁxed-window predictions (Harutyunyan et al., 2017; Lee et al., 2018; Luck et al., 2017) who did not discuss or evaluate this apparent drawback.

Original: That aside, comparisons between generalized parametric- and the dominant semi-parametric Cox-Proportional Hazards approach (Katzman et al., 2016; Luck et al., 2017; Joshi & Reeves, 2006) should be priority future work.
Revised: That aside, comparisons between generalized parametric- and the dominant semi-parametric CoxProportional Hazards approach (Katzman et al., 2016; Luck et al., 2017; Joshi & Reeves, 2006) should be priority future work.

Original: The latter can only answer a subset of the probabilistic queries but it is theoretically possible to compare models in terms of their predicted rankings between subjects, calculating Pr( Y i < Y j ) .
Revised: The latter can only answer a subset of the probabilistic queries but it is theoretically possible to compare models in terms of their predicted rankings between subjects, calculating Pr( Y i < Y j ) .

Original: As we found no prior work on architecture agnostic semi-parametric approaches we propose this as future work.
Revised: As we found no prior work on architecture agnostic semi-parametric approaches we propose this as future work.

Original: Finally, for extensions of our work we propose studies on how to extend and evaluate it for asynchronously predicted time to event and how multivariate time to event can be connected to temporal multi-class classiﬁcation.
Revised: Finally, for extensions of our work we propose studies on how to extend and evaluate it for asynchronously predicted time to event and how multivariate time to event can be connected to temporal multi-class classiﬁcation.

Original: ORE DETAILS ON EXPERIMENTS OF SECTION
Revised: ORE DETAILS ON EXPERIMENTS OF SECTION

Original: To validate our results we made a total of 452 individual training runs.
Revised: To validate our results we made a total of 452 individual training runs.

Original: All logs, results and code to recreate the experiments will be released online.
Revised: All logs, results and code to recreate the experiments will be released online.

Original: The binary baselines were run for every dataset, threshold ( τ = 10 , 30 , 90 , 300 ) and model (MLP,
Revised: The binary baselines were run for every dataset, threshold ( τ = 10 , 30 , 90 , 300 ) and model (MLP,

Original: CNN, RNN) using both the last (biased) binary target value (labeled a ’Bin’ in tables) and when removing the τ last timesteps of the training set (labeled as Bin ∗ ).
Revised: CNN, RNN) using both the last (biased) binary target value (labeled a ’Bin’ in tables) and when removing the τ last timesteps of the training set (labeled as Bin ∗ ).

Original: Every setting was repeated for different random seeds; at least 5 times for LastFm1k (98 successful runs in total), 8 times for BPI (64) and at least 4 times for linux dataset (69).
Revised: Every setting was repeated for different random seeds; at least 5 times for LastFm1k (98 successful runs in total), 8 times for BPI (64) and at least 4 times for linux dataset (69).

Original: HazardNet was run for every dataset, distribution (Exponential, Weibull, LogLogistic, Pareto and their MixedHazards-versions) and model.
Revised: HazardNet was run for every dataset, distribution (Exponential, Weibull, LogLogistic, Pareto and their MixedHazards-versions) and model.

Original: Binary evaluation was calculated for every τ .
Revised: Binary evaluation was calculated for every τ .

Original: Every setting was repeated for different random seeds; at least 2 times for LastFm1k (63 successful runs in total), 3 times for BPI (76) and at least 3 times for linux dataset (82 runs in total).
Revised: Every setting was repeated for different random seeds; at least 2 times for LastFm1k (63 successful runs in total), 3 times for BPI (76) and at least 3 times for linux dataset (82 runs in total).

Original: The TTE was artiﬁcially censored s.t TTE < 300 .
Revised: The TTE was artiﬁcially censored s.t TTE < 300 .

Original: This was found to increase numerical stability and reduce overﬁt.
Revised: This was found to increase numerical stability and reduce overﬁt.

Original: We found little or no advantage in the more expressive often multimodal MixedHazardsdistributions.
Revised: We found little or no advantage in the more expressive often multimodal MixedHazardsdistributions.

Original: We speculate that this is due to the tasks being inherently hard and exact timings of events unpredictable.
Revised: We speculate that this is due to the tasks being inherently hard and exact timings of events unpredictable.

Original: We present summaries of this large comparison matrix below.
Revised: We present summaries of this large comparison matrix below.

Original: Here we also show results when using a smaller 1-layer 5-hidden units RNN.
Revised: Here we also show results when using a smaller 1-layer 5-hidden units RNN.

Original: Figure 5,6, 7 shows the best metrics (HazardNet vs Binary baseline) evaluated per epoch.
Revised: Figure 5,6, 7 shows the best metrics (HazardNet vs Binary baseline) evaluated per epoch.

Original: It is clear that the HazardNet approach is almost always better or at least as good and seems less prone to overﬁt.
Revised: It is clear that the HazardNet approach is almost always better or at least as good and seems less prone to overﬁt.

Original: Table 5, 6, 7 summarises the ﬁnal results in more detail.
Revised: Table 5, 6, 7 summarises the ﬁnal results in more detail.

Original: Figure 8 and 9 shows the test-set loss broken down per distribution.
Revised: Figure 8 and 9 shows the test-set loss broken down per distribution.

Original: This was evaluated differently (using the whole test set, not just the timesteps after the end of training set).
Revised: This was evaluated differently (using the whole test set, not just the timesteps after the end of training set).

Original: When comparing the binary scores for HazardNet among distributions evaluated in the more rigourous fashion proposed in Section.
Revised: When comparing the binary scores for HazardNet among distributions evaluated in the more rigourous fashion proposed in Section.

Original: 4 the overall trends are the same.
Revised: 4 the overall trends are the same.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: l a s tf m 1 k l a s tf m 1 k l a s tf m 1 k l a s tf m 1 k 10 bp i 30 bp i
Revised: l a s tf m 1 k l a s tf m 1 k l a s tf m 1 k l a s tf m 1 k 10 bp i 30 bp i

Original: [Figure]
Revised: [Figure]

Original: li nu x
Revised: li nu x

Original: [Figure]
Revised: [Figure]

Original: li nu x
Revised: li nu x

Original: [Figure]
Revised: [Figure]

Original: li nu x
Revised: li nu x

Original: [Figure]
Revised: [Figure]

Original: li

Original: [Figure] [Figure][Figure]
Revised: [Figure] [Figure]

Original: l a s tf m
Revised: l a s tf m

Original: [Figure]
Revised: [Figure]

Original: l a s tf m
Revised: l a s tf m

Original: [Figure]
Revised: [Figure]

Original: k l a s tf m 1 k
Revised: k l a s tf m 1 k

Original: [Figure]
Revised: [Figure]

Original: bp i
Revised: bp i

Original: [Figure]
Revised: [Figure]

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: li nu
Revised: li nu

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: l a s tf m 1 k l a s tf m 1 k l a s tf m 1 k l a s tf m 1 k 10 bp i 30 bp i
Revised: l a s tf m 1 k l a s tf m 1 k l a s tf m 1 k l a s tf m 1 k 10 bp i 30 bp i

Original: [Figure]
Revised: [Figure]

Original: li nu x
Revised: li nu x

Original: [Figure]
Revised: [Figure]

Original: li nu x
Revised: li nu x

Original: [Figure]
Revised: [Figure]

Original: li nu x
Revised: li nu x

Original: [Figure]
Revised: [Figure]

Original: li nu x
Revised: li nu x

Original: [Figure]
Revised: [Figure]

Original: Lower is better
Revised: Lower is better.

Original: [Figure] [Figure][Figure]
Revised: [Figure] [Figure]