P OINT
P ROCESS F LOWS
Anonymous authors
Paper under double-blind review
A BSTRACT
Event sequences can be modeled by temporal point processes (TPPs) to capture their asynchronous and probabilistic nature.
We propose an intensity-free framework that directly models the point process as a non-parametric distribution by utilizing normalizing ﬂows.
This approach is capable of capturing highly complex temporal distributions and does not rely on restrictive parametric forms.
Comparisons with state-of-the-art baseline models on both synthetic and challenging real-life datasets show that the proposed framework is effective at modeling the stochasticity of discrete event sequences.
I NTRODUCTION
Data in real-life takes various forms.
Event sequences, as a special form of data, are discrete events in continuous time.
This type of data is prevalent in a broad spectrum of areas, for example, patient visits to hospitals, user behavior on social media, credit card transactions, etc.
In this setting, each event is discrete, and the temporal dynamics of the events are complex and asynchronous.
It is crucial to understand the characteristics and dynamics of such data, so that plausible future prediction, as well as other downstream applications, such as intervention or recommendation, can be performed.
Despite recent success in modeling images, videos and texts with the power of deep neural networks (DNNs), the asynchronous and probabilistic nature of event sequence data makes it challenging to utilize the power of off-the-shelf DNN-based models.
Temporal point processes (TPPs; Daley & Vere-Jones 2007) provide us with an elegant and effective mathematical framework for modeling event sequences data.
A temporal point process is deﬁned as a stochastic process whose realizations consist of a list of events with their corresponding occurring times.
These occurring times can either be real numbers from an index set (deﬁned from prior knowledge) or sampled from an intensity function.
While other time-series models learn temporal patterns synchronously (with each time-step being treated as an input to the model), TPP-based frameworks directly model the time intervals between events as random variables.
With such a setup, it allows for modeling long sequences without vanishing gradients or costly memory issues.
Although the temporal point process has shown to be useful in modeling events sequences, it is usually not trivial to come up with a simple yet ﬂexible intensity function.
An intensity function encodes the rate an event occurs at a speciﬁed time-step.
Poisson process (Kingman, 1992) has been a popular hand-crafted design for the intensity which assumes that events are independent of each other.
More sophisticated design choices are investigated in the self-exciting (Hawkes, 1971) and self-correcting process (Isham & Westcott, 1979).
The key contribution of these models is to ﬁnd a functional form of intensity that ﬁts data distribution well by making various parametric assumptions on the underlying generative process of the data.
Although shown effective in modeling simple synthetic datasets, parametric assumptions make such frameworks lack the ﬂexibility to model the generative process for real-life and complex data, hindering wider adoption of TPP-based frameworks.
Recently, learning the intensity function using recurrent neural networks (RNNs) to encode the history has received an increasing amount of attention (Du et al., 2016; Zhong et al., 2018; Mei &
Eisner, 2017; Jing & Smola, 2017).
In this line of work, history information is encoded and exploited in learning the intensity of the point process distribution.
In this case, the explicit parametric assumption on the forms of the intensity functions is relaxed.
However, the maximum likelihood training criteria on these models still requires the intensity function to be simple for the likelihood to be tractable.
Recent work by Mehrasa et al.
(2019) proposes a probabilistic framework based on variational autoencoders for modeling point process, further facilitating the stochastic generative process.
All the literature above is built upon explicit modeling of a temporal point process using the intensity function.
It is not necessary to explicitly model the intensity; a few works have tried to formulate TPP in an intensity-free manner.
WGANTPP (Xiao et al., 2017; 2018) introduce an intensity-free framework for modeling the point process distribution using Wasserstein distance.
The model is built upon a generative adversarial network (GAN).
RLPP (Li et al., 2018) formulates this problem in a reinforcement learning framework and treats future event predictions as actions taken by an agent.
Both of these models are optimized by trying to generate sequences of samples that are indistinguishable from the ground-truth sequences (by a discriminator in WGANTPP and policy learning in RLPP).
Although these models are capable of generating realistic sequences, such training criteria fail to model the data distribution, resulting in intractable likelihood.
In this work, We propose a novel intensity-free point process model based on continuous normalizing ﬂow and variational autoencoders.
The proposed point process ﬂow (PPF) model utilizes a recurrent variational autoencoder to encode the history of a given event sequence and makes probabilistic predictions on the next event.
It preserves the non-parametric characteristics of point process distributions with normalizing ﬂow.
The predicted non-parametric point process distribution is capable of capturing complex time distributions of arbitrary shape, leading to more accurate modeling of event sequences.
Extensive experiments are conducted on synthetic and real datasets to evaluate the performance of the proposed model.
Experimental results show that our model is capable of capturing complex point process distributions as well as performing accurate stochastic forecast.
The contributions are summarized as follows: (1) A novel intensity-free point process model built upon continuous normalizing ﬂow.
The proposed PPF is capable of capturing highly complex temporal distributions and does not rely on restrictive parametric forms; (2) PPF can be optimized by maximizing the exact likelihood using change of variable formula, relaxing the constraint in previous works that likelihood has to be tractable; (3) Evaluation on both synthetic and challenging real-life datasets shows improvement over the state-of-the-art point process models.
P RELIMINARIES 2. T EMPORAL POINT PROCESS
A temporal point process (TPP; Daley & Vere-Jones 2007) is a mathematical framework for modeling asynchronous sequences of actions.
It is a stochastic process whose realization is a sequence of discrete events in time t 1: n = { t 1 , t 2 , ..., t n } where t n is the time when the n th event occurred.
A temporal point process distribution is modeled by specifying the probability density function of the time of the next event:
[Equation]
where the intensity function λ ( t |H t ) is the conditional intensity function.
It encodes the expected rate of event happening in a small area around t and H t = { t 1 , t 2 , ..., t n − 1 | t n − 1 < t } is the sequence of event times up to time t .
Many works explored different design choices of intensity function to capture the phenomena of interest.
Here we review two popular hand-crafted design choices:
Poisson Process.
Poisson process (Kingman, 1992) is based on the assumption that events happen independent of each other where the intensity is a ﬁxed positive constant λ ( t ) =
λ and λ > 0 .
In a more general case, λ could be a function of time λ ( t |H t ) =
λ ( t ) but still independent of other events, which is called inhomogeneous Poisson process.
Self-exciting Process (Hawkes process).
Self-exciting process (Hawkes, 1971) assumes that occurrence of an event increases the probability of other events happening in near future.
Its intensity function has the functional form of λ ( t |H t ) =
µ + α (cid:80) t
i <t exp( − ( t − t i )) , where µ and α are positive constants and t i < t are all the events happening before time t.
2. N ORMALIZING FLOW
Normalizing ﬂows are generative models that allow both density estimation and sampling.
They map simple distributions to complex ones using bijective functions.
Speciﬁcally, if our interest is to estimate the density function p X of a random vector X ∈ R d , then normalizing ﬂows assume
X
= g θ ( Z ) , where g θ : R d → R d is a bijective function, and Z ∈ R d is a random vector with a tractable density function p Z .
We further denote the inverse of g θ by f θ .
On one hand, the probability density function can be evaluated using the change of variables formula:
[Equation]
where ∂f θ /∂ x denotes the Jacobian matrix of f θ .
On the other hand, sampling from p X can be done by ﬁrst drawing a sample from the simple distribution z ∼ p Z , and then apply the bijection x
= g θ ( z ) .
Given the expressive power of deep neural networks, it is natural to construct g θ as a neural network.
However, it requires the bijection g θ to be invertible, and the determinant of the Jacobian matrix should be efﬁcient to compute.
Several methods have been proposed along this research direction (Rezende & Mohamed, 2015;
Dinh et al., 2014; 2017; Kingma et al., 2016; Kingma & Dhariwal, 2018;
Papamakarios et al., 2017).
An extensive overview of normalizing ﬂow models is given by
Kobyzev et al.
2. C ONTINUOUS NORMALIZING FLOW
From a dynamical systems perspective, the residual network can be regarded as the discretization of an ordinary differential equation (ODE; Haber & Ruthotto 2017; Chang et al. 2018;
Lu et al. 2018).
Inspired by that, Chen et al.
(2018) propose neural ODE, where the continuous dynamics of hidden units is parameterized using an ordinary differential equation speciﬁed by a neural network:
[Equation]
dt
[Equation]
The neural ODE can be used to construct a continuous normalizing ﬂow.
The invertibility is naturally guaranteed by the theorem of the existence and uniqueness of the solution of the ODE.
Furthermore, using the instantaneous change of variables formula, similar to Equation 2, the log-density can be evaluated by solving the following ODE:
[Equation]
Grathwohl et al.
(2019) propose an improved version of neural ODE, named FFJORD, which has lower computational cost by using an unbiased stochastic estimation of the trace of a matrix.
P ROPOSED FRAMEWORK
We propose an intensity-free ﬂow framework to model the timing of events in point process sequences.
More speciﬁcally, we learn a non-parametric distribution over the timing of asynchronous event sequences by transforming a simple base probability density through continuous normalizing ﬂow, i .
e ., a series of invertible transformations.
With our proposed framework, we are able to model complex point process distributions without making any assumption on the functional form of the distribution while being able to evaluate the likelihood of sequences under our model.
3. N ON - PARAMETRIC POINT PROCESS FLOWS
Let the input be a sequence of asynchronous events { t 1 , t 2 , ... } , where t i ∈ R + represents the starting time of the i -th event.
We deﬁne the inter-arrival time τ n as the time difference between the starting time of events t n − 1 and t n .
Our goal is to model the distribution over inter-arrival time τ n given the past history of events inter-arrival times τ 1: n − 1 , i .
e ., learning to model the conditional distribution p ( τ n | τ 1: n − 1 ) .
Our approach is to construct the distribution over inter-arrival time τ n by transforming a simple base distribution through normalizing ﬂow transformations.
At time-step n , we assume that inter-arrival time τ n was generated by ﬁrst sampling from a simple distribution p ( z n ) and then transforming the drawn sample z n through an invertible transformation g θ : R → R + parametrized by θ : z n
[Equation]
p
[Equation] [Equation]
[Equation]
With this assumption and the change of variable formula discussed in subsection 2.2, we can write the distribution over inter-arrival time τ n as:
[Equation] [Equation]
where f θ ( τ n ) = g − 1 θ ( τ n ) =
z n , and the scalar Jacobin value df θ /dτ n shows the changes in the density when moving from τ n to z n .
We dropped the determinant term in the change of variable formula, because in our case, the inter-arrival time τ n is a one-dimensional variable.
With this formulation, we are able to model the inter-arrival time distribution, without any speciﬁc assumption on the functional form of the distribution.
Exact samples of the inter-arrival time distribution can be obtained by sampling from the base distribution z n ∼ p ( z n ) and transforming it through ﬂow transformation τ n = g θ ( z n ) .
We are also able to compute the exact likelihood of τ n by computing the likelihood of f θ ( τ n ) and multiplying it with the associated Jacobian term (cid:12)(cid:12)(cid:12)
[Equation]
Current formulation models the inter-arrival distribution of each time-step independent of past history.
The future event timing might depend on previous events in a very complex way, so its important to take the history information into consideration while modeling future events.
To capture this dependency, we adapt our formulation to construct the point process distribution by learning normalizing ﬂow parameters conditioned on the history.
More speciﬁcally, we learn the parameters of ﬂow base distribution by a time-dependent model parametrized by ρ that encodes history and provides the conditional base distribution p ρ ( z n | τ 1: n − 1 ) at each time-step.
The overal procedure can be seen in Figure 1.
In our framework, the base distribution is assumed to follow a Gaussian distribution:
[Equation]
where ( µ ρ n , σ 2 ρ n ) are the parameters of the base distribution.
These parameters can be obtained by encoding the history τ 1: n − 1 using various approaches.
In this work, we propose two approaches to construct the base distribution of the ﬂow: 1) Base distribution with deterministic parameters. 2)
Base distribution with stochastic parameters.
In the following sections, we describe each approach in more details.
3. B ASE DISTRIBUTION WITH DETERMINISTIC PARAMETERS
As discussed in subsection 3.1, we aim to model the conditional distribution p ( τ n | τ 1: n − 1 ) using history information in learning the base distribution parameters.
Recurrent neural networks (RNNs) have shown to be powerful deterministic models in capturing temporal dependencies.
Recent works adapt RNNs as a non-linear mapping of the history to the intensity function to deﬁne temporal point process distributions (Du et al., 2016; Zhong et al., 2018; Mei & Eisner, 2017).
As a ﬁrst attempt, we use RNNs to learn the parameters of the base distribution of the ﬂow using the history information.
Figure 2 part (a) illustrates the overall structure of our model.
To construct the conditional distribution p θ,ρ ( τ n | τ 1: n − 1 ) , the RNN takes the history of the past inter-arrival times τ 1: n − 1 and produces the conditional base distribution p ρ ( z n | τ 1: n − 1 ) =
N ( µ ρ n , σ 2 ρ n ) for the next time-step.
Then, the base distribution is transformed into the conditional inter-arrival time distribution over τ n through normalizing ﬂow transformations g θ .
The RNN is jointly optimized with the ﬂow module by maximizing the log-likelihood of observed sequence τ 1: N under the predicted distribution:
[Equation]
3. B ASE DISTRIBUTION WITH PROBABILISTIC PARAMETERS
It is known that there is a trade-off between the complexity of the bijective transformation and the form of base distribution (Jaini et al., 2019).
With the complexity of the bijective transformation ﬁxed, a more ﬂexible base distribution will lead to a more expressive model.
In our proposed framework, the fact that ﬂow transformations are shared across time-steps and the true underlying distribution across time-steps might vary a lot, makes our model more sensitive to the choice of base distribution family.
We believe that, if we choose to model base distributions as Gaussian distributions with deterministic parameters, the bijective transformation might not be able to estimate underlying distributions well.
We further support our claim by proving Proposition 1 which, intuitively speaking, says more ﬂexible base-distribution yields more expressive model.
Motivated by this, our second move is to have a more ﬂexible base-distribution where the parameters are probabilistic. In order to achieve this, we utilize the variational auto-encoder (VAE; Kingma &
Welling 2014) paradigm in modeling the conditional base-distributions.
To better illustrate the importance of having more ﬂexible base-distribution, we provide a motivating example in Appendix A.
To avoid confusion, at time-step n , we use the notation z τn for the random variable of the normalizing ﬂow base distribution and z vaen to refer to the VAE latent space.
We start by explaining the generation phase, i .
e ., how the distributions over inter-arrival time τ n are generated by stacking the normalizing ﬂow module on top of the VAE backbone and then describing the training process.
Generation.
Figure 2 part (b) shows an overview of the generation process.
Here, we adapt a recurrent VAE framework consisting of a time-variant prior network parametrized by ψ which takes the history of past actions τ 1: n − 1 and provides the latent distribution p ψ ( z vaen | τ 1: n − 1 ) .
Then, a sample of this distribution is passed to the VAE’s decoder which produces a non-parametric distribution over the inter-arrival time τ n by ﬁrst generating the normalizing ﬂow base distribution p ρ ( z τn | z vaen ) and then transforming it through ﬂow transformation g θ .
By applying the change of variable formula discussed in Equation 6, we can write the distribution over inter-arrival time τ n as:
[Equation]
Training.
At time-step n of training, the VAE module takes the sequence of inter-arrival times τ 1: n to approximate the true distribution over the latent space z vaen via the help of the recurrent inference network q φ ( z vaen | τ 1: n ) which is parametrized with φ .
A time-dependent prior network is also adapted to help the model to take use of history information in generation phase p ψ ( z vaen | τ 1: n − 1 ) .
Both prior and posterior distributions are assumed to follow conditional multivariate Gaussian distributions with diagonal covariance:
[Equation]
At each time-step during training, a latent code z vaen is taken from the posterior and is passed to the decoder which aims to generate the conditional distribution p θ,ρ ( τ n | z vaen ) .
The VAE backbone is jointly trained with the ﬂow module by optimizing the variational lower bound using the reparameterization trick (Kingma & Welling, 2014):
[Equation]
where we compute the log-likelihood term log p θ,ρ ( τ n | z vaen ) by applying the change of variable formula of Equation 9.
E VALUATION
To show the effectiveness of our non-parametric approach, we evaluate the performance of our model on synthetic and real-world datasets and compare it with the state-of-the-art point process models.
Please refer to Appendix A for architecture and implementation details.
4. D ATASETS AND B ASELINES
Synthetic Datasets.
We create three types of synthetic datasets as follow: (I) Inhomogeneous
Poisson Process (IP) deﬁnes the intensity as a function of time but independent of the history.
We simulate sequences of IP process with λ ( t ) =
(cid:80) ki
=
α i (2 πσ 2 i ) − 1 / 2 exp( − ( t − c i ) 2 /σ 2 i )
where k = 6 , α = (14 , 18 , 13 , 17 , 10 , 13) ,
c = (3 , 6 , 9 , 12 , 15 , 18) and σ = (5 , 5 , 5 , 5 , 5 , 5) .
(II) Selfexciting Process (SE) assumes that the occurrence of an event increases the probability of other events happening in the near future.
It is characterized by λ ( t ) = µ + β (cid:80) t i <t g ( t − t i ) ,
where in our case g ( t ) = exp( − t ) , µ = 1 . 0 , and β = 0 .
(III) IP
+ SE is created by combining the simulated data from the self-exciting process and the inhomogeneous process.
For each of IP and
SE, we generate 20000 sequences, where the length of each is 60, and split the sequences into train, validation and test sets with proportions of 0.7, 0.1, 0.2, respectively.
Real-world Datasets.
We also evaluate our models on real datasets that cover the areas of social media, healthcare, and human activity as follow: (I) LinkedIn data is collected from over
LinkedIn accounts and contains their job-hopping records with information including the time and company.
Our model predicts the time-interval before a user’s next job-hopping.
After pruning users with only one job-hopping record, we collect 2439 sequences.
(II) MIMIC (Medical Information
Mart for Intensive Care III; Johnson et al. 2016; Pollard 2016) is a publicly available, large-scale dataset which contains the medical records of more than 40000 anonymous patients.
Our method models the inter-arrival time of patients’ admissions to hospital.
We keep the record of patients who have at least three visits to hospitals and collected 2377 sequences.
(III) Breakfast dataset (Kuehne et al., 2014) contains 1712 videos with 48 action classes related to breakfast preparation.
On this dataset, we model the inter-arrival times of the actions as well as actions categories.
We explain this extension in more detail in subsection 4.2.
For the LinkedIn and MIMIC dataset, we also split the dataset into train, validation and test sets with proportions of 0.7, 0.1, 0.2, respectively.
For the
Breakfast dataset, we use the standard train and test split proposed by Kuehne et al.
Baselines.
We compare our proposed ﬂow-based approach with the state-of-the-art point process models: (I) APP-LSTM 1 is an LSTM that takes the history of past events and predicts the interarrival time distribution for the next time-step by mapping the history into the intensity of a point [Table] process distribution.
We train it by maximizing the likelihood of observed sequences under the predicted distribution.
This deterministic baseline has comparable performance to Du et al.
It only differs in the way that intensity is deﬁned; unlike Du et al.
(2016), its intensity doesn’t explicitly depend on time.
Zhong et al.
(2018) compare these two design choices, and implicit dependence was shown to be more effective in modeling point process distribution.
(II) APP-VAE (Mehrasa et al., 2019) is a latent variable framework for modeling marked temporal point process.
The model makes conditional predictions by learning a conditional latent space.
Given a history of past actions, APP-VAE generates two distributions for the next action: one over its timing (by predicting the conditional intensity and using it to deﬁne point process distributions) and one over its category.
On breakfast dataset, we use their original setup with the use of both time and mark data to have a fair comparison with APP-VAE; using mark data could help better capturing the temporal dependencies. For the rest of the datasets, we modify their approach to predict the time distribution only.
4. R ESULTS
Log-likelihood Comparison.
We report log-likelihood (LL) of test sequences across all models.
For our PPF model with the probabilistic approach in learning base distribution parameters (PPF-P; introduced in subsection 3.3) and APP-VAE baseline, we report the importance weighted autoencoder (IWAE) bound, which is a lower bound of the real log-likelihood.
To compute IWAE, at each time-step, we draw 1500 samples from the VAE’s posterior distribution and follow the standard procedure for computing IWAE.
We report the average of log-likelihood along all the time-steps of all sequences in the test dataset.
The experimental results are shown in Table 1.
The results indicate the better capability of our normalizing ﬂow-based approaches at modeling point process sequence data, especially the real-world data with complicated underlying distributions.
Our probabilistic PPF-P approach robustly outperforms state-of-the-art intensity-based baselines across all the datasets.
Without any assumption on the functional form of intensity, we are able to model the point process distribution effectively.
Our probabilistic approach also has a better performance in comparison to our deterministic approach introduced in subsection 3.2 (PPF-D).
This demonstrates the advantages of using a more ﬂexible base distribution in the ﬂow in the probabilistic approach.
Point Estimate Comparison.
We also report the mean absolute error (MAE) to evaluate the performance of our model in estimating future events timing.
The MAE between the samples of predicted time distribution and the ground-truth is reported.
To compute MAE for PPF-P and APP-VAE, at time-step i , we have two stages of sampling: (1) First, we draw samples from the prior distribution z vaei ∼ p ψ ( z vaei | τ 1: i − 1 ) .
Then, we pass each to the decoder and (2) draw samples from each predicted distribution τ i ∼ p ρ ( τ i | z vaei ) .
The MAE computation at time-step i is as follow:
[Equation]
13) where τ ∗ i is the ground-truth inter-arrival at time-step i .
We follow a similar procedure for computing the MAE for the deterministic approaches:
[Equation]
For PPF-P and APP-VAE, to estimate the expected error, we draw 100 samples from prior distribution p ψ ( z vaei | τ 1: i − 1 ) and 15 samples from each predicted base distribution p θ ( τ i | z vaei ) .
For
[Table] [Table] PPF-D and APP-LSTM, we sample 1500 predictions from the output distributions p ( τ i | τ 1: i − 1 ) at each step.
For our PPF approaches (both deterministic and probabilistic), the corresponding samples of predicted inter-arrival time distribution are obtained using Equation 5.
We report the average of
MAE along all the time-steps of all sequences in the test dataset.
Table 2 shows the experimental results for MAE metric.
We can see that our PPF-P approach is comparable to the APP-VAE baseline on the synthetic datasets.
On the more challenging real datasets, our PPF-based frameworks consistenlty outperforms baseline models (PPF-D on MIMIC, and PPF-P on LinkedIn).
The better log-likelihood estimations is also conformed by lower/competitive MAE which reﬂects the better quality of generated samples from our PPF approaches.
Extension to the Marked Temporal Point Process.
On Breakfast dataset, for our model to learn a more powerful encoding of history information, we extend our approach to marked point process which models both the inter-arrival time distribution of future event and also the distribution over its category.
At time-step i , given the history of past events including both time and mark information, in addition to modeling the time distribution of event at time-step i + 1 , we also model its category distribution.
Here, we assume that action category follows a multinomial distribution and accordingly, the log-likelihood of action category distribution modeling is added to training objective and evaluation criterion of our deterministic and probabilistic PPF models.
3 APP-LSTM is also extended to the marked case similar to Du et al.
For the experiments on Breakfast dataset, in addition to MAE, we also report the accuracy of predicting the next action category.
To compute accuracy at time-step i , for probabilistic approaches, we draw 100 samples from the prior distribution p ψ ( z vaei | τ 1: i − 1 ) and for each predicted category distribution, we select the action category with maximum probability as the predicted class.
For each time-step, the most frequently predicted type is reported as the model’s prediction.
Table 3 shows the experimental results of comparing log-likelihood, MAE, and accuracy on Breakfast dataset.
This dataset is much more challenging in comparison to LinkedIn and Mimic datasets, because of it contains various types of actions.
We can see that our probabilistic approach has a better performance in all the metrics which shows the effectiveness of our proposed model in capturing the underlying point process distribution.
C ONCLUSION
In this paper, we propose Point Process Flows (PPF), an intensity-free framework that directly models the point process as a non-parametric distribution by utilizing normalizing ﬂows.
The proposed model is capable of capturing arbitrary complex time distributions as well as performing stochastic future prediction.
The proposed PPF can be optimized by maximizing the likelihood using change of variable formula, relaxing the strict tractable likelihood constraint in previous works.
Extensive
A A PPENDIX A. A RCHITECTURE
The overall architectures are illustrated in Figure 2.
For the deterministic approach introduced in subsection 3.2, a long short-term memory (LSTM; Hochreiter & Schmidhuber 1997) network is used to model the conditional distribution p ρ ( z n | τ 1: n − 1 )
=
N ( µ ρ n , σ 2 ρ n ) .
The ﬂow module g θ is a neural ODE model described in subsection 2. where the derivative function h ( · ) is modeled by a multilayer perceptron (MLP).
For the probabilistic approach in subsection 3.
both the prior distribution p ψ ( z vae n | τ 1: n − 1 ) and the approximate posterior distribution q φ ( z vaen | τ 1: n ) are modeled by LSTMs.
The log-likelihood term log p θ,ρ ( τ n | z vaen ) is computed in two steps.
First a decoder network maps the latent variable z vaen to a base distribution p ρ ( z τn | z vaen ) which is also a normal distribution.
The decoder network is a
MLP that outputs the parameters of the base distribution.
After that, the ﬂow module g θ generates the distribution of the inter-arrival time p θ,ρ ( τ n | z vaen ) .
The architecture of g θ is the same as in the deterministic approach. [Figure] becausethere
A. I MPLEMENTATION D ETAILS
For the LSTM cells, we choose hidden size to be 128 for the synthetic data and Breakfast dataset, 64 for LinkedIn and MIMIC datasets.
The dimension of the latent space of VAE models is set to be 256 for Breakfast and synthetic datasets and 64 for LinkedIn and MIMIC .
For PPF-P model, the latent code was decoded into the mean and variance of the base distribution by two separate decoders, each with two hidden layers of size 256.
For all continuous normalizing modules, we use one block of network with 3 hidden layers of 64 dimension.
We use Adam optimizer (Kingma &
Ba, 2015) for all models with a learning rate of 0.001.
A. A MOTIVATING EXAMPLE FOR PPF-P
When using ﬂow techniques for density estimation, the expressiveness of the model is not only limited by the complexity of normalizing ﬂow transformations, but also by the class of basedistributions.
In our proposed framework, the fact that the ﬂow transformations are shared across time-steps and the underlying distribution across time-steps might vary a lot makes our model more sensitive to the choice of the base-distribution family.
By introducing a latent variable z vaen such that the z τn follows different Gaussian distributions conditioned on different samples of z vaen , the distributions of z τn after marginalizing z vaen becomes highly ﬂexible.
To motivate this argument, we make the following proposition, show its proof and substantiate it with experiment results on PPF-P and PPF-D models.
Proposition 1.
Let f : R → R be a bijective singular mapping that satisﬁes the following: z ∼
N ( µ 0 , σ 20 ) and x = f ( z ) follows a mixture of Gaussian distribution of two components, N ( µ 1 , σ 21 ) and N ( µ 2 , σ 22 ) , with weights a and 1 − a for some a ∈ (0 , 1) .
There exists i ∈ { 1 , 2 } , such that if x is sampled from component i of the Gaussian mixture distribution x ∼ N ( µ i , σ 2 i ) , f − 1 ( x ) does not follow a Gaussian distribution.
In summary, Proposition 1 says if a normalizing ﬂow can maps a Gaussian distribution to a mixture of Gaussian distributions, which is multi-modal, we can not obtain one of the mixture distribution’s components by applying the same normalizing ﬂow to another Gaussian distribution.
Continuous normalizing ﬂow deﬁnes a continuous bijective mapping from R to R and therefore it must be singular increasing or decreasing.
The normalizing ﬂow we used is shared for all time steps and its derivative is independent of the parameters of the base distribution.
It is also worth noting that this proposition can be extended to scenarios of any Gaussian mixture distributions with ﬁnite components.
Proof.
Without loss of generality, consider the following two cases:
Case 1 σ 21 > σ 22 .
We show that the inverse mapping of x ∼ N ( µ 1 , σ 21 ) is not a Gaussian by contradiction.
Suppose z = f − 1 ( x ) follows a Gaussian distribution N ( µ 3 , σ 23 ) .
By our assumptions and the change of varaible theorems we have the following
[Equation]
Jacobian is independent of the distribution. )
where
[Equation]
Rewriting the equality above, we get
[Equation]
for some constants c , d , e , f
and g where one of e and f is non-zero.
Since σ 21 < σ 22 , we know (cid:16) 12 σ 21 − 12 σ(cid:17) x
2 + c x + d → −∞ and LHS → a > 0 as x → ∞ or −∞ .
When x → ∞ or −∞ , we have z → ∞ or −∞ as well since f is bijective on R and singular.
However, the RHS can only converge to 0 or diverge to ∞ when taking the limit of z .
We get a contradiction.
Case 2 σ 21 =
σ 22 .
We also show that the inverse mapping of X ∼ N ( µ 1 , σ 21 ) is not a Gaussian by contradiction.
Following similar steps in Case 1 , we get
[Equation]
[Table]
for some constant c , d , e , f and g where c must be non-zero and at least one of e and f is non-zero.
Taking the limit of x such that c x + b → −∞ , we have LHS → a .
By the bijectivity and singularity of f , we know z goes to either ∞ or −∞ .
However, in either case, the RHS can only diverge to ∞ or converge to 0 .
We get a contradiction.
To complement Proposition 1, we generate sequential data with the following property to train PPFD and PPF-P models to ﬁt the data: the underlying distribution of values at each time-step switches between a mixture of Gaussians distribution and one component of the mixture distribution.
More speciﬁcally, the underlying distribution of observations at even time steps follows N (4 , 1) and the underlying distribution for odd time steps follow a Gaussian mixture distribution of two components
N (4 , 1) and N (10 , 1) with equal weights.
We created a dataset of 1000 sequences where each sequence has the length of 15, and trained both PPF-D and PPF-P on this dataset.
Figure 3 shows the experimental results for this experiment.
We can see that PPF-D is not able to handle data generated by this distribution very well.
The output distribution of PPF-D tries to cover both components of the Gaussian mixture distribution, but most of the samples are concentrated in an area of low probability.
In contrast, we can see that PPF-P, with more ﬂexible base distribution, is much better at modeling sequences sampled from our synthetic switching distribution.
Most of the data sampled from PPF-P model lie in the high-probability region: At odd time step, the sampled data can hit both components of the Gaussian mixture model and at even time step, the sampled data can also recover the ground truth distribution.
Table 4 shows the log-likelihood of test sequences under the distribution learned from our model vs. the log-likelihood under the true distribution.
The better estimation of PPF-P is conformed by a higher log-likelihood.
We also reported the difference of log-likelihood under the true distribution and log-likelihood under the learned model.
PPF-P has a lower score which shows it performs better in estimating the true underlying distribution.