I NTERPOLATION -P REDICTION N ETWORKS FOR I RREGULARLY S AMPLED T IME S ERIES
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper, we present a new deep learning architecture for addressing the problem of supervised learning with sparse and irregularly sampled multivariate time series.
The architecture is based on the use of a semi-parametric interpolation network followed by the application of a prediction network.
The interpolation network allows for information to be shared across multiple dimensions of a multivariate time series during the interpolation stage, while any standard deep learning model can be used for the prediction network.
This work is motivated by the analysis of physiological time series data in electronic health records, which are sparse, irregularly sampled, and multivariate.
We investigate the performance of this architecture on two datasets for both classiﬁcation and regression tasks, showing that our approach outperforms a range of baseline and recently proposed models.
I NTRODUCTION
Over the last several years, there has been signiﬁcant progress in developing specialized models and architectures that can accommodate sparse and irregularly sampled time series as input (Marlin et al., 2012; Li & Marlin, 2015; 2016; Lipton et al., 2016;
Che et al., 2018a; Futoma et al., 2017).
An irregularly sampled time series is a sequence of samples with irregular intervals between their observation times.
Irregularly sampled data are considered to be sparse when the intervals between successive observations are often large.
Of particular interest in the supervised learning setting are methods that perform end-to-end learning directly using multivariate sparse and irregularly sampled time series as input without the need for a separate interpolation or imputation step.
In this work, we present a new model architecture for supervised learning with multivariate sparse and irregularly sampled data: Interpolation-Prediction Networks.
The architecture is based on the use of several semi-parametric interpolation layers organized into an interpolation network, followed by the application of a prediction network that can leverage any standard deep learning model.
In this work, we use GRU networks (Chung et al., 2014) as the prediction network.
The interpolation network allows for information contained in each input time series to contribute to the interpolation of all other time series in the model.
The parameters of the interpolation and prediction networks are learned end-to-end via a composite objective function consisting of supervised and unsupervised components.
The interpolation network serves the same purpose as the multivariate Gaussian process used in the work of Futoma et al.
(2017), but remove the restrictions associated with the need for a positive deﬁnite covariance matrix.
Our approach also allows us to compute an explicit multi-timescale representation of the input time series, which we use to isolate information about transients (short duration events) from broader trends.
Similar to the work of Lipton et al.
(2016) and Che et al.
(2018a)
, our architecture also explicitly leverages a separate information channel related to the pattern of observed values.
However, our representation uses a semi-parametric intensity function representation of this information that is more closely related to the work of Lasko (2014) on modeling medical event point processes.
Our architecture thus produces three output time series for each input time series: a smooth interpolation modeling broad trends in the input, a short time-scale interpolation modeling transients, and an intensity function modeling local observation frequencies.
This work is motivated by problems in the analysis of electronic health records (EHR) (Marlin et al., 2012; Lipton et al., 2016;
Che et al., 2018a; Futoma et al., 2017).
It remains rare for hospital systems to capture dense physiological data streams.
Instead, it is common for the physiological time series data in electronic health records to be both sparse and irregularly sampled.
The additional issue of the lack of alignment in the observation times across physiological variables is also very common.
We evaluate the proposed architecture on two datasets for both classiﬁcation and regression tasks.
Our approach outperforms a variety of simple baseline models as well as the basic and advanced
GRU models introduced by Che et al.
(2018a) on both tasks across several metrics.
We also compare our model with Gaussian process adapter (Li & Marlin, 2016) and though our model performs equally well in comparison, it runs 50x faster.
Further, we perform full ablation testing of the information channels our architecture can produce to assess their impact on classiﬁcation and regression performance.
R ELATED W ORK
The problem of interest in this work is learning supervised machine learning models from sparse and irregularly sampled multivariate time series.
As described in the introduction, a sparse and irregularly sampled time series is a sequence of samples with large and irregular intervals between their observation times.
Such data commonly occur in electronic health records, where they can represent a signiﬁcant problem for both supervised and unsupervised learning methods (Yadav et al., 2018).
Such sparse and irregularly sampled data also commonly occur in a range of other areas with similar complex observation processes including climate science (Schulz & Stattegger, 1997), ecology (Clark & Bjørnstad, 2004), biology (Ruf, 1999), and astronomy (Scargle, 1982).
A closely related problem, is the much more prevalent problem of performing supervised learning in the presence of missing data (Batista & Monard, 2003).
The primary difference is that the missing data problem is generally deﬁned with respect to a ﬁxed-dimensional feature space (Little & Rubin, 2014).
In the irregularly sampled time series problem, observations typically occur in continuous time and there may be no notion of a “normal” or “expected” sampling frequency for some domains.
Methods for dealing with missing data in supervised learning include the pre-application of imputation methods (Sterne et al., 2009), and learning joint models of features and labels (Williams et al., 2005).
Joint models can either be learned generatively to optimize the joint likelihood of features and labels, or discriminately to optimize the conditional likelihood of the labels.
The problem of irregular sampling can be converted to a missing data problem by discretizing the time axis into non-overlapping intervals.
Intervals with no observations are then said to contain missing values.
This is the approach taken to deal with irregular sampling by Marlin et al.
(2012) as well as Lipton et al.
The disadvantage of this approach is that it forces a choice of discretization interval length.
When the intervals are long, there will be less missing data, but there can also be multiple observations in the same interval, which must be accounted for using ad-hoc methods.
When the intervals are shorter, most intervals will contain at most one value, but many intervals may be empty.
Learning is generally harder as the amount of missing data increases, so choosing a discretization interval length must be dealt with as a hyper-parameter of such a method.
One important feature of missing data problems is the potential for the pattern of observed and missing values for itself to be informative (Little & Rubin, 2014).
Since the set of missing data indicators is always observed, this information is typically easy to condition on.
This technique has been used successfully to improve models in the domain of recommender systems (Salakhutdinov et al., 2007).
It was also used by Lipton et al.
(2016) to improve performance of their GRU model.
The alternative to pre-discretizing irregularly sampled time series to convert the problem of irregular sampling into the problem of missing data is to construct models with the ability to directly use an irregularly sampled time series as input.
The machine learning and statistics literature include several models with this ability.
In the probabilistic setting, Gaussian process models have the ability to represent continuous time data via the use of mean and covariance functions (Rasmussen, 2006).
These models have non-probabilistic analogues that are similarly deﬁned in terms of kernels.
For example, Lu et al.
(2008) present a kernel-based method that can be used to produce a similarity function between two irregularly sampled time series.
Li & Marlin (2015) subsequently provided a signiﬁcant generalization of this approach to the case of kernels between Gaussian process models.
Li & Marlin (2016) showed how the re-parameterization trick (Kingma et al., 2015) could be used to extend these ideas to enable end-to-end training of a deep neural network model (feed-forward, convolutional, or recurrent) stacked on top of a Gaussian process layer via backpropagation.
While the basic model of Li & Marlin (2016) was only applied to univariate time series, in follow-up work the model was extended to multivariate time series using a multi-output Gaussian process regression model (Futoma et al., 2017).
However, modeling multivariate time series within this framework is quite challenging due to the constraints on the covariance function used in the GP layer.
Futoma et al.
(2017) deal with this problem using a sum of separable kernel functions (Bonilla et al., 2008), which limit the expressiveness of the model.
An important property of the above models is that they allow for incorporating all of the information from all available time points into a global interpolation model.
Variants differ in terms of whether they only leverage the posterior mean when the ﬁnal supervised problem is solved, or whether the whole posterior over interpolations is leveraged.
A separate line of work has looked at the use of more local interpolation methods while still operating directly over continuous time inputs.
For example, Che et al.
(2018a) presented several methods based on gated recurrent unit (GRU) networks (Chung et al., 2014) combined with simple imputation methods including mean imputation and forward ﬁlling with past values.
Che et al.
(2018a) additionally considered an approach that takes as input a sequence consisting of both the observed values and the timestamps at which those values were observed.
The previously observed input value is decayed over time toward the overall mean.
In another variant the hidden states are similarly decayed toward zero.
Yoon et al.
(2017) presented another similar approach based on multi-directional RNN which operate across streams in addition to within streams.
However, these models are limited to using global information about the structure of the time series via its empirical mean value, and current or past information about observed values.
The global structure of the time series is not directly taken into account.
Che et al.
(2018b) focus on a similar problem of modeling multi-rate multivariate time series data.
This is similar to the problem of interest in a way that the observations across time series could be asynchronous.
The difference is that the observations in each time series is uniformly spaced which is a simpler case.
In case of missing data, they use forward or linear interpolation which again does not capture the global structure of time series.
Our interpolation network can work as a backend in such case to model irregular sampling/ missing data.
Similarly, Binkowski et al.
(2018) presented an autoregression-type framework for regression tasks in irregularly sampled time series data.
It is not clear how it can be extended for classiﬁcation tasks.
On the contrary, our interpolation-prediction network is ﬂexible and can be applied for both classiﬁcation and regression tasks.
The model proposed in this work is similar to that of Li & Marlin (2016) and Futoma et al.
(2017) in the sense that it consists of global interpolation layers.
The primary difference is that these prior approaches used Gaussian process representations within the interpolation layers.
The resulting computations can be expensive and, as noted, the design of covariance functions in the multivariate case can be challenging.
By contrast, our proposed model uses semi-parametric, deterministic, feed-forward interpolation layers.
These layers do not encode uncertainty, but they do allow for very ﬂexible interpolation both within and across layers.
Also similar to Li & Marlin (2016) and Futoma et al.
(2017), the interpolation layers in our architecture produce regularly sampled interpolants that can serve as inputs for arbitrary, unmodiﬁed, deep classiﬁcation and regression networks.
This is in contrast to the approach of Che et al.
(2018a), where a recurrent network architecture was directly modiﬁed, reducing the modularity of the approach.
Finally, similar to Lipton et al.
(2016), our model includes information about the times at which observations occur.
However, instead of pre-discretizing the inputs and viewing this information in terms of a binary observation mask or set of missing data indicators, we directly model the sequence of observation events as a point process in continuous time using a semi-parametric intensity function (Lasko, 2014).
M ODEL F RAMEWORK
In this section, we present the proposed modeling framework.
We begin by presenting notation, followed by the model architecture and learning criteria.
3. N OTATION
We let D =
{ ( s i , y i ) | i = 1 , ..., N } represent a data set containing N data cases.
An individual data case consists of a single target value y n (discrete in the case of classiﬁcation and real-valued in the case of regression), as well as a D -dimensional, sparse and irregularly sampled multivariate time series s n .
Different dimensions d of the multivariate time series can have observations at different times, as well as different total numbers of observations L dn .
Thus, we represent time series d for data case n as a tuple s dn =
( t dn , x dn ) where t dn =
[ t 1 dn , ..., t L dn dn ] is the list of time points at which observations are deﬁned and x dn =
[ x 1 dn , ..., x L dn dn ] is the corresponding list of observed values.
3. M ODEL A RCHITECTURE
The overall model architecture consists of two main components: an interpolation network and a prediction network.
The interpolation network interpolates the multivariate, sparse, and irregularly sampled input time series against a set of reference time points r =
[ r 1 , ..., r T ] .
We assume that all of the time series are deﬁned within a common time interval (for example, the ﬁrst 24 or 48 hours after admission for MIMIC-III dataset).
The T reference time points r t are chosen to be evenly spaced within that interval.
In this work, we propose a two-layer interpolation network with each layer performing a different type of interpolation.
The second component, the prediction network, takes the output of the interpolation network as its input and produces a prediction ˆ y n for the target variable.
The prediction network can consist of any standard supervised neural network architecture (fully-connected feedforward, convolutional, recurrent, etc).
Thus, the architecture is fully modular with respect to the use of different prediction networks.
In order to train the interpolation network, the architecture also includes an auto-encoding component to provide an unsupervised learning signal in addition to the supervised learning signal from the prediction network.
Figure 1 shows the architecture of the proposed model.
We describe the components of the model in detail below.
3.2. I NTERPOLATION N ETWORK
We begin by describing the interpolation network.
The goal of the interpolation network is to provide a collection of interpolants of each of the D dimensions of an input multivariate time series deﬁned at the T reference time points r =
[ r 1 , ..., r T ] .
In this work, we use a total of two interpolation layers and C = 3 outputs for each of the D input time series.
The three outputs (discussed in detail below) capture long-term trends, transients, and frequency information.
We refer to the interpolation network mapping as the function ˆ s n = f θ ( s n ) where the dimensions of ˆ s n is ( DC ) × T .
The ﬁrst interpolation layer in the interpolation network performs a semi-parametric univariate interpolation for each of the D time series separately.
The interpolation is based on a radial basis function network.
This results in a set of values ˆ x 1 c kdn for each data case n , each input time series d , and each reference time point r k , and each interpolation channel c as shown in Equation 1.
[Equation]
The second interpolation layer merges information from across all of the D time series at each reference time point by taking into account learnable correlations ρ dd (cid:48) from across all time series.
This results in a set of values ˆ x 2 ckdn for each data case n , each input time series d , each reference time point r k , and each interpolation channel c as shown in Equation 2.
Here, the terms i kdn represent the intensity of the observations on input dimension d for data case n .
The more observations that occur near reference time point r k , the larger the value of i kdn .
This ﬁnal interpolation layer thus models the interpolant ˆ x 2 ckdn as a weighted linear combination of the ﬁrst layer interpolants, while focusing the combination on the time series d (cid:48) for which data are actually available at nearby time points.
[Equation]
In the experiments presented in the next section, we use a total of three interpolation outputs corresponding to a smooth interpolant to capture trends, a non-smooth interpolant to capture transients, and the intensity function to retain information about where observations occur.
The smooth interpolant corresponds to the ﬁrst interpolant ˆ x 21 dn =
[ˆ x 211 dn , ..., ˆ x 21 Tdn ] 1 .
The collection of model parameters associated with this interpolant are the cross-dimension correlation coefﬁcients ρ dd (cid:48) , and the RBF network bandwidths α d 1 for all d and d (cid:48) .
For the non-smooth interpolant, we start with a second interpolant ˆ x 12 dn =
[ˆ x 121 dn , ..., ˆ x 12 Tdn ] .
The collection of model parameters associated with this interpolant are the RBF network parameters α dfor all d .
To accomplish the goal of having this component represent a less smooth interpolation than ˆ x 21 dn , we enforce the relationship α d 2 = κα d 1 for all d for a value of κ greater than one.
This ensures that the temporal similarity decays faster for the component intended to model transients.
To further minimize any redundancy between ˆ x 21 dn and ˆ x 12 dn , we subtract the smooth interpolant from the non-smooth interpolant leaving the non-smooth residual: ˆ x (cid:48) 12 dn = ˆ x 12 dn − ˆ x 21 dn 1 .
Finally, for the intensity function, we use i 1 dn =
[ i 11 dn , ..., i 1 Tdn ] (Eq 2).
This component shares its
RBF network parameters with ˆ x 21 dn .
In the experiments, we study the end-to-end impact of each of these interpolation outputs.
3.2. P REDICTION N ETWORK
Following the application of the interpolation network, all D dimensions of the input multivariate time series have been re-represented in terms of C outputs deﬁned on the regularly spaced set of reference time points r 1 , ..., r T (again, in our experiments, we use C = 3 as described in the previous section).
Again, we refer to the complete set of interpolation network outputs as ˆ s n = f θ ( s n ) , which can be represented as a matrix of size ( DC ) × T .
The prediction network must take ˆ s n as input and output a prediction ˆ y n = g φ (ˆ s n )
=
g φ ( f θ ( s n )) of the target value y n for data case n .
There are many possible choices for this component of the model.
For example, the matrix ˆ s n can be converted into a single long vector and provided as input to a standard multi-layer feedforward network.
A temporal convolutional model or a recurrent model like a GRU or LSTM can instead be applied to time slices of the matrix ˆ s n .
In this work, we conduct experiments leveraging a GRU network as the prediction network.
3.2. L EARNING
To learn the model parameters, we use a composite objective function consisting of a supervised component and an unsupervised component.
This is due to the fact that the supervised compo- nent alone is insufﬁcient to learn reasonable parameters for the interpolation network parameters given the amount of available training data.
The unsupervised component used corresponds to an autoencoder-like loss function.
However, the semi-parametric RBF interpolation layers have the ability to exactly ﬁt the input points by setting the RBF kernel parameters to very large values.
To avoid this solution and force the interpolation layers to learn to properly interpolate the input data, it is necessary to hold out some observed data points x jdn during learning and then to compute the reconstruction loss only for these data points.
This is a well-known problem with high-capacity autoencoders, and past work has used similar strategies to avoid the problem of trivially memorizing the input data without learning useful structure.
To implement the autoencoder component of the loss, we introduce a set of masking variables m jdn for each data point ( t jdn , x jdn ) .
If m jdn = 1 , then we remove the data point ( t jdn , x jdn ) as an input to the interpolation network, and include the predicted value of this time point when assessing the autoencoder loss.
We use the shorthand notation m n (cid:12) s n to represent the subset of values of s n that are masked out, and (1 − m n ) (cid:12) s n to represent the subset of values of s n that are not masked out.
The value ˆ x jdn that we predict for a masked input at time point t jdn is the value of the smooth interpolant at that time point, calculated based on the un-masked input values:
ˆ x jdn = f 21 ( t jdn , (1 − m n ) (cid:12) s n ) .
Using these deﬁnitions, we can now deﬁne the learning problem for the proposed framework.
We let (cid:96) P be the loss for the prediction network (we use cross-entropy loss for classiﬁcation and squared error for regression).
We let (cid:96) I be the interpolation network autoencoder loss (we use standard squared error).
We also include (cid:96) 2 regularizers for both the interpolation and prediction networks parameters.
[Equation]
E XPERIMENTS AND R ESULTS
Our experiments focus on both classiﬁcation and regression task with sparse and irregularly sampled multivariate time series.
In both cases, the input to the prediction task is a sparse and irregularly sampled time series, and the output is a single scalar representing either the predicted class or the regression target variable.
We test the model framework on two publicly available real-world datasets:
MIMIC-III 2 − multivariate time series dataset consisting of sparse and irregularly sampled physiological signals collected at Beth Israel Deaconess Medical Center from 2001 to 2012 (Johnson et al., 2016), and UWaveGesture 3 − an univariate time series data consisting of simple gesture patterns divided into eight categories (Liu et al., 2009).
Details of each dataset can be found in the Appendix
A.1.
We use mortality and length of stay prediction task on MIMIC-III as an example classiﬁcation and regression task respectively.
We use gesture (univariate time series) classiﬁcation task on
UWave dataset for comparing the training time and performance of baseline models on test set.
4. B ASELINE M ODELS
We compare our proposed model with a number of baseline approaches including off-the-shelf classiﬁcation and regression models learned using basic features, as well as more recent approaches based on customized neural network models.
4.1. N ON -N
EURAL
N
ETWORK
B ASELINES
For non-neural network baselines, we evaluate Logistic Regression (Hosmer Jr et al., 2013), Support Vector Machines (SVM) (Cortes & Vapnik, 1995), Random Forest (RF) (Breiman, 2001) and
AdaBoost (Freund & Schapire, 1997) for the classiﬁcation task.
For the length of stay prediction task, we apply Linear Regression (Hastie et al., 2001), Support
Vector Regression (SVR), AdaBoost Regression (Drucker, 1997) and Random Forest Regression.
Standard instances of all of these models require ﬁxed-size feature representations.
We use forward ﬁlling to create ﬁxed-size representation in case of missing data and use this representation as feature set for non-neural network baselines.
4.1. N EURAL N ETWORK M ODELS
We compare to several existing deep learning baselines built on GRUs using simple interpolation or imputation approaches.
In addition, we compare to current state-of-the-art models for mortality prediction including the work of Che et al.
(2018a).
Their work proposed to handle irregularly sampled and missing data using recurrent neural networks (RNNs) by introducing temporal decays in the input and/or hidden layers.
We also evaluate the scalable end-to-end Gaussian process adapter for irregularly sampled time series classiﬁcation (Li & Marlin, 2016).
This work is discussed in detail in Section 2.
The complete set of models that we compare to is as follows:
• GP-GRU: Scalable end-to-end Gaussian process adapter with GRU as classiﬁer.
• GRU-M: Missing observations replaced with the global mean of the variable across the training examples.
• GRU-F: Missing values set to last observed measurement within that time series (referred to as forward ﬁlling).
• GRU-S: Missing variable replaced with the global mean and the input is concatenated with masking variable and time interval indicating how long the particular variable is missing.
• GRU-D:
In order to capture richer information, decay is introduced in the input as well as hidden layer on top of the structure of GRU-S. Instead of just replacing the missing value with the last measurement, it is decayed over time towards the empirical mean.
• GRU-HD: A variation of GRU-D where decay in only introduced in the hidden layer.
4. R ESULTS
In this section, we present the results of the classiﬁcation and regression experiments, as well as the results of ablation testing of the internal structure of the interpolation network for the proposed model.
We use UWaveGesture dataset to compare the training time and classiﬁcation performance of our baseline models.
We have standard train and test set in this case whose details are given in appendix A.1.
We report the training time taken for convergence and performance in terms of accuracy on test set.
For MIMIC-III, we create our own dataset 4 (appendix A.1) and report the results from the 5-fold cross validation in terms of the average area under the ROC curve (AUC score), average area under precision-recall curve (AUPRC score) and average cross-entropy loss for classiﬁcation task and average median absolute error and average fraction of explained variation score (EV) for regression task.
We also report the standard deviation over cross validation folds.
Because of complexity issues with Gaussian processes we evaluate it only for the univariate time series data.
Training and implementation details can be found in appendix A.2.
Figure 2 shows the classiﬁcation performance on the UWaveGesture dataset.
The proposed model and the Gaussian process adapter (Li & Marlin, 2016) signiﬁcantly outperform the rest of the baselines.
It is worth noting that the proposed model achieves similar performance to Gaussian process with a 50x speed up.
However, modeling multivariate time series within GP adapter framework is quite challenging due to the constraints on the covariance function used in the GP layer.
On the other hand, the training time of the proposed model is around the same order as other GRU-based model but it achieves much better accuracy.
We note that GRU-M and GRU-F are same model for this case because of the absence of missing data in this dataset.
Table 1 compares the predictive performance of the mortality and length of stay prediction task on
MIMIC-III.
We note that in highly skewed datasets as is the case of MIMIC-III, AUPRC (Davis &
Goadrich, 2006) can give better insights about the classiﬁcation performance as compared to AUC
[Figure]
Training time in seconds (log scale)
[Figure]
[Figure] [Figure]
cc
[Figure]
cy
LogReg
SVM
AdaBoost
Random Forest
GRU-
[Figure] [Figure]
/HD
GP-GRU Proposed
[Table]
score.
The proposed model consistently achieves the best average score over all the metrics.
We note that a paired t-test indicates that the proposed model results in statistically signiﬁcant improvements over all baseline models ( p < 0 . 01) with respect to all the metrics except median absolute error.
The version of the proposed model used in this experiment includes all three interpolation network outputs (smooth interpolation, transients
[Table]
and intensity function).
Ablation study shows that the results on regression task can further be improved by using only two outputs (transients, and intensity function), achieving statistically signiﬁcant improvements over all the baselines.
Results for the ablation study are given in appendix A.3.
D ISCUSSION AND C ONCLUSIONS
In this paper
[Table]
we have presented a new framework for dealing with the problem of supervised learning in the presence of sparse and irregularly sampled time series.
The proposed framework is fully modular.
It uses an interpolation network to accommodate the complexity that results from using sparse and irregularly sampled data as supervised learning inputs, followed by the application of a prediction network that operates over the regularly spaced and fully observed, multi-channel output provided by the interpolation network.
The proposed approach also addresses some difﬁculties with prior approaches including the complexity of the Gaussian process interpolation layers used in (Li &
Marlin, 2016), and the lack of modularity in the approach of Che et al.
(2018a).
Our framework also introduces novel elements including the use of semi-parametric, feed-forward interpolation layers, and the decomposition of an irregularly sampled input time series into multiple distinct information channels.
Our results show statistically signiﬁcant improvements for both classiﬁcation and regression tasks over a range of baseline and state-of-the-art methods
A A PPENDIX A. D ATASET D ESCRIPTION A.1. MIMIC-III D ATASET
We evaluate our model framework on the publicly available MIMIC-III dataset (Johnson et al., 2016).
It is a de-identiﬁed dataset collected at Beth Israel Deaconess Medical Center from 2001 to 2012.
It consists of approximately 58,000 hospital admission records.
This data set contains sparse and irregularly sampled physiological signals, medications, diagnostic codes, in-hospital mortality, length of stay and more.
We focus on predicting in-hospital mortality and length of stay using the ﬁrst 48 hours of data.
We extracted 12 standard physiological variables from each of the 53, records obtained after removing hospital admission records with length of stay less than 48 hours.
Table 2 shows the features, sampling rates (per hour) and their missingness information computed using the union of all time stamps that exist in any dimension of the input time series.
[Table]
Prediction Tasks
In our experiments, each admission record corresponds to one data case ( s n , y n ) .
Each data case n consists of a sparse and irregularly sampled time series s n with D = 12 dimensions.
Each dimension d of s n corresponds to one of the 12 vital sign time series mentioned above.
In the case of classiﬁcation, y n is a binary indicator where y n = 1 indicates that the patient died at any point within the hospital stay following the ﬁrst 48 hours and y n = 0 indicates that the patient was discharged at any point after the ﬁrst 48 hours.
There are 4310 (8.1%) patients with a y n =mortality label.
The complete data set is D = { ( s n , y n ) | n = 1 , ..., N } , and there are N = 53 ,data cases.
The goal in the classiﬁcation task is to learn a classiﬁcation function h of the form
ˆ y n ← h
( s n ) where ˆ y n is a discrete value.
In the case of regression, y n is a real-valued regression target corresponding to the length of stay.
Since the data set include some very long stay durations, we let y n represent the log of the length of stay in days for all models.
We convert back from the log number of days to the number of days when reporting results.
The complete data set is again D
= { ( s n , y n ) | n = 1 , ..., N } with
N = 53 , 211 data cases (we again require 48 hours worth of data).
The goal in the regression task is to learn a regression function h of the form
ˆ y n ← h
( s n ) where ˆ y n is a continuous value.
A.1. UW AVE D ATASET
UWave dataset is an univariate time series data consisting of simple gesture patterns divided into eight categories.
The dataset has been split into 3582 train and 896 test instances.
Out of the training data, 30% is used for validation.
Each time series contains 945 observations.
We follow the same data preparation method as in Li & Marlin (2016) where we randomly sample 10% of the observations points from each time series to create a sparse and irregularly sampled data.
A. I MPLEMENTATION D ETAILS A.2. P ROPOSED M ODEL
The model is learned using the Adam optimization method in TensorFlow with gradients provided via automatic differentiation.
However, the actual multivariate time series representation used during learning is based on the union of all time stamps that exist in any dimension of the input time series.
Undeﬁned observations are represented as zeros and a separate missing data mask is used to keep track of which time series have observations at each time point.
Equations 1 & 2 are modiﬁed such that data that are not available are not taken into account at all.
This implementation is exactly equivalent to the computations described in Equations 1 & 2, but support parallel computation across all dimensions of the time series for a given data case.
Finally, we note that the learning problem can be solved using a doubly stochastic gradient based on the use of mini batches combined with re-sampling the artiﬁcial missing data masks used in the interpolation loss.
In practice, we randomly select 20% of the observed data points to hold out from every input time series.
A.2. B ASELINES
The Logistic Regression model is trained with cross entropy loss with regularization strength set to 1.
The support vector classiﬁer is used with a RBF kernel and trained to minimize the soft margin loss.
We use the cross entropy loss on the validation set to select the optimal number of estimators in case of Adaboost and Random Forest.
Similar to the classiﬁcation setting, the optimal number of estimators for regression task in Adaboost and Random Forest is chosen on the basis of squared error on validation set.
MIMIC-III D ATASET
We evaluate all models using a ﬁve-fold cross-validation estimate of generalization performance.
In the classiﬁcation setting, all the deep learning baselines are trained to minimize the cross entropy loss while the proposed model uses a composite loss consisting of cross-entropy loss and interpolation loss (with δ = 1 ) as described in section 3 . 2 . 3 .
In the case of the regression task, all baseline models are trained to minimize squared error and the proposed model is again trained with a composite loss consisting of squared error and interpolation loss.
For all of the GRU-based models, we use the already speciﬁed parameters (Che et al., 2018a).
The models are learned using the Adam optimization.
Early stopping is used on a validation set subsampled from the training folds.
In the classiﬁcation case, the ﬁnal outputs of the GRU hidden units are used in a logistic layer that predicts the class.
In the regression case, the ﬁnal outputs of the GRU hidden units are used as input for a dense hidden layer with 50 units, followed by a linear output layer.
UW AVE D ATASET
We independently tune the hyperparameters of each baseline method.
For GRU-based method, hidden units are searched over the range { 2 5 , 2 6 , · · · , 2 11 } .
Learning is done in same way as described above.
We evaluate all the baseline models on the test set and compare the training time and performance based on accuracy.
For Gaussian process, we use the squared exponential covariance function.
We use same number of inducing points for both the Gaussian process and the proposed model.
Gaussian process is jointly trained with GRU using stochastic gradient descent with Nesterov momentum.
We apply early stopping based on the validation set.
A. A DDITIONAL E XPERIMENTS
In this section, we address the question of the relative information content of the different outputs produced by the interpolation network used in the proposed model for MIMIC-III dataset.
Recall that for each of the D = 12 vital sign time series, the interpolation network produces three outputs: a smooth interpolation output (SI), a non-smooth or transient output (T), and an intensity function (I).
The above results use all three of these outputs.
To assess the impact of each of the interpolation network outputs, we conduct a set of ablation experiments where we consider using all sub-sets of outputs for both the classiﬁcation task and for the regression task.
Table 3 shows the results from ﬁve-fold cross validation mortality and length of stay prediction experiments.
When using each output individually, smooth interpolation (SI) provides the best performance in terms of classiﬁcation.
Interestingly, the intensity output is the best single information source for the regression task and provides at least slightly better mean performance than any of the baseline methods shown in Table 2.
Also interesting is the fact that the transients output performs signiﬁcantly worse when used alone than either the smooth interpolation or the intensity outputs in classiﬁcation task.
When considering combinations of interpolation network components, we can see that the best performance is obtained when all three outputs are used simultaneously in classiﬁcation tasks.
For regression task, intensity output provides better performance in terms of median absolute error while combination of intensity and transients output provide better explained variance score.
However, the use of the transients output contributes almost no improvement in the case of the AUC and cross entropy loss for classiﬁcation relative to using only smooth interpolation and intensity.
Interestingly, in the classiﬁcation case, there is a signiﬁcant boost in performance by combining smooth interpolation and intensity relative to using either output on its own.
In the regression setting, smooth interpolation appear to carry little information.
[Table]