M ULTI -T IME A TTENTION N ETWORKS FOR I RREGULARLY S AMPLED T IME S ERIES
A BSTRACT
Irregular sampling occurs in many time series modeling applications where it presents a signiﬁcant challenge to standard deep learning models.
This work is motivated by the analysis of physiological time series data in electronic health records, which are sparse, irregularly sampled, and multivariate.
In this paper, we propose a new deep learning framework for this setting that we call MultiTime Attention Networks .
Multi-Time Attention Networks learn an embedding of continuous time values and use an attention mechanism to produce a ﬁxed-length representation of a time series containing a variable number of observations.
We investigate the performance of this framework on interpolation and classiﬁcation tasks using multiple datasets.
Our results show that the proposed approach performs as well or better than a range of baseline and recently proposed models while offering signiﬁcantly faster training times than current state-of-the-art methods.
I NTRODUCTION
Irregularly sampled time series occur in application domains including healthcare, climate science, ecology, astronomy, biology and others.
It is well understood that irregular sampling poses a signiﬁcant challenge to machine learning models, which typically assume fully-observed, ﬁxed-size feature representations (Marlin et al., 2012; Yadav et al., 2018).
While recurrent neural networks (RNNs) have been widely used to model such data because of their ability to handle variable length sequences, basic RNNs assume regular spacing between observation times as well as alignment of the time points where observations occur for different variables (i e., fully-observed vectors).
In practice, both of these assumptions can fail to hold for real-world sparse and irregularly observed time series.
To respond to these challenges there has been signiﬁcant progress over the last decade on building and adapting machine learning models that can better capture the structure of irregularly sampled multivariate time series (Li & Marlin, 2015; 2016; Lipton et al., 2016;
Futoma et al., 2017; Che et al., 2018; Shukla & Marlin, 2019; Rubanova et al., 2019).
In this work, we introduce a new model for multivariate, sparse and irregularly sampled time series that we refer to as Multi-Time Attention networks or mTANs.
mTANs are fundamentally continuous-time, interpolation-based models.
Their primary innovations are the inclusion of a learned continuous-time embedding mechanism coupled with a time attention mechanism that replaces the use of a ﬁxed similarity kernel when forming representation from continuous time inputs.
This gives mTANs more representational ﬂexibility than previous interpolation-based models (Shukla & Marlin, 2019).
Our approach re-represents an irregularly sampled time series at a ﬁxed set of reference points.
The proposed time attention mechanism uses reference time points as queries and the observed time points as keys.
We propose an encoder-decoder framework for end-to-end learning using an mTAN module to interface with given multivariate, sparse and irregularly sampled time series inputs.
The encoder takes the irregularly sampled time series as input and produces a ﬁxed-length latent representation over a set of reference points, while the decoder uses the latent representations to produce reconstructions conditioned on the set of observed time points.
Learning uses established methods for variational autoencoders (Rezende et al., 2014; Kingma & Welling, 2014).
The main contributions of the mTAN model framework are: (1) It provides a ﬂexible approach to modeling multivariate, sparse and irregularly sampled time series data (including irregularly sampled time series of partially observed vectors) by leveraging a time attention mechanism to learn temporal similarity from data instead of using ﬁxed kernels.
(2) It uses a temporally distributed latent representation to better capture local structure in time series data.
(3) It provides interpolation and classiﬁcation performance that is as good as current state-of-the-art methods or better, while providing signiﬁcantly reduced training times.
R ELATED W ORK
An irregularly sampled time series is a time series with irregular time intervals between observations.
In the multivariate setting, there can also be a lack of alignment across different variables within the same multivariate time series.
Finally, when gaps between observation times are large, the time series is also considered to be sparse.
Such data occur in electronic health records (Marlin et al., 2012; Yadav et al., 2018), climate science (Schulz & Stattegger, 1997), ecology (Clark & Bjørnstad, 2004), biology (Ruf, 1999), and astronomy (Scargle, 1982).
It is well understood that such data cause signiﬁcant issues for standard supervised machine learning models that typically assume fully observed, ﬁxed-size feature representations (Marlin et al., 2012).
A basic approach to dealing with irregular sampling is ﬁxed temporal discretization.
For example,
Marlin et al.
(2012) and Lipton et al.
(2016) discretize continuous-time observations into hour-long bins.
This has the advantage of simplicity, but requires ad-hoc handling of bins with more than one observation and results in missing data when bins are empty.
The alternative to temporal discretization is to construct models with the ability to directly use an irregularly sampled time series as input.
Che et al.
(2018) present several methods based on gated recurrent unit networks (GRUs, Chung et al.
(2014)), including an approach that takes as input a sequence consisting of observed values, missing data indicators, and time intervals since the last observation.
Pham et al.
(2017) proposed to capture time irregularity by modifying the forget gate of an LSTM (Hochreiter & Schmidhuber, 1997), while Neil et al.
(2016) introduced a new time gate that regulates access to the hidden and cell state of the LSTM.
While these approaches allow the network to handle event-based sequences with irregularly spaced vector-valued observations, they do not support learning directly from vectors that are partially observed, which commonly occurs in the multivariate setting because of lack of alignment of observation times across different variables.
Another line of work has looked at using observations from the future as well as from the past for interpolation.
Yoon et al.
(2019) and Yoon et al.
(2018) presented an approach based on the multi-directional RNN (M-RNN) that can leverage observations from the relative past and future of a given time point.
Shukla & Marlin (2019) proposed the interpolation-prediction network framework, consisting of several semi-parametric RBF interpolation layers that interpolate multivariate, sparse, and irregularly sampled input time series against a set of reference time points while taking into account all observed data in a time series.
Horn et al.
(2020) proposed a set function-based approach for classifying time-series with irregularly sampled and unaligned observation.
Chen et al.
(2018) proposed a variational auto-encoder model (Kingma & Welling, 2014; Rezende et al., 2014) for continuous time data based on the use of a neural network decoder combined with a latent ordinary differential equation (ODE) model.
They model time series data via a latent continuous-time function that is deﬁned via a neural network representation of its gradient ﬁeld.
Building on this, Rubanova et al.
(2019) proposed a latent ODE model that uses an ODE-RNN model as the encoder.
ODE-RNNs use neural ODEs to model the hidden state dynamics and an RNN to update the hidden state in the presence of a new observation.
De Brouwer et al.
(2019) proposed GRU-ODE-Bayes, a continuous-time version of the Gated Recurrent Unit (Chung et al., 2014).
Instead of the encoder-decoder architecture where the ODE is decoupled from the input processing, GRU-ODE-Bayes provides a tighter integration by interleaving the ODE and the input processing steps.
Several recent approaches have also used attention mechanisms to model irregularly sampled time series (Song et al., 2018; Tan et al., 2020;
Zhang et al., 2019) as well as medical concepts (Peng et al., 2019;
Cai et al., 2018).
Most of these approaches are similar to Vaswani et al.
(2017) where they replace the positional encoding with an encoding of time and model sequences using self-attention.
However, instead of adding the time encoding to the input representation as in Vaswani et al.
, they concatenate it with the input representation.
These methods use a ﬁxed time encoding similar to the positional encoding of Vaswani et al.
Xu et al.
(2019) learn a functional time representation and concatenate it with the input event embedding to model time-event interactions.
Like Xu et al.
and Kazemi et al.
(2019), our proposed method learns a time representation.
However, instead of concatenating it with the input embedding, our model learns to attend to observations at different time points by computing a similarity weighting using only the time embedding.
Our proposed model uses the time embedding as both the queries and keys in the attention formulation.
It learns an interpolation over the query time points by attending to the observed values at key time points.
Our proposed method is thus similar to kernel-based interpolation, but learning the time attention based similarity kernel gives our model more ﬂexibility compared to methods like that of Shukla & Marlin (2019) that use similarity kernels with ﬁxed functional forms.
Another important difference relative to many of these previous methods is that our proposed approach attends only to the observed data dimensions at each time point and hence does not require a separate imputation step to handle vector valued observations with an arbitrary collection of dimensions missing at any given time point.
T HE M ULTI -T IME A TTENTION M ODULE
In this section, we present the proposed Multi-Time Attention Module (mTAN).
The role of this module is to re-represent a sparse and irregularly sampled time series in a ﬁxed-dimensional space.
This module uses multiple continuous-time embeddings and attention-based interpolation.
We begin by presenting notation followed by the time embedding and attention components.
Notation: In the case of a supervised learning task, we let D = { ( s n , y n )
| n = 1 , ..., N } represent a data set containing N data cases.
An individual data case consists of a single target value y n (discrete for classiﬁcation), as well as a D -dimensional, sparse and irregularly sampled multivariate time series s n .
Different dimensions d of the multivariate time series can have observations at different times, as well as different total numbers of observations L dn .
Thus, we represent time series d for data case n as a tuple s dn =
( t dn , x dn ) where t dn =
[ t 1 dn , ..., t L dn dn ] is the list of time points at which observations are deﬁned and x dn =
[ x 1 dn , ..., x L dn dn ] is the corresponding list of observed values.
In the case of an unsupervised task such as interpolation, each data case consists of a multivariate time series s n only.
We drop the data case index n for brevity when the context is clear.
Time Embedding: Time attention module is based on embedding continuous time points into a vector space.
We generalize the notion of a positional encoding used in transformer-based models to continuous time.
Time attention networks simultaneously leverage H embedding functions φ h ( t ) , each outputting a representation of size d r .
Dimension i of embedding h is deﬁned as follows:
[Equation]
where the ω ih ’s and α ih ’s are learnable parameters.
The periodic terms can capture periodicity in time series data.
In this case, ω ih and α ih represent the frequency and phase of the sine function.
The linear term, on the other hand, can capture non-periodic patterns dependent on the progression of time.
For a given difference ∆ , φ h ( t + ∆) can be represented as a linear function of φ h ( t ) .
Learning the periodic time embedding functions is equivalent to using a one-layer fully connected network with a sine function non-linearity to map the time values into a higher dimensional space.
By contrast, the positional encoding used in transformer models is deﬁned only for discrete positions.
We note that our time embedding functions subsume positional encodings when evaluated at discrete positions.
Multi-Time Attention:
The time embedding component described above takes a continuous time point and embeds it into H different d r -dimensional spaces.
In this section, we describe how we leverage time embeddings to produce a continuous-time embedding module for sparse and irregularly sampled time series.
This multi-time attention embedding module mTAN ( t, s ) takes as input a query time point t and a set of keys and values in the form of a D -dimensional multivariate sparse and irregularly sampled time series s (as deﬁned in the notation section above), and returns a J - [Figure] dimensional embedding at time t .
This process leverages a continuous-time attention mechanism applied to the H time embeddings.
The complete computation is described below.
[Equation]
As shown in Equation 2, dimension j of the mTAN embedding mTAN ( t, s )
[ j ] is given by a linear combination of intermediate univariate continuous-time functions ˆ x hd ( t, s ) .
There is one such function deﬁned for each input data dimension d and each time embedding h .
The parameters U hdj are learnable linear combination weights.
As shown in Equation 3, the structure of the intermediate continuous-time function ˆ x hd ( t, s ) is essentially a kernel smoother applied to the d th dimension of the time series.
However, the interpolation weights κ h ( t, t id ) are deﬁned based on a time attention mechanism that leverages time embeddings, as shown in Equation 4 .
As we can see, the same time embedding function φ h ( t ) is applied for all data dimensions.
The form of the attention mechanism is a softmax function over the observed time points t id for dimension d .
The activation within the softmax is a scaled inner product between the time embedding φ h ( t ) of the query time point t and the time embedding φ h ( t id ) of the observed time point, the key.
The parameters w and v are each d r × d k matrices where d k ≤ d r .
We use a scaling factor 1 √ d k to normalize the dot product to counteract the growth in the dot product magnitude with increase in the dimension d k .
Learning the time embeddings provides our model with ﬂexibility to learn complex temporal kernel functions κ h ( t, t (cid:48) ) .
The use of multiple simultaneous time embeddings φ h ( t ) and a ﬁnal linear combination across time embedding dimensions and data dimensions means that the ﬁnal output representation function mTAN ( t, s ) is extremely ﬂexible.
Different input dimensions can leverage different time embeddings via learned sparsity patterns in the parameter tensor U .
Information from different data dimensions can also be mixed together to create compact reduced dimensional representations.
We note that all of the required computations can be parallelized using masking variables to deal with unobserved dimensions, allowing for efﬁcient implementation on a GPU.
Discretization: Since the mTAN module deﬁnes a continuous function of t given s , it can not be directly incorporated into neural network architectures that expect inputs in the form of ﬁxeddimensional vectors or discrete sequences.
However, the mTAN module can easily be adapted to [Figure] produce such an output representation by materializing its output at a set of reference time points r =
[ r 1 , ..., r K ] .
In some cases, we may have a ﬁxed set of such points.
In other cases, the set of reference time points may need to depend on s itself.
In particular, we deﬁne the auxiliary function ρ ( s ) to return the set of time points at which there is an observation on any dimension of s .
Given a collection of reference time points r , we deﬁne the discretized mTAN module mTAND ( r , s ) as mTAND ( r , s )
[ i ] = mTAN
( r i , s ) .
This module takes as input the set of reference time points r and the time series s and outputs a sequence of mTAN embeddings of length | r | , each of dimension J .
The architecture of the mTAND module is shown in Figure 1.
The mTAND module can be used to interface sparse and irregularly sampled multivariate time series data with any deep neural network layer type including fully-connected, recurrent, and convolutional layers.
In the next section, we describe the construction of a temporal encoder-decoder architecture leveraging the mTAND module, which can be applied to both classiﬁcation and interpolation tasks.
E NCODER -D ECODER F RAMEWORK
As described in the last section, we leverage the discretized mTAN module in an encoder-decoder framework as the primary model in this paper, which we refer to as an mTAN network.
We develop the encoder-decoder framework within the variational autoencoder (VAE) framework in this section.
The architecture for the model framework is shown in Figure 2.
Model Architecture: As we are modeling time series data, we begin by deﬁning a sequence of latent states z i .
Each of these latent states are IID-distributed according to a standard multivariate normal distribution p ( z i ) .
We deﬁne the set of latent states z =
[ z 1 , ..., z K ] at K reference time points.
We deﬁne a three-stage decoder.
First, the latent states are processed through an RNN decoder module to induce temporal dependencies resulting in a ﬁrst set of deterministic latent variables h decRNN =
[ h dec 1 ,RNN , ..., h decK,RNN ] .
Second, the output of the RNN decoder stage and the K time points h dec RNN are provided to the mTAND module along with a set of T query time points t .
The mTAND module outputs a sequence of embeddings h
decTAN =
[ h dec 1 ,TAN , ..., h decT,TAN ] of length | t | .
Third, the mTAN embeddings are independently decoded using a fully connected decoder f dec () and the result is used to parameterize an output distribution.
In this work, we use a diagonal covariance
Gaussian distribution with mean given by the ﬁnal decoded representation and a ﬁxed variance σ 2 .
The ﬁnal generated time series is given by ˆ s =
( t , x ) with all data dimensions observed.
The full generative process is shown below.
We let p θ ( x | z , t ) deﬁne the probability distribution over the values of the time series x given the time points t and the latent variables z .
θ represents the parameters of all components of the decoder.
[Equation]
For an encoder, we simply invert the structure of the generative process.
We begin by mapping the input time series s through the mTAND module along with a collection of K reference time points r .
We apply an RNN encoder to the mTAND model that outputs h encTAN to encode longer-range temporal structure.
Finally, we construct a distribution over latent variables at each reference time point using a diagonal Gaussian distribution with mean and variance output by fully connected layers applied to the RNN outputs h encRNN .
The complete encoder architecture is described below.
We deﬁne q γ ( z | r , s ) to be the distribution over the latent variables induced by the input time series s and the reference time points r .
γ represents all of the parameters in all of the encoder components.
[Equation]
Unsupervised Learning: To learn the parameters of our encoder-decoder model given a data set of sparse and irregularly sampled time series, we follow a slightly modiﬁed VAE training approach and maximize a normalized variational lower bound on the log marginal likelihood based on the evidence lower bound or ELBO.
The learning objective is deﬁned below where p θ ( x jdn | z , t n ) and q γ ( z | r , s n ) are deﬁned in the previous section.
[Equation]
Since irregularly sampled time series can have different numbers of observations across different dimensions as well as across different data cases, it can be helpful to normalize the terms in the standard ELBO objective to avoid the model focusing more on sequences that are longer at the expense of sequences that are shorter.
The objective above normalizes the contribution of each data case by the total number of observations it contains.
The fact that all data dimensions are not observed at all time points is accounted for in Equation 14.
In practice, we use k samples from the variational distribution q γ ( z | r , s n ) to compute the learning objective.
Supervised Learning: We can also augment the encoder-decoder model with a supervised learning component that leverages the latent states as a feature extractor.
We deﬁne this component to be of the form p δ ( y n | z ) where δ are the model parameters.
This leads to an augmented learning objective as shown in Equation 15 where the λ term trades off the supervised and unsupervised terms.
[Equation]
In this work, we focus on classiﬁcation as an illustrative supervised learning problem.
For the classiﬁcation model p δ ( y n | z ) , we use a GRU followed by a 2-layer fully connected network.
We use a small number of samples to approximate the required intractable expectations during both learning and prediction.
Predictions are computed by marginalizing over the latent variable as shown below.
[Equation]
[Equation]
E XPERIMENTS
In this section, we present interpolation and classiﬁcation experiments using a range of models and three real-world data sets (Physionet Challenge 2012, MIMIC-III, and a Human Activity dataset).
Additional illustrative results on synthetic data can be found in Appendix A.2.
Datasets: The PhysioNet Challenge 2012 dataset (Silva et al., 2012) consists of multivariate time series data with 37 variables extracted from intensive care unit (ICU) records.
Each record contains sparse and irregularly spaced measurements from the ﬁrst 48 hours after admission to ICU.
We follow the procedures of Rubanova et al.
and round the observation times to the nearest minute.
This leads to 2880 possible measurement times per time series.
The data set includes 4000 labeled instances and 4000 unlabeled instances.
We use all 8000 instances for interpolation experiments and the 4000 labeled instances for classiﬁcation experiments.
We focus on predicting in-hospital mortality.
8% of examples are in the positive class.
The MIMIC-III data set (Johnson et al., 2016) is a multivariate time series dataset consisting of sparse and irregularly sampled physiological signals collected at Beth Israel Deaconess Medical Center from 2001 to 2012.
Following the procedures of Shukla & Marlin (2019), we extract 53 , 211 records each containing 12 physiological variables.
We focus on predicting in-hospital mortality using the ﬁrst 48 hours of data.
1% of the instances have positive labels.
The human activity dataset consists of 3D positions of the waist, chest and ankles collected from ﬁve individuals performing various activities including walking, sitting, lying, standing, etc.
We follow the data preprocessing steps of Rubanova et al.
and construct a dataset of 6 , 554 sequences with 12 channels and 50 time points.
We focus on classifying each time point in the sequence into one of eleven types of activities.
Experimental Protocols: We conduct interpolation experiments using the 8000 data cases in the
PhysioNet data set.
We randomly divide the data set into a training set containing 80% of the instances, and a test set containing the remaining 20% of instances.
We use 20% of the training data for validation.
In the interpolation task, we condition on a subset of available points and predict values for rest of the time points.
We perform interpolation experiments with a varying percentage of observed points ranging from 50% to 90% of the available points.
At test time, the values of observed points are conditioned on and each model is used to infer the values at rest of the available time points in the test instance.
We repeat each experiment ﬁve times using different random seeds to initialize the model parameters.
We assess performance using mean squared error (MSE).
We use the labeled data in all three data sets to conduct classiﬁcation experiments.
The PhysioNet and MIMIC III problems are whole time series classiﬁcation problems.
Note that for the human activity dataset, we classify each time point in the time series.
We treat this as a smoothing problem and condition on all available observations when producing the classiﬁcation at each time-point (similar to labeling in a CRF).
We use bidirectional RNNs as the RNN-based baselines for the human activity dataset.
We randomly divide each data set into a training set containing 80% of the time series, and a test set containing the remaining 20% of instances.
We use 20% of the training set for validation.
We repeat each experiment ﬁve times using different random seeds to initialize the model parameters.
Due to class imbalance in the Physionet and MIMIC-III data sets, we assess classiﬁcation performance using area under the ROC curve (the AUC score).
For the Human Activity dataset, we evaluate models using accuracy.
For both interpolation and prediction, we select hyper-parameters on the held-out validation set using grid search, and then apply the best trained model to the test set.
The hyper-parameter ranges searched for each model/dataset/task are fully described in Appendix A.5.
Models:
The model we focus on is the encoder-decoder architecture based on the discretized multitime attention module ( mTAND-Full ).
In the classiﬁcation experiments, the hidden state at the last observed point is passed to a two-layer binary classiﬁcation module for all models.
For each data set, the structure of this classiﬁer is the same for all models.
For the proposed model, the sequence of latent states is ﬁrst passed through a GRU and then the ﬁnal hidden state is passed through the same classiﬁcation module.
For the classiﬁcation task only, we consider an ablation of the full model that uses the proposed mTAND encoder, which consists of our mTAND module followed by a GRU to extract a ﬁnal hidden state, which is then passed to the classiﬁcation module ( mTAND-Enc ).
We compare to several deep learning models that expand on recurrent networks to accommodate irregular [Table] sampling.
We also compare to several encoder-decoder approaches.
The full list of model variants is brieﬂy described below.
We use a Gated Recurrent Unit (GRU) (Chung et al., 2014) module as the recurrent network throughout.
Architecture details can be found in Appendix A.4.
• RNN-Impute: Missing observations replaced with weighted average of last observed measurement within that time series and global mean of the variable across training examples (Che et al., 2018).
RNN-
∆ t : Input is concatenated with masking variable and time interval ∆ t indicating how long the particular variable is missing.
• RNN-Decay: RNN with exponential decay on hidden states (Mozer et al., 2017; Che et al., 2018).
• GRU-D: combining hidden state decay with input decay (Che et al., 2018).
• Phased-LSTM:
Captures time irregularity by a time gate that regulates access to the hidden and cell state of the LSTM (Neil et al., 2016) with forward ﬁlling to handle partially observed vectors.
• IP-Nets: Interpolation prediction networks, which use several semi-parametric RBF interpolation layers, followed by a GRU (Shukla & Marlin, 2019).
• SeFT:
Uses a set function based approach where all the observations are modeled individually before pooling them together using an attention based approach (Horn et al., 2020).
• RNN-VAE:
A VAE-based model where the encoder and decoder are standard RNN models.
• ODE-RNN: Uses neural ODEs to model hidden state dynamics and an RNN to update the hidden state in presence of a new observation (Rubanova et al., 2019).
• L-ODE-RNN: Latent ODE where the encoder is an RNN and decoder is a neural ODE (Chen et al., 2018).
• L-ODE-ODE: Latent ODE where the encoder is an ODE-RNN and decoder is a neural ODE (Rubanova et al., 2019).
Physionet Experiments: Table 1 compares the performance of all methods on the interpolation task where we observe 50% − 90% of the values in the test instances.
As we can see, the proposed method (mTAND-Full) consistently and substantially outperforms all of the previous approaches across all of the settings of observed time points.
We note that in this experiment, different columns correspond to different setting (for example, in the case of 70% , we condition on 70% of data and predict the rest of the data; i.e., 30% ) and, hence the results across columns are not comparable. We perform a variant of this task in Appendix A.3, where we compute predictions on the same set of time points across different settings.
Table 2 compares predictive performance on the PhysioNet mortality prediction task.
The full
Multi-Time Attention network model (mTAND-Full) and the classiﬁer based only on the Multi-Time
Attention network encoder (mTAND-Enc) achieve signiﬁcantly improved performance relative to the current state-of-the-art methods (ODE-RNN and L-ODE-ODE) and other baseline methods.
We also report the time per epoch in minutes for all the methods.
We note that the ODE-based models require substantially more run time than other methods due to the required use of an ODE solver (Chen et al., 2018; Rubanova et al., 2019).
These methods also require taking the union of all observation time points in a batch, which further slows down the training process.
As we can see, the proposed full Multi-Time Attention network (mTAND-Full) is over 85 times faster than ODE-RNN and over 100 times faster than L-ODE-ODE, the best-performing ODE-based models.
MIMIC-III Experiments: Table 2 compares the predictive performance of the models on the mortality prediction task on MIMIC-III.
The Multi-Time Attention network-based encoder-decoder [Table] framework (mTAND-Full) achieves better performance than the recent IP-Net and SeFT model as well as all of the RNN baseline models.
While ODE-RNN and L-ODE-ODE both have slightly better mean AUC than mTAND-Full, the differences are not statistically signiﬁcant.
Further, as shown on the PhysioNet classiﬁcation problem, mTAND-Full is more than an order of magnitude faster than the ODE-based methods.
Human Activity Experiments: Table 2 shows that the mTAND-based classiﬁers achieve signiﬁcantly better performance than the baseline models on this prediction task, followed by ODE-based models and IP-Nets.
Additional Experiments: In Appendix A.2, we demonstrate the effectiveness of learning temporally distributed latent representations with mTANs on a synthetic dataset.
We show that mTANs are able to capture local structure in the time series better than latent ODE-based methods that encode to a single time point.
This property of mTANs helps to improve the interpolation performance in terms of mean squared error.
We also perform ablation experiments to show the performance gain achieved by learning similarity kernels and time embeddings in Appendix A.1.
In particular, we show that learning the time embedding improves classiﬁcation performance compared to using ﬁxed positional encodings.
We also demonstrate the effectiveness of learning the similarity kernel by comparing to an approach that uses ﬁxed RBF kernels.
Appendix A.1 shows that learning the similarity kernel using the mTAND module performs as well as or better than using a ﬁxed RBF kernel.
D ISCUSSION AND C ONCLUSIONS
In this paper, we have presented the Multi-Time Attention (mTAN) module for learning from sparse and irregularly sampled data along with a VAE-based encoder-decoder model leveraging this module.
Our results show that the resulting model performs as well or better than a range of baseline and state-of-the-art models on both the interpolation and classiﬁcation tasks, while offering training times that are one to two orders of magnitude faster than previous state of the art methods.
While in this work we have focused on a VAE-based encoder-decoder architecture, the proposed mTAN module can be used to provide an interface between sparse and irregularly sampled time series and many different types of deep neural network architectures including GAN-based models.
Composing the mTAN module with convolutional networks instead of recurrent architectures may also provide further computational enhancements due to improved parallelism. A CKNOWLEDGEMENTS
Research reported in this paper was partially supported by the National Institutes of Health under award numbers 5U01CA229445 and 1P41EB028242.
A A PPENDIX A. A BLATION S TUDY
In this section, we perform ablation experiments to show the performance gain achieved by learning similarity kernel and time embedding.
Table 3 shows the ablation results by substituting ﬁxed positional encoding (Vaswani et al., 2017) in place of learnable time embedding deﬁned in Equation 1 in mTAND-Full model on PhysioNet and MIMIC-III dataset for classiﬁcation task.
We report the average AUC score over 5 runs.
As we can see from Table 3, learning the time embedding improves
AUC score by 1% as compared to using ﬁxed positional encodings.
[Table]
Since mTANs are fundamentally continuous-time interpolation-based models [Table] we perform an ablation study by comparing mTANs with the IP-nets (Shukla & Marlin, 2019) [Table]
IP-Nets use several semiparametric RBF interpolation layers [Table] followed by a GRU to model irregularly sampled time series.
In this framework, we replace the RBK kernel with a learnable similarity kernel using mTAND module, the corresponding model is mTAND-Enc.
Table 4 compares the performance of the two methods on classiﬁcation task on PhysioNet, MIMIC-III and Human Activity dataset.
We report the average AUC score over 5 runs.
Table 4 shows that learning the similarity kernel using mTAND module performs as well or better than using a ﬁxed RBF kernel.
[Table]
A. S YNTHETIC I NTERPOLATION E XPERIMENTS
To demonstrate the capabilities of our model on the interpolation task, we generate a synthetic dataset consisting of 1000 trajectories each of 100 time points sampled over t ∈ [0 , 1] .
We ﬁx 10 reference points and use RBF kernel with a ﬁxed bandwidth of 100 for constructing local interpolations at 100 time points over [0 , 1] .
The values at the reference points are drawn from a standard normal distribution.
We randomly sample 20 observations from each trajectory to simulate a sparse and irregularly sampled multivariate time series.
We use 80% of the data for training and 20% for testing.
At test time, encoder conditions on 20 irregularly sampled time points and the decoder generates interpolations on all 100 time points.
Figure 3 illustrates the interpolation results on the test set for the Multi-Time
Attention Network and Latent ODE model with ODE encoder (Rubanova et al., 2019).
For both the models, we draw 100 samples from the approximate posterior distribution.
As we can see from Figure 3, the ODE interpolations are much smoother and haven’t been able to capture the local structure as well as mTANS.
[Table]
Table 5 compares the proposed model with best performing baseline Latent-ODE with ODE encoder (L-ODE-ODE) on reconstruction and interpolation task.
For both the tasks, we condition on theirregularly sampled time points and reconstruct the input points (reconstruction) and the whole set of 100 time points (interpolation).
We report the mean squared error on test set.
A. I NTERPOLATION ON P HYSIO N ET
This is a slight variation of the interpolation task shown in Table 1.
Here, we condition on a subset of available points and reconstruct/predict values for all points.
We perform interpolation experiments with a varying percentage of observed points ranging from 50% to 90% of the available points.
At test time, the values of observed points are conditioned on and each model is used to infer the values at all available time points in the test instance.
We repeat each experiment ﬁve times using different random seeds to initialize the model parameters.
We assess performance using mean squared error (MSE).
Table 6 compares the performance of all methods on the interpolation task where we observe 50% − 90% of the values in the test instances.
As we can see, the proposed method (mTAND-Full) consistently and substantially outperforms all of the previous approaches.
Compared to Table 1, here the results are comparable across columns since in each we report the points
[Table]
A. A RCHITECTURE D ETAILS
Multi-Time Attention Network (mTAND-Full) :
In our proposed encoder-decoder framework (Figure 2), we use bi-directional GRU as the recurrent model in both encoder and decoder.
In encoder, we use a 2 layer fully connected network with 50 hidden units and ReLU activations to map the RNN hidden state at each reference point to mean and variance.
Similarly in decoder, mTAN embeddings are independently decoded using a 2 layer fully connected network with 50 hidden units and ReLU activations, and the result is used to parameterize the output distribution.
For classiﬁcation tasks, we use a separate GRU layer on top of the latent states followed by a 2-layer fully connected layer with 300 units and ReLU activations to output the class probabilities.
Multi-Time Attention Encoder (mTAND-Enc) :
As we show in the experiments, the proposed mTAN module can standalone be used for classiﬁcation tasks.
The mTAND-Enc consists of MultiTime attention module followed by GRU to extract the ﬁnal hidden state which is then passed to a 2-layer fully connected layer to output the class probabilities.
Loss Function:
For computing the evidence lower bound (ELBO) during training, we use negative log-likelihood with ﬁxed variance as the reconstruction loss.
For all the datasets, we use a ﬁxed variance of 0 . 01 .
For computing ELBO, we use 5 samples for interpolation task and 1 sample for classiﬁcation tasks.
We use cross entropy loss for classiﬁcation.
For the classiﬁcation tasks, we tune the λ parameter in the supervised learning loss function (Equation 15).
We achieved best performance using λ as 100 and 5 for Physionet, MIMIC-III respectively.
For human activity dataset, we achieved best results without using the regulaizer or ELBO component.
We found that KL annealing with coeff 0 .
99 improved the performance of interpolation and classiﬁcation tasks on Physionet.
A. H YPERPARAMETERS
Baselines: For Physionet and Human Activity dataset, we use the reported hyperparameters for
RNN baselines as well as ODE models from Rubanova et al.
For MIMIC-III dataset, we independently tune the hyperparameters of the baseline models on the validation set.
We search for
GRU hidden units, latent dimension, number of hidden units in fully connected network for ODE function in recognition and generative model over the range { 20 , 32 , 64 , 128 , 256 } .
For ODEs, we also searched the number of layers in fully connected network in the range { 1 , 2 , 3 } .
mTAN: We learn time embeddings of size 128 .
The number of embeddings H ∈ { 1 , 2 , 4 } .
The linear projection matrices used for projecting time embedding W are each d k ∗ d k /h where d k is the embedding size.
We search the latent dimension and GRU encoder hidden size over the range { 32, 64, 128 } .
We keep GRU decoder hidden size at 50 .
For the classiﬁcation tasks, we use 128 reference points.
For interpolation task, we search number of reference points over the range { 8 , 16 , 32 , 64 , 128 } .
We use Adam Optimizer for training the models.
For classiﬁcation, experiments are run for 300 iteration with learning rate 0 . 0001 , while for interpolation task experiments are run for 500 iterations with learning rate 0 .
Best hyperparameters are reported in the code.
A. V ISUALIZING A TTENTION W EIGHTS
In this section, we visualize the attention weights learned by our proposed model.
We experiment using synthetic dataset (described in A.2) which consists of univariate time series.
Figure 4 shows the attention weights learned by the encoder mTAND module.
The input shown in the ﬁgure is the irregularly sampled time points and the edges show how the output at reference points attends to [Figure] the values on the input time points.
The ﬁnal output can be computed by substituting the attention weights in Equation 3.
A. T RAINING D ETAILS A.7. D ATA GENERATION AND PREPROCESSING
All the datasets used in the experiments are publicly available and can be downloaded using the following links:
PhysioNet: https://physionet.org/content/challenge-2012/
MIMIC-III: https://mimic.physionet.org/
Human Activity: https://archive.ics.uci.edu/ml/datasets/Localization+
Data+for+Person+Activity .
We rescale each feature to be between 0 and 1 for Physionet and MIMIC-III dataset.
We also rescale the time to be in [0 , 1] for all datasets.
In case of MIMIC-III dataset, for the time series missing entirely, we follow the preprocessing steps of Shukla & Marlin (2019) and assign the starting point (time t=0) value of the time series to the global mean for that variable.
A.7. S OURCE C ODE
The code for reproducing the results in this paper is available at https://github.com/ reml-lab/mTAN .
A.7. C OMPUTING I NFRASTRUCTURE
All experiments were run on a Nvidia Titan X GPU.