S ET F UNCTIONS FOR T IME S ERIES
Anonymous authors
Paper under double-blind review
A BSTRACT
Despite the eminent successes of deep neural networks, many architectures are often hard to transfer to irregularly-sampled and asynchronous time series that occur in many real-world datasets, such as healthcare applications.
This paper proposes a novel framework for classifying irregularly sampled time series with unaligned measurements, focusing on high scalability and data efﬁciency.
Our method S E FT ( Se t F unctions for T ime Series) is based on recent advances in differentiable set function learning, extremely parallelizable, and scales well to very large datasets and online monitoring scenarios.
We extensively compare our method to competitors on multiple healthcare time series datasets and show that it performs competitively whilst signiﬁcantly reducing runtime.
I NTRODUCTION
With the increasing digitalization, measurements over extensive time periods are becoming ubiquitous.
Nevertheless, in many application domains, in particular healthcare (Yadav et al., 2018), measurements might not necessarily be observed at a regular rate or could be misaligned.
Moreover, the presence or absence of a measurement and its observation frequency may carry information of its own (Little & Rubin, 2014), such that imputing the missing values is not always desired.
While some algorithms can be readily applied to datasets with varying length, these methods usually assume regular sampling of the data and/or require the measurements across modalities to be aligned/synchronized, preventing their application to the aforementioned settings.
Existing approaches for unaligned measurements, by contrast, typically rely on imputation to obtain a regularlysampled version of a dataset for classiﬁcation.
Learning a suitable imputation scheme, however, requires understanding the underlying dynamics of a system; this task is signiﬁcantly more complicated and not necessarily required when classiﬁcation is the main goal.
Furthermore, even though a decoupled imputation scheme followed by classiﬁcation is generally more scalable, it may lose information (in terms of “missingness patterns”) that could be crucial for prediction tasks.
In addition, the fact that decoupled schemes perform worse than methods that are trained end-to-end has been has been empirically demonstrated by Li & Marlin (2016).
Approaches that jointly optimize both tasks also add a large computational overhead, thus suffering from poor scalability or high memory requirements.
Our method is motivated by the understanding that, while RNNs and similar architectures are well suited for capturing and modelling the dynamics of a time series and thus excel at tasks such as forecasting, retaining the order of an input sequence can even be a disadvantage in classiﬁcation scenarios (Vinyals et al., 2015).
We show that by relaxing the condition that a sequence must be processed in order, we can naturally derive an architecture that directly accounts for (i) irregular sampling, and (ii) unsynchronized measurements.
Our method S E FT:
Se t F unctions for T ime Series, extends recent advances in set function learning to irregular sampled time series classiﬁcation tasks, yields state-of-the-art performance, is highly scalable and improves over current approaches by almost an order of magnitude in terms of runtime.
With S E FT, we propose to rephrase the problem of classifying time series as classifying a set of observations.
We show how set functions can be exploited to learn classiﬁers that are naturally applicable to unaligned and irregularly sampled time series, leading to state-of-the-art performance in irregularly-sampled time series classiﬁcation tasks.
Our approach can be interpreted as learning dataset-speciﬁc summary statistics of time series which are optimized to separate instances by class.
Furthermore, our method is highly parallelizable and can be readily extended to an online monitoring setup with up to thousands of patients.
R ELATED W ORK
This paper focuses on classifying time series with irregular sampling and potentially unaligned measurements.
We brieﬂy discuss recent work in this ﬁeld; all approaches can be broadly grouped into the following three categories.
Irregular sampling as missing data While the problem of supervised classiﬁcation in the presence of missing data is closely related to irregular sampling on time series, there are some core differences.
Missing data is usually deﬁned with respect to a number of features that could be observed, whereas time series themselves can have different lengths and a “typical” number of observed values might not exist.
Generally, an irregularly-sampled time series can be converted into a missing data problem by discretizing the time axis into non-overlapping intervals, and declaring intervals in which no data was sampled as missing.
This approach is followed by Marlin et al.
(2012), where a Gaussian Mixture Model was used to do semi-supervised clustering on electronic health records.
Similarly, Lipton et al.
(2016) discretize the time series into intervals, aggregate multiple measurements within an interval, and add missingness indicators to the input of a Recurrent Neural
Network.
By contrast, Che et al.
(2018) present several variants of the Gated Recurrent Unit (GRU) combined with imputation schemes.
Most prominently, the GRU-model was extended to include a decay term (GRU-D), such that the last observed value is decayed to the empirical mean of the time series via a learnable decay term.
While these approaches are applicable to irregularly-sampled data, they either rely on imputation schemes or empirical global estimates on the data distribution (our method, by contrast, requires neither), without directly exploiting the global structure of the time series.
Frameworks supporting irregular sampling Some frameworks support missing data.
For example, Lu et al.
(2008) directly deﬁned a kernel on irregularly-sampled time series, permitting subsequent classiﬁcation and regression with kernel-based classiﬁers or regression schemes.
Furthermore, Gaussian Processes (Williams & Rasmussen, 2006) constitute a common probabilistic model for time series; they directly permit modelling of continuous time data using mean and covariance functions.
Along these lines, Li & Marlin (2015) derived a kernel on Gaussian Process Posteriors, allowing the comparison and classiﬁcation of irregularly-sampled time series using kernel-based classiﬁers.
Nevertheless, all of these approaches still rely on separate tuning/training of the imputation method and the classiﬁer so that structures supporting the classiﬁcation could be potentially missed in the imputation step.
An emerging line of research employs Hawkes processes (Hawkes, 1971; Liniger, 2009), i.e. a speciﬁc class of self-exciting point processes, for time series modelling and forecasting (Mei & Eisner, 2017;
Yang et al., 2017; Xiao et al., 2017).
While Hawkes processes exhibit extraordinary performance in these domains, there is no standardised way of using them for classiﬁcation.
Previous work (Lukasik et al., 2016) trains multiple Hawkes processes (one for each label) and classiﬁes a time series by assigning it the label that maximises the respective likelihood function.
Since this approach does not scale to our datasets, we were unable to perform a fair comparison.
We conjecture that further research will be required to make Hawkes processes applicable to general time series classiﬁcation scenarios.
End-to-end learning of imputation schemes Methods of this type are composed of two modules with separate responsibilities, namely an imputation scheme and a classiﬁer, where both components are trained discriminatively and end-to-end using gradient-based training.
Recently, Li & Marlin (2016) proposed the Gaussian Process Adapters (GP Adapters) framework, where the parameters of a Gaussian Process Kernel are trained alongside a classiﬁer.
The Gaussian Process gives rise to a ﬁxed-size representation of the irregularly-sampled time series, making it possible to apply any differentiable classiﬁcation architecture.
This approach was further extended to multivariate time series by Futoma et al.
(2017) using Multi-task Gaussian Processes (MGPs) (Bonilla et al., 2008), which allow correlations between the imputed channels.
Moreover, Futoma et al.
(2017) made the approach more compatible with time series of different lengths by applying a Long Short Term
Memory (LSTM) (Hochreiter & Schmidhuber, 1997) classiﬁer.
Motivated by the limited scalability of approaches based on GP Adapters, Shukla & Marlin (2019) suggest an alternative imputation mmtttm
Multivariate Time Series   ( t 3 ,z 3 ,m 1 ) , ( t 5 ,z 5 ,m 1 ) ,...
( t 1 ,z 1 ,m 2 ) , ( t 4 ,z 4 ,m 2 ) ,...
( t 2 ,z 2 ,m 3 ) ,... ( t 11 ,z 11 ,m 3 )  
Set Encoding f 0 f 0 fagg attn
Embedding, Aggregation, and Attention
Classiﬁcation scheme, the interpolation prediction networks .
It applies multiple semi-parametric interpolation schemes to obtain a regularly-sampled time series representation.
The parameters of the interpolation network are trained with the classiﬁer in an end-to-end setup.
P ROPOSED M ETHOD
Our paper focuses on the problem of time series classiﬁcation of irregularly sampled and unaligned time series.
We ﬁrst deﬁne the required terms before describing our models
3. N OTATION & REQUIREMENTS
Deﬁnition 1 (Time series) .
We describe a time series of an instance i as a set S i of M :
= len( S i ) observations s j such that S i :
= { s 1 , . . .
, s M } .
We assume each observation s j to be represented as a tuple ( t j , z j , m j ) , consisting of a time t j ∈ R + , an observed value z j ∈ R , and a modality indicator m j ∈ { 1 . . .
D } , where D represents the dimensionality of the time series.
We write Ω ⊆ R
+ × R × N +
to denote the domain of observations.
An entire time series can thus be represented as
[Equation]
where for notational convenience we omitted the index i .
We leave this deﬁnition very general on purpose, allowing the length of each time series (comprising all channels, such as “heart rate”, “respiratory rate”, etc. of one instance) to differ, since our models are capable of handling this.
Likewise, we neither enforce nor expect all time series to be synchronized, i.e. being sampled at the same time, but rather we permit unaligned or non-synchronized observations in the sense of not having to observe all modalities at each time point.
Time series are collected in a dataset D .
Deﬁnition 2 (Dataset) .
We consider a dataset D to contain n time series.
Elements of D are tuples, i.e. D :=
{ ( S 1 , y 1 ) , . . .
, ( S N , y N ) } , where S i denotes the i th time series and y i ∈ { 1 , . . .
, C } its associated class label.
Figure 1 gives a high-level overview of our method, including the individual steps required to perform classiﬁcation.
To get a more intuitive grasp of these deﬁnitions, we brieﬂy illustrate our time series notation with an example.
Let instance i be an in-hospital patient, while the time series represent measurements of two channels of vital parameters during a hospital stay, namely heart rate (HR) and mean arterial blood pressure (MAP).
We enumerate those channels as modalities and 2.
Counting from admission time, a HR of 60 and 65 beats per minute was measured after 0.5 h and 3.0 h, respectively, whereas MAP values of 80, 85, and 87 mmHg were observed after 0.5 h, 1.7 h, and 2.5 h.
According to Deﬁnition 1, the time series is thus represented as
S i
= { (0 . 5 , 60 , 1) , (3 , 65 , 1) , (0 . 5 , 80 , 2) , (1 . 7 , 85 , 2) , (3 , 87 , 2) } .
In this example, observations are ordered by modality to increase readability; in practice, we are dealing with unordered sets.
Deﬁnition 3 (Non-synchronized time series) .
We call a D -dimensional time series nonsynchronized if there is at least one time point t j ∈ R + at which at least one modality is not observed, i.e. if there exists t j ∈ R + such that |{ ( t k , z k , m k ) | t k = t j }| (cid:54) =
D .
Furthermore, we assume that no two measurements of the same modality m k occur at the same time, i.e. t i (cid:54) =
t j for i (cid:54) =
j has to be satisﬁed for all measurements in m k .
This assumption is not required for technical reasons but for consistency .
It also makes it possible to interpret the results later on.
To summarize our generic setup, we do not require M , the number of observations per time series, to be the same, i.e. len( S i ) (cid:54) =
len( S j ) for i (cid:54) =
j is permitted, nor do we assume that the time points and modalities of the observations are the same across time series.
This setting is common in biomedical time series, for example.
Since typical machine learning algorithms are designed to operate on data of a ﬁxed dimension, novel approaches to this non-trivial problem are required.
3. O UR MODEL
In the following, we describe an approach inspired by differentiable learning of functions that operate on sets (Zaheer et al., 2017; Wagstaff et al., 2019).
We phrase the problem of classifying time series on irregular grids as learning a function f on a set of arbitrarily many time series observations following Deﬁnition 1, i.e. S =
( t 1 , z 1 , m 1 ) , . . .
, ( t M , z M , m M ) } , such that f : S → R C , where
S represents a generic time series of arbitrary cardinality and R C corresponds to the logits of the C classes in the dataset.
As we previously discussed, we interpret each time series as an unordered set of measurements, where all information is conserved because the observation time is included for each set element.
Speciﬁcally, we deﬁne f to be a set function, i.e. a function that operates on a set and thus has to be invariant to the ordering of the elements in the set.
Multiple architectures are applicable to constructing set functions such as Transformers (Lee et al., 2019; Vaswani et al., 2017), or Deep Sets (Zaheer et al., 2017).
Due to preliminary experiments, where Transformers suffered from lower generalization performance in our setting 1 , we base this work on the framework of Zaheer et al.
Intuitively, this can be seen as computing multivariate dataset-speciﬁc summary statistics, which are optimized to maximize classiﬁcation performance.
Thus, we sum-decompose the set function f into the form
[Equation]
where h : Ω → R d and g : R d → R C are neural networks, d ∈ N + determines the dimensionality of the latent representation, and s j represents a single observation of the time series S .
We can view the averaged representations 1 / |S| (cid:80) s j ∈S h ( s j ) in general as a dataset-speciﬁc summary statistic learned to best distinguish the class labels.
Equation 2 also implies the beneﬁcial scalability properties of our approach: each embedding can be calculated independently of the others; hence, the constant computational cost of passing a single observation through the function h is scaled by the number of observations, resulting in a runtime of O ( M ) for a time series of length M .
Recently, Wagstaff et al.
(2019) derived requirements for a practical universal function representation of sum-decomposable set functions, i.e the requirements necessary for a sum-decomposable function to represent an arbitrary set-function given that h and g are arbitrarily expressive.
In particular, they show that a universal function representation can only be guaranteed provided that d ≥ max i len( S i ) is satisﬁed.
During hyperparameter search we thus independently sample the dimensionality of the aggregation space, and allow it to be in the order of the number of observations that are to be expected in the dataset.
Further, we explored the utilization of max, sum, and mean as alternative aggregation functions inspired by Zaheer et al.
; Garnelo et al.
Intuition Our method can be connected to Takens’s embedding theorem (Takens, 1981) for dynamical systems: we also observe a set of samples from some unknown (but deterministic) dynamical process; provided the dimensionality of our architecture is sufﬁciently large 2 , we are capable of reconstructing the system up to diffeomorphism.
The crucial difference is that we do not have to construct a time-delay embedding but rather, we let the network learn an embedding that is suitable for classiﬁcation.
Time encoding In order to represent the time point of an observation on a normalized scale, we employ variant of positional encodings , as introduced by Vaswani et al.
Preliminary results indicated that this encoding scheme reduces the sensitivity towards initialization and training hyperparameters of a model.
Speciﬁcally, the time encoding converts the one-dimensional time axis into a multi-dimensional input by passing the time t of each observation through multiple sine and cosine functions of varying frequencies.
Given a dimensionality τ ∈ N + of the time encoding, we refer to the encoded position as x ∈ R τ , where
[Equation]
with k ∈ { 0 , . . .
, τ / 2 } and max ts representing the maximal time scale that is expected in the data.
Intuitively, we select the wavelengths using a geometric progression from 2 π to max ts · 2 π , and treat the number of steps and the maximum timescale max ts as hyperparameters of the model.
For all experiments time encodings were used, such that an observation is represented as s j = ( x ( t j ) , z j , m j ) .
Loss function If not mentioned otherwise, we choose h and g in Equation 2 to be multilayer perceptron deep neural networks, parametrized by weights θ and ψ , respectively.
We thus denote these neural networks by h θ and g ψ ; their parameters are shared across all instances per dataset.
In our training setup, we follow Zaheer et al.
and apply the devised set function to the complete time series, i.e. to the set of all observations for each time series.
Overall, we optimize a loss function that is deﬁned as
[Equation]
where (cid:96) ( · ) represents a task-speciﬁc loss function.
In out setup, we either utilize the binary crossentropy in combination with a sigmoid activation function in the last layer for binary classiﬁcation or multi-label classiﬁcation tasks and categorical cross-entropy in combination with a softmax activation function in the last layer for multi-class classiﬁcation tasks.
3. A TTENTION - BASED AGGREGATION
So far, our method permits encoding sets of arbitrary sizes into a ﬁxed-size representation.
For increasingly large set sizes, however, many irrelevant observations could inﬂuence the result of the set function.
The mean aggregation function is particularly susceptible to this because the inﬂuence of an observation to the embedding shrinks proportionally to the size of the set.
We thus suggest to use a weighted mean in order to allow the model to decide which observations are relevant and which should be considered irrelevant.
This is equivalent to computing an attention a ( S , s j ) over the set input elements, and subsequently, computing the sum over all elements in the set.
Our approach is based on scaled dot-product attention with multiple heads i ∈ { 1 , . . .
, m } in order to be able to cover different aspects of the aggregated set 3 .
We deﬁne a ( · ) , i.e. the attention weight function of an individual time series, to depend on the overall set of observations.
This is achieved by computing an embedding of the set elements using a smaller set function f (cid:48) , and projecting the concatenation of the set representation and the individual set elements into a d -dimensional space.
Speciﬁcally, we have K j,i =
[ f (cid:48) ( S ) , s j ] T W i where W i ∈
R
(im( f (cid:48) )
+ | s j | ) ×
d and K ∈ R |S|× d .
Furthermore, we deﬁne a matrix of query points Q ∈ R m × d , which allow the model to summarize different aspects of the dataset via
[Equation]
where a j,i represents the amount of attention that head i gives to set element j .
The head-speciﬁc row Q i of the query matrix Q allows a head to focus on individual aspects (such as the distribution of one or multiple modalities) of a time series.
For each head, we multiply the set element embeddings computed via the set function f with the attentions derived for the individual instances, i.e. r i = (cid:80) j a j,i f ( s j ) .
The computed representation is concatenated and passed to the aggregation network h θ as in a regular set function, i.e. r ∗ =
[ r 1 . . .
r m ] .
In our setup, we initialize Q with zeros, such that at the beginning of training, the attention mechanism is equivalent to computing the unweighted mean over the set elements.
Overall, this aggregation function is similar to Transformers (Vaswani et al., 2017), but differs from them in a few key aspects.
Standard Transformer blocks would use the information from all set elements in order to compute the embedding of an individual set element, leading to a runtime and space complexity of O ( n 2 ) .
In contrast, our approach computes the embeddings of set elements independently, leading lower runtime and memory complexity of O ( n ) .
Further, we observed that computing embeddings with information from other set elements (as the Transformer does) actually decreases generalization performance (see Table 1 for details).
E XPERIMENTS
We executed all experiments and implementations in a uniﬁed code base, which we also make available 4 to the community.
While some of the datasets used subsequently have access restrictions, anybody can gain access after satisfying the deﬁned requirements.
This ensures the reproducibility of our results.
Please consult Appendix A.2 for further details.
4. D ATASETS
In order to benchmark the proposed method we selected 4 datasets with irregularly-sampled and non-synchronized measurements.
Healing MNIST
The H-MNIST dataset was introduced by Krishnan et al.
(2015) in order to simulate characteristics which typically occur in medical time series.
In our setup, we use a variant of this dataset.
Every instance of the dataset contains 10 frames, derived from a single instance of
MNIST dataset, where the digit is rotated according to an angle uniformly sampled between −90 ◦ to 90 ◦ .
Furthermore, 3 randomly-selected consecutive frames are augmented by a square artefact in the top left corner of the image in order to indicate seasonality in the time series.
Finally, 60 % of the data points are randomly discarded in order to yield a ﬁnal high-dimensional irregularly-sampled time series with non-synchronized measurements.
Using these settings each instance has on average 3 , 136 observations.
MIMIC-III Tasks MIMIC-III (Johnson et al., 2016) is a widely-used, freely-accessible dataset containing around 50 , 000 distinct ICU stays.
The median length of stay is 2.1 d and a wide range of physiological measurements (e.g. arterial blood pressure, respiration rate, heart rate) are recorded with a resolution of 1 h. Furthermore, laboratory test results, collected at irregular time intervals are available.
Recently, Harutyunyan et al.
(2019) deﬁned a set of machine learning tasks, labels, and benchmarks using a subset of the MIMIC-III dataset.
We trained and evaluated our method and competing methods on the binary mortality prediction task ( M3-Mortality ) and on the multiclass problem of phenotype classiﬁcation ( M3-Phenotyping ), while applying additional ﬁltering described in Appendix A.1.
The goal of the mortality prediction task is to predict whether a patient will die during his/her hospital stay using only data from the ﬁrst 48 hours of the ICU stay.
This dataset contains around 21 , 000 stays of which approximately 10 % result in death.
The phenotype classiﬁcation task consists of 40 , 000 patients, each of which can suffer from a multitude of 25 acute care conditions.
Physionet Mortality Prediction Challenge The 2012 Physionet challenge dataset (Goldberger et al., 2000), which we abbreviate P-Mortality , contains 12 ,
ICU stays each of which lasts at least 48 h. For each stay, a set of general descriptors (such as gender, age, height, weight) were collected at admission time.
Depending on the course of the stay and patient status, up to 37 time series variables were measured (e.g. blood pressure, lactate, respiration rate, temperature).
While some modalities might be measured in regular time intervals (e.g. hourly or daily), some are only collected when required.
Not all variables are available for each stay.
The goal of the challenge was to predict if—and with which certainty —a patient will die during the hospital stay.
The training set consists of 8 , 000 stays while the testing set comprises 4 , 000 ICU visits.
Both datasets are similarly imbalanced, with a prevalence of around 14 %.
For simplicity, the general descriptors (such as age and weight), were included as time points with a single observation at the beginning of the stay.
This treatment is similar to the approach by Harutyunyan et al.
in the MIMIC-III benchmarking datasets.
Please refer to Table A.1, Table A.2, and Table A.3 in the appendix for a more detailed enumeration of samples sizes and label distributions.
The total number of samples may slightly deviate from the originally published splits, as time series of excessive length prevented ﬁtting some methods in reasonable time, and were therefore excluded.
4. C OMPETITOR METHODS
GRU-simple GRU- SIMPLE
Che et al., 2018) augments the input at time t of a Gated-RecurrentUnit RNN with a measurement mask m dt and a δ t matrix, which contains the time since the last measurement of the corresponding modality d , such that
[Equation]
  s t − s t − 1 + δ dt −t > 1 , m dt − 1 =
s t − s t −t > 1 , m dt − 1 =
t =where s t represents the time associated with time step t .
Phased-LSTM The P HASED -LSTM
(Neil et al., 2016) introduced a biologically inspired time dependent gating mechanism which regulates access to the hidden and cell state of a Long short-term
RNN cell (Hochreiter & Schmidhuber, 1997).
While this allows the network to handle event-based sequences with irregularly spaced observations, the approach does not support unaligned measurements.
In order to still provide the architecture with all relevant information, we augment the input in a similar fashion as described for the GRU- SIMPLE approach.
GRU-D GRU-D or GRU-Decay (Che et al., 2018) contains modiﬁcations to the GRU RNN cell, allowing it to decay past observations to the mean imputation of a modality using a learnable decay rate.
By additionally providing the measurement masks as an input the recurrent neural network the last feed in value.
Learns how fast to decay back to a mean imputation of the missing data modality.
Interpolation Prediction Networks IP-N ETWORKS (Shukla & Marlin, 2019) apply multiple semiparametric interpolation schemes to irregularly-sampled time series to obtain regularly-sampled representations that cover long-term trends, transients, and also sampling information.
The method combines a univariate interpolation step with a subsequent multivariate interpolation; the parameters of the interpolation network are trained with the classiﬁer in an end-to-end fashion.
Transformer
In the T RANSFORMER architecture (Vaswani et al., 2017) the elements of a sequence are encoded simultaneously and information between sequence elements is captured using MultiHead-Attention blocks.
In our case, an individual sequence element corresponds to all measurements available at a given time point, augmented with a measurement indicator.
Transformers are normally used for sequence-to-sequence modelling tasks and in our setup were adapted to classiﬁcation tasks by mean-aggregating the ﬁnal representation.
This representation is then fed into a one-layer MLP to predict logits for the individual classes. [Figure]
4. E XPERIMENTAL SETUP
To permit a fair comparison between the methods, we executed hyperparameter searches for each model on each dataset, composed of uniformly sampling 20 parameters according to Appendix A.3.
Training was stopped after 20 epochs without improvement of the validation loss, the hyperparameters with the best overall validation performance were selected for quantifying the performance on the test set.
The train, validation, and test splits were the same for all models and all evaluations.
Final performance on the test set was calculated by 3 independent runs of the models; evaluation took place after the model was restored to the state with the best validation loss.
In all subsequent benchmarks, we use the standard deviation of the test performance of these runs as generalization performance estimates.
4. R ESULTS
The results are shown in Table 1.
Overall, our proposed method exhibits the lowest per-epoch runtime on most datasets, while either yielding competitive or state-the-art performance.
Further, the trade-off between runtime and performance of the proposed method is very good on all datasets (see
Figure A.1 and Figure A.2 in the appendix for a visualization of this argument).
In order to elucidate the contribution of individual model components, we also provide an ablation study in Table A.4.
Here we see that the attention mechanism contributes more to the model performance, while the positional encoding seems to be beneﬁcial for datasets with highly-varying time series lengths, in particular M3-Phenotyping .
Opening the black box In the medical domain, it is of particular interest to understand the decisions a model makes based on the input it is provided with.
The formulation of our model and its per observation perspective on time series gives it the unique property of being able to quantify to which extent an individual observation contributed to the output of the model.
We exemplify this in
Figure 2 with a patient time series that was combined with our models attention values, displayed for a set of clinically relevant variables.
After reviewing these records with our medical expert, we ﬁnd that our model is able to pick up regions with drastic changes in individual modalities.
Moreover, it is able to inspect other modalities at the same associated time (for instance, at hour 20).
This is behaviour similar to what one would expect from an alerted clinician reviewing the logged medical records.
Interestingly, we observe that the model attends to known trends (that are consisting with domain knowledge about patient deterioration ultimately resulting in death) such as increase in lactate or hemodynamic instability, as indicated by drops in blood pressure.
Furthermore, the model appears to be alerted by persisting low urine output.
After several hours, this can be indicative of kidney failure.
[Table]
C ONCLUSION
In this work, we presented a novel approach for classifying time series with irregularly-sampled and unaligned, that is non-synchronized , observations.
Our approach yields state-of-the-art to strongly competitive performance on numerous simulated and real-world datasets, while reducing runtime by almost half.
Moreover, we demonstrated that combining the perspective of individual observations with an attention mechanism permits increasing the interpretability of the model.
This is particularly relevant for the medical and healthcare applications.
For future work, we reserve a more extensive exploration of the learned latent representation to evaluate its utility for clustering of time series or visualization of their similarity.
A A PPENDIX
[Table] [Table]
A. D ATA FILTERING
Due to memory requirements of some of the competitor methods, it was necassary to excluded time series with extremly high number of measurements.
For the M3-Phenotyping patients with more than 2000 distinct time points were discarded from training.
For M3-Mortality patients with more than 1000 time points were discarded as they contained dramatically different measuring frequencies compared to the rest of the dataset.
A. I MPLEMENTATIONAL DETAILS
All experiments were run using tensorflow 0.15.0rc0 and training was performed on
NVIDIA Geforce GTX 1080 GPUs.
In order to allow a fair comparison between methods, the input processing pipeline cached model speciﬁc representations and transformations of the data.
To further increase efﬁciency of the RNNs, sequences were binned in to buckets of jointly trained instances depending on their sequence length.
The buckets were determined according to the (0 . 25 , 0 . 5 , 0 . 75) quantiles of the length distributions of the datasets.
A. T RAINING , M ODEL A RCHITECTURES AND H YPERPARAMETER S EARCH
General All models were trained using the Adam optimizer, while randomly sampling the learning rate from (0 . 001 , 0 . 0005 , 0 . 00025 , 0 . 0001) .
Further, the batch size of all methods was sampled from the values (32 , 64 , 128 , 256) .
Recurrent neural networks For the RNN based methods (GRU-S IMPLE ,
P HASEDLSTM, GRU-D and IP-N ETS ), the number of units was sampled in from the values (16 , 32 , 64 , 128 , 256 , 512) .
Further, recurrent dropout and input dropout were sampled from the values (0 . 0 , 0 . 1 , 0 . 2 , 0 . 3) .
Solely, for the P HASED -LSTM method, we did not apply dropout to the recurrent state and the inputs, as the learnt frequencies were hypothesized to fulﬁll a similar function as dropout (Neil et al., 2016).
S E FT
We vary the number of layers, dropout in between the layers and the number of nodes per layer for both the encoding network h θ and the aggregation network g ψ from the same ranges.
The number of layers is randomly sampled between 1 and 5 , the number of nodes in a layer are uniformly sampled from the range (16 , 32 , 64 , 128 , 256 , 512) and the dropout fraction is sampled from the values (0 . 0 , 0 . 1 , 0 . 2 , 0 . 3) .
The width of the embedding space prior to aggregation is sampled from the values (32 , 64 , 128 , 256 , 512 , 1024 , 2048) .
The aggregation function selected to be one of mean , sum and max .
The number of dimensions used for the positional embedding τ is selected uniformly from (4 , 8 , 16) and max ts us selected from the values (10 , 100 , 1000) .
S E
FT-Attn
The parameters for the encoding and aggregation networks are sampled in a similar fashion as for S E FT.
In contrast we set the aggregation function to be sum as described in the text.
[Table]
Disorders of lipid metabolism
[Table]
Essential
hypertension
[Table]
Fluid
[Table]
Gastrointestinal
[Table]
Hypertension with complications and secondary hypertension
[Table]
Other
[Table]
Pneumonia
[Table]
except that
caused by tuberculosis or sexually transmitted disease)
[Table]
Respiratory failure; insufﬁciency; arrest (adult
[Table]
(except in labor)
[Table]
Further we use a constant architecture for the attention network f (cid:48) with 2 layers, 64 nodes per layer, 4 heads and a dimensionality of the dot product space d of 128 .
We solely sample the amount of attention dropout uniformly from the values (0 . 0 , 0 . 1 , 0 .
Transformer
We utilize the same model architecture as deﬁned in Vaswani et al.
(2017), where we use a one hidden layer MLP as a feed-forward network, with dimensionality of the hidden layer selected to be twice the model dimensionality.
The parameters for the Transformer network were sampled according to the following criteria.
The dimensionality of the model was sampled uniformly from the values (64 , 128 , 256 , 512 , 1024) , the number of attention heads per layer from the values (2 , 4 , 8) and the number of layers from the range [1 , 6] ∈ N .
Further, we sampled the amount of dropout of the residual connections and the amount of attention dropout uniformly from the values (0 . 0 , 0 . 1 , 0 . 2 , 0 . 3 , 0 . 5) , and the maximal timescale for the time embedding from the values (10 , 100 , 1000) (similar to the S E FT approach).
[Table]
[Figure]
Runtime in s
AU
C i n %
M3-Mortality
Runtime in s
AU
[Figure]
n %
P-Mortality
[Figure] [Figure]
[Figure] [Figure]
[Figure]
W e i gh t e d
AU
C i n %
H-MNIST
Runtime in s
W e i gh t e d
AU
C i n %
M3-Phenotyping
[Figure]