G RAPH -G UIDED N ETWORK FOR I RREGULARLY S AMPLED M ULTIVARIATE T IME S ERIES
A BSTRACT
In many domains, including healthcare, biology, and climate science, time series are irregularly sampled with varying time intervals between successive readouts and different subsets of variables (sensors) observed at different time points.
Here, we introduce R AINDROP , a graph neural network that embeds irregularly sampled and multivariate time series while also learning the dynamics of sensors purely from observational data.
R AINDROP represents every sample as a separate sensor graph and models time-varying dependencies between sensors with a novel message passing operator.
It estimates the latent sensor graph structure and leverages the structure together with nearby observations to predict misaligned readouts.
This model can be interpreted as a graph neural network that sends messages over graphs that are optimized for capturing time-varying dependencies among sensors.
We use
R AINDROP to classify time series and interpret temporal dynamics on three healthcare and human activity datasets.
R AINDROP outperforms state-of-the-art methods by up to 11.4% (absolute F1-score points), including techniques that deal with irregular sampling using ﬁxed discretization and set functions.
R AINDROP shows superiority in diverse setups, including challenging leave-sensor-out settings.
I NTRODUCTION
Multivariate time series are prevalent in a variety of domains, including healthcare, space science, cyber security, biology, and ﬁnance (Ravuri et al., 2021;
Sousa et al., 2020; Sezer et al., 2020;
Fawaz et al., 2019).
Practical issues often exist in collecting sensor measurements that lead to various types of irregularities caused by missing observations, such as saving costs, sensor failures, external forces in physical systems, medical interventions, to name a few (Choi et al., 2020).
While temporal machine learning models typically assume fully observed and ﬁxed-size inputs, irregularly sampled time series raise considerable challenges (Shukla & Marlin, 2021;
Hu et al., 2021).
For example, observations of different sensors might not be aligned, time intervals among adjacent observations are different across sensors, and different samples have different numbers of observations for different subsets of sensors recorded at different time points (Horn et al., 2020; Wang et al., 2011).
Prior methods for dealing with irregularly sampled time series involve ﬁlling in missing values using interpolation, kernel methods, and probabilistic approaches (Schafer & Graham, 2002).
However, the absence of observations can be informative on its own (Little & Rubin, 2014) and thus imputing missing observations is not necessarily beneﬁcial (Agniel et al., 2018).
While modern techniques involve recurrent neural network architectures ( e.g. , RNN, LSTM, GRU) (Cho et al., 2014) and transformers (Vaswani et al., 2017), they are restricted to regular sampling or assume aligned measurements across modalities.
For misaligned measurements, existing methods tend to rely on a two-stage approach that ﬁrst imputes missing values to produce a regularly-sampled dataset and then optimizes a model of choice for downstream performance.
This decoupled approach does not fully exploit informative missingness patterns or deal with irregular sampling, thus producing suboptimal performance (Wells et al., 2013; Li & Marlin, 2016).
Thus, recent methods circumvent the imputation stage and directly model irregularly sampled time series (Che et al., 2018; Horn et al., 2020).
Previous studies (Wu et al., 2021; Li et al., 2020a; Zhang et al., 2019) have noted that inter-sensor correlations bring rich information in modeling time series.
However, only few studies consider relational structure of irregularly sampled time series, and those which do have limited ability in capturing inter-sensor connections (Wu et al., 2021; Shukla & Marlin, 2018).
In contrast, we integrate recent advances in graph neural networks to take advantage of relational structure among sensors.
We learn latent graphs from multivariate time series and model time-varying inter-sensor dependencies through neural message passing, establishing graph neural networks as a way to model sample-varying and time-varying structure in complex time series.
[Figure]
Present work.
To address the characteristics of irregularly sampled time series, we propose to model temporal dynamics of sensor dependencies and how those relationships evolve over time.
Our intuitive assumption is that the observed sensors can indicate how the unobserved sensors currently behave, which can further improve the representation learning of irregular multivariate time series.
We develop R AINDROP 1 , a graph neural network that leverages relational structure to embed and classify irregularly sampled multivariate time series.
R AINDROP takes samples as input, each sample containing multiple sensors and each sensor consisting of irregularly recorded observations ( e.g. , in clinical data, an individual patient’s state of health is recorded at irregular time intervals with different subsets of sensors observed at different times).
R AINDROP model is inspired by how raindrops hit a surface at varying times and create ripple effects that propagate through the surface.
Mathematically, in R AINDROP , observations ( i.e. , raindrops) hit a sensor graph ( i.e. , surface) asynchronously and at irregular time intervals.
Every observation is processed by passing messages to neighboring sensors ( i.e. , creating ripples), taking into account the learned sensor dependencies (Figure 1).
As such, R AINDROP can handle misaligned observations, varying time gaps, arbitrary numbers of observations, and produce multi-scale embeddings via a novel hierarchical attention.
We represent dependencies with a separate sensor graph for every sample wherein nodes indicate sensors and edges denote relationships between them.
Sensor graphs are latent in the sense that graph connectivity is learned by R AINDROP purely from observational time series.
In addition to capturing sensor dependencies within each sample, R AINDROP i) takes advantage of similarities between different samples by sharing parameters when calculating attention weights, and ii) considers importance of sequential sensor observations via temporal attention.
R AINDROP adaptively estimates observations based on both neighboring readouts in the temporal domain and similar sensors as determined by the connectivity of optimized sensor graphs.
We compare R AINDROP to ﬁve state-of-the-art methods on two healthcare datasets and an activity recognition dataset across three experimental settings, including a setup where a subset of sensors in the test set is malfunctioning ( i.e. , have no readouts at all).
Experiments show that R AINDROP outperforms baselines on all datasets with an average AUROC improvement of 3.5% in absolute points on various classiﬁcation tasks.
Further, R AINDROP improves prior work by a 9.3% margin (absolute points in accuracy) when varying subsets of sensors malfunction.
R ELATED W ORK
Our work here builds on time-series representation learning and notions of graph neural networks and attempts to resolve them by developing a single, uniﬁed approach for analysis of complex time series.
Learning with irregularly sampled multivariate time series.
Irregular time series are characterized by varying time intervals between adjacent observations (Zerveas et al., 2021;
Tipirneni &
Reddy, 2021; Chen et al., 2020).
In a multivariate case, irregularity means that observations can be misaligned across different sensors, which can further complicate the analysis.
Further, because of a multitude of sampling frequencies and varying time intervals, the number of observations can also vary considerably across samples (Fang & Wang, 2020; Kidger et al., 2020).
Predominant downstream tasks for time series are classiﬁcation ( i.e. , predicting a label for a given sample, e.g. ,
Tan et al.
; Ma et al.
and forecasting ( i.e. , anticipating future observations based on historical observations, e.g. , Wu et al.
(2020a)).
The above mentioned characteristics create considerable challenges for models that expect well-aligned and ﬁxed-size inputs (Shukla & Marlin, 2020).
An intuitive way to deal with irregular time series is to impute missing values and process them as regular time series (Mikalsen et al., 2021; Li & Marlin, 2020; Shan & Oliva, 2021).
However, imputation methods can distort the underlying distribution and lead to unwanted distribution shifts.
To this end, recent methods directly learn from irregularly sampled time series (Chen et al., 2018).
For example, Che et al.
(2018) develop a decay mechanism based on gated recurrent units ( GRU-D ) and binary masking to capture long-range temporal dependencies.
SeFT (Horn et al., 2020) takes a set-based approach and transforms irregularly sampled time series datasets into sets of observations modeled by set functions insensitive to misalignment.
mTAND (Shukla & Marlin, 2021) leverages a multi-time attention mechanism to learn temporal similarity from non-uniformly collected measurements and produce continuous-time embeddings.
IP-Net (Shukla & Marlin, 2018) and DGM 2 (Wu et al., 2021) adopt imputation to interpolate irregular time series against a set of reference points using a kernel-based approach.
The learned inter-sensor relations are static ignoring sample-speciﬁc and time-speciﬁc characteristics.
In contrast with the above methods, R AINDROP leverages dynamic graphs to address the characteristics of irregular time series and produce high-quality representations.
Learning with graphs and neural message passing.
There has been a surge of interest in applying neural networks to graphs, leading to the development of graph embeddings (Zhou et al., 2020; Li et al., 2021), graph neural networks (Wu et al., 2020b), and message passing neural networks (Gilmer et al., 2017).
To address the challenges of irregular time series, R AINDROP speciﬁes a message passing strategy to exchange neural message along edges of sensor graphs and deal with misaligned sensor readouts (Riba et al., 2018;
Nikolentzos et al., 2020; Galkin et al., 2020; Fey et al., 2020;
Lin et al., 2018; Zhang et al., 2020).
In particular, R AINDROP considers message passing on latent sensor graphs, each graph describing a different sample ( e.g. , patient, Figure 1), and it speciﬁes a message-passing network with learnable adjacency matrices.
The key difference with the predominant use of message passing is that R AINDROP uses it to estimate edges (dependencies) between sensors rather than applying it on a ﬁxed, apriori-given graph.
To the best of our knowledge, prior work did not utilize sensor dependencies for irregularly sampled time series.
While prior work used message passing for regular time series (Wang et al., 2020; Wu et al., 2020c; Kalinicheva et al., 2020; Zha et al., 2022), its utility for irregularly sampled time series has not yet been studied.
R AINDROP
Let D =
{ ( S i , y i ) | i = 1 , . . .
, N } denote an irregular time series dataset with N labeled samples (Figure 2).
Every sample S i is an irregular multivariate time series with a corresponding label y i ∈ { 1 , . . .
, C } , indicating which of the C classes S
i is associated with.
Each sample contains M non-uniformly measured sensors that are denoted as u , v , etc.
R AINDROP can also work on samples with only a subset of active sensors (see Sec. 4.1).
Each sensor is given by a sequence of observations ordered by time.
For sensor u in sample S i , we denote a single observation as a tuple ( t, x ti,u ) , meaning that sensor u was recorded with value x ti,u ∈ R at timestamp t ∈ R + .
We omit sample index i and sensor index u in timestamp t .
Sensor observations are irregularly recorded, meaning that time intervals between successive observations can vary across sensors.
For sensor u in sample S i , we use
T i,u to denote the set of timestamps that u , or at least one of u ’s L -hop neighbors ( L is the number of layers in R AINDROP ’s message passing) is recorded.
We use || and T to denote concatenation and transpose, respectively.
We omit layer index l ∈ { 1 , . . .
, L } for simplicity when clear from the text.
Problem (Representation learning for irregularly sampled multivariate time series).
A dataset
D of irregularly sampled multivariate time series is given, where each sample S i has multiple sensors and each sensor has a variable number of observations.
R AINDROP learns a function f : S i
→ z i that maps S i to a ﬁxed-length representation z i suitable for downstream task of interest, such as classiﬁcation.
Using learned z i , R AINDROP can predict label ˆ y i ∈ { 1 , . .
, C } for S i .
R AINDROP learns informative embeddings for irregularly samples time series.
The learned embeddings capture temporal patterns of irregular observations and explicitly consider varying dependencies between sensors.
While we focus on time-series classiﬁcation in this work, the proposed method can be easily extended to broader applications such as regression, clustering and generation tasks.
3. O VERVIEW OF R AINDROP
[Figure]
R AINDROP aims to learn a ﬁxed-dimensional embedding z
i for a given sample S i and predict the associated label ˆ y i .
To this end, it generates sample embeddings using a hierarchical architecture composed of three levels to model observations (sensor readouts), sensors, and whole samples (Figure 2).
Without loss of generality, we describe R AINDROP ’s procedure as if observations arrive one at a time (one sensor is observed at time t and other sensors do not have observations).
If there are multiple observations at the same time, R AINDROP can effortlessly process them in parallel.
R AINDROP ﬁrst constructs a graph for every sample where nodes represent sensors and edges indicate relations between sensors (Sec. 3.2).
We use G i to denote the sensor graph for sample S i and e i,uv to represent the weight of a directed edge from sensor u to sensor v in G i .
Sensor graphs are automatically optimized considering sample-wise and time-wise speciﬁcity.
The key idea of R AINDROP is to borrow information from u ’s neighbors based on estimated relationships between u and other sensors.
This is achieved via message passing carried out on S
i ’s dependency graph and initiated at node u in the graph.
When an observation ( t, x ti,u ) is recorded for sample S i at time t , R AINDROP ﬁrst embeds the observation at active sensor u ( i.e. , sensor whose value was recorded) and then propagates messages ( i.e. , the observation embeddings) from u to neighboring sensors along edges in sensor dependency graph G i .
As a result, recording the value of u can affect u ’s embedding as well as embeddings of other sensors that related to u (Sec. 3.3).
Finally, R AINDROP generates sensor embeddings by aggregating all observation embeddings for each sensor (across all timestamps) using temporal attention weights (Sec. 3.4).
At last, R AINDROP embeds sample S i based on sensor embeddings (Sec. 3.5) and feeds the sample embedding into a downstream predictor.
3. C ONSTRUCTING S ENSOR D EPENDENCY G RAPHS
We build a directed weighted graph G i
=
{V , E i } for every sample S i and refer to it as the sensor dependency graph for S i .
Nodes V represent sensors and edges E i describe dependencies between sensors in sample S i that R AINDROP infers.
As we show in experiments, R AINDROP can be directly used with samples that only contain a subset of sensors in V .
We denote edge from u to v as a triplet ( u, e i,uv , v ) , where e i,uv ∈ [0 , 1] represents the strength of relationship between sensors u and v in sample S
i .
Edge ( u, e i,uv , v ) describes the relationship between u and v : when u receives an observation, it will send a neural message to v following edge e i,uv .
If e i,uv = 0 , there is no exchange of neural information between u and v , indicating that the two sensors are unrelated.
We assume that the importance of u to v is different than the importance of v to u , and so we treat sensor dependency graphs as directed, i.e. , e i,uv (cid:54) =
e i,vu .
All graphs are initialized as fully-connected graphs ( i.e. , e i,uv = 1 for any u , v and S i ) and edge weights e i,uv are updated following Eq.
3 during model training.
If available, it is easy to integrate additional domain knowledge into graph initialization.
3. G ENERATING E MBEDDINGS OF I NDIVIDUAL O BSERVATIONS
Let u indicate active sensor at time t ∈ T i,u , i.e., sensor whose value x ti,u is observed at t , and let u be connected to v through edge ( u, e i,uv , v ) .
We next describe how to produce observation embeddings h ti,u ∈ R d h and h ti,v ∈ R d h for sensors u and v , respectively (Figure 3a).
We omit layer index l and note that the proposed strategy applies to any number of layers.
[Figure]
Embedding an observation of an active sensor.
Let u denote an active sensor whose value has just been observed as x ti,u .
For sufﬁcient expressive power (Veli ˇ ckovi ´
c et al., 2018)
we map observation x ti,u to a high-dimensional space using a nonlinear transformation: h ti,u
= σ ( x ti,u R u ) .
We use sensorspeciﬁc transformations because values recorded at different sensors can follow different distributions, which is achieved by trainable weight vectors R u depending on what sensor is activated (Li et al., 2020b).
Alternatives, such as a multilayer perceptron, can be considered to transform x ti,u into h ti,u .
As h ti,u represents information brought on by observing x ti,u , we regard h ti,u as the embedding of u ’s observation at t .
Sensor-speciﬁc weight vectors R u are shared across samples.
Passing messages along sensor dependency graphs.
For sensors that are not active at timestamp t but are neighbors of the active sensor u in the sensor dependency graph G i , R AINDROP uses relationships between u and those sensors to estimate observation embeddings for them.
We proceed by describing how R AINDROP generates observation embedding h t i,v for sensor v assuming v is a neighbor of u in G i .
Given h ti,u and edge ( u, e i,uv , v ) , we ﬁrst calculate inter-sensor attention weight α ti,uv ∈ [0 , 1] , representing how important u is to v via the following equation:
[Equation]
where r v ∈ R d r is a trainable weight vector that is speciﬁc to the sensor receiving the message ( i.e. , h ti,u ).
Vector r v allows the model to learn distinct attention weights for different edges going out from the same sensor u .
Further, p ti ∈ R d t is the time representation obtained by converting a 1-dimensional timestamp t into a multi-dimensional vector p ti by passing t through a series of trigonometric functions (Horn et al., 2020).
See Appendix A.1 for details.
R AINDROP uses p ti to calculate attention weights that are sensitive to time.
Finally, D is a trainable weight matrix mapping
h ti,u from d h dimensions to ( d r + d t ) dimensions.
Taken this together, we can estimate the embedding h ti,v for u ’s neighbor v as follows:
[Equation]
where w u , w v ∈ R
d
h are trainable weight vectors shared across all samples.
The w u is speciﬁc to active sensor u and w v is speciﬁc to neighboring sensor v .
In the above equation, e i,uv denotes edge weight shared across all timestamps.
The above message passing describes the processing of a single observation at a single timestamp.
In case multiple sensors are active at time t and connected with v , we normalize α ti,uv (with softmax function) across active sensors and aggregate messages at v .
Overall, R AINDROP produces observation embedding h ti,v for sensor v through its relational connection with u , even though there is no direct measurement of v at time t .
These message passing operations are performed to adaptively and dynamically estimate missing observations in the embedding space based on recorded information and learned graph structure.
Updating sensor dependency graphs.
We describe the update of edge weights and prune of graph structures in the situation that stacks multiple R AINDROP layers (Figure 3).
Here we explicitly show layer index l because multiple layers are involved in the computation.
As no prior knowledge is assumed, we initialize the graph as all sensors connected with each other.
However, the fully connected edges may bridge sensors that should be independent, which will introduce spurious correlations and prevent the model from paying attention to the truly important connections.
Addressing this issue,
R AINDROP automatically updates edge weights and prunes out less important edges.
Based on the aggregated temporal inﬂuence driven by the inter-sensor attention weights α ( l ) ,t i,uv , we update edge weights e ( l ) i,uv in each layer l ∈ { 1 , . . .
, L } by:
[Equation]
where T i,u denotes the set of all timestamps where there is message passes from u to v .
In particular, we set e (0) i,uv = 1 in the initialization of graph structures.
We use L = 2 in all our experiments.
In every layer, we order the estimated values e ( l ) i,uv for all edges in sample S
i and prune bottom K % edges with smallest edge weights (Yang et al., 2021).
Pruned edges are not re-added in later layers.
3. G ENERATING S ENSOR E MBEDDINGS
Next we describe how to aggregate observation embeddings into sensor embeddings z i,v , taking sensor v as an example (Figure 3b).
Previous step (Sec. 3.3) generates observation embeddings for every timestamp when either v or v ’s neighbor is observed.
The observation embeddings at different timestamps have unequal importance to the the sensor embedding (Zerveas et al., 2021).
We use the temporal attention weight (scalar) β ti,v to represent the importance of observation embedding at t .
We use T i,v = { t 1 , t 2 , . . .
, t T } to denote all the timestamps when a readout is observed in v (we can directly generate h ti,v ) or in v ’s neighbor (we can generate h ti,v through message passing).
The β ti,v is the corresponding element of vector β i,v which include the temporal attention weights at all timestamps t ∈ T i,v .
We use temporal self-attention to calculate β i,v , which is different from the standard self-attention (Hu et al., 2020; Yun et al., 2019).
The standard dot-product self-attention generates an attention matrix with dimension of T × T (where T = |T
i,v | can vary across samples) that has an attention weight for each pair of observation embeddings.
In our case, we only need a single attention vector where each element denotes the temporal attention weight of an observation embedding when generating the sensor embedding.
Thus, we modify the typical self-attention model to ﬁt our case: using a trainable s ∈ R T × 1 to map the self-attention matrix ( R T × T ) to T -dimensional vector β
i,v ( R T × 1 ) through matrix product (Appendix A.2).
The following steps describe how to generate sensor embeddings.
We ﬁrst concatenate observation embedding h ti,v with time representation p ti to include information of timestamp.
Then, we stack the concatenated embeddings [ h ti,v || p ti ] for all t ∈ T i,v into a matrix H i,v .
The H i,v contains all information of observations and timestamps for sensor v .
We calculate β t i,v through:
[Equation]
where Q i,v and K i,v are two intermediate matrices that are derived from the stacked observation embeddings.
In practice, Q i,v = H i,v W Q and K i,v = H i,v W K are linearly mapped from H i,v parameterized by W Q and W K , respectively (Vaswani et al., 2017).
The √ d k is a scaling factor where d k is the dimension after linear mapping.
Based on the learned temporal attention weights β ti,v , we calculate sensor embedding z i,v through:
[Equation]
where weight matrix W is a linear projector shared by all sensors and samples.
It is worth to mention that all attention weights (such as α ti,uv and β i,v ) can be multi-head.
In this work, we describe the model in the context of single head for brevity.
Using attentional aggregation, R AINDROP can learn a ﬁxed-length sensor embedding for arbitrary number of observations.
Meanwhile, R AINDROP is capable of focusing on the most informative observation embeddings.
We process all observation embeddings as a whole instead of sequentially, which allows parallel computation for faster training and also mitigates the performance drop caused by modeling long dependencies sequentially.
In the case of sensors with very large number of observations, we can reduce the length of time series by subsampling or splitting a long series into multiple short series.
3. G ENERATING S AMPLE E MBEDDINGS
Finally, for sample S i , we aggregate sensor embeddings z i,v (Eq. 5) across all sensors to obtain an embedding z i ∈ R d z through a readout function g as follows:
z i = g
( z i,v | v = 1 , 2 , . . .
, M ) (such as concatenation).
When a sample contains a large number of sensors, R AINDROP can seamlessly use a set-based readout function such as averaging aggregation (Appendix A.3).
Given an input sample
S i , R AINDROP ’s strategy outlined in Sec. 3.2-3.5 produces a sample embedding z
i
that can be further optimized for downstream tasks.
3. I MPLEMENTATION AND P RACTICAL C ONSIDERATIONS
Loss function.
R AINDROP ’s loss function is formulated as:
L =
L CE + λ L r , where L r =
M 2 (cid:80) u,v ∈V (cid:80) i,j ∈V || e i,uv − e j,uv
|| 2 / ( N − 1) 2 , where L CE is cross entropy and L r is a regularizer to encourage the model to learn similar sensor dependency graphs for similar samples.
The L r measures averaged Euclidean distance between edge weights across all samples pairs, in all sensor pairs (including self-connections).
The λ is a user-deﬁned coefﬁcient.
Practically, as N can be large, we calculate L r only for samples in a batch.
Downstream tasks.
If a sample has auxiliary attributes ( e.g. , a patient’s demographics) that do not change over time, we can project the attribute vector to a d a -dimensional vector a i with a fullyconnected layer and concatenate it with the sample embedding, getting [ z i || a i ] .
At last, we feed [ z i
|| a
i ] (or only z i if a i is not available) into a neural classiﬁer ϕ :
R d z + d a → { 1 , . . .
, C } .
In our experiments, ϕ is a 2-layer fully-connected network with C neurons at the output layer returning prediction ˆ y i = ϕ
([ z i || a i ]) for sample S i .
Sensor dependencies.
While modeling sensor dependencies, we involve observation embedding ( h ti,u , Eq. 1) of each sample in the calculation of attention weights.
Similarly, to model time-wise speciﬁcity in graph structures, we consider time information ( p ti , Eq. 1) when measuring α ti,uv .
R AINDROP can capture similar graph structures across samples from three aspects (Appendix A.4): (1) the initial graphs are the same in all samples; (2) the parameters in message passing ( R u ; w u , w v , Eq. 2), inter-sensor attention weights calculation ( D , Eq. 1), and temporal attention weights calculation ( s , Eq. 4; W , Eq. 5) are shared by all samples; (3) we encourage the model to learn similar graph structures by adding a penalty to disparity of structures ( L r ).
Scalability.
R AINDROP is efﬁcient because embeddings can be learned in parallel.
In particular, processing of observation embeddings is independent across timestamps.
Similarly, sensor embeddings can be processed independently across different sensors (Figure 3).
While the complexity of temporal self-attention calculation grows quadratically with the number of observations, it can be practically implemented using highly-optimized matrix multiplication.
E XPERIMENTS
Datasets.
Below we brieﬂy overview healthcare and human activity datasets.
(1) P19 (Reyna et al., 2020) includes 38,803 patients that are monitored by 34 sensors.
Each patient is associated with a binary label representing the occurrence of sepsis.
(2) P12 (Goldberger et al., 2000) records temporal measurements of 36 sensors of 11,988 patients in the ﬁrst 48-hour stay in ICU.
The samples are labeled based on hospitalization length.
(3) PAM (Reiss & Stricker, 2012) contains 5,333 segments from 8 activities of daily living that are measured by 17 sensors.
Details are in Appendix A.5.
Baselines.
We compare R AINDROP with ﬁve state-of-the-art baselines:
Transformer (Vaswani et al., 2017), Trans-mean , GRU-D (Che et al., 2018), SeFT (Horn et al., 2020), and mTAND (Shukla &
Marlin, 2021).
The Trans-mean is an imputation method combining transformer architecture with commonly used average interpolation ( i.e. , missing values are replaced by average observations in each sensor).
The mTAND (Shukla & Marlin, 2021) method has been shown to outperform numerous recurrent models including RNN-Impute (Che et al., 2018), RNN-Simple , and Phased-LSTM (Neil et al., 2016), along with ordinary differential equations (ODE)-based models such as LATENT-ODE and ODE-RNN (Chen et al., 2018).
For this reason, we compare with mTAND and do not report comparison with those techniques in this paper.
Even though, to better show the superiority of
R AINDROP , we provide extensive comparison with popular approaches, such as DGM 2 -O
(Wu et al., 2021) and MTGNN (Wu et al., 2020c), that are designed for forecasting tasks.
Further details are in
Table 1 and Appendix A.11.
Details on hyperparameter selection and baselines are in Appendix A.6, and evaluation metrics are presented in Appendix A.7.
D
ETTINGS
Setting 1: Classic time series classiﬁcation.
Setup. Setup.
Setup. Setup.
Setup. Setup.
Setup. Setup.
Setup. Setup.
Setup. Setup.
Setup. Setup.
Setup. Setup.
Setup.
We randomly split the dataset into training (80%), validation (10%), and test (10%) set.
The indices of these splits are ﬁxed across all methods.
Results.
Results. Results.
Results. Results.
Results. Results.
Results. Results.
Results. Results.
Results. Results.
Results. Results.
Results. Results.
As shown in Table 1, R AINDROP obtains the best performance across three benchmark datasets, suggesting its strong performance for time series classiﬁcation.
In particular, in binary classiﬁcation (P19 and P12), R AINDROP outperforms the strongest baselines by 5.3% in AUROC and 4.8% in AUPRC on average.
In a more challenging 8-way classiﬁcation on the PAM dataset,
R AINDROP outperforms existing approaches by 5.7% in accuracy and 5.5% in F1 score.
Further exploratory analyses and benchmarking results are shown in Appendix A.9-A.10.
[Table]
Setting 2: Leave-ﬁxed-sensors-out.
Setup. Setup.
Setup. Setup.
Setup. Setup.
Setup. Setup.
Setup. Setup.
Setup. Setup.
Setup. Setup.
Setup. Setup.
Setup.
R AINDROP can compensate for missing sensor observations by exploiting dependencies between sensors.
To this end, we test whether R AINDROP can achieve good performance when a subset of sensors are completely missing.
This setting is practically relevant in situations when, for example, sensors fail or are unavailable.
We select a fraction of sensors and hide all their observations in both validation and test sets (training samples are not changed).
In particular, we leave out the most informative sensors as deﬁned by information gain analysis (Appendix A.8).
The left-out sensors are ﬁxed across samples and models.
Results. Results.
Results. Results.
Results. Results.
Results. Results.
Results. Results.
Results. Results.
Results. Results.
Results. Results.
Results.
We report results taking PAM as an example.
In Table 2 (left block), we observe that R AINDROP achieves top performance in 18 out of 20 settings when the number of left-out sensors goes from 10% to 50%.
With the increased amount of missing data, R AINDROP yield greater performance improvements.
R AINDROP outperforms baselines by up to 24.9% in accuracy, 50.3% in precision, 29.3% in recall, and 42.8% in F1 score.
Setting 3: Leave-random-sensors-out.
Setup.
Setup. Setup. Setup. Setup.
Setup. Setup. Setup. Setup.
Setup. Setup. Setup.
Setup.
Setup.
Setup. Setup.
Setup.
Setting 3 is similar to Setting 2 except that left-out sensors are randomly selected in each sample instead of being ﬁxed.
In each test sample, we select a subset of sensors and regard them as missing by replacing all of their observations with zeros.
Results.
Results. Results.
Results. Results.
Results. Results.
Results. Results.
Results. Results.
Results. Results.
Results. Results.
Results. Results.
We provide results for the PAM dataset in Table 2 (right block).
We ﬁnd that R AINDROP achieves better performance than baselines in 16 out of 20 settings and that Trans-mean and GRU-D are the strongest competitors.
Further, we evaluated R AINDROP in another setting where the model is trained on one group of samples ( e.g. , females) and tested on another group not seen during training ( e.g. , males).
Experimental setup and results are detailed in Appendix A.13.
[Table] [Table]
A BLATION S TUDY AND V ISUALIZATION OF O PTIMIZED S ENSOR G RAPHS
Ablation study.
Considering the PAM dataset and a typical setup (Setting 1)
[Table]
we conduct an ablation study to evaluate how much various R AINDROP ’s components contribute towards its ﬁnal performance.
We examine the following components: inter-sensor dependencies (further decomposed into weights including e i,uv , r v , p ti
[Table]
and α ti,uv ), temporal attention
[Table]
and sensor-level concatenation.
We show in Appendix A.14 (Table 7) that all model components are necessary and that regularization L r contributes positively to R AINDROP ’s performance.
Visualizing sensor dependency graphs.
We investigate whether samples with the same labels get more similar sensor dependency graphs than samples with different labels.
To this end, we visualize inter-sensor dependencies (P19; Setting 1) and explore them.
Figure 4 shows distinguishable patterns between graphs of negative and positive samples, indicating that R AINDROP can extract relationships that are speciﬁc to downstream sample labels.
Further differential analysis provides insights that can inform early detection of sepsis from P19 clinical data.
Details are in Appendix A.15.
C ONCLUSION
We introduce R AINDROP , a graph-guided network for irregularly sampled time series.
R AINDROP learns a distinct sensor dependency graph for every sample capturing time-varying dependencies between sensors.
The ability to leverage graph structure gives R AINDROP unique capability to naturally handle misaligned observations, non-uniform time intervals between successive observations, and sensors with varying numbers of recorded observations.
Our ﬁndings have implications for using message passing as a way to leverage relational information in multivariate time series.
A CKNOWLEDGMENTS
This material is based upon work supported by the Under Secretary of Defense for Research and
Engineering under Air Force Contract No. FA8702-15-D-0001.
M.Z. is supported, in part, by
NSF under nos.
IIS-2030459 and IIS-2033384, Harvard Data Science Initiative, Amazon Research
Award, Bayer Early Excellence in Science Award, AstraZeneca Research, and Roche Alliance with
Distinguished Scientists Award.
Any opinions, ﬁndings, conclusions or recommendations expressed in this material are those of the authors and do not necessarily reﬂect the views of the funders.
The authors declare that there are no conﬂict of interests.
R EPRODUCIBILITY S TATEMENT
We ensure the reproducibility of our work by clearly presenting the model and providing publicly accessible code and data.
For all datasets used in this work, we share downloadable links to the raw sources and processed and ready-to-run datasets with the research community through this link: https://github.com/mims-harvard/Raindrop.
We specify all training details (e.g., preprocessing, data splits, hyperparameters, sensor selection) in the main text and Appendix.
Python implementation of
R AINDROP and all baseline methods is available at the aforementioned link.
Detailed description of data, scripts, and conﬁgurations along with examples of usage are also provided.
E THICS S TATEMENT
The ability of R AINDROP to learn robust information about sensors’ representations and dependencies creates new opportunities for applications, where time series are predominant, e.g. , in healthcare, biology, and ﬁnance.
In all these ﬁelds, especially in healthcare applications, our method should be used with caution.
Although our model can gain valuable insights from time series, users must consider the limitations of machine-guided predictions.
As with all data-driven solutions, our model may make biased predictions.
In the case of biomedical data, biases can exist within the data itself, which can be, for example, caused by considering demographic attributes, such as age, weight, and gender, that might correlate with protected/regulated attributes.
When target classes are highly imbalanced, our model can mitigate the issues by upsampling minority classes in every processed batch.
All datasets in this paper are publicly available and are not associated with any privacy or security concern.
Further, all data are anonymized to guard against breaching patients’ protected health information.
We followed PhysioNet privacy policy and guidelines (https://archive.physionet.org/ privacy.shtml) when experimenting with P12 and P19 datasets.
A A PPENDIX A. E NCODING T IMESTAMPS
For a given time value t , we pass it to trigonometric functions with the frequency of 10,000 (Vaswani et al., 2017) and generate time representation p t ∈ R ξ (omit sample index i for brevity) through (Horn et al., 2020):
[Equation]
where ξ is the expected dimension.
In this work, we set ξ = 16 in all experimental settings for all models.
Please note, we encode the time value which is a continuous timestamp, instead of time position which is a discrete integer indicating the order of observation in time series.
A. A DDITIONAL INFORMATION ON THE CALCULATION OF TEMPORAL ATTENTION WEIGHT
The Eq. 4 describes how we learn the temporal attention weights vector β
i,v for sensor v , following the self-attention formalism.
Different from the standard self-attention mechanism that generates an self-attention matrix, we generate a temporal attention weight vector.
The reason is that we only need an attention weight vector (instead of a matrix) to aggregate the observation embeddings into a single sensor embedding through weighted sum.
In the standard self-attention matrix, each element denotes the dependency of an observation embedding on another observation embedding.
Similarly, each row describes the dependencies of an observation embedding on all other observation embeddings (all the observations belong to the same sensor).
Our intuition is to aggregate a row in the self-attention matrix into a scalar that denotes the importance of the observation embedding to the whole sensor embedding.
In practice, we apply the weighted aggregation, parameterized by s , to every row in the self-attention matrix and concatenate the generated scalars into an attention vector.
Next, we give a concrete example to speciﬁcally describe the meaning of s .
Each row, j , of the self-attention matrix captures relationships of observation embedding h t j i,v to all observation embeddings { h t k
i,v : k = 1 , ..., T } .
Then, using the learnable weight vector s , these correlations between observations are aggregated across time to obtain temporal importance weight β t j i,v .
The β t j i,v represents the importance of the corresponding observation to the whole sensor embedding.
A. A DDITIONAL INFORMATION ON SAMPLE EMBEDDING
As we generate sample embedding by concatenating all sensor embeddings, the sample embedding could be relatively long when there is a large number of sensors.
To alleviate this issue, on one hand, we can reduce the dimension of sample embeddings by adding a neural layer (such as a simple fully-connected layer) after the concatenation.
On the other hand, when the number of sensors is super large, our model is ﬂexible and can effortlessly switch the concatenation to other readout functions (such as averaging aggregation): this will naturally solve the problem of long vectors.
We empirically show that concatenation works better than averaging in our case.
We see a boost in the AUROC score by 0.6% using concatenation instead of averaging for generating sample embeddings(P19; Setting 1).
A. A DDITIONAL INFORMATION ON SAMPLE SIMILARITIES
In this work, we assume all samples share some common characteristics to some extent.
When modeling the similarities across samples, we do not consider the situation where the samples are similar within latent groups and different across groups.
Our study focuses on the question of irregularity rather than the question of distribution shifts in time series.
To this end, in our experiments, we ﬁrst rigorously benchmark Raindrop using a standard evaluating setup (Setting 1, which is classiﬁcation of irregular time series).
This is the only setup that most existing methods consider (e.g., Shukla & Marlin (2021);
Che et al.
and we want to make sure our comparisons are fair.
In order to provide a more rigorous assessment of Raindrop’s performance, we also consider more challenging setups in our experiments (i.e., Settings 2-4) when the dataset is evaluated in a non-standard manner and the split is informed by a select data attribute.
[Table]
Our results on Setting 1 are consistent with those on Settings 2-4.
Results on harder Settings 2-4 show that Raindrop can perform comparably better than baselines.
Results across these diverse settings increase our conﬁdence that Raindrop is quite ﬂexible and widely applicable.
A. F URTHER DETAILS ON DATASETS
P19: PhysioNet Sepsis Early Prediction Challenge 2019.
P19 dataset (Reyna et al., 2020) contains 38,803 patients and each patient is monitored by 34 irregularly sampled sensors including 8 vital signs and 26 laboratory values.
The original dataset has 40,336 patients, we remove the samples with too short or too long time series, remaining 38,803 patients (the longest time series of the patient has more than one and less than 60 observations).
Each patient is associated with a static vector indicating attributes: age, gender, time between hospital admission and ICU admission, ICU type, and ICU length of stay (days).
Each patient has a binary label representing occurrence of sepsis within the next 6 hours.
The dataset is highly imbalanced with only ∼ 4% positive samples.
P12: PhysioNet Mortality Prediction Challenge 2012.
P12 dataset (Goldberger et al., 2000) includes 11,988 patients (samples), after removing 12 inappropriate samples following (Horn et al., 2020).
Each patient contains multivariate time series with 36 sensors (excluding weight), which are collected in the ﬁrst 48-hour stay in ICU.
Each sample has a static vector with 9 elements including age, gender, etc.
Each patient is associated with a binary label indicating length of stay in ICU, where negative label means hospitalization is not longer than 3 days and positive label marks hospitalization is longer than 3 days.
P12 is imbalanced with ∼ 93% positive samples.
PAM: PAMAP2 Physical Activity Monitoring.
PAM dataset (Reiss & Stricker, 2012) measures daily living activities of 9 subjects with 3 inertial measurement units.
We modify it to suit our scenario of irregular time series classiﬁcation.
We excluded the ninth subject due to short length of sensor readouts.
We segment the continuous signals into samples with the time window ofand the overlapping rate of 50%.
PAM originally has 18 activities of daily life.
We exclude the ones associated with less than 500 samples, remaining 8 activities.
After modiﬁcation, PAM dataset contains 5,333 segments (samples) of sensory signals.
Each sample is measured by 17 sensors and contains 600 continuous observations with the sampling frequency 100 Hz.
To make time series irregular, we randomly remove 60% of observations.
To keep fair comparison, the removed observations are randomly selected but kept the same for all experimental settings and approaches.
PAM is labelled by 8 classes where each class represents an activity of daily living.
PAM does not include static attributes and the samples are approximately balanced across all 8 categories.
To feed given data into neural networks, we set the input as zero if no value was measured.
In highly imbalanced datasets (P19 and P12) we perform batch minority class upsampling, which means that every processed batch has the same number of positive and negative class samples.
The dataset statistics including sparse ratio are provided in Table 3.
A. F URTHER DETAILS ON MODEL HYPERPARAMETERS
Baseline hyperparameters.
The implementation of baselines follows the corresponding papers including SeFT (Horn et al., 2020), GRU-D (Che et al., 2018), and mTAND (Shukla & Marlin, 2021).
We follow the settings of Transformer baseline in (Horn et al., 2020) while implementing
Transformer in our work.
For average imputation in Trans-mean, we replace the missing values by the global mean value of observations in the sensor (Shukla & Marlin, 2020).
We use batch size of
128 and learning rate of 0.0001.
Note that we upsample the minority class in each batch to make the batch balance (64 positive samples and 64 negative samples in each batch).
The chosen hyperparameters are the same across datasets (P19, P12, PAM), models (both baselines and R AINDROP ), and experimental settings.
Remarkably, we found that all the baselines make dummy predictions (classify all testing samples as the majority label) on PAM in Setting 2-3 while
R AINDROP makes reasonable predictions.
For the comparison to make sense ( i.e. , the baselines can make meaningful predictions), we use learning rate of 0.001 for baselines on PAM.
GRU-D haslayers while other models have 2 layers.
We run all models for 20 epochs, store the parameters that obtain the highest AUROC in the validation set, and use it to make predictions for testing samples.
We use the Adam algorithm for gradient-based optimization (Kingma & Ba, 2014).
R AINDROP hyperparameters.
Next, we report the setting of unique hyperparameters in our R AINDROP .
In the generation of observation embedding, we set R u as a 4-dimensional vector, thus the produced observation embedding has 4 dimensions.
The dimensions of time representation p t and r v are both 16.
The trainable weight matrix D has shape of 4 × 32 .
The dimensions of w u and w v are the same as the number of sensors: 34 in P19, 36 in P12, and 17 in PAM.
We set the number of
R AINDROP layers L as 2 while the ﬁrst layer prunes edges and the second layer does not.
We set the proportion of edge pruning as 50% (K=50), which means we remove half of the existing edges that have the lowest weights.
The d k is set to 20, while the shape of W is 20 × 20 .
All the activation functions, without speciﬁc clariﬁcation, are sigmoid functions.
The d a is set equal to the number of sensors.
The ﬁrst layer of ϕ has 128 neurons while the second layer has C neurons ( i.e. , 2 for
P19 and P12; 8 for PAM).
We set λ = 0 .
02 to adjust L r regularization scale.
All the preprocessed datasets and implementation codes are made available online.
Further details are available through
R AINDROP ’s code and dataset repository.
Readout function.
Here we discuss the selection of readout function g in section 3.5.
Our preliminary experiments show that concatenation outperforms other popular aggregation functions such as averaging (Errica et al., 2021) and squeeze-excitation readout function (Kim et al., 2021;
Hu et al., 2018).
While any of those aggregation functions can be considered, we used concatenation throughout all experiments in this manuscript.
A. P ERFORMANCE METRICS
Since P19 and P12 datasets are imbalanced, we use the Area Under a ROC Curve (AUROC) and
Area Under Precision-Recall Curve (AUPRC) to measure performance.
As the PAM dataset is nearly balanced, we also report accuracy, precision, recall and F1 score.
We report mean and standard deviation values over 5 independent runs.
Model parameters that achieve the best AUROC value on the validation set are used for test set.
A. F URTHER DETAILS ON S ETUP DETAILS FOR S ETTING
In Setting 2, the selected missing sensors are ﬁxed across different models and chosen in the following way.
First, we calculate the importance score for each sensor and rank them in a descending order.
The importance score is based on information gain, which we calculate with feeding the observations into a Random Forest classiﬁer with 20 decision trees.
In particular, we treat each sample as only having one sensor, then feed the single sensor into random forest classiﬁer and record the AUROC.
The higher AUROC indicates the sensor provides higher information gain.
When we have sensors ranked by their AUROC values, we choose the ﬁrst n sensors (the ones with highest AUROC values) and replace all observations in these sensors by zeros in all samples in validation and test set.
The number of missing sensors is deﬁned indirectly from the user with the sensors’ missing ratio which ranges from 0.1 to 0.
. A. A DDITIONAL INFORMATION ON MISSING PATTERN
This work propose R AINDROP which is a novel solution for irregularity in multivariate time series through inter-sensor dependencies.
R AINDROP is not in conﬂict with other solutions (such as missing pattern and temporal decay) for irregularity.
However, as the missing pattern is widely discussed in modelling incomplete time series (Che et al., 2018), we explore how to combine the advantages of relational structures and missing pattern.
We adopt mask matrix as a proxy of missing pattern as in Che et al.
Taking the architecture of R AINDROP , we concatenate the observation x ti,u with a binary mask indicator b ti,u as input.
The indicator b ti,u is set as 1 when there is an observation of sensor i at time t and set as 0 otherwise.
All the experimental settings and hyperparameters are the same as in R AINDROP (P19; Setting 1).
The experimental results show that taking advantage of missing pattern can slightly boost the AUROC by 1.2% and AUPRC by 0.9% in P19.
This empirically shed the light for future research on integrating multiple characteristics in representation of irregularly time series.
A. C OMPARISON BETWEEN TEMPORAL ATTENTION AND LSTM
We conduct extensive experiments to compare the effectiveness of temporal attention and LSTM.
To this end, we replace the temporal attention in sensor embedding generation (Eq 4-5) in R AINDROP by
LSTM layer which processes all observation embeddings sequentially.
We use zero padding to convert the irregular observations into ﬁxed-length time series so the data can be fed into LSTM architecture.
We regard the last output of LSTM as generated sensor embedding.
The number of LSTM cells equal to the dimension of observation embedding.
All the model structures are identical except in the part of temporal attention and LSTM.
We keep all experimental settings (P19; Setting 1) and hyperparameter selections the same.
The experimental results show that the temporal self-attention outperform LSTM by 1.8% (AUROC) and additionally saved 49% of the training time.
One potential reason is that the self-attention mechanism avoids recursion and allows parallel computation and also reduces performance degradation caused by long-term dependencies (Ganesh et al., 2021; Vaswani et al., 2017).
A. A DDITIONAL INFORMATION ON METHOD BENCHMARKING
Taking experimental Setting 1 ( i.e. , classic time series classiﬁcation) as an example, we conduct extensive experiments to compare Raindrop with ODE-RNN (Chen et al., 2020), DGM 2 -O (Wu et al., 2021), EvoNet (Hu et al., 2021), and MTGNN (Wu et al., 2020c).
As IP-Net (Shukla & Marlin, 2018) and mTAND (Shukla & Marlin, 2021) are from the same authors, we only compare with mTAND which is the latest model.
For the baselines, we follow the settings as provided in their public codes.
For methods, which cannot deal with irregular data ( e.g. , EvoNet and MTGNN), we ﬁrst impute the missing data using mean imputation and then feed data into the model.
For forecasting models ( e.g. ,
MTGNN) which are strictly not comparable with the proposed classiﬁcation model, we formulate the task as a single-step forecasting, concatenate the learned representations from all sensors and feed into a fully-connected layer (work as classiﬁer) to make prediction, and use cross-entropy to quantify the loss.
A. R ESULTS FOR P19 (S ETTINGS 2-3)
Here we report the experimental results for P19 in Setting 2 (Table 4) and Setting 3 (Table 5).
[Table]
[Table] [Table]
[Table]
. E VALUATION ON GROUP - WISE TIME SERIES CLASSIFICATION
To understand whether R AINDROP can adaptively adjust its structure and generalize well to other groups of samples which were not observed while training the model.
In this setting we split the data into two groups, based on a speciﬁc static attribute.
The ﬁrst split attribute is age , where we classify people into young ( < 65 years) and old ( ≥ 65 years) groups.
We also split patients into male and female by gender attribute.
Given the split attribute, we use one group as a train set and randomly split the other group into equally sized validation and test set.
Taking P19 as an example, we present the classiﬁcation results when the training and testing samples are from different groups.
As shown in Table 8, R AINDROP achieves the best results over all of the four given cross-group scenarios.
For instance, R AINDROP claims large margins (with 4.8% in
AUROC and 13.1% in AUPRC absolute improvement) over the second best model while training on males and testing on female patients.
Although R AINDROP is not designed to address domain adaptation explicitly, the results show that
R AINDROP performs better than baselines when transferring from one group of samples to another.
One reason for our good performance is that the learned inter-sensor weights and dependency graphs are sample-speciﬁc and their learning is based on the sample’s observations.
Thus, the proposed
R AINDROP has the power, to some extent, to adaptively learn the inter-sensor dependencies based on the test sample’s measurements.
R AINDROP is not generalizing to new groups, but generalizing to new samples, which leads to a good performance even though our model is not designed for domain adaptation.
We validate the reason empirically.
We remove the inter-sensor dependencies (set all sensors isolated in the dependency graph; set all α ti,uv and e ti,uv as 0) in R AINDROP and evaluate the model in group-wise time series classiﬁcation.
The experimental results show that the performance drops a lot when excluding dependency graphs and message passing in R AINDROP (Table 6).
Without inter-sensor dependencies our model is on par with other baselines and does not outperform them by a large margin.
[Figure] [Figure]
[Table]
A. F URTHER DETAILS ON ABLATION STUDY
We provide ablation study, taking PAM at Setting 1 as an example, in Table 7.
In the setup of ‘W/o sensor level concatenation’, we take the average of all sensor embeddings (in stead of concatenating them together) to obtain sample embedding.
Experimental results show that the full R AINDROP model achieves the best performance
[Table]
indicating every component or designed structure is useful to the model.
For example, we ﬁnd that excluding inter-sensor attention weights α ti,uv will cause a decrease of 3.9% in accuracy while excluding edge weights e i,uv (i
[Table]
e., dependency graphs) will drop the accuracy by 7.1%.
A. V ISUALIZATION OF INTER - SENSOR DEPENDENCY GRAPHS LEARNED BY R AINDROP
We visualize the learned inter-sensor dependencies ( i.e. , e i,uv before the averaging operation in Eq. 3) on P19 in early sepsis prediction.
The visualizations are implemented with Cytoscape (Shannon et al., 2003).
The data shown are for testing set of P19 including 3,881 samples (3708 negative and 173 positive).
As R AINDROP learns the speciﬁc graph for each sample, we take average of all positive samples and visualize it in Figure 4b; and visualize the average of all negative samples in
Figure 4b.
As we take average, the edges with weights smaller than 0.1 (means they rarely appear in graphs) are ignored.
The averaged edge weights range from 0.1 to 1.
We initialize all sample graphs as complete graph that has 1,156 = 34 × 34 edges, then prune out 50% of them in training phase, remaining 578 edges.
The 34 nodes in ﬁgures denote 34 sensors measured in P19, as listed https://physionet.org/content/challenge-2019/1.0.0/.
We list the sensor names here: 0: HR; 1: O2Sat; 2: Temp; 3: SBP; 4: MAP; 5: DBP; 6: Resp; 7: EtCO2; 8: BaseExcess; 9: HCO3; 10: FiO2; 11: pH; 12: PaCO2; 13: SaO2; 14: AST; 15: BUN; 16: Alkalinephos; 17: Calcium; 18: Chloride; 19:
Creatinine; 20: Bilirubin_direct; 21: Glucose; 22: Lactate; 23: Magnesium; 24: Phosphate; 25:
Potassium; 26: Bilirubin_total; 27: TroponinI; 28: Hct; 29: Hgb; 30: PTT; 31: WBC; 32: Fibrinogen; 33: Platelets.
We also visualize the differential inter-sensor connections between the learned dependency graphs from patients who are likely to have sepsis and the graphs from patients who are unlikely to suffer from sepsis.
Based on the aggregated graph structures of positive and negative samples, we calculate the divergence between two groups of patients and report the results in Figure 5.
In detail, we sort edges by the absolute difference of edge weights across negative and positive samples.
On top of the visualization of the 50 most distinctive edges, we can have a series of concrete insights.
For example, the dependency between node 6 (Respiration rate) to node 25 (Potassium) is important to the early prediction of sepsis.
Note these data-driven observations could be biased and still need conﬁrmation and future analysis from healthcare professionals.
The edges in both Figure 4 and
Figure 5 are directed.
The edge arrows might be difﬁcult to recognize due to the small ﬁgure size.
We will provide high-resolution ﬁgures to our public repository.
Furthermore, we statistically measure the similarities across samples within the same class and dissimilarities across samples from different classes.
Speciﬁcally, for every sample, we calculate: 1) the average Euclidean distance between its dependency graph and the dependency graphs of all samples from the same class; 2) the average distance with all samples from the different classes.
The
P19 dataset has 38,803 samples including 1,623 positive samples and 37,180 negative samples.
For a fair comparison, we randomly select 1,623 samples from the negative cohort, then mixed them with an equal number of positive samples to measure the averaged Euclidean distances intra- and inter-classes.
We select the cohort for 5 independent times with replacement.
We ﬁnd that the distance ( (8 . 6 ± 1 . 7) × 10 − 5 ) among dependency graphs of positive samples is smaller than the distance ( (12 . 9 ± 3 . 1) × 10 − 5 ) across samples.
The results show that the learned dependency graphs are similar within the same class and dissimilar across classes, which demonstrates R AINDROP can learn label-sensitive dependency graphs.