G RAPH -G UIDED N ETWORK FOR I RREGULARLY S AMPLED M ULTIVARIATE T IME
S
ERIES
Anonymous authors
Paper under double-blind review
A BSTRACT
In many domains, including healthcare, biology, and climate science, time series are irregularly sampled with variable time between successive observations and different subsets of variables (sensors) are observed at different time points, even after alignment to start events.
These data create multiple challenges for prevailing models that assume fully observed and ﬁxed-length feature representations.
To address these challenges, it is essential to understand the relationships between sensors and how they evolve over time.
Here, we introduce R AINDROP , a graphguided network for learning representations of irregularly sampled multivariate time series.
R AINDROP represents every sample as a graph, where nodes indicate sensors and edges represent dependencies between them.
R AINDROP models dependencies between sensors using neural message passing and temporal selfattention.
It considers both inter-sensor relationships shared across samples and those unique to each sample that can vary with time, and it adaptively estimates misaligned observations based on nearby observations.
We use R AINDROP to classify time series and interpret temporal dynamics of three healthcare and human activity datasets.
R AINDROP outperforms state-of-the-art methods by up to 11.4% (absolute points in F1 score), including methods that deal with irregular sampling using ﬁxed discretization and set functions, and even in challenging leave-sensorout settings and setups that require generalizing to new patient groups.
I NTRODUCTION
Multivariate time series are prevalent in a variety of domains including healthcare, space science, cybersecurity, biology, and ﬁnance (Ravuri et al., 2021;
Sousa et al., 2020; Sezer et al., 2020;
Fawaz et al., 2019; Abanda et al., 2019; Tang et al., 2018).
Practical issues often exist in collecting sensor measurements that lead to various types of irregularities caused by missing observations, such as cost saving, sensor failures, external forces in physical scenarios, medical interventions, to name a few (Choi et al., 2020).
While temporal machine learning models usually assume fully observable and ﬁxed-size input data, irregularly sampled time series raise considerable challenges.
For example, the observations of multiple sensors are not well-aligned; the time intervals among adjacent observations are different across sensors; and different samples have different numbers of observations for different subsets of sensors recorded at different time points.
Prior methods for dealing with irregularly sampled time series involve ﬁlling in missing values, using interpolation, kernel methods, and probabilistic approaches (Schafer & Graham, 2002).
The missingness of observations carry informative power (Little & Rubin, 2014) and thus imputation of missing data is not always beneﬁcial (Agniel et al., 2018).
While modern techniques involve recurrent neural network architectures ( e.g. , RNN, LSTM, GRU) (Hochreiter & Schmidhuber, 1997;
Cho et al., 2014) and transformers (Vaswani et al., 2017), they are restricted to regular sampling or assume aligned measurements across modalities.
For misaligned measurements, existing methods tend to rely on a two-stage process of imputation to get a regularly-sampled version of a dataset, and then performing a downstream task such as classiﬁcation.
This decoupled approach might not fully exploit informative missingness patterns that could be essential for the downstream task thus achieving suboptimal performance (Wells et al., 2013; Li & Marlin, 2016).
Thus, several approaches circumvent imputation and directly model irregularly sampled time series data (Che et al., 2018; Horn et al., 2020; Shukla & Marlin, 2021; 2018).
However, few explicitly consider relational structure to address the characteristics of irregularly sampled multivariate time series and none leverages graph neural networks.
[Figure]
Present work.
To address the characteristics of irregularly sampled time series data, we propose to model temporal dynamics of sensor dependencies and how those relationships evolve over time. Previous studies (Wu et al., 2021;
Li et al., 2020a; Zhang et al., 2019) have noted that the inter-sensor correlations bring rich information in modeling time series.
Our intuitive assumption is that the observed sensors can indicate how the unobserved sensors currently behave, which can further improve the representation learning of irregular multivariate time series.
We develop R AINDROP 1 , a graph-guided network that leverages relational structure to embed and classify irregularly sampled multivariate time series.
R AINDROP takes samples as input, each sample containing multiple sensors and each sensor consisting of irregularly recorded observations ( e.g. , in clinical data, an individual patient’s state of health, recorded at irregular time intervals with different subsets of sensors observed at different times).
R AINDROP model is inspired by the idea of raindrops falling into a pool at sequential but non-uniform time intervals and thereby creating ripple effects that propagate across the pool.
Mathematically, in R AINDROP , observations ( i.e. , raindrops) hit a sensor graph ( i.e. , pool) asynchronously and at irregular time intervals, and each observation is processed by passing messages to neighboring sensors ( i.e. , causes a ripple effect in the pool) taking into account the learned sensor dependencies (Figure 1).
As such, R AINDROP can handle misaligned observations, varying time gaps, arbitrary numbers of observations, and produce embeddings via a novel hierarchical attention.
R AINDROP is the ﬁrst to explicitly model sensor dependencies in learning representations of irregularly sampled time series.
We represent dependencies with a separate sensor graph for every sample, wherein nodes indicate sensors and edges denote relationships between them.
In addition to capturing distinct sensor dependencies within each sample, R AINDROP i) takes advantage of similarities between different samples via sharing of parameters in the calculation of attention weights, and ii) considers importance of successive observations via temporal attention.
R AINDROP adaptively estimates missing observations based on recorded information and the learned graph structure.
We compare R AINDROP to ﬁve state-of-the-art methods across three datasets and four experimental settings, including a setup where a subset of sensors in the test set have malfunctioned ( i.e. , have no readouts at all).
Experiments show that R AINDROP outperforms baselines on all datasets with an average AUROC improvement of 3.5% in absolute points on classiﬁcation tasks.
Further, R AINDROP achieves a considerable margin (9.3% absolute points in accuracy on activity recognition) when a subset of sensors malfunction.
R ELATED W ORK
Learning with irregularly sampled multivariate time series.
Irregular time series indicate that the time intervals between adjacent observations are varying (Zerveas et al., 2021;
Tipirneni & Reddy, 2021; Chen et al., 2020).
In the multivariate case, irregularity means that observations are misaligned across different sensors.
Further, because of a multitude of sampling frequencies and varying time intervals, the number of observations can vary considerably across samples (Fang & Wang, 2020;
Kidger et al., 2020).
Predominant downstream tasks for time series are classiﬁcation ( i.e. , predicting a label for a given sample, e.g. , Tan et al.
; Ma et al.
and forecasting ( i.e. , anticipating future observations based on historical observations, e.g. , Wu et al.
(2020a)).
The above mentioned data characteristics create considerable challenges for machine learning models that expect wellaligned and ﬁxed-size input (Shukla & Marlin, 2020).
An intuitive way to deal with irregular time series is to impute missing values and process them as a regular time series dataset (Mikalsen et al., 2021; Li & Marlin, 2020; Shan & Oliva, 2021).
However, imputation methods can distort underlying distributions and introduce unwanted bias in the dataset.
To this end, recent methods directly learn from irregularly sampled time series.
For example, Che et al.
(2018) propose a decay mechanism based on gated recurrent units ( GRU-D ), incorporating binary masking indicators and time intervals to capture long-term temporal dependencies.
SeFT (Horn et al., 2020) re-formulates irregularly sampled time series into a set of observations and learn representations through set functions that are insensitive to alignment.
Chen et al.
(2018) model continuous-time hidden dynamics by latent ordinary differential equations ( Latent-ODE ), combined with an RNN for temporal representations.
mTAND (Shukla & Marlin, 2021) leverages multi-time attention mechanism to learn temporal similarity from non-uniformly collected measurements and produces continuous-time embeddings. IP-Net (Shukla & Marlin, 2018) and DGM 2 (Wu et al., 2021) adopt similar imputation paradigms to interpolate the irregular time series against a set of reference points using a kernel-based method. The learned inter-sensor relations are static without considering sample-speciﬁc and time-speciﬁc characteristics.
In contrast with these methods, R AINDROP leverages dynamic graphs to address the characteristics of irregular time series and improve the quality of learned representations.
Learning with graphs and neural message passing.
There has been a surge of interest in applying neural networks to graphs, leading to the development of graph embeddings (Zhou et al., 2020; Li et al., 2021), graph neural networks (Wu et al., 2020b), and message passing neural networks (Gilmer et al., 2017).
Our approach is mainly related to methods that perform message passing along edges to update node representations through neural transformations (Riba et al., 2018; Nikolentzos et al., 2020; Galkin et al., 2020; Fey et al., 2020; Lin et al., 2018; Zhang et al., 2020).
However, in contrast to message passing used to make predictions on graphs, we are interested in leveraging it to meet the challenges of irregularly sampled time series.
In particular, we consider message passing where nodes are sensors, describing a particular sample ( e.g. , patient, Figure 1), and we design a message-passing network with learnable adjacency matrices.
The key difference with the predominant use of message passing is that R AINDROP uses it to estimate edges (dependencies) between sensors rather than applying it on a ﬁxed, apriori-given graph.
To the best of our knowledge, no prior work utilized sensor dependencies for learning representations of irregularly sampled multivariate time series.
Finally, while prior work used message passing for regular time series (Wang et al., 2020; Wu et al., 2020c;
Kalinicheva et al., 2020), no modeling of irregularly sampled time series has been attempted.
R AINDROP
[Figure]
[Figure] ) ( Section 3.4 ) (Section [Figure] [Table]
Let D =
{ ( S i , y i ) | i = 1 , . . .
, N } denote an irregular time series dataset with N labeled samples (Figure 2).
Every sample S i is an irregular multivariate time series with a corresponding label y i ∈ { 1 , . . .
, C } , indicating which of the C classes S
i is associated with.
Each sample contains M non-uniformly measured sensors that are denoted as u , v , etc.
R AINDROP can also work on samples with a subset of sensors (see Sec. 4.1).
Each sensor is given by a sequence of observations ordered by time.
For sensor u in sample S i , we denote a single observation as a tuple ( t, x ti,u ) , meaning that sensor u was recorded with value x ti,u ∈ R at timestamp t ∈ R + .
We omit sample index i and sensor index u in timestamp t .
Sensor observations are irregularly recorded, meaning that time intervals between successive observations can vary across sensors.
For sensor u in sample S i , we use T i,u to denote the set of timestamps that u , or at least one of u ’s L -hop neighbors ( L is the number of layers in R AINDROP ’s message passing) is recorded.
We use || and T to denote concatenation and transpose, respectively.
We omit layer index l ∈ { 1 , . . .
, L } for simplicity when text is clear.
Problem (Representation learning for irregularly sampled multivariate time series).
A dataset D of irregularly sampled multivariate time series is given, where each sample S i has multiple sensors and each sensor has a variable number of observations.
R AINDROP learns a function f : S i
→ z i that maps S i to a ﬁxed-length representation z i suitable for downstream task of interest, such as classiﬁcation.
Using learned z i , R AINDROP can predict label ˆ y i ∈ { 1 , . .
, C } for S i .
R AINDROP learns informative embeddings for irregularly samples time series.
The learned embeddings capture temporal patterns of irregular observations and explicitly consider varying dependencies between sensors.
While we focus on time-series classiﬁcation in this work, the proposed method can be easily extended to broader applications such as regression, clustering and generation tasks.
3. O VERVIEW OF R AINDROP
R AINDROP aims to learn a ﬁxed-dimensional embedding z i for given S i , and then predict the associated label ˆ y i .
R AINDROP learns sample embeddings in a hierarchical architecture composed of observation embedding, sensor embedding and sample embedding (Figure 2).
We describe the procedure of R AINDROP in the context of at most one observation at a speciﬁc time t (one sensor has observation while all residual sensors do not have observation).
If there are multiple observations at the same time, R AINDROP can effortlessly process them all in parallel.
We ﬁrst construct a graph for each sample where nodes represent sensors and edges indicate relations between sensors (Sec. 3.2).
Each sample S i corresponds to a graph G i , where we use e i,uv to denote the weight of directed edge going from sensor u to sensor v .
The graph structures are automatically learned while considering sample-wise and time-wise speciﬁcity.
The key idea of R AINDROP is to borrow information from u ’s neighbors based on estimated relationships between u and other sensors.
This is achieved via message passing carried out on S
i ’s dependency graph and initiated at node u in the graph.
When an observation ( t, x ti,u ) is recorded for sample S i at time t , R AINDROP ﬁrst embeds the observation at active sensor u ( i.e. , sensor whose value was recorded) and then propagates messages ( i.e. , the observation embeddings) from u to neighboring sensors along edges in sensor dependency graph G i .
As a result, recording the value of u can affect u ’s embedding as well as embeddings of other sensors that related to u (Sec. 3.3).
Finally, R AINDROP generates sensor embeddings by aggregating all observation embeddings for each sensor (across all timestamps) using temporal attention weights (Sec. 3.4).
At last, R AINDROP embeds every sample S i based on the embeddings of all sensors (Sec. 3.5) and feeds the learned sample embedding into downstream classiﬁer.
3. C ONSTRUCTING S ENSOR D EPENDENCY G RAPHS
We build a directed weighted graph G i
=
{V , E i } for every sample S i and refer to it as the sensor dependency graph for S i .
Nodes V represent sensors and edges E i describe dependencies between sensors in sample S i that R AINDROP infers.
As we show in experiments, R AINDROP can be directly used with samples that only contain a subset of sensors in V .
We denote edge from u to v as a triplet ( u, e i,uv , v ) , where e i,uv ∈ [0 , 1] represents the strength of relationship between sensors u and v in sample S
i .
Edge ( u, e i,uv , v ) describes the relationship between u and v : when u receives an observation, it will send a neural message to v following edge e i,uv .
If e i,uv = 0 , there is no exchange of neural information between u and v , indicating that the two sensors are unrelated.
We assume that the importance of u to v is different than the importance of v to u , and so we treat sensor dependency graphs as directed, i.e. , e i,uv (cid:54) =
e i,vu .
All graphs are initialized as fully-connected graphs ( i.e. , e i,uv = 1 for any u , v and S i ) and edge weights e i,uv are updated following Eq.
3 during model training.
If available, it is easy to integrate additional domain knowledge into graph initialization.
3. G ENERATING E MBEDDINGS OF I NDIVIDUAL O BSERVATIONS
Let sensor u be activated at time t ∈ T i,u and receive observation x ti,u and let u be connected to v through edge ( u, e i,uv , v ) .
We next describe how to produce observation embeddings h ti,u ∈ R d h and h ti,v ∈ R d h for sensors u and v , respectively (Figure 3a).
We omit layer index l and note that the proposed strategy applies to any number of layers.
Embedding observation of an active sensor.
We treat u as an active sensor that has received observation x ti,u .
For sufﬁcient expressive power (Veliˇckovi´c et al., 2018), we map observation x ti,u to a high-dimensional space using a nonlinear transformation: h ti,u
= σ ( x ti,u R u ) .
Here, we use sensorspeciﬁc transformations because values recorded at different sensors can follow different distributions, which is achieved by trainable weight vectors R u depending on what sensor is activated (Li et al., 2020b).
Alternative functions (such as a multilayer perceptron) can be considered for transforming
[Figure] x ti,u into h ti,u . As h ti,u represents information brought on by measuring x ti,u , we regard h ti,u as u ’s observation embedding at t .
The sensor-speciﬁc weight vectors R u are shared across all samples
Message passing in sensor dependency graphs.
For sensors that are not active at timestamp t but are neighbors of the active sensor u in the sensor dependency graph G i , R AINDROP uses relationships between u and those sensors to estimate observation embeddings for them.
We proceed by describing how R AINDROP generates observation embedding h ti,v for sensor v assuming v is a neighbor of u in
G i .
Given h ti,u and edge ( u, e i,uv , v ) , we ﬁrst calculate inter-sensor attention weight α ti,uv ∈ [0 , 1] , representing how important u is to v via the following equation:
[Equation]
where r v ∈ R d r is a trainable weight vector that is speciﬁc to the sensor receiving the message ( i.e. , h ti,u ).
Vector r v allows the model to learn distinct attention weights for different edges going out from the same sensor u .
Further, p ti ∈ R d t is the time representation obtained by converting a 1-dimensional timestamp t into a multi-dimensional vector p ti by passing t through a series of trigonometric functions (Horn et al., 2020).
See Appendix A.1 for details.
R AINDROP uses p ti to calculate attention weights that are sensitive to time.
Finally, D is a trainable weight matrix mapping
h ti,u from d h dimensions to ( d r + d t ) dimensions.
Taken this together, we can estimate the embedding h ti,v for u ’s neighbor v as follows:
[Equation]
where w u , w v ∈ R
d
h are trainable weight vectors shared across all samples. The w u is speciﬁc to active sensor u and w v is speciﬁc to neighboring sensor v .
In the above equation, e i,uv denotes edge weight shared across all timestamps.
The above message passing describes the processing of a single observation at a single timestamp.
In case multiple sensors are active at time t and connected with v , we normalize α ti,uv (with softmax function) across active sensors and aggregate messages at v .
Overall, R AINDROP produces observation embedding h ti,v for sensor v through its relational connection with u , even though there is no direct measurement of v at time t .
These message passing operations are performed to adaptively and dynamically estimate missing observations in the embedding space based on recorded information and learned graph structure.
Updating sensor dependency graphs.
We describe the update of edge weights and prune of graph structures in the situation that stacks multiple R AINDROP layers (Figure 3).
Here we explicitly show layer index l because multiple layers are involved in the computation.
As no prior knowledge is assumed, we initialize the graph as all sensors connected with each other.
However, the fully connected edges may bridge sensors that should be independent, which will introduce spurious correlations and prevent the model from paying attention to the truly important connections.
Addressing this issue,
R AINDROP automatically updates edge weights and prunes out less important edges.
Based on the aggregated temporal inﬂuence driven by the inter-sensor attention weights α ( l ) ,t i,uv , we update edge weights e ( l ) i,uv in each layer l ∈ { 1 , . . .
, L } by:
[Equation]
where T i,u denotes the set of all timestamps where there is message passes from u to v .
In particular, we set e (0) i,uv = 1 in the initialization of graph structures.
We use L = 2 in all our experiments.
In every layer, we order the estimated values e ( l ) i,uv for all edges in sample S
i and prune bottom K % edges with smallest edge weights (Yang et al., 2021).
Pruned edges will not re-appear in later layers.
3. G ENERATING S ENSOR E MBEDDINGS
Next we describe how to aggregate observation embeddings into sensor embeddings, taking sensor v as an example (Figure 3b).
Previous step outlined in Sec.
3.3 outputs observation embeddings for every timestamp when either v or v ’s neighbor in S
i ’s sensor dependency graph are observed.
The following step aggregates those embeddings into a sensor embedding z
i,v via temporal attention.
We concatenate observation embedding h ti,v with time representation p ti to include information of timestamp.
Then, we stack the concatenated embeddings [ h ti,v || p ti ] for all t ∈ T i,v into a matrix H i,v .
The T i,v = { t 1 , t 2 , . . .
, t T } includes all timestamps when a readout is observed in v (we can directly generate h ti,v ) or v ’s neighbor (we can generate h ti,v through message passing).
We calculate the temporal attention weight β ti,v using self-attention (Hu et al., 2020;
Yun et al., 2019; Vaswani et al., 2017).
The β ti,v represents the importance of observation embedding to the whole sensor embedding.
The β ti,v is the corresponding element of vector β i,v that is calculated through:
[Equation]
where query matrix Q i,v = H i,v W Q and
key matrix
K i,v = H i,v W K are two matrices linearly mapped from H i,v parameterized by W Q and W K , respectively.
The √ d k is a scaling factor where d k is the dimension after linear mapping.
Our calculation of β i,v is different from the typical selfattention by having mapping weight vector s .
The standard dot-product self-attention generates an attention matrix with dimension of T × T (where T = |T
i,v | can vary across samples) that has an attention weight for each pair of observation embeddings.
In our case, we only need a single attention vector where each element denotes the attention we should pay to an observation embedding when generating the sensor embedding.
Thus, we modify the self-attention model to ﬁt our case and use a trainable s ∈ R T × 1 to map the results of dot-product from matrix ( R T × T ) to T -dimensional vector β i,v ( R T × 1 ) through matrix product (Appendix A.2).
We calculate sensor embedding z i,v through:
[Equation]
where weight matrix W is a linear projector shared by all sensors and samples.
It’s worth to mention that all attention weights (such as α ti,uv and β i,v ) can be multi-head.
In this work, we describe the model using a single head for brevity.
Using attentional aggregation, R AINDROP can learn a ﬁxed-length sensor embedding for arbitrary number of observations.
Meanwhile, R AINDROP is capable of focusing on the most informative observation embeddings.
We process all observation embeddings as a whole instead of sequentially, which allows parallel computation for faster training and also mitigates the performance drop caused by modeling long dependencies sequentially.
In the case of sensors with very large number of observations, we can reduce the length of time series by subsampling or splitting a long series into multiple short series.
3. G ENERATING S AMPLE E MBEDDINGS
Finally, we aggregate sensor embeddings z i,v (Eq. 5) to obtain an embedding z i ∈ R d z for sample S i , as follows:
z i =
[ z i, 1 ||
z i, 2 || · · · || z i,M ] across all sensors v = 1 , 2 , . . .
, M .
Our preliminary experiments show that concatenation outperforms other popular aggregation functions such as averaging (Errica et al., 2021) and squeeze-excitation readout function (Kim et al., 2021;
Hu et al., 2018).
While any of those aggregation functions can be considered, we used concatenation throughout all experiments in this manuscript (Appendix A.3).
Taken together Sec. 3.2-3.5, given an input of sample S i , R AINDROP returns a sample embedding z
i
that can in turn be fed into a downstream task.
3. I MPLEMENTATION AND P RACTICAL C ONSIDERATIONS
Loss function.
The R AINDROP ’s loss function is formulated as:
L =
L CE + λ L r , where L r =
M 2 (cid:80) u,v ∈V (cid:80) i,j ∈V || e i,uv − e j,uv
|| 2 / ( N − 1) 2 , where L CE is a cross-entropy classiﬁcation loss, and L r is a regularizer that encourages the model to learn similar sensor dependency graphs for similar samples.
The L r measures averaged Euclidean distance of edge weights across all samples pairs, in all sensor pairs (including self-connections). The λ is a user-deﬁned coefﬁcient.
Practically, as N can be large, we calculate L r only for samples in each batch.
Downstream tasks.
If a sample has auxiliary attributes ( e.g. , a patient’s demographics) that do not change over time, we can project the attribute vector to a d a -dimensional vector a i with a fullyconnected layer and concatenate it with the sample embedding, getting [ z i || a i ] .
At last, we feed [ z i
|| a
i ] (or only z i if a i is not available) into a neural classiﬁer ϕ :
R d z + d a → { 1 , . . .
, C } .
In our experiments, ϕ is a 2-layer fully-connected network with C neurons at the output layer returning prediction ˆ y i = ϕ
([ z i || a i ]) for sample S i .
Sensor dependencies.
While modeling sensor dependencies, we involve observation embedding ( h ti,u , Eq. 1) of each sample in the calculation of attention weights.
Similarly, to model time-wise speciﬁcity in graph structures, we consider time information ( p ti , Eq. 1) when measuring α ti,uv .
R AINDROP can capture similar graph structures across samples from three aspects (Appendix A.4): (1) the initial graphs are the same in all samples; (2) the parameters in message passing ( R u ; w u , w v , Eq. 2), inter-sensor attention weights calculation ( D , Eq. 1), and temporal attention weights calculation ( s , Eq. 4; W , Eq. 5) are shared by all samples; (3) we encourage the model to learn similar graph structures by adding a penalty to disparity of structures ( L r ).
Scalability.
R AINDROP is efﬁcient because embeddings can be learned in parallel.
In particular, processing of observation embeddings is independent across timestamps.
Similarly, sensor embeddings can be processed independently across different sensors (Figure 3).
While the complexity of temporal self-attention calculation grows quadratically with the number of observations, it can be practically implemented using highly-optimized matrix multiplication.
E XPERIMENTS
Datasets.
Below we brieﬂy overview healthcare and human activity datasets.
(1) P19 (Reyna et al., 2020) includes 38,803 patients that are monitored by 34 sensors.
Each patient is associated with a binary label representing the occurrence of sepsis.
(2) P12 (Goldberger et al., 2000) records temporal measurements of 36 sensors of 11,988 patients in the ﬁrst 48-hour stay in ICU.
The samples are labeled based on hospitalization length.
(3) PAM (Reiss & Stricker, 2012) contains 5,333 segments from 8 activities of daily living that are measured by 17 sensors.
Details are in Appendix A.5.
Baselines.
We compare R AINDROP with ﬁve state-of-the-art time series classiﬁcation methods:
Transformer (Vaswani et al., 2017), Trans-mean , GRU-D (Che et al., 2018), SeFT (Horn et al., 2020), and mTAND (Shukla & Marlin, 2021).
The Trans-mean is an imputation method combining transformer architecture with commonly used average interpolation ( i.e. , missing values are replaced by average observations in each sensor).
Further details are in Section 2.
The mTAND (Shukla &
Marlin, 2021) method has been shown to outperform numerous recurrent models including RNNImpute (Che et al., 2018), RNN-Simple , Phased-LSTM (Neil et al., 2016), and IP-Nets (Shukla &
Marlin, 2018), along with ordinary differential equations (ODE)-based models such as LATENT-ODE and ODE-RNN (Chen et al., 2018).
For this reason, we compare with mTAND and do not report comparison with those techniques in this paper.
Details on hyperparameter selection of R AINDROP and baselines are in Appendix A.6.
Chosen evaluation metrics are presented in Appendix A.7.
4. R ESULTS ACROSS A M ULTITUDE OF E VALUATION S ETTINGS
Setting 1: Classic time series classiﬁcation.
(1) Setup.
Setup. Setup.
Setup. Setup.
Setup. Setup.
Setup. Setup.
Setup. Setup.
Setup. Setup.
Setup. Setup.
Setup. Setup.
We randomly split the dataset into training (80%), validation (10%), and test (10%) set.
The indices of these splits are ﬁxed across all methods.
(2) Results.
Results. Results.
Results. Results.
Results. Results.
Results. Results.
Results. Results.
Results. Results.
Results. Results.
Results. Results.
As shown in Table 1, R AINDROP obtains the best performance across three benchmark datasets, implying our superiority in standard time series classiﬁcation scenario.
In particular, in binary classiﬁcation (P19 and P12), R AINDROP outperforms the strongest baselines by 5.3% in
AUROC and 4.8% in AUPRC on average.
In a much more challenging task of 8-class classiﬁcation,
R AINDROP also outperforms existing approaches by 5.7% in accuracy and 5.5% in F1 score on PAM.
More experiments on exploration and comparison are shown in Appendix A.9-A.10.
Comparison with extensive baselines is in Appendix A.11.
[Table]
We evaluate whether R AINDROP can learn dependencies among sensors, thus achieve good performance even some sensors are completely missing.
This setting is practically meaningful, such as facing sensor failure.
or some sensors are unavailable in speciﬁc scenes.
Our intuition is that R AINDROP can compensate for the missing information from nearby observations by exploiting relational dependencies.
In this setting, we select a proportion of sensors, and set all their observations as zero in validation and test set (training samples are not changed).
We mask out the most informative sensors which are selected through information gain (Appendix A.8).
The selected sensors are ﬁxed across samples and models.
(2) Results.
Results. Results.
Results. Results.
Results. Results.
Results. Results.
Results. Results.
Results. Results. Results.
Results. Results.
Results.
We report results taking PAM as an example.
In Table 2 (left columns), we observe that R AINDROP achieves the highest values in 18 out of 20 settings while the missing data ratio ranges from 10% to 50%.
The missing data ratio refers to the proportion of sensors we mask out.
With the increase of missing data ratio, our model has greater relative gains.
R AINDROP outperforms competitive baselines by up to 24.9% in accuracy, 50.3% in precision, 29.3% in recall, and 42.8% in F1 score.
Setting : Leave-random-sensors-out.
Setup.
Setup. Setup.
Setup. Setup.
Setup. Setup.
Setup. Setup.
Setup. Setup.
Setup. Setup.
Setup. Setup.
Setup. Setup.
Setting 3 is similar to Setting 2 except that the missing sensors in this setting are randomly selected instead of ﬁxed.
In each test sample, we select a subset of sensors and regard them as missing through replacing all of their observations with zeros.
(2) Results.
Results. Results.
Results. Results.
Results. Results.
Results. Results.
Results. Results.
Results. Results.
Results. Results.
Results. Results.
We provide results of PAM in Table 2 (right columns).
Similar to Setting 2, we observe that R AINDROP achieves better performance than baselines in 16 out of 20 settings while Trans-mean and GRU-D are competitive methods.
Setting 4: Group-wise time series classiﬁcation.
(1) Setup.
Setup. Setup.
Setup. Setup.
Setup. Setup.
Setup. Setup.
Setup. Setup.
Setup. Setup.
Setup. Setup.
Setup. Setup.
To understand whether R AINDROP can adaptively adjust its structure and generalize well to other groups of samples which were not observed while training the model.
In this setting we split the data into two groups, based on a speciﬁc static attribute.
The ﬁrst split attribute is age , where we classify people into young ( < 65 years) and old ( ≥ 65 years) groups.
We also split patients into male and female by gender attribute.
Given the split attribute, we use one group as a train set and randomly split the other group into equally sized validation and test set.
(2) Results.
Results. Results.
Results. Results.
Results. Results.
Results. Results.
Results. Results.
Results. Results.
Results. Results.
Results. Results.
Taking P19 as an example, we present the classiﬁcation results when the training and testing samples are from different groups.
As shown in Table 3, R AINDROP achieves the best results over all of the four given cross-group scenarios (Appendix A.13).
For instance, R AINDROP claims large margins (with 4.8% in AUROC and 13.1% in AUPRC absolute improvement) over the second best model while training on males and testing on female patients. [Table] [Table] [Table]
A BLATION S TUDY AND V ISUALIZATION OF L EARNED G RAPH S TRUCTURE
Ablation study.
Taking PAM in the most common task (Setting 1) as an example, we conduct an ablation study to evaluate the necessity of three fundamental steps of R AINDROP : inter-sensor dependency (further decomposed to key weights including e i,uv , r v , p ti , and α ti,uv ), temporal attention, and sensor-level concatenation.
In Appendix A.14 (Table 9), we observe that all of the components and the regularization term L r are important to improve model performance.
Visualizing learned sensor dependency graphs.
We investigate whether samples with the same label have similar graph structure by visualizing the learned inter-sensor dependencies (P19; Setting 1).
As shown in Figure 4, we can observe distinguishable patterns between the graph structures of negative and positive samples, indicating R AINDROP can adaptively learn relationships that are sensitive to the classiﬁcation task.
The differential analysis can also provide inspirations to domain experts such as intensivists.
More details in Appendix A.15.
C ONCLUSION
We develop R AINDROP , a graph-guided network for learning with irregularly sampled multivariate time series.
R AINDROP learns a distinct sensor dependency graph for every sample that captures time-varying dependencies between sensors in the sample.
The ability to leverage the relational structure gives R AINDROP unique ability to naturally handle misaligned observations, non-uniform time intervals between successive observations, and sensors with a varying number of recorded observations.
Our novel ﬁndings have implications for using message passing as a powerful approach to take advantage of relational structure in multivariate time series.
R EPRODUCIBILITY S TATEMENT
We ensure the reproducibility of our work by clearly presenting the model and provide publicly accessible code and data.
For all datasets used in this work, we share the downloadable links to the raw sources, and well-processed and ready-to-run datasets, with the research community through link https://anonymous.4open.science/r/Raindrop.
We specify all the training details (e.g., preprocessing, data splits, hyperparameters, sensor selection) in the main text and Appendix.
Python implementation of R AINDROP and all baseline methods is available at the aforementioned link.
Detailed description of all data and scripts, conﬁgurations, along with examples of usage, will also be provided.
E THICS S TATEMENT
The ability of R AINDROP model to learn powerful information about sensors’ representations and dependencies creates new opportunities for applications, where time series data is predominant, e.g. , in healthcare, biology, and ﬁnance.
In all these ﬁelds, especially in patient care, our method should be used with caution.
Although our model can gain useful insights from time series data, users must be aware of the restrictions in terms of machine-guided predictions.
The use of our model in real-life clinical settings should always be veriﬁed by qualiﬁed physicians.
As with all data-driven solutions, it is possible that our model makes biased predictions.
In the case of biomedical data, biases can exist within the data itself, which can be, for example, caused by considering static attributes, such as age, weight, and gender.
When the ratio of target labels is highly imbalanced, our model mitigates the possible bias by performing minority class upsampling for every processed batch.
All datasets in this paper are publicly available and are not associated with any privacy or security concerns.
Further, all data are anonymized to guard against breaching patient’s protected health information (PHI).
We have followed PhysioNet privacy policy and guidelines (https://archive.
physionet.org/privacy.shtml) when experimenting with P12 and P19 datasets.
A A PPENDIX A. T IMESTAMP ENCODING
For a given time value t , we pass it to trigonometric functions with the frequency of 10,000 (Vaswani et al., 2017) and generate time representation p t ∈ R ξ (omit sample index i for brevity) through (Horn et al., 2020):
[Equation]
where ξ is the expected dimension.
In this work, we set ξ = 16 in all experimental settings for all models.
Please note, we encode the time value which is a continuous timestamp, instead of time position which is a discrete integer indicating the order of observation in time series. A. I NTUITION OF VARIABLE s
Here we report the intuition and the meaning for the designed transformation variable s in Eq. 4.
The target of Eq. 4 is to learn the temporal attention weights vector β
i,v for sensor v , taking advantage of self-attention mechanism.
However, the standard self-attention generates an attention matrix but we ask for an attention vector.
In the attention matrix, each row represents the importance of the observation embedding at a certain time to the observation embeddings at all other time steps. Our intuition is to aggregate a row in the self-attention matrix into a scalar that denotes the importance of the observation embedding at a speciﬁc time step to the whole sensor embedding. We apply the weighted aggregation, parameterized by s , to every row in the attention matrix and concatenate the generated scalars into an attention vector. Next, we give a concrete example to speciﬁcally describe the meaning of s .
Each row, j , of the self-attention matrix captures relationships of observation embedding h t j i,v to all observation embeddings { h t k
i,v : k = 1 , ..., T } .
Then, using the learnable weight vector s , these correlations between observations are aggregated across time to obtain temporal importance weight β t j i,v .
The β t j i,v represents the importance of the corresponding observation to the whole sensor embedding.
A. A DDITIONAL INFORMATION ON SAMPLE EMBEDDING
As we generate sample embedding by concatenating all sensor embeddings, the sample embedding could be relatively long when there is a large number of sensors.
To alleviate this issue, on one hand, we can reduce the dimension of sample embeddings by adding a neural layer (such as a simple fully-connected layer) after the concatenation.
On the other hand, when the number of sensors is super large, our model is ﬂexible and can effortlessly switch the concatenation to other readout functions (such as averaging aggregation): this will naturally solve the problem of long vectors.
We empirically show that concatenation works better than averaging in our case.
We see a boost in the AUROC score by 0.6% using concatenation instead of averaging for generating sample embeddings(P19; Setting 1).
A. A DDITIONAL INFORMATION ON SAMPLE SIMILARITIES
In this work, we assume all samples share some common characteristics to some extent.
When modeling the similarities across samples, we do not consider the situation where the samples are similar within latent groups and different across groups.
Our study focuses on the question of irregularity rather than the question of distribution shifts in time series.
To this end, in our experiments, we ﬁrst rigorously benchmark Raindrop using a standard evaluating setup (Setting 1, which is classiﬁcation of irregular time series).
This is the only setup that most existing methods consider (e.g., Shukla & Marlin (2021);
Che et al.
and we want to make sure our comparisons are fair.
In order to provide a more rigorous assessment of Raindrop’s performance, we also consider more challenging setups in our experiments (i.e., Settings 2-4) when the dataset is evaluated in a non-standard manner and the split is informed by a select data attribute.
Our results on Setting 1 are consistent with those on Settings 2-4.
Results on harder Settings 2-4 show that Raindrop can perform comparably better than baselines.
Results across these diverse settings increase our conﬁdence that Raindrop is quite ﬂexible and widely applicable.
[Table]
A. F URTHER DETAILS ON DATASETS
P19: PhysioNet Sepsis Early Prediction Challenge 2019.
P19 dataset (Reyna et al., 2020) contains 38,803 patients and each patient is monitored by 34 irregularly sampled sensors including 8 vital signs and 26 laboratory values.
The original dataset has 40,336 patients, we remove the samples with too short or too long time series, remaining 38,803 patients (the longest time series of the patient has more than one and less than 60 observations).
Each patient is associated with a static vector indicating attributes: age, gender, time between hospital admission and ICU admission, ICU type, and ICU length of stay (days).
Each patient has a binary label representing occurrence of sepsis within the next 6 hours.
The dataset is highly imbalanced with only ∼ 4% positive samples.
P12: PhysioNet Mortality Prediction Challenge 2012.
P12 dataset (Goldberger et al., 2000) includes 11,988 patients (samples), after removing 12 inappropriate samples following (Horn et al., 2020).
Each patient contains multivariate time series with 36 sensors (excluding weight), which are collected in the ﬁrst 48-hour stay in ICU.
Each sample has a static vector with 9 elements including age, gender, etc.
Each patient is associated with a binary label indicating length of stay in ICU, where negative label means hospitalization is not longer than 3 days and positive label marks hospitalization is longer than 3 days.
P12 is imbalanced with ∼ 93% positive samples.
PAM: PAMAP2 Physical Activity Monitoring.
PAM dataset (Reiss & Stricker, 2012) measures daily living activities of 9 subjects with 3 inertial measurement units.
We modify it to suit our scenario of irregular time series classiﬁcation.
We excluded the ninth subject due to short length of sensor readouts.
We segment the continuous signals into samples with the time window ofand the overlapping rate of 50%.
PAM originally has 18 activities of daily life.
We exclude the ones associated with less than 500 samples, remaining 8 activities.
After modiﬁcation, PAM dataset contains 5,333 segments (samples) of sensory signals.
Each sample is measured by 17 sensors and contains 600 continuous observations with the sampling frequency 100 Hz.
To make time series irregular, we randomly remove 60% of observations.
To keep fair comparison, the removed observations are randomly selected but kept the same for all experimental settings and approaches.
PAM is labelled by 8 classes where each class represents an activity of daily living.
PAM does not include static attributes and the samples are approximately balanced across all 8 categories.
To feed given data into neural networks, we set the input as zero if no value was measured.
In highly imbalanced datasets (P19 and P12) we perform batch minority class upsampling, which means that every processed batch has the same number of positive and negative class samples.
The dataset statistics including sparse ratio are provided in Table 4.
A. F URTHER DETAILS ON MODEL HYPERPARAMETERS
Baseline hyperparameters.
The implementation of baselines follows the corresponding papers including SeFT (Horn et al., 2020), GRU-D (Che et al., 2018), and mTAND (Shukla & Marlin, 2021).
We follow the settings of Transformer baseline in (Horn et al., 2020) while implementing
Transformer in our work.
For average imputation in Trans-mean, we replace the missing values by the global mean value of observations in the sensor (Shukla & Marlin, 2020).
We use batch size of 128 and learning rate of 0.0001.
Note that we upsample the minority class in each batch to make the batch balance (64 positive samples and 64 negative samples in each batch).
The chosen hyperparameters are the same across datasets (P19, P12, PAM), models (both baselines and R AINDROP ), and experimental settings.
Remarkably, we found that all the baselines make dummy predictions (classify all testing samples as the majority label) on PAM in Setting 2-3 while
R AINDROP makes reasonable predictions.
For the comparison to make sense ( i.e. , the baselines can make meaningful predictions), we use learning rate of 0.001 for baselines on PAM.
GRU-D haslayers while other models have 2 layers.
We run all models for 20 epochs, store the parameters that obtain the highest AUROC in the validation set, and use it to make predictions for testing samples.
We use the Adam algorithm for gradient-based optimization (Kingma & Ba, 2014).
R AINDROP hyperparameters.
Next, we report the setting of unique hyperparameters in our R AINDROP .
In the generation of observation embedding, we set R u as a 4-dimensional vector, thus the produced observation embedding has 4 dimensions.
The dimensions of time representation p t and r v are both 16.
The trainable weight matrix D has shape of 4 × 32 .
The dimensions of w u and w v are the same as the number of sensors: 34 in P19, 36 in P12, and 17 in PAM.
We set the number of
R AINDROP layers L as 2 while the ﬁrst layer prunes edges and the second layer does not.
We set the proportion of edge pruning as 50% (K=50), which means we remove half of the existing edges that have the lowest weights.
The d k is set to 20, while the shape of W is 20 × 20 .
All the activation functions, without speciﬁc clariﬁcation, are sigmoid functions.
The d a is set equal to the number of sensors.
The ﬁrst layer of ϕ has 128 neurons while the second layer has C neurons ( i.e. , 2 for P19 and
P12; 8 for PAM).
We set λ = 0 .
02 to adjust the scale of regularization term L r .
All the preprocessed datasets and implementation codes are made available online.
More details can be found through our publicly accessible link.
A. P ERFORMANCE METRICS
Since P19 and P12 datasets are imbalanced, we use the Area Under a ROC Curve (AUROC) and
Area Under Precision-Recall Curve (AUPRC) to measure performance.
As the PAM dataset is nearly balanced, we also report accuracy, precision, recall and F1 score. We report mean and standard deviation values over 5 independent runs. Model parameters that achieve the best AUROC value on the validation set are used for test set. A. F URTHER DETAILS ON S ETUP DETAILS FOR S ETTING
In Setting 2, the selected missing sensors are ﬁxed across different models and chosen in the following way.
First, we calculate the importance score for each sensor and rank them in a descending order.
The importance score is based on information gain, which we calculate with feeding the observations into a Random Forest classiﬁer with 20 decision trees.
In particular, we treat each sample as only having one sensor, then feed the single sensor into random forest classiﬁer and record the AUROC.
The higher AUROC indicates the sensor provides higher information gain. When we have sensors ranked by their AUROC values, we choose the ﬁrst n sensors (the ones with highest AUROC values) and replace all observations in these sensors by zeros in all samples in validation and test set.
The number of missing sensors is deﬁned indirectly from the user with the sensors’ missing ratio which ranges from 0.1 to 0.5. A. A DDITIONAL INFORMATION ON MISSING PATTERN
This work propose R AINDROP which is a novel solution for irregularity in multivariate time series through inter-sensor dependencies. R AINDROP is not in conﬂict with other solutions (such as missing pattern and temporal decay) for irregularity. However, as the missing pattern is widely discussed in modelling incomplete time series (Che et al., 2018), we explore how to combine the advantages of relational structures and missing pattern. We adopt mask matrix as a proxy of missing pattern as in
Che et al.
Taking the architecture of R AINDROP , we concatenate the observation x ti,u with a binary mask indicator b t [ i, u ] as input.
The indicator b t [ i, u ] is set as 1 when there is an observation of sensor i at time t and set as 0 otherwise.
All the experimental settings and hyperparameters are the same as in R AINDROP (P19; Setting 1).
The experimental results show that taking advantage of missing pattern can slightly boost the AUROC by 1.2% and AUPRC by 0.9% in P19.
This empirically shed the light for future research on integrating multiple characteristics in representation of irregularly time series
A. COMPARISON BETWEEN TEMPORAL ATTENTION AND LSTM
We conduct extensive experiments to compare the effectiveness of temporal attention and LSTM.
To this end, we replace the temporal attention in sensor embedding generation (Eq 4-5) in R AINDROP by
LSTM layer which processes all observation embeddings sequentially.
We use zero padding to convert the irregular observations into ﬁxed-length time series so the data can be fed into LSTM architecture.
We regard the last output of LSTM as generated sensor embedding.
The number of LSTM cells equal to the dimension of observation embedding.
All the model structures are identical except in the part of temporal attention and LSTM.
We keep all experimental settings (P19; Setting 1) and hyperparameter selections the same.
The experimental results show that the temporal self-attention outperform LSTM by 1.8% (AUROC) and additionally saved 49% of the training time.
One potential reason is that the self-attention mechanism avoids recursion and allows parallel computation and also reduces performance degradation caused by long-term dependencies (Ganesh et al., 2021; Vaswani et al., 2017).
A. A DDITIONAL INFORMATION ON METHOD BENCHMARKING
We conduct extensive experiments to compare Raindrop with ODE-RNN (Chen et al., 2020), IPNet (Shukla & Marlin, 2018), DGM 2 -O (Wu et al., 2021), EvoNet (Hu et al., 2021), and MTGNN (Wu et al., 2020c). For the ﬁve newly added baselines, we follow the settings as provided in their public codes. For methods, which cannot deal with irregular data (i.e., EvoNet and MTGNN), we ﬁrst impute the missing data using mean imputation and then feed data into the model. For forecasting models (i.e., MTGNN) which are strictly not comparable with the proposed classiﬁcation model, we formulate the task as a single-step forecasting, concatenate the learned representations from all sensors and feed into a fully-connected layer (work as classiﬁer) to make classiﬁcation, and use cross-entropy to quantify the loss.
The results are from the PAM dataset under experimental Setting 1. As shown in the following Table 5, the experimental results conﬁrm that Raindrop outperforms the strongest baseline in accuracy by 4.7%. The added experiments increase our conﬁdence in the effectiveness of R AINDROP .
[Table]
A. R ESULTS FOR P19 (S ETTINGS 2-3)
Here we report the experimental results for P19 in Setting 2 (Table 6) and Setting 3 (Table 7).
[Table] [Table] [Table] [Table]
A. A DDITIONAL INFORMATION ON GROUP - WISE CLASSIFICATION
In experimental Setting 4, we evaluate R AINDROP under a challenging scenario.
Although R AINDROP is not designed to address domain adaptation explicitly, the results show that R AINDROP performs better than baselines when transferring from one group of samples to another.
One reason for our good performance is that the learned inter-sensor weights and dependency graphs are sample-speciﬁc and their learning is based on the sample’s observations.
Thus, the proposed R AINDROP has the power, to some extent, to adaptively learn the inter-sensor dependencies based on the test sample’s measurements.
R AINDROP is not generalizing to new groups, but generalizing to new samples, which leads to a good performance even though our model is not designed for domain adaptation. The reason is validated empirically. We remove the inter-sensor dependencies (set all sensors isolated in the dependency graph; set all α ti,uv and e ti,uv as 0) in R AINDROP and evaluate the model in Setting 4. The experimental results show that the performance drops a lot when excluding dependency graphs and message passing in R AINDROP (Table 8).
Without inter-sensor dependencies our model is on par with other baselines and does not outperform them by a large margin. [Figure] [Figure]
[Figure]
A. F URTHER DETAILS ON ABLATION STUDY
We provide ablation study, taking PAM at Setting 1 as an example, in Table 9. In the setup of ‘W/o sensor level concatenation’, we take the average of all sensor embeddings (in stead of concatenating them together) to obtain sample embedding. Experimental results show that the full R AINDROP model achieves the best performance, indicating every component or designed structure is useful to the model. For example, we ﬁnd that excluding inter-sensor attention weights α ti,uv will cause a decrease of 3.9% in accuracy while excluding edge weights e i,uv (i.e., dependency graphs) will drop the accuracy by 7.1%.
A. V ISUALIZATION OF INTER - SENSOR DEPENDENCY GRAPHS LEARNED BY R AINDROP
We visualize the learned inter-sensor dependencies ( i.e. , e i,uv before the averaging operation in Eq. 3) on P19 in early sepsis prediction. The visualizations are implemented with Cytoscape (Shannon et al., 2003).
The data shown are for testing set of P19 including 3,881 samples (3708 negative and 173 positive).
As R AINDROP learns the speciﬁc graph for each sample, we take average of all positive samples and visualize it in Figure 4b; and visualize the average of all negative samples in Figure 4b.
As we take average, the edges with weights smaller than 0.1 (means they rarely appear in graphs) are ignored.
The averaged edge weights range from 0.1 to 1.
We initialize all sample graphs as complete graph that has 1,156 = 34 × 34 edges, then prune out 50% of them in training phase, remaining 578 edges.
The 34 nodes in ﬁgures denote 34 sensors measured in P19, as listed https://physionet.org/content/challenge-2019/1.0.0/.
We list the sensor names here: 0: HR; 1: O2Sat; 2: Temp; 3: SBP; 4: MAP; 5: DBP; 6: Resp; 7: EtCO2; 8: BaseExcess; 9: HCO3; 10: FiO2; 11: pH; 12: PaCO2; 13: SaO2; 14: AST; 15: BUN; 16: Alkalinephos; 17: Calcium; 18: Chloride; 19: Creatinine; 20: Bilirubin_direct; 21: Glucose; 22: Lactate; 23: Magnesium; 24: Phosphate; 25:
Potassium; 26: Bilirubin_total; 27: TroponinI; 28: Hct; 29: Hgb; 30: PTT; 31: WBC; 32: Fibrinogen; 33: Platelets.
We also visualize the differential inter-sensor connections between the learned dependency graphs from patients who are likely to have sepsis and the graphs from patients who are unlikely to suffer from sepsis.
Based on the aggregated graph structures of positive and negative samples, we calculate
[Figure]
the divergence between two groups of patients and report the results in Figure 5.
In detail, we sort edges by the absolute difference of edge weights across negative and positive samples.
On top of the visualization of the 50 most distinctive edges, we can have a series of concrete insights.
For example, the dependency between node 6 (Respiration rate) to node 25 (Potassium) is important to the early prediction of sepsis.
Note these data-driven observations could be biased and still need conﬁrmation and future analysis from healthcare professionals.
The edges in both Figure 4 and Figure 5 are directed. The edge arrows might be difﬁcult to recognize due to the small ﬁgure size. We will provide high-resolution ﬁgures to our public repository.
Furthermore, we statistically measure the similarities across samples within the same class and dissimilarities across samples from different classes.
Speciﬁcally, for every sample, we calculate: 1) the average Euclidean distance between its dependency graph and the dependency graphs of all samples from the same class; 2) the average distance with all samples from the different classes.
The
P19 dataset has 38,803 samples including 1,623 positive samples and 37,180 negative samples.
For a fair comparison, we randomly select 1,
623 samples from the negative cohort, then mixed them with an equal number of positive samples to measure the averaged Euclidean distances intra- and inter-classes.
We select the cohort for 5 independent times with replacement.
We ﬁnd that the distance ( (8 . 6 ± 1 . 7) × 10 − 5 ) among dependency graphs of positive samples is smaller than the distance ( (12 . 9 ± 3 . 1) × 10 − 5 ) across samples.
The results show that the learned dependency graphs are similar within the same class and dissimilar across classes, which demonstrates R AINDROP can learn label-sensitive dependency graphs.