MGP-A TT TCN: A N I NTERPRETABLE M ACHINE L EARNING M ODEL FOR THE P REDICTION OF S EPSIS Anonymous authors Paper under double-blind review
Abstract
With a mortality rate of 5.4 million lives worldwide every year and a healthcare cost of more than 16 billion dollars in the USA alone, sepsis is one of the leading causes of hospital mortality and an increasing concern in the ageing western world.
Recently, medical and technological advances have helped re-deﬁne the illness criteria of this disease, which is otherwise poorly understood by the medical society.
Together with the rise of widely accessible Electronic Health Records, the advances in data mining and complex nonlinear algorithms are a promising avenue for the early detection of sepsis.
This work contributes to the research effort in the ﬁeld of automated sepsis detection with an open-access labelling of the medical MIMIC-III data set.
Moreover, we propose MGP-AttTCN: a joint multitask Gaussian Process and attention-based deep learning model to early predict the occurrence of sepsis in an interpretable manner.
We show that our model outperforms the current state-of-the-art and present evidence that different labelling heuristics lead to discrepancies in task difﬁculty.
I NTRODUCTION
Every year, it is estimated that 31.5 million people worldwide contract sepsis.
With a mortality rate of 17% in its benign state and 26% for its severe state (Fleischmann et al., 2016), sepsis is one of the leading causes of hospital mortality (Vincent et al., 2014), costing the healthcare system more than 16 billion dollars in the USA alone (Angus et al., 2001).
Studies demonstrated that early treatment has a signiﬁcant positive effect on the survival rate (Kumar et al., 2006; Nguyen et al., 2007).
In particular, Castellanos-Ortega et al.
(2010) demonstrated that each hour delay in treating a patient results in a 7.6% increase in mortality.
Current methods of screening, such as Modiﬁed Early Warning System (MEWS) and Systemic
Inﬂammatory Response Syndrome (SIRS) have been criticised for their lack of speciﬁcity, leading to low accuracies and high false alarm rates.
In 2015, the Third International Consensus Deﬁnitions for Sepsis (Singer et al., 2016; Seymour et al., 2016; Shankar-Hari et al., 2016) committee worked towards incorporating medical and technological advances into an up-to-date deﬁnition of sepsis, providing scientists with widely acknowledged illness criteria.
Together with the rise of Electronic
Health Records (EHR), the scientiﬁc community is now armed with both the data and labelling techniques to experiment with novel prediction methods (Islam et al., 2019;
Henry et al., 2015;
Ghosh et al., 2017; Calvert et al., 2016; Desautels et al., 2016), which are already proving effective in increasing survival rate (Shimabukuro et al., 2017) and promising in decreasing costs.
New models developed so far either relied on some interpretable yet simple prediction methods, such as logistic regression (Calvert et al., 2016) and decision tree based classiﬁers (Mao et al., 2018;
Delahanty et al., 2019), or on effective yet black-box methods such as Recurrent Neural Networks (Futoma et al., 2017b).
Moreover, the results achieved by different authors are rarely comparable: although most use the MIMIC-III data set, the disparities in labelling rules result in highly variable data sets (eg.
Raghu et al.
(2018) have 17,898 septic patients vs. 2,577 for Desautels et al.
This work presents an attempt at reconciling interpretability and predictive performance on the sepsis prediction task and makes the following contributions:
• Gold standard for labelling.
We provide a gold standard for Sepsis-3 labelling implemented on the MIMIC-III data set. [Figure]
• Novel interpretable model.
We present an explainable and end-to-end trainable model based on Multitask Gaussian Processes and Attentive Neural Networks for the early prediction of sepsis.
• Empirical evaluation.
We assess our model on real-world medical data and report superior predictive performance and interpretability compared to previous methods.
An overview of our proposed method is shown in Figure 1.
R ELATED WORK
Medical time series diagnosis Multiple researchers have tackled the task of predicting sepsis and septic shock.
Works on septic shock include exploration of survival models (Henry et al., 2015) and Hidden Markov Models (Ghosh et al., 2017).
However, these models rely on the assumption that a patient has already developed sepsis and focus on patterns of patients’ further deterioration.
Other authors (Calvert et al., 2016; Desautels et al., 2016; Mao et al., 2018; Delahanty et al., 2019) use linear models and decision trees on engineered features to predict sepsis onset, thus failing to capture temporal patterns.
More recently, Kam & Kim (2017) and Raghu et al.
(2018) employed recurrent neural networks to better capture time dependencies.
Crucially, all these models rely on either averaging or forward imputation of data points to create equidistant inputs.
This transformation creates data artefacts and discards relevant uncertainty: in the medical ﬁeld, the absence of data is a conscious decision made by professionals implying an underlying belief of the patient state.
Futoma et al.
(2017b)
and Moor et al.
(2019) tackled this issue with Multitask Gaussian Processes (MGPs), however their models lack the interpretability necessary in the medical ﬁeld.
Irregularly sampled time series The most common solution to missing values is forward imputation (Calvert et al., 2016).
Lipton et al.
(2016) utilise forward imputation coupled with a missingness indicator fed into a black-box model.
Although this method retains information about data presence, it is not clear how the information is later interpreted by the model and hence does not meet our transparency criteria.
Ghassemi et al.
(2015) use MGPs to ﬁt sparse medical data, however they optimise their model for the data ﬁt and use the parametrisation as input for a classiﬁer rather than optimising the model for a classiﬁcation task.
Both Futoma et al.
(2017a)
and Moor et al.
(2019) use MGPs with end-to-end training, although their temporal covariance function is shared across all variables.
Finally, Futoma et al.
(2017b) uses MGPs with multiple time kernels in a similar fashion to our model, although they infer the number of kernels from hyperparameter tuning rather than the data itself.
Attention based neural networks Attention was ﬁrst introduced on the topic of machine translation (Bahdanau et al., 2014).
Since then, the concept has been used in natural language processing (Yang et al., 2016; Yu et al., 2018) and image analysis (Mnih et al., 2014; Schlemper et al., 2019).
In the same spirit, Qin et al.
(2017) used attention mechanisms to improve the performance of a time series prediction model.
Although their model easily explains the variable importance, its attention mechanism is based on Long Short Term Memory encodings of the time series.
At any given time, such an encoding contains both the information of the current time point and all previous time points seen by the recurrent model.
As such, the time domain attention does not allow for easy interpretation.
More similar to our implementation is the RETAIN model (Choi et al., 2016), which generates its attention weights through reversed recurrent networks and applies them to a simple embedding of the time series.
The model employs recurrent neural networks which are slower to train and suffer from the vanishing gradient problem.
Furthermore, the initial and ﬁnal embeddings decrease the model’s interpretablity.
Other authors using Temporal Convolutional Network (TCN) based attention include Lin et al.
(2019), who only attend to time.
M ETHOD 3. N OTATION
Let us ﬁrst deﬁne some notation for the problem at hand.
For each patient encounter p , several features y p,t i ,k are recorded at times t p,k,i from admission, where k ∈ { 1 , . . .
, M } is the feature identiﬁer.
These features are often vital signs and laboratory results.
As such, they are rarely observed at the same times.
Hence, we have a sparse matrix representation of observations
[Equation]
where N p is the patient’s observation period length.
We also deﬁne static features s p = { s p,M +1 , ..
s p,M + Q } with features identiﬁers k ∈ { M + 1 , . .
, M + Q } , corresponding to timeindependent quantities, such as age, gender and ﬁrst admission unit.
Finally, we deﬁne sepsis labels l p ∈ { 0 , 1 } .
Given the sparsity of the data, we can deﬁne the compact representation of all observed values:
[Equation]
The goal of the model is, for a given set { t p , y p , s p } to predict the label l p .
In order to remove clutter, we will from now on drop the patient-speciﬁc subscript p from all notation, and the feature subscript k from time notation, simplifying t p,k,i to t i .
3. M ULTITASK G AUSSIAN P ROCESS (MGP)
Gaussian processes are commonly known for their ability to generate coherent function ﬁts to a set of irregular samples, by modelling the data covariance.
As they easily account for uncertainty and do not require homogeneously sampled data, Gaussian processes are the perfect candidate model to deal with relatively small amounts of medical data.
Following Bonilla et al.
(2008), we use a Multitask Gaussian Process (MGP) to capture feature correlation and Li & Marlin (2016)
’s end-to-end training framework, in a similar manner to Futoma et al.
(2017a).
Given an hourly spaced time series { t (cid:48) i }
i = − N p (where 0 is the time of prediction), the
MGP layer produces a set of posterior predictions for each feature, which will then be fed into a classiﬁcation model.
We deﬁne a patient-independent prior over the true values of { y i,k } by { f k ( t i ) } such that
[Equation]
where { K tt l ( t i , t j ) } l ∈ L are time point covariances varying in smoothness, { K k l ( k, k (cid:48) )
} l ∈ L are feature covariances at a given smoothness level, independent of time, and L are smoothness clusters.
Over all variables and time points, the multivariate model has covariance
[Equation]
5) where D = diag ( σ k ) are the noise terms associated to each feature and ⊗ is the Kronecker product.
The posterior over t (cid:48)
=
{ t (cid:48) i }
i = − N p is a multivariate Gaussian with mean and covariance:
[Equation]
In order to approximate the posterior distribution, we then take Monte Carlo samples y MC from
Y MGP ∼ N ( µ , Σ) .
Note that there are two main feature clusters: vital signs (vitals) and laboratory results (labs).
Vitals are noisier and sampled more often, whereas labs are more monotone and rarely sampled.
As opposed to Futoma et al.
(2017b), we do not treat the number of clusters L as hyperparameters but set L =and deﬁne
[Equation]
as Ornstein-Uhlenbeck (OU) kernels with lengths λ 1 and λ 2 , each representing a cluster smoothness.
OU kernels are well suited to capture local variations and do not assume inﬁnite differentiability as
Squared Exponential kernels do.
In our case, differentiablity implies a level of smoothness which does not apply to medical records and only introduces unnecessary bias.
In addition, given the scarce availability of labs, all short lengthscales would be an ill ﬁt to the data.
We hence discarded kernels varying over lengthscales such as the Cauchy and the Rational Quadratic kernels.
K kl ( k, k (cid:48) ) are free-form covariance matrice that are learned by gradient descent.
To feed the MGP samples into the classiﬁer, we ﬁx the model time window to N = 25 by either zero padding or truncating the beginning of the time series.
We choose to do so at the beginning of the time series in order to align prediction times as the last step of the temporal classiﬁcation model.
Here, we also integrate the static variables by broadcasting them over each time point 1 .
A TTENTION T IME C ONVOLUTIONAL N ETWORK (A TT TCN)
The concept of attention was born in machine translation (Bahdanau et al., 2014): given an input sentence embedding S
= { h 1 , . . .
h | S | } , the attention mechanism produces weights { α i 1 , . . .
α i | S | } such that α ij ∈ [0 , 1] , (cid:80)
j α ij = 1 , and a context vector c i =
(cid:80)
j α ij h j used to predict target word i .
The weights α ij can therefore be interpreted as the importance of the input sentence’s j th word to produce the i th word of the translation.
More recently, Choi et al.
(2016) have applied attention to clinical time series.
Given a time series { x 1 , . . .
x T } ⊂ R r , the authors ﬁrst create a time-independent embedding of the data { v 1 , . . .
v T } ⊂ R m .
They then use inversed recurrent neural networks (RNN) to create weights α ∈ R T and β ∈ R T × m , where α j ∈ [0 , 1] and β ij ∈ [ − 1 , 1] , with softmax and tanh activations respectively.
The context vectors then take the form c i = (cid:80) j ≤ i α j β j (cid:12) v j and are fed into a multilayer perceptron with softmax activation to yield a prediction.
The attention model we devised borrows some ideas from Choi et al.
The interpolated data y MC ∈ R N ×
M + Q ) is directly fed into two temporal convolutional networks (TCNs) (Lea et al.
and generates embeddings z =
[ z 1 , . .
. , z N ] ∈ R N × ( M + Q ) and z (cid:48) =
[ z (cid:48) 1 , . . .
, z (cid:48) N ] ∈ R N × ( M + Q ) .
TCNs are a class of neural networks composed of causal convolutions stacked into Residual Blocks.
A causal convolution is a 1D convolutional layer which only takes inputs from the past to generate its output, avoiding any information leakage from the future.
Residual Blocks are made of two causal convolutional layers together with ReLU activation functions, dropout and L2 regularisations.
The
Residual Blocks also include an identity map from the input of the block added to the output.
As we only use up to 12 layers, this last step is omitted in our architecture.
TCNs have shown to outperform
RNNs (Bai et al.
(2018)), are faster at training and do not suffer from vanishing gradients.
Given the latter, inverting the time series similarly to Lea et al.
(2017) also becomes an unnecessary step which we omit.
We generate the attention weights α and β as
[Equation]
[Equation]
such that α =
[ α 0 , α 1 ] ∈ R N × 2 and β =
[ β 0 , β 1 ] ∈ R N × ( M + Q ) × 2 .
We then create two context vectors, one for each of negative and positive label predictions
[Equation]
where y MC ,j is broadcast to meet the dimensionality of β j,δ .
We then predict the labels as
[Equation]
In our case, we are only interested in making predictions with the latest available data.
We therefore only use ˆl last to train the model.
This of course can be easily modiﬁed to suit any speciﬁc use-case.
Since the MGP output is directly multiplied by weights c i , the classiﬁcation model can be interpreted as a scoring mechanism where each past point y MC ,ij contributes α i, 0 β ij, 0 to the time series being classiﬁed as positive, and α i, 1 β ij, 1 to the time series being classiﬁed as negative.
The positive and negative scores are then normalised to represent probabilities of the positive or negative labelling.
As we design both α and β to be non-negative, we can hence directly look at the average α and β over
Monte Carlo samples to see which time points and features contribute most strongly to the network’s positive or negative decision.
D ATA
Sepsis is deﬁned as a life-threatening organ dysfunction caused by a dysregulated host response to infection (Singer et al., 2016).
The latter is usually interpreted as the administration of antibiotics coupled with the culture of blood samples, generating a suspicion of infection window, whereas the former is interpreted as a two point increase in Sequential Organ Failure Assessment (SOFA) within such a suspected infection window.
We make use of the MIMIC-III data set (Johnson et al., 2016) and encode the Sepsis-3 criteria following Johnson & Pollard (2018)’s code available on GitHub, with the help of Moor et al.
(2019)’s code that the authors have generously provided.
One key difference between our assumptions and Moor et al.
(2019)’s is the handling of missing
SOFA contributor values: if one or more SOFA contributors are missing, Moor et al.
do not calculate the total score.
On the other hand, we assume such a contributor to be within a healthy norm, implying a zero contribution.
The latter heuristics is in line with the ofﬁcial Sepsis 3 deﬁnition in Singer et al.
In order to validate our results, we carry out all experiments using both labelling techniques.
We proceed to extract times series of case and control patients for a set of commonly recorded vitals, labs and static variables and normalise their values.
Following Moor et al.
(2019), in order to keep the data set length balanced, we match the time series lengths of control patients to those of case patients using the class balance ratio.
In addition, we create up to seven copies of each time series and truncate the last zero to six hours of data, effectively creating early prediction patients and augmenting our data set.
We remove excessively noisy or computationally intensive data and train the model over
[Figure]
different hyperparameters, randomly resampling an equal number of case and control patients to counteract the data set imbalance.
E XPERIMENTAL R ESULTS
We compare our model’s performance to the performance of the InSight algorithm (Calvert et al., 2016) and to the state-of-the-art MGP-TCN algorithm
(Moor et al., 2019).
Figure 3 shows the predictive performance of the models for different time horizons.
5. C OMPARISON BETWEEN DIFFERENT DATA LABELS
The ﬁrst result is the difference in performance of models applied to the different labelling methods.
The SOFA contributor assumption Moor et al.
make has two main implications.
Firstly, it considerably restricts the number of patients.
Assuming that sicker patients receive more medical attention, the patients included are likely to be in worse conditions than the septic patients excluded and hence easier to classify.
Secondly, it delays sepsis onset.
For example, a patient having a severe liver failure with few other recorded vitals, followed by an overall collapse further in time will have septic onset at the time of its liver failure in our records, whereas it will only be considered septic at the time of the overall collapse in Moor et al.
’s labels.
On the other hand, the labels we produce reﬂect the incomplete nature of medical data: even if only a part of all the potentially relevant tests are carried out at any given time, a doctor must be able to assess a patient’s well-being and foresee potential complications.
The difference in labels implies a discrepancy in task difﬁculty: Moor et al.’s
labels present an easier learning problem, however they deﬁne a more narrow use-case in real-world scenarios.
Indeed, when assessing the performance of the different models on the two different data labellings, it becomes evident that our proposed labels are harder to ﬁt.
This means that predicting sepsis in a realistic setting on the intensive care unit is probably much harder than previous work would suggest.
5. M ODEL PERFORMANCE
We ﬁnd that our MGP-AttTCN model has a better performance when presented with patients further in time from sepsis onset.
In the case of Moor et al.’s labels the difference is clearly noticeable, whereas with our labels it is of lower statistical signiﬁcance.
With our labels, both MGP-TCN and
MGP-AttTCN have a stronger performance than InSight.
The intuition behind this result is the robustness of the models to missing data: both MGP-TCN and MGP-AttTCN account for the data uncertainty and hence have a better performance on lower resolution and more irregular data.
[Figure]
bicarbonatecreatininechlorideglucosehematocrithemoglobinlactateplateletpotassiumpttinrptsodiumbunwbcmagnesiumph
[Figure]
for time covariance lenth 1.
for time covariance lenth 64.
On Moor et al.’s labels, the MGP-TCN model does not seem to signiﬁcantly outperform the InSight model, suggesting that those labels might be easy enough to not require a particularly pronounced robustness to missing data.
However, the additional attention of the proposed MGP-AttTCN model does seem to gain a clearer advantage here than on our labels, presumably due to a more complete set of features that can be attended to.
5. MGP INTERPRETABILITY
Inspecting the learned covariances (Figure 4), we notice that the two OU lengthscales converged to represent two clusters within the selected variables: a shorter lengthscale (around two hours) represents noisy data, whereas a larger lengthscale (around 64 hours) represents smoother observations.
In addition, the feature covariance matrix for the short lengthscale puts more emphasis on vitals, while the one for the long lengthscale puts more emphasis on labs, ﬁtting our initial intuition that vitals vary more rapidly.
Graphically, one can observe this by inspecting the diagonals on the covariance heatmaps.
On a more granular level, the two covariance matrices also provide insights about the underlying variables.
One can for instance observe that the body temperature ( tempc ) has a larger variance than the systolic and diastolic blood pressure ( sysbp , diabp ), following the general clinical intuition.
Moreover, we can observe correlations between different features, such as a negative correlation between temperature and heart rate, which also seems to coincide with the general medical expectation.
These covariances can then for instance be used by the model to extrapolate a full function from a single INR observation with an inverse correlation to the pulse oximetry observations (Fig. 5).
5. A TTENTION WEIGHTS
One important beneﬁt of our model compared to current approaches is its interpretability due to the attention mechanism.
Once the samples have been drawn, the weights α and β provide us with more information about the importance of different time points and features for the model’s behaviour.
The attention weights for an exemplary patient trajectory are depicted in Figure 5.
Overall, the absolute values of α are small for points further from the prediction time and increasingly larger closer to it.
A good example of this behaviour is the fourth row in Figure 5, where feature importance increases in time.
We can also see there, that different features can have opposing effects on the prediction.
While the elevated heart rate close to the prediction time increases the likelihood of a sepsis prediction (ﬁrst column, yellow weights), the lowered prothrombin values reduce this likelihood (third column, blue weights).
Interestingly, the low prothrombin values are not actually measured in this example, but predicted by the MGP purely based on the other measured features and the learned covariances.
Heart Rate - var.
Pulse oximetry (SpO2) - var.
Prothrombin time (INR) - var.
var.var.var.var.var.var.var.
[Figure]
Finally, α × β × y MC gives the individual score contribution of each feature at each time point.
These weights are shown in the last row of the ﬁgure.
It can again be seen that the attention weights are generally larger in magnitude closer to the prediction time.
Moreover, about half of the features have signiﬁcant non-zero attention weights, while the others seem to not be important for the prediction in this example.
These visualizations could be used by doctors to make an informed decision about whether or not to trust the prediction of the model for each given patient, thus facilitating the interpretability and accountability that is crucial in medical applications.
C ONCLUSION
We have shown that current data sets for the early prediction of sepsis underestimate the true difﬁculty of the problem and proposed a new labelling for the MIMIC-III data set that corresponds more closely to a realistic intensive care setting.
Moreover, we have proposed a new machine learning model, the MGP-AttTCN, which outperforms the state-of-the-art approaches on the easier labels from the literature as well as on our proposed harder labels.
Additionally, our model provides an interpretable attention mechanism that would allow clinicians to make more informed decisions about trusting its predictions on a case-by-case basis.
Potential avenues for future work include a more thorough discussion with clinicians to potentially make our proposed labels even more representative of the real-world task, and architectural improvements, for instance by meta-learning the MGP prior (Fortuin & Rätsch, 2019), amortizing the latent MGP inference for performance gains (Fortuin et al., 2019), or discretizing the latent space for improved interpretability (Fortuin et al., 2018).
A PPENDIX A D ATA PROCESSING A. D ATA LABELLING
Please refer to link omitted for anonymity for more details how we derived the MIMIC-III sepsis labels.
A. D ATA EXTRACTION
Patient Inclusion We ﬁlter for patients admitted to Intensive Care Units (ICU) who are more than 14 years old and with valid records.
Case patients are patients having sepsis onset within their ICU stay, whereas control patients have not developed sepsis nor have an ICD discharge code referring to sepsis.
Starting with 58’976 patients, we ﬁnd 14’071 control patients and 7’936 case patients using our labels, versus 1’797 cases using Moor et al.
labels.
Feature extraction Reviewing sepsis related literature and commonly extracted laboratory and vital recordings, we extracted all features which were reported at least once for more than 75% of the included population.
The ﬁnal 24 dynamic features are reported in Table 3.
We also extracted static features - age, gender and ﬁrst ICU admission department.
Case-control matching As the goal is to predict sepsis prior to onset, the cases data was extracted between ICU admission and sepsis onset.
Note that sepsis onset happens early within ICU admission, with the median patient getting sick at 3.4 hours of admission.
On the other hand, patients not developing sepsis are more likely to recover completely, and do so in a lengthier time frame.
In addition, once they are close to discharge, their vitals and labs are within the norms.
Hence, both the length and the values of the time series are strong discriminatory factors which ease the classiﬁcation.
We hence carry out a matching strategy similar to Moor et al.
(2019): following the class imbalance ratio, we associate each control time series to a case time series and truncate the control to have the same length as the case from ICU admission.
We then discard patients with less than 40 data points within the selected window, and - for computational tractability - truncate the ﬁrst N p − 250 initial values of patients’ time series in order to keep a maximum of 250 data points per patient.
Horizon augmentation As our goal is to predict sepsis early, we augment the data by creating new shorter time series.
For each time series, we create six copies, where each copy represents a different horizon to onset.
We then proceed to truncate the last one to six hours prior to onset from the time series copies.
In order to keep data consistency, we once again discard time series with less thanobservations.
In Tables 1 and 2 we illustrate the data distribution per horizon.
[Table]
Data split Finally we split the data into training, validation and testing sets, respectively capturing 80%, 10% and 10% of the data.
We then normalise the data by subtracting the training set mean and dividing by the training set standard deviation of each feature.
[Table]
[Table]
A PPENDIX B B ASELINES B. D ATA PREPARATION
In order to benchmark our MGP model, we build some baselines homogenising the data sampling.
For each hour and variable, we take the average of the available observations.
If a given hour has no observations, we carry forward the average of the previous hour.
In this manner, we generate an hourly sampled time series for each patient.
We then proceed to normalise the size of each patient matrix by setting a time window of observation N .
For patients having more than N observations
N p , we discard the ﬁrst N − N p observation; whereas for patients having less than N observations we pad the beginning of the matrix with zeros.
  y p, 1 ,t. . .
y p, 1 ,t Np .
y p,M,t. . .
y p,M,t
Np 
 carry forward −−−−−−−→   y p, 1 ,. . .
y p, 1 ,N p .
y p,M
[Equation]
normalise
[Equation]
 
[Equation]
We choose to align the end of the time series as opposed to the beginning as the relative importance of time points is to when a patient becomes sick rather to when he is admitted to the ICU.
As a next step, we augment the data to focus on different time series in a similar manner than for irregularly sampled data.
We create seven copies of each time series, for each copy we discard the last zero to six hours, then normalise the matrix as above.
We hence generate a dataset Y BL
= { Y } q = {{ y q,ij } N,M i,j =1 } q where q represent all augmented the time series.
B. I N S IGHT
The InSight scoring model is one of the few machine learning algorithms to surpass the proof-ofconcept stage with multiple research, economic and clinical trials (Calvert et al., 2016; Desautels et al., 2016; Calvert et al., 2017; Mao et al., 2018).
We therefore include it as a baseline to our model.
The key concept of the model is to use few largely available vitals, build some handcrafted features and train a simple classiﬁcation model.
Here is an account of our interpretation of the author’s method.
The features extracted are based on a six consecutive hour window.
For each six hour window, we extract each variable’s
mean M i and difference
D i (last observation minus ﬁrst observation) over the window.
We also extract variables pairs correlation D ij and triplet correlation D ijk ; where i, j, k are observed variables.
We interpret the latter as a relaxation of the Pearson correlation: if the correlation between two variables is
[Equation] [Equation]
then we deﬁne the triplet correlation as
[Equation]
We then classify the difference and correlations as either positive, negligible or negative using their distribution quantiles over every patient and six hour window observed.
Note that given the high level of data missingness, many variables are calculated by forward imputation and hence have no variance over six hours.
To adjust for the high number of zero correlations, we calcualte the quantiles of non-zero correlations and deﬁne:
[Equation]
where q ∗ is the adjusted quantile function.
We proceed in a similar manner for the correlations and triplet correlations.
In order to keep the results comparable to the AttTCN ﬁxed window N , we extract N − (6 − 1) six consecutive hour window and vectorise the resulting features, generating in total
[Equation]
features per patient.
Although the original paper does not specify which classiﬁcation method the authors employ, we derive by their description of a dimensionless score that the method is a logistic regression.
A PPENDIX C M ODEL DETAILS C. TCN PROPERTIES
TCNs are a class of neural networks composed of causal convolutions stacked into Residual Blocks.
A causal convolution is a 1D convolutional layer which only takes inputs from the past to generate its output, avoiding any data leakage.
Residual Blocks are made of two causal convolutional layers together with ReLU activation functions, dropout and L2 regularisations.
C. S TATIC VARIABLES
In our model implementation, we decide to integrate the static variables to the MGP output.
Another choice we considered is to integrate the data to the output of the attention model.
Once the weights α and β have been created, we can introduce a bias term to
[Equation]
[Table]
[Table]
where b static is generated by the output of a two layers multilayer perceptron applied to the static data.
Although this solution is computationally lighter and provides a more elegant interpretation [Table] it does not allow the attention mechanism to utilise the information about the patient’s static state when making a decision about its vitals and labs values.
For the scope of this paper [Table] we hence decided for earlier static data integration [Table]
C. P ARAMETER HYPERSEARCH AND TRAINING
As the datasets are highly unbalanced [Table] we carry out a case set oversampling: we randomly resample the case set to have the same size as the control set.
In addition, at each iteration we sample equally the same number of cases and controls, then feed a shufﬂed version into the model.
In this manner the model will see an equal number of controls and cases and will not become biased towards zero labels.
This procedure does not happen for either of the validations and test sets, as the results would not compare to real life settings.
For both our core model MGP-AttTCN and all baselines, in order to select the best possible hyperparameters, we performed a hyperparameter random search, as described in Table 4. C. N UMERICAL RESULTS The results are shown in Tables 5 and 6. [Table]