C LUSTER - BASED F EATURE I MPORTANCE L EARNING FOR E LECTRONIC H EALTH R ECORD T IME - SERIES
Anonymous authors
Paper under double-blind review
A BSTRACT
The recent availability of Electronic Health Records (EHR) has allowed for the development of algorithms predicting inpatient risk of deterioration and trajectory evolution.
However, prediction of disease progression with EHR is challenging since these data are sparse, heterogeneous, multi-dimensional, and multi-modal time-series.
As such, clustering is used to identify similar groups within the patient cohort to improve prediction.
Current models have shown some success in obtaining cluster representation of patient trajectories, however, they i) fail to obtain clinical interpretability for each cluster, and ii) struggle to learn meaningful cluster numbers in the context of the imbalanced distribution of disease outcomes.
We propose a supervised deep learning model to cluster EHR data based on the identification of clinically understandable phenotypes with regard to both outcome prediction and patient trajectory.
We introduce novel loss functions to address the problems of class imbalance and cluster collapse, and furthermore propose a feature-time attention mechanism to identify cluster-based phenotype importance across time and feature dimensions.
We tested our model in over 100,000 unique trajectories from hospitalised patients with Type-II respiratory failure to predict four different outcomes.
Our model yielded added interpretability to cluster formation and outperformed benchmarks by at least 5% in mean AUROC.
I NTRODUCTION
Chronic conditions such as Chronic Obstructive Pulmonary Disease (COPD) and Cardiovascular
Disease (CVD) describe a broad spectrum of medical ailments, and affect a significant percentage of the overall population (Adeloye et al., 2015).
Such diseases are characterized by the existence of multiple distinct patient subgroups, largely distinguished by differences in pathology and in the response to different treatments and medical interventions Turner et al.
; Vogelmeier et al.
Exacerbation of COPD, a condition of respiratory failure, can result in emergency hospital admission and mortality if it is not well treated and managed.
Early identification of COPD patients’ subgroups is therefore of high medical importance and relevance.
EHR time-series data are typically used to determine clinically relevant COPD inpatient subgroups, and have been applied to detect risk of deterioration (Pikoula et al., 2019).
However, modelling disease progression and risk prediction is challenging due to the extreme data heterogeneity nature of EHRs.
Firstly, EHR data contains a mixture of demographic or static variables (i.e. time independent such as age and sex), and multi-dimensional time-series (e.g Heart Rate,
HR, and laboratory measurements, such as blood tests).
Secondly, EHR time-series are multi-modal as different features are collected from different devices, representing distinct clinical properties of relevance.
Similarly, time-series features are sampled at different times and have low and distinct sampling rates, as well as different missing value properties.
Furthermore, each feature is associated with different noise and evolution patterns.
Recent advances in deep learning (DL) approaches have shown promising results in EHR modelling due to their capacity to handle complex data (Rajkomar et al., 2018).
Nonetheless, DL approaches lack relevant interpretability frameworks to be scaled and applied in hospital settings.
Several such models have since been proposed to tackle this issue (Mayhew et al., 2018), however, most of them focus on a subset of EHR features (usually vital signs only) and fail to provide a clinically-focused phenotypic analysis of learnt patient sub-groups (via clustering).
This work builds on previous research in literature to introduce a cluster-based feature-time attention mechanism to predict patient outcomes based on EHR data.
Our method also leverages phenotypic information to aid in clinical interpretability, not only making use of demographics and vital-signs information but also of relevant laboratory measurements (all present in the EHR) to provide a more complete patient physiological status.
Our contributions include the following:
• An end-to-end DL supervised model to cluster EHR patient data based on the identification of clinically understandable cluster phenotypes with regard to both outcome prediction and patient trajectory in a multi-class setting; • A weighted loss to address data imbalance for both tasks of clustering and prediction, a common issue in the medical domain; • The incorporation of a novel loss mechanism in the model, to address the issue of cluster collapse and promote sample assignment to all available clusters; • Finally, the inclusion of a novel interpretability framework, derived from a cluster-based feature-time attention layer, aiming to identify relevant timestamps and feature variables to represent the patient physiology, cluster assignment and, ultimately, outcome prediction.
This paper is structured as follows.
In Section 2, we describe previous research in EHR time-series modelling, clustering and attention methods.
Section 3 introduces the dataset used for analysis and provides description of the proposed model.
The experimental setup and results of our analysis are presented in Section 4 and discussion takes place in Section 5.
Finally, concluding remarks and future work are available in Section 6.
R ELATED W ORK
EHR data comprise complex time-series data, being high-dimensional, multi-modal and heterogeneous, and thus presenting challenges when used in machine learning models (Keogh & Kasetty (2003); Rani & Sikka (2012)).
An important goal in a medical setting is to identify phenotypically separable clusters with distinct phenotypic profiles (which we denote as phenotypic clustering hereafter).
For the purpose of this work, cluster phenotypes result from the combination of two distinct components: a) the evolution profile of patient trajectories’ within the cluster, and b) the characterisation of the cluster with regards to clinical variables of interest.
The latter may include features not used for clustering and may provide information about the underlying or future health status.
Traditional clustering models such as K-Means or hierarchical clustering have been shown to fail to capture the existing time-dependent feature relationships.
As such, variants have been proposed to mitigate this problem.
A temporal version of the K-Means algorithm, Time-Series K-Means (TSKM, Tavenard et al.
(2020)), models the distance between time-series of different datapoints, using the Euclidean distance (which is equivalent to considering all temporal observations as an independent feature value for the corresponding patient admission), or time-series alignment strategies such as Dynamic-Time Warping (DTW, Berndt & Clifford (1994)) and soft-DTW (Cuturi &
Blondel (2017)).
Recent DL architectures, ranging from Auto-Encoders (AE, Ma et al.
, Convolutional Neural
Networks (CNN, Munir et al.
and others, have shown great promise when applied to timeseries data across a variety of domains.
Fortuin et al.
(2019) proposed a Self-Organising Map - Variational Auto-Encoder (SOM-VAE), is a state-of-the-art, unsupervised, DL clustering algorithm which extends a variational auto-encoder architecture (Kingma & Welling, 2013) for observation learning and representation, through the addition of a Markov model (Gagniuc, 2017), to infer temporal evolution within the latent space.
Clustering is performed in the low dimensional latent space through the use of self-organising maps (Kohonen, 1982) to obtain a discrete, topologically-interpretable latent representation of the learnt clusters.
In a supervised setting, AC-TPC (Lee & Van Der Schaar, 2020) serves as the current state-of-the-art for identifying phenotypically separable clusters in patient trajectories in EHR data.
AC-TPC maps EHR data into a latent space via an encoder, and uses an actor-critic network (Konda & Tsitsiklis, 2000) which leverages clinical outcomes to aid in cluster formation and obtaining cluster phenotypes.
Neither SOM-VAE and AC-TPC provide clinically meaningful interpretation of feature-time importance or outcome of interest.
Attention mechanisms have recently been proposed to provide greater interpretability to Recurrent
Neural Networks (RNN) and to aid in dealing with long-term dependencies (Vaswani et al., 2017; Xu et al., 2015), and have also been used in modelling EHR time-series (Schwab et al., 2017; Shashikumar et al., 2018).
RETAIN (Choi et al., 2016) proposes a two-level reverse attention mechanism to mimic physician’s decision process and predict a future diagnosis.
In other recent works, attention mechanisms based on bi-directional RNN and CNN outperformed standard classification models in predicting high risk vascular diseases with the addition of medication information as input data (Kim et al., 2017).
A drawback of such attention mechanisms is the focus on temporal interpretability only, and inability to look at individual features, which is key in a medical setting.
To solve this issue, Shamout et al.
(2019) considered independent RNN per feature, with a concatenation of the resulting latent vectors.
However, the latter does not allow the joint modelling across both feature and time dimensions.
Alternatively, Kaji et al.
; Gandin et al.
(2021) proposed learning attention weights directly on the original inputs, prior to being transformed by a RNN, which does not allow modelling of the resulting latent representations.
To the best of our knowledge, no existing models have been proposed that jointly leverage both feature and time dimensions (feature-time) to determine clinical observation relevance on clustered EHR data.
M ETHODS 3. D ATASET AND P RE -P ROCESSING
Our dataset was retrieved from a retrospective database of routinely collected observations from concluded hospital admissions between March 2014 and March 2018 (the HAVEN project, REC reference: 16/SC/0264 and Confidential Advisory Group reference 08/02/1394).
The database includes EHR measurements of adult patients admitted to four hospitals from the Oxford University
Hospitals NHS Foundation Trust.
Note that the HAVEN dataset does not include data from Intensive Care Units (ICU), and we have excluded observations taken in the Emergency Department.
Key characteristics of HAVEN cohort data include a) heterogeneity, b) multi-modality , and difference in: c) noise distributions, d) sampling rates, e) missing values, etc.
Such properties are common across
EHR settings, and are challenging with respect to learning useful representations and predictions.
We used the protocol defined in Pimentel et al.
(2019) to subset the cohort to those patients at risk of developing Type-II Respiratory Failure (T2RF) in hospital (a diagram of the data selection steps can be found in Figure A.1 in the Appendix).
Four patient outcomes were considered in our analysis: i) no event during hospital stay, leading to successful discharge from the hospital, or the first instance of one of three possible events, ii) unplanned entry to ICU, iii) cardiac arrest (also named ”Cardiac” hereafter) and iv) death.
Outcome groups are not clearly separable (see Tables A.2, A.3 in the
Appendix), so patient clusters will naturally contain a mix of different admission outcomes.
In this setting, the clinically relevant component of a cluster phenotype (henceforth denoted as cluster outcome propensity or cluster outcome ) is represented as a categorical distribution indicating the corresponding propensity for cluster-assigned patients to each corresponding outcome.
For each admission, observations were grouped according to mean window observation value into 4 hour blocks based on the time to outcome (discharge in the case of no event during stay) - only observations within 24 and 72 hours before the outcome were considered.
This time window was selected based on those traditionally used for validating Early Warning Score (EWS) systems (baseline models used by UK NHS staff to track inpatient physiology, (Royal College of Physicians, 2017) and clinical input, such that the target phenotype represents the patient status in the subsequenthours.
Features were transformed according to min-max normalization due to skewness and heterogeneity in their distributions.
Patient admissions were randomly split into train, validation and test sets.
Missing values were imputed based on the previously observed time block - all remaining missing observations were imputed according to the feature median from the aggregated validation and test data (see Section 4 for description of train-test data split).
Imputed values were flagged in a three-dimensional mask matrix.
After processing, input data contained over 100,000 unique patient trajectories corresponding to 4,266 unique patient admissions (only last patients admissions were considered in our analysis).
Original trajectories for the patient cohort are shown in the Appendix in Figures
A.4, A.5, A. for different variables/features.
A lack of clear outcome group separability can be observed across temporal and static variables.
Furthermore, we note the high degree of imbalance in the dataadmissions with no event account for over 86.8% of the total number of admissions, while event classes correspond to 10.3% Death, 1.8% ICU and 1.1% Cardiac.
3. P ROPOSED MODEL
We propose a novel model, which we denote by C luster-b A sed i M portanc E L earning f O r T imeseries (CAMELOT).
Our proposed methodology is displayed in Figure 1 1 .
Our model builds on previous literature on 3 key items: a) a modified loss function to target the multi-class imbalance, b) a novel loss function to ensure cluster assignment and phenotype are representative, and c) a novel feature-time attention-level framework to boost representation and introduce feature-time interpretability for cluster assignment. [Figure]
Let N denote the number of patients and D f the number of input features.
Input data consists of a set of patient trajectories
X = {{ x n,t } T n t =1 } Nn =1 , where T n is the maximum number of temporal observations for patient n , and a set of patient outcomes
Y = { y n } Nn =1 .
Input trajectory data for the n -th patient is represented as X n = [ x n, 1 , ..., x n,T n ] , where each x n,t ∈ R D f is referred to as an observation (vector), with a maximum of observed D f feature values.
The corresponding patient outcome is a one-hot encoded vector y n ∈ R 4 (more generally, the dimension of y n equals to the number of possible outcomes).
Our DL model can be decomposed into 3 neural network blocks: an Encoder, Identifier and Predictor.
We refer the action of each network respectively as E, I and P (for example, I ( x ) denotes the output of the Identifier given some input vector x ).
Both the Identifier and Predictor are Multilayer
Perceptrons (MLP), networks of stacked feed-forward dense layers.
On the other hand, the Encoder block can be further sub-divided into a) a stack of RNN layers and b) our proposed custom attention layer (see Section 3.3 for further details).
Separately, we also consider a set of trainable cluster representation vectors, C = { c 1 , ..., c K } .
We assign the outcome for cluster i as P ( c i ) .
A model call is as follows:
Given the n -th patient input trajectory data X n , the Encoder network returns a latent representation z n := E ( X n ) ∈ R l .
Consequently, the Identifier network computes cluster assignment probabilities, π n := I ( z n ) ∈ R K .
Each element of π n , π in , represents the probability assignment of z n to cluster i , given a total of K clusters.
A cluster, k n samp is selected according to categorical sampling ( k n samp ∼ Cat ( π n ) ), and the corresponding cluster representation, c n samp :=
c k n samp is then selected from C .
The output of the model is y pred :=
P ( c n samp ) ∈ R 4 .
We note that k n samp is only sampled during a training phase ; at prediction stage, cluster selection follows the equation k n pred = arg max i =1 ,...,K π in .
3. E NCODER N ETWORK AND A C USTOM A TTENTION L AYER
The diagram of our proposed Encoder network is presented in Figure 2.
The Encoder contains (i) a Recurrent Neural Network (RNN) block of stacked Long Short-Term Memory (LSTM) layers, and (ii) a customised attention layer, which computes a latent representation by comparing input data with the sequence of output states from the RNN block. We use the same notation as above, [Figure]
𝒙 𝒕 𝒙 𝒕+
[Figure]
𝒙 [Figure] … 𝒙 𝒕 … … … … … … … … … … 𝒐 𝒕− 𝒐 𝒕 𝒐 𝒕+ 𝒐 𝒕 𝑹 𝒕 𝑹 𝒕 𝑹 𝒕− 𝑹 𝒕+ 𝜶 𝜶 𝑡 𝜶 ,𝜷 𝜷 𝒛 𝒛 𝑡−1 𝑡 𝑡+1 𝑡𝑡 … … and write the sequence of output states of the final LSTM layer as o n, 1 , ..., o n,T n , with o n,i ∈ R l .
Theoretically, each o n,t corresponds to a representative summary of input patient information up until time t .
We propose to approximate o n,t as a linear combination of latent representations of each individual feature, thereby allowing the separation of output states into contributions from each feature.
Note that it is important feature transformations be time-independent , to avoid overparametrising the model, over-fitting and ensure feature representation maps are similar across time.
Our attention layer behaves as a set of D f feed-forward neural network layers, U 1 , ..., U D f , jointly represented by: (i) a matrix of learnable kernel weights D ∈ R l × D f .
We write D =
[ D 1 , ..., D D f ] ; (ii) a matrix of learnable bias vectors B ∈ R l × D f .
Similarly, we can write B =
[ B 1 , ..., B D f ] ; and (iii) an activation function, σ which matches the output activation of the RNN block.
Input data for patient n , X n is fed as input to the RNN block, which outputs a sequence of latent output states ( o n,t ) T n t =1 .
For t = 1 , ..., T n , we compute D f feature representations in latent space as:
[Equation]
where R n,t =
[ R 1 n,t , ..., R D f n,t ] is our collection of feature representations, σ is applied element-wise and A :
= D ⊙ x n,t is a matrix satisfying A i,j = D i,j ( x n,t ) j .
Equivalently, R in,t is the output of a dense layer, U i with kernel D i , bias B i , activation σ and input ( x n , t ) i .
We approximate o n,t ≈ (cid:80) D f i
=
α it R in,t = R n,t α t .
This approximation is minimised following a least squares criterion, which has a well-known solution, (cid:98) α t , and corresponding optimal approximation (cid:98) o n,t .
Given, ˆ o n,t , a similar procedure is used to compute a context vector as z := (cid:80) t β t ˆ o n,t , where weights β are learned to provide a more representative context vector.
3. A TTENTION M AP V ISUALISATION
Given cluster representation vectors, c k , we can compute a cluster-wise feature-time attention visualisation map as follows.
First, we normalise feature-weights (cid:98) α t according to a softmax function, s t =
σ ( (cid:98) α t ) ∈
R D f , where σ is the softmax function: σ ( x ) = exp
| x | ∥ exp | x |∥ 1 .
Secondly, we compute cluster-wise weights, γ k t according to a least-square approximation of c k ≈ (cid:80) T n t =1 (cid:98) o n,t γ k n,t , and solved as before.
We similarly normalise γ kn,t to obtain cluster temporal scores, e kn,t = σ ( γ kn,t ) .
Finally, we can compute K scoring matrices, M 1 n , ..., M Kn ∈ R T n × D f : (cid:0) M kn (cid:1) t,f = e kn,t s ft .
Note that: ∥
[Equation]
t e kn,t (cid:80) f s ft = (cid:80) t e kn,t = 1 .
Given that Matrices M kn are normalised, they may be consequently, visualised as a normalised feature-time map for cluster assignment relevance and provide further model interpretability.
3. L OSS OPTIMISATION
The model is optimised through consideration of three distinct loss functions.
We introduce a weighted cross-entropy loss function:
[Equation]
where c ′ is the true outcome for a particular patient.
We propose inversely proportional normalised weights: (cid:80)
Cc =1 w c = 1 and w c is inversely proportional to the class distribution, i.e., w c ∝ NN c with
N being the number of patients and N c being the number of patients with outcome label c .
Class weighting penalises misclassification more heavily on less sampled classes.
We also propose a novel distribution loss function, L dist ( π ) .
We define the average cluster probability of assignment as π C :=
1 N (cid:80) n π n .
Then we introduce L dist ( π ) =
− H ( π C ) , where H denotes entropy.
Note that L dist is minimised when π C is uniform, ensuring all clusters are ’explored’ and have comparable number of samples.
While clusters should not necessarily be assigned the same number of samples, this loss helps to overcome a concern of cluster collapse , where clusters do not separate and samples are assigned to a very small number of non-representative clusters.
Finally, to separate cluster representation vectors, we define the cluster separation loss as
L
[Equation]
To optimise our model, iterative gradients are applied according to weighted combinations of the above loss functions with hyper-parameter weights α, β :
Outcome Predictor is updated according to L pred ; 2.
Encoder and Identifier are trained according to L pred + αL dist ; 3.
Cluster representation vectors are updated with regards to L pred + βL clus .
I NITIALISATION
Our proposed model also follows a set of initialisation pre-training procedures.
Firstly, the Encoder and Outcome Predictor are pre-trained according to a classification task (˜ y
=
P ( E ( x ))) , with corresponding loss L pred .
Latent state representations E ( x ) are clustered through a K-means algorithm with K clusters across the whole training set.
Cluster representation vectors are initialised as given by the resulting cluster centroids, and finally the Cluster Identifier network is pre-trained to identify clusters as predicted by the K-Means algorithm with categorical cross-entropy loss.
Implementation was completed in Python, with TensorFlow 2, scikit-learn and NumPy.
All experiments were run with 1 Tesla v100 GPU, and 8 CPUs Intel(R) Xeon(R)
Gold 6246 @ 3.30GHz.
R ESULTS
For Benchmark purposes, we considered TSKM as a classic clustering benchmark, and SOM-VAE and AC-TPC as state-of-the-art phenotypic clustering methods.
AC-TPC considers temporal subsequences of a complete patient set of observations - for comparison purposes, we consider only the model output for the complete patient sequence.
For simplicity, we present results with all input features considered.
All models were trained on the same training set (60% of the complete input data) and evaluated against the same test set (remaining 40%).
For DL models, we further split the training set into a purely training and validation sets.
All experiments with varying hyper-parameters were repeatedtimes with a fixed set of 10 distinct seeds, and results are reported according to average metric and standard deviation.
A complete list of the hyper-parameters considered for each model is included in Table A7 in the Appendix.
In bold, top-performing hyper-parameters are indicated.
Optimal integer hyper-parameters ( K , l ) were selected according to an “Occam’s Razor” approach - for each parameter, we assign it the highest value such that increasing this amount does not lead to a significant increase in performance according to mean AUROC and a Friedman’s hypothesis test.
Neural network size parameters were kept consistent across all DL models where applicable.
All other optimal hyper-parameters were selected according to highest AUROC performance conditional on the model predicting each class (e.g. not making predictions solely for no event or Death Events).
We evaluated clustering performance through standard clustering metrics, including Silhouette score (SIL, Rousseeuw (1987)), Davies-Bouldin Index (DBI, Davies (1979)), Variance Ratio Criterion (VRI, Cali´nski & Harabasz (1974)).
Results for all clustering models are displayed in Table 3.
In Table 4, we evaluated the (multi-class) prediction performance with regards to Area-under-theReceiver-Operating-Curve (AUROC), unweighted mean F1-score, unweighted mean Recall, and
Normalised Mutual Information (NMI).
For purely unsupervised models (SOM-VAE and TSKM), an outcome predictive pipeline was constructed by assigning patient admissions to clusters, and consequently to the empirical outcome distribution in the corresponding cluster.
The prediction task was also benchmarked against traditional classifiers for outcome prediction in EHR data, namely
Support Vector Machines (SVM), XGBoost (XGB) and NEWS2 (i.e., the National Early Warning
Score used in the UK hospitals).
Furthermore, to evaluate other Neural Network models as benchmarks and also justify both proposed mechanisms, we furthermore considered two other benchmarks: a) ENC-PRED: a stacked LSTM Encoder, followed by a MLP network for outcome prediction, and b) ATTEP; a model equivalent to CAMELOT, except the original entropy loss is considered over the clustering dist loss.
Supervised performance for all the above models is included in Table4.
As noted, convergence was particularly difficult for ATTEP due to cluster collapse.
[Table] [Table]
On top of performance evaluation with regards to clustering separability and outcome prediction, we display a comparison between the learnt cluster phenotypes of the proposed model and that of phenotypic clustering benchmark AC-TPC.
For each cluster, the corresponding outcome propensity
P ( c ) is shown as a bar plot over the 4 possible outcomes with corresponding probability value.
We also display cluster outcome propensity plots for both TSKM and SOM-VAE in the Appendix (Figures A.8 and A.9).
We note that the cluster outcome propensity distributions learnt by CAMELOT also align with the empirical outcome relevance in the learnt clusters (Table A.12).
Lastly, we also display feature-time cluster relevance attention maps in Figure 6.
For each cluster, a patient was randomly selected from the set of patients in the corresponding cluster, and a corresponding featuretime attention matrix and visualised as a heatmap.
D ISCUSSION
Our proposed model shows an improvement in clustering performance ( see Table 3) when compared to the current phenotypic clustering benchmark (AC-TPC), and outperforms SOM-VAE according [Figure] [Figure] to VRI.
Although the cluster separability metrics are superior in the case of TSKM, this is expected given a metric bias towards convex clusters, and the convexity of the K-Means based algorithm.
In particular, DL clustering occurs in a latent space, which, unfortunately, is not easily comparable with an algorithm targetting the input space (such as K-Means).
Furthermore, we argue clusters learnt by
TSKM are less relevant that our model’s as a) TSKM clusters are extremely hard to distinguish with regards to outcome propensity (as evidenced by very low performance on a prediction task (Table 4)), and b) TSKM clusters are less separable with regards to trajectory evolution, as there is less separation of mean HR trajectories, and less cluster separation when data is projected to a two-dimensional domain with t-stochastic neighbour embedding (tSNE) - both figures are in the Appendix,
Figures A10 and A11. With regards to predictive power, it can be seen in Table 4 that our model outperforms both standard classifiers (at least 8%) and benchmarking clustering methods (at least 5%) according to mean AUROC.
A similar increase can be seen in other classification metrics, with the exception of F 1 -score, where model performance is slightly below to that of AC-TPC.
Our model is able to more accurately determine patterns in the data than the previously proposed models, as EHR data is extremely complex and heterogeneous.
It is particularly promising that the model obtains good predictive task results despite a clustering bottleneck (i.e. sample predicted outcomes are done through the assigned cluster, as opposed to tailored to the precise input data).
While it is possible that other models could show better performance on the direct task of outcome prediction given EHR input, such models can potentially be associated with lack of robustness or input sensitivity difficulties.
Furthermore, it is likely they would struggle with identifying relevant trends and properties of clinical interest.
As such, for new admissions, such models could provide a prediction for the overall outcome, but no robust understanding of how this outcome will occur, and how to prevent potential risks of deterioration, let alone the ability to pool data from other similar patients.
Figure 5 shows the advantage of two key aspects of our methodology.
Our model identifies clear, separable cluster outcome distribution and provides a useful layer of interpretability to clinicians to understand a potential risk of deterioration.
Our model also identifies a more diverse set of cluster outcomes than AC-TPC, which only picks up 3 different cluster outcome distributions, and doesn’t identify the presence of the ”ICU” and ”Cardiac” classes. We also show other cluster-phenotype benchmark results in Appendix A.8 and A.9.
With regards to clusters learnt by CAMELOT, clusters 0 and 3 are the clusters with most ill cohort - they are largely representative of death and cardiac events on the subsequent 24 hours. On the other hand, while cluster 2 , 5 are healthier, with a smaller chance of adverse events. Clusters 1 and 4 are largely “healthy” cluster, with reduced risks of the most intense adverse events.
We note cluster outcome propensity distributions learnt by AC-TPC are unable to provide this level of detailed information. Furthermore, the propensity distribution learnt by our model matches with the empirical number outcome events observed in each cluster (displayed in Appendix Table A12).
Note, furthermore, that the model managed to successfully navigate a heavy class-imbalance setting.
Representative clusters are able to capture different-sized sub-populations, yet still identify potential risks of deterioration.
On the other hand, learnt cluster attention maps introduced in Figure 6 introduce yet another layer of interpretability to our proposed clustering model.
The personalised attention maps highlight the relevant feature-time pairs driving patient cluster assignment, and can be used to identify the most important clinical variables.
For instance, analysing Figure 5 suggests clusters with highest propensity for either of Death or Cardiac Arrest events are Cluster 0, 3 (and 2 to a slightly smaller extent).
This reflects in the resulting attention maps, where SBP and FIO 2 are highlighted as key clinical variables for cluster assignment. This conclusion is further corroborated when considering descriptive statistics of CAMELOT clusters (Table A13), as SBP and FIO 2 are some of the few variables with some significant separation across clusters, and when considering trajectory evolutions (Figure A14). Lastly, note that feature-time weights are also relevant if potential deterioration events did not take place - so that we are more confident on a patient’s health status.
As an example, attention maps for clusters 1 and 4 (reasonably healthy clusters) indicate FIO 2 as very relevant towards the latter stages of the admission - this is likely due to these patients not showing an increase in oxygen intake (as they did not need it). Thus, attention maps can be very versatile.
C ONCLUSION AND F UTURE W ORK
In this work, we propose a novel deep learning model for the task of identification of phenotypically separable clusters applied to EHR data for.
As part of our proposed model, we propose 2 distinct loss functions and introduce a novel feature-time attention layer to better represent patient data and to introduce a feature-time relevance map for each cluster.
Our experiments show promising results with the addition of both methodological tools above, on both cluster separability and outcome prediction performance.
The addition of the feature-time layer has the added benefit of introducing key interpretability tools for researchers to understand relevant regions for good patient physiology representation as well as an indication of what can lead to patient deterioration.
There are multiple interesting avenues of investigation building on this work.
On the other hand, the current attention layer mechanisms could potentially be improved with the addition of temporal weight smoothness, or, alternatively, weight regularization to encourage exploration of the complete feature-time space.
Alternatively, cluster selection through a neural network mechanism introduces high capacity at the potential cost of robustness and cluster collapse.
Potentially, more traditional methods incorporated into a similarly complex pipeline can achieve better performing through a clearer identification of cluster regions in latent space.
Furthermore, methodological improvements will also benefit from a more extensive testing across other diverse datasets and other potential areas of application.
A A PPENDIX A. D ATA
A description of the complete pipeline of data re-processing, following the protocol defined in Pimentel et al.
(2019), is shown in Figure A.1.
A.1: HAVEN processing
A total of 26 input features were considered.
Firstly, 4-hourly vital-sign sets which includedfeatures: Heart Rate (HR), Respiratory Rate (RR), Systolic Blood Pressure (SBP), Diastolic Blood
Pressure (DBP), peripheral Oxygen Saturation (SpO 2 ), Temperature (TEMP), level of consciousness via the AVPU scale - Alert, Verbal, Pain, Unresponsive - and estimated Fraction of Inspired Oxygen (FiO 2 , available when an oxygen mask is applied to the patient).
Each set consisted of a timestamp and the vital-sign numerical values.
Secondly, 4 demographic variables were selected (modelled as static variables): age, sex, and admission type (elective or surgical).
Thirdly, we included 6 features resulting from biochemistry blood tests, denoted as ’Serum’:
Serum levels of urea, albumin, creatinine, sodium, potassium and C-reactive protein.
Finally, 8 haematological blood test features were also included: white and haemaglobin cell counts, concentration of eosinophils, basophils, neutrophils, and lymphocytes, as well as eosinophil-to-basophil and neutrophil-to-lymphocyte ratios.
These features were selected based on domain knowledge of features related to severity in the prognosis and outcome of T2RF inpatients.
Descriptive statistics for all input variables is described in Table A.2.
Median and inter-quartile range (IQR) is displayed for continuous and categorical variables, while binary variables are shown according to number of counts in the dataset and corresponding cohort proportion.
Statistics are displayed for the complete data (”All”), but also for each sub-cohort defined by the overall outcome.
We can observe that these sub-cohorts are not clearly separable and are hard to identify solely from this information.
A summary of the patient cohort in relation to outcomes and target phenotypes can be seen in
Table A.3.
Challenges with regards to obtaining phenotypically separable clusters can similarly be observed - there is no clear significant difference between the target outcome sub-cohorts with regards to demographic input variables.
With regards to outcome distribution, we also note the high degree of imbalance in the dataset - the large majority of the patients in our dataset suffered from no adverse events (over 86 %), while only 48 had a Cardiac event, and 76 were re-directed to the ICU.
The lack of outcome sub-cohort separability can further observed in a temporal domain.
Figures
A.4, A.5, A.6 plot the mean trajectories of different temporal variables sets for each outcome subcohort, respectively, according to vital signs, haematological and serum features.
Mean is calculated based on the time to outcome, and missing observations are disregarded and ignored.
[Table]
[Table]
A list indicating the grid-search range of hyper-parameters considered in our experiments are indicated in Table A.7.
For simplicity, we define P := { 0 . 001 , 0 . 01 , 0 .
1 , 1 , 10 } , L := { 32 , 64 , 128 , 256 } and K := { 3 , ..., 20 } .
In bold, top-performing hyper-parameters according to target metrics defined in Section 4 are highlighted.
A. R ESULTS C OMPARISON
In Figures A.8 and Figures A.9 we display cluster outcome propensity distributions for some of our experiments with benchmark clustering models SOM-VAE and TSKM, respectively.
Both models do not naturally associate clusters with a distribution - we estimate the cluster outcome as the empirical outcome distribution for the patient cohort assigned to the corresponding cluster.
We note that clusters learnt by both clustering benchmark models have identical outcomes, which provides no useful clinical interpretability to the cluster-defined populations, as well as likely not assisting models to learn relevant cluster representations We go further in comparing clusters learnt by TSKM and by our proposed model. We argue clusters learnt by CAMELOT are much more relevant towards our goal. We show this through two distinct plots.
Firstly, in Figure A10, we display a scatter plot of patients in each cluster (CAMELOT on the right and TSKM clusters on the left) after projection to two dimensions. Projection was completed through a principal component analysis reduction to 50 dimensions, followed by t-stochastic neighbour embedding dimensionality projection to two. Furthermore, we also demonstrated that TSKM does not learn as separable cluster trajectory evolution profiles as CAMELOT. This is shown in Figure A11, where Heart-Rate mean trajectories for each cluster (i
[Table]
e., average HR observations aligned to the same time until end of observations for patients in the clusters) are displayed. It is clear that CAMELOT cluster trajectories are easier to separate. A complete description of the number of patient admissions with a given outcome per learnt cluster in our proposed model can be seen in Table A.
[Table]
- - - { ’polynomial’ , ’rbf’ } - C - - - -
P ( 10 ) - n-estimators - - - - - { 100 , 200 , 300 } depth - - - - - { 1 , 3 , 5 , 10 } min-child-weight - - - - - { 1 , 2 , 3 , 5 }
[Table]
We also computed summary statistics for the learnt CAMELOT clusters. For each of the resulting clusters, median, and quartile values were computed and plotted, except on the case of binary variables, where only the number of positive occurrences (and the corresponding proportion in the cluster) are shown. Lastly, we plot the mean cluster trajectory evolution for SBP and FIO2 to present supportive evidence for the personalised attention maps in Figure 6. These two features were selected from attention map analysis.