V ARIATIONAL PSOM: D EEP P ROBABILISTIC C LUSTERING WITH S ELF -O RGANIZING M APS
Anonymous authors
Paper under double-blind review
A BSTRACT
Generating visualizations and interpretations from high-dimensional data is acommon problem in many ﬁelds.
Two key approaches for tackling this problemare clustering and representation learning.
There are very performant deep clustering models on the one hand and interpretable representation learning techniques,often relying on latent topological structures such as self-organizing maps, on theother hand.
However, current methods do not yet successfully combine these twoapproaches.
We present a new deep architecture for probabilistic clustering, VarPSOM and its extension to time series data, VarTPSOM.
We show that they achievesuperior clustering performance compared to current deep clustering methods onstatic MNIST/Fashion-MNIST data as well as medical time series, while inducingan interpretable representation.
Moreover, on the medical time series, VarTPSOMsuccessfully predicts future trajectories in the original data space
I NTRODUCTION
Information visualization techniques are essential in areas where humans have to make decisionsbased on large amounts of complex data.
Their goal is to ﬁnd an interpretable representation ofthe data that allows the integration of humans into the data exploration process.
This encouragesvisual discoveries of relationships in the data and provides guidance to downstream tasks.
In thisway, a much higher degree of conﬁdence in the ﬁndings of the exploration is attained (Keim, 2002).
An interpretable representation of the data, in which the underlying factors are easily visualized, isparticularly important in domains where the reason for obtaining a certain prediction is as valuableas the prediction itself.
However, ﬁnding a meaningful and interpretable representation of complexdata can be challenging.
Clustering is one of the most natural ways for retrieving interpretable information from raw data.
Long-established methods such as k-means (MacQueen, 1967) and Gaussian Mixture Models(Bishop, 2006) represent the cornerstone of cluster analysis.
Their applicability, however, is oftenconstrained to simple data and their performance is limited in high-dimensional, complex, real-worlddata sets, which do not exhibit a clustering-friendly structure.
Deep generative models have recently achieved tremendous success in representation learning.
Some of the most commonly used and efﬁcient approaches are Autoencoders (AEs), Variational
Autoencoders (VAEs) and Generative Adversarial Networks (GANs) (Kingma & Welling, 2013;
Goodfellow et al., 2014).
The compressed latent representation generated by these models has beenproven to ease the clustering process (Aljalbout et al., 2018).
As a result, the combination of deepgenerative models for feature extraction and clustering results in a dramatic increase of the clusteringperformance (Xie et al., 2015).
Although very successful, most of these methods do not investigatethe relationship among clusters and the clustered feature points live in a high-dimensional latentspace that cannot be easily visualized or interpreted by humans.
The Self-Organizing Map (SOM) (Kohonen, 1990) is a clustering method that provides such aninterpretable representation.
It produces a low-dimensional (typically 2-dimensional), discretizedrepresentation of the input space by inducing a ﬂexible neighbourhood structure over the clusters.
Alas, its applicability is often constrained to simple data sets similar to other classical clusteringmethods.
To resolve the above issues, we propose a novel deep architecture, the Variational Probabilistic SOM(VarPSOM), that jointly trains a VAE and a SOM to achieve an interpretable discrete representationwhile exhibiting state-of-the-art clustering performance.
Instead of hard assignment of data pointsto clusters, our model uses a centroid-based probability distribution.
It minimizes its KullbackLeibler divergence against an auxiliary target distribution, while enforcing a SOM-friendly space.
To highlight the importance of an interpretable representation for different purposes, we extendedthis model to deal with temporal data, yielding VarTPSOM.
We discuss related work in Section2.
Extensive evidence of the superior clustering performance of both models, on MNIST/FashionMNIST images as well as real-world medical time series is presented in Section 4.
Our main contributions are:
• A novel architecture for deep clustering, yielding an interpretable discrete representationthrough the use of a probabilistic self-organizing map.
• An extension of this architecture to time series, improving clustering performance on this
data type and enabling temporal predictions.
• A thorough empirical assessment of our proposed models, showing superior performanceon benchmark tasks and challenging medical time series from the intensive care unit.
R ELATED W ORK
Self-Organizing Maps have been widely used as a means to visualize information from largeamounts of data (Tirunagari et al., 2014) and as a form of clustering in which the centroids are connected by a topological neighborhood structure (Flexer, 1999).
Since their early inception, severalvariants have been proposed to enhance their performance and scope.
The adaptive subspace SOM,
ASSOM (Kohonen, 1995), for example, proposed to combine PCA and SOMs to map data into areduced feature space.
(Tokunaga & Furukawa, 2009) combine SOMs with multi-layer perceptronsto obtain a modular network.
(Liu et al., 2015) proposed Deep SOM (DSOM), an architecture composed of multiple layers similar to Deep Neural Networks.
There exist several methods tailoredto representation learning on time series, among them (Franceschi et al., 2019; Fortuin & R ¨ atsch,2019; Fortuin et al., 2019), which are however not based on SOMs.
Extensions of SOM optimizedfor temporal data include the Temporal Kohonen map (Chappell & Taylor, 1993) and its improvedversion Recurrent SOM (McQueen et al., 2004) as well as Recursive SOM (Voegtlin, 2002).
While SOM and its variants are particularly effective for data visualization (Liu et al., 2015), it was rarelyattempted to combine their merits in this respect with modern state-of-the-art clustering methods,which often use deep generative models in combination with probabilistic clustering.
In particular, recent works on clustering analysis have shown that combining clustering algorithmswith the latent space of AEs greatly increases the clustering performance (Aljalbout et al., 2018).(Xie et al., 2015) proposed DEC, a method that sequentially applies embedding learning using
Stacked Autoencoders (SAE), and the Clustering Assignment Hardening method on the obtainedrepresentation.
An improvement of this architecture, IDEC (Guo et al., 2017), includes the decodernetwork of the SAE in the learning process, so that training is affected by both the clustering lossand the reconstruction loss.
Similarly, DCN (Yang et al., 2016) combines a k-means clustering losswith the reconstruction loss of SAE to obtain an end-to-end architecture that jointly trains representations and clustering.
These models achieve state-of-the-art clustering performance but they donot investigate the relationship among clusters.
An exception is the work by (Li et al., 2018), inwhich they present an unsupervised method that learns latent embeddings and discovers multi-facetclustering structure.
Relationships among clusters were discovered, however, they do not provide alatent space that can be easily interpreted and which eases the process of analytical reasoning.
While there exist previous efforts to endow VAEs with a hierarchical latent space (Vikram et al.
Goyal et al., 2017), to the best of our knowledge, only two models used deep generativemodels in combination with a SOM structure in the latent space.
The SOM-VAE model (Fortuinet al., 2018), inspired by the VQ-VAE architecture (van den Oord et al., 2017) (which itself waslater extended in (Razavi et al., 2019)), uses an AE to embed the input data points into a latent spaceand then applies a SOM-based clustering loss on top of this latent representation.
It features hardassignments of points to centroids, as well as the use of a Markov model for temporal data, bothof which yield inferior expressivity compared to our method.
The Deep Embedded SOM, DESOM
(Forest et al., 2019), improved the previous model by using a Gaussian neighborhood window withexponential radius decay and by learning the SOM structure in a continuous setting.
Both methodsfeature a topologically interpretable neighborhood structure and yield promising results in visualizing state spaces.
However, those works did not feature empirical comparisons to state-of-the-artdeep clustering techniques and did not make use of many of the design principles that have recentlyproven to be successful in this space.
P ROBABILISTIC CLUSTERING WITH V ARIATIONAL PSOM
Given a set of data samples { x i } i =1 ,...,n , where x i ∈ R d , the goal is to partition the data into a setof clusters { S i } i =1 ,...,K , while retaining a topological structure over the cluster centroids.
The proposed architecture for static data is presented in Figure 1a.
The input vector x i is embeddedinto a latent representation
z i using a VAE.
This latent vector is then clustered using PSOM , anew SOM clustering strategy that extends the Clustering Assignment Hardening method (Xie et al.,2015).
The VAE and PSOM are trained jointly to learn a latent representation with the aim to boostthe clustering performance.
To prevent the network from outputting a trivial solution, the decodernetwork reconstructs the input from the latent embedding, encouraging it to be as similar as possibleto the original input.
The obtained loss function is a linear combination of the clustering loss andthe reconstruction loss.
To deal with temporal data, we propose another model variant, which isdepicted in Figure 1b.
[Figure]
(a) VarPSOM architecture for cluster- ing of static data. Data points x i are mapped to a continuous embedding z i using a VAE (parameterized by Φ ).
The loss function is the sum of a SOM- based clustering loss and the ELBO.
(b) VarTPSOM architecture, composed of VarPSOM modules con- nected by LSTMs across the time axis, which predict the continuous embedding z t +1 of the next time step. This architecture allows to unroll future trajectories in the latent space as well as the original data space by reconstructing the x t using the VAE.
3. B ACKGROUND
A Self-Organizing Map is comprised of K nodes connected to form a grid M ⊆
N 2 , where thenode m i,j , at position ( i, j ) of the grid, corresponds to a centroid vector, µ i,j in the input space.
Thecentroids are tied by a neighborhood relation N ( µ i,j ) =
{ µ i − 1 ,j , µ i +1 ,j , µ i,j − 1 , µ i,j +1 } .
Given a 116 random initialization of the centroids, the SOM algorithm randomly selects an input x i and updatesboth its closest centroid µ i,j and its neighbors N ( µ i,j ) to move them closer to x i .
For a completedescription of the SOM algorithm, we refer to the appendix (A).
The Clustering Assignment Hardening method has been recently introduced by the DEC model (Xie et al., 2015) and was shown to perform well in the latent space of AEs (Aljalbout et al., 2018).
Given an embedding function z i = f
( x i ) , it uses a Student’s t-distribution ( S ) as a kernel to measure the
[Equation]
It improves the cluster purity by enforcing the distribution S to approach a target distribution, T :
[Equation]
By taking the original distribution to the power of γ and normalizing it, the target distribution puts more emphasis on data points that are assigned a high conﬁdence.
We follow (Xie et al., 2015) in choosing γ =2, which leads to larger gradient contributions of points close to cluster centers, as they show empirically.
The resulting clustering loss is deﬁned as:
[Equation]
3. P ROBABILISTIC SOM (PSOM) CLUSTERING
Our proposed clustering method, called PSOM, expands Clustering Assignment Hardening to include a SOM neighborhood structure over the centroids.
We add an additional loss to (1) to achieve an interpretable representation.
This loss term maximizes the similarity between each data point and the neighbors of the closest centroids.
For each embedded data point z i and each centroid µ j the loss is deﬁned as the negative sum of all the neighbors of µ j , { e : µ e ∈ N ( µ j ( x i )) } , of the probability that z i is assigned to e , deﬁned as s ie .
This sum is weighted by the similarity s ij between z i and the centroid µ j :
[Equation]
The complete PSOM clustering loss is then:
[Equation]
We note that for β =
it becomes equivalent to Clustering Assignment Hardening.
3. V AR PSOM: VAE FOR FEATURE EXTRACTION
In our method, the nonlinear mapping between the input x i and embedding z
i is realized by a VAE.
Instead of directly embedding the input x i into a latent embedding z i , the VAE learns a probability distribution q φ ( z | x i ) parametrized as a multivariate normal distribution whose mean and variance are ( µ φ , Σ φ ) =
f φ ( x i ) .
Similarly, it also learns the probability distribution of the reconstructed output given a sampled latent embedding, p θ ( x i | z ) where ( µ θ , Σ θ ) =
f θ ( z i ) .
Both f φ and f θ are neural networks, respectively called encoder and decoder.
The ELBO loss is:
[Equation]
where p ( z ) is an isotropic Gaussian prior over the latent embeddings.
The second term can be interpreted as a form of regularization, which encourages the latent space to be compact.
For each data point x i the latent embedding z
i is sampled from q φ ( z | x i ) .
Adding the ELBO loss to the
PSOM loss from the previous subsection, we get the overall loss function of VarPSOM:
[Equation]
To the best of our knowledge, no previous SOM methods attempted to use a VAE to embed theinputs into a latent space.
There are many advantages of a VAE over an AE for realizing our goals.
Its prior on the latent space encourages structured and disentangled factors (Higgins et al. , 2016),which could help clustering.
The suitability of VAEs for anomaly detection (An & Cho, 2015)means that points with a higher variance in the latent space could be treated as less accurate andtrustworthy.
The regularization term of the VAE can be used to prevent the network from scatteringthe embedded points discontinuously in the latent space, which naturally facilitates the ﬁtting of the SOM.
To test if the use of CNNs can boost clustering performance on image data, we introduceanother model variant called VarCPSOM, which uses convolutional ﬁlters as part of the VAE.
3. V AR TPSOM: E XTENSION TO TIME SERIES DATA
To extend our proposed model to time series data, we add a temporal component to the architecture.
Given a set of N time series of length T , { x t,i } t =1 ,...,T ; i =1 ,...,N , the goal is to learn interpretable trajectories on the SOM grid.
To do so, the VarPSOM could be used directly but it would treat each time step t of the time series independently, which is undesirable.
To exploit temporal information and enforce smoothness in the trajectories, we add an additional loss to (3):
[Equation]
where u i t ,i t +1 =
g ( z i,t , z
i,t +1 ) is the similarity between z i,t and z i,t +1 using a Student’s t- 133 distribution and z i,t refers to the embedding of time series x i at time index t .
It maximizes thesimilarity between latent embeddings of adjacent time steps, such that large jumps in the latent statebetween time points are discouraged.
One of the main goals in time series modeling is to predict future data points, or alternatively, future embeddings.
This can be achieved by adding a long short-term memory network (LSTM) across the latent embeddings of the time series, as shown in Fig 1b.
Each cell of the LSTM takes as input the latent embedding z t at time step t , and predicts a probability distribution over the next latent embedding, p ω ( z t +1 | z t ) .
We parametrize this distribution as a Multivariate Normal Distribution whose mean and variance are learnt by the LSTM.
The prediction loss is the log-likelihood between the learned distribution and a sample of the next embedding z t +1 :
[Equation]
The ﬁnal loss of VarTPSOM, which is trainable in a fully end-to-end fashion, is
[Equation]
E XPERIMENTS
First, we evaluate VarPSOM and VarCPSOM and compare them with state-of-the-art noninterpretable as well as SOM-based clustering methods on MNIST (Lecun et al., 1998) and FashionMNIST (Xiao et al., 2017) data.
Here, particular focus is laid on the comparison of VarPSOM andthe clustering models DEC and IDEC, to investigate the role of the VAE and the SOM loss.
Wethen present visualizations of the obtained 2D representations, to illustrate how our method couldease visual reasoning about the data.
Finally, we present extensive evidence of the performance of
VarTPSOM on real-world complex time series from the eICU data set (Pollard et al., 2018), andillustrate how it allows visualization of patient health state trajectories in an easily understandable2D domain.
For details on the data sets, we refer to the appendix (B.1).
Baselines We used two different types of baselines.
The ﬁrst category contains clustering methodsthat do not provide any interpretable discrete latent representation.
Those include k-means, the DECmodel, as well as its improved version IDEC, whose clustering methods are related to ours.
We alsoinclude a modiﬁed version of IDEC that we call VarIDEC, in which we substitute the AE with a
VAE, to investigate the role of the VAE.
For all these methods we use 64 clusters.
In the secondcategory, we include state-of-the-art clustering methods based on SOMs.
Here, we used a standard
SOM (minisom), AE+SOM, an architecture composed of an AE and a SOM applied on top of thelatent representation (trained sequentially), SOM-VAE and DESOM.
Finally, we create a modiﬁedversion of our model, called AEPSOM, in which we substitute the VAE with an AE (similarly to
VarIDEC).
For all SOM-based methods we set the SOM grid size to (8 × 8) .
For different gridconﬁgurations we refer to the appendix, (B.3).
Implementation In implementing our models we focused on retaining a fair comparison with thebaselines.
Hence we decided to use a standard network structure, with fully connected layers ofdimensions d − 500 − 500 − 2000 − l , to implement both the VAE of our models and the AE of thebaselines.
The latent dimension, l , is set to 100 for the VAE, and to 10 for the AEs.
Since the priorin the VAE enforces the latent embeddings to be compact, it also requires more dimensions to learn
[Table]
a meaningful latent space. On the other hand, providing the AE models with a higher-dimensionallatent space, needed for the VAE, resulted in a dramatic decrease of performance (see appendix B.2).
VarCPSOM is composed of 4 convolutional layers of feature maps [32 , 64 , 128 , 256] and kernel size3 × 3 for all layers.
For all architectures, no greedy layer-wise pretraining was used to tune the
VAE.
Instead we simply run the VAE without the clustering loss for a few epochs for initialization.
A standard SOM was then used to produce an initial conﬁguration of the centroids/neighbourhoodrelation.
Finally, the entire architecture is trained for 100 , 000 iterations.
To avoid ﬁne-tuning hyperparameters, given the unsupervised setting, α is set to 10 for all experiments while the otherhyperparameters are modiﬁed accordingly to maintain the same order of magnitude of the differentloss components.
Clustering Evaluation Table 1 shows the clustering quality results of VarPSOM and VarCPSOMon MNIST and Fashion-MNIST data, compared with the baselines.
Purity and Normalized Mutual
Information are used as evaluation metrics.
We observe that our proposed models outperform thebaselines of both categories and achieve state-of-the-art clustering performance.
VarPSOM vs. IDEC VarPSOM is inspired by IDEC but it has two major differences.
It uses a
VAE instead of an AE and it improves interpretability in the latent space by adding a new loss that enforces a SOM structure.
Since both VarIDEC and VarPSOM show superior clustering performancecompared to IDEC and AEPSOM respectively (Table 1), we conclude that the VAE indeed succeeds in capturing a more meaningful latent representation compared to a standard AE.
Regardingthe second difference, the SOM structure was expected to slightly decrease the clustering performance, due to a trade-off between interpretability and raw clustering performance.
However, we donot observe this in our results.
Adding the SOM loss rather leads to an increase of the clusteringperformance.
We suspect this is due to the regularization effect of the SOM’s topological structure.
Overall, VarPSOM outperforms both DEC and IDEC.
Improvement over Training After obtaining the initial conﬁguration of the SOM structure, bothclustering and feature extraction using the VAE are trained jointly.
To illustrate that our architectureimproves clustering performance over the initial conﬁguration, we plotted NMI and Purity againstthe number of training iterations in Figure 2.
We observe that the performance is stable whenincreasing the number of epochs and no overﬁtting is visible.
Role of the SOM loss To investigate the inﬂuence of the SOM loss component, we plot the clustering performance of VarPSOM against the weight ( β ) of L SOM in Fig. 3, using the MNIST dataset.
With β = 30 , the KL term (responsible for improving clustering purity) and the L SOM term (responsible for enforcing a SOM structure over the centroids) are almost equal.
It is interesting to observethe different trends in NMI and purity.
The NMI performance increases for increasing values of βwhile purity slightly decreases.
Overall, enforcing a more interpretable latent space results in a morerobust clustering model with higher NMI clustering performance.
Time Series Evaluation We evaluate the clustering performance of our proposed models on theeICU dataset, comprised of complex medical time series.
We compare them against SOM-VAE,as this is the only method among the baselines that is suited for temporal data.
Table 2 shows thecluster cell enrichment in terms of NMI for three different labels, the current (APACHE-0) and worstfuture (APACHE-6/12 hours) physiology scores.
VarTPSOM clearly achieves superior clusteringperformance compared to SOM-VAE.
This, we hypothesize, is due to the better feature extractionusing a VAE as well as the improved treatment of uncertainty using PSOM, which features softassignments, whereas SOM-VAE contains a deterministic AE and hard assignments.
Moreover,both the smoothness loss and the prediction loss seem to increase the clustering performance.
Moreresults on ICU time series are reported in the appendix (B.4).
To quantify the performance of VarTPSOM in unrolling future trajectories, we predict the ﬁnal6 latent embeddings of each time series.
For each predicted embedding we reconstruct the inputusing the decoder of the VAE.
Finally, we measure the MSE between the original input and thereconstructed inputs for the last 6 hours of the ICU admission.
As baselines, we used an LSTM thattakes as input the ﬁrst 66 hours of the time series and then predicts the next 6 hours.
Since mostof the trajectories tend to stay in the same state over long periods of time, another strong baselineis obtained by duplicating the last seen embedding over the ﬁnal 6 hours.
The results (Table 3)indicate that the joint training of clustering and prediction used by VarTPSOM clearly outperformsthe 2 baselines.
[Table] [Table]
[Figure]
Interpretability To illustrate the topological structure in the latent space, we present reconstructions of the VarPSOM centroids, arranged in a (8 × 8) grid, on static MNIST/Fashion-MNIST datain Figure 4.
On the ICU time series data [Table] we show example trajectories for one patient dying at theend of the ICU stay, as well as two control patients which are dispatched healthily from the ICU.
Weobserve that the trajectories are located in different parts of the SOM grid
[Table]
and form a smooth andinterpretable representation (Fig. 5).
For further results, including a more quantitative evaluationusing randomly sampled trajectories, enrichment for future mortality as well as an illustration ofhow the uncertainty generated by the soft assignments can help in data visualization, we refer to theappendix (B.5).
C ONCLUSION
We presented two novel methods for interpretable unsupervised clustering, VarPSOM and VarTPSOM.
Both models make use of a VAE and a novel clustering method, PSOM, that extends theclassical SOM algorithm to include a centroid-based probability distribution.
Our models achievesuperior clustering performance compared to state-of-the-art deep clustering baselines on benchmark data sets and real-world medical time series.
The use of a VAE for feature extraction, insteadof an AE, used in previous methods, and the use of soft assignments of data points to clusters resultin an interpretable model that can quantify uncertainty in the data.
[Figure]
A PPENDIXA S ELF -O RGANIZING M APS
Among various existing interpretable unsupervised learning algorithms, Kohonen’s self-organizingmap (SOM) (Kohonen, 1990) is one of the most popular models.
It is comprised of K neuronsconnected to form a discrete topological structure.
The data are projected onto this topographic mapwhich locally approximates the data manifold.
Usually it is a ﬁnite two-dimensional region whereneurons are arranged in a regular hexagonal or rectangular grid.
Here we use a grid, M ⊆ N 2 ,because of its simplicity and its visualization properties.
Each neuron m ij , at position ( i, j ) of thegrid, for i, j = 1 , . .
. , √ K , corresponds to a centroid vector, µ i,j in the input space.
The centroidsare tied by a neighborhood relation, here deﬁned as N ( µ i,j ) = { µ i − 1 ,j , µ i +1 ,j , µ i,j − 1 , µ i,j +1 } .
Given a random initialization of the centroids, the SOM algorithm randomly selects an input x i andupdates both its closest centroid µ i,j and its neighbors N ( µ i,j ) to move them closer to x i .
Thealgorithm (1) then iterates these steps until convergence.
Algorithm
Require: 0 < α ( t ) < 1;
[Equation]
repeat
At each time t , present an input x ( t ) and select the winner,
[Equation]
Update the weights of the winner and its neighbours,
[Equation]
until the map converges
The range of SOM applications includes high dimensional data visualization, clustering, imageand video processing, density or spectrum proﬁle modeling, text/document mining, management systems and gene expression data analysis
B E XPERIMENTAL AND IMPLEMENTATION DETAILSB. D ATASETS
• MNIST:
It consists of 70000 handwritten digits of 28-by-28 pixel size.
Digits range from0 to 9, yielding 10 patterns in total.
The digits have been size-normalized and centered in aﬁxed-size image Lecun et al.
(1998).• Fashion MNIST:
A dataset of Zalando’s article images consisting of a training set of60,000 examples and a test set of 10,000 examples Xiao et al.
Each example isa 28 × 28 grayscale image, associated with a label from 10 classes.• eICU
For temporal data we use vital sign/lab measurements of intensive care unit (ICU)patients resampled to a 1-hour based grid using forward ﬁlling and ﬁlling with populationstatistics from the training set if no measurements were available.
From all ICU stays, we excluded ICU stays, which were shorter than 1 day, longer than 30 days or which had atleast one gap in the continuous vital sign monitoring, which we deﬁne by a interval between2 HR measurements of at least 1 hour.
This yielded N =
ICU stays from the eICU database.
d vitals = 14 vital sign variables and d lab = 84 lab measurement variableswere included, giving an overall data dimension of d = 98 .
The last 72 hours of these multivariate time series were used for the experiments.
As labels we use a variant of the
current dynamic APACHE physiology score (APACHE-0) as well as the worst APACHEscore in the next 6 and 12 hours (APACHE-6/12), and the mortality in the next 24 hours.
Only those variables from the APACHE score deﬁnition which are recorded in the eICUdatabase were taken into account.
Each dataset is divided into training, validation and test sets for both our models and the baselines.
B. L ATENT SPACE DIMENSION
We evaluate the DEC model for different latent space dimensions.
Table S1 shows that the AE, usedin the DEC model, performs better when a lower dimensional latent space is used.
[Table]
B. N UMBER OF CLUSTERS
We evaluate the NMI and purity clustering performance of our model, VarPSOM, with a varyingnumber of clusters on the MNIST dataset.
Since IDEC represents the main competitor we includeit in this analysis.
Figure S1 shows that VarPSOM outperforms IDEC for all the different conﬁgurations. In particular, it is interesting to observe that NMI decreases with an increasing number ofclusters in both models. This is because the entropy of the clustering increases with the number ofclusters. B. L EARNING HEALTH STATE REPRESENTATIONS IN THE ICU
By enforcing a SOM structure, VarPSOM, as well as VarTPSOM, project the cluster centroids ontoa discrete 2D grid.
Such a grid is particularly suited for visualization purposes and relations betweencentroids become immediately intuitive.
In Fig.
S2 a heat-map (colored according to enrichment inthe current APACHE score, as well as mortality risk in the next 24 hours) shows compact enrichmentstructures.
VarTPSOM succeeds in creating a meaningful and smooth neighbourhood structure.
Itdistinguishes risk proﬁles with practically zero mortality risk from high mortality risk, reaching upto
≈ 15 %, in different regions of the map, even though it is learned in a purely unsupervised fashion.
Remarkably, the two heat-maps (S2b and S2a) show different enrichment patterns.
Clusters whichare enriched in health states with higher APACHE scores often do not correspond exactly to clusterswith a higher mortality risk.
This suggests that traditional representations of physiologic values,such as the APACHE score, fail to fully use all complex multivariate relationships present in the
ICU recordings, and are not associated with dynamic mortality in a simple way.
[Figure]
B. V ISUALIZING HEALTH STATE TRAJECTORIES IN THE ICU
To analyze the trend of the patient pathology, VarTPSOM induces trajectories on the 2D SOM gridwhich can be easily visualized.
Fig.
S3 shows 20 randomly sampled patient trajectories obtainedby our model.
Trajectories ending in the death of the patient are shown in red, healthily dispatchedpatients are shown in green.
One of the main advantage of VarTPSOM over the traditional SOM algorithm is the use of softassignments of data points to clusters which results in a better ability to quantify uncertainty in thedata.
For visualizing health states in the ICU, this property is very important.
In Fig S4 we plot anexample patient trajectory, where 6 different time-steps (in temporal order) of the trajectory werechosen.
Our model yields a soft centroid-based probability distribution which evolves with time andwhich allows estimation of likely discrete health states at a given point in time.
For each time-stepthe distribution of probabilities is plotted using a heat-map, whereas the overall trajectory is plottedusing a black line.
The circle and cross indicate ICU admission and dispatch, respectively.