A bumpy journey: exploring deep Gaussian mixture models
Abstract
The deep Gaussian mixture model (DGMM) is a framework directly inspired by the ﬁnite mixture of factor analysers model (MFA) and the deep learning architecture composed of multiple layers.
The MFA is a generative model that considers a data point as arising from a latent variable (termed the score) which is sampled from a standard multivariate Gaussian distribution and then transformed linearly.
The linear transformation matrix (termed the loading matrix) is speciﬁc to a component in the ﬁnite mixture.
The DGMM consists of stacking MFA layers, in the sense that the latent scores are no longer assumed to be drawn from a standard Gaussian, but rather are drawn from a mixture of factor analysers model.
Thus the latent scores are at one point considered to be the input of an MFA and also to have latent scores themselves.
The latent scores of the DGMM’s last layer only are considered to be drawn from a standard multivariate Gaussian distribution.
In recent years, the
DGMM has gained prominence in the literature: intuitively, this model should be able to capture complex distributions more precisely than a simple Gaussian mixture model.
We show in this work that while the DGMM is an original and novel idea, in certain cases it is challenging to infer its parameters.
In addition, we give some insights to the probable reasons of this difﬁculty.
Experimental results are provided on github: https:
//github.com/ansubmissions/ICBINB , alongside an
R package that implements the algorithm and a number of ready-to-run R scripts.
Mixture of Factor Analysers
The mixture of factor analysers model [3] is an extension of the factor analysis model to a ﬁnite mixture of G factor analysis models.
We assume that an observation x i ∈ R J can be expressed through a latent random vector z i ∈ R R such that R ≤ J .
In addition, the latent labels { v 1 , . .
, v n } are assumed to be independent unobserved realisations of a categorical G -dimensional random vector, such that v ig = 1 indicates that x i is generated by the g th factor analyser.
The generative process of this model is as follows:
[Equation]
where g is the index of the element of v i that is equal to 1 , Λ g is a J × R matrix and η g ∈ R J is the mean vector of the g th factor analysis model.
In addition, ψ g is a diagonal covariance matrix of dimension J , and π = ( π 1 , . . .
, π G ) are the mixing proportions.
The latent variable v i =
( v i 1 , . . .
, v iG ) represents the partitions for observation i and p ( v ig = 1)
=
π g .
The marginal density of x i is then a Gaussian mixture model (GMM):
[Equation]
where θ =
( η g , Λ g , ψ g , π g ) g ∈{ 1 ,...,G } denotes all parameters and:
• f g ( . ; η g , Σ g ) is the Gaussian probability density function for component g with mean η g and covariance matrix Σ g and • Σ g = Λ g Λ Tg + ψ g .
Hence, the observed-data log-likelihood of N samples x
= ( x 1 , . .
, x N ) is given by:
l o ( θ ; x ) = N (cid:88) i = log G (cid:88) g = π g f g ( x i ; η g , Σ g ) .
Deep Gaussian Mixture Models 2.1 Coupling Deep Learning and Gaussian Mixture Models
One of the earliest papers to deﬁne a model that stacks MFA layers in a deep learning fashion is that of [6] where the authors refer to the model as the “Deep Mixture of Factor Analysers".
In this model, the layers are not fully-connected and the architecture is a tree: a neuron receives an input from only one neuron of the previous layer.
In [7], the authors deﬁne a DGMM model with fully-connected layers, but the matrices of weights Λ are assumed to be square.
In other words, there is no dimension reduction between each MFA layer, which leads to important unidentiﬁability issues.
In the model proposed by [9], instead of using the MFA model as a layer, the authors suggest using a mixture of factor analysers model with common factor loadings (MCFA) [1]: the matrices Λ g of the same layer are therefore assumed to be equal for all g .
Finally, [8] deﬁnes the deep Gaussian mixture model (DGMM) with dimension reduction at each layer and provides an EM algorithm for the inference of the parameters.
This present work aims at investigating the statistical efﬁciency of this algorithm.
2.2 Deﬁnition of the Deep Gaussian Mixture Model
This section relies on [8].
Let us assume that there are L layers and that the data x = ( x i ) i ∈{ 1 ,...,N } are generated as follows:
[Equation]
with prob.
π (1) g 1 , g
[Equation]
, G (1) }
[Equation]
with prob.
π (2) g 2 , g 2 ∈ { 1 , . . .
, G (2) }
[Equation]
with prob.
[Equation]
( l +
[Equation]
1 , g l +
[Equation] [Equation]
[Equation]
[Equation]
L ) i + u ( L ) i with prob.
π ( L ) g L , g L ∈ { 1 , . . .
, G ( L ) } , where:
• z ( L ) i is a vector of size R ( L ) and is assumed to be drawn from the Gaussian distribution
N ( 0 , I R ( L ) ) , • u ( l ) i is a speciﬁc random error that follows a Gaussian distribution with expectation 0 and covariance matrix ψ ( l ) g l , • η ( l ) g l is a vector of length R ( l ) , • Λ ( l ) g l is a matrix of dimension R ( l − 1) × R ( l ) , • J > . . .
> R ( l − 1) >
R ( l ) for all l .
The vectors u ( l ) i are assumed to be independent of the scores z ( l ) i .
From these considerations, we see that at each layer l , the conditional distribution of z ( l ) i given z ( l +1)
i is a mixture of Gaussian distributions such that:
[Equation]
[Equation]
Figure 1 represents the graphical model of this DGMM in the case of L = 2 .
Therefore, the DGMM has parameters θ =
( η ( l ) g l , Λ ( l ) g l , ψ ( l ) g l , π ( l ) g l ) l ∈{ 1 ,...,L } ; g l ∈{ 1 ,...,G ( l ) } .
It also has the latent variables v ( l ) and z ( l ) where v ( l ) corresponds to the partition of the l th layer.
In addition, it is important to notice that the marginal probability distribution f ( x i ; θ ) is a GMM where each component corresponds to one of the possible paths of the network.
Therefore, by noting
G as the set of all possible paths through the network, the DGMM can be written:
[Equation]
where g = ( g 1 , . .
. , g L ) is one of the possible paths of the network,
[Equation]
We refer to the GMM of Equation (3) as the “global GMM" of the DGMM so as not to confuse it with the GMMs of the layers as expressed in Equation (2).
In addition, the number of possible paths through the network is denoted as G .
In fact, Equation (3) can be generalised to all layers: the marginal distributions of all the scores z ( l ) i are Gaussian mixtures.
So, by noting ˜ G ( l ) as all the possible paths from the l th layer, and by integrating out the latent variables of the layers that follow the l th layer, we have:
[Equation]
where ˜ g ( l )
=
( g l +1 , . .
. , g L ) is a possible path of the network from the l th layer, [Equation] and
[Equation]
2.3 Inference of the model
Similar to MFA, the DGMM has parameters and latent variables to estimate.
Therefore, the EM algorithm [2] is a candidate to optimise the log-likelihood function with respect to the parameters and latent variables.
However, in the MFA model, the computation of the expectation of the complete data log-likelihood does not involve p ( z ; θ ) since z is assumed to be drawn from the standard Gaussian distribution N ( 0 , I ) .
For the DGMM, unless l = L , z ( l ) l is assumed to be drawn from an MFA with parameters ( π ( l +1)
g l +1 , η ( l +1)
g l +1 , Λ ( l +1) g l +1 , ψ ( l +1)
g l +1 )
g l +1 ∈{ 1 ,...,G ( l +1) } .
Thus, the parameters of the model are involved in the computation of p ( z ( l ) ; θ ) which makes it difﬁcult to optimise the auxiliary function with all the layers at the same time.
A solution to this issue is to perform the optimisation layer by layer in iterations of the EM algorithm.
In this case, when we optimise the auxiliary function with respect to the parameters of layer l , the fact that p ( z ( l ) ; θ ) depends on the parameters of layer l +is not a problem and an EM algorithm can be performed.
We refer to [8] for further details about the EM algorithm steps.
Finally, choosing the number of clusters for each layer can be done with a model selection criterion such as the BIC [4].
2.4 Interpreting the inferred partitions under the DGMM
There are two main ways to use the inferred partitions of the DGMM.
The ﬁrst way focuses on the ﬁrst layer of the network, and then have a look at the partitions v (1) .
In this case, the number of clusters is equal to G (1) and the distribution of the observations of a cluster is assumed not to be Gaussian since the latent scores z (1) are assumed to be drawn through an MFA, as detailed in
Equation (2).
The second way of using the DGMM is to observe the clusters of the global GMM of
Equation (3).
In this case, the number of clusters is equal to G (1) × . . .
× G ( L ) , and the observations of a cluster are now assumed to follow a Gaussian distribution.
Experiments with the Deep Gaussian Mixture Model
We simulate data via the generative process of the DGMM with known parameters and partitions.
Then, we run the EM algorithm to see if it estimates the parameters and partitions well.
Because the number of parameters is large, we focus on the Adjust Rand Index (ARI), which gives an indicator of the performances of the algorithm on the estimation of the partitions and hence on the estimation of the parameters too.
In these experiments we start by focusing on the partitions of the ﬁrst layer, and then we focus on the partitions of the global GMM.
3.1 Settings for the simulated data set
For the data set x , we have N = 4 , 000 and J = 17 .
The network was built with three layers ( L = 3 ) and G (1) = 3 , G (2)
= 3 and G (3) = 2 .
In addition, the latent scores dimensions were set to
R (1) = 11 , R (2) = 6 and R (2) = 2 respectively; note this DGMM is identiﬁed in a factorial sense..
The parameters π ( l ) g were set to 1 /G
( l ) for all g and for all l .
The parameters ( η ( l ) g , Λ ( l ) g , ψ ( l ) g ) g,l were sampled from Gaussian distributions whose parameters can be found in the supplementary material ﬁles.
3.2 Results on the ﬁrst layer
In this section, we focus on the partitions of the ﬁrst layer v (1) .
We run 20 times the DGMM EM algorithm and the classic GMM by using the R package mclust [5].
For the DGMM runs, we ﬁx
G (2) and G (3) to their ground-truth values and vary G (1) from 2 to 6 .
In addition, the algorithm is run with random starting values.
For the GMM runs, we vary the number of components G from 2 to 6 .
Table 1 shows for each run the number of clusters with the best BIC value.
We see that mclust was not able to ﬁnd the true number of clusters.
This result is expected since the clusters of the ﬁrst layer are not supposed to be Gaussian: mclust overﬁts the data by ﬁnding more clusters than there truly are.
However, the DGMM seems to yield the same behaviour, which is not expected since its ﬁrst layer is supposed to capture non-Gaussian distributions.
DGMM                    mclust
Figure 2 represents the ARI of the runs with the best BIC value.
We see that the DGMM yields better results than mclust .
However, these results are not satisfying because the data was generated through the DGMM generative process: one would hope that at least some of the runs should achieve an ARI equal to 1 .
Let us note that similar behaviour was observed across a large number of data sets generated with a range of settings of the DGMM architecture but for brevity only one such is presented here.
3.3 Results on the global GMM
For every experiment described below, the number of clusters ( G ( l ) ) l are ﬁxed to their true value.
The DGMM’s inference algorithm is run 20 times with random starting values.
Figure 3 shows the resulting ARI.
The DGMM yields poor results since the highest ARI value achieved is 0 .
67 even though the data set was generated through the DGMM’s generative process.
In Figure 4, we plot the BIC values over the iterations of the EM algorithm for a single random initialisation; the BIC reaches a plateau, meaning that the EM algorithm achieves at least a local maximum of the likelihood.
If the EM algorithm is able to ﬁnd local maxima, but fails to ﬁnd the right partitions, it might mean that there are many local maxima.
To verify this, we run the EM algorithm a hundred times and compute the ARI of each resulting clustering with respect to every other resulting clustering.
The mutual ARI are plotted in Figure 5 (left and right), where we see that none of the mutual ARI values are 1 (except the ARI of a result with respect to itself).
This means that the EM algorithm founddifferent local maxima leading to very different partitions.
The last result we detail regards another unexpected behaviour of the DGMM’s inference algorithm.
When we list the resulting partition for each observation on each layer, we can see that all the clusters of every layer are represented.
However, when we list the resulting partitions of the global GMM as in Table 2, we see that not all the clusters are represented.
This means that even if each neuron of each layer has observations, not every combination of neurons of the three layers does.
Therefore, there is a severe problem of empty clusters at the global GMM’s level.
Finally, let us note that the package mclust was not able to ﬁnd a solution for G = (cid:81) l G ( l ) = 18 .
From this result, the DGMM can be seen as a candidate when the number of clusters is high, even if the estimation of partitions is not perfect. [Table] [Table] [Table]
We see that the latent scores ( z ( l ) )
l do not seem to be estimated well.
3.4 Possible reasons for the poor results
There are several possible reasons for the poor results described in the previous section.
First of all, the MFA model already suffers from unindentiﬁability issues that could worsen with the multilayer architecture of the DGMM.
The most common solutions to tackle the unidentiﬁablity issues are:
• To constrain Λ ( l ) g so that it is orthonormal and to order the columns by decreasing variance of the corresponding latent scores.
• To constrain Λ ( l ) g T ψ ( l ) g Λ ( l ) g to be diagonal.
• To constrain Λ ( l ) g so that it is a lower triangular full rank matrix with diagonal elements strictly positive.
• To choose an informative rotation matrix.
Many heuristic methods try to ﬁnd rotation matrices P that can be used to modify Λ ( l ) g (and the latent factors) so as to try to increase the interpretability.
While we already tried to apply these constraints (results not detailed in this document), we did not obtained better results regarding the partitions and parameters estimation.
Second, the DGMM has many latent variables, which can be a severe problem when it comes to estimation.
Indeed, in the context of the conﬁguration of Section 3.1, the latent scores alone represent
N × ( R (1) + R (2) + R (3) )
= 4 , 000 × (11 + 6 + 2) =
76 , 000 continuous values to estimate, which is very high.
This large number of latent variables is for us one of the most important obstacles to correct estimation.
To support this point, Table 3 shows the ARI that the EM algorithm achieved at each layer.
At the ﬁrst layer, the ARIs are globally higher than the other layers and can even be equal to 1 .
However, the ARI for the other layers worsens with the depth: the second layer’s higher ARI value is 0 .
51 and the third layer’s ARI value are all near to zero.
This probably means that the latent scores are poorly estimated since the DGMM of layer 1 depends on the data x , the DGMM of layer 2 depends on the scores z (1) and the DGMM of layer 3 depends on the scores z (2) .
A solution to bypass the latent score estimation is to use another kind of algorithm, such as the Newton-Raphson algorithm and its variants.
However, we encountered different problems when using this kind of algorithm such as computational errors due to matrix inversions.
Finally, we have also observed that even though the clusters of each layer do not necessarily become empty, the clusters of the global GMM do so, which is also a problem for estimation of the partition.
Conclusion
In theory, the DGMM is a powerful model that can estimate complex distributions by stacking MFA layers in a deep learning fashion.
We have seen in this Section 3.3 that this algorithm is able to provide an estimation of partitions and parameters in cases where mclust is not able to do so, for instance when the number of clusters is high.
However, this estimation is not correct and we have seen that the EM algorithm suffers from drawbacks and can struggle to ﬁnd the global maxima.
Acknowledgments and Disclosure of Funding
We would like to warmly thank Cinzia Viroli and Geoff McLachlan for their helpful discussions from which the work beneﬁtted.
ICG was supported by the Science Foundation Ireland funded Insight
Centre for Data Analytics (SFI/12/RC/2289_P2).