Original: G RADIENT - BASED TRAINING OF G AUSSIAN M IXTURE M ODELS IN H IGH -D IMENSIONAL S PACES
Revised: G RADIENT - BASED TRAINING OF G AUSSIAN M IXTURE M ODELS IN H IGH -D IMENSIONAL S PACES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We present an approach for efﬁciently training Gaussian Mixture Models (GMMs) with Stochastic Gradient Descent (SGD) on large amounts of high-dimensional data (e.g., images).
Revised: We present an approach for efﬁciently training Gaussian Mixture Models (GMMs) with Stochastic Gradient Descent (SGD) on large amounts of high-dimensional data (e.g., images).

Original: In such a scenario, SGD is strongly superior in terms of execution time and memory usage, although it is conceptually more complex than the traditional Expectation-Maximization (EM) algorithm.
Revised: In such a scenario, SGD is strongly superior in terms of execution time and memory usage, although it is conceptually more complex than the traditional Expectation-Maximization (EM) algorithm.

Original: For enabling SGD training, we propose three novel ideas:
Revised: For enabling SGD training, we propose three novel ideas:

Original: First, we show that minimizing an upper bound to the GMM log likelihood instead of the full one is feasible and numerically much more stable way in high-dimensional spaces.
Revised: First, we show that minimizing an upper bound to the GMM log likelihood instead of the full one is feasible and numerically much more stable way in high-dimensional spaces.

Original: Secondly, we propose a new annealing procedure that prevents SGD from converging to pathological local minima.
Revised: Secondly, we propose a new annealing procedure that prevents SGD from converging to pathological local minima.

Original: And lastly, we present a simple method for enforcing the constraints inherent to

Original: GMM training when using SGD. We also propose an SGD-compatible simpliﬁcation to the full GMM model based on local principal directions, which avoids excessive memory use in high-dimensional spaces due to quadratic growth of covariance matrices.
Revised: We also propose an SGD-compatible simpliﬁcation to the full GMM model based on local principal directions, which avoids excessive memory use in highdimensional spaces due to quadratic growth of covariance matrices.

Original: Experiments on several standard image datasets show the validity of our approach and we provide a publicly available TensorFlow implementation
Revised: Experiments on several standard image datasets show the validity of our approach and we provide a publicly available TensorFlow implementation.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: This contribution is in the context of Gaussian Mixture Models (GMMs), which is a probabilistic unsupervised method for clustering and data modeling.
Revised: This contribution is in the context of Gaussian Mixture Models (GMMs), which is a probabilistic unsupervised method for clustering and data modeling.

Original: GMMs have been used in a wide range of scenarios, e.g., Melnykov et al.
Revised: GMMs have been used in a wide range of scenarios, e.g., Melnykov et al.

Original: Traditionally, free parameters of a GMM are estimated using the so-called Expectation-Maximization (EM) algorithm, which has the appealing property of requiring no learning rates and which automatically enforces all constraints that GMMs impose.
Revised: Traditionally, free parameters of a GMM are estimated using the so-called Expectation-Maximization (EM) algorithm, which has the appealing property of requiring no learning rates and which automatically enforces all constraints that GMMs impose.

Original: 1. M OTIVATION
Revised: 1. M OTIVATION

Original: GMMs have several properties that make their application to image data appealing, like the ability to generate samples from the modeled distribution, or their intrinsic outlier detection ability.
Revised: GMMs have several properties that make their application to image data appealing, like the ability to generate samples from the modeled distribution, or their intrinsic outlier detection ability.

Original: However, for such high-dimensional and data-intensive scenarios which require a high degree of parallelization to be efﬁciently solvable, the traditional way of training GMMs by EM is reaching its limits, both in terms of memory consumption and execution speed.
Revised: However, for such high-dimensional and data-intensive scenarios which require a high degree of parallelization to be efﬁciently solvable, the traditional way of training GMMs by EM is reaching its limits, both in terms of memory consumption and execution speed.

Original: Memory requirements skyrocket mainly because GMM is intrinsically a batch-type algorithm that needs to store and process all samples in memory in order to be parallelizable.
Revised: Memory requirements skyrocket mainly because GMM is intrinsically a batch-type algorithm that needs to store and process all samples in memory in order to be parallelizable.

Original: In addition, computation time becomes excessive because EM involves covariance matrix inversions, which become prohibitive for high data dimensionality.
Revised: In addition, computation time becomes excessive because EM involves covariance matrix inversions, which become prohibitive for high data dimensionality.

Original: Here, an online or mini-batch type of optimization such as Stochastic Gradient Descent (SGD) has strong advantages w.r.t.
Revised: Here, an online or mini-batch type of optimization such as Stochastic Gradient Descent (SGD) has strong advantages w.r.t.

Original: the memory consumption in particular, but also w.r.t.
Revised: the memory consumption in particular, but also w.r.t.

Original: computation time since it requires no matrix inversions.
Revised: computation time since it requires no matrix inversions.

Original: Therefore, we seek to develop a Stochastic Gradient Descent algorithm for GMMs, making it possible to efﬁciently train GMMs on large collections of images.
Revised: Therefore, we seek to develop a Stochastic Gradient Descent algorithm for GMMs, making it possible to efﬁciently train GMMs on large collections of images.

Original: Additionally, since the number of free GMM parameters depends quadratically on the data dimensionality
Revised: Additionally, since the number of free GMM parameters depends quadratically on the data dimensionality

Original: D , we seek to develop simpliﬁed GMM models with a less severe impact of D .
Revised: D , we seek to develop simpliﬁed GMM models with a less severe impact of D .

Original: 1. R ELATED W ORK
Revised: 1. R ELATED W ORK

Original: There are several proposals for what is termed “incremental” or “online” EM for training GMMs
Revised: There are several proposals for what is termed “incremental” or “online” EM for training GMMs

Original: Vlassis & Likas (2002); Engel & Heinen (2010); Pinto & Engel (2015); Cederborg et al.
Revised: Vlassis & Likas (2002); Engel & Heinen (2010); Pinto & Engel (2015); Cederborg et al.

Original: Song & Wang (2005); Kristan et al.
Revised: Song & Wang (2005); Kristan et al.

Original: (2008), allowing GMMs to be trained by providing one sample at a time.
Revised: (2008), allowing GMMs to be trained by providing one sample at a time.

Original: However, none of these models reproduce the original GMM algorithm faithfully, which is why most of them additionally resort to component pruning or merging heuristics for components, leading to learning dynamics that are difﬁcult to understand.
Revised: However, none of these models reproduce the original GMM algorithm faithfully, which is why most of them additionally resort to component pruning or merging heuristics for components, leading to learning dynamics that are difﬁcult to understand.

Original: This is also the case for the Locally
Revised: This is also the case for the Locally

Original: Weighted Projection Regression (LWPR) algorithm Vijayakumar et al.
Revised: Weighted Projection Regression (LWPR) algorithm Vijayakumar et al.

Original: (2005) which can be interpreted as a GMM model adapted for online learning.
Revised: (2005) which can be interpreted as a GMM model adapted for online learning.

Original: To our knowledge, there is only a single work proposing to train GMMs by SGD Hosseini & Sra (2015).
Revised: To our knowledge, there is only a single work proposing to train GMMs by SGD Hosseini & Sra (2015).

Original: Here, constraint enforcement is ensured by using manifold optimization techniques and a carefully chosen set of regularizers, leading to a rather complex model with several additional and hard-to-choose hyper-parameters.
Revised: Here, constraint enforcement is ensured by using manifold optimization techniques and a carefully chosen set of regularizers, leading to a rather complex model with several additional and hard-to-choose hyper-parameters.

Original: The idea of including regularization/annealing into GMMs (in the context of EM) was proposed in Verbeek et al.
Revised: The idea of including regularization/annealing into GMMs (in the context of EM) was proposed in Verbeek et al.

Original: (2005); Ormoneit & Tresp (1998), although their regularizes signiﬁcantly differs from ours.
Revised: (2005); Ormoneit & Tresp (1998), although their regularizes signiﬁcantly differs from ours.

Original: Approximating the full GMM log-likelihood is used in Verbeek et al.
Revised: Approximating the full GMM log-likelihood is used in Verbeek et al.

Original: ; Pinheiro & Bates (1995);
Revised: ; Pinheiro & Bates (1995);

Original: Dognin et al.
Revised: Dognin et al.

Original: (2009), although together with EM training.
Revised: (2009), although together with EM training.

Original: There has been signiﬁcant work on simplifying GMM models (e.g., Jakovljevi´c (2014)), leading to what is called parsimonious GMM models , which is reviewed in Bouveyron & Brunet-Saumard (2014).
Revised: There has been signiﬁcant work on simplifying GMM models (e.g., Jakovljevi´c (2014)), leading to what is called parsimonious GMM models , which is reviewed in Bouveyron & Brunet-Saumard (2014).

Original: 1. C ONTRIBUTIONS
Revised: 1. C ONTRIBUTIONS

Original: The main novel contributions of this article are:
Revised: The main novel contributions of this article are:

Original: • a generic and novel method for training GMMs using standard SGD that enforces all constraints and is numerically stable for high dimensional data (e.g., D > 500 ) • an annealing procedure that ensures convergence from a wide range of initial conditions in a streaming setting (excluding, e.g., centroid initialization by k-means) • a simpliﬁcation of the basic GMM model that is particularly suited for applying GMMs to high-dimensional data • an analysis of the link between SGD-trained GMM models and self-organizing maps
Revised: • a generic and novel method for training GMMs using standard SGD that enforces all constraints and is numerically stable for high dimensional data (e.g., D > 500 ) • an annealing procedure that ensures convergence from a wide range of initial conditions in a streaming setting (excluding, e.g., centroid initialization by k-means) • a simpliﬁcation of the basic GMM model that is particularly suited for applying GMMs to high-dimensional data • an analysis of the link between SGD-trained GMM models and self-organizing maps

Original: Apart from these novel contributions, we provide a publicly available TensorFlow implementation.
Revised: Apart from these novel contributions, we provide a publicly available TensorFlow implementation.

Original:  D ATASETS
Revised:  D ATASETS

Original: We use two datasets for all experiments.
Revised: We use two datasets for all experiments.

Original: MNIST (LeCun et al., 1998) consists of gray scale images of handwritten digits ( 0 - 9 ) and is a common benchmark for computer vision systems and classiﬁcation problems.
Revised: MNIST (LeCun et al., 1998) consists of gray scale images of handwritten digits ( 0 - 9 ) and is a common benchmark for computer vision systems and classiﬁcation problems.

Original: SVHN (Netzer et al., 2011) is a 10 -class benchmark based on color images of house numbers ( 0 - 9 ).
Revised: SVHN (Netzer et al., 2011) is a 10 -class benchmark based on color images of house numbers ( 0 - 9 ).

Original: We use the cropped and centered digit format.
Revised: We use the cropped and centered digit format.

Original: For both benchmarks, we use either the full images ( 32 × 32 × 3 for SVHN, 28 × 28 × 1 for MNIST), or a crop of the central 6 × 6 patch of each image.
Revised: For both benchmarks, we use either the full images ( 32 × 32 × 3 for SVHN, 28 × 28 × 1 for MNIST), or a crop of the central 6 × 6 patch of each image.

Original: The class information is not used since GMM is a purely unsupervised method.
Revised: The class information is not used since GMM is a purely unsupervised method.

Original:  M ETHODS : S TOCHASTIC G RADIENT D ESCENT FOR GMM S
Revised:  M ETHODS : S TOCHASTIC G RADIENT D ESCENT FOR GMM S

Original: Gaussian Mixture Models (GMMs) are normally formulated in terms of K component probabilities, modeled by multi-variate Gaussians.
Revised: Gaussian Mixture Models (GMMs) are normally formulated in terms of K component probabilities, modeled by multi-variate Gaussians.

Original: It is assumed that each data sample x , represented as a single row of the data matrix X , has been sampled from a single Gaussian component k , selected with a priori probability π k .
Revised: It is assumed that each data sample x , represented as a single row of the data matrix X , has been sampled from a single Gaussian component k , selected with a priori probability π k .

Original: Sampling is performed from the Gaussian conditional probability p ( (cid:126)x | k ) , whose parameters are the centroids µ k and covariance matrices Σ k : p ( x | k ) ∼ N ( x ; µ , Σ ) .
Revised: Sampling is performed from the Gaussian conditional probability p ( (cid:126)x | k ) , whose parameters are the centroids µ k and covariance matrices Σ k : p ( x | k ) ∼ N ( x ; µ , Σ ) .

Original: A
Revised: A

Original: GMM aims to explain the data by minimizing the log-likelihood function
Revised: GMM aims to explain the data by minimizing the log-likelihood function

Original: [Equation]
Revised: [Equation]

Original: Free parameters to be adapted are the component probabilities π k , the component centroids µ k and the component covariance matrices Σ k .
Revised: Free parameters to be adapted are the component probabilities π k , the component centroids µ k and the component covariance matrices Σ k .

Original: 3. M AX - COMPONENT A PPROXIMATION
Revised: 3. M AX - COMPONENT A PPROXIMATION

Original: As a ﬁrst step to implement Stochastic Gradient Descent (SGD) for optimizing the log-likelihood (see Eq. 1), we observe that the component weights π k and the conditional probabilities p ( x | k ) are positive by deﬁnition.
Revised: As a ﬁrst step to implement Stochastic Gradient Descent (SGD) for optimizing the log-likelihood (see Eq. 1), we observe that the component weights π k and the conditional probabilities p ( x | k ) are positive by deﬁnition.

Original: It is therefore evident that any single component of the inner sum over the components k is a lower bound for the whole inner sum.
Revised: It is therefore evident that any single component of the inner sum over the components k is a lower bound for the whole inner sum.

Original: The largest of these K lower bounds is given by the maximum over the components, thus giving
Revised: The largest of these K lower bounds is given by the maximum over the components, thus giving

Original: [Equation]
Revised: [Equation]

Original: This is what we term the max-component approximation of Eq. 2.
Revised: This is what we term the max-component approximation of Eq. 2.

Original: Since L MC ≥ L , we can decrease
Revised: Since L MC ≥ L , we can decrease

Original: L by minimizing L MC .
Revised: L by minimizing L MC .

Original: The advantage of L MC is that it is not affected by numerical instabilities as L is.
Revised: The advantage of L MC is that it is not affected by numerical instabilities as L is.

Original: An easy way to see this is by considering the case of a single Gaussian component with weight
Revised: An easy way to see this is by considering the case of a single Gaussian component with weight

Original: π k = 1 , centroid µ k = 0 and whose covariance matrix Σ k has only diagonal entries which are all equal to 0 .
Revised: π k = 1 , centroid µ k = 0 and whose covariance matrix Σ k has only diagonal entries which are all equal to 0 .

Original: In this case, an 1 000 -dimensional input vector x , x i = 1 ∀ i would give a conditional probability of p ( x | k ) =
Revised: In this case, an 1 000 -dimensional input vector x , x i = 1 ∀ i would give a conditional probability of p ( x | k ) =

Original: (2 π 0 . 1) − 500 e − 5000 which will, depending on ﬂoating point inaccuracy, produce NaN or inﬁnity for the ﬁrst factor due to overﬂow, and zero for the second factor due to underﬂow, resulting in an undeﬁned behavior.
Revised: (2 π 0 . 1) − 500 e − 5000 which will, depending on ﬂoating point inaccuracy, produce NaN or inﬁnity for the ﬁrst factor due to overﬂow, and zero for the second factor due to underﬂow, resulting in an undeﬁned behavior.

Original: 3. C ONSTRAINT E NFORCEMENT
Revised: 3. C ONSTRAINT E NFORCEMENT

Original: GMMs require the weights to be normalized: (cid:80)
Revised: GMMs require the weights to be normalized: (cid:80)

Original: k
Revised: k

Original: π k = 1 and elements of the diagonal matrices Σ to be non-negative: Σ ij ≥ 0 ∀ i, j .
Revised: π k = 1 and elements of the diagonal matrices Σ to be non-negative: Σ ij ≥ 0 ∀ i, j .

Original: In traditional GMM training via Expectation-Maximization, these constraints are taken care of by the method of Lagrangian multipliers.
Revised: In traditional GMM training via Expectation-Maximization, these constraints are taken care of by the method of Lagrangian multipliers.

Original: In SGD, they must be enforced in different manner.
Revised: In SGD, they must be enforced in different manner.

Original: For the weights π k , we adopt the approach proposed in Hosseini &
Revised: For the weights π k , we adopt the approach proposed in Hosseini &

Original: Sra (2015), which replaces them by other free parameters ξ k from which the π k are computed such that normalization is ensured.
Revised: Sra (2015), which replaces them by other free parameters ξ k from which the π k are computed such that normalization is ensured.

Original: One such computational scheme is
Revised: One such computational scheme is

Original: [Equation]
Revised: [Equation]

Original: For ensuring non-negativeness of the covariances, we identify all covariances inferior to a certain minimal value Σ min after each gradient descent step, and subsequently clip them to Σ min .
Revised: For ensuring non-negativeness of the covariances, we identify all covariances inferior to a certain minimal value Σ min after each gradient descent step, and subsequently clip them to Σ min .

Original: 3. A NNEALING PROCEDURE
Revised: 3. A NNEALING PROCEDURE

Original: A major issue for SGD optimization of GMMs are local minima that obviously correspond to undesirable solutions.
Revised: A major issue for SGD optimization of GMMs are local minima that obviously correspond to undesirable solutions.

Original: Prominently among these is what we term the single-component solution : here, a single component k ∗ has a weight close to 1 , with its centroid and covariance matrix being given by the mean and covariance of the data:
Revised: Prominently among these is what we term the single-component solution : here, a single component k ∗ has a weight close to 1 , with its centroid and covariance matrix being given by the mean and covariance of the data:

Original: [Equation]
Revised: [Equation]

Original: Another undesirable local minimum is the degenerate solution where all components have the same weight, centroid and covariance matrix, the latter being identical to the single-component solution.
Revised: Another undesirable local minimum is the degenerate solution where all components have the same weight, centroid and covariance matrix, the latter being identical to the single-component solution.

Original: See Fig.
Revised: See Fig.

Original: 4 (left) for a visualization of the latter.
Revised: 4 (left) for a visualization of the latter.

Original: Such solutions usually evolve from an incorrect initialization of the GMM model, and it is easy to show that the gradient w.r.t.
Revised: Such solutions usually evolve from an incorrect initialization of the GMM model, and it is easy to show that the gradient w.r.t.

Original: all model parameters is zero in both of these scenarios.
Revised: all model parameters is zero in both of these scenarios.

Original: This is easy to show: As the gradients read (with γ nl denoting the responsibility of component l for sample n )
Revised: This is easy to show: As the gradients read (with γ nl denoting the responsibility of component l for sample n )

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: it is easy to see that when either all or single centroid/covariance matrix are equal to the total mean and variance of the data ( µ lm = 1 N x nm and Σ lm = 2 N ( x nm − µ lm ) 2) ), we obtain zero gradients for degenerate solutions ( π i = γ i = 1 /K ) and single-component solutions ( ∃ !
Revised: it is easy to see that when either all or single centroid/covariance matrix are equal to the total mean and variance of the data ( µ lm = 1 N x nm and Σ lm = 2 N ( x nm − µ lm ) 2 ), we obtain zero gradients for degenerate solutions ( π i = γ i = 1 /K ) and single-component solutions ( ∃ !

Original: π i ∗ , γ i ∗ = 1 ), respectively.
Revised: π i ∗ , γ i ∗ = 1 ), respectively.

Original: Our approach for avoiding these undesirable solutions in principle is to punish their characteristic response patterns by an appropriate modiﬁcation of the loss function that is minimized, i.e., L MC .
Revised: Our approach for avoiding these undesirable solutions in principle is to punish their characteristic response patterns by an appropriate modiﬁcation of the loss function that is minimized, i.e., L MC .

Original: The following guidelines should be adhered to:
Revised: The following guidelines should be adhered to:

Original: • single-component and degenerate solution should be punishedstrongly by the annealing procedure • as we mainly wish to ensure a good starting point for optimization, annealing should be active only at the beginning of the training process • as training advances, annealed loss should smoothly transition into the original loss • the number of free parameters introduced by the annealing procedure should be low
Revised: • single-component and degenerate solution should be punished by the annealing procedure • as we mainly wish to ensure a good starting point for optimization, annealing should be active only at the beginning of the training process • as training advances, annealed loss should smoothly transition into the original loss • the number of free parameters introduced by the annealing procedure should be low

Original: These requirements are met by what we term the smoothed max- component log -likelihood ˜ 
Revised: These requirements are met by what we term the smoothed max-component log-likelihood ˜ L MC :

Original: L

Original: MC

Original: [Equation]
Revised: [Equation]

Original: Here, we assign a normalized coefﬁcient vector g k to each Gaussian mixture component k .
Revised: Here, we assign a normalized coefﬁcient vector g k to each Gaussian mixture component k .

Original: The entries of g k are computed in the following fashion:
Revised: The entries of g k are computed in the following fashion:

Original: • Assume that the K Gaussian components are arranged on a 1D grid of dimensions (1 , K ) or on a 2D grid of dimensions ( √ K , √ K ) .
Revised: • Assume that the K Gaussian components are arranged on a 1D grid of dimensions (1 , K ) or on a 2D grid of dimensions ( √ K , √ K ) .

Original: Each linear component index k has thus an unique associated 1D or 2D coordinate c ( k ) .
Revised: Each linear component index k has thus an unique associated 1D or 2D coordinate c ( k ) .

Original: • Assume that the vector g k of length K is actually representing a 1D structure of dimension (1 , K ) or a 2D structure of dimension ( √ K , √ K ) .
Revised: • Assume that the vector g k of length K is actually representing a 1D structure of dimension (1 , K ) or a 2D structure of dimension ( √ K , √ K ) .

Original: Each linear vector index j in g k has thus a unique associated 1D or 2D coordinate c ( j ) .
Revised: Each linear vector index j in g k has thus a unique associated 1D or 2D coordinate c ( j ) .

Original: • The entries of the vector g k are computed as
Revised: • The entries of the vector g k are computed as

Original: [Equation]
Revised: [Equation]

Original: and subsequently normalize to have unit sum.
Revised: and subsequently normalize to have unit sum.

Original: Thus, Eq. 7 essentially represents a convolution of the probabilities p ( (cid:126)x | k ) , arranged in an 1D or 2D grid, with a Gaussian convolution ﬁlter, thus amounting to a smoothing operation.
Revised: Thus, Eq. 7 essentially represents a convolution of the probabilities p ( (cid:126)x | k ) , arranged in an 1D or 2D grid, with a Gaussian convolution ﬁlter, thus amounting to a smoothing operation.

Original: The 1D or 2D variance σ in Eq.
Revised: The 1D or 2D variance σ in Eq.

Original: 8 is a parameter that must be set as a function of the grid size such that Gaussians are neither homogeneous nor delta peaks.
Revised: 8 is a parameter that must be set as a function of the grid size such that Gaussians are neither homogeneous nor delta peaks.

Original: Thus, the loss function in Eq.
Revised: Thus, the loss function in Eq.

Original: 7 is maximized if the log probabilities follow an unimodal Gaussian proﬁle of variance σ , whereas single-component and degenerate solutions are punished.
Revised: 7 is maximized if the log probabilities follow an unimodal Gaussian proﬁle of variance σ , whereas single-component and degenerate solutions are punished.

Original: It is trivial to so see that the annealed loss function in Eq.
Revised: It is trivial to so see that the annealed loss function in Eq.

Original: 7 reduces to the non-annealed form Eq.in the limit where σ → 0 .
Revised: 7 reduces to the non-annealed form Eq.in the limit where σ → 0 .

Original: This is because the vectors g k approach Kronecker deltas in this case, with only a single entry of value 1 , which essentially removes the inner sum in Eq.
Revised: This is because the vectors g k approach Kronecker deltas in this case, with only a single entry of value 1 , which essentially removes the inner sum in Eq.

Original: By making σ time-dependent, starting at an intermediate value of σ ( t 0 ) ≡ σ 0 and then let it approach a small ﬁnal value σ ( t ∞ )
Revised: By making σ time-dependent, starting at an intermediate value of σ ( t 0 ) ≡ σ 0 and then let it approach a small ﬁnal value σ ( t ∞ )

Original: ≡ σ ∞ , we can smoothly transition the annealed loss function Eq.
Revised: ≡ σ ∞ , we can smoothly transition the annealed loss function Eq.

Original: 7 to the original max-component log-likelihood Eq.
Revised: 7 to the original max-component log-likelihood Eq.

Original: Time dependency of σ ( t ) can thus be chosen to be:
Revised: Time dependency of σ ( t ) can thus be chosen to be:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the time constant in the exponential is chosen as τ = log σ 0 − σ ∞ t ∞ − t 0 to ensure a smooth transition.
Revised: where the time constant in the exponential is chosen as τ = log σ 0 − σ ∞ t ∞ − t 0 to ensure a smooth transition.

Original: 3. F ULL GMM M ODEL T RAINED BY SGD AND ITS H YPER -P ARAMETERS
Revised: 3. F ULL GMM M ODEL T RAINED BY SGD AND ITS H YPER -P ARAMETERS

Original: Putting everything together, training Gaussian Mixture Models with SGD is performed by minimizing the smoothed max-component log-likelihood of Eq. 7, enforcing the constraints on the component weights and covariances as detailed in Sec. 3.2 and transitioning from the smoothed to the ‘’bare” loss function as detailed in Sec. 3.3.
Revised: Putting everything together, training Gaussian Mixture Models with SGD is performed by minimizing the smoothed max-component log-likelihood of Eq. 7, enforcing the constraints on the component weights and covariances as detailed in Sec. 3.2 and transitioning from the smoothed to the “bare” loss function as detailed in Sec. 3.3.

Original: In order to avoid convergence to other undesirable local minima by SGD, we introduce three weighting constants λ π , λ µ and λ Σ , all in the [0 , 1] range, for controlling the relative adaptation speeds for the three groups of free parameters in the GMM models (setting, e.g., λ π = 0 would disable the adaptation of weights).
Revised: In order to avoid convergence to other undesirable local minima by SGD, we introduce three weighting constants λ π , λ µ and λ Σ , all in the [0 , 1] range, for controlling the relative adaptation speeds for the three groups of free parameters in the GMM models (setting, e.g., λ π = 0 would disable the adaptation of weights).

Original: We have thus the update rules:
Revised: We have thus the update rules:

Original: [Equation]
Revised: [Equation]

Original: Centroids are initialized to random values in range [ − µ i , µ i ] , weights are chosen equiprobable, that is to say 1 /K , for all components, and covariance matrix entries are uniformly initialized to very small positive values
Revised: Centroids are initialized to random values in range [ − µ i , µ i ] , weights are chosen equiprobable, that is to say 1 /K , for all components, and covariance matrix entries are uniformly initialized to very small positive values

Original: Σ i (a good ad-hoc choice is 0 . 15 ).
Revised: Σ i (a good ad-hoc choice is 0 . 15 ).

Original: Last but not least, a learning rate (cid:15) is required for performing SGD.
Revised: Last but not least, a learning rate (cid:15) is required for performing SGD.

Original: We ﬁnd that our algorithm depends very weakly on (cid:15) , and that a ﬁxed value of (cid:15) = 0 .
Revised: We ﬁnd that our algorithm depends very weakly on (cid:15) , and that a ﬁxed value of (cid:15) = 0 .

Original: 01 is usually a good choice.
Revised: 01 is usually a good choice.

Original: We summarize good practices for choosing hyper-parameters of the proposed SGD approach for GMMs in App.
Revised: We summarize good practices for choosing hyper-parameters of the proposed SGD approach for GMMs in App.

Original: C.
Revised: C.

Original: Please not that it is not required to initialize the component centroids by k-means as it is usually recommended when training GMMs by EM.
Revised: Please not that it is not required to initialize the component centroids by k-means as it is usually recommended when training GMMs by EM.

Original: 3. S IMPLIFICATIONS AND M EMORY /P ERFORMANCE I SSUES
Revised: 3. S IMPLIFICATIONS AND M EMORY /P ERFORMANCE I SSUES

Original: [Figure] For the full GMM model, regardless of whether it is trained by EM or SGD, the memory requirements of a GMM with K components depend strongly on the data dimensionality D .
Revised: For the full GMM model, regardless of whether it is trained by EM or SGD, the memory requirements of a GMM with K components depend strongly on the data dimensionality D .

Original: A good measure for memory requirements is the number of free model parameters M computed as
Revised: A good measure for memory requirements is the number of free model parameters M computed as

Original: M = K ( D 2 + D
Revised: M = K ( D 2 + D

Original: + 1) .
Revised: + 1) .

Original: To reduce this number, we propose several strategies that are compatible with training by SGD (see also Fig. 1): 
Revised: To reduce this number, we propose several strategies that are compatible with training by SGD (see also Fig. 1): [Figure]

Original: Diagonal Covariance Matrix A well-known simpliﬁcation of GMMs consists of using a diagonal covariance matrix instead of a full one.
Revised: Diagonal Covariance Matrix A well-known simpliﬁcation of GMMs consists of using a diagonal covariance matrix instead of a full one.

Original: This reduces the number of trainable parameters to
Revised: This reduces the number of trainable parameters to

Original: K (2 D + 1) , where D is the dimensionality of the data vectors and K the number of components.
Revised: K (2 D + 1) , where D is the dimensionality of the data vectors and K the number of components.

Original: Local Principal Directions A potentially very powerful simpliﬁcation consist of using a diagonal covariance matrix of S < D diagonal entries, and letting SGD adapt the local principal directions along which these variances are measured for each component (in addition to the other free parameters, of course).
Revised: Local Principal Directions A potentially very powerful simpliﬁcation consist of using a diagonal covariance matrix of S < D diagonal entries, and letting SGD adapt the local principal directions along which these variances are measured for each component (in addition to the other free parameters, of course).

Original: The expression of the conditional component probabilities now includes, for each component k , the S normalized principal direction vectors d ks , s = 1 , . .
Revised: The expression of the conditional component probabilities now includes, for each component k , the S normalized principal direction vectors d ks , s = 1 , . .

Original: , S and reads:
Revised: , S and reads:

Original: [Equation]
Revised: [Equation]

Original: This leads to a value of M = K (( S + 1) D + 1) , and thus achieves satisfactory model simpliﬁcation only if S can be chosen very small and D is high.
Revised: This leads to a value of M = K (( S + 1) D + 1) , and thus achieves satisfactory model simpliﬁcation only if S can be chosen very small and D is high.

Original: In order for the model to be fully determined, another constraint needs to be enforced after each SGD step, namely the orthogonality of the local principal directions for each component k : p Tkl p km = δ lm ∀ l, m < S .
Revised: In order for the model to be fully determined, another constraint needs to be enforced after each SGD step, namely the orthogonality of the local principal directions for each component k : p Tkl p km = δ lm ∀ l, m < S .

Original: For now, the matrix D k whose rows are the S vectors d ks is subjected to a QR decomposition D k = Q k R k and setting D k = Q k .
Revised: For now, the matrix D k whose rows are the S vectors d ks is subjected to a QR decomposition D k = Q k R k and setting D k = Q k .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Unless otherwise stated, the experiments in this section will always be conducted with the following parameter values (see App.
Revised: Unless otherwise stated, the experiments in this section will always be conducted with the following parameter values (see App.

Original: C for a justiﬁcation of these choices): total iterations T = 1 000 · √ K , t 0 = 0 .
Revised: C for a justiﬁcation of these choices): total iterations T = 1 000 · √ K , t 0 = 0 .

Original: 3 T , t ∞ = 0 .
Revised: 3 T , t ∞ = 0 .

Original: 8 T , mini-batch size B = 1 , µ i = 0 .
Revised: 8 T , mini-batch size B = 1 , µ i = 0 .

Original: 01 , σ 0 = 1 . 2 , σ ∞ = 0 .
Revised: 01 , σ 0 = 1 . 2 , σ ∞ = 0 .

Original: 011 , (cid:15) = 0 .
Revised: 011 , (cid:15) = 0 .

Original: 011 , Σ min = 0 .
Revised: 011 , Σ min = 0 .

Original: 15 and K = 5 · 5 .
Revised: 15 and K = 5 · 5 .

Original: The adaptation strengths λ π , λ µ and λ σ are all set to 1 . 0 .
Revised: The adaptation strengths λ π , λ µ and λ σ are all set to 1 . 0 .

Original: Except for
Revised: Except for

Original: Sec. 3.5, we will use a diagonal covariance matrix for each of the K components.
Revised: Sec. 3.5, we will use a diagonal covariance matrix for each of the K components.

Original: Training/test data are taken from MNIST or SVHN, either using full images or the central 6 × 6 patches.
Revised: Training/test data are taken from MNIST or SVHN, either using full images or the central 6 × 6 patches.

Original: 4. V ALIDITY OF THE M AX -C OMPONENT A PPROXIMATION AND C OMPARISON TO EM
Revised: 4. V ALIDITY OF THE M AX -C OMPONENT A PPROXIMATION AND C OMPARISON TO EM

Original: Since we minimize only an approximation to the GMM log-likelihood, a ﬁrst important question is about the quality of this approximation.
Revised: Since we minimize only an approximation to the GMM log-likelihood, a ﬁrst important question is about the quality of this approximation.

Original: To this end, we plot both energy functions over time when training on full MNIST and SVHN images, see Fig.
Revised: To this end, we plot both energy functions over time when training on full MNIST and SVHN images, see Fig.

Original: This ﬁgure shows that the log-likelihood obtained by EM-based training (using the implementation of sklearn ) with the same conﬁguration (using 10 iterations and a k-means-based initialization) achieves a very similar log-likelihood, which conﬁrms that SGD is indeed comparable in performance.
Revised: This ﬁgure shows that the log-likelihood obtained by EM-based training (using the implementation of sklearn ) with the same conﬁguration (using 10 iterations and a k-means-based initialization) achieves a very similar log-likelihood, which conﬁrms that SGD is indeed comparable in performance.

Original: [Figure]
Revised: [Figure]

Original: 4. E FFECTS AND B ENEFITS OF ANNEALING
Revised: 4. E FFECTS AND B ENEFITS OF ANNEALING

Original: In order to demonstrate the beneﬁcial effects of the annealing procedure, we conduct the basic experiment on full MNIST with annealing effectively turned off, which is achieved by setting σ 0 = σ ∞ + 0 .
Revised: In order to demonstrate the beneﬁcial effects of the annealing procedure, we conduct the basic experiment on full MNIST with annealing effectively turned off, which is achieved by setting σ 0 = σ ∞ + 0 .

Original: 001 = 0 .
Revised: 001 = 0 .

Original: As a result, in Fig.
Revised: As a result, in Fig.

Original: 4 we observe typical single-component solutions when visualizing the centroids and weights, along with a markedly inferior (higher) loss value of ≈ 155 , in contrast to ≈ 138 when annealing is turned on.
Revised: 4 we observe typical single-component solutions when visualizing the centroids and weights, along with a markedly inferior (higher) loss value of ≈ 155 , in contrast to ≈ 138 when annealing is turned on.

Original: As a side effect, annealing enforces a topological ordering of centroids in the manner of a SOM (see also App. B for a mathematical analysis), which is only of aesthetic value for the moment, but might be exploited in the future.
Revised: As a side effect, annealing enforces a topological ordering of centroids in the manner of a SOM (see also App. B for a mathematical analysis), which is only of aesthetic value for the moment, but might be exploited in the future.

Original: 4. B ASIC F EASIBILITY AND P ARAMETER S PACE
Revised: 4. B ASIC F EASIBILITY AND P ARAMETER S PACE

Original: In this experiment, we put everything together and train an GMM using SGD on various datasets of very high and intermediate dimensionality.
Revised: Here, we put everything together and train an GMM using SGD on various datasets of very high and intermediate dimensionality (results: see Fig. 2 and App. D).

Original: In particular, we analyze how the choice of the relative adaptation strengths λ π , λ Σ and number of components K affects the ﬁnal log-likelihood.
Revised: In particular, we analyze how the choice of the relative adaptation strengths λ π , λ Σ and number of components K affects the ﬁnal log-likelihood.

Original: The results presented in Tab.
Revised: The results presented in Tab.

Original: 1 suggest that increasing K is always beneﬁcial, and that the covariances and weights should be adapted more slowly than the centroids.
Revised: 1 suggest that increasing K is always beneﬁcial, and that the covariances and weights should be adapted more slowly than the centroids.

Original: The batch size B had an effect on convergence speed but no signiﬁcant impact on the ﬁnal loss value (not shown here).
Revised: The batch size B had an effect on convergence speed but no signiﬁcant impact on the ﬁnal loss value (not shown here).

Original: Parameter
Revised: Parameter

Original: K = K =
Revised: K = K =

Original: K =
Revised: K =

Original: K =λ π , λ Σ = 1 . 0 ,
Revised: K =λ π , λ Σ = 1 . 0 ,

Original: B =138 . 1 , 735 .136 .
Revised: B =138 . 1 , 735 .136 .

Original: 1 , 682 .λ π , λ Σ = 0 .
Revised: 1 , 682 .λ π , λ Σ = 0 .

Original: B =138 . 3 , 715 .134 . 6 , 706 .133 . 5 , 691 .130 .
Revised: B =138 . 3 , 715 .134 . 6 , 706 .133 . 5 , 691 .130 .

Original: 1 , 691 .λ
Revised: 1 , 691 .λ

Original: π , λ Σ = 0 .
Revised: π , λ Σ = 0 .

Original: B =138 . 1 , 722 .135 .
Revised: B =138 . 1 , 722 .135 .

Original: 4. R OBUSTNESS TO INITIAL CONDITIONS
Revised: 4. R OBUSTNESS TO INITIAL CONDITIONS

Original: [Figure]
Revised: [Figure]

Revised: represents the same value in all diagrams, the seeming difference is due to scaling as initial loss values in some experiments are much higher (i.e., worse).

Original: To test whether SGD converges as a function of the initial conditions, we train three GMMs on
Revised: To test whether SGD converges as a function of the initial conditions, we train three GMMs on

Original: MNIST using three different initializations: µ i ∈ { 1 . 0 , 0 .
Revised: MNIST using three different initializations: µ i ∈ { 1 . 0 , 0 .

Original: All of these choices led to nonconvergence of the EM-based GMM implementation of sklearn .
Revised: All of these choices led to nonconvergence of the EM-based GMM implementation of sklearn .

Original: Differently from the other experiments, we choose a learning rate of (cid:15) = 0 .
Revised: Differently from the other experiments, we choose a learning rate of (cid:15) = 0 .

Original: 02 since the original value does not lead to convergent learning.
Revised: 02 since the original value does not lead to convergent learning.

Original: The alternative is to double the training time which works as well.
Revised: The alternative is to double the training time which works as well.

Original: Fig.
Revised: Fig.

Original: 5 shows the development of the loss function in each of the three cases, and we observe that all three cases indeed converge by comparing the ﬁnal loss values to Fig.
Revised: 5 shows the development of the loss function in each of the three cases, and we observe that all three cases indeed converge by comparing the ﬁnal loss values to Fig.

Original: 3 (middle) and to the EM baseline.
Revised: 3 (middle) and to the EM baseline.

Original: This behavior persists across all datasets, but only MNIST is shown due to space limitations.
Revised: This behavior persists across all datasets, but only MNIST is shown due to space limitations.

Original: 4. SGD T RAINING OF S IMPLIFIED GMM S WITH P RINCIPAL L OCAL D IRECTIONS
Revised: 4. SGD T RAINING OF S IMPLIFIED GMM S WITH P RINCIPAL L OCAL D IRECTIONS

Original: We train a GMM with S ∈ { 26 , 50 , 100 } local principal directions on MNIST and SVHN patches and log the ﬁnal loss function values in order to assess model quality.
Revised: We train a GMM with S ∈ { 26 , 50 , 100 } local principal directions on MNIST and SVHN patches and log the ﬁnal loss function values in order to assess model quality.

Original: We ﬁnd that learned centroids and ﬁnal loss values approach those of the diagonal model for S ≥ 20 for both datasets.
Revised: We ﬁnd that learned centroids and ﬁnal loss values approach those of the diagonal model for S ≥ 20 for both datasets.

Original: Please see
Revised: Please see

Original: App.
Revised: App.

Original: A for a visualization of the ﬁnal centroids as a function of S .
Revised: A for a visualization of the ﬁnal centroids as a function of S .

Original: For the full datasets, we ﬁnd that
Revised: For the full datasets, we ﬁnd that

Original: MNIST and SVHN require values of S ≥ 100 and S ≥ 300 , respectively for achieving similar loss values as the diagonal model.
Revised: MNIST and SVHN require values of S ≥ 100 and S ≥ 300 , respectively for achieving similar loss values as the diagonal model.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We showed that GMMs can be trained on high-dimensional image data by SGD, at very small batch sizes, in a purely online fashion.
Revised: We showed that GMMs can be trained on high-dimensional image data by SGD, at very small batch sizes, in a purely online fashion.

Original: Memory and execution time requirements are very modest, and can be reduced even more by using intelligent simpliﬁcations of the GMM model.
Revised: Memory and execution time requirements are very modest, and can be reduced even more by using intelligent simpliﬁcations of the GMM model.

Original: Here, we would like to discuss a few noteworthy points and suggest some avenues for improving the model:
Revised: Here, we would like to discuss a few noteworthy points and suggest some avenues for improving the model:

Original: Robust Convergence Unlike EM approaches which require a careful selection of starting point, the presented SGD scheme is very robust to initial conditions due to the proposed annealing scheme.
Revised: Robust Convergence Unlike EM approaches which require a careful selection of starting point, the presented SGD scheme is very robust to initial conditions due to the proposed annealing scheme.

Original: In all experiments with high-dimensional image data, we found no initializations that did not, given reasonable time, converge to a regular solution.
Revised: In all experiments with high-dimensional image data, we found no initializations that did not, given reasonable time, converge to a regular solution.

Original: Numerical Stability While we are not optimizing the full log-likelihood here, the max-component approximation is actually a good one, and has the advantage of absolute numerical stability.
Revised: Numerical Stability While we are not optimizing the full log-likelihood here, the max-component approximation is actually a good one, and has the advantage of absolute numerical stability.

Original: Clearly, clever ways might be found to avoid instabilities in the log-likelihood function computation, but when it comes to automatically computed gradients this is no longer possible, and indeed we found that it was mainly the (inaccessible) gradient computations where NaN values happened mostly.
Revised: Clearly, clever ways might be found to avoid instabilities in the log-likelihood function computation, but when it comes to automatically computed gradients this is no longer possible, and indeed we found that it was mainly the (inaccessible) gradient computations where NaN values happened mostly.

Original: Complex annealing procedure The complex way of regularizing the model, which is similar in spirit to simulated annealing approaches, may seem daunting, although the time constants can be determined by simple heuristics, see App.
Revised: Complex annealing procedure The complex way of regularizing the model, which is similar in spirit to simulated annealing approaches, may seem daunting, although the time constants can be determined by simple heuristics, see App.

Original: C. A next step will be to replace this procedure by an automated scheme based on an evaluation of the loss function: when it is stationary, σ can be decreased slightly.
Revised: C. A next step will be to replace this procedure by an automated scheme based on an evaluation of the loss function: when it is stationary, σ can be decreased slightly.

Original: Repeating this strategy until a small minimal value is reached should remove the need to ﬁx annealing parameters except for time constants which can be chosen as a function of dataset size.
Revised: Repeating this strategy until a small minimal value is reached should remove the need to ﬁx annealing parameters except for time constants which can be chosen as a function of dataset size.

Original: Free Parameters Apart from the annealing mechanism, the model contains the same parameters any SGD approach would, the learning rate (cid:15) and the weighting constants for the model parameters:
Revised: Free Parameters Apart from the annealing mechanism, the model contains the same parameters any SGD approach would, the learning rate (cid:15) and the weighting constants for the model parameters:

Original: λ π , λ µ and λ Σ .
Revised: λ π , λ µ and λ Σ .

Original: We found no experimental evidence suggesting that the latter require values different from 1 , although from general principles it might be sensible to adapt the variances and weights slower than the centroids.
Revised: We found no experimental evidence suggesting that the latter require values different from 1 , although from general principles it might be sensible to adapt the variances and weights slower than the centroids.

Original: Further, it is required to understand how these constants can be used to obtain better solutions.
Revised: Further, it is required to understand how these constants can be used to obtain better solutions.

Original: Interesting is that standard DNN optimizers like Adam Kingma & Ba (2014) did not seem to produce satisfactory solutions which also requires looking into.
Revised: Interesting is that standard DNN optimizers like Adam Kingma & Ba (2014) did not seem to produce satisfactory solutions which also requires looking into.

Original: Approximate Gradient Descent
Revised: Approximate Gradient Descent

Original: The procedure described in Sec.
Revised: The procedure described in Sec.

Original: 3.4 performs gradient descent on an approximation to the full log-likelihood Eq. 1.
Revised: 3.4 performs gradient descent on an approximation to the full log-likelihood Eq. 1.

Original: This might seem unsatisfactory, so we would like to point out that the original EM procedure does nothing else: it optimizes a lower bound of the log-likelihood.
Revised: This might seem unsatisfactory, so we would like to point out that the original EM procedure does nothing else: it optimizes a lower bound of the log-likelihood.

Original: In particular, the M-step is not guaranteed to improve the log-likelihood at all: all we know is that it will never make it worse.
Revised: In particular, the M-step is not guaranteed to improve the log-likelihood at all: all we know is that it will never make it worse.

Original: In our SGD approach, there is a simple way to ﬁx this shortcoming, which we will tackle as an next step: we can replace the max-operation in Eq.
Revised: In our SGD approach, there is a simple way to ﬁx this shortcoming, which we will tackle as an next step: we can replace the max-operation in Eq.

Original: 2 by a softmax function with an initially large steepness parameter ω which will make it indistinguishable from a discrete max operation.
Revised: 2 by a softmax function with an initially large steepness parameter ω which will make it indistinguishable from a discrete max operation.

Original: After convergence, ω can be relaxed, which will result in a smooth transition to optimizing the full log-likelihood, but from an already well-converged state.
Revised: After convergence, ω can be relaxed, which will result in a smooth transition to optimizing the full log-likelihood, but from an already well-converged state.

Original:  C ONCLUSION AND O UTLOOK
Revised:  C ONCLUSION AND O UTLOOK

Original: On a more abstract level, we developed the presented method because of our interest in continual or incremental learning, which is essentially about eliminating the well-known catastrophic forgetting effect.
Revised: On a more abstract level, we developed the presented method because of our interest in continual or incremental learning, which is essentially about eliminating the well-known catastrophic forgetting effect.

Original: We believe GMMs are an essential building block for continual learning models since their parameter updates are purely local in the sense that only components that are close to the currently best-matching component are updated.
Revised: We believe GMMs are an essential building block for continual learning models since their parameter updates are purely local in the sense that only components that are close to the currently best-matching component are updated.

Original: Next steps will consist of constructing deep network classi- ﬁers from convolutional GMM layers, with a readout layer on top (all layers being trained by SGD).
Revised: Next steps will consist of constructing deep network classiﬁers from convolutional GMM layers, with a readout layer on top (all layers being trained by SGD).

Original: Further, we will investigate how to sample efﬁciently from such hierarchical convolutional GMMs, allow generating large batches of samples in a replay-based architecture for continual learning.
Revised: Further, we will investigate how to sample efﬁciently from such hierarchical convolutional GMMs, allow generating large batches of samples in a replay-based architecture for continual learning.

Original: A L OCAL PRINCIPAL DIRECTIONS
Revised: A L OCAL PRINCIPAL DIRECTIONS

Original: Here, we give a visualization of the resulting centroids for various choices of S when training on
Revised: Here, we give a visualization of the resulting centroids for various choices of S when training on

Original: MNIST patches.
Revised: MNIST patches.

Original: B R IGOROUS L INK TO S ELF -O RGANIZING M APS
Revised: B R IGOROUS L INK TO S ELF -O RGANIZING M APS

Original: It is known that the self-organizing map (SOM, Kohonen (1990)) rule has no energy function it is minimizing.
Revised: It is known that the self-organizing map (SOM, Kohonen (1990)) rule has no energy function it is minimizing.

Original: However, some modiﬁcations (see Heskes (1999)) have been proposed that ensure the existence of a C ∞ energy function.
Revised: However, some modiﬁcations (see Heskes (1999)) have been proposed that ensure the existence of a C ∞ energy function.

Original: These energy-based SOM models reproduce all features of the original model and use a learning rule that the original SOM algorithm is actually approximating very closely.
Revised: These energy-based SOM models reproduce all features of the original model and use a learning rule that the original SOM algorithm is actually approximating very closely.

Original: In the notation of this article, SOMs model the data through K prototypes µ k and K neighborhood functions g k deﬁned on a periodic 2D grid, and their energy function is written as
Revised: In the notation of this article, SOMs model the data through K prototypes µ k and K neighborhood functions g k deﬁned on a periodic 2D grid, and their energy function is written as

Original: [Equation]
Revised: [Equation]

Original: Discarding constant terms, we ﬁnd that this is actually identical to the max-component loglikelihood approximation given in Eq.
Revised: Discarding constant terms, we ﬁnd that this is actually identical to the max-component loglikelihood approximation given in Eq.

Original: 2 with constant equiprobable weights π k and a constant diagonal Σ with equal diagonal entries.
Revised: 2 with constant equiprobable weights π k and a constant diagonal Σ with equal diagonal entries.

Original: To our knowledge, this is the ﬁrst time that a rigorous link between SOMs and GMMs has been established based on a comparison of energy functions, showing that SOMs are actually implementing a annealing-type approximation to the full GMM model, with component weights and variances chosen in a special way.
Revised: To our knowledge, this is the ﬁrst time that a rigorous link between SOMs and GMMs has been established based on a comparison of energy functions, showing that SOMs are actually implementing a annealing-type approximation to the full GMM model, with component weights and variances chosen in a special way.

Original: C R ULES -O F -T HUMB FOR SGD-T RAINING OF GMM S ON IMAGES
Revised: C R ULES -O F -T HUMB FOR SGD-T RAINING OF GMM S ON IMAGES

Original: When training DNNs by SGD, several parameters need to be set according to heuristics.
Revised: When training DNNs by SGD, several parameters need to be set according to heuristics.

Original: Here, we present some rules-of-thumb for performing this selection with GMMs.
Revised: Here, we present some rules-of-thumb for performing this selection with GMMs.

Original: Generally, we can always set the batch size to 1 , Σ min = 0 .
Revised: Generally, we can always set the batch size to 1 , Σ min = 0 .

Original: 15 and (cid:15) = 0 .
Revised: 15 and (cid:15) = 0 .

Original: In all of our experiments, it was always feasible to set t 0 to half an epoch, and t ∞ − t 0 to half an epoch, too.
Revised: In all of our experiments, it was always feasible to set t 0 to half an epoch, and t ∞ − t 0 to half an epoch, too.

Original: However, tuning these parameters can speed up training signiﬁcantly.
Revised: However, tuning these parameters can speed up training signiﬁcantly.

Original: The number of prototypes is a critical choice, but follows a simple “the more the better” rule.
Revised: The number of prototypes is a critical choice, but follows a simple “the more the better” rule.

Original: From the mathematical foundations of GMMs, it is evident that more components must always be able to reach a lower loss (except for possible local minima).
Revised: From the mathematical foundations of GMMs, it is evident that more components must always be able to reach a lower loss (except for possible local minima).

Original: The relative adaptation strengths λ π , λ Σ and λ µ can always be set to 1 (at least we did not ﬁnd any examples where that did not work).
Revised: The relative adaptation strengths λ π , λ Σ and λ µ can always be set to 1 (at least we did not ﬁnd any examples where that did not work).

Original: Choose the number of iterations a an increasing function of K : the more components, the more free parameters, the more time to train is needed 2.
Revised: Choose the number of iterations a an increasing function of K : the more components, the more free parameters, the more time to train is needed 2.

Original: Flatten images, and put them as rows with random indices (as usual for SGD) into a tensor X 3.
Revised: Flatten images, and put them as rows with random indices (as usual for SGD) into a tensor X 3.

Original: Choose a quadratic number of components as n4.
Revised: Choose a quadratic number of components as n4.

Original: Initialize σ 0 to n/ 3 . 5.
Revised: Initialize σ 0 to n/ 3 . 5.

Original: Preliminary train the GMM with constant σ 0 .
Revised: Preliminary train the GMM with constant σ 0 .

Original: Select t 0 as the ﬁrst time step where loss does no longer decrease.
Revised: Select t 0 as the ﬁrst time step where loss does no longer decrease.

Original: It never hurts if t 0 is chosen too large, it just takes longer.
Revised: It never hurts if t 0 is chosen too large, it just takes longer.

Original: Select t ∞ by trial and error on the train set: start with very large values and use the smallest value that givesthe same ﬁnal lossvalue. Larger values for t ∞ never hurt but take longer.
Revised: gives loss

Original: Trainwith the chosen parameters!
Revised: Train the chosen parameters!

Revised: D M ORE R ESULTS

Revised: Visualization of prototypes training on different datasets (extracted 10 random classes, see Pf¨ulb &

Revised: Gepperth (2019) for details about the datasets).

Revised: Each result (in the form of centroid visualizations) is recorded after training for 14 epoch on a given dataset with K = 25 (see Figure 7 and Figure 8).