A
D
ISCRIMINATIVE G AUSSIAN M IXTURE
M
ODEL WITH
S PARSITY
Anonymous authors
Paper under double-blind review
A BSTRACT
In probabilistic classiﬁcation, a discriminative model based on the softmax function has a potential limitation in that it assumes unimodality for each class in the feature space.
The mixture model can address this issue, although it leads to an increase in the number of parameters.
We propose a sparse classiﬁer based on a discriminative GMM, referred to as a sparse discriminative Gaussian mixture (SDGM).
In the SDGM, a GMM-based discriminative model is trained via sparse
Bayesian learning.
Using this sparse learning framework, we can simultaneously remove redundant Gaussian components and reduce the number of parameters used in the remaining components during learning; this learning method reduces the model complexity, thereby improving the generalization capability.
Furthermore, the SDGM can be embedded into neural networks (NNs), such as convolutional NNs, and can be trained in an end-to-end manner.
Experimental results demonstrated that the proposed method outperformed the existing softmax-based discriminative models.
I NTRODUCTION
In probabilistic classiﬁcation, a discriminative model is an approach that assigns a class label c to an input sample x by estimating the posterior probability P ( c | x ) .
The posterior probability P ( c | x ) should correctly be modeled because it is not only related to classiﬁcation accuracy, but also to the conﬁdence of decision making in real-world applications such as medical diagnosis support.
In general, the model calculates the class posterior probability after nonlinear feature extraction using the softmax function.
Classically, a combination of the kernel method and the softmax function has been used.
The recent mainstream method is to use a deep neural network for representation learning and softmax for the calculation of the posterior probability.
Such a general procedure for developing a discriminative model potentially contains a limitation due to unimodality.
The softmax-based model, such as a fully connected (FC) layer with a softmax function that is often used in deep neural networks (NNs), assumes a unimodal Gaussian distribution for each class (details are shown in Appendix A).
Therefore, even if the feature space is transformed into discriminative space via the feature extraction part, P ( c | x ) cannot correctly be modeled if the multimodality remains, which leads to a decrease in accuracy.
Mixture models can address this issue.
Mixture models are widely used for generative models, with a Gaussian mixture model (GMM) as a typical example.
Mixture models are also effective in discriminative models; for example, discriminative GMMs have been applied successfully in various ﬁelds, e.g., speech recognition (T¨uske et al.
2015; Wang 2007).
However, the number of parameters increases if the number of mixture components increases, which may lead to over-ﬁtting and an increase in memory usage; this is useful if we can reduce the number of redundant parameters while maintaining multimodality.
In this paper, we propose a discriminative model with two important properties; multimodality and sparsity.
The proposed model is referred to as the sparse discriminative Gaussian mixture (SDGM).
In the SDGM, a GMM-based discriminative model is formulated and trained via sparse Bayesian learning.
This learning algorithm reduces memory usage without losing generalization capability by obtaining sparse weights while maintaining the multimodality of the mixture model.
[Figure]
The technical highlight of this study is twofold: One is that the SDGM ﬁnds the multimodal structure in the feature space and the other is that redundant Gaussian components are removed owing to sparse learning.
Figure 1 shows a comparison of the decision boundaries with other discriminative models.
The two-class data are from Ripley’s synthetic data (Ripley 2006), where two Gaussian components are used to generate data for each class.
The FC layer with the softmax function, which is often used in the last layer of deep NNs, assumes a unimodal Gaussian for each class, resulting in an inappropriate decision boundary.
Kernel Bayesian methods, such as the Gaussian process (GP) classiﬁer (Wenzel et al. 2019) and relevance vector machine (RVM) (Tipping 2001), estimate nonlinear decision boundaries using nonlinear kernels, whereas these methods cannot ﬁnd multimodal structures.
Although the discriminative GMM ﬁnds multimodal structure, this model retains redundant Gaussian components.
However, the proposed SDGM ﬁnds a multimodal structure of data while removing redundant components, which leads to an accurate decision boundary.
Furthermore, the SDGM can be embedded into NNs, such as convolutional NNs (CNNs), and trained in an end-to-end manner with an NN.
The proposed SDGM is also considered as a mixture, nonlinear, and sparse expansion of the logistic regression, and thus the SDGM can be used as the last layer of an NN for classiﬁcation by replacing it with the fully connected (FC) layer with a softmax activation function.
The contributions of this study are as follows:
• We propose a novel sparse classiﬁer based on a discriminative GMM.
The proposed SDGM has both multimodality and sparsity, thereby ﬂexibly estimating the posterior distribution of classes while removing redundant parameters.
Moreover, the SDGM automatically determines the number of components by simultaneously removing the redundant components during learning.
• From the perspective of the Bayesian kernel methods, the SDGM is considered as the expansion of the GP and RVM.
The SDGM can estimate the posterior probabilities more ﬂexibly than the GP and RVM owing to multimodality.
The experimental comparison using benchmark data demonstrated superior performance to the existing Bayesian kernel methods.
• This study connects both ﬁelds of probabilistic models and NNs.
From the equivalence of a discriminative model based on a Gaussian distribution to an FC layer, we demonstrate that the SDGM can be used as a module of a deep NN.
We also demonstrate that the SDGM exhibits superior performance to the FC layer with a softmax function via end-toend learning with an NN on the image recognition task.
R ELATED W ORK AND P OSITION OF T HIS S TUDY
The position of the proposed SDGM among the related methods are summarized in Figure 2.
Interestingly, by summarizing the relationships, we can conﬁrm that the three separately developed ﬁelds, generative models, discriminative models, and kernel Bayesian methods, are related to each other.
Starting from the Gaussian distribution, all the models shown in Figure 2 are connected via four kinds of arrows.
There is an undeveloped area in the upper right part, and the development of the area is the contribution of this study.
[Figure]
A (unimodal) Gaussian distribution is used as the most naive generative model in machine learning and is the foundation of this relationship diagram.
A GMM is the mixture expansion of the Gaussian distributions.
Since the GMM can express (almost) arbitrary continuous distributions using multiple Gaussian components, it has been utilized for a long time.
Since Gaussian ﬁtting requires numerous parameters, the sparsiﬁed versions of Gaussian (Hsieh et al.
and GMM (Gaiffas & Michel 2014) have been proposed.
The discriminative models and the generative models are mutually related (Lasserre et al. 2006; Minka 2005).
According to Lasserre et al.
(2006), the only difference between these models is their statistical parameter constraints.
Therefore, given a generative model, we can derive a corresponding discriminative model.
For example, discriminative models corresponding to the Gaussian mixture model have been proposed (Axelrod et al. 2006; Bahl et al. 1996;
Klautau et al. 2003;
Tsai & Chang 2002; Tsuji et al.
1999; T ¨ uske et al. 2015; Wang 2007).
They indicate more ﬂexible ﬁtting capability for classiﬁcation problems than the generative GMM because the discriminative models have a lower statistical bias than the generative models.
Furthermore, as shown by T ¨ uske et al.
; Variani et al.
, these models can be used as the last layer of the NN because these models output the class posterior probability.
From the perspective of the kernel Bayesian methods, the GP classiﬁer (Wenzel et al. 2019) and the mixture of GPs (MGP) (Luo & Sun 2017) are the Bayesian kernelized version of the logistic regression and the discriminative GMM, respectively.
The SDGM with kernelization is also regarded as a kernel Bayesian method because the posterior distribution of weights is estimated during learning instead of directly estimating the weights as points, as with the GP and MGP.
The RVM (Tipping 2001) is the sparse version of the GP classiﬁer and is the most important related study.
The learning algorithm of the SDGM is based on that of the RVM; however, it is extended for the mixture model.
If we use kernelization, the SDGM becomes one of the kernel Bayesian methods and is considered as the mixture expansion of the RVM or sparse expansion of the MGP.
Therefore, the classiﬁcation capability and sparsity are compared with kernel Bayesian methods in Section 4.1.
Otherwise, the
SDGM is considered as one of the discriminative models and can be embedded in an NN.
The comparison with other discriminative models is conducted in Section 4.2 via image classiﬁcation by combining a CNN.
S PARSE D ISCRIMINATIVE G AUSSIAN M IXTURE (SDGM)
The SDGM takes a continuous variable as its input and outputs the posterior probability of each class, acquiring a sparse structure by removing redundant components via sparse Bayesian learning.
Figure 3 shows how the SDGM is trained by removing unnecessary components while maintaining discriminability.
In this training, we set the initial number of components to three for each class.
As the training progresses, one of the components for each class gradually becomes small and is removed.
3. N OTATION
Let x ∈ R D be a continuous input variable and t c ( c ∈ { 1 , . .
, C } , C is the number of classes) be a discrete target variable that is coded in a one-of- C form, where t c = 1 if x belongs to class c , t c
=otherwise.
Also, let z cm be a discrete latent variable, and z cm = 1 when x from class c belongs to the m -th component ( m ∈ { 1 , . . .
, M c } , M c is the number of components for class c ), z cm =otherwise.
For simplicity, in this paper, the probabilities for classes and components are described using only c and m ; e.g., we use P ( c, m | x ) instead of P ( t c = 1 , z cm = 1 | x ) .
[Figure]
3. M ODEL FORMULATION
The posterior probabilities of each class c given x are calculated as follows:
[Equation]
where π cm is the mixture weight that is equivalent to the prior of each component P ( c, m ) .
It should be noted that we use w cm ∈ R H , which is the weight vector representing the m -th Gaussian component of class c .
The dimension of w cm , i.e., H , is the same as that of φ ; namely, H = 1 + D ( D + 3) / 2 .
Derivation.
Utilizing a Gaussian distribution as a conditional distribution of x given c and m , P ( x | c, m ) , the posterior probability of c given x , P ( c | x ) , is calculated as follows:
[Equation]
where µ cm ∈ R D and Σ cm ∈ R D
× D are the mean vector and the covariance matrix for component m in class c .
Since the calculation inside an exponential function in (4) is quadratic form, the conditional distributions can be transformed as follows:
[Equation]
where
[Equation]
Here, s cmij is the ( i, j )-th element of Σ − 1 cm .
3. D UAL FORM VIA KERNELIZATION
Since φ is a second-order polynomial form, we can derive the dual form of the SDGM using polynomial kernels.
By kernelization, we can treat the SDGM as the kernel Bayesian method as described in Section 2.
Let ψ cm ∈ R N be a novel weight vector for the c, m -th component.
Using ψ cm and the training dataset { x n } Nn =1 , the weight of the original form w cm is represented as
[Equation]
where φ ( x n ) is the transformation x n of using (2).
Then, (5) is reformulated as follows:
[Equation]
where K ( X , x ) is an N -dimensional vector that contains kernel functions deﬁned as k ( x n , x )
=
φ ( x n ) T φ ( x ) =
( x T n x + 1) 2 for its elements and X is a data matrix that has x T n in the n -th row.
Whereas the computational complexity of the original form in Section 3.2 increases in the order of the square of the input dimension D , the dimensionality of this dual form is proportional to N .
When we use this dual form, we use N and k ( x n , · ) instead of H and φ ( · ) , respectively.
3. L EARNING ALGORITHM
[Equation]
A set of training data and target value { x n , t nc } ( n = 1 , · · · , N ) is given.
We also deﬁne π and z as vectors that comprise π cm and z ncm as their elements, respectively.
As the prior distribution of the weight w cmh , we employ a Gaussian distribu- tion with a mean of zero.
Using a different precision parameter (inverse of the variance)
α cmh for each weight w cmh , the joint probability of all the weights is represented as follows:
[Equation]
where w and α are vectors with w cmh and α cmh as their elements, respectively.
During learning, we update not only w but also α .
If α cmh → ∞ , the prior (9) is 0; hence a sparse solution is obtained by optimizing α as shown in Figure 4.
Using these variables, the expectation of the log-likelihood function over z , J , is deﬁned as follows:
[Equation]
where T is a matrix with t nc as its element.
The variable r ncm in the right-hand side corresponds to
P ( m | c, x n ) and can be calculated as r ncm =
P ( c, m | x n ) /P
( c | x n ) .
The posterior probability of the weight vector w is described as follows:
[Equation]
An optimal w is obtained as the point where (10) is maximized.
The denominator of the right-hand side in (10) is called the evidence term, and we maximize it with respect to α .
However, this maximization problem cannot be solved analytically; therefore we introduce the Laplace approximation described as the following procedure.
With α ﬁxed, we obtain the mode of the posterior distribution of w .
The solution is given by the point where the following equation is maximized:
[Equation]
[Equation]
where A = diag α cmh .
We obtain the mode of (11) via Newton’s method.
The gradient and Hessian required for this estimation can be calculated as follows:
[Equation]
Each element of ∇ J and ∇∇ J is calculated as follows:
[Equation]
[Equation]
where δ cc (cid:48) mm (cid:48) is a variable that takes 1 if both c = c (cid:48) and m = m (cid:48) , 0 otherwise.
Hence, the posterior distribution of w can be approximated by a Gaussian distribution with a mean of ˆ w and a covariance matrix of Λ , where
[Equation]
Since the evidence term can be represented using the normalization term of this Gaussian distribu- tion, we obtain the following updating rule by calculating its derivative with respect to α cmh .
[Equation]
where λ cmh is the diagonal component of Λ .
The mixture weight π cm can be estimated using r ncm as follows:
[Equation]
(18) where N c is the number of training samples belonging to class c .
As described above, we obtain a sparse solution by alternately repeating the update of hyper-parameters, as in (17) and (18), and the posterior distribution estimation of w using the Laplace approximation.
As a result of the optimization, some of α cmh approach to inﬁnite values, and w cmh corresponding to α cmh have prior distributions with mean and variance both zero as shown in (4); hence such w cmh are removed because their posterior distributions are also with mean and variance both zero.
During the procedure, the { c, m } -th component is eliminated if π cm becomes 0 or all the weights w cmh corresponding to the component become 0.
The learning algorithm of the SDGM is summarized in Algorithm 1.
In this algorithm, the optimal weight is obtained as maximum a posterior solution.
We can obtain a sparse solution by optimizing the prior distribution set to each weight simultaneously with weight optimization.
E XPERIMENTS 4. C OMPARATIVE STUDY USING BENCHMARK DATA
To evaluate the capability of the SDGM quantitatively, we conducted a classiﬁcation experiment using benchmark datasets.
The datasets used in this experiment were Ripley’s synthetic data (Ripley 2006)
(Ripley hereinafter) and four datasets cited from (R¨atsch et al. 2001);
Banana, Waveform,
Titanic, and Breast Cancer.
Ripley is a synthetic dataset that is generated from a two-dimensional ( D = 2 ) Gaussian mixture model, and 250 and 1,000 samples are provided for training and test, [Table]
respectively. The number of classes is two ( C = 2 ), and each class comprises two components.
The remaining four datasets are all two-class ( C = 2 ) datasets, which comprise different data size and dimensionality.
Since they contain 100 training/test splits, we repeated experiments 100 times and then calculated average statistics.
For comparison, we used three kernel Bayesian methods: a GP classiﬁer, an MPG classiﬁer (Tresp 2001;
Luo & Sun 2017), and an RVM (Tipping 2001), which are closely related to the SDGM from the perspective of sparsity, multimodality, and Bayesian learning, as described in Section 2.
In the evaluation, we compared the recognition error rates for discriminability and the number of nonzero weights for sparsity on the test data.
The results of RVM were cited from (Tipping 2001).
By way of summary, the statistics were normalized by those of the SDGM, and the overall mean was shown.
Table 1 shows the recognition error rates and the number of nonzero weights for each method.
The results in Table 1 demonstrated that the SDGM achieved abetter accuracy on average compared to the other kernel Bayesian methods.
The SDGM is developed based on a Gaussian mixture model and is particularly effective for data where a Gaussian distribution can be assumed, such as the Ripley dataset.
Since the SDGM explicitly models multimodality, it could more accurately represent the sharp changes in decision boundaries near the border of components compared to the RVM, as shown in Figure 1.
Although the SDGM did not necessarily outperform the other methods in all datasets, it achieved the best accuracy on average.
In terms of sparsity, the number of initial weights for the SDGM is the same as MGP, and the SDGM reduced 90.0–99.5% of weights from the initial state due to the sparse Bayesian learning, which leads to drastically efﬁcient use of memory compared to non-sparse classiﬁers (GP and MGP).
The results above indicated that the SDGM demonstrated generalization capability and a sparsity simultaneously.
4. I MAGE CLASSIFICATION
In this experiment, the SDGM is embedded into a deep neural network.
Since the SDGM is differentiable with respect to the weights, SDGM can be embedded into a deep NN as a module and is trained in an end-to-end manner.
In particular, the SDGM plays the same role as the softmax function since the SDGM calculates the posterior probability of each class given an input vector.
We can show that a fully connected layer with the softmax is equivalent to the discriminative model based on a single Gaussian distribution for each class by applying a simple transformation (see Appendix
A), whereas the SDGM is based on the Gaussian mixture model.
To verify the difference between them, we conducted image classiﬁcation experiments.
Using a
CNN with a softmax function as a baseline, we evaluated the capability of SDGM by replacing softmax with the SDGM.
We also used a CNN with a softmax function trained with L 1 regularization, a CNN with a large margin softmax (Liu et al. 2016), and a CNN with the discriminative GMM as other baselines.
In this experiment, we used the original form of the SDGM.
To achieve sparse optimization during end-to-end training, we employed an approximated sparse Bayesian learning based on Gaussian dropout proposed by Molchanov et al.
This is because it is difﬁcult to execute the learning algorithm in Section 3.4 with backpropagation due to large computational costs for inverse matrix calculation of the Hessian in (16), which takes O ( N 3 ) .
4.2. D ATASETS AND EXPERIMENTAL SETUPS
We used the following datasets and experimental settings in this experiment. [Figure] [Table] [Figure]
[Table]
[Figure] [Table] [Figure] MNIST :
This dataset includes 10 classes of handwritten binary digit images of size 28 × 28 (LeCun et al. 1998).
We used 60,000 images as training data and 10,000 images as testing data.
As a feature extractor, we used a simple CNN that consists of ﬁve convolutional layers with four max pooling layers between them and a fully connected layer.
To visualize the learned CNN features, we ﬁrst set the output dimension of the fully connected layer of the baseline CNN as two ( D = 2 ).
Furthermore, we tested by increasing the output dimension of the fully connected layer from two to ten ( D = 10 ).
Fashion-MNIST : Fashion-MNIST (Xiao et al. 2017) includes 10 classes of binary fashion images with a size of 28 × 28 .
It includes 60,000 images for training data and 10,000 images for testing data.
We used the same CNN as in MNIST with 10 as the output dimension.
CIFAR-10 and CIFAR-100 : CIFAR-10 and CIFAR-100 (Krizhevsky & Hinton 2009) consist of 60,000 32 × 32 color images in 10 classes and 100 classes, respectively.
There are 50,000 training images and 10,000 test images for both datasets.
For these datasets, we trained DenseNet (Huang et al. 2017) with a depth of 40 and a growth rate of 12 as a baseline CNN.
For each dataset, the network was trained with a batch size of 64 for 100 epochs with a learning rate of 0.01 We used a weight decay of 1 .
× 10 − 5 and the Nesterov optimization algorithm (Sutskever et al. 2013) with a momentum of 0.9.
The network weights were initialized using the Glorot uniform (Glorot & Bengio 2010).
It should be noted that we did not employ additional techniques to increase classiﬁcation accuracy such as data augmentation, hyperparameter tuning, and pre-trained models; therefore, the accuracy of the baseline model did not reach the state-of-the-art.
This is because we considered that it is not essential to conﬁrm the effectiveness of the proposed method.
4.2. R ESULTS
Figure 5 shows the two-dimensional feature embeddings on the MNIST dataset.
Different feature embeddings were acquired for each method.
When softmax was used, the features spread in a fan shape and some parts of the distribution overlapped around the origin.
However, when the SDGM was used, the distribution for each class exhibited an ellipse shape and margins appeared between the class distributions.
This is because the SDGM is based on a Gaussian mixture model and functions to push the features into a Gaussian shape.
Table 2 shows the recognition error rates on each dataset.
SDGM achieved better performance than softmax.
Although sparse learning was ineffective in two out of ﬁve comparisons according to the comparison with the discriminative GMM, replacing softmax with SDGM was effective in all the comparisons.
As shown in Figure 5, SDGM can create margins between classes by pushing the features into a Gaussian shape.
This phenomenon positively affected classiﬁcation capability.
Although large-margin softmax, which has the effect of increasing the margin, and the discriminative
GMM, which can represent multimodality, also achieved relatively high accuracy, the SDGM can achieve the same level of accuracy with sparse weights.
C ONCLUSION
In this paper, we proposed a sparse classiﬁer based on a Gaussian mixture model (GMM), which is named sparse discriminative Gaussian mixture (SDGM).
In the SDGM, a GMM-based discriminative model was trained by sparse Bayesian learning.
This learning algorithm improved the generalization capability by obtaining a sparse solution and automatically determined the number of components by removing redundant components.
The SDGM can be embedded into neural networks (NNs) such as convolutional NNs and could be trained in an end-to-end manner.
In the experiments, we demonstrated that the SDGM could reduce the number of weights via sparse
Bayesian learning, thereby improving its generalization capability.
The comparison using benchmark datasets suggested that SDGM outperforms the conventional kernel Bayesian classiﬁers.
We also demonstrated that SDGM outperformed the fully connected layer with the softmax function when it was used as the last layer of a deep NN.
One of the limitations of this study is that the proposed sparse learning reduces redundant Gaussian components but cannot obtain the optimal number of components, which should be improved in future work.
Since the learning of the proposed method can be interpreted as the incorporation of the EM algorithm into the sparse Bayesian learning, we will tackle a theoretical analysis by utilizing the proofs for the EM algorithm (Wu 1983) and the sparse Bayesian learning (Faul &
Tipping 2001).
Furthermore, we would like to tackle the theoretical analysis of error bounds using the PAC-Bayesian theorem.
We will also develop a sparse learning algorithm for a whole deep NN structure including the feature extraction part.
This will improve the ability of the CNN for larger data classiﬁcation.
Further applications using probabilistic property of the proposed model such as semi-supervised learning, uncertainty estimation, and conﬁdence calibration will be considered.
S UPPLEMENTARY MATERIALS A R ELATIONSHIP BETWEEN THE DISCRIMINATIVE G AUSSIAN AND LOGISTIC REGRESSION
We show that a fully connected layer with the softmax function, or logistic regression, can be regarded as a discriminative model based on a Gaussian distribution by utilizing transformation of the equations.
Let us consider a case in which the class-conditional probability P ( x | c ) is a Gaussian distribution.
In this case, we can omit m from the equations (3)–(6).
If all classes share the same covariance matrix and the mixture weight π cm , the terms π cm in (1), x 21 , x 1 x 2 , · · · , x 1 x D , x 22 , x 2 x 3 , · · · , x 2 x D , · · · , x 2 D in (2), and − 12 s c 11 , · · · , − 12 s cDD in (6) can be canceled; hence the calculation of the posterior probability P ( c | x ) is also simpliﬁed as
[Equation]
where
[Equation]
This is equivalent to a fully connected layer with the softmax function, or linear logistic regression.
B E VALUATION OF CHARACTERISTICS USING SYNTHETIC DATA
To evaluate the characteristics of the SDGM, we conducted classiﬁcation experiments using synthetic data.
The dataset comprises two classes.
The data were sampled from a Gaussian mixture model with eight components for each class.
The numbers of training data and test data wereand 1,600, respectively.
The scatter plot of this dataset is shown in Figure 6.
In the evaluation, we calculated the error rates for the training data and the test data, the number of components after training, the number of nonzero weights after training, and the weight reduction ratio (the ratio of the number of the nonzero weights to the number of initial weights), by varying the number of initial components as 2 , 4 , 8 , . . .
We repeated evaluation ﬁve times while regenerating the training and test data and calculated the average value for each evaluation criterion.
We used the dual form of the SDGM in this experiment.
Figure 6 displays the changes in the learned class boundaries according to the number of initial components.
When the number of components is small, such as that shown in Figure 6(a), the decision boundary is simple; therefore, the classiﬁcation performance is insufﬁcient.
However, according to the increase in the number of components, the decision boundary ﬁts the actual class boundaries.
It is noteworthy that the SDGM learns the GMM as a discriminative model instead of a generative model; an appropriate decision boundary was obtained even if the number of components for the model is less than the actual number (e.g., 6(c)).
Figure 7 shows the evaluation results of the characteristics.
Figures 7(a), (b), (c), and (d) show the recognition error rate, number of components after training, number of nonzero weights after training, and weight reduction ratio, respectively.
The horizontal axis shows the number of initial components in all the graphs.
In Figure 7(a), the recognition error rates for the training data and test data are almost the same with the few numbers of components and decrease according to the increase in the number of initial components while it is 2 to 6.
This implied that the representation capability was insufﬁcient when the number of components was small, and that the network could not accurately separate the classes.
Meanwhile, changes in the training and test error rates were both ﬂat when the number of initial components exceeded eight, even though the test error rates were slightly higher than the training error rate.
In general, the training error decreases and the test error increases when the complexity of
[Figure] [Figure]
the classiﬁer is increased.
However, the SDGM suppresses the increase in complexity using sparse
Bayesian learning, thereby preventing overﬁtting.
In Figure 7(b), the number of components after training corresponds to the number of initial components until the number of initial components is eight.
When the number of initial components exceeds ten, the number of components after training tends to be reduced.
In particular, eight components are reduced when the number of initial components is 20.
The results above indicate the
SDGM can reduce unnecessary components.
From the results in Figure 7(c), we conﬁrm that the number of nonzero weights after training increases according to the increase in the number of initial components.
This implies that the complexity of the trained model depends on the number of initial components, and that the minimum number of components is not always obtained.
Meanwhile, in Figure 7(d), the weight reduction ratio increases according to the increase in the number of initial components.
This result suggests that the larger the number of initial weights, the more weights were reduced.
Moreover, the weight reduction ratio is greater than 99 % in any case.
The results above indicate that the SDGM can prevent overﬁtting by obtaining high sparsity and can reduce unnecessary components.
C D ETAILS OF INITIALIZATION
In the experiments during this study, each trainable parameters for the m -th component of the c -th class were initialized as follows ( H = 1 + D
( D + 3) / 2 , where D is the input dimension, for the original form and H = N , where N is the number of the training data, for the kernelized form):
• w cm (for the original form): A zero vector 0 ∈ R H .
• ψ cm (for the kernelized form): A zero vector 0 ∈ R H .
• α cm : An all-ones vector 1 ∈ R H .
• π cm : A scalar 1 (cid:80) Cc
=
1 M c , where C is the number of classes and M c is the number of components for the c -th class.
• r ncm :
Initialized based on the results of k -means clustering applied to the training data; r ncm
= 1 if the n -th sample belongs to class c and is assigned to the m -th component by k -means clustering, r ncm
= 0 otherwise.