Original: SDGM: S PARSE B AYESIAN C LASSIFIER B ASED ON A D ISCRIMINATIVE G AUSSIAN M IXTURE M ODEL
Revised: SDGM: S PARSE B AYESIAN C LASSIFIER B ASED ON A D ISCRIMINATIVE G AUSSIAN M IXTURE M ODEL

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In probabilistic classiﬁcation, a discriminative model based on Gaussian mixture exhibits ﬂexible ﬁtting capability.
Revised: In probabilistic classiﬁcation, a discriminative model based on Gaussian mixture exhibits ﬂexible ﬁtting capability.

Original: Nevertheless, it is difﬁcult to determine the number of components.
Revised: Nevertheless, it is difﬁcult to determine the number of components.

Original: We propose a sparse classiﬁer based on a discriminative
Revised: We propose a sparse classiﬁer based on a discriminative

Original: Gaussian mixture model (GMM), which is named sparse discriminative Gaussian mixture (SDGM).
Revised: Gaussian mixture model (GMM), which is named sparse discriminative Gaussian mixture (SDGM).

Original: In the SDGM, a GMM-based discriminative model is trained by sparse Bayesian learning.
Revised: In the SDGM, a GMM-based discriminative model is trained by sparse Bayesian learning.

Original: This learning algorithm improves the generalization capability by obtaining a sparse solution and automatically determines the number of components by removing redundant components.
Revised: This learning algorithm improves the generalization capability by obtaining a sparse solution and automatically determines the number of components by removing redundant components.

Original: The SDGM can be embedded into neural networks (NNs) such as convolutional NNs and can be trained in an end-to-end manner.
Revised: The SDGM can be embedded into neural networks (NNs) such as convolutional NNs and can be trained in an end-to-end manner.

Original: Experimental results indicated that the proposed method prevented overﬁtting by obtaining sparsity.
Revised: Experimental results indicated that the proposed method prevented overﬁtting by obtaining sparsity.

Original: Furthermore, we demonstrated that the proposed method outperformed a fully connected layer with the softmax function in certain cases when it was used as the last layer of a deep NN.
Revised: Furthermore, we demonstrated that the proposed method outperformed a fully connected layer with the softmax function in certain cases when it was used as the last layer of a deep NN.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In supervised classiﬁcation, probabilistic classiﬁcation is an approach that assigns a class label c to an input sample x by estimating the posterior probability P ( c | x ) .
Revised: In supervised classiﬁcation, probabilistic classiﬁcation is an approach that assigns a class label c to an input sample x by estimating the posterior probability P ( c | x ) .

Original: This approach is primarily categorized into two types of models: discriminative model and generative model.
Revised: This approach is primarily categorized into two types of models: discriminative model and generative model.

Original: The former optimizes the posterior distribution P ( c | x ) directly on a training set, whereas the latter ﬁnds the class conditional distribution P ( x
Revised: The former optimizes the posterior distribution P ( c | x ) directly on a training set, whereas the latter ﬁnds the class conditional distribution P ( x

Original: | c ) and class prior P ( c ) and subsequently derives the posterior distribution P ( c | x ) using Bayes’ rule.
Revised: | c ) and class prior P ( c ) and subsequently derives the posterior distribution P ( c | x ) using Bayes’ rule.

Original: The discriminative model and generative model are mutually related (Lasserre et al., 2006; Minka, 2005).
Revised: The discriminative model and generative model are mutually related (Lasserre et al., 2006; Minka, 2005).

Original: According to Lasserre et al.
Revised: According to Lasserre et al.

Original: (2006), the only difference between these models is their statistical parameter constraints.
Revised: (2006), the only difference between these models is their statistical parameter constraints.

Original: Therefore, given a certain generative model, we can derive a corresponding discriminative model.
Revised: Therefore, given a certain generative model, we can derive a corresponding discriminative model.

Original: For example, the discriminative model corresponding to a unimodal
Revised: For example, the discriminative model corresponding to a unimodal

Original: Gaussian distribution is logistic regression (see Appendix A for derivation).
Revised: Gaussian distribution is logistic regression (see Appendix A for derivation).

Original: Several discriminative models corresponding to the Gaussian mixture model (GMM) have been proposed (Axelrod et al., 2006; Bahl et al., 1996; Klautau et al., 2003; Tsai & Chang, 2002; Tsuji et al., 1999; T¨uske et al., 2015; Wang, 2007).
Revised: Several discriminative models corresponding to the Gaussian mixture model (GMM) have been proposed (Axelrod et al., 2006; Bahl et al., 1996; Klautau et al., 2003; Tsai & Chang, 2002; Tsuji et al., 1999; T¨uske et al., 2015; Wang, 2007).

Original: They indicate more ﬂexible ﬁtting capability than the generative GMM and have been applied successfully in ﬁelds such as speech recognition (Axelrod et al., 2006; T¨uske et al., 2015; Wang, 2007).
Revised: They indicate more ﬂexible ﬁtting capability than the generative GMM and have been applied successfully in ﬁelds such as speech recognition (Axelrod et al., 2006; T¨uske et al., 2015; Wang, 2007).

Original: The problem to address in mixture models such as the GMM is the determination of the number of components M .
Revised: The problem to address in mixture models such as the GMM is the determination of the number of components M .

Original: Classically, Akaike’s information criterion and the Bayesian information criterion have been used; nevertheless, they require a considerable computational cost because a likelihood must be calculated for every candidate component number.
Revised: Classically, Akaike’s information criterion and the Bayesian information criterion have been used; nevertheless, they require a considerable computational cost because a likelihood must be calculated for every candidate component number.

Original: In the generative GMM, methods that optimize M during learning exist (Crouse et al., 2011; ˇStep´anov´a & Vavreˇcka, 2018).
Revised: In the generative GMM, methods that optimize M during learning exist (Crouse et al., 2011; ˇStep´anov´a & Vavreˇcka, 2018).

Original: However, in a discriminative GMM, a method to optimize M simultaneously during learning has not been clearly formulated.
Revised: However, in a discriminative GMM, a method to optimize M simultaneously during learning has not been clearly formulated.

Original: In this paper, we propose a novel GMM having two important properties: sparsity and discriminability, which is named sparse discriminative Gaussian mixture (SDGM).
Revised: In this paper, we propose a novel GMM having two important properties: sparsity and discriminability, which is named sparse discriminative Gaussian mixture (SDGM).

Original: In the SDGM, a GMM-based discriminative model is trained by sparse Bayesian learning.
Revised: In the SDGM, a GMM-based discriminative model is trained by sparse Bayesian learning.

Original: This learning algorithm improves the generalization capability by obtaining a sparse solution and determines the number of components automatically by removing redundant components.
Revised: This learning algorithm improves the generalization capability by obtaining a sparse solution and determines the number of components automatically by removing redundant components.

Original: Furthermore, the SDGM can be embedded into
Revised: Furthermore, the SDGM can be embedded into

Original: [Figure]
Revised: [Figure]

Original: neural networks (NNs) such as convolutional NNs and trained in an end-to-end manner with an NN.
Revised: neural networks (NNs) such as convolutional NNs and trained in an end-to-end manner with an NN.

Original: To the authors best knowledge, there is no GMM that has both of sparsity and discriminability.
Revised: To the authors best knowledge, there is no GMM that has both of sparsity and discriminability.

Original: The contributions of this study are as follows:
Revised: The contributions of this study are as follows:

Original: • We propose a novel sparse classiﬁer based on a discriminative GMM.
Revised: • We propose a novel sparse classiﬁer based on a discriminative GMM.

Original: The proposed SDGM has both sparsity and discriminability, and determines the number of components automatically.
Revised: The proposed SDGM has both sparsity and discriminability, and determines the number of components automatically.

Original: The SDGM can be considered as the theoretical extension of the discriminative GMM and the relevance vector machine (RVM)
Revised: The SDGM can be considered as the theoretical extension of the discriminative GMM and the relevance vector machine (RVM)

Original: (Tipping, 2001).
Revised: (Tipping, 2001).

Original: • This study attempts to connect both ﬁelds of probabilistic models and NNs.
Revised: • This study attempts to connect both ﬁelds of probabilistic models and NNs.

Original: From the equivalence of a discriminative model based on
Revised: From the equivalence of a discriminative model based on

Original: Gaussian distribution to a fully connected layer, we demonstrate that the SDGM can be used as a module of a deep NN.
Revised: Gaussian distribution to a fully connected layer, we demonstrate that the SDGM can be used as a module of a deep NN.

Original: We also show that the SDGM can show superior performance than the fully connected layer with a softmax function via an end-to-end learning with an NN on the image recognition task.
Revised: We also show that the SDGM can show superior performance than the fully connected layer with a softmax function via an end-to-end learning with an NN on the image recognition task.

Original:  S PARSE D ISCRIMINATIVE G AUSSIAN M IXTURE (SDGM)
Revised:  S PARSE D ISCRIMINATIVE G AUSSIAN M IXTURE (SDGM)

Original: An SDGM takes a continuous variable x ∈ R D as its input and outputs its posterior probability
Revised: An SDGM takes a continuous variable x ∈ R D as its input and outputs its posterior probability

Original: P ( c | x ) for each class c ∈ { 1 , . . . , C } .
Revised: P ( c | x ) for each class c ∈ { 1 , . . . , C } .

Original: An SDGM acquires a sparse structure by removing redundant components via sparse Bayesian learning.
Revised: An SDGM acquires a sparse structure by removing redundant components via sparse Bayesian learning.

Original: Figure 1 shows how the SDGM is trained by removing unnecessary components while keeping discriminability.
Revised: Figure 1 shows how the SDGM is trained by removing unnecessary components while keeping discriminability.

Original: The two-class training data are from Ripley’s synthetic data (Ripley, 2006), where a Gaussian mixture model with two components is used for generating data of each class.
Revised: The two-class training data are from Ripley’s synthetic data (Ripley, 2006), where a Gaussian mixture model with two components is used for generating data of each class.

Original: In this training, we set the initial number of components to three for each class.
Revised: In this training, we set the initial number of components to three for each class.

Original: As the training progresses, one of the components for each class becomes small gradually and is removed.
Revised: As the training progresses, one of the components for each class becomes small gradually and is removed.

Original: 2. M ODEL FORMULATION
Revised: 2. M ODEL FORMULATION

Original: The posterior probabilities for each class c is calculated as follows:
Revised: The posterior probabilities for each class c is calculated as follows:

Original: [Equation]
Revised: [Equation]

Original: ) where M c is the number of components for class c and π cm is the mixture weight that is equivalent to the prior of each component P ( c, m ) .
Revised: ) where M c is the number of components for class c and π cm is the mixture weight that is equivalent to the prior of each component P ( c, m ) .

Original: It should be noted that we use w cm ∈ R H , which is the weight vector representing the m -th Gaussian component of class c .
Revised: It should be noted that we use w cm ∈ R H , which is the weight vector representing the m -th Gaussian component of class c .

Original: The dimension of w cm , i.e.,
Revised: The dimension of w cm , i.e.,

Original: H , is the same as that of φ ; namely, H = 1 + D
Revised: H , is the same as that of φ ; namely, H = 1 + D

Original: ( D + 3) / 2 .
Revised: ( D + 3) / 2 .

Original: Derivation.
Revised: Derivation.

Original: Utilizing Gaussian distribution as a conditional distribution of x given c and m ,
Revised: Utilizing Gaussian distribution as a conditional distribution of x given c and m ,

Original: P ( x | c, m ) , the posterior probability of c given x , P ( c | x ) , is calculated as follows:
Revised: P ( x | c, m ) , the posterior probability of c given x , P ( c | x ) , is calculated as follows:

Original: [Equation]
Revised: [Equation]

Original: where µ cm ∈ R D and Σ cm ∈ R D
Revised: where µ cm ∈ R D and Σ cm ∈ R D

Original: × D are the mean vector and the covariance matrix for component m in class c .
Revised: × D are the mean vector and the covariance matrix for component m in class c .

Original: Since the calculation inside an exponential function in (5) is quadratic form, the conditional distributions can be transformed as follows:
Revised: Since the calculation inside an exponential function in (5) is quadratic form, the conditional distributions can be transformed as follows:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: i =
Revised: i =

Original: D (cid:88)
Revised: D (cid:88)

Original: j = s cmij µ cmi µ cmj ,
Revised: j = s cmij µ cmi µ cmj ,

Original: D (cid:88) i = s cmi 1 µ cmi , . . . ,
Revised: D (cid:88) i = s cmi 1 µ cmi , . . . ,

Original: D (cid:88) i =
Revised: D (cid:88) i =

Original: s cmiD µ cmi , − 1 2 s cm 11 , − s cm 12 , . .
Revised: s cmiD µ cmi , − 1 2 s cm 11 , − s cm 12 , . .

Original: , − s cm 1 D , − 1 2 s cm 22 , . .
Revised: , − s cm 1 D , − 1 2 s cm 22 , . .

Original: , − 1 2 s cmDD (cid:35) T .
Revised: , − 1 2 s cmDD (cid:35) T .

Original: Here, s cmij is the ( i, j )-th element of Σ − 1 cm .
Revised: Here, s cmij is the ( i, j )-th element of Σ − 1 cm .

Original: 2. L EARNING ALGORITHM
Revised: 2. L EARNING ALGORITHM

Original: Algorithm 1 shows the training of the SDGM.
Revised: Algorithm 1 shows the training of the SDGM.

Original: In this algorithm, the optimal weight is obtained as maximum a posteriori solution.
Revised: In this algorithm, the optimal weight is obtained as maximum a posteriori solution.

Original: We can obtain a sparse solution by optimizing the prior distribution set to each weight simultaneously with weight optimization.
Revised: We can obtain a sparse solution by optimizing the prior distribution set to each weight simultaneously with weight optimization.

Original: A set of training data and target value { x n , t nc } ( n = 1 , · · · , N ) is given.
Revised: A set of training data and target value { x n , t nc } ( n = 1 , · · · , N ) is given.

Original: The target t nc is coded in a one-of- K form, where t nc = 1 if the n -th sample belongs to class c , t nc = 0 otherwise.
Revised: The target t nc is coded in a one-of- K form, where t nc = 1 if the n -th sample belongs to class c , t nc = 0 otherwise.

Original: A binary random variable z ncm is introduced.
Revised: A binary random variable z ncm is introduced.

Original: The variable z ncm = 1 when the n -th sample from class c belongs to the m -th component.
Revised: The variable z ncm = 1 when the n -th sample from class c belongs to the m -th component.

Original: Otherwise, z ncm = 0 .
Revised: Otherwise, z ncm = 0 .

Original: This variable is required for the optimization of the mixture weight π cm .
Revised: This variable is required for the optimization of the mixture weight π cm .

Original: We also deﬁne π and z as vectors that comprise π cm and z ncm as their elements, respectively.
Revised: We also deﬁne π and z as vectors that comprise π cm and z ncm as their elements, respectively.

Original: As the prior distribution of the weight w cmh , we employ a
Revised: As the prior distribution of the weight w cmh , we employ a

Original: Gaussian distribution with a mean of zero.
Revised: Gaussian distribution with a mean of zero.

Original: Using a different precision parameter (inverse of the variance)
Revised: Using a different precision parameter (inverse of the variance)

Original: α cmh for each weight w cmh , the joint probability of all the weights is represented as follows:
Revised: α cmh for each weight w cmh , the joint probability of all the weights is represented as follows:

Original: [Equation]
Revised: [Equation]

Original: where w and α are vectors with w cmh and α cmh as their elements, respectively.
Revised: where w and α are vectors with w cmh and α cmh as their elements, respectively.

Original: During learning, we update not only w but also α .
Revised: During learning, we update not only w but also α .

Original: If α cmh → ∞ , the prior (8) is 0; hence a sparse solution is obtained by optimizing α .
Revised: If α cmh → ∞ , the prior (8) is 0; hence a sparse solution is obtained by optimizing α .

Original: Using these variables, the expectation of the log-likelihood function over z , J , is deﬁned as follows:
Revised: Using these variables, the expectation of the log-likelihood function over z , J , is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: where T is a matrix with t nc as its element.
Revised: where T is a matrix with t nc as its element.

Original: The training data matrix X contains x T n in the n -th row.
Revised: The training data matrix X contains x T n in the n -th row.

Original: The variable r ncm in the right-hand side corresponds to P ( m | c, x n ) and can be calculated as r ncm =
Revised: The variable r ncm in the right-hand side corresponds to P ( m | c, x n ) and can be calculated as r ncm =

Original: P ( c, m | x n ) /P
Revised: P ( c, m | x n ) /P

Original: ( c | x n ) .
Revised: ( c | x n ) .

Original: The posterior probability of the weight vector w is described as follows:
Revised: The posterior probability of the weight vector w is described as follows:

Original: [Equation]
Revised: [Equation]

Original: An optimal w is obtained as the point where (10) is maximized.
Revised: An optimal w is obtained as the point where (10) is maximized.

Original: The denominator of the right-hand side in (10) is called the evidence term, and we maximize it with respect to α .
Revised: The denominator of the right-hand side in (10) is called the evidence term, and we maximize it with respect to α .

Original: However, this maximization problem cannot be solved analytically; therefore we introduce the Laplace approximation described as the following procedure.
Revised: However, this maximization problem cannot be solved analytically; therefore we introduce the Laplace approximation described as the following procedure.

Original: With α ﬁxed, we obtain the mode of the posterior distribution of w .
Revised: With α ﬁxed, we obtain the mode of the posterior distribution of w .

Original: The solution is given by the point where the following equation is maximized:
Revised: The solution is given by the point where the following equation is maximized:

Original: [Equation]
Revised: [Equation]

Original: where A = diag α cmh .
Revised: where A = diag α cmh .

Original: We obtain the mode of (11) via Newton’s method.
Revised: We obtain the mode of (11) via Newton’s method.

Original: The gradient and Hessian required for this estimation can be calculated as follows:
Revised: The gradient and Hessian required for this estimation can be calculated as follows:

Original: [Equation]
Revised: [Equation]

Original: Each element of ∇ J and ∇∇ J is calculated as follows:
Revised: Each element of ∇ J and ∇∇ J is calculated as follows:

Original: [Equation]
Revised: [Equation]

Original:  where δ cc (cid:48) mm (cid:48) is a variable that takes 1 if both c = c (cid:48) and m = m (cid:48) , 0 otherwise.
Revised: 15) where δ cc (cid:48) mm (cid:48) is a variable that takes 1 if both c = c (cid:48) and m = m (cid:48) , 0 otherwise.

Original: Hence, the posterior distribution of w can be approximated by a Gaussian distribution with a mean of ˆ w and a covariance matrix of Λ , where
Revised: Hence, the posterior distribution of w can be approximated by a Gaussian distribution with a mean of ˆ w and a covariance matrix of Λ , where

Original: [Equation]
Revised: [Equation]

Original: Because the evidence term can be represented using the normalization term of this Gaussian distribution, we obtain the following updating rule by calculating its derivative with respect to α cmh .
Revised: Because the evidence term can be represented using the normalization term of this Gaussian distribution, we obtain the following updating rule by calculating its derivative with respect to α cmh .

Original: [Equation]
Revised: [Equation]

Original: where λ cmh is the orthogonal component of Λ .
Revised: where λ cmh is the diagonal component of Λ .

Original: The mixture weight π cm can be estimated using r ncm as follows:
Revised: The mixture weight π cm can be estimated using r ncm as follows:

Original: [Equation]
Revised: [Equation]

Original: where N c is the number of training samples belonging to class c .
Revised: where N c is the number of training samples belonging to class c .

Original: As described above, we obtain a sparse solution by alternately repeating the update of hyper-parameters, as described in (17) and (18) and the posterior distribution estimation of w using the Laplace approximation.
Revised: As described above, we obtain a sparse solution by alternately repeating the update of hyper-parameters, as described in (17) and (18) and the posterior distribution estimation of w using the Laplace approximation.

Original: During the procedure, the { c, m } -th component is eliminated if π cm becomes 0 or all the weights w cmh corresponding to the component become 0.
Revised: During the procedure, the { c, m } -th component is eliminated if π cm becomes 0 or all the weights w cmh corresponding to the component become 0.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original:  E XPERIMENTS 3. E VALUATION OF CHARACTERISTICS USING SYNTHETIC DATA
Revised:  E XPERIMENTS 3. E VALUATION OF CHARACTERISTICS USING SYNTHETIC DATA

Original: To evaluate the characteristics of the SDGM
Revised: To evaluate the characteristics of the SDGM

Original: [Figure]
Revised: [Figure]

Original: we conducted classiﬁcation experiments using synthetic data.
Revised: we conducted classiﬁcation experiments using synthetic data.

Original: The dataset comprises two classes.
Revised: The dataset comprises two classes.

Original: The data were sampled from a Gaussian mixture model with eight components for each class.
Revised: The data were sampled from a Gaussian mixture model with eight components for each class.

Original: The numbers of training data and test data wereand 1,600, respectively.
Revised: The numbers of training data and test data wereand 1,600, respectively.

Original: The scatter plot of this dataset is shown in Figure 2.
Revised: The scatter plot of this dataset is shown in Figure 2.

Original: In the evaluation, we calculated the error rates for the training data and the test data, the number of components after training, the number of nonzero weights after training, and the weight reduction ratio (the ratio of the number of the nonzero weights to the number of initial weights), by varying the number of initial components as 2 , 4 , 8 , . . .
Revised: In the evaluation, we calculated the error rates for the training data and the test data, the number of components after training, the number of nonzero weights after training, and the weight reduction ratio (the ratio of the number of the nonzero weights to the number of initial weights), by varying the number of initial components as 2 , 4 , 8 , . . .

Original: Figure 2 displays the changes in the learned class boundaries according to the number of initial components.
Revised: Figure 2 displays the changes in the learned class boundaries according to the number of initial components.

Original: When the number of components is small, such as that shown in Figure 2(a), the decision boundary is simple; therefore, the classiﬁcation performance is insufﬁcient.
Revised: When the number of components is small, such as that shown in Figure 2(a), the decision boundary is simple; therefore, the classiﬁcation performance is insufﬁcient.

Original: However, according to the increase in the number of components, the decision boundary ﬁts the actual class boundaries.
Revised: However, according to the increase in the number of components, the decision boundary ﬁts the actual class boundaries.

Original: It is noteworthy that the SDGM learns the GMM as a discriminative model instead of a generative model; an appropriate decision boundary was obtained even if the number of components for the model is less than the actual number (e.g., 2(c)).
Revised: It is noteworthy that the SDGM learns the GMM as a discriminative model instead of a generative model; an appropriate decision boundary was obtained even if the number of components for the model is less than the actual number (e.g., 2(c)).

Original: Figure 3 shows the evaluation results of the characteristics.
Revised: Figure 3 shows the evaluation results of the characteristics.

Original: Figures 3(a), (b), (c), and (d) show the recognition error rate, number of components after training, number of nonzero weights after training, and weight reduction ratio, respectively.
Revised: Figures 3(a), (b), (c), and (d) show the recognition error rate, number of components after training, number of nonzero weights after training, and weight reduction ratio, respectively.

Original: The horizontal axis shows the number of initial components in all the graphs.
Revised: The horizontal axis shows the number of initial components in all the graphs.

Original: In Figure 3(a), the recognition error rates for the training data and test data are almost the same with the few number of components, and decrease according to the increase in the number of initial components while it is 2 to 6.
Revised: In Figure 3(a), the recognition error rates for the training data and test data are almost the same with the few number of components, and decrease according to the increase in the number of initial components while it is 2 to 6.

Original: This implied that the representation capability was insufﬁcient when the number of components was small, and that the network could not accurately separate the classes.
Revised: This implied that the representation capability was insufﬁcient when the number of components was small, and that the network could not accurately separate the classes.

Original: Meanwhile, changes in the training and test error rates were both ﬂat when the number of initial components exceeded eight, even though the test error rates were slightly higher than the training error rate.
Revised: Meanwhile, changes in the training and test error rates were both ﬂat when the number of initial components exceeded eight, even though the test error rates were slightly higher than the training error rate.

Original: In general, the training error decreases and the test error increases when the complexity of the classiﬁer is increased.
Revised: In general, the training error decreases and the test error increases when the complexity of the classiﬁer is increased.

Original: However, the SDGM suppresses the increase in complexity using sparse
Revised: However, the SDGM suppresses the increase in complexity using sparse

Original: Bayesian learning, thereby preventing overﬁtting.
Revised: Bayesian learning, thereby preventing overﬁtting.

Original: [Table]
Revised: [Table]

Original: In Figure 3(b)
Revised: In Figure 3(b)

Original: [Table]
Revised: [Table]

Original: the number of components after training corresponds to the number of initial components until the number of initial components is eight.
Revised: the number of components after training corresponds to the number of initial components until the number of initial components is eight.

Original: When the number of initial components exceeds ten, the number of components after training tends to be reduced.
Revised: When the number of initial components exceeds ten, the number of components after training tends to be reduced.

Original: In particular, eight components are reduced when the number of initial components is 20.
Revised: In particular, eight components are reduced when the number of initial components is 20.

Original: The results above indicate the
Revised: The results above indicate the

Original: SDGM can reduce unnecessary components.
Revised: SDGM can reduce unnecessary components.

Original: From the results in Figure 3(c)
Revised: From the results in Figure 3(c)

Original: [Table]
Revised: [Table]

Original: we conﬁrm that the number of nonzero weights after training increases according to the increase in the number of initial components.
Revised: we conﬁrm that the number of nonzero weights after training increases according to the increase in the number of initial components.

Original: This implies that the complexity of the trained model depends on the number of initial components
Revised: This implies that the complexity of the trained model depends on the number of initial components

Original: [Table]
Revised: [Table]

Original: and that the minimum number of components is not always obtained.
Revised: and that the minimum number of components is not always obtained.

Original: Meanwhile, in Figure 3(d), the weight reduction ratio increases according to the increase in the number of initial components.
Revised: Meanwhile, in Figure 3(d), the weight reduction ratio increases according to the increase in the number of initial components.

Original: This result suggests that the larger the number of initial weights, the more weights were reduced.
Revised: This result suggests that the larger the number of initial weights, the more weights were reduced.

Original: Moreover, the weight reduction ratio is greater than 99 % in any case.
Revised: Moreover, the weight reduction ratio is greater than 99 % in any case.

Original: The results above indicate that the SDGM can prevent overﬁtting by obtaining high sparsity and can reduce unnecessary components.
Revised: The results above indicate that the SDGM can prevent overﬁtting by obtaining high sparsity and can reduce unnecessary components.

Original: 3. C OMPARATIVE STUDY USING BENCHMARK DATA
Revised: 3. C OMPARATIVE STUDY USING BENCHMARK DATA

Original: To evaluate the capability of the SDGM quantitatively, we conducted a classiﬁcation experiment using benchmark datasets.
Revised: To evaluate the capability of the SDGM quantitatively, we conducted a classiﬁcation experiment using benchmark datasets.

Original: The datasets used in this experiment were Ripley’s synthetic data (Ripley, 2006)
Revised: The datasets used in this experiment were Ripley’s synthetic data (Ripley, 2006)

Original: (Ripley hereinafter) and four datasets cited from R ¨ atsch et al.
Revised: (Ripley hereinafter) and four datasets cited from R ¨ atsch et al.

Original: ; Banana, Waveform,
Revised: ; Banana, Waveform,

Original: Titanic, and Breast Cancer.
Revised: Titanic, and Breast Cancer.

Original: Ripley is a synthetic dataset that is generated from a two-dimensional ( D = 2 ) Gaussian mixture model, and 250 and 1,000 samples are provided for training and test, respectively.
Revised: Ripley is a synthetic dataset that is generated from a two-dimensional ( D = 2 ) Gaussian mixture model, and 250 and 1,000 samples are provided for training and test, respectively.

Original: The number of classes is two ( C = 2 ), and each class comprises two components.
Revised: The number of classes is two ( C = 2 ), and each class comprises two components.

Original: The remaining four datasets are all two-class ( C = 2 ) datasets, which comprise different data size and dimensionality.
Revised: The remaining four datasets are all two-class ( C = 2 ) datasets, which comprise different data size and dimensionality.

Original: Since they contain 100 training/test splits, we repeated experiments for 100 times and then calculated average statistics.
Revised: Since they contain 100 training/test splits, we repeated experiments for 100 times and then calculated average statistics.

Original: For comparison, we used three classiﬁers that can obtain a sparse solution: a linear logistic regression (LR) with l 1 constraint, a support vector machine (SVM) (Cortes & Vapnik, 1995) and a relevance vector machine (RVM) (Tipping, 2001).
Revised: For comparison, we used three classiﬁers that can obtain a sparse solution: a linear logistic regression (LR) with l 1 constraint, a support vector machine (SVM) (Cortes & Vapnik, 1995) and a relevance vector machine (RVM) (Tipping, 2001).

Original: In the evaluation, we compared the recognition error rates for discriminability and number of nonzero weights for sparsity on the test data.
Revised: In the evaluation, we compared the recognition error rates for discriminability and number of nonzero weights for sparsity on the test data.

Original: The results of SVM and RVM were cited from Tipping (2001).
Revised: The results of SVM and RVM were cited from Tipping (2001).

Original: For ablation study, we also tested our
Revised: For ablation study, we also tested our

Original: SDGM without sparse learning by omitting the update of α .
Revised: SDGM without sparse learning by omitting the update of α .

Original: By way of summary, the statistics were normalized by those of the SDGM and the overall mean was shown.
Revised: By way of summary, the statistics were normalized by those of the SDGM and the overall mean was shown.

Original: Table 1 shows the recognition error rates and number of nonzero weights for each method.
Revised: Table 1 shows the recognition error rates and number of nonzero weights for each method.

Original: The results in Table 1 show that the SDGM achieved an equivalent or greater accuracy compared with the SVM and RVM on average.
Revised: The results in Table 1 show that the SDGM achieved an equivalent or greater accuracy compared with the SVM and RVM on average.

Original: The SDGM is developed based a Gaussian mixture model and is particularly effective for data where a Gaussian distribution can be assumed, such as the Ripley dataset.
Revised: The SDGM is developed based a Gaussian mixture model and is particularly effective for data where a Gaussian distribution can be assumed, such as the Ripley dataset.

Original: On the number of nonzero weights, understandably, the LR showed the smallest number since it is a linear model.
Revised: On the number of nonzero weights, understandably, the LR showed the smallest number since it is a linear model.

Original: Among the remaining nonlinear classiﬁers, the SDGM achieved relatively small number of nonzero weights thanks to its sparse Bayesian learning.
Revised: Among the remaining nonlinear classiﬁers, the SDGM achieved relatively small number of nonzero weights thanks to its sparse Bayesian learning.

Original: The results above indicated that the SDGM demonstrated generalization capability and a sparsity simultaneously.
Revised: The results above indicated that the SDGM demonstrated generalization capability and a sparsity simultaneously.

Original: 3. I MAGE CLASSIFICATION
Revised: 3. I MAGE CLASSIFICATION

Original: In this experiment, the SDGM is embedded into a deep neural network.
Revised: In this experiment, the SDGM is embedded into a deep neural network.

Original: Since the SDGM is differentiable with respect to the weights, SDGM can be embedded into a deep NN as a module and is
Revised: Since the SDGM is differentiable with respect to the weights, SDGM can be embedded into a deep NN as a module and is

Original: [Table]
Revised: [Table]

Original: trained in an end-to-end manner.
Revised: trained in an end-to-end manner.

Original: In particular, the SDGM plays the same role as the softmax function since the SDGM calculates the posterior probability of each class given an input vector.
Revised: In particular, the SDGM plays the same role as the softmax function since the SDGM calculates the posterior probability of each class given an input vector.

Original: We can show that a fully connected layer with the softmax is equivalent to the discriminative model based on a single Gaussian distribution for each class by applying a simple transformation (see Appendix
Revised: We can show that a fully connected layer with the softmax is equivalent to the discriminative model based on a single Gaussian distribution for each class by applying a simple transformation (see Appendix

Original: A), whereas the SDGM is based on the Gaussian mixture model.
Revised: A), whereas the SDGM is based on the Gaussian mixture model.

Original: To verify the difference between them, we conducted image classiﬁcation experiments.
Revised: To verify the difference between them, we conducted image classiﬁcation experiments.

Original: Using a
Revised: Using a

Original: CNN with a softmax function as a baseline, we evaluated the capability of SDGM by replacing softmax with the SDGM.
Revised: CNN with a softmax function as a baseline, we evaluated the capability of SDGM by replacing softmax with the SDGM.

Original: 3.3. D ATASETS AND EXPERIMENTAL SETUPS
Revised: 3.3. D ATASETS AND EXPERIMENTAL SETUPS

Original: We used the following datasets and experimental settings in this experiment.
Revised: We used the following datasets and experimental settings in this experiment.

Original: MNIST :
Revised: MNIST :

Original: This dataset includes 10 classes of handwritten binary digit images of size 28 × 28 (LeCun et al., 1998).
Revised: This dataset includes 10 classes of handwritten binary digit images of size 28 × 28 (LeCun et al., 1998).

Original: We used 60,000 images as training data and 10,000 images as testing data.
Revised: We used 60,000 images as training data and 10,000 images as testing data.

Original: As a feature extractor, we used a simple CNN that consists of ﬁve convolutional layers with four max pooling layers between them and a fully connected layer.
Revised: As a feature extractor, we used a simple CNN that consists of ﬁve convolutional layers with four max pooling layers between them and a fully connected layer.

Original: To visualize the learned CNN features, we ﬁrst set the output dimension of the fully connected layer of the baseline CNN as two ( D = 2 ).
Revised: To visualize the learned CNN features, we ﬁrst set the output dimension of the fully connected layer of the baseline CNN as two ( D = 2 ).

Original: Furthermore, we tested by increasing the output dimension of the fully connected layer from two to ten ( D = 10 ).
Revised: Furthermore, we tested by increasing the output dimension of the fully connected layer from two to ten ( D = 10 ).

Original: Fashion-MNIST : Fashion-MNIST (Xiao et al., 2017) includes 10 classes of binary fashion images with a size of 28 × 28 .
Revised: Fashion-MNIST : Fashion-MNIST (Xiao et al., 2017) includes 10 classes of binary fashion images with a size of 28 × 28 .

Original: It includes 60,000 images for training data and 10,000 images for testing data.
Revised: It includes 60,000 images for training data and 10,000 images for testing data.

Original: We used the same CNN as in MNIST with 10 as the output dimension.
Revised: We used the same CNN as in MNIST with 10 as the output dimension.

Original: CIFAR
Revised: CIFAR

Original: -10 : CIFAR-
Revised: -10 : CIFAR-

Original: Krizhevsky
Revised: Krizhevsky

Original: Hinton,
Revised: Hinton,

Original: 2009) is the labeled subsets of an 80 million tiny image dataset.
Revised: 2009) is the labeled subsets of an 80 million tiny image dataset.

Original: This dataset consists of 60,000 32x32 color images in 10 classes, with 6,000 images per class.
Revised: This dataset consists of 60,000 32x32 color images in 10 classes, with 6,000 images per class.

Original: There are 50,000 training images and 10,000 test images.
Revised: There are 50,000 training images and 10,000 test images.

Original: For CIFAR-10, we trained
Revised: For CIFAR-10, we trained

Original: DenseNet (Huang et al., 2017) with a depth of 40 and a growth rate of 12.
Revised: DenseNet (Huang et al., 2017) with a depth of 40 and a growth rate of 12.

Original: For each dataset, the network was trained with a batch size of 64 for 100 epochs with a learning rate of 0.01 We used a weight decay of 1 . 

Revised: For each dataset, the network was trained with a batch size of 64 for 100 epochs with a learning rate of 0.01 We used a weight decay of 1 .

Original: × 10 − 5 and the Nesterov optimization algorithm (Sutskever et al., 2013) with a momentum of 0.9.
Revised: × 10 − 5 and the Nesterov optimization algorithm (Sutskever et al., 2013) with a momentum of 0.9.

Original: The network weights were initialized using the Glorot uniform (Glorot & Bengio, 2010).
Revised: The network weights were initialized using the Glorot uniform (Glorot & Bengio, 2010).

Original: 3.3. R ESULTS
Revised: 3.3. R ESULTS

Original: Figure 4 shows the two-dimensional feature embeddings on the MNIST dataset.
Revised: Figure 4 shows the two-dimensional feature embeddings on the MNIST dataset.

Original: Different feature embeddings were acquired for each method.
Revised: Different feature embeddings were acquired for each method.

Original: When softmax was used, the features spread in a fan shape and some part of the distribution overlapped around the origin.
Revised: When softmax was used, the features spread in a fan shape and some part of the distribution overlapped around the origin.

Original: However, when the SDGM was used, the distribution for each class exhibited an ellipse shape and margins appeared between the class distributions.
Revised: However, when the SDGM was used, the distribution for each class exhibited an ellipse shape and margins appeared between the class distributions.

Original: This is because the SDGM is based on a Gaussian mixture model and functions to push the samples into a Gaussian shape.
Revised: This is because the SDGM is based on a Gaussian mixture model and functions to push the samples into a Gaussian shape.

Original: Table 2 shows the recognition error rates on each dataset.
Revised: Table 2 shows the recognition error rates on each dataset.

Original: SDGM achieved better performance than softmax.
Revised: SDGM achieved better performance than softmax.

Original: As shown in Figure 4, SDGM can create margins between classes by pushing the features into a Gaussian shape.
Revised: As shown in Figure 4, SDGM can create margins between classes by pushing the features into a Gaussian shape.

Original: This phenomenon positively affected the classiﬁcation capability.
Revised: This phenomenon positively affected the classiﬁcation capability.

Original:  R ELATED W ORK AND P OSITION OF T HIS S TUDY
Revised:  R ELATED W ORK AND P OSITION OF T HIS S TUDY

Original: Figure 5 illustrates the relationship of our study with other studies.
Revised: Figure 5 illustrates the relationship of our study with other studies.

Original: This study is primarily consists of three factors: discriminative model, Gaussian mixture model, and Sparse Bayesian learning.
Revised: This study is primarily consists of three factors: discriminative model, Gaussian mixture model, and Sparse Bayesian learning.

Original: This study is the ﬁrst that combines these three factors and expands the body of knowledge in these ﬁelds.
Revised: This study is the ﬁrst that combines these three factors and expands the body of knowledge in these ﬁelds.

Original: From the perspective of the sparse Bayesian classiﬁer, the RVM (Tipping, 2001) is the most important related study.
Revised: From the perspective of the sparse Bayesian classiﬁer, the RVM (Tipping, 2001) is the most important related study.

Original: An RVM is combines logistic regression and sparse Bayesian learning.
Revised: An RVM is combines logistic regression and sparse Bayesian learning.

Original: Since
Revised: Since

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: the logistic regression is equivalent to the discriminative model of a unimodal Gaussian model, the
Revised: the logistic regression is equivalent to the discriminative model of a unimodal Gaussian model, the

Original: SDGM can be considered as an extended RVM using a GMM.
Revised: SDGM can be considered as an extended RVM using a GMM.

Original: furthermore, from the perspective of the probabilistic model, the SDGM is considered as the an extended discriminative GMM (Klautau et al., 2003) using sparse Bayesian learning, and an extended sparse GMM (Gaiffas & Michel, 2014) using the discriminative model.
Revised: furthermore, from the perspective of the probabilistic model, the SDGM is considered as the an extended discriminative GMM (Klautau et al., 2003) using sparse Bayesian learning, and an extended sparse GMM (Gaiffas & Michel, 2014) using the discriminative model.

Original: Sparse methods have often been used in machine learning.
Revised: Sparse methods have often been used in machine learning.

Original: Three primary merits of using sparse learning are as follows: improvements in generalization capability, memory reduction
Revised: Three primary merits of using sparse learning are as follows: improvements in generalization capability, memory reduction

Original: [Figure]
Revised: [Figure]

Original: and interpretability.
Revised: and interpretability.

Original: Several attempts have been conducted to adapt sparse learning to deep NNs.
Revised: Several attempts have been conducted to adapt sparse learning to deep NNs.

Original: Graham (2014) proposed a spatially-sparse convolutional neural network.
Revised: Graham (2014) proposed a spatially-sparse convolutional neural network.

Original: Liu et al.
Revised: Liu et al.

Original: (2015) proposed a sparse convolution neural network.
Revised: (2015) proposed a sparse convolution neural network.

Original: Additionally, sparse Bayesian learning has been applied in many ﬁelds.
Revised: Additionally, sparse Bayesian learning has been applied in many ﬁelds.

Original: For example, an application to EEG classiﬁcation has been reported (Zhang et al., 2017).
Revised: For example, an application to EEG classiﬁcation has been reported (Zhang et al., 2017).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we proposed a sparse classiﬁer based on a GMM, which is named SDGM.
Revised: In this paper, we proposed a sparse classiﬁer based on a GMM, which is named SDGM.

Original: In the
Revised: In the

Original: SDGM, a GMM-based discriminative model was trained by sparse Bayesian learning.
Revised: SDGM, a GMM-based discriminative model was trained by sparse Bayesian learning.

Original: This learning algorithm improved the generalization capability by obtaining a sparse solution and automatically determined the number of components by removing redundant components.
Revised: This learning algorithm improved the generalization capability by obtaining a sparse solution and automatically determined the number of components by removing redundant components.

Original: The SDGM could be embedded into NNs such as convolutional NNs and could be trained in an end-to-end manner.
Revised: The SDGM could be embedded into NNs such as convolutional NNs and could be trained in an end-to-end manner.

Original: In the experiments, we demonstrated that the SDGM could reduce the amount of weights via sparse Bayesian learning, thereby improving its generalization capability.
Revised: In the experiments, we demonstrated that the SDGM could reduce the amount of weights via sparse Bayesian learning, thereby improving its generalization capability.

Original: The comparison using benchmark datasets suggested that SDGM outperforms the conventional sparse classiﬁers.
Revised: The comparison using benchmark datasets suggested that SDGM outperforms the conventional sparse classiﬁers.

Original: We also demonstrated that SDGM outperformed the fully connected layer with the softmax function when it was used as the last layer of a deep NN.
Revised: We also demonstrated that SDGM outperformed the fully connected layer with the softmax function when it was used as the last layer of a deep NN.

Original: One of the limitations of this study is that sparse Bayesian learning was applied only when the
Revised: One of the limitations of this study is that sparse Bayesian learning was applied only when the

Original: SDGM was trained stand-alone.
Revised: SDGM was trained stand-alone.

Original: In future work, we will develop a sparse learning algorithm for a whole deep NN structure including the feature extraction part.
Revised: In future work, we will develop a sparse learning algorithm for a whole deep NN structure including the feature extraction part.

Original: This will improve the ability of the
Revised: This will improve the ability of the

Original: CNN for larger data classiﬁcation.
Revised: CNN for larger data classiﬁcation.

Original: A A PPENDIX
Revised: A A PPENDIX

Original: We explain that a fully connected layer with the softmax function, or logistic regression, can be regarded as a discriminative model based on a Gaussian distribution by utilizing transformation of the equations.
Revised: We explain that a fully connected layer with the softmax function, or logistic regression, can be regarded as a discriminative model based on a Gaussian distribution by utilizing transformation of the equations.

Original: Let us consider a case in which the class-conditional probability P ( x | c ) is a Gaussian distribution.
Revised: Let us consider a case in which the class-conditional probability P ( x | c ) is a Gaussian distribution.

Original: In this case, we can omit m from the equations (4)–(7).
Revised: In this case, we can omit m from the equations (4)–(7).

Original: If all classes share the same covariance matrix and the mixture weight π cm , the terms π cm in (2), x 21 , x 1 x 2 , · · · , x 1 x D , x 22 , x 2 x 3 , · · · , x 2 x D , · · · , x 2 D in (3), and − 12 s c 11 , · · · , − 12 s cDD in (7) can be canceled; hence the calculation of the posterior probability P ( c | x ) is also simpliﬁed as
Revised: If all classes share the same covariance matrix and the mixture weight π cm , the terms π cm in (2), x 21 , x 1 x 2 , · · · , x 1 x D , x 22 , x 2 x 3 , · · · , x 2 x D , · · · , x 2 D in (3), and − 12 s c 11 , · · · , − 12 s cDD in (7) can be canceled; hence the calculation of the posterior probability P ( c | x ) is also simpliﬁed as

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: This is equivalent to a fully connected layer with the softmax function.
Revised: This is equivalent to a fully connected layer with the softmax function.