Original: A Reparametrization-Invariant Sharpness Measure Based on Information Geometry
Revised: A Reparametrization-Invariant Sharpness Measure Based on Information Geometry

Original: Abstract
Revised: Abstract

Original: It has been observed that the generalization performance of neural networks correlates with the sharpness of their loss landscape.
Revised: It has been observed that the generalization performance of neural networks correlates with the sharpness of their loss landscape.

Original: Dinh et al.
Revised: Dinh et al.

Original: (2017) [8] have observed that existing formulations of sharpness measures fail to be invariant with respect to scaling and reparametrization.
Revised: (2017) [8] have observed that existing formulations of sharpness measures fail to be invariant with respect to scaling and reparametrization.

Original: While some scale-invariant measures have recently been proposed, reparametrization-invariant measures are still lacking.
Revised: While some scale-invariant measures have recently been proposed, reparametrization-invariant measures are still lacking.

Original: Moreover, they often do not provide any theoretical insights into generalization performance nor lead to practical use to improve the performance.
Revised: Moreover, they often do not provide any theoretical insights into generalization performance nor lead to practical use to improve the performance.

Original: Based on an information geometric analysis of the neural network parameter space, in this paper we propose a reparametrization-invariant sharpness measure that captures the change in loss with respect to changes in the probability distribution modeled by neural networks, rather than with respect to changes in the parameter values.
Revised: Based on an information geometric analysis of the neural network parameter space, in this paper we propose a reparametrization-invariant sharpness measure that captures the change in loss with respect to changes in the probability distribution modeled by neural networks, rather than with respect to changes in the parameter values.

Original: We reveal some theoretical connections of our measure to generalization performance.
Revised: We reveal some theoretical connections of our measure to generalization performance.

Original: In particular, experiments confirm that using our measure as a regularizer in neural network training significantly improves performance.
Revised: In particular, experiments confirm that using our measure as a regularizer in neural network training significantly improves performance.

Original:  Introduction
Revised:  Introduction

Original: From recent discoveries in deep learning it has been conjectured that the generalization performance of neural networks correlates with the sharpness of their loss landscape.
Revised: From recent discoveries in deep learning it has been conjectured that the generalization performance of neural networks correlates with the sharpness of their loss landscape.

Original: In particular, lower generalization performance of large-batch training of stochastic gradient descent (SGD) has been attributed to its convergence to sharper minima [22, 49] and several optimization methods have shown better generalization performance by actively finding solutions with lower sharpness [7, 19, 4, 43, 12, 24].
Revised: In particular, lower generalization performance of large-batch training of stochastic gradient descent (SGD) has been attributed to its convergence to sharper minima [22, 49] and several optimization methods have shown better generalization performance by actively finding solutions with lower sharpness [7, 19, 4, 43, 12, 24].

Original: Minimum description length (MDL)-based arguments [16, 17] and the generalization upper bound of the PAC-Bayes theory for neural networks [10] also suggest that the flatter the loss landscape, the better the generalization performance.
Revised: Minimum description length (MDL)-based arguments [16, 17] and the generalization upper bound of the PAC-Bayes theory for neural networks [10] also suggest that the flatter the loss landscape, the better the generalization performance.

Original: Furthermore, in [21, 32], measures based on the sharpness concept show better performance in predicting the generalization performance of models among various generalization measures considered in the past including the margin [5] and norm [34, 33].
Revised: Furthermore, in [21, 32], measures based on the sharpness concept show better performance in predicting the generalization performance of models among various generalization measures considered in the past including the margin [5] and norm [34, 33].

Original: Various measures for sharpness and flatness have been proposed in the literature.
Revised: Various measures for sharpness and flatness have been proposed in the literature.

Original: Hochreiter &
Revised: Hochreiter &

Original: Schmidhuber (1997)
Revised: Schmidhuber (1997)

Original: [17] propose the concept of ‘flat minima,’ in which the flatness of a minimum is interpreted as the volume of the connected region in the parameter space over which the loss has approximately similar values.
Revised: [17] propose the concept of ‘flat minima,’ in which the flatness of a minimum is interpreted as the volume of the connected region in the parameter space over which the loss has approximately similar values.

Original: Similarly, the sharpness has often been measured by the maximum loss value (inside a Euclidean ball) near the minimum [22] or by the spectral norm of the Hessian at the minimum [49].
Revised: Similarly, the sharpness has often been measured by the maximum loss value (inside a Euclidean ball) near the minimum [22] or by the spectral norm of the Hessian at the minimum [49].

Original: The above definitions of sharpness have some notable critical flaws.
Revised: The above definitions of sharpness have some notable critical flaws.

Original: Dinh et al.
Revised: Dinh et al.

Original: (2017) [8] point out that certain parameter scalings (e.g., for neural networks with ReLU activation functions [31]) and reparametrizations may have no effect on the model output or overall generalization performance, and yet lead to wildly different values of sharpness.
Revised: (2017) [8] point out that certain parameter scalings (e.g., for neural networks with ReLU activation functions [31]) and reparametrizations may have no effect on the model output or overall generalization performance, and yet lead to wildly different values of sharpness.

Original: 1 Other sharpness measures have been proposed to address the lack of scale-invariance [26, 46, 40, 39], but these measures fail to be invariant with respect to reparametrizations; Section 5 of [8] offers several nonlinear reparametrization examples.
Revised: 1 Other sharpness measures have been proposed to address the lack of scale-invariance [26, 46, 40, 39], but these measures fail to be invariant with respect to reparametrizations; Section 5 of [8] offers several nonlinear reparametrization examples.

Original: The connections between sharpness and generalization performance also remain elusive.
Revised: The connections between sharpness and generalization performance also remain elusive.

Original: In this paper we address the lack of scale- and reparametrization-invariance of existing sharpness measures, as well as the problem that they often do not explain the generalization performance of deep learning networks nor lead to practical use to improve the performance.
Revised: In this paper we address the lack of scale- and reparametrization-invariance of existing sharpness measures, as well as the problem that they often do not explain the generalization performance of deep learning networks nor lead to practical use to improve the performance.

Original: Our approach rests on the observation and insight that sharpness should be measured taking into proper account the geometry of the underlying parameter space.
Revised: Our approach rests on the observation and insight that sharpness should be measured taking into proper account the geometry of the underlying parameter space.

Original: For this purpose we draw upon tools from information geometry, paying attention to the fact that in most classification problems, probabilistic classification models are parametrized in terms of neural networks.
Revised: For this purpose we draw upon tools from information geometry, paying attention to the fact that in most classification problems, probabilistic classification models are parametrized in terms of neural networks.

Original: In information geometry, the Fisher information matrix (FIM) serves as a natural Riemannian metric for the parameter space [3] and allows for measuring the change in probability density with respect to changes in the parameter values.
Revised: In information geometry, the Fisher information matrix (FIM) serves as a natural Riemannian metric for the parameter space [3] and allows for measuring the change in probability density with respect to changes in the parameter values.

Original: The eigenspectra of the FIM of a neural network are observed to have a small number of positive outliers (the number is usually equated with the number of classes) and a bulk consisting of small eigenvalues [41, 37, 36, 48, 14].
Revised: The eigenspectra of the FIM of a neural network are observed to have a small number of positive outliers (the number is usually equated with the number of classes) and a bulk consisting of small eigenvalues [41, 37, 36, 48, 14].

Original: This observation indicates that the number of principal varying components of the probability densities modeled by neural networks is significantly lower compared to the number of parameters.
Revised: This observation indicates that the number of principal varying components of the probability densities modeled by neural networks is significantly lower compared to the number of parameters.

Original: Therefore, identically weighting every possible parameter change direction in the parameter space – that is, applying Euclidean geometry – can be problematic, since it places too much weight on parameter change directions that are meaningless with respect to changes in the probabilistic model, or equivalently, not enough weight is placed along meaningful directions.
Revised: Therefore, identically weighting every possible parameter change direction in the parameter space – that is, applying Euclidean geometry – can be problematic, since it places too much weight on parameter change directions that are meaningless with respect to changes in the probabilistic model, or equivalently, not enough weight is placed along meaningful directions.

Original: We argue that one should consider these implications and use the change in probabilistic models as a ‘ruler’ when defining a sharpness measure of neural network loss landscapes that does not depend on how the model is parametrized.
Revised: We argue that one should consider these implications and use the change in probabilistic models as a ‘ruler’ when defining a sharpness measure of neural network loss landscapes that does not depend on how the model is parametrized.

Original: Building on the above geometric analysis of the neural network parameter space, in this paper we propose a new sharpness measure based on information geometry.
Revised: Building on the above geometric analysis of the neural network parameter space, in this paper we propose a new sharpness measure based on information geometry.

Original: This measure is by definition reparametrization-invariant.
Revised: This measure is by definition reparametrization-invariant.

Original: Additionally, we show that this sharpness measure simultaneously satisfies the scale-invariance for neural networks with ReLU activation functions, since it is invariant for parameter transforms that do not change the model output.
Revised: Additionally, we show that this sharpness measure simultaneously satisfies the scale-invariance for neural networks with ReLU activation functions, since it is invariant for parameter transforms that do not change the model output.

Original: Hence our measure is free from all the reparametrization- and scale-variance issues of previous sharpness measures raised by [8].
Revised: Hence our measure is free from all the reparametrization- and scale-variance issues of previous sharpness measures raised by [8].

Original: As a second contribution of this paper, we also show that our geometric sharpness measure sheds an important insight on generalization performance.
Revised: As a second contribution of this paper, we also show that our geometric sharpness measure sheds an important insight on generalization performance.

Original: As detailed in Section 3, when the FIM is replaced with the Hessian in our measure, the measure is in a form similar to Takeuchi’s information criterion (TIC), which quantifies the asymptotic bias of the log-likelihood value (usually the negative loss) evaluated at the maximum likelihood estimates hence corresponding to the expectation of the generalization gap [44].
Revised: As detailed in Section 3, when the FIM is replaced with the Hessian in our measure, the measure is in a form similar to Takeuchi’s information criterion (TIC), which quantifies the asymptotic bias of the log-likelihood value (usually the negative loss) evaluated at the maximum likelihood estimates hence corresponding to the expectation of the generalization gap [44].

Original: TIC for neural networks has empirically shown a strong correlation with the generalization gap [45].
Revised: TIC for neural networks has empirically shown a strong correlation with the generalization gap [45].

Original: One can apply the same argument to our measure since the Hessian can be approximated by the FIM under mild conditions for deep neural networks [28].
Revised: One can apply the same argument to our measure since the Hessian can be approximated by the FIM under mild conditions for deep neural networks [28].

Original: In addition, we show that our measure can be linked to generalization in another geometrical way.
Revised: In addition, we show that our measure can be linked to generalization in another geometrical way.

Original: The measure is associated with a margin defined in an information geometric sense, enabling the interpretation that a smaller sharpness measure indicates larger margins.
Revised: The measure is associated with a margin defined in an information geometric sense, enabling the interpretation that a smaller sharpness measure indicates larger margins.

Original: We also demonstrate experimentally that using our measure as a regularizer to train neural networks can significantly improve generalization performance.
Revised: We also demonstrate experimentally that using our measure as a regularizer to train neural networks can significantly improve generalization performance.

Original: Our contributions can be summarized as follows:
Revised: Our contributions can be summarized as follows:

Original: • We provide an information geometric analysis of the neural network parameter space by investigating the eigensubspace of the Fisher information matrix (FIM) of neural networks.
Revised: • We provide an information geometric analysis of the neural network parameter space by investigating the eigensubspace of the Fisher information matrix (FIM) of neural networks.

Original: • We propose a reparametrization- and scale-invariant sharpness measure based on information geometry that is free from the problems of existing sharpness measures posed by [8], and discuss the relation between the measure and generalization performance.
Revised: • We propose a reparametrization- and scale-invariant sharpness measure based on information geometry that is free from the problems of existing sharpness measures posed by [8], and discuss the relation between the measure and generalization performance.

Original: to as is called a represent a a space obtained by applying a
Revised: to as is called a represent a a space obtained by applying a

Original: • We use the proposed measure as a regularizer when training neural networks, and demonstrate improved generalization performance.
Revised: • We use the proposed measure as a regularizer when training neural networks, and demonstrate improved generalization performance.

Original: We provide an information geometric analysis of the neural network parameter space in Section 2.
Revised: We provide an information geometric analysis of the neural network parameter space in Section 2.

Original: Building on this analysis, Section 3 presents our reparametrization-invariant sharpness measure based on information geometry and connects the proposed measure to generalization.
Revised: Building on this analysis, Section 3 presents our reparametrization-invariant sharpness measure based on information geometry and connects the proposed measure to generalization.

Original: In Section 4, we perform numerical experiments using our measure as a regularizer to improve generalization performance in neural network training.
Revised: In Section 4, we perform numerical experiments using our measure as a regularizer to improve generalization performance in neural network training.

Original:  An information geometric analysis of the neural network parameter space
Revised:  An information geometric analysis of the neural network parameter space

Original: In this paper, we focus on the probabilistic classification models parametrized by neural networks.
Revised: In this paper, we focus on the probabilistic classification models parametrized by neural networks.

Original: Let x ∈ R D denote data, and y ∈ { 1 , . .
Revised: Let x ∈ R D denote data, and y ∈ { 1 , . .

Original: , C } denote class labels with C as the number of classes.
Revised: , C } denote class labels with C as the number of classes.

Original: Let p ( x, y ) and p ( y | x ; θ ) respectively denote the data generating distribution and the parametric model of the probability density of classes given data.
Revised: Let p ( x, y ) and p ( y | x ; θ ) respectively denote the data generating distribution and the parametric model of the probability density of classes given data.

Original: Here the parameter θ ∈ R m is the set of all the weight and bias parameters of neural networks.
Revised: Here the parameter θ ∈ R m is the set of all the weight and bias parameters of neural networks.

Original: We consider a neural network as a function f ( · ; θ ) : R D → R C , x (cid:55)→ f ( x ; θ ) =
Revised: We consider a neural network as a function f ( · ; θ ) : R D → R C , x (cid:55)→ f ( x ; θ ) =

Original: ( f 1 ( x ; θ ) , . . .
Revised: ( f 1 ( x ; θ ) , . . .

Original: , f C ( x ; θ )) ⊤ , where f j ( · ; θ ) :
Revised: , f C ( x ; θ )) ⊤ , where f j ( · ; θ ) :

Original: R D → R is a function that returns the j -th logit for j = 1 , . . .
Revised: R D → R is a function that returns the j -th logit for j = 1 , . . .

Original: , C .
Revised: , C .

Original: Our parametric model is then represented as p ( y | x ; θ ) =
Revised: Our parametric model is then represented as p ( y | x ; θ ) =

Original: exp( f y ( x )) (cid:80) j exp( f j ( x )) , and we denote the negative log-likelihood by l ( y, f ( x ; θ ))
Revised: exp( f y ( x )) (cid:80) j exp( f j ( x )) , and we denote the negative log-likelihood by l ( y, f ( x ; θ ))

Original: = − log p ( y | x ; θ ) .
Revised: = − log p ( y | x ; θ ) .

Original: Suppose data { ( x 1 , y 1 ) , . . .
Revised: Suppose data { ( x 1 , y 1 ) , . . .

Original: , ( x N , y N ) } are drawn i.i.d.
Revised: , ( x N , y N ) } are drawn i.i.d.

Original: from the data generating distribution p ( x, y ) .
Revised: from the data generating distribution p ( x, y ) .

Original: The cross-entropy loss, our training objective function throughout the paper, can be written as follows:
Revised: The cross-entropy loss, our training objective function throughout the paper, can be written as follows:

Original: [Equation]
Revised: [Equation]

Original: where l i ( θ ) = l ( y
Revised: where l i ( θ ) = l ( y

Original: [Equation]
Revised: [Equation]

Original: 2.1 The Fisher information matrix (FIM)
Revised: 2.1 The Fisher information matrix (FIM)

Original: The Fisher information matrix (FIM) for the family of probability density functions p ( x, y ; θ ) = p ( x ) p ( y | x ; θ ) parametrized by the neural network parameters θ is defined as
Revised: The Fisher information matrix (FIM) for the family of probability density functions p ( x, y ; θ ) = p ( x ) p ( y | x ; θ ) parametrized by the neural network parameters θ is defined as

Original: F (
Revised: F (

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where the dependence of l ( y, f ( x ; θ )) to y, f ( x ; θ ) is omitted for simplicity and J ( x ) = ∂f∂θ ( x ) ∈
Revised: where the dependence of l ( y, f ( x ; θ )) to y, f ( x ; θ ) is omitted for simplicity and J ( x ) = ∂f∂θ ( x ) ∈

Original: R C × m is the Jacobian of the logits with respect to the parameters.
Revised: R C × m is the Jacobian of the logits with respect to the parameters.

Original: In deriving (2), the expectation with respect to p ( x ) is approximated by the finite sum over data points x 1 , . . .
Revised: In deriving (2), the expectation with respect to p ( x ) is approximated by the finite sum over data points x 1 , . . .

Original: , x N drawn i.i.d.
Revised: , x N drawn i.i.d.

Original: from p ( x ) , J i =
Revised: from p ( x ) , J i =

Original: J ( x i ) , and we use the fact that E p ( y | x i ; θ ) (cid:20)(cid:16) ∂l∂f (cid:17) ⊤ (cid:16) ∂l∂f (cid:17)(cid:21) =
Revised: J ( x i ) , and we use the fact that E p ( y | x i ; θ ) (cid:20)(cid:16) ∂l∂f (cid:17) ⊤ (cid:16) ∂l∂f (cid:17)(cid:21) =

Original: diag ( p i ) − p i
Revised: diag ( p i ) − p i

Original: p ⊤ i , where p i
Revised: p ⊤ i , where p i

Original: =
Revised: =

Original: ( p (1 | x i ; θ ) , . . .
Revised: ( p (1 | x i ; θ ) , . . .

Original: , p ( C | x i ; θ ))
Revised: , p ( C | x i ; θ ))

Original: ⊤ ∈ R C is the model’s prediction on the probability that x i belongs to each class and diag
Revised: ⊤ ∈ R C is the model’s prediction on the probability that x i belongs to each class and diag

Original: ( p i ) ∈
Revised: ( p i ) ∈

Original: R C
Revised: R C

Original: × C is a diagonal matrix whose ( j, j ) entry is ( p i ) j .
Revised: × C is a diagonal matrix whose ( j, j ) entry is ( p i ) j .

Original: In this paper, among many of the important properties and applications of the FIM (e.g., being the
Revised: In this paper, among many of the important properties and applications of the FIM (e.g., being the

Original: Cramer-Rao lower bound of estimator variances in estimation theory and statistics), we focus on the fact that the FIM can serve as a metric to measure the distance between the parametric probability density models in information geometry.
Revised: Cramer-Rao lower bound of estimator variances in estimation theory and statistics), we focus on the fact that the FIM can serve as a metric to measure the distance between the parametric probability density models in information geometry.

Original: Differential geometrically speaking, the FIM acts as a natural
Revised: Differential geometrically speaking, the FIM acts as a natural

Original: Riemannian metric on the statistical manifold, a space where the family of probability densities modeled with smoothly varying parameters θ ∈ R m is gathered.
Revised: Riemannian metric on the statistical manifold, a space where the family of probability densities modeled with smoothly varying parameters θ ∈ R m is gathered.

Original: The FIM makes it possible to
Revised: The FIM makes it possible to

Original: [Figure]
Revised: [Figure]

Original: measure geometric quantities such as length, angle, and volume on the manifold.
Revised: measure geometric quantities such as length, angle, and volume on the manifold.

Original: (We refer the reader to [6, 11] for the backgrounds on the Riemannian manifolds and differential geometry, and [3, 35, 28] for those on the information geometry and the Fisher information matrix.)
Revised: (We refer the reader to [6, 11] for the backgrounds on the Riemannian manifolds and differential geometry, and [3, 35, 28] for those on the information geometry and the Fisher information matrix.)

Original: Note that when the FIM is not full-rank, e.g., in the case of overparameterized neural networks with m ≫ N , it can be used as a pseudo metric.
Revised: Note that when the FIM is not full-rank, e.g., in the case of overparameterized neural networks with m ≫ N , it can be used as a pseudo metric.

Original: The FIM is closely related to the KL-divergence, which is frequently used as an information theoretic measure of discrepancy between probability density functions.
Revised: The FIM is closely related to the KL-divergence, which is frequently used as an information theoretic measure of discrepancy between probability density functions.

Original: The KL-divergence between two probability density functions with infinitesimal parameter difference dθ ∈
Revised: The KL-divergence between two probability density functions with infinitesimal parameter difference dθ ∈

Original: R
Revised: R

Original: m can be approximated as
Revised: m can be approximated as

Original: [Equation]
Revised: [Equation]

Original: 2.2 An analysis of the eigensubspace of the FIM
Revised: 2.2 An analysis of the eigensubspace of the FIM

Original: This section discusses the characteristics of the eigensubspace of FIM of neural networks.
Revised: This section discusses the characteristics of the eigensubspace of FIM of neural networks.

Original: In the analysis of the eigenspectra of deep neural network Hessians of [37], the FIM in (2) has been decomposed as follows:
Revised: In the analysis of the eigenspectra of deep neural network Hessians of [37], the FIM in (2) has been decomposed as follows:

Original: [Equation]
Revised: [Equation]

Original: where M i = diag ( √ p i ) (cid:0) I − 1 p ⊤ i (cid:1) ∈ R C × C , √ p i =
Revised: where M i = diag ( √ p i ) (cid:0) I − 1 p ⊤ i (cid:1) ∈ R C × C , √ p i =

Original: ( (cid:112) ( p i ) 1 , . .
Revised: ( (cid:112) ( p i ) 1 , . .

Original: , (cid:112) ( p i ) C )
Revised: , (cid:112) ( p i ) C )

Original: ⊤ ∈ R C , and 1 = (1 , . .
Revised: ⊤ ∈ R C , and 1 = (1 , . .

Original: . , 1) ⊤ ∈ R C is a vector whose elements are all one.
Revised: . , 1) ⊤ ∈ R C is a vector whose elements are all one.

Original: According to the decomposition, the FIM can be thought of as a non-centralized second moment of the (modified) logit gradients, i.e., each row of M
Revised: According to the decomposition, the FIM can be thought of as a non-centralized second moment of the (modified) logit gradients, i.e., each row of M

Original: i J i ∈ R C × m whose j -th row vector is represented as
Revised: i J i ∈ R C × m whose j -th row vector is represented as

Original: [Equation]
Revised: [Equation]

Original: where J i,j ∈ R 1 × m is the j -th row of J i .
Revised: where J i,j ∈ R 1 × m is the j -th row of J i .

Original: It has been observed that these logit gradients form a kind of hierarchical structure after sufficient training.
Revised: It has been observed that these logit gradients form a kind of hierarchical structure after sufficient training.

Original: In the observation from [36], the gradients of the c ′ ̸ = c -th logit calculated from the data in class c are gathered to form a cluster.
Revised: In the observation from [36], the gradients of the c ′ ̸ = c -th logit calculated from the data in class c are gathered to form a cluster.

Original: A matrix obtained from averaging the outer products of the averaged logit gradients belonging to each cluster is then attributed to the outliers of the eigenspectra, and the outliers of the eigenvalues appear as much as the number of classes.
Revised: A matrix obtained from averaging the outer products of the averaged logit gradients belonging to each cluster is then attributed to the outliers of the eigenspectra, and the outliers of the eigenvalues appear as much as the number of classes.

Original: That is, the principal eigenvalues of the FIM (and possibly the eigenvectors) can be closely connected with distinguishing each class c from the rest of the c ′ ̸ = c classes (as can be implied from averaging (5) for j ̸ = c and by assuming ( p i ) c ≈ 1 ).
Revised: That is, the principal eigenvalues of the FIM (and possibly the eigenvectors) can be closely connected with distinguishing each class c from the rest of the c ′ ̸ = c classes (as can be implied from averaging (5) for j ̸ = c and by assuming ( p i ) c ≈ 1 ).

Original: This becomes more evident in
Revised: This becomes more evident in

Original: Appendix A, which assumes the prediction is balanced as ( p i ) c ′ =
Revised: Appendix A, which assumes the prediction is balanced as ( p i ) c ′ =

Original: ϵ for all c ′ ̸
Revised: ϵ for all c ′ ̸

Original: = c with ϵ ≪ 1 .
Revised: = c with ϵ ≪ 1 .

Original: In order to more intuitively understand the characteristics of the eigensubspace of FIM implied from the above analysis, we provide a classification example for two-dimensional synthetic data generated from mixtures of three Gaussians in Figure 1.
Revised: In order to more intuitively understand the characteristics of the eigensubspace of FIM implied from the above analysis, we provide a classification example for two-dimensional synthetic data generated from mixtures of three Gaussians in Figure 1.

Original: We trained a three-layer fully connected neural network with 70-dimensional hidden units (the number of parameters = 5,393).
Revised: We trained a three-layer fully connected neural network with 70-dimensional hidden units (the number of parameters = 5,393).

Original: Figure 1 (a) shows the top twenty eigenvalues of the FIM of the trained neural network.
Revised: Figure 1 (a) shows the top twenty eigenvalues of the FIM of the trained neural network.

Original: Note that there are few outliers among the eigenvalues in the entire 5,393-dimensional parameter space, with the remainder being almost 0.
Revised: Note that there are few outliers among the eigenvalues in the entire 5,393-dimensional parameter space, with the remainder being almost 0.

Original: The eigenvalues indicate the rate of change of the probability density (modeled by the neural network) measured by the KL-divergence for a unit change of parameter values along the corresponding eigenvectors.
Revised: The eigenvalues indicate the rate of change of the probability density (modeled by the neural network) measured by the KL-divergence for a unit change of parameter values along the corresponding eigenvectors.

Original: The existence of a few outliers means that the density change mainly occurs in only a few specific directions and that the amount of change in probability density can vary significantly according to the direction of change in parameter values.
Revised: The existence of a few outliers means that the density change mainly occurs in only a few specific directions and that the amount of change in probability density can vary significantly according to the direction of change in parameter values.

Original: This fact can also be observed in decision boundaries obtained from neural networks with the parameter values perturbed along some eigenvectors shown in Figure 1 (b-d).
Revised: This fact can also be observed in decision boundaries obtained from neural networks with the parameter values perturbed along some eigenvectors shown in Figure 1 (b-d).

Original: When perturbing the values along the principal eigenvectors, a significant change appears in the decision boundary, increasing or decreasing the margin of certain classes.
Revised: When perturbing the values along the principal eigenvectors, a significant change appears in the decision boundary, increasing or decreasing the margin of certain classes.

Original: On the other hand, for eigenvectors with small eigenvalues, there is almost no change in the decision boundary according to the same level of change in parameter values (in the Euclidean sense).
Revised: On the other hand, for eigenvectors with small eigenvalues, there is almost no change in the decision boundary according to the same level of change in parameter values (in the Euclidean sense).

Original: These examples imply that it would be more meaningful to reflect the different influences on the probabilistic model according to the change direction when we measure the quantities concerning changes in the parameter values, e.g., the sharpness of the loss landscape.
Revised: These examples imply that it would be more meaningful to reflect the different influences on the probabilistic model according to the change direction when we measure the quantities concerning changes in the parameter values, e.g., the sharpness of the loss landscape.

Original: By reflecting on these implications and using the unit change in probability distribution modeled by neural networks as our ‘ruler’ to measure the sharpness, that is, applying the information geometry, the following section presents a sharpness measure that does not rely on how the model is parametrized.
Revised: By reflecting on these implications and using the unit change in probability distribution modeled by neural networks as our ‘ruler’ to measure the sharpness, that is, applying the information geometry, the following section presents a sharpness measure that does not rely on how the model is parametrized.

Original:  An information geometric sharpness measure
Revised:  An information geometric sharpness measure

Original: Reflecting the above analysis, we define an information geometric sharpness (IGS) measure as follows:
Revised: Reflecting the above analysis, we define an information geometric sharpness (IGS) measure as follows:

Original: [Equation]
Revised: [Equation]

Original: where F ( θ ) † ∈ R m × m is the pseudo-inverse of F ( θ ) and l i = l ( y i , f ( x i ; θ )) is the cross-entropy loss evaluated at the i -th data ( x i , y i ) .
Revised: where F ( θ ) † ∈ R m × m is the pseudo-inverse of F ( θ ) and l i = l ( y i , f ( x i ; θ )) is the cross-entropy loss evaluated at the i -th data ( x i , y i ) .

Original: Our measure evaluates the squared norm of the loss gradients calculated at each data point using the (pseudo-inverse of) FIM and averages them for all data points.
Revised: Our measure evaluates the squared norm of the loss gradients calculated at each data point using the (pseudo-inverse of) FIM and averages them for all data points.

Original: 4 Since (cid:0) ∂l i ∂θ (cid:1)
Revised: 4 Since (cid:0) ∂l i ∂θ (cid:1)

Original: ⊤ ∈
Revised: ⊤ ∈

Original: R m belongs to the span of the eigenvectors of F ( θ ) with non-zero eigenvalues, our sharpness measure is well-defined in the sense that there are not any components of (cid:0) ∂l
Revised: R m belongs to the span of the eigenvectors of F ( θ ) with non-zero eigenvalues, our sharpness measure is well-defined in the sense that there are not any components of (cid:0) ∂l

Original: i ∂θ (cid:1) neglected in measuring its information geometric norm (we elaborate on this in Appendix B).
Revised: i ∂θ (cid:1) neglected in measuring its information geometric norm (we elaborate on this in Appendix B).

Original: Our sharpness measure in (6) is an intrinsic quantity, i.e., coordinate-invariant (in differential geometric terms).
Revised: Our sharpness measure in (6) is an intrinsic quantity, i.e., coordinate-invariant (in differential geometric terms).

Original: To see why, observe that under a local coordinate transformation (or a reparametrization) ϕ : R m → R m , θ (cid:55)→ θ ′ =
Revised: To see why, observe that under a local coordinate transformation (or a reparametrization) ϕ : R m → R m , θ (cid:55)→ θ ′ =

Original: ϕ ( θ ) , i.e., the model is now represented with respect to a different parameter θ ′ as ˜ f ( · ; θ ′ )
Revised: ϕ ( θ ) , i.e., the model is now represented with respect to a different parameter θ ′ as ˜ f ( · ; θ ′ )

Original: =
Revised: =

Original: f ( · ; ϕ − 1 ( θ ′ ))
Revised: f ( · ; ϕ − 1 ( θ ′ ))

Original: =
Revised: =

Original: f ( · ; θ ) , F and (cid:0) ∂l i ∂θ (cid:1) transform according to the following rules: (i) F (cid:55)→ F ′ =
Revised: f ( · ; θ ) , F and (cid:0) ∂l i ∂θ (cid:1) transform according to the following rules: (i) F (cid:55)→ F ′ =

Original: Φ −⊤ F Φ − 1 , where Φ =
Revised: Φ −⊤ F Φ − 1 , where Φ =

Original: ∂ϕ∂θ ∈
Revised: ∂ϕ∂θ ∈

Original: R m × m ; (ii) (cid:0) ∂l i ∂θ (cid:1) (cid:55)→ (cid:0) ∂l i ∂θ ′ (cid:1) =
Revised: R m × m ; (ii) (cid:0) ∂l i ∂θ (cid:1) (cid:55)→ (cid:0) ∂l i ∂θ ′ (cid:1) =

Original: (cid:0) ∂l i ∂θ (cid:1)
Revised: (cid:0) ∂l i ∂θ (cid:1)

Original: Φ − 1 , where it can be verified that the (cid:0) ∂l i ∂θ (cid:1) F ( θ ) † (cid:0) ∂l i ∂θ (cid:1) ⊤ remains the same.
Revised: Φ − 1 , where it can be verified that the (cid:0) ∂l i ∂θ (cid:1) F ( θ ) † (cid:0) ∂l i ∂θ (cid:1) ⊤ remains the same.

Original: Since the above measure gives the same value regardless of which parametrization (e.g., θ or θ ′ )
Revised: Since the above measure gives the same value regardless of which parametrization (e.g., θ or θ ′ )

Original: the statistical model is parametrized, it is free from the reparametrization-variant issues raised in [8].
Revised: the statistical model is parametrized, it is free from the reparametrization-variant issues raised in [8].

Original: We can also define other reparametrization-invariant measures by measuring the information geometric norm of the gradients of the entire loss or the mini-batch losses.
Revised: We can also define other reparametrization-invariant measures by measuring the information geometric norm of the gradients of the entire loss or the mini-batch losses.

Original: For mini-batches of size b , the corresponding measure can be defined as follows:
Revised: For mini-batches of size b , the corresponding measure can be defined as follows:

Original: [Equation]
Revised: [Equation]

Original: where E p ( B ) [ · ] denotes the expectation with respect to some distribution p ( B ) of mini-batches
Revised: where E p ( B ) [ · ] denotes the expectation with respect to some distribution p ( B ) of mini-batches

Original: B ⊂ { x 1 , . . .
Revised: B ⊂ { x 1 , . . .

Original: , x N } of size | B | = b .
Revised: , x N } of size | B | = b .

Original: We discuss the relation between (6) and (7) in Appendix C. Note that this definition can be linked to the concept of m-sharpness, which measures the sharpness of the loss landscape by averaging the sharpness of mini-batch losses and shows a better correlation to the generalization as the batch size reduces [12].
Revised: We discuss the relation between (6) and (7) in Appendix C. Note that this definition can be linked to the concept of m-sharpness, which measures the sharpness of the loss landscape by averaging the sharpness of mini-batch losses and shows a better correlation to the generalization as the batch size reduces [12].

Original: 3.1 Transformation invariance
Revised: 3.1 Transformation invariance

Original: Neural networks composed of activation functions such as ReLU and leaky ReLU possess scaleinvariant properties.
Revised: Neural networks composed of activation functions such as ReLU and leaky ReLU possess scaleinvariant properties.

Original: In the case of layer-wise scaling, for consecutive layers of linear transforms and
Revised: In the case of layer-wise scaling, for consecutive layers of linear transforms and

Original: ReLU, e.g., f ( x ; { W 1 , . . .
Revised: ReLU, e.g., f ( x ; { W 1 , . . .

Original: , W L } )
Revised: , W L } )

Original: =
Revised: =

Original: W L · ReLU ( W L − 1 · · · ReLU ( W 1 x )) , where W l ∈ R d l × d l − 1 for l = 1 , . . .
Revised: W L · ReLU ( W L − 1 · · · ReLU ( W 1 x )) , where W l ∈ R d l × d l − 1 for l = 1 , . . .

Original: , L with d 0 = D and d L = C , if each weight W l is multiplied by a constant c l > 0 and the constants satisfy (cid:81)
Revised: , L with d 0 = D and d L = C , if each weight W l is multiplied by a constant c l > 0 and the constants satisfy (cid:81)

Original: Ll =1 c l = 1 , there is no change in the outputs of the neural network under the same input values.
Revised: Ll =1 c l = 1 , there is no change in the outputs of the neural network under the same input values.

Original: A similar concept includes the node-wise scaling, multiplying weights entering a node (or a hidden variable in neural networks) by a positive constant and dividing weights out of the node by the same constant.
Revised: A similar concept includes the node-wise scaling, multiplying weights entering a node (or a hidden variable in neural networks) by a positive constant and dividing weights out of the node by the same constant.

Original: These kinds of weight scaling can be viewed as a transformation (or a mapping between identical parameter spaces) that ensures that the neural network model maintains the same probability density function, i.e., remains equivalent.
Revised: These kinds of weight scaling can be viewed as a transformation (or a mapping between identical parameter spaces) that ensures that the neural network model maintains the same probability density function, i.e., remains equivalent.

Original: If a transformation is differentiable and locally invertible without changing the probability density function that the neural network parameter models for all parameters on a given neighborhood U ⊆ R m of θ , our measures defined in (6) and (7) become invariant with respect to such a transformation.
Revised: If a transformation is differentiable and locally invertible without changing the probability density function that the neural network parameter models for all parameters on a given neighborhood U ⊆ R m of θ , our measures defined in (6) and (7) become invariant with respect to such a transformation.

Original: This fact can be expressed as the following proposition:
Revised: This fact can be expressed as the following proposition:

Original: Proposition 3.1.
Revised: Proposition 3.1.

Original: Suppose there exist open subsets U, V ⊆
Revised: Suppose there exist open subsets U, V ⊆

Original: R m and an invertible and locally differentiable transformation g : U → V that satisfies f ( x ; θ ) =
Revised: R m and an invertible and locally differentiable transformation g : U → V that satisfies f ( x ; θ ) =

Original: f ( x ; g ( θ )) for all x ∈ R D and θ ∈ U with g ( U )
Revised: f ( x ; g ( θ )) for all x ∈ R D and θ ∈ U with g ( U )

Original: ⊆
Revised: ⊆

Original: V .
Revised: V .

Original: The information geometric sharpness measures in (6) and (7) are invariant to such a transformation, i.e., IGS ( θ ) =
Revised: The information geometric sharpness measures in (6) and (7) are invariant to such a transformation, i.e., IGS ( θ ) =

Original: IGS ( g ( θ )) (or IGS b ( θ ) =
Revised: IGS ( g ( θ )) (or IGS b ( θ ) =

Original: IGS b ( g ( θ )) ) for all θ ∈ U .
Revised: IGS b ( g ( θ )) ) for all θ ∈ U .

Original: The proof of Proposition 3.1 is provided in Appendix D.1.
Revised: The proof of Proposition 3.1 is provided in Appendix D.1.

Original: Since the layer-wise and node-wise scalings considered for neural networks with ReLU satisfy the assumptions of Proposition 3.1, our measures in (6) and (7) are invariant to such scalings.
Revised: Since the layer-wise and node-wise scalings considered for neural networks with ReLU satisfy the assumptions of Proposition 3.1, our measures in (6) and (7) are invariant to such scalings.

Original: Proposition 3.1 is more general than the scale-invariance, and Appendix D.2 provides examples of transformations other than parameter scalings that satisfy the assumptions in the proposition.
Revised: Proposition 3.1 is more general than the scale-invariance, and Appendix D.2 provides examples of transformations other than parameter scalings that satisfy the assumptions in the proposition.

Original: 3.2 A comparison to some previous sharpness measures
Revised: 3.2 A comparison to some previous sharpness measures

Original: [Figure]
Revised: [Figure]

Original: After [8] pointed out the critical scale- and reparametrization-variant issues in the sharpness measures considered in [17, 22, 49], various scale-invariant sharpness measures have been proposed that suit to neural networks with activation functions satisfying the non-negative homogeneity conditions (i.e., σ ( a · x ) = a · σ ( x ) for a > 0 ) such as ReLU [40, 39, 46, 26].
Revised: After [8] pointed out the critical scale- and reparametrization-variant issues in the sharpness measures considered in [17, 22, 49], various scale-invariant sharpness measures have been proposed that suit to neural networks with activation functions satisfying the non-negative homogeneity conditions (i.e., σ ( a · x ) = a · σ ( x ) for a > 0 ) such as ReLU [40, 39, 46, 26].

Original: The measure presented in [40] is based on the geometry of the quotient manifold obtained from an equivalence class of neural networks, establishing the scale-invariance depending on the considered equivalence types.
Revised: The measure presented in [40] is based on the geometry of the quotient manifold obtained from an equivalence class of neural networks, establishing the scale-invariance depending on the considered equivalence types.

Original: The work in [39] considers a layer-wise flatness measure for neural networks and additionally explores the conditions for the measure to explain the generalization well.
Revised: The work in [39] considers a layer-wise flatness measure for neural networks and additionally explores the conditions for the measure to explain the generalization well.

Original: Also, a scale-invariant sharpness measure based on the PAC-Bayes theory is developed in [46].
Revised: Also, a scale-invariant sharpness measure based on the PAC-Bayes theory is developed in [46].

Original: The FisherRao norm is proposed as a capacity measure for neural networks in [26] and defined as θ ⊤ F ( θ ) θ , which is scale-invariant.
Revised: The FisherRao norm is proposed as a capacity measure for neural networks in [26] and defined as θ ⊤ F ( θ ) θ , which is scale-invariant.

Original: However, the above measures do not satisfy the invariance to general nonlinear reparametrizations, hence solving the problem raised by [8] only partially.
Revised: However, the above measures do not satisfy the invariance to general nonlinear reparametrizations, hence solving the problem raised by [8] only partially.

Original: That is, for a nonlinear reparametrization θ (cid:55)→ η = g ( θ ) , the measures evaluated on the reparametrized model ˜ f
Revised: That is, for a nonlinear reparametrization θ (cid:55)→ η = g ( θ ) , the measures evaluated on the reparametrized model ˜ f

Original: ( · ; η ) = f ( · ; g − 1 ( η ))
Revised: ( · ; η ) = f ( · ; g − 1 ( η ))

Original: =
Revised: =

Original: f ( · ; θ ) (with respect to η ) can arbitrarily vary from those evaluated on the original model f ( · ; θ ) (with respect to θ ).
Revised: f ( · ; θ ) (with respect to η ) can arbitrarily vary from those evaluated on the original model f ( · ; θ ) (with respect to θ ).

Original: In Figure 2, we empirically demonstrate the existence of this problem in conventional sharpness measures such as the trace of the Hessian, the measures of [26], [40], and [39] (respectively denoted as Tr ( H ) , Fisher-Rao, Rangamani et al., and Petzka et al.
Revised: In Figure 2, we empirically demonstrate the existence of this problem in conventional sharpness measures such as the trace of the Hessian, the measures of [26], [40], and [39] (respectively denoted as Tr ( H ) , Fisher-Rao, Rangamani et al., and Petzka et al.

Original: on the x-axis of the figure; see Appendix G. for details).
Revised: on the x-axis of the figure; see Appendix G. for details).

Original: We can observe that the magnitudes of these measures can vary significantly.
Revised: We can observe that the magnitudes of these measures can vary significantly.

Original: Compared with the previous measures, only our measure (denoted as IGS on the x-axis of Figure 2) is invariant to the considered reparametrizations.
Revised: Compared with the previous measures, only our measure (denoted as IGS on the x-axis of Figure 2) is invariant to the considered reparametrizations.

Original: These experiments empirically demonstrate that our measure satisfies the reparametrization-invariance, hence providing a solution that properly resolves the issues posed by [8].
Revised: These experiments empirically demonstrate that our measure satisfies the reparametrization-invariance, hence providing a solution that properly resolves the issues posed by [8].

Original: 3.3 Connections to the generalization 3.3.1 Connections to Takeuchi’s information criterion (TIC)
Revised: 3.3 Connections to the generalization 3.3.1 Connections to Takeuchi’s information criterion (TIC)

Original: Our measures can be linked to generalization in different aspects.
Revised: Our measures can be linked to generalization in different aspects.

Original: When we train parametric models via the maximum likelihood estimation (this is equivalent to minimizing the negative log-likelihood of (1)), the obtained maximum likelihood estimate is asymptotically unbiased, guaranteed theoretically by the asymptotic normality.
Revised: When we train parametric models via the maximum likelihood estimation (this is equivalent to minimizing the negative log-likelihood of (1)), the obtained maximum likelihood estimate is asymptotically unbiased, guaranteed theoretically by the asymptotic normality.

Original: However, the log-likelihood value (or the negative loss) evaluated at the obtained estimate does not enjoy such a property.
Revised: However, the log-likelihood value (or the negative loss) evaluated at the obtained estimate does not enjoy such a property.

Original: The well-known Akaike’s information criterion (AIC) in the model selection literature calculates this bias under the assumption that the parametric model contains the actual data distribution [1].
Revised: The well-known Akaike’s information criterion (AIC) in the model selection literature calculates this bias under the assumption that the parametric model contains the actual data distribution [1].

Original: Takeuchi’s information criterion (TIC) is a generalized form of the AIC by considering the case of misspecified models, i.e., the parametric models do not contain the actual data distribution [44, 9, 45].
Revised: Takeuchi’s information criterion (TIC) is a generalized form of the AIC by considering the case of misspecified models, i.e., the parametric models do not contain the actual data distribution [44, 9, 45].

Original: The bias b (ˆ θ ) of the log-likelihood value evaluated at the maximum likelihood estimate is defined as follows:
Revised: The bias b (ˆ θ ) of the log-likelihood value evaluated at the maximum likelihood estimate is defined as follows:

Original: [Equation]
Revised: [Equation]

Original: where D = { ( x 1 , y 1 ) , . .
Revised: where D = { ( x 1 , y 1 ) , . .

Original: , ( x N , y N ) } is the set of N training data with ( x i , y i ) ∼ p ( x, y ) and ˆ θ ( D ) denotes the maximum likelihood estimate obtained from using
Revised: , ( x N , y N ) } is the set of N training data with ( x i , y i ) ∼ p ( x, y ) and ˆ θ ( D ) denotes the maximum likelihood estimate obtained from using

Original: D [9].
Revised: D [9].

Original: Since our loss is the negative log-likelihood, the first and second terms inside E D [ · ] correspond to the negative training and test losses, respectively.
Revised: Since our loss is the negative log-likelihood, the first and second terms inside E D [ · ] correspond to the negative training and test losses, respectively.

Original: Consequently, the bias in (8) becomes the expectation of the parametric model’s generalization gap (i.e., test loss – training loss).
Revised: Consequently, the bias in (8) becomes the expectation of the parametric model’s generalization gap (i.e., test loss – training loss).

Original: TIC calculates this bias under asymptotic assumptions, i.e., in the limit N → ∞ , as follows [9]:
Revised: TIC calculates this bias under asymptotic assumptions, i.e., in the limit N → ∞ , as follows [9]:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where θ 0 ∈ R m is a local maximum of the expected log-likelihood, H ( θ 0 )
Revised: where θ 0 ∈ R m is a local maximum of the expected log-likelihood, H ( θ 0 )

Original: ∈
Revised: ∈

Original: R m × m is the Hessian of the (expected) loss, and C ( θ 0 ) =
Revised: R m × m is the Hessian of the (expected) loss, and C ( θ 0 ) =

Original: E p ( x,y ) (cid:20)(cid:16) ∂l ( y,f ( x
Revised: E p ( x,y ) (cid:20)(cid:16) ∂l ( y,f ( x

Original: [Equation]
Revised: [Equation]

Original: θ ))
Revised: θ ))

Original: ∂θ
Revised: ∂θ

Original: [Equation]
Revised: [Equation]

Original: (cid:17)(cid:21)(cid:12)(cid:12)(cid:12)(cid:12)
Revised: (cid:17)(cid:21)(cid:12)(cid:12)(cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: m × m is the non-centered covariance of the loss gradients.
Revised: m × m is the non-centered covariance of the loss gradients.

Original: Here both H ( θ 0 ) and C ( θ 0 ) are evaluated using the data generating distribution p ( x, y ) .
Revised: Here both H ( θ 0 ) and C ( θ 0 ) are evaluated using the data generating distribution p ( x, y ) .

Original: The formulation in (9) is very similar to our information geometric sharpness measure when evaluated at a local minimum ˆ θ ∈
Revised: The formulation in (9) is very similar to our information geometric sharpness measure when evaluated at a local minimum ˆ θ ∈

Original: R m of the loss in (1).
Revised: R m of the loss in (1).

Original: Compared to TIC, our measure in (6) contains the FIM instead of the Hessian, and the expectation for C ( θ ) is taken with respect to the empirical distribution rather than the true distribution.
Revised: Compared to TIC, our measure in (6) contains the FIM instead of the Hessian, and the expectation for C ( θ ) is taken with respect to the empirical distribution rather than the true distribution.

Original: After a model is sufficiently trained, the Hessian can be approximated to the FIM [36, 28], indicating that our sharpness measure is closely related to TIC.
Revised: After a model is sufficiently trained, the Hessian can be approximated to the FIM [36, 28], indicating that our sharpness measure is closely related to TIC.

Original: It has been observed that TIC predicts the generalization gap of neural network models well when the expectation with respect to p ( x, y ) is approximated by a finite sum of the integrands over the test data [45].
Revised: It has been observed that TIC predicts the generalization gap of neural network models well when the expectation with respect to p ( x, y ) is approximated by a finite sum of the integrands over the test data [45].

Original: These findings are also confirmed by experiments using our measures.
Revised: These findings are also confirmed by experiments using our measures.

Original: In Figure 3, compared to the other sharpness measures (labeled the same as in Figure 2), we can observe that our measure correlates better with the generalization gap.
Revised: In Figure 3, compared to the other sharpness measures (labeled the same as in Figure 2), we can observe that our measure correlates better with the generalization gap.

Original: The experimental details are provided in Appendix G.2.
Revised: The experimental details are provided in Appendix G.2.

Original: [Figure]
Revised: [Figure]

Original: N 
Revised: N I S T [Figure] Tr

Original: [Figure]
Revised: [Figure]

Original: H
Revised: H

Original: [Figure]
Revised: [Figure]

Revised: Fisher

Revised: [Figure]

Original: et al.
Revised: et al.

Original: Petzka et al.
Revised: Petzka et al.

Original: [Figure]
Revised: [Figure]

Original: C
Revised: C

Original: I F
Revised: I F

Original: A R

Revised: A R

Original: R=0.
Revised: R=0.

Original: [Figure]
Revised: [Figure]

Original:  t i o n g a p
Revised: a t i o n g a p

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: 3.3.2 Connections to the margin
Revised: 3.3.2 Connections to the margin

Original: Our measure can also be linked to generalization in terms of margin.
Revised: Our measure can also be linked to generalization in terms of margin.

Original: Unlike the usual definition of the margin as the distance from the decision boundary to its closest data, we devise a similar concept developed in the parametric model space rather than the input space.
Revised: Unlike the usual definition of the margin as the distance from the decision boundary to its closest data, we devise a similar concept developed in the parametric model space rather than the input space.

Original: Given a model parameter value, if we perturb the value so that the model’s decision boundary passes through the location of a nearby data as shown in Figure 4, the difference between the initial and the perturbed parameter values would correspond to a sort of margin defined in the model parameter space.
Revised: Given a model parameter value, if we perturb the value so that the model’s decision boundary passes through the location of a nearby data as shown in Figure 4, the difference between the initial and the perturbed parameter values would correspond to a sort of margin defined in the model parameter space.

Original: Since the parameter space dimension is high, we can perturb the model parameter value in various directions.
Revised: Since the parameter space dimension is high, we can perturb the model parameter value in various directions.

Original: Considering the perturbations of parameter values along the eigenvectors of the
Revised: Considering the perturbations of parameter values along the eigenvectors of the

Original: FIM results in an interesting connection between this margin and our sharpness measure.
Revised: FIM results in an interesting connection between this margin and our sharpness measure.

Original: We now formalize this relationship for the binary classification case, while that for the multi-class case is derived in Appendix E.
Revised: We now formalize this relationship for the binary classification case, while that for the multi-class case is derived in Appendix E.

Original: We define the binary cross-entropy loss as l i ( θ ) =
Revised: We define the binary cross-entropy loss as l i ( θ ) =

Original: l BCE ( x i , z i ; θ ) =
Revised: l BCE ( x i , z i ; θ ) =

Original: −
Revised: −

Original: z
Revised: z

Original: i log p ( x i ; θ ) − (1 − z i ) log(1 − p ( x i ; θ )) , where z i ∈ { 0 , 1 } denotes  binary class labels and p ( x i ; θ ) denotes the model’s prediction on the probability thatan input x i∈ R D belongs to class one.
Revised: i log p ( x i ; θ ) − (1 − z i ) log(1 − p ( x i ; θ )) , where z i ∈ { 0 , 1 } denotes the binary class label for an input x i ∈ R D and p ( x i ; θ ) denotes the model’s prediction on the probability that x i belongs to class one.

Original: The condition for x i∈ R D to lie on the decision boundary is p ( x i ) = 12 , which is equal to l i ( θ ) = log 2 regardless of z i ∈ { 0 , 1 } .
Revised: The condition for x i to lie on the decision boundary is p ( x i ) = 12 , which is equal to l i ( θ ) = log 2 regardless of z i ∈ { 0 , 1 } .

Original: Suppose we perturb the neural network parameter value from θ to θ + ϵ i,k ∆ θ k , where ∆ θ k ∈ R m is the k -th eigenvector of the FIM associated with the k -th eigenvalue λ k and ϵ i,k is a scalar.
Revised: Suppose we perturb the neural network parameter value from θ to θ + ϵ i,k ∆ θ k , where ∆ θ k ∈ R m is the k -th eigenvector of the FIM associated with the k -th eigenvalue λ k and ϵ i,k is a scalar.

Original: For the decision boundary of the perturbed model to lie on x i , the scalar ϵ i,k should satisfy log 2 =
Revised: For the decision boundary of the perturbed model to lie on x i , the scalar ϵ i,k should satisfy log 2 =

Original: l i ( θ + ϵ i,k ∆ θ k )
Revised: l i ( θ + ϵ i,k ∆ θ k )

Original: ≈ l
Revised: ≈ l

Original: i ( θ ) + ϵ i,k (cid:0) ∂l i ∂θ (cid:1) ∆ θ k , where we apply the first-order Taylor expansion by assuming small ϵ i,k .
Revised: i ( θ ) + ϵ i,k (cid:0) ∂l i ∂θ (cid:1) ∆ θ k , where we apply the first-order Taylor expansion by assuming small ϵ i,k .

Original: The scalar ϵ i,k can then be approximated as ϵ i,k ≈ log2 − l i ( θ ) (cid:16) ∂li∂θ (cid:17) ∆ θ k .
Revised: The scalar ϵ i,k can then be approximated as ϵ i,k ≈ log2 − l i ( θ ) (cid:16) ∂li∂θ (cid:17) ∆ θ k .

Original: A sort of margin can be derived in an information geometric sense by computing the squared norm of the perturbation vector
Revised: A sort of margin can be derived in an information geometric sense by computing the squared norm of the perturbation vector

Original: ϵ i,k ∆ θ k using the inner product based on the FIM as follows:
Revised: ϵ i,k ∆ θ k using the inner product based on the FIM as follows:

Original: [Equation]
Revised: [Equation]

Original: The summation over k = 1 , . . .
Revised: The summation over k = 1 , . . .

Original: , m ′ (with m ′ as the largest k with non-zero λ k ) of the reciprocal of these squared norms (hence weighing more on small ϵ 2 i,k values which would reduce the error from approximations) can then be related to our sharpness measure as follows:
Revised: , m ′ (with m ′ as the largest k with non-zero λ k ) of the reciprocal of these squared norms (hence weighing more on small ϵ 2 i,k values which would reduce the error from approximations) can then be related to our sharpness measure as follows:

Original: [Equation]
Revised: [Equation]

Original: where we have used F ( θ ) † = (cid:80) m
Revised: where we have used F ( θ ) † = (cid:80) m

Original: ′ k
Revised: ′ k

Original: =1 1 λ k ∆ θ k ∆ θ ⊤ k in deriving the last identity.
Revised: =1 1 λ k ∆ θ k ∆ θ ⊤ k in deriving the last identity.

Original: Note that our sharpness measure at each data ( x i , y i ) appears in the numerator of (11).
Revised: Note that our sharpness measure at each data ( x i , y i ) appears in the numerator of (11).

Original: This indicates that the smaller our sharpness measure, the larger the squared norms ( ϵ 2 i,k λ k ) in the LHS, which means that the margin becomes larger in an information geometric sense.
Revised: This indicates that the smaller our sharpness measure, the larger the squared norms ( ϵ 2 i,k λ k ) in the LHS, which means that the margin becomes larger in an information geometric sense.

Original: [Figure]
Revised: [Figure]

Original:  Using the sharpness measure as a regularizer to train neural networks
Revised:  Using the sharpness measure as a regularizer to train neural networks

Original: We now apply our sharpness measure as a regularizer to train neural networks to see if regularizing the measure can improve the generalization performance.
Revised: We now apply our sharpness measure as a regularizer to train neural networks to see if regularizing the measure can improve the generalization performance.

Original: The corresponding loss function is written as
Revised: The corresponding loss function is written as

Original: [Equation]
Revised: [Equation]

Original: where ρ is a coefficient for the regularization term.
Revised: where ρ is a coefficient for the regularization term.

Original: We consider a toy example and then consider image classification tasks involving MNIST and CIFAR-10/100 data sets.
Revised: We consider a toy example and then consider image classification tasks involving MNIST and CIFAR-10/100 data sets.

Original: We also provide some tractable and possibly efficient ways to regularize our sharpness measure.
Revised: We also provide some tractable and possibly efficient ways to regularize our sharpness measure.

Original: All the experiments have been performed using the PyTorch library [38].
Revised: All the experiments have been performed using the PyTorch library [38].

Original: 4.1 A toy example
Revised: 4.1 A toy example

Original: We first apply our regularizer to a toy example.
Revised: We first apply our regularizer to a toy example.

Original: In [18], they have trained neural networks on poison data with the wrong labels to hamper the generalization performance and make the loss landscape extremely sharp while fitting all the training data.
Revised: In [18], they have trained neural networks on poison data with the wrong labels to hamper the generalization performance and make the loss landscape extremely sharp while fitting all the training data.

Original: We follow this experimental setting and check whether the obtained solution can avoid the undesirable generalization performance and loss landscapes when applying our regularizer (calculated on the training data).
Revised: We follow this experimental setting and check whether the obtained solution can avoid the undesirable generalization performance and loss landscapes when applying our regularizer (calculated on the training data).

Original: The experimental details, as well as how the IGS is approximated for this example, are provided in Appendix G.3.
Revised: The experimental details, as well as how the IGS is approximated for this example, are provided in Appendix G.3.

Original: The experimental results are shown in Figure 5.
Revised: The experimental results are shown in Figure 5.

Original: The results from SGD without regularization show very irregular decision boundaries with tiny margins, whereas relatively soft boundaries with larger margins appear when our regularizer is used, especially around the training data.
Revised: The results from SGD without regularization show very irregular decision boundaries with tiny margins, whereas relatively soft boundaries with larger margins appear when our regularizer is used, especially around the training data.

Original: Applying our regularizer also shows a lower generalization gap, i.e., the difference between the training and test losses, than the SGD without regularization.
Revised: Applying our regularizer also shows a lower generalization gap, i.e., the difference between the training and test losses, than the SGD without regularization.

Original: 4.2 MNIST and CIFAR-10/ 
Revised: 4.2 MNIST and CIFAR-10/

Original: To check the effect of the regularizer in more realistic settings, we apply our regularizer to the classification of MNIST and CIFAR-10/100 data sets.
Revised: To check the effect of the regularizer in more realistic settings, we apply our regularizer to the classification of MNIST and CIFAR-10/100 data sets.

Original: For this experiment, we regularize the mini-batch IGS defined in (7).
Revised: For this experiment, we regularize the mini-batch IGS defined in (7).

Original: For an efficient implementation of our regularizer, the following approximation on perturbed loss functions with a perturbation δ = ρF ( θ ) † (cid:0) ∂l∂θ (cid:1) ⊤ ∈ R m (with a small ρ > 0 ) is useful:
Revised: For an efficient implementation of our regularizer, the following approximation on perturbed loss functions with a perturbation δ = ρF ( θ ) † (cid:0) ∂l∂θ (cid:1) ⊤ ∈ R m (with a small ρ > 0 ) is useful:

Original: [Equation]
Revised: [Equation]

Original: Note that for l = 1 b (cid:80) x i ∈ B l i with a mini-batch B of size | B | = b , minimizing (13) becomes minimizing a stochastic version of (12) (with IGS b ( θ ) ), where the stochasticity comes from sampling a mini-batch B .
Revised: Note that for l = 1 b (cid:80) x i ∈ B l i with a mini-batch B of size | B | = b , minimizing (13) becomes minimizing a stochastic version of (12) (with IGS b ( θ ) ), where the stochasticity comes from sampling a mini-batch B .

Original: Therefore we minimize (13) in our experiments with ρ as a tunable hyperparameter.
Revised: Therefore we minimize (13) in our experiments with ρ as a tunable hyperparameter.

Original: To obtain δ , we resort to an EKFAC-based approximation for the natural gradients [13] as detailed in
Revised: To obtain δ , we resort to an EKFAC-based approximation for the natural gradients [13] as detailed in

Original: Appendix G.4.1.
Revised: Appendix G.4.1.

Original: A three-layer fully connected neural network (3FCN) is used for the experiments using MNIST data, and various convolutional neural networks such as VGG [42], ResNet [15], and WideResNet [50] are used for CIFAR-10/100 experiments, with data augmentation, cosine annealing [27], and label
Revised: A three-layer fully connected neural network (3FCN) is used for the experiments using MNIST data, and various convolutional neural networks such as VGG [42], ResNet [15], and WideResNet [50] are used for CIFAR-10/100 experiments, with data augmentation, cosine annealing [27], and label

Original: [Table]
Revised: [Table]

Original: smoothing [30] methods additionally applied.
Revised: smoothing [30] methods additionally applied.

Original: For comparison, we consider the gradient regularization (GR) method [4, 43], sharpness-aware minimization (SAM) method [12]
Revised: For comparison, we consider the gradient regularization (GR) method [4, 43], sharpness-aware minimization (SAM) method [12]

Original: [Table]
Revised: [Table]

Original: and the adaptive SAM (ASAM) method [24], an extension of SAM to involve the scale-invariance.
Revised: and the adaptive SAM (ASAM) method [24], an extension of SAM to involve the scale-invariance.

Original: The detailed experimental settings are explained in Appendix G.4.2.
Revised: The detailed experimental settings are explained in Appendix G.4.2.

Original: We provide the averages and standard errors of the test classification accuracies obtained from three runs of each method in Table 1.
Revised: We provide the averages and standard errors of the test classification accuracies obtained from three runs of each method in Table 1.

Original: As can be seen from the table, one can confirm that the generalization performance of SGD is significantly improved with our regularizer.
Revised: As can be seen from the table, one can confirm that the generalization performance of SGD is significantly improved with our regularizer.

Original: Furthermore, our method shows better performance than the GR method and a comparable performance improvement with the SAM and ASAM methods.
Revised: Furthermore, our method shows better performance than the GR method and a comparable performance improvement with the SAM and ASAM methods.

Original:  Conclusion
Revised:  Conclusion

Original: Various empirical results observed from running deep learning algorithms such as SGD have suggested that minima with a flatter loss landscape tend to show better generalization performance.
Revised: Various empirical results observed from running deep learning algorithms such as SGD have suggested that minima with a flatter loss landscape tend to show better generalization performance.

Original: However, the previous definitions of sharpness/flatness have faced a severe dependence on reparametrizations or parameter scalings that do not change the model output and the generalization performance [8].
Revised: However, the previous definitions of sharpness/flatness have faced a severe dependence on reparametrizations or parameter scalings that do not change the model output and the generalization performance [8].

Original: In this paper, based on an information geometric analysis of the neural network parameter space, we have proposed an information geometric sharpness measure which is reparametrization- and scaleinvariant, making it free from the issues posed by [8].
Revised: In this paper, based on an information geometric analysis of the neural network parameter space, we have proposed an information geometric sharpness measure which is reparametrization- and scaleinvariant, making it free from the issues posed by [8].

Original: We have also discussed the connection of the measure to generalization.
Revised: We have also discussed the connection of the measure to generalization.

Original: In addition, a regularizer that reduces the suggested sharpness measure is proposed, showing a significant improvement in generalization performance under practical settings.
Revised: In addition, a regularizer that reduces the suggested sharpness measure is proposed, showing a significant improvement in generalization performance under practical settings.

Original: Unlike many other sharpness measures, the Fisher information matrix (FIM) plays a crucial role in our measure.
Revised: Unlike many other sharpness measures, the Fisher information matrix (FIM) plays a crucial role in our measure.

Original: Since it can be computationally demanding to calculate the FIM and our sharpness measure for large-scale neural networks, more efficient evaluation methods should be developed to apply our measure to such networks.
Revised: Since it can be computationally demanding to calculate the FIM and our sharpness measure for large-scale neural networks, more efficient evaluation methods should be developed to apply our measure to such networks.

Original: Concerning the regularization of our measure during training, exploring better ways to reduce the time complexity and increase generalization performance is left for future work.
Revised: Concerning the regularization of our measure during training, exploring better ways to reduce the time complexity and increase generalization performance is left for future work.

Original: Furthermore, analyzing the generalization properties of models obtained from deep learning algorithms such as SGD or natural gradient descent [2, 28] using our measure would be another intriguing future work.
Revised: Furthermore, analyzing the generalization properties of models obtained from deep learning algorithms such as SGD or natural gradient descent [2, 28] using our measure would be another intriguing future work.

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: C. Jang and Y.-K. Noh were supported by IITP Artificial Intelligence Graduate School Program for Hanyang University funded by MSIT (Grant No. 2020-0-01373).
Revised: C. Jang and Y.-K. Noh were supported by IITP Artificial Intelligence Graduate School Program for Hanyang University funded by MSIT (Grant No. 2020-0-01373).

Original: S. Lee was supported by a
Revised: S. Lee was supported by a

Original: KIAS Individual Grant (AP083601) via the Center for AI and Natural Sciences at Korea Institute for Advanced Study.
Revised: KIAS Individual Grant (AP083601) via the Center for AI and Natural Sciences at Korea Institute for Advanced Study.

Original: F. C. Park was supported in part by SRRC NRF grant 2016R1A5A1938472,
Revised: F. C. Park was supported in part by SRRC NRF grant 2016R1A5A1938472,

Original: IITP-MSIT grant 2022-0-00480 (Training and Inference Methods for Goal-Oriented AI Agents), SNUAIIS, SNU-IAMD, and the SNU Institute for Engineering Research.
Revised: IITP-MSIT grant 2022-0-00480 (Training and Inference Methods for Goal-Oriented AI Agents), SNUAIIS, SNU-IAMD, and the SNU Institute for Engineering Research.

Original: Y.-K. Noh was partly supported by NRF/MSIT (Grant No. 2018R1A5A7059549, 2021M3E5D2A01019545) and IITP/MSIT (Grant
Revised: Y.-K. Noh was partly supported by NRF/MSIT (Grant No. 2018R1A5A7059549, 2021M3E5D2A01019545) and IITP/MSIT (Grant

Original: No. IITP-2021-0-02068).
Revised: No. IITP-2021-0-02068).

Original: Appendix A The Fisher information matrix and class margins
Revised: Appendix A The Fisher information matrix and class margins

Original: As mentioned in Section 2.2 of the manuscript, [37] suggested a hierarchy of the logit gradients based on the following decomposition of the Fisher information matrix:
Revised: As mentioned in Section 2.2 of the manuscript, [37] suggested a hierarchy of the logit gradients based on the following decomposition of the Fisher information matrix:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where M i = diag ( √ p i ) (cid:0) I − 1 p ⊤ i (cid:1) ∈ R C × C , √ p i =
Revised: where M i = diag ( √ p i ) (cid:0) I − 1 p ⊤ i (cid:1) ∈ R C × C , √ p i =

Original: ( (cid:112) ( p i ) 1 , . .
Revised: ( (cid:112) ( p i ) 1 , . .

Original: , (cid:112) ( p i ) C ) ∈ R C , and 1 = (1 , . . .
Revised: , (cid:112) ( p i ) C ) ∈ R C , and 1 = (1 , . . .

Original: , 1) ∈ R C is a vector whose elements are all one.
Revised: , 1) ∈ R C is a vector whose elements are all one.

Original: According to the proposed hierarchy, the dominant component of the FIM attributing the outliers of the eigenspectra turns out to be
Revised: According to the proposed hierarchy, the dominant component of the FIM attributing the outliers of the eigenspectra turns out to be

Original: [Equation]
Revised: [Equation]

Original: where N c is the number of data belonging to class c .
Revised: where N c is the number of data belonging to class c .

Original: Here the vector δ c ∈ R m is defined as
Revised: Here the vector δ c ∈ R m is defined as

Original: [Equation]
Revised: [Equation]

Original: where δ i,j ∈ R m is the j -th row vector of the matrix
Revised: where δ i,j ∈ R m is the j -th row vector of the matrix

Original: M i J i ∈ R C
Revised: M i J i ∈ R C

Original: × m (also defined in (5)) and
Revised: × m (also defined in (5)) and

Original: I c =
Revised: I c =

Original: { i ∈ { 1 , . . .
Revised: { i ∈ { 1 , . . .

Original: , N }| y i
Revised: , N }| y i

Original: = c } is the set of indices for data in class c .
Revised: = c } is the set of indices for data in class c .

Original: The dominant component of FIM in (15) implies that the FIM can be related to the difference between the c -th component and c ′ ̸
Revised: The dominant component of FIM in (15) implies that the FIM can be related to the difference between the c -th component and c ′ ̸

Original: = c -th components in the logit derivatives.
Revised: = c -th components in the logit derivatives.

Original: This relation can be seen more evidently using some assumptions on the predicted probability p i .
Revised: This relation can be seen more evidently using some assumptions on the predicted probability p i .

Original: Assuming the prediction for the classes c ′ ̸
Revised: Assuming the prediction for the classes c ′ ̸

Original: = c is balanced, i.e., ( p i ) c ′ , the c ′ -th component of p i for c ′ ̸
Revised: = c is balanced, i.e., ( p i ) c ′ , the c ′ -th component of p i for c ′ ̸

Original: = c , are all equal to a constant ϵ ≪ 1 , we can approximate δ i,c ′ and δ c as follows:
Revised: = c , are all equal to a constant ϵ ≪ 1 , we can approximate δ i,c ′ and δ c as follows:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where J i,j ∈ R 1 × m denotes the j -th row of the logit derivative J i ∈ R C × m .
Revised: where J i,j ∈ R 1 × m denotes the j -th row of the logit derivative J i ∈ R C × m .

Original: In this case, δ c can be interpreted as the gradient of the differences in the averaged logit values that can discriminate class c from the other classes c ′ ̸
Revised: In this case, δ c can be interpreted as the gradient of the differences in the averaged logit values that can discriminate class c from the other classes c ′ ̸

Original: = c .
Revised: = c .

Original: Since the dominant component of the FIM is insisted to be the G term in (15), we conjecture that the eigensubspace of the FIM can be related to the class margins as long as the hierarchy from [37] is valid.
Revised: Since the dominant component of the FIM is insisted to be the G term in (15), we conjecture that the eigensubspace of the FIM can be related to the class margins as long as the hierarchy from [37] is valid.

Original: Similar derivations can be found in [36].
Revised: Similar derivations can be found in [36].

Original: B On the well-definedness of IGS
Revised: B On the well-definedness of IGS

Original: From (2), the Fisher information matrix is written as
Revised: From (2), the Fisher information matrix is written as

Original: [Equation]
Revised: [Equation]

Original: where J i = (cid:16) ∂f∂θ ( x ) (cid:17)(cid:12)(cid:12)(cid:12) x = x i ∈ R C
Revised: where J i = (cid:16) ∂f∂θ ( x ) (cid:17)(cid:12)(cid:12)(cid:12) x = x i ∈ R C

Original: × m and p i
Revised: × m and p i

Original: = ( p (1 | x i ; θ ) , . . .
Revised: = ( p (1 | x i ; θ ) , . . .

Original: , p ( C | x i ; θ ))
Revised: , p ( C | x i ; θ ))

Original: ∈ R C is the model’s prediction on the probability that x i belongs to each class.
Revised: ∈ R C is the model’s prediction on the probability that x i belongs to each class.

Original: The FIM is positive semi-definite when the number of parameters is very large, e.g., for the overparametrized models with m ≫ N .
Revised: The FIM is positive semi-definite when the number of parameters is very large, e.g., for the overparametrized models with m ≫ N .

Original: Therefore, we need to verify if our definition of information geometric measure in (6) involving the pseudo-inverse of F ( θ ) is well-defined in the sense that there are not any components of (cid:0) ∂l
Revised: Therefore, we need to verify if our definition of information geometric measure in (6) involving the pseudo-inverse of F ( θ ) is well-defined in the sense that there are not any components of (cid:0) ∂l

Original: i ∂θ (cid:1) neglected in measuring the information geometric norm by (6).
Revised: i ∂θ (cid:1) neglected in measuring the information geometric norm by (6).

Original: For this purpose, we characterize the null space of F ( θ ) and compare it to that of the loss derivative (cid:0) ∂l
Revised: For this purpose, we characterize the null space of F ( θ ) and compare it to that of the loss derivative (cid:0) ∂l

Original: i ∂θ (cid:1) ∈
Revised: i ∂θ (cid:1) ∈

Original: R 1 × m .
Revised: R 1 × m .

Original: From (19) the null space of F ( θ ) , Null ( F ( θ ))
Revised: From (19) the null space of F ( θ ) , Null ( F ( θ ))

Original: =
Revised: =

Original: N 1 ∪ N 2 , where N 1 and N 2 are defined as
Revised: N 1 ∪ N 2 , where N 1 and N 2 are defined as

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Note that the condition for N 2 comes from the fact that the matrix (cid:0) diag ( p i ) − p i p ⊤ i (cid:1) ∈ R C × C in (19) possesses the vector (1 , . .
Revised: Note that the condition for N 2 comes from the fact that the matrix (cid:0) diag ( p i ) − p i p ⊤ i (cid:1) ∈ R C × C in (19) possesses the vector (1 , . .

Original: , 1) ⊤ ∈ R C as its eigenvector with the eigenvalue of zero.
Revised: , 1) ⊤ ∈ R C as its eigenvector with the eigenvalue of zero.

Original: For ˙ θ ∈ Null ( F ( θ )) , we can easily show that (cid:0) ∂l i ∂θ (cid:1) ˙
Revised: For ˙ θ ∈ Null ( F ( θ )) , we can easily show that (cid:0) ∂l i ∂θ (cid:1) ˙

Original: θ = 0 for all i = 1 , . .
Revised: θ = 0 for all i = 1 , . .

Original: N , since (cid:0) ∂l i ∂θ (cid:1)
Revised: N , since (cid:0) ∂l i ∂θ (cid:1)

Original: = − ( e ⊤ y
Revised: = − ( e ⊤ y

Original: i − (cid:80) j
Revised: i − (cid:80) j

Original: ( p i )
Revised: ( p i )

Original: j e ⊤ j ) J i ˙ θ = 0 for both the cases of ˙ θ
Revised: j e ⊤ j ) J i ˙ θ = 0 for both the cases of ˙ θ

Original: ∈ N 1 and ˙ θ ∈ N 2 .
Revised: ∈ N 1 and ˙ θ ∈ N 2 .

Original: Therefore, Null ( F ( θ )) ⊂ Null ( (cid:0) ∂l i ∂θ (cid:1) ) holds, and we can say that the vector (cid:0) ∂l i ∂θ (cid:1)
Revised: Therefore, Null ( F ( θ )) ⊂ Null ( (cid:0) ∂l i ∂θ (cid:1) ) holds, and we can say that the vector (cid:0) ∂l i ∂θ (cid:1)

Original: ⊤ ∈
Revised: ⊤ ∈

Original: R m belongs to the span of the eigenvectors of F ( θ ) with non-zero eigenvalues for all i = 1 , . . .
Revised: R m belongs to the span of the eigenvectors of F ( θ ) with non-zero eigenvalues for all i = 1 , . . .

Original: , N .
Revised: , N .

Original: Hence our information geometric measure in (6) does not neglect any components in (cid:0)
Revised: Hence our information geometric measure in (6) does not neglect any components in (cid:0)

Original: ∂l i ∂θ (cid:1) .
Revised: ∂l i ∂θ (cid:1) .

Original: C On the relationship between IGS and mini-batch IGS
Revised: C On the relationship between IGS and mini-batch IGS

Original: To obtain the relationship between IGS in (6) and mini-batch IGS in (7), we express them as follows:
Revised: To obtain the relationship between IGS in (6) and mini-batch IGS in (7), we express them as follows:

Original: [Equation]
Revised: [Equation]

Original: where S ( θ ) = 1 N (cid:80) Ni =1 (cid:0) ∂l i ∂θ (cid:1) ⊤ (cid:0) ∂l
Revised: where S ( θ ) = 1 N (cid:80) Ni =1 (cid:0) ∂l i ∂θ (cid:1) ⊤ (cid:0) ∂l

Original: i ∂θ (cid:1) ∈
Revised: i ∂θ (cid:1) ∈

Original: R m × m is the second moment of the (individual) loss gradients and S b ( θ ) =
Revised: R m × m is the second moment of the (individual) loss gradients and S b ( θ ) =

Original: E B (cid:104)(cid:0) 1 b (cid:80) x i ∈ B
Revised: E B (cid:104)(cid:0) 1 b (cid:80) x i ∈ B

Original: ∂l i ∂θ (cid:1) ⊤ (cid:0) 1 b (cid:80) x i ∈ B
Revised: ∂l i ∂θ (cid:1) ⊤ (cid:0) 1 b (cid:80) x i ∈ B

Original: ∂l i ∂θ (
Revised: ∂l i ∂θ (

Original: cid:1)(cid:105) ∈
Revised: cid:1)(cid:105) ∈

Original: R m × m is that of the mini-batch loss gradients with a batch size of b .
Revised: R m × m is that of the mini-batch loss gradients with a batch size of b .

Original: For our purpose, the relationship between the covariance of loss gradients and that of mini-batch loss gradients is useful.
Revised: For our purpose, the relationship between the covariance of loss gradients and that of mini-batch loss gradients is useful.

Original: The relationship is established in [47] as C b ( θ ) = γ b,N b C ( θ ) , where
Revised: The relationship is established in [47] as C b ( θ ) = γ b,N b C ( θ ) , where

Original: C ( θ ) , C b ( θ ) ∈
Revised: C ( θ ) , C b ( θ ) ∈

Original: R m × m are respectively the covariance of the (individual) loss gradients and minibatch loss gradients (with a batch size of b ), and γ b,N = N − b N − 1 if mini-batches are sampled without replacement and 1 if sampled with replacement.
Revised: R m × m are respectively the covariance of the (individual) loss gradients and minibatch loss gradients (with a batch size of b ), and γ b,N = N − b N − 1 if mini-batches are sampled without replacement and 1 if sampled with replacement.

Original: From the relations between the covariances and second moments given as C ( θ ) =
Revised: From the relations between the covariances and second moments given as C ( θ ) =

Original: S ( θ ) − S N ( θ ) and C b ( θ ) =
Revised: S ( θ ) − S N ( θ ) and C b ( θ ) =

Original: S b ( θ ) − S N ( θ ) with
Revised: S b ( θ ) − S N ( θ ) with

Original: S N ( θ ) = (cid:16) 1 N (cid:80) Ni
Revised: S N ( θ ) = (cid:16) 1 N (cid:80) Ni

Original: =1 ∂l i ∂θ (cid:17) ⊤ (cid:16) 1 N (cid:80) Ni
Revised: =1 ∂l i ∂θ (cid:17) ⊤ (cid:16) 1 N (cid:80) Ni

Original: =1 ∂l i ∂θ (cid:17)
Revised: =1 ∂l i ∂θ (cid:17)

Original: ∈ R m × m , we can relate S ( θ ) and S b ( θ ) as follows:
Revised: ∈ R m × m , we can relate S ( θ ) and S b ( θ ) as follows:

Original: [Equation]
Revised: [Equation]

Original: By applying (25) to (23), we can obtain the relationship between IGS and mini-batch IGS as follows:
Revised: By applying (25) to (23), we can obtain the relationship between IGS and mini-batch IGS as follows:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: By defining other information geometric sharpness measures as IGS-C ( θ ) ≡ IGS ( θ ) − IGS N ( θ ) and
Revised: By defining other information geometric sharpness measures as IGS-C ( θ ) ≡ IGS ( θ ) − IGS N ( θ ) and

Original: IGS b -C ( θ ) ≡ IGS b ( θ ) − IGS N ( θ ) , which can be considered as replacing the second moments with covariances in (22) and (23), the relationship between them becomes as follows:
Revised: IGS b -C ( θ ) ≡ IGS b ( θ ) − IGS N ( θ ) , which can be considered as replacing the second moments with covariances in (22) and (23), the relationship between them becomes as follows:

Original: [Equation]
Revised: [Equation]

Original: D On the transformation invariance of IGS D.1 Proof of Proposition 3. 
Revised: D On the transformation invariance of IGS D.1 Proof of Proposition 3.

Original: Proof.
Revised: Proof.

Original: To prove IGS ( θ ) =
Revised: To prove IGS ( θ ) =

Original: IGS ( g ( θ )) (or IGS b ( θ ) =
Revised: IGS ( g ( θ )) (or IGS b ( θ ) =

Original: IGS b ( g ( θ )) ) for all θ ∈ U , we have to show whether the following equality holds for any θ ∈ U and i, j ∈ { 1 , . . .
Revised: IGS b ( g ( θ )) ) for all θ ∈ U , we have to show whether the following equality holds for any θ ∈ U and i, j ∈ { 1 , . . .

Original: , N } :
Revised: , N } :

Original: [Equation]
Revised: [Equation]

Original: By differentiating f ( x ; θ ) =
Revised: By differentiating f ( x ; θ ) =

Original: f ( x ; g ( θ )) with respect to θ , we get (cid:16) ∂f∂θ (cid:17) = (cid:16)
Revised: f ( x ; g ( θ )) with respect to θ , we get (cid:16) ∂f∂θ (cid:17) = (cid:16)

Original: ∂f∂g
Revised: ∂f∂g

Original: ( θ ) (cid:17) (cid:16) ∂g ( θ ) ∂θ (cid:17) , where (cid:16) ∂g
Revised: ( θ ) (cid:17) (cid:16) ∂g ( θ ) ∂θ (cid:17) , where (cid:16) ∂g

Original: [Equation]
Revised: [Equation]

Original: θ
Revised: θ

Original: [Equation]
Revised: [Equation]

Original: ∂θ
Revised: ∂θ

Original: [Equation]
Revised: [Equation]

Original: ∈ R m × m is invertible by the assumption.
Revised: ∈ R m × m is invertible by the assumption.

Original: Applying this identity to the definition of the
Revised: Applying this identity to the definition of the

Original: Fisher information matrix gives the transformation rule of F ( θ ) = (cid:16) ∂g ( θ ) ∂θ (cid:17) ⊤ F ( g ( θ )) (cid:16) ∂g ( θ ) ∂θ (cid:17) .
Revised: Fisher information matrix gives the transformation rule of F ( θ ) = (cid:16) ∂g ( θ ) ∂θ (cid:17) ⊤ F ( g ( θ )) (cid:16) ∂g ( θ ) ∂θ (cid:17) .

Original: Furthermore, from the chain rule, (cid:0) ∂l i ∂θ (cid:1) =
Revised: Furthermore, from the chain rule, (cid:0) ∂l i ∂θ (cid:1) =

Original: (cid:16) ∂l i ∂g ( θ ) (cid:17) (cid:16) ∂g ( θ )
Revised: (cid:16) ∂l i ∂g ( θ ) (cid:17) (cid:16) ∂g ( θ )

Original: ∂θ (cid:17) .
Revised: ∂θ (cid:17) .

Original: Combining these two identities and using the fact that (cid:16) ∂g ( θ ) ∂θ (cid:17) is invertible and ( AMA ⊤ ) † =
Revised: Combining these two identities and using the fact that (cid:16) ∂g ( θ ) ∂θ (cid:17) is invertible and ( AMA ⊤ ) † =

Original: A −⊤ M † A − 1 for an invertible matrix
Revised: A −⊤ M † A − 1 for an invertible matrix

Original: A ∈ R m × m and any M ∈ R m × m give the desired (29).
Revised: A ∈ R m × m and any M ∈ R m × m give the desired (29).

Original: D.2 Other transformation examples
Revised: D.2 Other transformation examples

Original: We provide several examples of transformations that satisfy the assumption of Proposition 3.1 as follows:
Revised: We provide several examples of transformations that satisfy the assumption of Proposition 3.1 as follows:

Original: • For the fully-connected neural networks, the network function remains the same if we apply identical permutations for the i -th layer weights row-wisely and the i + 1 -th layer weights column-wisely.
Revised: • For the fully-connected neural networks, the network function remains the same if we apply identical permutations for the i -th layer weights row-wisely and the i + 1 -th layer weights column-wisely.

Original: (For convolutional neural networks, applying permutations channel-wisely would maintain the network function the same.)
Revised: (For convolutional neural networks, applying permutations channel-wisely would maintain the network function the same.)

Original: • For the fully-connected neural networks, when we apply odd functions (i.e., ϕ ( − u ) =
Revised: • For the fully-connected neural networks, when we apply odd functions (i.e., ϕ ( − u ) =

Original: − ϕ ( u ) ) as an activation function, the network function remains the same if we apply identical sign change to some rows of the i -th layer weights and corresponding columns of the i + 1 -th layer weights.
Revised: − ϕ ( u ) ) as an activation function, the network function remains the same if we apply identical sign change to some rows of the i -th layer weights and corresponding columns of the i + 1 -th layer weights.

Original: • As a bit more complicated nonlinear example, consider a neural network with twodimensional input x = ( x 1 , x 2 ) , three-dimensional hidden layer, and one-dimensional output.
Revised: • As a bit more complicated nonlinear example, consider a neural network with twodimensional input x = ( x 1 , x 2 ) , three-dimensional hidden layer, and one-dimensional output.

Original: Consider s ( x ) =
Revised: Consider s ( x ) =

Original: x 2 as the (element-wise) nonlinear activation function to apply.
Revised: x 2 as the (element-wise) nonlinear activation function to apply.

Original: Denote by W (1) ∈ R 3 × 2 the input-to-hidden weights and by W (2)
Revised: Denote by W (1) ∈ R 3 × 2 the input-to-hidden weights and by W (2)

Original: ∈ R 1 × 3 the hidden-tooutput weights.
Revised: ∈ R 1 × 3 the hidden-tooutput weights.

Original: The network function then becomes
Revised: The network function then becomes

Original: [Equation]
Revised: [Equation]

Original: where W ij is the ( i, j ) entry of W .
Revised: where W ij is the ( i, j ) entry of W .

Original: Set another network with the same architecture as
Revised: Set another network with the same architecture as

Original: [Equation]
Revised: [Equation]

Original: using weights
Revised: using weights

Original: [Equation]
Revised: [Equation]

Original: The condition for the two networks modeling identical function can be expressed as follows:
Revised: The condition for the two networks modeling identical function can be expressed as follows:

Original: [Equation]
Revised: [Equation]

Original: If we rearrange the above for each coefficient of x 21 , x 1 x 2 , x 22 , we obtain
Revised: If we rearrange the above for each coefficient of x 21 , x 1 x 2 , x 22 , we obtain

Original: [Equation]
Revised: [Equation]

Original: For the set of W (1) , W (2) , ˜ W (1) , ˜ W (2) that satisfy above equations, the two network functions f ( x ; W (1) , W (2) ) and ˜ f ( x ; ˜ W (1) , ˜ W (2) ) become identical.
Revised: For the set of W (1) , W (2) , ˜ W (1) , ˜ W (2) that satisfy above equations, the two network functions f ( x ; W (1) , W (2) ) and ˜ f ( x ; ˜ W (1) , ˜ W (2) ) become identical.

Original: Consider finding ˜ W (2) for given W (1) , W (2) , and ˜ W (1)
Revised: Consider finding ˜ W (2) for given W (1) , W (2) , and ˜ W (1)

Original: =
Revised: =

Original: ϕ 1 ( W (1) ) for a bijective nonlinear function ϕ : R 3 × 2 → R 3 × 2 .
Revised: ϕ 1 ( W (1) ) for a bijective nonlinear function ϕ : R 3 × 2 → R 3 × 2 .

Original: Since there are three equations, i.e., (30), (31), and (32), and three variables of ( ˜ W (2)11 , ˜ W (2)12 , ˜ W (2)13 ) in ˜ W (2) , we can find the solution of ˜ W (2) and represent it as ˜ W (2) =
Revised: Since there are three equations, i.e., (30), (31), and (32), and three variables of ( ˜ W (2)11 , ˜ W (2)12 , ˜ W (2)13 ) in ˜ W (2) , we can find the solution of ˜ W (2) and represent it as ˜ W (2) =

Original: ϕ 2 ( W (1) , W (2) ) .
Revised: ϕ 2 ( W (1) , W (2) ) .

Original: We can then figure out that two different weight configurations can model identical network functions, with a nonlinear relationship between each other given as ˜ W (1) =
Revised: We can then figure out that two different weight configurations can model identical network functions, with a nonlinear relationship between each other given as ˜ W (1) =

Original: ϕ 1 ( W (1) ) and ˜ W (2) =
Revised: ϕ 1 ( W (1) ) and ˜ W (2) =

Original: ϕ 2 ( W (1) , W (2) ) .
Revised: ϕ 2 ( W (1) , W (2) ) .

Original: Even if the input or output dimension increases, if the hidden variable dimension increases appropriately, we can find mappings between weights similarly that can model identical neural network functions.
Revised: Even if the input or output dimension increases, if the hidden variable dimension increases appropriately, we can find mappings between weights similarly that can model identical neural network functions.

Original: E An information geometric margin for multi-class problems
Revised: E An information geometric margin for multi-class problems

Original: An information geometric margin can also be formulated in the multi-class classification case similarly as done in Section 3.3.
Revised: An information geometric margin can also be formulated in the multi-class classification case similarly as done in Section 3.3.

Original: For a data point ( x i , y i ) of class y i
Revised: For a data point ( x i , y i ) of class y i

Original: =
Revised: =

Original: c , we consider the decision boundary between class c and class c ′ for all c ′ ̸
Revised: c , we consider the decision boundary between class c and class c ′ for all c ′ ̸

Original: = c .
Revised: = c .

Original: The condition for a point x ∈ R D to lie on this decision boundary between classes c and c ′ is given as f c ( x ; θ ) =
Revised: The condition for a point x ∈ R D to lie on this decision boundary between classes c and c ′ is given as f c ( x ; θ ) =

Original: f c ′ ( x ; θ ) , where f j ( · ; θ ) :
Revised: f c ′ ( x ; θ ) , where f j ( · ; θ ) :

Original: R D → R is the j -th logit function (or the j -th output of the neural network).
Revised: R D → R is the j -th logit function (or the j -th output of the neural network).

Original: Again, we consider perturbing the parameter values along the eigenvectors of the Fisher information matrix to make the data ( x i , y i ) lie on this decision boundary.
Revised: Again, we consider perturbing the parameter values along the eigenvectors of the Fisher information matrix to make the data ( x i , y i ) lie on this decision boundary.

Original: Suppose we perturb the parameter value from θ to θ + ϵ i,k,c ′ ∆ θ k , where ∆ θ k is the k -th eigenvector (associated with the k -th eigenvalue λ k ) and ϵ i,k,c ′ is a scalar.
Revised: Suppose we perturb the parameter value from θ to θ + ϵ i,k,c ′ ∆ θ k , where ∆ θ k is the k -th eigenvector (associated with the k -th eigenvalue λ k ) and ϵ i,k,c ′ is a scalar.

Original: To make the data ( x i , y i ) lie on the decision boundary, the perturbation step size ϵ i,k,c ′ should satisfy the following:
Revised: To make the data ( x i , y i ) lie on the decision boundary, the perturbation step size ϵ i,k,c ′ should satisfy the following:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where f i,j ( θ ) ∈ R is the f j ( x i ; θ ) .
Revised: where f i,j ( θ ) ∈ R is the f j ( x i ; θ ) .

Original: From the first-order Taylor expansion assuming small ϵ i,k,c ′ , the following approximations hold:
Revised: From the first-order Taylor expansion assuming small ϵ i,k,c ′ , the following approximations hold:

Original: [Equation]
Revised: [Equation]

Original: where J i,j = (cid:16) ∂f i,j ∂θ (cid:17) ∈ R 1 × m .
Revised: where J i,j = (cid:16) ∂f i,j ∂θ (cid:17) ∈ R 1 × m .

Original: Note that
Revised: Note that

Original: ϵ i,k,c ′ can have different signs for different c ′ s according to the sign of the ( J i,c ′ − J i,c ) ∆ θ k term.
Revised: ϵ i,k,c ′ can have different signs for different c ′ s according to the sign of the ( J i,c ′ − J i,c ) ∆ θ k term.

Original: Considering the weighted sum of (36) with the weights of the probability p i,c ′ , i.e., the probability of x i being class c ′ predicted from the model, we can approximate the directional derivative of the cross-entropy loss using ϵ i,k,c ′ as follows:
Revised: Considering the weighted sum of (36) with the weights of the probability p i,c ′ , i.e., the probability of x i being class c ′ predicted from the model, we can approximate the directional derivative of the cross-entropy loss using ϵ i,k,c ′ as follows:

Original: [Equation]
Revised: [Equation]

Original: where we have used (cid:0)
Revised: where we have used (cid:0)

Original: [Equation]
Revised: [Equation]

Original: Our sharpness measure can then be related to the (signed) norm of the perturbation vector √ λ k ϵ i,k,c ′ as follows:
Revised: Our sharpness measure can then be related to the (signed) norm of the perturbation vector √ λ k ϵ i,k,c ′ as follows:

Original: [Equation]
Revised: [Equation]

Original: where the last approximation comes from applying (37).
Revised: where the last approximation comes from applying (37).

Original: Therefore as in the binary classification case, we can observe that the smaller our sharpness measure, the larger the magnitude of the margin, i.e., the (signed) norm √ λ k ϵ i,k,c ′ of the perturbation vector, in the information geometric sense.
Revised: Therefore as in the binary classification case, we can observe that the smaller our sharpness measure, the larger the magnitude of the margin, i.e., the (signed) norm √ λ k ϵ i,k,c ′ of the perturbation vector, in the information geometric sense.

Original: F Adaptability of IGS to the square loss
Revised: F Adaptability of IGS to the square loss

Original: Our measure can be defined for the square loss by considering the log-likelihood for a Gaussian, i.e., l i ( θ )
Revised: Our measure can be defined for the square loss by considering the log-likelihood for a Gaussian, i.e., l i ( θ )

Original: =
Revised: =

Original: l ( y i , f ( x i ; θ ))
Revised: l ( y i , f ( x i ; θ ))

Original: = − log p ( y i | x i ; θ ) =
Revised: = − log p ( y i | x i ; θ ) =

Original: 12 ( y i − f ( x i ; θ )) 2 + const.
Revised: 12 ( y i − f ( x i ; θ )) 2 + const.

Original: in (1) for p ( y | x ; θ ) =
Revised: in (1) for p ( y | x ; θ ) =

Original: C exp( − 1 2 ( y − f ( x ; θ )) 2 ) with C as a normalization constant.
Revised: C exp( − 1 2 ( y − f ( x ; θ )) 2 ) with C as a normalization constant.

Original: The corresponding reparametrizationand scale-invariance properties, as well as the connections with generalization in terms of TIC and margin, can all be seen as well.
Revised: The corresponding reparametrizationand scale-invariance properties, as well as the connections with generalization in terms of TIC and margin, can all be seen as well.

Original: More specifically, the invariance properties are satisfied by definition.
Revised: More specifically, the invariance properties are satisfied by definition.

Original: Since we do not assume any specific form of the likelihood during the derivation of TIC, the TIC can be well-defined for the square loss [9], and the link between our measure and the TIC comes again from the fact that the FIM and the Hessian are approximately equal when the neural network is sufficiently trained for the square loss [28].
Revised: Since we do not assume any specific form of the likelihood during the derivation of TIC, the TIC can be well-defined for the square loss [9], and the link between our measure and the TIC comes again from the fact that the FIM and the Hessian are approximately equal when the neural network is sufficiently trained for the square loss [28].

Original: In terms of our definition of margin, if we let l i = 12 ( z i − f ( x i ; θ )) 

Revised: In terms of our definition of margin, if we let l i = 12 ( z i − f ( x i ; θ ))

Original: ( z i ∈ { 0 , 1 } ) and set the condition for data lying on the decision boundary as f ( x ; θ ) = 12 , we can show the connection of the measure to the margin without much difficulty according to a discussion similar to Section 3.3.2.
Revised: ( z i ∈ { 0 , 1 } ) and set the condition for data lying on the decision boundary as f ( x ; θ ) = 12 , we can show the connection of the measure to the margin without much difficulty according to a discussion similar to Section 3.3.2.

Original: G Experimental details G.1 Reparametrization invariance
Revised: G Experimental details G.1 Reparametrization invariance

Original: We examine the reparametrization-invariance of the IGS and compare it to other sharpness measures such as the trace of the Hessian, the spectral norm (or the top eigenvalue) of the Hessian, the
Revised: We examine the reparametrization-invariance of the IGS and compare it to other sharpness measures such as the trace of the Hessian, the spectral norm (or the top eigenvalue) of the Hessian, the

Original: Fisher-Rao norm [26], the Rangamani et al.
Revised: Fisher-Rao norm [26], the Rangamani et al.

Original: ’s measure [40], and the Petzka
Revised: ’s measure [40], and the Petzka

Original: et al.
Revised: et al.

Original: ’s measure [39].
Revised: ’s measure [39].

Original: We consider a two-layer fully connected neural network with 20-dimensional hidden units and ReLU activation functions for this experiment.
Revised: We consider a two-layer fully connected neural network with 20-dimensional hidden units and ReLU activation functions for this experiment.

Original: The neural network is trained on two-dimensional synthetic data generated from mixtures of three Gaussians via SGD with a learning rate of 0.1 and a batch size of 40 until the training loss gets lower than 5e-3.
Revised: The neural network is trained on two-dimensional synthetic data generated from mixtures of three Gaussians via SGD with a learning rate of 0.1 and a batch size of 40 until the training loss gets lower than 5e-3.

Original: After training ten neural networks initialized with different random seeds, we obtain reparametrized neural networks for each of them by considering an element-wise nonlinear reparametrization η = g ( θ ) =
Revised: After training ten neural networks initialized with different random seeds, we obtain reparametrized neural networks for each of them by considering an element-wise nonlinear reparametrization η = g ( θ ) =

Original: ( | θ − ˆ θ | 2 + b ) a ( θ − ˆ θ )+ ˆ η , considered in Figure 5 in Section 5 of [8].
Revised: ( | θ − ˆ θ | 2 + b ) a ( θ − ˆ θ )+ ˆ η , considered in Figure 5 in Section 5 of [8].

Original: We consider several pairs of ( a, b ) for a ∈ {− 0 .
Revised: We consider several pairs of ( a, b ) for a ∈ {− 0 .

Original: 5 } and b ∈ { 0 . 01 , 0 .
Revised: 5 } and b ∈ { 0 . 01 , 0 .

Original: 1 , 1 } in defining g ( θ ) , with ˆ θ set as the obtained parameter and ˆ η sampled from the standard normal distribution.
Revised: 1 , 1 } in defining g ( θ ) , with ˆ θ set as the obtained parameter and ˆ η sampled from the standard normal distribution.

Original: The above sharpness measures are then calculated for both the original models (with respect to θ ) and their reparametrized ones ˜ f
Revised: The above sharpness measures are then calculated for both the original models (with respect to θ ) and their reparametrized ones ˜ f

Original: ( · ; η ) = f ( · ; g − 1 ( η ))
Revised: ( · ; η ) = f ( · ; g − 1 ( η ))

Original: =
Revised: =

Original: f ( · ; θ ) (with respect to η )
Revised: f ( · ; θ ) (with respect to η )

Original: [Figure]
Revised: [Figure]

Original: ( y = log 10 ( x ) ) denote the case of reparametrizationinvariance.
Revised: ( y = log 10 ( x ) ) denote the case of reparametrizationinvariance.

Original: In Figure 6, we plot the sharpness measures calculated for the original models on the x-axis and the logarithm of those for the reparametrized ones on the y-axis.
Revised: In Figure 6, we plot the sharpness measures calculated for the original models on the x-axis and the logarithm of those for the reparametrized ones on the y-axis.

Original: Points with the same x-axis values correspond to reparametrizations from the identical original model with different choices of ( a, b ) .
Revised: Points with the same x-axis values correspond to reparametrizations from the identical original model with different choices of ( a, b ) .

Original: The solid lines ( y = log 10 ( x ) ) denote the case in which the sharpness measures evaluated for the original model and the reparameterized model are the same.
Revised: The solid lines ( y = log 10 ( x ) ) denote the case in which the sharpness measures evaluated for the original model and the reparameterized model are the same.

Original: From the figure, we can observe that only the IGS is invariant to the considered reparametrizations, while the other measures including the scale-invariant ones can be arbitrarily varied in their magnitudes significantly.
Revised: From the figure, we can observe that only the IGS is invariant to the considered reparametrizations, while the other measures including the scale-invariant ones can be arbitrarily varied in their magnitudes significantly.

Original: These figures are merged into a single figure in Figure 2 of the manuscript, reporting the ratio of the sharpness measures evaluated for the reparameterized models to those evaluated for the original models.
Revised: These figures are merged into a single figure in Figure 2 of the manuscript, reporting the ratio of the sharpness measures evaluated for the reparameterized models to those evaluated for the original models.

Original: G.2 Correlations of the IGS and other sharpness measures to the generalization gap
Revised: G.2 Correlations of the IGS and other sharpness measures to the generalization gap

Original: To investigate the correlations of the IGS and other sharpness measures to the generalization performances, we experiment using LeNet-5 [25] trained on MNIST and CIFAR-10 data sets via SGD.
Revised: To investigate the correlations of the IGS and other sharpness measures to the generalization performances, we experiment using LeNet-5 [25] trained on MNIST and CIFAR-10 data sets via SGD.

Original: To obtain models possessing different generalization performances for MNIST data sets, we train the models via SGD with learning rates in { 0 . 002 , 0 .
Revised: To obtain models possessing different generalization performances for MNIST data sets, we train the models via SGD with learning rates in { 0 . 002 , 0 .

Original: and batch sizes in { 64 , 128 , 256 , 512 , 1024 } , and the training is terminated when the training loss becomes lower than 0.01.
Revised: and batch sizes in { 64 , 128 , 256 , 512 , 1024 } , and the training is terminated when the training loss becomes lower than 0.01.

Original: For CIFAR-10 data sets, we use learning rates in { 0 . 001 , 0 . 002 , 0 . 005 , 0 . 01 , 0 . 02 , 0 . 05 } and the identical batch sizes to the above, and the training is terminated when the training loss gets lower than 0.1.
Revised: For CIFAR-10 data sets, we use learning rates in { 0 . 001 , 0 . 002 , 0 . 005 , 0 . 01 , 0 . 02 , 0 . 05 } and the identical batch sizes to the above, and the training is terminated when the training loss gets lower than 0.1.

Original: All the experiments are performed for two different random seeds, and only the models with loss converged in a specified number of iterations are considered.
Revised: All the experiments are performed for two different random seeds, and only the models with loss converged in a specified number of iterations are considered.

Original: For the trained models, we compute the IGS and other sharpness measures such as the trace of the Hessian, the Fisher-Rao norm [26], the Rangamani et al.
Revised: For the trained models, we compute the IGS and other sharpness measures such as the trace of the Hessian, the Fisher-Rao norm [26], the Rangamani et al.

Original: ’s measure [40], and the Petzka
Revised: ’s measure [40], and the Petzka

Original: et al.
Revised: et al.

Original: ’s measure [39] similarly as explained in Appendix G.1.
Revised: ’s measure [39] similarly as explained in Appendix G.1.

Original: (When calculating our measure, we consider the expectation taken with respect to the data distribution (similarly to [45]), which is approximated by a finite sum of the integrands over the test data.)
Revised: (When calculating our measure, we consider the expectation taken with respect to the data distribution (similarly to [45]), which is approximated by a finite sum of the integrands over the test data.)

Original: In Figure 3, the sharpness values are normalized to be in [0,1].
Revised: In Figure 3, the sharpness values are normalized to be in [0,1].

Original: We can observe that the IGS has better correlation with the generalization gap, i.e., the difference between the train and test losses, than other sharpness measures in the figure.
Revised: We can observe that the IGS has better correlation with the generalization gap, i.e., the difference between the train and test losses, than other sharpness measures in the figure.

Original: In calculating the IGS, a crucial difference to the experiments in [45] is that we use LeNet-5, of which the number of parameters exceeds 60,000.
Revised: In calculating the IGS, a crucial difference to the experiments in [45] is that we use LeNet-5, of which the number of parameters exceeds 60,000.

Original: Therefore, it is difficult to compute the exact values of
Revised: Therefore, it is difficult to compute the exact values of

Original: IGS for the obtained models.
Revised: IGS for the obtained models.

Original: Here we briefly explain how the IGS is evaluated in our experiments.
Revised: Here we briefly explain how the IGS is evaluated in our experiments.

Original: We first obtain the top- m ′ eigenvalues/eigenvectors of the FIM numerically via the power iteration, using subsampled data points { x 1 , . . .
Revised: We first obtain the top- m ′ eigenvalues/eigenvectors of the FIM numerically via the power iteration, using subsampled data points { x 1 , . . .

Original: , x N s } (with N s = 1 , 024 in our experiments) from the data set as explained in Appendix G.5.1.
Revised: , x N s } (with N s = 1 , 024 in our experiments) from the data set as explained in Appendix G.5.1.

Original: The dimension m ′ of eigensubspace to consider (for the pseudoinverse of the approximated FIM) is chosen to be the least number that makes the sum of top- m ′ eigenvalues exceed 95 % of Tr ( F ( θ )) .
Revised: The dimension m ′ of eigensubspace to consider (for the pseudoinverse of the approximated FIM) is chosen to be the least number that makes the sum of top- m ′ eigenvalues exceed 95 % of Tr ( F ( θ )) .

Original: To further reduce the computational costs, the mini-batch IGS in (7) is computed from mini-batch gradients (with a batch size of 128) obtained by iterating for an epoch as follows:
Revised: To further reduce the computational costs, the mini-batch IGS in (7) is computed from mini-batch gradients (with a batch size of 128) obtained by iterating for an epoch as follows:

Original: [Equation]
Revised: [Equation]

Original: where B is the considered set of mini-batches.
Revised: where B is the considered set of mini-batches.

Original: We then convert the mini-batch IGS to IGS using (27).
Revised: We then convert the mini-batch IGS to IGS using (27).

Original: We perform these procedures for five different subsamples to approximate the FIM, and average the obtained IGS values.
Revised: We perform these procedures for five different subsamples to approximate the FIM, and average the obtained IGS values.

Original: G.3 Regularization experiments – a toy example G.3.1 A finite difference-based approximation of IGS
Revised: G.3 Regularization experiments – a toy example G.3.1 A finite difference-based approximation of IGS

Original: Consider the following eigendecomposition of the Fisher information matrix
Revised: Consider the following eigendecomposition of the Fisher information matrix

Original: [Equation]
Revised: [Equation]

Original: where λ k , ∆ θ k are respectively the k -th eigenvalue and eigenvector of F ( θ ) , and m
Revised: where λ k , ∆ θ k are respectively the k -th eigenvalue and eigenvector of F ( θ ) , and m

Original: ′ is the largest index of non-zero λ k .
Revised: ′ is the largest index of non-zero λ k .

Original: The sharpness measure is then approximated as follows:
Revised: The sharpness measure is then approximated as follows:

Original: [Equation]
Revised: [Equation]

Original: where E ϵ [ · ] denotes the expectation with respect to an m ′ -dimensional
Revised: where E ϵ [ · ] denotes the expectation with respect to an m ′ -dimensional

Original: vector ϵ
Revised: vector ϵ

Original: =
Revised: =

Original: ( ϵ 1 , . . .
Revised: ( ϵ 1 , . . .

Original: , ϵ m ′ )
Revised: , ϵ m ′ )

Original: ∼
Revised: ∼

Original: N (0 , I ) in (42), and the first-order approximation is used to derive (42) with α > 0 as a small scalar.
Revised: N (0 , I ) in (42), and the first-order approximation is used to derive (42) with α > 0 as a small scalar.

Original: To avoid possible numerical instabilities stemming from the varying scales of the eigenvalues, we can consider following regularizer using a fixed amount of perturbation from θ in the first l i ( · ) -term in (42):
Revised: To avoid possible numerical instabilities stemming from the varying scales of the eigenvalues, we can consider following regularizer using a fixed amount of perturbation from θ in the first l i ( · ) -term in (42):

Original: [Equation]
Revised: [Equation]

Original: where v ϵ
Revised: where v ϵ

Original: [Figure]
Revised: [Figure]

Original: θ k is a perturbation vector with η ≥ 0 included for numerical stability.
Revised: θ k is a perturbation vector with η ≥ 0 included for numerical stability.

Original: G.3.2 Data sets, models, and hyperparameters
Revised: G.3.2 Data sets, models, and hyperparameters

Original: For the toy examples in Section 4.1, we consider classifying two-dimensional synthetic data generated from mixtures of three Gaussians.
Revised: For the toy examples in Section 4.1, we consider classifying two-dimensional synthetic data generated from mixtures of three Gaussians.

Original: As shown in Figure 7, we assign distinct labels to each Gaussian for training data.
Revised: As shown in Figure 7, we assign distinct labels to each Gaussian for training data.

Original: Test data are labeled analogously to the training data.
Revised: Test data are labeled analogously to the training data.

Original: For poison data, we assign labels different from training and test data to each Gaussian by randomly choosing labels for each data among the other two labels.
Revised: For poison data, we assign labels different from training and test data to each Gaussian by randomly choosing labels for each data among the other two labels.

Original: We sample 60 data for each training, test, and poison data.
Revised: We sample 60 data for each training, test, and poison data.

Original: [Figure]
Revised: [Figure]

Original: A six-layer fully connected neural network with 100-dimensional hidden units and hyperbolic tangent (Tanh) activation functions is used for the experiments.
Revised: A six-layer fully connected neural network with 100-dimensional hidden units and hyperbolic tangent (Tanh) activation functions is used for the experiments.

Original: We use a learning rate of 0.1 and a batch size of 30 for both SGD with and without regularization.
Revised: We use a learning rate of 0.1 and a batch size of 30 for both SGD with and without regularization.

Original: The hyperparameters for regularization are set as α = 0 . 5 , ˜
Revised: The hyperparameters for regularization are set as α = 0 . 5 , ˜

Original: ρ = ρ/α 2 = 2 . 0 , and η = 0 .
Revised: ρ = ρ/α 2 = 2 . 0 , and η = 0 .

Original: We use the top three eigenvalues and eigenvectors (i.e., m ′ = 3 for v ϵ in (43)) obtained from the PyHessian library [48] 6 to approximate the regularization term as in (43) and update them every 20 iterations.
Revised: We use the top three eigenvalues and eigenvectors (i.e., m ′ = 3 for v ϵ in (43)) obtained from the PyHessian library [48] 6 to approximate the regularization term as in (43) and update them every 20 iterations.

Original: We refer the reader to Appendix G.5.1 for additional details of the eigendecomposition.
Revised: We refer the reader to Appendix G.5.1 for additional details of the eigendecomposition.

Original: For a better interpretation of the loss trajectory, we have smoothed the trajectory by averagingconsecutive loss values in Figures 5 (c) and (d).
Revised: For a better interpretation of the loss trajectory, we have smoothed the trajectory by averagingconsecutive loss values in Figures 5 (c) and (d).

Original: G.4 Regularization experiments – MNIST and CIFAR-
Revised: G.4 Regularization experiments – MNIST and CIFAR-

Original: G.4.1 Approximations of the natural gradients
Revised: G.4.1 Approximations of the natural gradients

Original: As mentioned in Section 4.2, it is required to approximate the natural gradient F ( θ ) † (cid:0) ∂l∂θ (cid:1)
Revised: As mentioned in Section 4.2, it is required to approximate the natural gradient F ( θ ) † (cid:0) ∂l∂θ (cid:1)

Original: ⊤ ∈ R m in (13) to regularize the mini-batch IGS.
Revised: ⊤ ∈ R m in (13) to regularize the mini-batch IGS.

Original: In this experiment, we consider some approximations based on the eigenvalue-corrected Kronecker factorization (EKFAC) method [13] 7 to efficiently calculate the FIM and its (pseudo-)inverse.
Revised: In this experiment, we consider some approximations based on the eigenvalue-corrected Kronecker factorization (EKFAC) method [13] 7 to efficiently calculate the FIM and its (pseudo-)inverse.

Original: (We provide a brief explanation of the EKFAC method in Appendix G.5.2.)
Revised: (We provide a brief explanation of the EKFAC method in Appendix G.5.2.)

Original: We devise three different methods for the approximation; the methods differ in detail from each other in terms of which labels are used to approximate the FIM or to calculate (cid:0) ∂l∂θ (cid:1) in the EKFAC method.
Revised: We devise three different methods for the approximation; the methods differ in detail from each other in terms of which labels are used to approximate the FIM or to calculate (cid:0) ∂l∂θ (cid:1) in the EKFAC method.

Original: The methods are classified as follows:
Revised: The methods are classified as follows:

Original: • When approximating F ( θ ) in [20], they use sampled labels according to the probabilistic model that the neural network models with current parameter values.
Revised: • When approximating F ( θ ) in [20], they use sampled labels according to the probabilistic model that the neural network models with current parameter values.

Original: We follow this idea and use sampled labels to approximate F ( θ ) and the original labels for (cid:0) ∂l∂θ (cid:1) in our first method denoted as Ours 1 in Table 2.
Revised: We follow this idea and use sampled labels to approximate F ( θ ) and the original labels for (cid:0) ∂l∂θ (cid:1) in our first method denoted as Ours 1 in Table 2.

Original: • The second method uses original labels for both F ( θ ) and (cid:0) ∂l∂θ (cid:1) and is denoted as Ours 2 in Table 2.
Revised: • The second method uses original labels for both F ( θ ) and (cid:0) ∂l∂θ (cid:1) and is denoted as Ours 2 in Table 2.

Original: This method can be considered as utilizing the empirical Fisher discussed in [23].
Revised: This method can be considered as utilizing the empirical Fisher discussed in [23].

Original: • The third method uses sampled labels for both F ( θ ) and (cid:0) ∂l∂θ (cid:1) and is denoted as Ours 3 in Table 2.
Revised: • The third method uses sampled labels for both F ( θ ) and (cid:0) ∂l∂θ (cid:1) and is denoted as Ours 3 in Table 2.

Original: (The results from this method are reported in Table 1, and the method is denoted as Ours in the table.)
Revised: (The results from this method are reported in Table 1, and the method is denoted as Ours in the table.)

Original: Table 2 reports the averages and standard errors of the test classification accuracies obtained from three runs of all of our regularization methods as well as SGD, GR, SAM, and ASAM methods.
Revised: Table 2 reports the averages and standard errors of the test classification accuracies obtained from three runs of all of our regularization methods as well as SGD, GR, SAM, and ASAM methods.

Original: All of our methods show superior performance to SGD, and the third method (Ours 3) shows the best performance among them for most of the considered settings.
Revised: All of our methods show superior performance to SGD, and the third method (Ours 3) shows the best performance among them for most of the considered settings.

Original: The experimental details are provided in the next section
Revised: The experimental details are provided in the next section

Original: [Table] [Table]
Revised: [Table] [Table]

Original: G.4.2 Models and hyperparameters
Revised: G.4.2 Models and hyperparameters

Original: For the experiments using MNIST data, a three-layer fully connected neural network with 200dimensional hidden units and ReLU activation functions is used.
Revised: For the experiments using MNIST data, a three-layer fully connected neural network with 200dimensional hidden units and ReLU activation functions is used.

Original: The neural network is trained with a learning rate of 0.1 and a batch size of 128 for 200 epochs.
Revised: The neural network is trained with a learning rate of 0.1 and a batch size of 128 for 200 epochs.

Original: For CIFAR-10/100 experiments, convolutional neural networks such as VGG [42], ResNet [15], and WideResNet [50] are used.
Revised: For CIFAR-10/100 experiments, convolutional neural networks such as VGG [42], ResNet [15], and WideResNet [50] are used.

Original: The neural networks are trained with a batch size of 128 and a weight decay coefficient of 5e-4 for 200 epochs.
Revised: The neural networks are trained with a batch size of 128 and a weight decay coefficient of 5e-4 for 200 epochs.

Original: We additionally apply data augmentation and label smoothing [30] methods, and the learning rates are scheduled according to the cosine annealing method [27] with the maximum learning rate of 0.1 and 10 epochs to warm up.
Revised: We additionally apply data augmentation and label smoothing [30] methods, and the learning rates are scheduled according to the cosine annealing method [27] with the maximum learning rate of 0.1 and 10 epochs to warm up.

Original: In approximating the natural gradient using the EKFAC method [13], we set the (Kronecker-factored) eigenbasis update frequency of 100 and 500 for the MNIST and CIFAR-10/100 experiments, respectively.
Revised: In approximating the natural gradient using the EKFAC method [13], we set the (Kronecker-factored) eigenbasis update frequency of 100 and 500 for the MNIST and CIFAR-10/100 experiments, respectively.

Original: We also apply the running average option in correcting the scaling of natural gradients.
Revised: We also apply the running average option in correcting the scaling of natural gradients.

Original: The
Revised: The

Original: [Figure]
Revised: [Figure]

Original: method are chosen among ρ ∈ { 0 . 01 , 0 . 02 , 0 . 03 , 0 . 05 , 0 . 1 } for the MNIST experiments and ρ ∈ { 0 . 001 , 0 . 002 , 0 . 003 , 0 .
Revised: method are chosen among ρ ∈ { 0 . 01 , 0 . 02 , 0 . 03 , 0 . 05 , 0 . 1 } for the MNIST experiments and ρ ∈ { 0 . 001 , 0 . 002 , 0 . 003 , 0 .

Original: 02 } for the CIFAR-10/100 experiments.
Revised: 02 } for the CIFAR-10/100 experiments.

Original: To provide the sensitivity to the hyperparameter ρ of our method (the thrid one proposed in Appendix G.4.1), we have depicted the results under various choices of ρ in Figure 8.
Revised: To provide the sensitivity to the hyperparameter ρ of our method (the thrid one proposed in Appendix G.4.1), we have depicted the results under various choices of ρ in Figure 8.

Original: We consider the regularization coefficient ρ ∈ { 5 e − 4 , 0 .
Revised: We consider the regularization coefficient ρ ∈ { 5 e − 4 , 0 .

Original: 03 , 0 . 05 } for the squared gradient norm in the GR method [43].
Revised: 03 , 0 . 05 } for the squared gradient norm in the GR method [43].

Original: We consider ρ ∈ { 0 . 05 , 0 . 1 , 0 .
Revised: We consider ρ ∈ { 0 . 05 , 0 . 1 , 0 .

Original: 5 , 1 . 0 } for the SAM method [12], and consider ρ ∈ { 0 . 3 , 0 .
Revised: 5 , 1 . 0 } for the SAM method [12], and consider ρ ∈ { 0 . 3 , 0 .

Original: 5 , 1 . 0 , 2 . 0 , 3 . 0 , 5 . 0 } and η = 0 .
Revised: 5 , 1 . 0 , 2 . 0 , 3 . 0 , 5 . 0 } and η = 0 .

Original: 01 for the ASAM method [24], where the hyperparameters ρ and η are defined as in [12, 24].
Revised: 01 for the ASAM method [24], where the hyperparameters ρ and η are defined as in [12, 24].

Original: We report the results obtained from hyperparameters showing the best test accuracy in Tables 1 and 2, where the results are averaged over trials from three different random seeds.
Revised: We report the results obtained from hyperparameters showing the best test accuracy in Tables 1 and 2, where the results are averaged over trials from three different random seeds.

Original: Due to the space constraints, in Table 1 of the manuscript, we report only the results from VGG11-BN, ResNet-56, and WRN-16-8 for CIFAR10/100 experiments and only the third method among the methods introduced in Appendix G.4.1.
Revised: Due to the space constraints, in Table 1 of the manuscript, we report only the results from VGG11-BN, ResNet-56, and WRN-16-8 for CIFAR10/100 experiments and only the third method among the methods introduced in Appendix G.4.1.

Original: Under our experimental settings, elapsed times per epoch (in seconds) during training for
Revised: Under our experimental settings, elapsed times per epoch (in seconds) during training for

Original: SGD/GR/SAM/ASAM/Ours are 1.68/3.50/3.60/3.85/5.70 in MNIST experiments.
Revised: SGD/GR/SAM/ASAM/Ours are 1.68/3.50/3.60/3.85/5.70 in MNIST experiments.

Original: For CIFAR-10/ experiments, those are 5.95/24.7/14.8/16.1/40.1 for VGG11-BN, 25.7/98.3/63.7/68.3/66.9 for ResNet56, and 24.2/145/47.8/48.0/80.7 for WRN-16-8.
Revised: For CIFAR-10/ experiments, those are 5.95/24.7/14.8/16.1/40.1 for VGG11-BN, 25.7/98.3/63.7/68.3/66.9 for ResNet56, and 24.2/145/47.8/48.0/80.7 for WRN-16-8.

Original: Most of the experiments have been performed using
Revised: Most of the experiments have been performed using

Original: NVIDIA Tesla V100 GPUs.
Revised: NVIDIA Tesla V100 GPUs.

Original: Note that in our regularization method, the process of finding the natural gradient in δ = ρF ( θ ) † (cid:0) ∂l∂θ (cid:1) ⊤ in (13) is added once every iteration compared to vanilla SGD.
Revised: Note that in our regularization method, the process of finding the natural gradient in δ = ρF ( θ ) † (cid:0) ∂l∂θ (cid:1) ⊤ in (13) is added once every iteration compared to vanilla SGD.

Original: Therefore, additional time complexity depends on the details of methods to obtain the natural gradients, i.e., the
Revised: Therefore, additional time complexity depends on the details of methods to obtain the natural gradients, i.e., the

Original: EKFAC method (see Appendix G.5.2).
Revised: EKFAC method (see Appendix G.5.2).

Original: G.5 Details for the eigendecomposition of the FIM
Revised: G.5 Details for the eigendecomposition of the FIM

Original: Several approximations have been used in our experiments for the eigendecomposition (required to get the pseudo-inverse) of the FIM.
Revised: Several approximations have been used in our experiments for the eigendecomposition (required to get the pseudo-inverse) of the FIM.

Original: The computations in Sections 3.3.1 and 4.1 are based on the power iteration, and those in Section 4.2 are based on the EKFAC method.
Revised: The computations in Sections 3.3.1 and 4.1 are based on the power iteration, and those in Section 4.2 are based on the EKFAC method.

Original: In this section, we explain some details of these approximations.
Revised: In this section, we explain some details of these approximations.

Original: G.5.1 The power iteration
Revised: G.5.1 The power iteration

Original: In Section 4.1, we utilize the functions of the PyHessian library [48], which are based on the power iteration (see Algorithm 2 in [48]), to obtain the top few eigenvalues/eigenvectors of the Hessian, i.e., λ k and ∆ θ k in (39).
Revised: In Section 4.1, we utilize the functions of the PyHessian library [48], which are based on the power iteration (see Algorithm 2 in [48]), to obtain the top few eigenvalues/eigenvectors of the Hessian, i.e., λ k and ∆ θ k in (39).

Original: (Note that we can consider the eigenvalues/eigenvectors of the Hessian here since we can approximate the Hessian to the FIM.)
Revised: (Note that we can consider the eigenvalues/eigenvectors of the Hessian here since we can approximate the Hessian to the FIM.)

Original: In Section 3.3.1 and Appendix G.2, to obtain the eigenvalues/eigenvectors of the FIM we first subsample N s data points { x 1 , . . .
Revised: In Section 3.3.1 and Appendix G.2, to obtain the eigenvalues/eigenvectors of the FIM we first subsample N s data points { x 1 , . . .

Original: , x N s } from the data set.
Revised: , x N s } from the data set.

Original: We then calculate the Jacobian of the loss for the sampled labels with respect to the parameters, i.e., ˜ J = ∂ ˜ l∂θ ∈ R N s × m , where ˜ l
Revised: We then calculate the Jacobian of the loss for the sampled labels with respect to the parameters, i.e., ˜ J = ∂ ˜ l∂θ ∈ R N s × m , where ˜ l

Original: =
Revised: =

Original: ( l (˜ y 1 , f ( x 1 ; θ )) , . . .
Revised: ( l (˜ y 1 , f ( x 1 ; θ )) , . . .

Original: , l (˜ y N s , f ( x N s ; θ )))
Revised: , l (˜ y N s , f ( x N s ; θ )))

Original: ∈ R N s and the labels ˜
Revised: ∈ R N s and the labels ˜

Original: y i are sampled according to the probabilistic model p ( y | x i ; θ ) that the neural network models for i = 1 , . .
Revised: y i are sampled according to the probabilistic model p ( y | x i ; θ ) that the neural network models for i = 1 , . .

Original: , N s as discussed in [20].
Revised: , N s as discussed in [20].

Original: The FIM can then be approximated as F ( θ )
Revised: The FIM can then be approximated as F ( θ )

Original: ≈ 1 N s ˜
Revised: ≈ 1 N s ˜

Original: J ⊤ ˜ J , and its top- m ′ eigenvalues/eigenvectors are numerically obtained via the power iteration in Algorithm 1.
Revised: J ⊤ ˜ J , and its top- m ′ eigenvalues/eigenvectors are numerically obtained via the power iteration in Algorithm 1.

Original: Note that the time complexity of the multiplication F ( θ ) v ≈ 1 N s ˜
Revised: Note that the time complexity of the multiplication F ( θ ) v ≈ 1 N s ˜

Original: J ⊤ ( ˜ Jv ) in line 5 of Algorithm 1 is O ( N s m 2 ) , where v ∈ R m is an approximation of the eigenvector during the iteration.
Revised: J ⊤ ( ˜ Jv ) in line 5 of Algorithm 1 is O ( N s m 2 ) , where v ∈ R m is an approximation of the eigenvector during the iteration.

Original: Algorithm
Revised: Algorithm

Original: Input: The Jacobian of the loss for sampled labels with respect to parameters ˜ J = ∂ ˜ l∂θ ∈ R N s × m , the dimension of top eigenvalues/eigenvectors to compute m ′ , the number of iterations N iter .
Revised: Input: The Jacobian of the loss for sampled labels with respect to parameters ˜ J = ∂ ˜ l∂θ ∈ R N s × m , the dimension of top eigenvalues/eigenvectors to compute m ′ , the number of iterations N iter .

Original: Initialize:
Revised: Initialize:

Original: The top eigenvalue vector λ = 0 ∈ R m ′ , the top eigenvector matrix V = 

Revised: The top eigenvalue vector λ = 0 ∈ R m ′ , the top eigenvector matrix V =

Original: ∈
Revised: ∈

Original: R m × m ′ .
Revised: R m × m ′ .

Original: Iteration: 1: for k = 1 , . . .
Revised: Iteration: 1: for k = 1 , . . .

Original: , m ′ do 2:
Revised: , m ′ do 2:

Original: Draw a random vector v ∈ R m from N (0 , I ) .
Revised: Draw a random vector v ∈ R m from N (0 , I ) .

Original: 3: for i = 1 , . . .
Revised: 3: for i = 1 , . . .

Original: , N iter do 4:
Revised: , N iter do 4:

Original: Project v to the orthogonal complement of V and normalize, v = ( I − V V ⊤ ) v ∥ ( I − V V ⊤ ) v ∥ .
Revised: Project v to the orthogonal complement of V and normalize, v = ( I − V V ⊤ ) v ∥ ( I − V V ⊤ ) v ∥ .

Original: Compute F ( θ ) v
Revised: Compute F ( θ ) v

Original: ≈ 1 N s ˜ J ⊤ ( ˜ Jv ) and reset v , v = F ( θ ) v .
Revised: ≈ 1 N s ˜ J ⊤ ( ˜ Jv ) and reset v , v = F ( θ ) v .

Original: 6: end for 7:
Revised: 6: end for 7:

Original: Project v to the orthogonal complement of V and normalize, v = ( I − V V ⊤ ) v ∥ ( I − V V ⊤ ) v ∥ .
Revised: Project v to the orthogonal complement of V and normalize, v = ( I − V V ⊤ ) v ∥ ( I − V V ⊤ ) v ∥ .

Original: Update the top eigenvector matrix V , set V k
Revised: Update the top eigenvector matrix V , set V k

Original: = v ∈ R m , where V k is the k -th column of V .
Revised: = v ∈ R m , where V k is the k -th column of V .

Original: Update the top eigenvalue vector λ , set λ k = v ⊤ F ( θ ) v , where λ k is the k -th element of λ .
Revised: Update the top eigenvalue vector λ , set λ k = v ⊤ F ( θ ) v , where λ k is the k -th element of λ .

Original: 10: end for
Revised: 10: end for

Original: Output
Revised: Output

Original: In implementing the power iteration, we handle matrices of size N s × m with m as the neural network parameter size.
Revised: In implementing the power iteration, we handle matrices of size N s × m with m as the neural network parameter size.

Original: Since memory usage is proportional to the network size, additional approximations may be needed to handle larger neural networks.
Revised: Since memory usage is proportional to the network size, additional approximations may be needed to handle larger neural networks.

Original: G.5.2 The EKFAC method
Revised: G.5.2 The EKFAC method

Original: The Kronecker factorization (KFAC) method is developed to efficiently implement the natural gradient descent algorithm [29].
Revised: The Kronecker factorization (KFAC) method is developed to efficiently implement the natural gradient descent algorithm [29].

Original: The method approximates the FIM as layer-wise block-diagonal and efficiently performs eigendecomposition via Kronecker factorization for each block.
Revised: The method approximates the FIM as layer-wise block-diagonal and efficiently performs eigendecomposition via Kronecker factorization for each block.

Original: The eigenvaluecorrected KFAC (EKFAC) method additionally performs an eigenvalue correction step to make the decomposition based on the Kronecker-factored eigenbasis closer to the FIM [13].
Revised: The eigenvaluecorrected KFAC (EKFAC) method additionally performs an eigenvalue correction step to make the decomposition based on the Kronecker-factored eigenbasis closer to the FIM [13].

Original: (Compared to the power iteration, these methods are based on stronger approximations and may be more inaccurate, but we can apply them to larger neural networks.)
Revised: (Compared to the power iteration, these methods are based on stronger approximations and may be more inaccurate, but we can apply them to larger neural networks.)

Original: The time complexity of the EKFAC method (for obtaining δ = ρF ( θ ) † (cid:0) ∂l∂θ (cid:1) ⊤ in (13)) becomes the addition of that for computing gradient (cid:0) ∂l∂θ (cid:

Revised: The time complexity of the EKFAC method (for obtaining δ = ρF ( θ ) † (cid:0) ∂l∂θ (cid:1) ⊤ in (13)) becomes the addition of that for computing gradient (cid:0) ∂l∂θ (cid:

Original: ) ⊤ , that for solving layer-wise eigenvalue problems to obtain F ( θ ) † , and that for the multiplication of F ( θ ) † with (cid:0) ∂l∂θ (cid:1) ⊤ .
Revised: ) ⊤ , that for solving layer-wise eigenvalue problems to obtain F ( θ ) † , and that for the multiplication of F ( θ ) † with (cid:0) ∂l∂θ (cid:1) ⊤ .

Original: Due to the Kronecker factorization, the time complexity for solving the eigenvalue problems per layer becomes O ( d 3 ) , where d denotes the typical layer input and output node sizes.
Revised: Due to the Kronecker factorization, the time complexity for solving the eigenvalue problems per layer becomes O ( d 3 ) , where d denotes the typical layer input and output node sizes.

Original: (Note that in the EKFAC method, the eigenvalue problem is usually solved per several tens or hundreds of iteration steps.)
Revised: (Note that in the EKFAC method, the eigenvalue problem is usually solved per several tens or hundreds of iteration steps.)

Original: After that, F ( θ ) † is not obtained explicitly but considered in the decomposed form, i.e., F ( θ ) † = US − 1 U ⊤ for the eigendecomposition of F ( θ ) =
Revised: After that, F ( θ ) † is not obtained explicitly but considered in the decomposed form, i.e., F ( θ ) † = US − 1 U ⊤ for the eigendecomposition of F ( θ ) =

Original: USU ⊤ .
Revised: USU ⊤ .

Original: The multiplication of F ( θ ) † with (cid:0) ∂l∂θ (cid:1) ⊤ is done layerwisely and efficiently by using the mixed Kronecker matrix-vector product with complexity O ( d 3 ) per layer [13].
Revised: The multiplication of F ( θ ) † with (cid:0) ∂l∂θ (cid:1) ⊤ is done layerwisely and efficiently by using the mixed Kronecker matrix-vector product with complexity O ( d 3 ) per layer [13].

Original: The final complexity for the EKFAC method is O ( bLd 2 )
Revised: The final complexity for the EKFAC method is O ( bLd 2 )

Original: + O ( Ld 3 ) , where O ( bLd 2 ) is that for computing gradient (hence that for SGD), L is the number of layers, and b is the mini-batch size.
Revised: + O ( Ld 3 ) , where O ( bLd 2 ) is that for computing gradient (hence that for SGD), L is the number of layers, and b is the mini-batch size.

Original: For a more detailed analysis of the coefficients of each term, we refer the reader to Section 8 of [29].
Revised: For a more detailed analysis of the coefficients of each term, we refer the reader to Section 8 of [29].