Original: D EEP H IERARCHICAL -H YPERSPHERICAL L EARNING (DH 2 L)
Revised: D EEP H IERARCHICAL -H YPERSPHERICAL L EARNING (DH 2 L)

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Regularization is known to be an inexpensive and reasonable solution to alleviate over-ﬁtting problems arising in inference models, including deep neural networks.
Revised: Regularization is known to be an inexpensive and reasonable solution to alleviate the overﬁtting problem arising in inference models, including deep neural networks.

Original: In this paper, we proposea hierarchical regularization which preserves the semantic structure of a sample distribution.
Revised: In this paper, we propose hierarchical regularization which preserves the semantic structure of a sample distribution.

Original: At the same time, this regularization promotes diversity by imposing  distance between parameter vectors enlarged within semantic structures.
Revised: At the same time, this regularization promotes diversity by imposing a distance between parameter vectors enlarged within semantic structures.

Original: To generate evenly distributed parameters which is considered less redundant, we constrain them to lie on hierarchical hyperspheres .
Revised: To generate evenly distributed parameters which is considered less redundant, we constrain them to lie on hierarchical hyperspheres .

Original: To deﬁne a hierarchical parameter space, we propose to reformulate the topology space with multiple hyperspheres.
Revised: To deﬁne a hierarchical parameter space, we propose to reformulate the topology space with multiple hyperspheres.

Original: On each hypersphere, a projection is parameterized by two individual parameters.
Revised: On each hypersphere, a projection is parameterized by two individual parameters.

Original: Since maximizing a groupwise and pairwise distance between points on hypersphere is nontrivial (generalized Thomson problem), we propose a new discrete metric integrated with  continuousangular metric.
Revised: Since maximizing a groupwise and pairwise distance between points on hypersphere is nontrivial (generalized Thomson problem), we propose a new discrete metric integrated with a continuous metric.

Original: The proposed method shows improved generalization performance on extensive experiments using publicly available datasets (CIFAR-10, CIFAR-100, CUB2002011, and Stanford Cars), especially when the number of super-classes is large.
Revised: The proposed method shows improved generalization performance on extensive experiments using publicly available datasets (CIFAR-10, CIFAR-100, CUB200-2011, and Stanford Cars), especially when the number of super-classes is large.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Diversity promoting learning has been widely adopted via enlarging pairwise distances (Xie et al., 2018; 2017a;
Revised: Diversity promoting learning has been widely adopted via enlarging pairwise distances (Xie et al., 2018; 2017a;

Original: Liu et al., 2018), increasing orthogonality (Xie et al., 2018), reducing covariance between parameters (Xie et al., 2017b), or reducing correlation on feature (Cogswell et al., 2016) to improve generalization performance.
Revised: Liu et al., 2018), increasing orthogonality (Xie et al., 2018), reducing covariance between parameters (Xie et al., 2017b), or reducing correlation on feature (Cogswell et al., 2016) to improve generalization performance.

Original: Among them, diversity promoting regularization (Xie et al., 2017a;b; Liu et al., 2018) by enforcing large diversity between projection parameters achieves a reasonable performance without modifying the model structure.
Revised: Among them, diversity promoting regularization (Xie et al., 2017a;b; Liu et al., 2018) by enforcing large diversity between projection parameters achieves a reasonable performance without modifying the model structure.

Original: While optimizing the objective function with a covariance matrix in (Xie et al., 2017b; 2018) is nontrivial, the diversity promoting regularization via minimizing energy of parameters of deep neural networks has been proposed (Liu et al., 2018) in a simpler way.
Revised: While optimizing the objective function with a covariance matrix in (Xie et al., 2017b; 2018) is nontrivial, the diversity promoting regularization via minimizing energy of parameters of deep neural networks has been proposed (Liu et al., 2018) in a simpler way.

Original: By minimizing a pairwise distance between parameters on a hypersphere with the known metric, they achieved the improved generalization performance.
Revised: By minimizing a pairwise distance between parameters on a hypersphere with the known metric, they achieved the improved generalization performance.

Original: Following an efﬁcient regularization on the hypersphere, we explore further this direction with three main concepts (hierarchical and hyperspherical learning with discrete metrics).
Revised: Following an efﬁcient regularization on the hypersphere we explore further this direction with three main concepts (hierarchical and hyperspherical learning with discrete metrics).

Original: 1) Why hierarchical learning?
Revised: 1) Why hierarchical learning?

Original: Hierarchical inference explains human intelligence .
Revised: Hierarchical inference explains human intelligence .

Original: In (Kurzweil, 2013), it states that “the neocortex contains about 300 million very general pattern recognizers, arranged in a hierarchy”.
Revised: In (Kurzweil, 2013), it states that “the neocortex contains about 300 million very general pattern recognizers, arranged in a hierarchy”.

Original: Applying the hierarchy of multiple classes based on semantic taxonomy is a natural choice to devise machine intelligence .
Revised: Applying the hierarchy of multiple classes based on semantic taxonomy is a natural choice to devise machine intelligence .

Original: Effectiveness of the hierarchical learning can be found in (Verma et al., 2012).
Revised: Effectiveness of the hierarchical learning can be found in (Verma et al., 2012).

Original: 2) Why hyperspherical learning?
Revised: 2) Why hyperspherical learning?

Original: Hypersphere is the set of points at the equidistance (radius) from a given point (centroid) in a certain dimensional space.
Revised: Hypersphere is the set of points at the equidistance (radius) from a given point (centroid) in a certain dimensional space.

Original: Due to the denominator in the unit-length normalization ( w (cid:107) w (cid:107) , w ∈ R d +1 ), the angular distance deﬁned on the hypersphere converges when the magnitude of w goes inﬁnity while Euclidean distance goes inﬁnity.
Revised: Due to the denominator in the unit-length normalization ( w (cid:107) w (cid:107) , w ∈ R d +1 ), the angular distance deﬁned on the hypersphere converges when the magnitude of w goes inﬁnity while Euclidean distance goes inﬁnity.

Original: Due to this bounded property, a hierarchical structure with multiple separated hyperspheres can be deﬁned.
Revised: Due to this bounded property, a hierarchical structure with multiple separated hyperspheres can be deﬁned.

Original: 3) Why discrete metric learning?
Revised: 3) Why discrete metric learning?

Original: If vector points form discontinuous series with discrete representation (e.g. multi-dimensional binary or ternary), they are isolated from each other with a certain margin.
Revised: If vector points form discontinuous series with discrete representation (e.g. multi-dimensional binary or ternary), they are isolated from each other with a certain margin.

Original: This property may ﬁt with a disconnected/groupwise manifold space problem.
Revised: This property may ﬁt with a disconnected/groupwise manifold space problem.

Original: We note that making points to be equidistributed where a pairwise distance is maximized is a nontrivial task.
Revised: We note that making points to be equidistributed where a pairwise distance is maximized is a nontrivial task.

Original: As
Revised: As

Original: points in a discrete metric space are ﬁnite and isolated, search efforts to satisfy those constraints may become easy.
Revised: points in a discrete metric space are ﬁnite and isolated, search efforts to satisfy those constraints can be reduced.

Original: In this paper, we propose to apply a hierarchical structure to parameter regularization on the multiple groupwise hyperspheres.
Revised: In this paper, we propose to apply a hierarchical structure to parameter regularization on the multiple groupwise hyperspheres.

Original: In order to ﬁnd an appropriate metric in this space, we explore a discrete angular metric.
Revised: In order to ﬁnd an appropriate metric in this space, we explore a discrete angular metric.

Original: We examine the proposed method on extensive experimental setups in terms of datasets and deep network models.
Revised: We examine the proposed method on extensive experimental setups in terms of datasets and deep network models.

Original:  M ULTIPLE SEPARATED HYPERSPHERES
Revised:  M ULTIPLE SEPARATED HYPERSPHERES

Original: Samples observed from the real world may be on disconnected manifolds.
Revised: Samples observed from the real world may be on disconnected manifolds.

Original: In other words, the disjoint union of those manifolds could generate the global manifold (Lee, 2000).
Revised: In other words, the disjoint union of those manifolds can generate the global manifold (Lee, 2000).

Original: In this section, we decompose the one space into multiple spaces (manifolds) and re-deﬁne the space interms of a hierarchical point of view.
Revised: In this section, we decompose the one space into multiple spaces (manifolds) and re-deﬁne the spaces in a hierarchical point of view.

Original: 2. D ISCONNECTED MANIFOLD VIA EQUIVALENT RELATIONS
Revised: 2. D ISCONNECTED MANIFOLD VIA EQUIVALENT RELATIONS

Original: Since it is not suitable to measure a pairwise distance between high dimensional vectors which have the hierarchical structure in the same space, we construct another identiﬁcation space which includes isolatedspaces, from the original space (via equivalence relation (Tu, 2010)).
Revised: Since it is not suitable to measure a pairwise distance between high dimensional vectors which have the hierarchical structure in the same space, we construct another identiﬁcation space where subspaces (sub-manifolds) are isolated (via equivalence relation (Tu, 2010)).

Original: Denote d -sphere S d to be the set of points that satisﬁes S d = {
Revised: Denote d -sphere S d to be the set of points that satisﬁes S d = {

Original: w ∈ R d +1 : (cid:107) w (cid:107) = 1 } .
Revised: w ∈ R d +1 : (cid:107) w (cid:107) = 1 } .

Original: We construct multiple separated hyperspheres using multiple identifying relations.
Revised: We construct multiple separated hyperspheres using multiple identifying relations.

Original: In Figure 1, we use a center vector w c and a surface vector w s to deﬁne ahypershpere space and a projection parameter  w 
Revised: In Figure 1, we use a center parameter and a surface parameter to deﬁne a projection parameter vector w which constitutes a hypershpere space.

Original: . 2. P RIOR DISTRIBUTION AND REGULARIZATION
Revised: 2. P RIOR DISTRIBUTION AND REGULARIZATION

Original: To make parameter vectors uniformly distributed on the unit hypersphere, parameters are sampled from the Gaussian normal distribution (Muller, 1959; Harman & Lacko, 2010).
Revised: To make parameter vectors uniformly distributed on the unit hypersphere, parameters are sampled from the Gaussian normal distribution (Muller, 1959; Harman & Lacko, 2010).

Original: This is because the normal distribution is spherically symmetric (Muller, 1959).
Revised: This is because the normal distribution is spherically symmetric (Muller, 1959).

Original: In a Bayesian point of view, neural networks with Gaussian priors are known to induce l 2 -norm regularization (Vladimirova et al., 2019).
Revised: In a Bayesian point of view, neural networks with Gaussian priors are known to induce l 2 -norm regularization (Vladimirova et al., 2019).

Original: From two evidences, we know that enforcing parameters to have the Gaussian prior is important for hyperspherical learning in neural networks.
Revised: From two evidences, we know that enforcing parameters to have the Gaussian prior is important for hyperspherical learning in neural networks.

Original: Note that a parameter which is calculated from the difference arithmetic operation with two parameters on the normal Gaussian distribution is on the normal difference distribution .
Revised: Note that a parameter which is calculated from the difference arithmetic operation with two parameters on the normal Gaussian distribution is on the normal difference distribution .

Original:  M ETHOD
Revised:  M ETHOD

Original: In deep neural networks, the objective function J with regularization R in addition to a loss L ,
Revised: In deep neural networks, the objective function J with regularization R in addition to a loss L ,

Original: J R ( W ) =
Revised: J R ( W ) =

Original: L ( x , W ) + λ R ( W ) , is optimized to ﬁnd the optimal W having a near minimum loss L , arg min W J R ( x , W ) , where x ∈ R d 0 denotes an input vector,
Revised: L ( x , W ) + λ R ( W ) , is optimized to ﬁnd the optimal W having a near minimum loss L , arg min W J R ( x , W ) , where x ∈ R d 0 denotes an input vector,

Original: W = {
Revised: W = {

Original: W i ∈ R
Revised: W i ∈ R

Original: d i − 1 × c
Revised: d i − 1 × c

Original: i : W i = { w j ∈
Revised: i : W i = { w j ∈

Original: R d i − 1 } , j = 1 , . . .
Revised: R d i − 1 } , j = 1 , . . .

Original: c i , i = 1 , . . .
Revised: c i , i = 1 , . . .

Original: , L } denotes a set of parameter matrices (i.e. neurons/kernels),
Revised: , L } denotes a set of parameter matrices (i.e. neurons/kernels),

Original: L denotes the number of layers, and λ > 0 is to control the degree of the regularization.
Revised: L denotes the number of layers, and λ > 0 is to control the degree of the regularization.

Original: For a classiﬁcation task, the cross entropy loss is used for the loss function L .
Revised: For a classiﬁcation task, the cross entropy loss is used for the loss function L .

Original: We propose a new regularization formulation R preserving a hierarchical parameter structure in Section 3.1.
Revised: We propose a new regularization formulation R preserving a hierarchical parameter structure in Section 3.1.

Original: 3. R EGULARIZATION FOR H IERARCHICAL AND H YPERSPHERICAL H YPOTHESES
Revised: 3. R EGULARIZATION FOR H IERARCHICAL AND H YPERSPHERICAL H YPOTHESES

Original: Denote w a parameter vector (called a projection parameter, an element of W at a certain layer) to transform a given input into an embedding space deﬁned in a Euclidean metric space: x ∈ R d +1 (cid:55)→ w T x ∈ R .
Revised: Denote w a parameter vector (called a projection parameter, an element of W at a certain layer) to transform a given input into an embedding space deﬁned in a Euclidean metric space: x ∈ R d +1 (cid:55)→ w T x ∈ R .

Original: By the deﬁnition of unit-length projection w (cid:107) w (cid:107) , a new parameter vector ˆ w can be deﬁned on d -sphere: S d = { ˆ w ∈ R d +1 : (cid:107) ˆ w (cid:107)
Revised: By the deﬁnition of unit-length projection w (cid:107) w (cid:107) , a new parameter vector ˆ w can be deﬁned on d -sphere: S d = { ˆ w ∈ R d +1 : (cid:107) ˆ w (cid:107)

Original: = 1 } where (cid:107) · (cid:107) denotes l 2 -norm and a center is zero.
Revised: = 1 } where (cid:107) · (cid:107) denotes l 2 -norm and a center is zero.

Original: In other word the parameter ˆ w can be deﬁned by a center parameter vector w c ∈ R d +1 and a surface parameter vector w s ∈ R d
Revised: In other word the parameter ˆ w can be deﬁned by a center parameter vector w c ∈ R d +1 and a surface parameter vector w s ∈ R d

Original: +1 using an arithmetic operation:
Revised: +1 using an arithmetic operation:

Original: ˆ w := w s − w c .
Revised: ˆ w := w s − w c .

Original: We deﬁne the d -sphere with  center and surface parameters: S d w c = { w s − w c ∈ R d
Revised: We deﬁne the d -sphere with the center and surface parameters: S d w c = { w s − w c ∈ R d

Original: +1 : (cid:107) w s − w c (cid:107) = 1 } .
Revised: +1 : (cid:107) w s − w c (cid:107) = 1 } .

Original: For a notation simplicity, we use w instead of ˆ w hereafter.
Revised: For a notation simplicity, we use w instead of ˆ w hereafter.

Original: While we consider a radius equals to 1 for simplicity, the parameter vector can have a radius r > 0 .
Revised: While we consider a radius equals to 1 for simplicity, the parameter vector can have a radius r > 0 .

Original: 3.1. H IERARCHICAL PARAMETERIZATION VIA LEVELWISE AND GROUPWISE PARAMETERS
Revised: 3.1. H IERARCHICAL PARAMETERIZATION VIA LEVELWISE AND GROUPWISE PARAMETERS

Original: We assume that the hierarchical structure consists of a levelwise structure with a notation ( l ) and a groupwise structure with a notation g below.
Revised: We assume that the hierarchical structure consists of a levelwise structure with a notation ( l ) and a groupwise structure with a notation g below.

Original: Through deﬁning these structures serially, we explain hierarchical parameters.
Revised: Through deﬁning these structures serially, we explain hierarchical parameters.

Original: Levelwise structure
Revised: Levelwise structure

Original: The projection parameter w ∈ R d +1 in S d w c can be deﬁned with a levelwise notation ( l ) as follows,
Revised: The projection parameter w ∈ R d +1 on S d w c can be deﬁned with a levelwise notation ( l ) as follows,

Original: [Equation]
Revised: [Equation]

Original: where S d w ( l ) c denotes the l -th level d -sphere centered at w ( l ) c .
Revised: where S d w ( l ) c denotes the l -th level d -sphere centered at w ( l ) c .

Original: In this paper, we deﬁne the parameter in a higher dimensional space than that shown in Figure 2.
Revised: In this paper, we deﬁne the parameter in a higher dimensional space than that shown in Figure 2.

Original: In a levelwise setting, w ( l ) s and w ( l ) c at the current level ( l ) are additively represented based on a center parameter of the previous level ( l − 1) : w ( l − 1) c
Revised: In a levelwise setting, w ( l ) s and w ( l ) c at the current level ( l ) are additively represented based on a center parameter of the previous level ( l − 1) : w ( l − 1) c

Original: [Equation]
Revised: [Equation]

Original: l ) c , where w
Revised: w

Original: [Equation]
Revised: [Equation]

Original: l  − 1) c 
Revised: l ) c , where w ( l − 1) c =

Original: [Equation]
Revised: [Equation]

Original: w
Revised: w

Original: [Equation]
Revised: [Equation]

Original: is the accumulated and −−→ ∆ w ( l ) denotes a connecting parameter from w ( l − 1) c to w ( l ) c .
Revised: is the accumulated and −−→ ∆ w ( l ) denotes a connecting parameter from w ( l − 1) c to w ( l ) c .

Original: By denoting −−→ ∆ w ( l ) as w ( l,l − 1) , the center vector at the l -th level is deﬁned as, w ( l ) c :=
Revised: By denoting −−→ ∆ w ( l ) as w ( l,l − 1) , the center vector at the l -th level is deﬁned as, w ( l ) c :=

Original: w ( l,l − 1) c
Revised: w ( l,l − 1) c

Original: + w ( l − 1) c , and the surface vector as w ( l ) s := w ( l,l − 1)
Revised: + w ( l − 1) c , and the surface vector as w ( l ) s := w ( l,l − 1)

Original: s + w ( l − 1) c .
Revised: s + w ( l − 1) c .

Original: Both the center parameter and the surface parameter at the current level are based on the center parameter at the previous level 1 .
Revised: Both the center parameter and the surface parameter at the current level are based on the center parameter at the previous level 1 .

Original: Hence, Eq. 1 is equivalent to
Revised: Hence, Eq. 1 is equivalent to

Original: [Equation]
Revised: [Equation]

Original: Note that we use a superscript ( l, l − 1) to denote parameters at the l -th level connected from the center parameter at the ( l − 1) -th level.
Revised: Note that we use a superscript ( l, l − 1) to denote parameters at the l -th level connected from the center parameter at the ( l − 1) -th level.

Original: Groupwise structure With a group notation g k , the center parameter in Eq.
Revised: Groupwise structure With a group notation g k , the center parameter in Eq.

Original: 2 can be rephrased as w ( l,l − 1) c,g k , on S d w ( l,l − 1) c,gk which denotes d -sphere of g k group at the l -th level.
Revised: 2 can be rephrased as w ( l,l − 1) c,g k , on S d w ( l,l − 1) c,gk which denotes d -sphere of g k group at the l -th level.

Original: Each group constitutes a group set at the l -th level g ( l ) :
Revised: Each group constitutes a group set at the l -th level g ( l ) :

Original: = { g k } | g ( l ) | k
Revised: = { g k } | g ( l ) | k

Original: =1 where g ( l ) ⊆
Revised: =1 where g ( l ) ⊆

Original: G ( l ) , G ( l ) denotes a batch group set at the l -th level, 
Revised: G ( l ) , G ( l ) denotes a batch group set at the l -th level, and | · | denotes the cardinality.

Original: The group set g  l )  the l -th level is conditioned on a group set at the ( l − 1) -th level, g ( l − 1) :
Revised: The group set g ( l ) at the l -th level is conditioned on a group set at the ( l − 1) -th level, g ( l − 1) :

Original: = { g k (cid:48) } | g ( l − 1) | k (cid:48)
Revised: = { g k (cid:48) } | g ( l − 1) | k (cid:48)

Original: = where g ( l − 1) ⊆
Revised: = where g ( l − 1) ⊆

Original: G ( l − 1) .
Revised: G ( l − 1) .

Original: By their groupwise relation over levels, an adjacency indication 2 P ( l,l − 1) ( { G ( l − 1) , G ( l ) } ) ∈ { 0 , 1 } | G ( l − 1) |×| G ( l ) | is given.
Revised: By their groupwise relation over levels, an adjacency indication 2 P ( l,l − 1) ( { G ( l − 1) , G ( l ) } ) ∈ { 0 , 1 } | G ( l − 1) |×| G ( l ) | is given.

Original: Hence, the i -th parameter vector in S d w ( l,l − 1) c,gk is deﬁned as follows,
Revised: Hence, the i -th parameter vector on S d w ( l,l − 1) c,gk is deﬁned as follows,

Original: [Equation]
Revised: [Equation]

Original: where { w ( l,l − 1) s,g k ,i , w ( l,l − 1) c,g k } ∀ i is calculated based on a center vector w ( l − 1) c,g k (cid:48) at the ( l − 1) -th level 3 , i = 1 , . .
Revised: where { w ( l,l − 1) s,g k ,i , w ( l,l − 1) c,g k } ∀ i is calculated based on a center vector w ( l − 1) c,g k (cid:48) at the ( l − 1) -th level 3 , i = 1 , . .

Original: , | g k | , and | g k | denotes the number of surface parameter vectors in the group g k .
Revised: , | g k | , and | g k | denotes the number of surface parameter vectors in the group g k .

Original:  H IERARCHICAL REGULARIZATION
Revised: 3.1. H IERARCHICAL REGULARIZATION

Original: In this section, we deﬁne a regularization term using the hierarchical parameters deﬁned above.
Revised: In this section, we deﬁne a regularization term using the hierarchical parameters deﬁned above.

Original: A set of parameters { W ( l,l − 1) s,g k [Equation] c [Equation]
Revised: A set of parameters { W ( l,l − 1) s,g k , w ( l,l − 1) c,g k , w ( l − 1) c,g (cid:48)

Original:  g[Equation] k 
Revised: k } ∈ W ∀ g k , ∀ g k (cid:48) where W ( l,l − 1) s,g k :

Original: [Equation]
Revised: [Equation]

Original: where W ( l,l −
Revised: { w ( l

Original: [Equation]
Revised: [Equation]

Original: s [Equation] g k 
Revised: l − 1) s,g k ,i } | g k | i =1 , is an optimizing target of hierarchical regularization as follows:

Original: [Equation]
Revised: [Equation]

Original: is an optimizing target of hierarchical regularization as follows:

Original: [Equation]
Revised: [Equation]

Original: where R l works on individual spheres S d w ( l,l − 1) c,gk , λ l > 0 , and C l aims to apply geometry-aware constraints across spheres.
Revised: where R l works on individual spheres S d w ( l,l − 1) c,gk , λ l > 0 , and C l aims to apply geometry-aware constraints across spheres.

Original: R l consists of two terms for regularization: 1) R l,p for projection parameters in the same group g k on S d w ( l,l − 1) c,gk and 2) R l,c for center parameters across groups in the same level on S d w( l −
Revised: R l consists of two terms for regularization: 1) R l,p for projection parameters in the same group g k on S d w ( l,l − 1) c,gk and 2) R l,c for center parameters across groups in the same level on S d w

Original: [Equation]
Revised: [Equation]

Original: c,g
Revised: where R l,p

Original: [Equation]
Revised: [Equation]

Original: [Equation] , (5) whereR l,p ( W ( l,l − 1) s,g k , w ( l,l − 1) c,g k ) : =
Revised: where w ( l,l − 1) g k ,i

Original: [Equation]
Revised: [Equation]

Original: G (cid:88)
Revised: , ( w

Original: [Equation]
Revised: [Equation]

Original: g k
Revised: l,l

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: i [Equation] [Equation] , w ( l,l
Revised: is similarly deﬁned), G = |{ g k ∈ g ( l ) }| ,

Original: [Equation] 1) c,g j ) , (7) where w ( l,l − 1) g k ,i := w ( l,l − 1) s,g k ,i − w ( l,l − 1) c,g k , ( w ( l,l − 1) g k ,j is similarly deﬁned), G = |{ i (cid:54) = j ∈ g k }| , G k
Revised: G k

Original: = |{ i (cid:54) =
Revised: = |{ i (cid:54) =

Original: j ∈ g k }| , C = |{ g i (cid:54) =
Revised: j ∈ g k }| , C = |{ g i (cid:54) =

Original: g j ∈ g ( l ) }| , and d ( · , · ) denotes a distance metric between parameters.
Revised: g j ∈ g ( l ) }| , and d ( · , · ) denotes a distance metric between parameters.

Original: The distance metric is deﬁned in Section 3.2.
Revised: The distance metric is deﬁned in Section 3.2.

Original: We explain regularization terms formulated using these parameters and pairwise distances  in Figure 4 of Appendix.
Revised: We explain regularization terms formulated using these parameters and pairwise distances further in Figure 4 of Appendix.

Original: When a minibatch of inputs ( m x : a set of inputs { x i } ) is given, the regularization term becomes: E( R
Revised: When a minibatch of inputs ( m x : a set of inputs { x i } ) is given, the regularization term becomes: E

Original: [Equation]
Revised: [Equation]

Original: The constraint term help construct geometry-aware relational parameters between different spheres on the same level and on the across levels.
Revised: The constraint term help construct geometry-aware relational parameters between different spheres on the same level and on the across levels.

Original: Multiple constraints are deﬁned as C l := (cid:80) k λ k C l,k , where C l,k is the k -th constraint between parameters at the l -th and the ( l − 1) -th level, and λ k >is a Lagrange multiplier.
Revised: Multiple constraints are deﬁned as C l := (cid:80) k λ k C l,k , where C l,k is the k -th constraint between parameters at the l -th and the ( l − 1) -th level, and λ k >is a Lagrange multiplier.

Original: We propose three constraints (deﬁned in Appendix) in a geometric point of view.
Revised: We propose three constraints (deﬁned in Appendix) in a geometric point of view.

Original: 3. D ISCRETE AND CONTINUOUS ANGULAR DISTANCE METRIC
Revised: 3. D ISCRETE AND CONTINUOUS ANGULAR DISTANCE METRIC

Original: Discrete metric is a good ﬁt for the above groupwise deﬁnition.
Revised: Discrete metric is a good ﬁt for the above groupwise deﬁnition.

Original: We expect that points projected by parameters in a discrete metric space, are isolated each other.
Revised: We expect that points projected by parameters in a discrete metric space are isolated each other.

Original: In Figure 3, we show that discrete 
Revised: In Figure 3, we show that discrete distances of parameter pairs have different values while their continuous distances are same. Hence, maximization of discrete distances of parameter pairs could help parameters distributed isolatedly and diversely.

Original:  | g k | i w ( l ) s,g k ,i .
Revised: g k | (cid:80) | g k | i w ( l ) s,g k ,i .

Original: The center vector for the group g k can be determined by a parameter vector at the ( l − 1) − th level using an adjust factor ( (cid:15) , | (cid:15) | < 1 ): w ( l,l − 1) c,g k = (cid:15) · w ( l − 1) g k (cid:48) ,i , where w ( l − 1) g k (cid:48) ,
Revised: The center vector for the group g k can be determined by a parameter vector at the ( l − 1) − th level using an adjust factor ( (cid:15) , | (cid:15) | < 1 ): w ( l,l − 1) c,g k = (cid:15) · w ( l − 1) g k (cid:48) ,i , where w ( l − 1) g k (cid:48) ,

Original: i ∈ S d w ( l − 1) c,g (cid:48) k .
Revised: i ∈ S d w ( l − 1) c,g (cid:48) k .

Original: distances of parameter pairs have different values while their continuous distances are same.

Original: Hence, maximization of discrete distances of parameter pairs could help parameters distributed isolatedly and diversely.

Original: Using parameter vectors w i and w j in R d
Revised: Using parameter vectors w i and w j in R d

Original: +1 , we deﬁne a discrete distance metric using a sign function as follows:
Revised: +1 , we deﬁne a discrete distance metric using a sign function as follows:

Original: [Equation]
Revised: [Equation]

Original: where sign ( x ) := (cid:110) 1 , if x ≥ 0 − 1 , otherwise , − 1 ≤ D h ≤ 1 , and w = { w ( k )
Revised: where sign ( x ) := (cid:110) 1 , if x ≥ 0 − 1 , otherwise , − 1 ≤ D h ≤ 1 , and w = { w ( k )

Original: | ∀ k
Revised: | ∀ k

Original: = 1 , . .
Revised: = 1 , . .

Original: , d + 1 } ∈ R d +1 } .
Revised: , d + 1 } ∈ R d +1 } .

Original: This is a normalized version of Hamming distance.
Revised: This is a normalized version of Hamming distance.

Original: For a ternary discrete, sign ( x ) ∈ {− 1 , 0 , 1 } is used.
Revised: For a ternary discrete, sign ( x ) ∈ {− 1 , 0 , 1 } is used.

Original: In order to consider the discrete distance as an angular distance within [0 , 1] , normalized one is deﬁned as D h 01 :
Revised: In order to consider the discrete distance as an angular distance within [0 , 1] , normalized one is deﬁned as D h 01 :

Original: =
Revised: =

Original: − D
Revised: − D

Original: h
Revised: h

Original: +1, where 0 ≤ D h 01 ≤ 1 .
Revised: +1, where 0 ≤ D h 01 ≤ 1 .

Original: The angular distance based on the above product can be rephrased as θ D h
Revised: The angular distance based on the above product can be rephrased as θ D h

Original: = D h 01 4 where 0 ≤ θ D h ≤ 1 .
Revised: = D h 01 4 where 0 ≤ θ D h ≤ 1 .

Original: As the discrete distance could underestimate to approximate the model distribution, we merge the above discrete distance metric with continuous angular distance metric ( 
Revised: As the discrete distance could underestimate to approximate the model distribution, we merge the above discrete distance metric with continuous angular distance metric ( [Equation] = [Equation]

Revised: arccos

Original: [Equation]
Revised: [Equation]

Original: 1 ) into a single metric.
Revised: into a single metric.

Original: We simply use the deﬁnition of Pythagorean means which consist of the arithmetic mean (AM), the geometric mean (GM), and the harmonic mean (HM).
Revised: We simply use the deﬁnition of Pythagorean means which consist of the arithmetic mean (AM), the geometric mean (GM), and the harmonic mean (HM).

Original: Pythagorean means using a pair of angular distances are deﬁned as follows:
Revised: Pythagorean means using a pair of angular distances are deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: In the angular distance 5 using a pair of angles { θ D h , θ } , a reversed form 1 − D { θ Dh ,θ } is adopted to maximize an angle in  optimization formulation as a form of minimization instead of ( · ) − s where s = 1 , 2 , . . .
Revised: In the angular distance 5 using a pair of angles { θ D h , θ } , a reversed form 1 − D { θ Dh ,θ } is adopted to maximize an angle in the optimization formulation by minimization instead of ( · ) − s where s = 1 , 2 , . . .

Original: which is used in Thomson problem that utilizes s -energy (Brauchart & Grabner, 2015).
Revised: which is used in Thomson problem that utilizes s -energy (Brauchart & Grabner, 2015).

Original: The cosine similarity using the pair of angles can be deﬁned as follows:
Revised: The cosine similarity using the pair of angles can be deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: then the cosine similarity functions are normalized with cos( · )+ to have a value within a range [0 , 1] .
Revised: then the cosine similarity functions are normalized with cos( · )+ to have a value within a range [0 , 1] .

Original: Finally, Pythagorean means of cosine similarities can be calculated as follows:
Revised: Finally, Pythagorean means of cosine similarities can be calculated as follows:

Original: [Equation]
Revised: [Equation]

Original: Metric functions deﬁned in (9), (10), and (11) satisfy metric conditions: non-negativity, symmetry, and triangle inequality.
Revised: Metric functions deﬁned in (9), (10), and (11) satisfy metric conditions: non-negativity, symmetry, and triangle inequality.

Original: The distance using the above metric function between any two parameter points is bounded, because the hypersphere is a compact manifold.
Revised: The distance using the above metric function between any two parameter points is bounded, because the hypersphere is a compact manifold.

Original: 3. G RADIENTS AND B ACKPROPAGATION
Revised: 3. G RADIENTS AND B ACKPROPAGATION

Original: As the sign function is not differentiable at the value 0, we adopt alternative backpropagation function.
Revised: As the sign function is not differentiable at the value 0, we adopt alternative backpropagation function.

Original: We adopt straight-through estimator (STE) (Bengio et al., 2013) in the backward path of the neural networks for the sign function in the discrete metric.
Revised: We adopt straight-through estimator (STE) (Bengio et al., 2013) in the backward path of the neural networks for the sign function in the discrete metric.

Original: The derivative of the sign function is substituted with 1 | w |≤ 1 in the backward pass, known as the saturated STE.
Revised: The derivative of the sign function is substituted with 1 | w |≤ 1 in the backward pass, known as the saturated STE.

Original: As the derivative of arccos( x ) ( − 1 √ 1 − x 2 ) is undeﬁned at the value x = ± 1 , we apply clamping to the cosine function to have x ∈ [ − 0 .
Revised: As the derivative of arccos( x ) ( − 1 √ 1 − x 2 ) is undeﬁned at the value x = ± 1 , we apply clamping to the cosine function to have x ∈ [ − 0 .

Original: 99 , 0 . 99] 6 where x = cos( θπ ) , and 0 ≤ θ ≤ 1 .
Revised: 99 , 0 . 99] 6 where x = cos( θπ ) , and 0 ≤ θ ≤ 1 .

Original:  E XPERIMENTS 4. E XPERIMENTAL SETUP
Revised:  E XPERIMENTS 4. E XPERIMENTAL SETUP

Original: Datasets We conduct experiments using four publicly available datasets including small size images (CIFAR-10 and CIFAR-100) and large size images (CUB200-2011 (Wah et al., 2011) and
Revised: Datasets We conduct experiments using four publicly available datasets including small size images (CIFAR-10 and CIFAR-100) and large size images (CUB200-2011 (Wah et al., 2011) and

Original: Stanford-Cars (Krause et al., 2013b), shortly CUB200 and Cars hereafter respectively).

Original: CIFARdataset is used to validate effectiveness of the proposed metric without hierarchical information.
Revised: CIFARdataset is used to validate effectiveness of the proposed metric without hierarchical information.

Original: Except CIFAR-10, we use two-level hierarchy pairs { parent, child } for the proposed hierarchical regularization on three datasets.
Revised: Except CIFAR-10, we use two-level hierarchy pairs { parent, child } for the proposed hierarchical regularization on three datasets.

Original: In Table 7 at Appendix, statistics of datasets in detail is provided.
Revised: In Table 7 at Appendix, statistics of datasets in detail is provided.

Original: Deep neural network models and training setting We used the deep residual neural network ( resnet )
Revised: Deep neural network models and training setting We used the deep residual neural network ( resnet )

Original: (He et al., 2016) with different sizes of parameters (weights) dependent on an image size of datasets.
Revised: (He et al., 2016) with different sizes of parameters (weights) dependent on an image size of datasets.

Original: For a small size input ( 32 × 32 pixels) of CIFAR-10 and CIFAR-100, we used resnet with smaller number of parameters (light models, resnet-20 (0.29M) and resnet-110 (1.73 M))
Revised: For a small size input ( 32 × 32 pixels) of CIFAR-10 and CIFAR-100, we used resnet with smaller number of parameters (light models, resnet-20 (0.29M) and resnet-110 (1.73 M))

Original: so as not to have redundant parameters leading to over-ﬁtting.
Revised: so as not to have redundant parameters leading to over-ﬁtting.

Original: For a ﬁne-grained input ( 224 × 224 pixels) of
Revised: For a ﬁne-grained input ( 224 × 224 pixels) of

Original: CUB200 and Cars, the original resnet with larger number of parameters (heavy models 7 , Resnet- (11.28M) and Resnet-50 (23.91M)) is used.
Revised: CUB200 and Cars, the original resnet with larger number of parameters (heavy models 7 , Resnet- (11.28M) and Resnet-50 (23.91M)) is used.

Original: In training of the deep neural network with the hierarchical regularization, we assume that the global hierarchical structure is not given.
Revised: In training of the deep neural network with the hierarchical regularization, we assume that the global hierarchical structure is not given.

Original: Instead, stochastic or partial hierarchical structure is shown within each mini-batch by hierarchy pairs.
Revised: Instead, stochastic or partial hierarchical structure is shown within each mini-batch by hierarchy pairs.

Original: Mini-batches, 512 for light models and 256 for heavy models, are used in the SGD optimizer.
Revised: Mini-batches, 512 for light models and 256 for heavy models, are used in the SGD optimizer.

Original: We applied  hierarchical regularization in the FC layer of the resnet.
Revised: We applied the hierarchical regularization in the FC layer of the resnet.

Original: Even though SGD is known as an unbiased estimation, a stochastic hierarchical structure could affect the overall approximation performance upon the class distribution within the mini-batch.
Revised: Even though SGD is known as an unbiased estimation, a stochastic hierarchical structure could affect the overall approximation performance upon the class distribution within the minibatch.

Original: Settings in more detail are provided in Appendix.
Revised: Settings in more detail are provided in Appendix.

Original: Baseline setting In terms of the regularization, we used three reference settings, called ‘baseline’, ‘baseline+ l 2 ’, and ‘E’ for a reference.
Revised: Baseline setting In terms of the regularization, we used three reference settings, denoted ‘baseline’, ‘baseline+ l 2 ’, and ‘E’ for a reference.

Original: First, in ‘baseline’, we examine neural networks without regularization.
Revised: First, in ‘baseline’, we examined neural networks without regularization.

Original: Then, in ‘baseline+ l 2 ’, we added l 2 -norm minimization based regularization (which is equivalent to a weight decay in SGD setting (Zhang et al., 2019)) via λ f (cid:80) k (cid:107) w k (cid:107) , where w k ∈ W and λ f > 0 to ‘baseline’.
Revised: Then, in ‘baseline+ l 2 ’, we added l 2 -norm minimization based regularization (which is equivalent to a weight decay in SGD setting (Zhang et al., 2019)) via λ f (cid:80) k (cid:107) w k (cid:107) , where w k ∈ W and λ f > 0 to ‘baseline’.

Original: Lastly, in ‘E’, energy minimization (Liu et al., 2018), diversity promoting regularization without hierarchical information, via λ e (cid:80) i (cid:54) =
Revised: Lastly, in ‘E’, energy minimization (Liu et al., 2018), diversity promoting regularization without hierarchical information, via λ e (cid:80) i (cid:54) =

Original: j d ( w i , w j ) , where d ( · , · ) is a pairwise distance and λ e > 0 , is added to baseline+ l 2 .
Revised: j d ( w i , w j ) , where d ( · , · ) is a pairwise distance and λ e > 0 , is added to baseline+ l 2 .

Original: When the proposed hierarchical regularization (Eq. 5) is applied to the FC layer, we named it as ‘H’.
Revised: When the proposed hierarchical regularization (Eq. 5) is applied to the FC layer, we denoted it as ‘H’.

Original: Hence, the regularization is incrementally adopted to the baseline: l 2 +‘E’+‘H’.
Revised: Hence, the regularization is incrementally adopted to the baseline: l 2 +‘E’+‘H’.

Original: In terms of a distance metric, we compared the proposed metric with other continuous metrics such as Euclidean distance with the unit-length projection (‘N-euclidean2’, (cid:80) 
Revised: In terms of a distance metric, we compared the proposed metric with other continuous metrics such as Euclidean distance with the unit-length projection (‘N-euclidean2’, (cid:80) i (cid:54) =

Original: [Equation] ) 8 , angular distance (‘angular2’, [Equation]
Revised: j (cid:107) w i (cid:107) w i (cid:107) − w i (cid:107) w i (cid:107) (cid:107) − 2 ) 8 , angular distance (‘angular2’, (cid:80) i (cid:54) = j arccos ( w

Original: [Equation]
Revised: [Equation]

Revised: · w j (cid:107) w i (cid:107)(cid:107) w j (cid:107) )

Original:  2 ) where ‘2’ is from Riesz s -energy shown higher accuracy, and cosine similarity (‘cosine’, (cid:80) i [Equation]
Revised: − 2 ) where ‘2’ is from Riesz s -energy shown higher accuracy, and cosine similarity (‘cosine’, (cid:80) i (cid:54) =

Original: j
Revised: j

Original: [Equation]
Revised: [Table]

Original: 4. R ESULTS
Revised: 4. R ESULTS

Original: Low resolution object classiﬁcation In Table 

Revised: Low resolution object classiﬁcation In Table

Original: , we showed test accuracy ( % ) using different metrics with energy minimization regularization without  hierarchical regularization 9 along a weight decay ( l 2 -norm) using CIFAR-10 and resnet-20.
Revised: , we showed test accuracy ( % ) using different metrics with energy minimization regularization without the hierarchical regularization 9 along a weight decay ( l 2 -norm) using CIFAR-10 and resnet-20.

Original:  ‘baseline’ method is examined without ( × ) or with ( ◦ ) the weight decay ( l 2 ).
Revised: The ‘baseline’ method is examined without ( × in Table 1) or with ( ◦ in Table 1) the weight decay ( l 2 ).

Original: The discrete angular metric based regularization ( D terh (ternary code), D binh (binary code), D cos( HM ) , and D (cid:48) HM cos ) on the baseline can improve the generalization performance in term of test accuracy compared to the other continuous metrics such as N-euclidean2, angular2, and cosine.
Revised: The discrete angular metric based regularization ( D terh (ternary code), D binh (binary code), D cos( HM ) , and D (cid:48) HM cos ) on the baseline can improve the generalization performance in term of test accuracy compared to the other continuous metrics such as N-euclidean2, angular2, and cosine.

Original: Due to the unit-length projection in N-euclidean2, their performance is comparable to that other angular metrics.
Revised: Due to the unit-length projection in N-euclidean2, their performance is comparable to that other angular metrics.

Original: D (cid:48) denotes an acrcosine adopted discrete distance used as 
Revised: D (cid:48) denotes an arccosine adopted discrete distance used as D (cid:48)

Original: D (cid:48) h (see footnote 4 in Section 3.2).
Revised: h (see footnote 4 in Section 3.2).

Original: Regularization methods are applied over all layers except Batch Normalization (BN) layers.
Revised: Regularization methods are applied over all layers except Batch Normalization (BN) layers.

Original: As explained in (van Laarhoven, 2017; Zhang et al., 2019), l 2 -norm minimization based regularization (weight decay) showed much improvement due to their effective learning rate in SGD.[Table] [Table]
Revised: As explained in (van Laarhoven, 2017; Zhang et al., 2019), l 2 -norm minimization based regularization (weight decay) showed much improvement due to their effective learning rate in SGD.

Original: [Table] [Table] [Table] As shown in Table 2, the regularization with proposed metrics shows better accuracy performance than that of the baseline on both resnet-20 and resnet-110 using CIFAR-100 dataset.
Revised: As shown in Table 2, the regularization with proposed metrics shows better accuracy performance than that of the baseline on both resnet-20 and resnet-110 using CIFAR-100 dataset.

Original: Comparing to the baseline+ l 2 , with energy minimization ‘E’ regularization, metrics such as D terh , D cos( AM ) , and
Revised: Comparing to the baseline+ l 2 , with energy minimization ‘E’ regularization, metrics such as D ter h , D cos( AM ) , and

Original: D (cid:48) cos( HM ) showed better performance than other metrics.
Revised: D (cid:48) cos( HM ) showed better performance than other metrics.

Revised: If the proposed hierarchical ‘H’ regular-

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: If the proposed hierarchical ‘H’ regularization is applied, the generalization performance is further improved in most metric cases on both resnet-20 and resnet-110.
Revised: [Table] [Table] [Table] ization is applied, the generalization performance is further improved in most metric cases on both resnet-20 and resnet-110.

Original: As the binary metric shows a better performance than that ternary, we adopt binary discretization for proposed angular metrics ( D • , D cos( • ) , and D • cos ) in experiments.
Revised: As the binary metric shows a better performance than that ternary, we adopt binary discretization for proposed angular metrics ( D • , D cos( • ) , and D • cos ) in experiments.

Original: Fine-grained visual categorization In this experiment
Revised: Fine-grained visual categorization In this experiment

Original: , we used two large size image datasets with a ﬁne-grained category.
Revised: , we used two large size image datasets with a ﬁne-grained category.

Original: One is with birds (CUB200) and another is with cars (Cars) that focus on single species of objects.
Revised: One is with birds (CUB200) and another is with cars (Cars) that focus on single species of objects.

Original: A hierarchy of CUB200 is deﬁned by the academic expert on birds whereas that of Cars is categorized manually based on a name of cars by a non-expert.
Revised: A hierarchy of CUB200 is deﬁned by the academic expert on birds whereas that of Cars is categorized manually based on a name of cars by a non-expert.

Original: The rate between the number of superclass (parent) per subclass of CUB200 (0.35) is much larger than that of Cars (0.0459) (as shown in Table 7 at Appendix).
Revised: Moreover, the rate between the number of superclass (parent) per subclass of CUB200 (0.35) is much larger than that of Cars (0.0459) (as shown in Table 7 at Appendix).

Original: That rate of Cars is even smaller than that of CIFAR-100.
Revised: That rate of Cars is even smaller than that of CIFAR-100.

Original: As shown in Table 3, the proposed hierarchical regularization signiﬁcantly improved the test accuracy over all metrics on both Resnet-18 and Resnet-50.
Revised: As shown in Table 3, the proposed hierarchical regularization signiﬁcantly improved the test accuracy over all metrics on both Resnet-18 and Resnet-50.

Original: Compared to CUB200, as shown in Table 4, the improvement of the proposed method is not that signiﬁcant for Cars dataset.
Revised: Compared to CUB200, as shown in Table 4, the improvement of the proposed method is not that signiﬁcant for Cars dataset.

Original: This is because
Revised: This is because

Original: CUB200 dataset has not only better hierarchical categorization by the expert but also more diversitiﬁed pairs between the superclass and the subclass.
Revised: CUB200 dataset has not only better hierarchical categorization by the expert but also more diversitiﬁed pairs between the superclass and the subclass.

Original: Ablation study In Table 5, we showed how metrics affect the generalization performance using resnet-20 and CIFAR-100 dataset with different proposed  showed signiﬁcantly improved performance  to baseline
Revised: Ablation study In Table 5, we showed how metrics affect the generalization performance using resnet-20 and CIFAR-100 dataset. The proposed method with different proposed metrics showed signiﬁcantly improved performance compared to the baseline+ l 2 .

Original: HM [Table] [Table] [Table]
Revised: Individual means (AM, GM, and

Original: [Table]
Revised: HM) show different improvement patterns.

Original: In Table [Table] we showed how heterogeneous metrics on convolutional (conv.)
Revised: In Table 6, we showed how heterogeneous metrics on convolutional (conv.)

Original: layers (with the energy minimization, ‘E’) and fully connected (FC) layers (with the hierarchical ‘H’ regularization, ‘H’) affect the generalization performance using resnet-20 and CIFAR-100 dataset.
Revised: layers (with the energy minimization, ‘E’) and fully connected (FC) layers (with the hierarchical ‘H’ regularization, ‘H’) affect the generalization performance using resnet-20 and CIFAR-100 dataset.

Original: The cases applying hierarchical regularization showed better performance than that the baseline applying only energy minimization regularization.
Revised: The cases applying hierarchical regularization showed better performance than that the baseline applying only energy minimization regularization.

Original: In this experiment, a combination GM and HM shows a better improvement than that other combinations
Revised: In this experiment, a combination GM and HM shows a better improvement than that other combinations. [Table] [Table]

Original: [Table]
Revised: [Table]

Original:  R ELATED WORKS
Revised:  R ELATED WORKS

Original: Promoting of diversity in an embedding space or model parameters is an widely adopted strategy in machine learning related area to improve the generalization performance (Cogswell et al., 2016;
Revised: Promoting of diversity in an embedding space or model parameters is an widely adopted strategy in machine learning related area to improve the generalization performance (Cogswell et al., 2016;

Original: Yang et al., 2019; Li et al., 2012; Ratzlaff & Fuxin, 2019; Xie et al., 2017b; 2018; 2017a;
Revised: Yang et al., 2019; Li et al., 2012; Ratzlaff & Fuxin, 2019; Xie et al., 2017b; 2018; 2017a;

Original: Liu et al., 2018).
Revised: Liu et al., 2018).

Original: This diversity based strategy can be applied in a variety of levels such as in a feature level (Cogswell et al., 2016; Xie et al., 2018), in a projection parameter level (Xie et al., 2017a; Liu et al., 2018), in a model ensemble level (Zhou et al., 2018;
Revised: This diversity based strategy can be applied in a variety of levels such as in a feature level (Cogswell et al., 2016; Xie et al., 2018), in a projection parameter level (Xie et al., 2017a; Liu et al., 2018), in a model ensemble level (Zhou et al., 2018;

Original: Ratzlaff & Fuxin, 2019), in a latent space model level (Ratzlaff & Fuxin, 2019; Liu et al., 2018), and in a generative model level (Yang et al., 2019;
Revised: Ratzlaff & Fuxin, 2019), in a latent space model level (Ratzlaff & Fuxin, 2019; Liu et al., 2018), and in a generative model level (Yang et al., 2019;

Original: Ratzlaff & Fuxin, 2019).
Revised: Ratzlaff & Fuxin, 2019).

Original: Among these approaches, some regularization methods require additional optimization effort.
Revised: Among these approaches, some regularization methods require additional optimization effort.

Original: Enlarging pairwise distance between features (Xie et al., 2018) requires computational efforts due to their covariance matrix.
Revised: Enlarging pairwise distance between features (Xie et al., 2018) requires computational efforts due to their covariance matrix.

Original: In (Xie et al., 2017b), unit-eigenvalue is utilized via singular value decomposition.
Revised: In (Xie et al., 2017b), unit-eigenvalue is utilized via singular value decomposition.

Original: To optimize a direction and magnitude of the parameter vector alternatively, an alternating direction method of multipliers (ADMM)
Revised: To optimize a direction and magnitude of the parameter vector alternatively, an alternating direction method of multipliers (ADMM)

Original: (Xie et al., 2017a) is used.
Revised: (Xie et al., 2017a) is used.

Original: In terms of learning on a hypersphere, (Liu et al., 2017) proposed that hyperspherical convolution (SphereCov) replaces the traditional inner-product based convolution in order to conduct learning angular representation.
Revised: In terms of learning on a hypersphere, (Liu et al., 2017) proposed that hyperspherical convolution (SphereCov) replaces the traditional inner-product based convolution in order to conduct learning angular representation.

Original: On a hypersphere, regularization via promoting diversity of parameters has been proposed (Liu et al., 2018) based on Minimum Hyperspherical Energy.
Revised: On a hypersphere, regularization via promoting diversity of parameters has been proposed (Liu et al., 2018) based on Minimum Hyperspherical Energy.

Original: In terms of hierarchical learning, (Nickel & Kiela, 2017; Ganea et al., 2018) proposed to apply hyperbolic spaces to embed the data in deep neural networks.
Revised: In terms of hierarchical learning, (Nickel & Kiela, 2017; Ganea et al., 2018) proposed to apply hyperbolic space to embed the data in deep neural networks.

Original: They showed that learning representations in the hyperbolic space is effective to the data preserving a latent hierarchy compared to non-Euclidean space.
Revised: They showed that learning representations in the hyperbolic space is effective to the data preserving a latent hierarchy compared to in the Euclidean space.

Original: Different from these works  for representation learning, our proposed method is focused on regularization learning using explicit (predeﬁned) hierarchy information.
Revised: Different from these works which are aimed for representation learning, our proposed method is focused on regularization learning using explicit (predeﬁned) hierarchy information.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We proposed the regularization method, which maximizes a pairwise distance between parameters preserving a hierarchical structure.
Revised: We proposed the regularization method, which maximizes a pairwise distance between parameters preserving a hierarchical structure.

Original: To deﬁne a hierarchical parameter space, we reformulated the topology space with multiple hyperspheres.
Revised: To deﬁne a hierarchical parameter space, we reformulated the topology space with multiple hyperspheres.

Original: In each hypersphere, a projection is parameterized by the surface parameter with the center parameter.
Revised: In each hypersphere, a projection is parameterized by the surface parameter with the center parameter.

Original: By imposing a maximumangular distance between hierarchical parameters, diversity of parameters preserving semantic structure was promoted.
Revised: By imposing a maximum distance between hierarchical parameters, diversity of parameters preserving semantic structure was promoted.

Original: For theangular distance metric in multiple separated spaces, we proposed a discrete metric integrated with a continuous metric.
Revised: For the distance metric in multiple separated spaces, we proposed a discrete metric integrated with a continuous metric.

Original: Extensive experiments using publicly available datasets (CIFAR-10, CIFAR100, CUB200-2011, and Stanford Cars), the deep neural network with our proposed regularization method showed superior classiﬁcation performance, especially when the number of super-classes is large.
Revised: Extensive experiments using publicly available datasets (CIFAR-10, CIFAR100, CUB200-2011, and Stanford Cars), the deep neural network with our proposed regularization method showed superior classiﬁcation performance, especially when the number of super-classes is large.

Original: For further exploration in future, our proposed method can be combined with hierarchical representation learning such as hyperbolic (or Poincar´e) embeddings (Nickel & Kiela, 2017; Ganea et al., 2018).
Revised: For further exploration in future, our proposed method can be combined with hierarchical representation learning such as hyperbolic (or Poincar´e) embeddings (Nickel & Kiela, 2017; Ganea et al., 2018)

Original: A PPENDIX
Revised: A PPENDIX

Original: Dataset acquisition details CIFAR-100 dataset provides a pair of superclass-subclass labels.
Revised: Dataset acquisition details CIFAR-100 dataset provides a pair of superclass-subclass labels.

Original: In
Revised: In

Original: CUB200 dataset, the pairs can be extracted from their ﬁlename.
Revised: CUB200 dataset, the pairs can be extracted from their ﬁlename.

Original: In Cars dataset, we parsed each subclass label to one of nine superclass vehicle types, such as “Sedan”, “SUV”, “Van” and etc., following (Krause et al., 2013a).
Revised: In Cars dataset, we parsed each subclass label to one of nine superclass vehicle types, such as “Sedan”, “SUV”, “Van” and etc., following (Krause et al., 2013a).

Original: [Table]
Revised: [Table]

Original: Deep neural network models and training details First, resnet-20 (0.29M) and resnet-110 (1.
Revised: Deep neural network models and training details First, resnet-20 (0.29M) and resnet-110 (1.

Original: M), which include a combination of Basic blocks with output channels [16 , 30 , 64] are adopted for light models.
Revised: M), which include a combination of Basic blocks with output channels [16 , 30 , 64] are adopted for light models.

Original: An input dimensionality of the fully connected (FC) layer (a classier) is 64 for both resnet-20 and resnet-110.
Revised: An input dimensionality of the fully connected (FC) layer (a classier) is 64 for both resnet-20 and resnet-110.

Original: Second, heavy models are adopted such as Resnet-18 (11.28M 10 ) and
Revised: Second, heavy models are adopted such as Resnet-18 (11.28M 10 ) and

Original: Resnet-50 (23.91M) which consists of the basic blocks (Resnet-18) or the bottleneck blocks with output channels [64, 128, 256, 512] in Conv.
Revised: Resnet-50 (23.91M) which consists of the basic blocks (Resnet-18) or the bottleneck blocks with output channels [64, 128, 256, 512] in Conv.

Original: layers.
Revised: layers.

Original: An input dimensionality of the FC layer isfor Resnet-18 and 2048 for Resnet-50.
Revised: An input dimensionality of the FC layer isfor Resnet-18 and 2048 for Resnet-50.

Original: Networks are optimized with SGD for both light and heavy models: we ﬁxed 1) the weight initialization with Random-Seed number ‘0’ in pytorch, 2) learning rate schedule [0.1, 0.01, 0.001], 3) with momentum 0.9, 4) regularization: l 2 -norm minimization with λ f = 0 .
Revised: Networks are optimized with SGD for both light and heavy models: we ﬁxed 1) the weight initialization with Random-Seed number ‘0’ in pytorch, 2) learning rate schedule [0.1, 0.01, 0.001], 3) with momentum 0.9, 4) regularization: l 2 -norm minimization with λ f = 0 .

Original: 0005 , 5) energy minimization λ
Revised: 0005 , 5) energy minimization λ

Original: e = { 0 . 1 , 1 } , and hierarchical minimization λ l = 0 .
Revised: e = { 0 . 1 , 1 } , and hierarchical minimization λ l = 0 .

Original: 1 ×{ 1 , 5 } .
Revised: 1 ×{ 1 , 5 } .

Original: All the regularization is not applied to the parameters of Batch Normalization (BN) layers, since BN is also a regularization function.
Revised: All the regularization is not applied to the parameters of Batch Normalization (BN) layers, since BN is also a regularization function.

Original: A bias term in the FC layer is not used.
Revised: A bias term in the FC layer is not used.

Original: The images in training and test, images are resized to 256 size.
Revised: The images in training and test, images are resized to 256 size.

Original: The images is cropped with 224 × 224 size at random location in training and at center location in test.
Revised: The images is cropped with 224 × 224 size at random location in training and at center location in test.

Original: Horizontal ﬂipping is applied in training.
Revised: Horizontal ﬂipping is applied in training.

Original: The light models are trained from scratch without the pretrained weights for 300 epochs.
Revised: The light models are trained from scratch without the pretrained weights for 300 epochs.

Original: The heavy model is trained using pretrained model provided by pytorch library 11 with 100 epochs.
Revised: The heavy model is trained using pretrained model provided by pytorch library 11 with 100 epochs.

Original: The experiments are conduced using GPU “NVIDIA TESLA
Revised: The experiments are conduced using GPU “NVIDIA TESLA

Original: P40”.
Revised: P40”.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: ( l,l − 1) c,g k ) =
Revised: ( l,l − 1) c,g k ) =

Original: d ( w ( l,l − 1) c,g i , w ( l,l − 1) c,g j ) + d ( w ( l,l − 1) c,g j , w ( l,l − 1) c,g k ) . . .
Revised: d ( w ( l,l − 1) c,g i , w ( l,l − 1) c,g j ) + d ( w ( l,l − 1) c,g j , w ( l,l − 1) c,g k ) . . .

Original: corresponds to Eq (7)
Revised: corresponds to Eq (7)

Original: Constraints in Eq.
Revised: Constraints in Eq.

Original: 4 C l := (cid:80) k λ k C l,k can be given as follows:
Revised: 4 C l := (cid:80) k λ k C l,k can be given as follows:

Original: Constraint 1 ( C 1 ): This constraint describes that a radius of an inner sphere must be smaller than that of its outer sphere.
Revised: Constraint 1 ( C 1 ): This constraint describes that a radius of an inner sphere must be smaller than that of its outer sphere.

Original: r
Revised: r

Original: [Equation]
Revised: [Equation]

Original: − 1) − r
Revised: − 1) − r

Original: [Equation]
Revised: [Equation]

Original: l ) ≥
Revised: l ) ≥

Original: [Equation]
Revised: [Equation]

Original: ⇒
Revised: ⇒

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: Constraint 2 ( C 2 ): This constraint describes that a center of an inner sphere must be located within its outer sphere.
Revised: Constraint 2 ( C 2 ): This constraint describes that a center of an inner sphere must be located within its outer sphere.

Original: [Equation]
Revised: [Equation]

Original: Constraint 3 ( C 3 ): This constraint describes that a margin between spheres must be larger than zero.
Revised: Constraint 3 ( C 3 ): This constraint describes that a margin between spheres must be larger than zero.

Original: (cid:107) w ( l,l − 1) c
Revised: (cid:107) w ( l,l − 1) c

Original: (cid:107) (2 − 2 cos θ ) 0 .
Revised: (cid:107) (2 − 2 cos θ ) 0 .

Original: 5 −
Revised: 5 −

Original: [Equation]
Revised: [Equation]

Original: r ( l )
Revised: r ( l )

Original: [Equation]
Revised: [Equation]

Original: ( l )
Revised: ( l )

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]