Original: D EEP H IERARCHICAL -H YPERSPHERICAL L EARNING (DH 2 L)
Revised: D EEP H IERARCHICAL -H YPERSPHERICAL L EARNING (DH 2 L)

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Regularization is known to be an inexpensive and reasonable solution to alleviate over-ﬁtting problems arising in inference models, including deep neural networks.
Revised: Regularization is known to be an inexpensive and reasonable solution to alleviate over-ﬁtting problems arising in inference models, including deep neural networks.

Original: In this paper, we propose a hierarchical regularization which preserves the semantic structure of a sample distribution.
Revised: In this paper, we propose a hierarchical regularization which preserves the semantic structure of a sample distribution.

Original: At the same time, this regularization promotes diversity by imposing distance between parameter vectors enlarged within semantic structures.
Revised: At the same time, this regularization promotes diversity by imposing distance between parameter vectors enlarged within semantic structures.

Original: To generate evenly distributed parameters which is considered less redundant, we constrain them to lie on hierarchical hyperspheres .
Revised: To generate evenly distributed parameters which is considered less redundant, we constrain them to lie on hierarchical hyperspheres .

Original: To deﬁne a hierarchical parameter space, we propose to reformulate the topology space with multiple hyperspheres.
Revised: To deﬁne a hierarchical parameter space, we propose to reformulate the topology space with multiple hyperspheres.

Original: On each hypersphere, a projection is parameterized by two individual parameters.
Revised: On each hypersphere, a projection is parameterized by two individual parameters.

Original: Since maximizing a groupwise and pairwise distance between points on hypersphere is nontrivial (generalized Thomson problem), we propose a new discrete metric integrated with continuous angular metric.
Revised: Since maximizing a groupwise and pairwise distance between points on hypersphere is nontrivial (generalized Thomson problem), we propose a new discrete metric integrated with continuous angular metric.

Original: The proposed method shows improved generalization performance on extensive experiments using publicly available datasets (CIFAR-10, CIFAR-100, CUB2002011, and Stanford Cars), especially when the number of super-classes is large.
Revised: The proposed method shows improved generalization performance on extensive experiments using publicly available datasets (CIFAR-10, CIFAR-100, CUB2002011, and Stanford Cars), especially when the number of super-classes is large.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Diversity promoting learning has been widely adopted via enlarging pairwise distances (Xie et al., 2018; 2017a;
Revised: Diversity promoting learning has been widely adopted via enlarging pairwise distances (Xie et al., 2018; 2017a;

Original: Liu et al., 2018), increasing orthogonality (Xie et al., 2018), reducing covariance between parameters (Xie et al., 2017b), or reducing correlation on feature (Cogswell et al., 2016) to improve generalization performance.
Revised: Liu et al., 2018), increasing orthogonality (Xie et al., 2018), reducing covariance between parameters (Xie et al., 2017b), or reducing correlation on feature (Cogswell et al., 2016) to improve generalization performance.

Original: Among them, diversity promoting regularization (Xie et al., 2017a;b; Liu et al., 2018) by enforcing large diversity between projection parameters achieves a reasonable performance without modifying the model structure.
Revised: Among them, diversity promoting regularization (Xie et al., 2017a;b; Liu et al., 2018) by enforcing large diversity between projection parameters achieves a reasonable performance without modifying the model structure.

Original: Optimizing the objective function with a covariance matrix in (Xie et al., 2017a;b) is nontrivial.
Revised: Optimizing the objective function with a covariance matrix in (Xie et al., 2017a;b) is nontrivial.

Original: In different circumstances, the diversity promoting regularization via minimizing energy of parameters of deep neural networks has been proposed (Liu et al., 2018).
Revised: In different circumstances, the diversity promoting regularization via minimizing energy of parameters of deep neural networks has been proposed (Liu et al., 2018).

Original: By minimizing a pairwise distance between parameters on hypersphere with the known metric, they achieved the improved generalization performance.
Revised: By minimizing a pairwise distance between parameters on hypersphere with the known metric, they achieved the improved generalization performance.

Original: Following an efﬁcient regularization on hypersphere space, we explore further this direction with three main concepts (hierarchical and hyperspherical learning with discrete metrics).
Revised: Following an efﬁcient regularization on hypersphere space, we explore further this direction with three main concepts (hierarchical and hyperspherical learning with discrete metrics).

Original: 1) Why hierarchical learning?
Revised: 1) Why hierarchical learning?

Original: Hierarchical inference explains human intelligence .
Revised: Hierarchical inference explains human intelligence .

Original: In

Original:  (Kurzweil, 2013) it states that “the neocortex contains about  million very 
Revised: In (Kurzweil, 2013), it states that “the neocortex contains about 300 million very general pattern recognizers, arranged in a hierarchy”.

Original: general pattern recognizers, arranged in

Original: a

Original: hierarchy”.

Original: Applying a hierarchy of multiple classes based on semantic taxonomy is a natural choice to devise machine intelligence .
Revised: Applying a hierarchy of multiple classes based on semantic taxonomy is a natural choice to devise machine intelligence .

Original: Effectiveness of the hierarchical learning can be found in (Verma et al., 2012)
Revised: Effectiveness of the hierarchical learning can be found in (Verma et al., 2012).

Original: 2) Why hyperspherical learning?
Revised: 2) Why hyperspherical learning?

Original: Hypersphere can be represented by a centroid and a radius.
Revised: Hypersphere is the set of points at the equidistance (radius) from a given point (centroid) in a certain dimensional space.

Original: Due to the denominator in the unit-length normalization ( w (cid:107) w (cid:107)  ), the  distance deﬁned on the hypersphere converges when the magnitude of w goes inﬁnity while Euclidean distance goes inﬁnity.
Revised: Due to the denominator in the unit-length normalization ( w (cid:107) w (cid:107) , w ), the angular distance deﬁned on the hypersphere converges when the magnitude of w goes inﬁnity while Euclidean distance goes inﬁnity.

Original: Due to this bounded property, a hierarchical structure with multiple separated hyperspheres can be deﬁned.
Revised: Due to this bounded property, a hierarchical structure with multiple separated hyperspheres can be deﬁned.

Original: 3) Why discrete metric learning?
Revised: 3) Why discrete metric learning?

Original: If the vector points form discontinuous series with discrete representation (e
Revised: If the vector points form discontinuous series with discrete representation (e

Original: g. multi-dimensional binary or ternary), they are isolated from each other with a certain margin.
Revised: g. multi-dimensional binary or ternary), they are isolated from each other with a certain margin.

Original: This property may ﬁt with a disconnected manifold or a groupwise space problem.
Revised: This property may ﬁt with a disconnected manifold or a groupwise space problem.

Original: Moreover, to make points to be equidistributed where a pairwise distance is maximized is a nontrivial
Revised: Moreover, to make points to be equidistributed where a pairwise distance is maximized is a nontrivial

Original: task.
Revised: task.

Original: Because of ﬁnite and isolated points, this metric may reduce search efforts to satisfy those constraints using a set of pairwise distance.
Revised: Because of ﬁnite and isolated points, this metric may reduce search efforts to satisfy those constraints using a set of pairwise distance.

Original: In this paper, we propose to apply  hierarchical structure to parameter regularization on the multiple groupwise hyperspherical spaces.
Revised: In this paper, we propose to apply a hierarchical structure to parameter regularization on the multiple groupwise hyperspherical spaces.

Original: In order to ﬁnd an appropriate metric on this space, we explore a discrete angular metric.
Revised: In order to ﬁnd an appropriate metric on this space, we explore a discrete angular metric.

Original: We examine the proposed method on extensive experimental setups in terms of datasets and deep network models.
Revised: We examine the proposed method on extensive experimental setups in terms of datasets and deep network models.

Original:  M ULTIPLE SEPARATED HYPERSPHERES
Revised:  M ULTIPLE SEPARATED HYPERSPHERES

Original: Samples observed from the real world may be on disconnected manifolds.
Revised: Samples observed from the real world may be on disconnected manifolds.

Original: In other words, disjoint union of those manifolds could generate the global manifold (Lee, 2000).
Revised: In other words, disjoint union of those manifolds could generate the global manifold (Lee, 2000).

Original: In this section, we decompose the one space into multiple spaces (manifolds) and re-deﬁne the space in terms of hierarchical point of view.
Revised: In this section, we decompose the one space into multiple spaces (manifolds) and re-deﬁne the space in terms of hierarchical point of view.

Original: 2. D ISCONNECTED MANIFOLD VIA EQUIVALENT RELATIONS
Revised: 2. D ISCONNECTED MANIFOLD VIA EQUIVALENT RELATIONS

Original: Since it is not suitable to measure a pairwise distance between high dimensional vectors which have the hierarchical structure in the same space, we construct another identiﬁcation space which includes isolated spaces, from the original space (via equivalence relation (Tu, 2010)).
Revised: Since it is not suitable to measure a pairwise distance between high dimensional vectors which have the hierarchical structure in the same space, we construct another identiﬁcation space which includes isolated spaces, from the original space (via equivalence relation (Tu, 2010)).

Original: Denote d -sphere S d to be the set of points that satisﬁes S d = {
Revised: Denote d -sphere S d to be the set of points that satisﬁes S d = {

Original: w ∈ R d +1 : (cid:107) w (cid:107) = 1 } .
Revised: w ∈ R d +1 : (cid:107) w (cid:107) = 1 } .

Original: We construct multiple separated hyperspheres using multiple identifying relations.
Revised: We construct multiple separated hyperspheres using multiple identifying relations.

Original: In Figure 1, we use a center vector w c and a surface vector w s to deﬁne a hypershpere space and a projection parameter w .
Revised: In Figure 1, we use a center vector w c and a surface vector w s to deﬁne a hypershpere space and a projection parameter w .

Original: 2. P RIOR DISTRIBUTION AND REGULARIZATION
Revised: 2. P RIOR DISTRIBUTION AND REGULARIZATION

Original: To make parameter vectors uniformly distributed on the unit hypersphere, parameters are sampled from the Gaussian normal distribution (Muller, 1959; Harman & Lacko, 2010).
Revised: To make parameter vectors uniformly distributed on the unit hypersphere, parameters are sampled from the Gaussian normal distribution (Muller, 1959; Harman & Lacko, 2010).

Original: This is because the normal distribution is spherically symmetric (Muller, 1959).
Revised: This is because the normal distribution is spherically symmetric (Muller, 1959).

Original: In a Bayesian point of view, neural networks with Gaussian priors are known to induce an l 2 -norm regularization (Vladimirova et al., 2019).
Revised: In a Bayesian point of view, neural networks with Gaussian priors are known to induce an l 2 -norm regularization (Vladimirova et al., 2019).

Original: From two evidences, we know that enforcing the parameters to have the Gaussian prior is important in hyperspherical learning in neural networks.
Revised: From two evidences, we know that enforcing the parameters to have the Gaussian prior is important in hyperspherical learning in neural networks.

Original: Note that a parameter which is calculated from the difference arithmetic operation with two parameters on the normal Gaussian distribution is on the normal difference distribution .
Revised: Note that a parameter which is calculated from the difference arithmetic operation with two parameters on the normal Gaussian distribution is on the normal difference distribution .

Original:  M ETHOD
Revised:  M ETHOD

Original: In deep neural networks, the objective function J with regularization R in addition to a loss L ,
Revised: In deep neural networks, the objective function J with regularization R in addition to a loss L ,

Original: J R ( W ) =
Revised: J R ( W ) =

Original: L ( x , W ) + λ R ( W ) , is optimized to ﬁnd the optimal W having a near minimum loss L , arg min W J R ( x , W ) , where x ∈ R d 0 denotes an input vector,
Revised: L ( x , W ) + λ R ( W ) , is optimized to ﬁnd the optimal W having a near minimum loss L , arg min W J R ( x , W ) , where x ∈ R d 0 denotes an input vector,

Original: W = {
Revised: W = {

Original: W i ∈ R
Revised: W i ∈ R

Original: d i − 1 × c
Revised: d i − 1 × c

Original: i : W i = { w j ∈
Revised: i : W i = { w j ∈

Original: R d i − 1 } , j = 1 , . . .
Revised: R d i − 1 } , j = 1 , . . .

Original: c i , i = 1 , . . .
Revised: c i , i = 1 , . . .

Original: , L } denotes a set of parameter matrices (i.e. neurons/kernels),
Revised: , L } denotes a set of parameter matrices (i.e. neurons/kernels),

Original: L denotes the number of layers, and λ > 0 is to control the degree of the regularization.
Revised: L denotes the number of layers, and λ > 0 is to control the degree of the regularization.

Original: For a classiﬁcation task, the cross entropy loss is used for the loss function L .
Revised: For a classiﬁcation task, the cross entropy loss is used for the loss function L .

Original: We propose a new regularization formulation R preserving a hierarchical parameter structure in Section 3.1.
Revised: We propose a new regularization formulation R preserving a hierarchical parameter structure in Section 3.1.

Original: 3. R EGULARIZATION FOR H IERARCHICAL  H YPERSPHERICAL H YPOTHESES
Revised: 3. R EGULARIZATION FOR H IERARCHICAL AND H YPERSPHERICAL H YPOTHESES

Original: Denote w a parameter vector (called a projection parameter, an element of W at a single layer) to transform a given input into an embedding space deﬁned in a Euclidean metric space: x ∈ R d +1 (cid:55)→ w T x ∈ R .
Revised: Denote w a parameter vector (called a projection parameter, an element of W at a certain layer) to transform a given input into an embedding space deﬁned in a Euclidean metric space: x ∈ R d +1 (cid:55)→ w T x ∈ R .

Original: By the deﬁnition of unit-length projection w (cid:107) w (cid:107) , a new parameter vector ˆ w can be deﬁned on d -sphere: S d = { ˆ w ∈ R d +1 : (cid:107) ˆ w (cid:107)
Revised: By the deﬁnition of unit-length projection w (cid:107) w (cid:107) , a new parameter vector ˆ w can be deﬁned on d -sphere: S d = { ˆ w ∈ R d +1 : (cid:107) ˆ w (cid:107)

Original: = 1 } where (cid:107) · (cid:107) denotes l 2 -norm and the center is zero.
Revised: = 1 } where (cid:107) · (cid:107) denotes l 2 -norm and a center is zero.

Original: In other word the  parametervector ˆ w can be deﬁned by a center point vector w c ∈ R d +1 and a surface  vector w s ∈ R d
Revised: In other word the projection parameter ˆ w can be deﬁned by a center parameter vector w c ∈ R d +1 and a surface parameter vector w s ∈ R d

Original: +1 using an arithmetic operation:
Revised: +1 using an arithmetic operation:

Original: ˆ w := w s − w c .
Revised: ˆ w := w s − w c .

Original: We deﬁne the d -sphere with center and surface vectors: S d w c = { w s − w c ∈ R d
Revised: We deﬁne the d -sphere with center and surface parameters: S d w c = { w s − w c ∈ R d

Original: +1 : (cid:107) w s − w c (cid:107) = 1 } .
Revised: +1 : (cid:107) w s − w c (cid:107) = 1 } .

Original: For a notation simplicity, we use w instead of ˆ w hereafter.
Revised: For a notation simplicity, we use w instead of ˆ w hereafter.

Original: While we consider a radius equals to 1 for simplicity, the parameter vector can have a radius r > 0 .
Revised: While we consider a radius equals to 1 for simplicity, the parameter vector can have a radius r > 0 .

Original: 3.1. H IERARCHICAL PARAMETERS DERIVED FROM LEVELWISE AND GROUPWISE CENTROID VECTORS
Revised: 3.1. H IERARCHICAL PARAMETERIZATION VIA LEVELWISE AND GROUPWISE PARAMETERS

Original: We assume that the hierarchical structure consists of levelwise structure with a notation ( l ) and groupwise structure with a notation g below.
Revised: We assume that the hierarchical structure consists of levelwise structure with a notation ( l ) and groupwise structure with a notation g below.

Original: We explain these two terms to deﬁne parameter vectors serially.
Revised: Through deﬁning these structures serially, we explain hierarchical parameters.

Original: Levelwise structure The above parameter vector on S d w c can be deﬁned with the levelwise notation ( l ) as follows,
Revised: Levelwise structure The projection parameter w ∈ R d +1 in S d w c can be deﬁned with the levelwise notation ( l ) as follows,

Original: [Equation]
Revised: [Equation]

Original: where the parameter is deﬁned on l -th  d -sphere, S d w ( l ) c .
Revised: where S d w ( l ) c denotes l -th level d -sphere centered at w ( l ) c .

Original: In Figure 2, an example is provided in a lower dimensional space.

Original: In this paper, we deﬁne thehierarchical parameter in a higher dimensional space than that in(b) and (c) in Figure 2.
Revised: In this paper, we deﬁne the parameter in a higher dimensional space than that in Figure 2.

Original: In a levelwise setting, w ( l ) s and w ( l ) c  are additively represented based on the center parameter (centroid) calculated from the previous level: w ( l − 1) c
Revised: In a levelwise setting, w ( l ) s and w ( l ) c at the current level ( l ) are additively represented based on a center parameter of the previous level ( l − 1) : w ( l − 1) c

Original: [Equation]
Revised: [Equation]

Original:  c , where w ( l
Revised: l ) c , where w ( l

Original: [Equation]
Revised: [Equation]

Original: 1) c=
Revised: 1) c

Original: [Equation]
Revised: [Equation]

Original: ∆ w
Revised: w

Original: [Equation]
Revised: [Equation]

Original: ) is the accumulatedcenter vector and −−→ ∆ w ( l ) denotes a newly connected parametervector from w ( l − 1) c to w ( l ) c .
Revised: is the accumulated and −−→ ∆ w ( l ) denotes a connecting parameter from w ( l − 1) c to w ( l ) c .

Original: By denoting −−→ ∆ w ( l ) as w ( l,l − 1) , the center vector at the l -level is deﬁned as, w ( l ) c :=
Revised: By denoting −−→ ∆ w ( l ) as w ( l,l − 1) , the center vector at the l -th level is deﬁned as, w ( l ) c :=

Original: w ( l,l − 1) c
Revised: w ( l,l − 1) c

Original: + w ( l − 1) c , and the surface vector w ( l ) s := w 
Revised: + w ( l − 1) c , and the surface vector w ( l ) s := w ( l,l − 1) s + w

Original: [Equation]
Revised: [Equation]

Original: l,l − 1)s + w
Revised: l − 1)

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Both the center vector and the surface vector at the current level are based on the center vector at the previous level 1 .
Revised: Both the center parameter and the surface parameter at the current level are based on the center parameter at the previous level 1 .

Original: Hence, Eq. 1 is equivalent to
Revised: Hence, Eq. 1 is equivalent to

Original: [Equation]
Revised: [Equation]

Original: Note that we use ( l, l − 1) to denote a connecting parameter from the center parameter at the ( l − 1) th level to parameters at the( l ) th level.
Revised: Note that we use ( l, l − 1) to denote a connecting parameter from the center parameter at the ( l − 1) -th level to parameters at the l -th level.

Original: Groupwise structure With a group notation g k , the center parameter in Eq.
Revised: Groupwise structure With a group notation g k , the center parameter in Eq.

Original: 2 can be rephrased as w ( l,l − 1) c,g k , on S d w ( l,l − 1) c,gk which denotes d -sphere of g k group at  l -th level, g ( l ) :
Revised: 2 can be rephrased as w ( l,l − 1) c,g k , on S d w ( l,l − 1) c,gk which denotes d -sphere of g k group at the l -th level.

Revised: Each group constitutes a group set at the l -th level g ( l ) :

Original: = { g k } | g ( l ) | k
Revised: = { g k } | g ( l ) | k

Original: =1 . g ( l ) ⊆
Revised: =1 where g ( l ) ⊆

Original: G ( l ) is a group set at the l th level, and | · | denotes the cardinality.
Revised: G ( l ) , and | · | denotes the cardinality.

Original: A group  g ( l ) at the l -th level is conditioned on a group at a parent ( l − 1) -th level, g ( l − 1) :
Revised: The group set g ( l ) at the l -th level is conditioned on a set of groups at the ( l − 1) - th level, g ( l − 1) :

Original: = { g k (cid:48) } | g ( l − 1) | k (cid:48)
Revised: = { g k (cid:48) } | g ( l − 1) | k (cid:48)

Original: = where g ( l − 1) ⊆
Revised: = where g ( l − 1) ⊆

Original: G ( l − 1) .
Revised: G ( l − 1) .

Original: With their groupwise relation over levels, an adjacency indication  P ( l,l − 1) ( { G ( l − 1) , G ( l ) } ) ∈ { 0 , 1 } | G ( l − 1) |×| G ( l ) | is given.
Revised: By their groupwise relation over levels, an adjacency indication 2 P ( l,l − 1) ( { G ( l − 1) , G ( l ) } ) ∈ { 0 , 1 } | G ( l − 1) |×| G ( l ) | is given.

Original: Hence, the  parameter vector at the lth level is determined as: w ( l ) g k ,i :=
Revised: Hence, the i -th parameter vector in S d w ( l,l − 1) c,gk is deﬁned as follows,

Original: w ( l,l − 1) s,g k ,i − w ( l,l − 1) c,g k on S d w ( l,l − 1) c,gk where i = 1 , . . , | g k | , { w ( l,l − 1) s,g k ,i , w ( l,l − 1) c,g k } ∀ i is calculated based on w ( l − 1) c,g ( l − 1) referring to the adjacency indication.
Revised: [Equation]

Original: A representative vector of the group g k at the l -th level is the center vector w ( l ) c,g k and it is equivalent to a mean vectorof w ( l ) s,g k ,i ∀ i : µ ( w ( l ) s,g k ,i ) = 
Revised: where { w ( l,l − 1) s,g k ,i , w ( l,l − 1) c,g k } ∀ i is calculated based on a center vector w ( l − 1) c,g k (cid:48) at the ( l − 1) -th level 3 , i = 1 , . .

Original: 1 | g k | (cid:80) | g k | i w ( l ) s,g k,i 3 .
Revised: , | g k | , and | g k | denotes the number of surface parameter vectors in the group g k .

Original: 3.1. H IERARCHICAL REGULARIZATION
Revised: 3.1. H IERARCHICAL REGULARIZATION

Original: In this section, we deﬁne a regularization term of the hierarchical parameter vectors deﬁned above.
Revised: In this section, we deﬁne a regularization term for the hierarchical parameters deﬁned above.

Original: A set of parameters { W ( l,l − 1) s,g k 
Revised: A set of parameters { W ( l,l − 1) s,g k , w ( l,l [Equation] 1) c,g k , w ( l [Equation]

Revised: 1) c

Original: [Equation]
Revised: [Equation]

Revised: g (cid:48) k } ∈ W

Revised: [Equation]

Revised: k

Revised: [Equation]

Original:  where W 
Revised: ∀ g k (cid:48) where W ( l,l − 1) s,g k := { [Equation] ( l [Equation]

Revised: l − 1) s,g k

Original: [Equation]
Revised: [Equation]

Original:  is an optimizing target of hierarchical regularizationterm as follows:
Revised: i } | g k | i =1 , is an optimizing target of hierarchical regularization as follows:

Original: [Equation]
Revised: [Equation]

Original: where R l works on individual spheres S d w ( l,l − 1) c,gk , λ l > 0 , and C l aims to apply geometry-aware constraints across spheres.
Revised: where R l works on individual spheres S d w ( l,l − 1) c,gk , λ l > 0 , and C l aims to apply geometry-aware constraints across spheres.

Original: R l consists of twoparts of regularization terms with: 1) R l,p for projection parameter vectors in the same group g k on S d w ( l,l − 1) c,gk and 2) R l,c for center parameter vectors acrossthe groups in the same level on S d w ( l − 1) c,g (cid:48) k
Revised: R l consists of two terms for regularization: 1) R l,p for projection parameters in the same group g k on S d w ( l,l − 1) c,gk and 2) R l,c for center parameters across groups in the same level on S d [Equation]

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: R l,p
Revised: R l,p

Original: [Equation]

Original: l

Original: [Equation]

Original: l

Original: [Equation]
Revised: [Equation]

Original:  l,l − 1) g k ,j ) , (5) R l,c (  w ( l,l − 1) c,g [Equation] ) := | C | ( | C | − 1) (cid:88)
Revised: g j ) , (7) where w ( l,l − 1) g k ,i := w ( l,l − 1) s,g k ,i − w ( l,l − 1) c,g k , ( w ( l,l − 1) g k ,j is similarly deﬁned), G = |{ i (cid:54) =

Original: M d ( w ( l,l − 1) c,g [Equation] , w ( l,l − 1) c [Equation]
Revised: j ∈ g k }| , G k

Original: ) , (6) where w ( l,l − 1) g k ,i := w ( l,l − 1) s,g k ,i − w ( l,l − 1) c,g k , ( w ( l,l − 1) g k ,j is similarly deﬁned),

Original: G = { i (cid:54) =
Revised: = |{ i (cid:54) =

Original: j ∈ g k } , C = { g i (cid:54) =
Revised: j ∈ g k }| , C = |{ g i (cid:54) =

Original: g j ∈ g ( l ) } , and d ( · , · ) denotes a distance metric between the parameter vectors.
Revised: g j ∈ g ( l ) }| , and d ( · , · ) denotes a distance metric between parameters.

Original: The distance metric is deﬁned in Section 3.2.
Revised: The distance metric is deﬁned in Section 3.2.

Original: We explain  these parameter vectors and pairwise distancesformulating regularization terms in Figure 4 in Appendix.
Revised: We explain regularization terms formulated using these parameters and pairwise distances in Figure 4 of Appendix.

Original: When a mini-batch of inputs ( m x : a set of inputs { x i } ) is given, the regularization term becomes: E ( R ( W)) =
Revised: When a minibatch of inputs ( m x : a set of inputs { x i } ) is given, the regularization term becomes: E ( R ( W

Original: [Equation]
Revised: [Equation]

Original: The constraint term help construct geometry-aware relational parameters between different spheres on the same level and on the across levels.
Revised: The constraint term help construct geometry-aware relational parameters between different spheres on the same level and on the across levels.

Original: Multiple constraints are deﬁned as C l := (cid:80) k λ k C l,k , where C l,k is k th constraint between parameters at l th and ( l − 1) th, and λ k > 0 is a Lagrange multiplier.
Revised: Multiple constraints are deﬁned as C l := (cid:80) k λ k C l,k , where C l,k is k th constraint between parameters at l -th and ( l − 1) -th level, and λ k > 0 is a Lagrange multiplier.

Original: We apply three constraints in a geometric point of view. The detailed formulation is deﬁned in appendix.
Revised: We propose three constraints in a geometric point of view deﬁned in Appendix.

Original: 3. D ISCRETE AND CONTINUOUS ANGULAR DISTANCE METRIC
Revised: 3. D ISCRETE AND CONTINUOUS ANGULAR DISTANCE METRIC

Original: Discrete(code) product metric might be a good ﬁt with the above group-wise deﬁnition.
Revised: Discrete metric might be a good ﬁt with the above groupwise deﬁnition.

Original: We expect thata projected point from the parametersformed in a discrete metric space, are isolatedfrom each other.
Revised: We expect that projected points by parameters in a discrete metric space, are isolated each other.

Original: In Figure 3,  discrete distance helps a distribution of  pairs having the same angle distance to be diversiﬁed.
Revised: In Figure 3, we show that discrete distances of parameter pairs have different values while their continuous distances are same.

Original: In order to maximize the distance between the parameters, maximization of discrete distance could helpthe distribution of parameters diverse.
Revised: Hence, maximization of discrete distances of parameter pairs could help parameters distributed isolatedly and diversely. w ( l ) s,g k ,i ) =

Original:  The center vector for the group g k can be determined by a parameter vector at the ( l − 1) − th level using an adjust factor ( (cid:15) , | (cid:15) | < 1 ): w ( l,l − 1) c,g k [Equation]
Revised: 1 | g k | (cid:80) | g k | i w ( l ) s,g k ,i . The center vector for the group g k can be determined by a parameter vector at the ( l − 1) − th level using an adjust factor ( (cid:15) , | (cid:15) | < 1 ): w ( l,l − 1) c,g k = (cid:15) · w ( l − 1) g k (cid:48) ,i , where w ( l − 1) g k (cid:48) ,

Original: ,i , where w ( l [Equation] k [Equation]
Revised: i ∈ S d w ( l − 1) c,g (cid:48) k .

Original: Using parameter vectors w i and w j in R d
Revised: Using parameter vectors w i and w j in R d

Original: +1 , we deﬁne a discrete distance metric using a sign function as follows:
Revised: +1 , we deﬁne a discrete distance metric using a sign function as follows:

Original: [Equation]
Revised: [Equation]

Original: where sign ( x ) := (cid:110) 1 , if x≥ 0 − 1 ,
Revised: where sign ( x ) := (cid:110) 1 , if x

Original: [Equation]
Revised: [Equation]

Original:  , − 1 ≤ D h ≤ 1 , and w = { w ( k ) | ∀ k
Revised: − 1 , otherwise , − 1 ≤ D h ≤ 1 , and w = { w ( k ) | ∀ k

Original: = 1 , . . .
Revised: = 1 , . . .

Original: , d } ∈ R d +1 } .
Revised: , d } ∈ R d +1 } .

Original: This is a normalized version of
Revised: This is a normalized version of

Original: Hamming distance.
Revised: Hamming distance.

Original: For a ternary discrete, {− 1 , 0 , 1 } is used.
Revised: For a ternary discrete, {− 1 , 0 , 1 } is used.

Original: In order to consider the discrete distance as an angle distance within [0 , 1] , normalized one is deﬁned as D h 01 :
Revised: In order to consider the discrete distance as an angular distance within [0 , 1] , normalized one is deﬁned as D h 01 :

Original: =
Revised: =

Original: − D h
Revised: − D h

Original: +1, 0 ≤ D h 01 ≤ 1 .
Revised: +1, 0 ≤ D h 01 ≤ 1 .

Original: The angle distance based on the above product can be rephrased as θ D h
Revised: The angular distance based on the above product can be rephrased as θ D h

Original: = D h 01where 0 ≤ θ D h ≤ 1 .
Revised: = D h 01where 0 ≤ θ D h ≤ 1 .

Original: As the discrete distance could underestimate to approximate the model distribution, we merge the above discrete distance metric with continuous angle distance metric (θ = 1 π arccos
Revised: As the discrete distance could underestimate to approximate the model distribution, we merge the above discrete distance metric with continuous angular distance metric (

Original: [Equation]
Revised: [Equation]

Original: w i
Revised: arccos

Original: [Equation]
Revised: [Equation]

Original: j [Equation] w i (cid:107)(cid:107) w j (cid:107) ) , 0 ≤ θ ≤ 1 ) into the single metric.
Revised: ) into a single metric.

Original: We simply use the deﬁnition of Pythagorean means which consist of the arithmetic mean (AM), the geometric mean (GM), and the harmonic mean (HM).
Revised: We simply use the deﬁnition of Pythagorean means which consist of the arithmetic mean (AM), the geometric mean (GM), and the harmonic mean (HM).

Original: Pythagorean means using the above angle pair is deﬁned as follows:
Revised: Pythagorean means using a pair of angular distances are deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: In the angular distance 5 using { θ D h , θ } , a reversed form 1 − D { θ Dh ,θ } is adopted to maximize an angle in optimization formulation as a form of minimization instead of ( · ) − s where s = 1 , 2 , . . .
Revised: In the angular distance 5 using { θ D h , θ } , a reversed form 1 − D { θ Dh ,θ } is adopted to maximize an angle in optimization formulation as a form of minimization instead of ( · ) − s where s = 1 , 2 , . . .

Original: which is used in Thomson problem that utilizes s -energy (Brauchart & Grabner, 2015).
Revised: which is used in Thomson problem that utilizes s -energy (Brauchart & Grabner, 2015).

Original: The cosine similarity of these angles can be deﬁned as follows:
Revised: The cosine similarity of these angles can be deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: then the cosine similarity functions are normalized with cos( · )+ to have adistance value within  [0 , 1] .
Revised: then the cosine similarity functions are normalized with cos( · )+ to have a value within a range [0 , 1] .

Original: Finally, Pythagorean means of each cosine similarity can be calculated as follows:
Revised: Finally, Pythagorean means of each cosine similarity can be calculated as follows:

Original: [Equation]
Revised: [Equation]

Original: The above metric functions deﬁned in(8), (9),  and (10) satisfy the metric conditions: non-negativity, symmetry, and triangle inequality.
Revised: The above metric functions deﬁned in (9), (10), and (11) satisfy the metric conditions: nonnegativity, symmetry, and triangle inequality.

Original: The distance using the above metric functions between any two points is bounded, because the hypersphere is a compact manifold.
Revised: The distance using the above metric functions between any two points is bounded, because the hypersphere is a compact manifold.

Original: 3. G RADIENTS AND B ACKPROPAGATION
Revised: 3. G RADIENTS AND B ACKPROPAGATION

Original: As the sign function is not differentiable at the value 0, we adopt alternative backpropagation function.
Revised: As the sign function is not differentiable at the value 0, we adopt alternative backpropagation function.

Original: We adopt straight-through estimator (STE) (Bengio et al., 2013) in the backward path of the neural networks for the sign function in the discrete metric.
Revised: We adopt straight-through estimator (STE) (Bengio et al., 2013) in the backward path of the neural networks for the sign function in the discrete metric.

Original: The derivative of the sign function is substituted with 1 | w |≤ 1 in the backward pass, known as the saturated STE.
Revised: The derivative of the sign function is substituted with 1 | w |≤ 1 in the backward pass, known as the saturated STE.

Original: As the derivative of arccos( x ) ( − 1 √ 1 − x 2 ) is undeﬁned at the value x = ± 1 , we apply clamping to the cosine function to have x ∈ [ − 0 .
Revised: As the derivative of arccos( x ) ( − 1 √ 1 − x 2 ) is undeﬁned at the value x = ± 1 , we apply clamping to the cosine function to have x ∈ [ − 0 .

Original: 99 , 0 . 99] 6 where x = cos( θπ ) , 0 ≤ θ ≤ 1 .
Revised: 99 , 0 . 99] 6 where x = cos( θπ ) , 0 ≤ θ ≤ 1 .

Original:  E XPERIMENTS 4. E XPERIMENTAL SETUP
Revised:  E XPERIMENTS 4. E XPERIMENTAL SETUP

Original: Datasets We conduct experiments using four publicly available datasets including small size images (CIFAR-10 and CIFAR-100) and large size images (CUB200-2011 (Wah et al., 2011) and
Revised: Datasets We conduct experiments using four publicly available datasets including small size images (CIFAR-10 and CIFAR-100) and large size images (CUB200-2011 (Wah et al., 2011) and

Original: Stanford-Cars (Krause et al., 2013b), shortly CUB200 and Cars respectively).
Revised: Stanford-Cars (Krause et al., 2013b), shortly CUB200 and Cars respectively).

Original: CIFAR-10 dataset is used to validate effectiveness of the proposed metric without hierarchical information.
Revised: CIFAR-10 dataset is used to validate effectiveness of the proposed metric without hierarchical information.

Original: Except
Revised: Except

Original: CIFAR-10, we use two-level hierarchy pairs { parent, child } for the proposed hierarchical regularization on three datasets.
Revised: CIFAR-10, we use two-level hierarchy pairs { parent, child } for the proposed hierarchical regularization on three datasets.

Original: In Table 7 at Appendix, statics of datasets in detail is provided.
Revised: In Table 7 at Appendix, statics of datasets in detail is provided.

Original: Deep neural network models and training setting We used the deep residual neural networks ( resnet )
Revised: Deep neural network models and training setting We used the deep residual neural networks ( resnet )

Original: (He et al., 2016) with different sizes of parameters (weights) dependent on an image size of datasets.
Revised: (He et al., 2016) with different sizes of parameters (weights) dependent on an image size of datasets.

Original: For a small size input ( 32 × 32 pixels) such on CIFAR-10 and CIFAR-100, we used resnet with smaller number of parameters (light models, resnet-20 (0.29M) and resnet-110 (1.73 M))
Revised: For a small size input ( 32 × 32 pixels) such as CIFAR-10 and CIFAR-100, we used resnet with smaller number of parameters (light models, resnet-20 (0.29M) and resnet-110 (1.73 M))

Original: so as not to have redundant parameters leading to over-ﬁtting.
Revised: so as not to have redundant parameters leading to over-ﬁtting.

Original: For a ﬁne-grained input ( 224 × 224 pixels) such on CUB200 and Cars, The original resnet with larger number of parameters (heavy models 7 ,
Revised: For a ﬁne-grained input ( 224 × 224 pixels) such as CUB200 and Cars, the original resnet with larger number of parameters (heavy models 7 ,

Original: Resnet-18 (11.28M) and Resnet-50 (23.91M)).
Revised: Resnet-18 (11.28M) and Resnet-50 (23.91M)) is used.

Original: In training  with the hierarchical regularization, we assume that the global hierarchical structure is not given.
Revised: In training of the deep neural network with the hierarchical regularization, we assume that the global hierarchical structure is not given.

Original: Instead, stochastic or partial hierarchical structure is shown within each mini-batch by the hierarchy pairs.
Revised: Instead, stochastic or partial hierarchical structure is shown within each mini-batch by the hierarchy pairs.

Original: Mini-batches, 512 for light models and 256 for heavy models, are used in the SGD optimizer.
Revised: Mini-batches, 512 for light models and 256 for heavy models, are used in the SGD optimizer.

Original: We applied hierarchical regularization in the FC layer of the resnet.
Revised: We applied hierarchical regularization in the FC layer of the resnet.

Original: Even though SGD is known as an unbiased estimation, a stochastic hierarchical structure could affect the overall approximation performance upon the class distribution within the minibatch.
Revised: Even though SGD is known as an unbiased estimation, a stochastic hierarchical structure could affect the overall approximation performance upon the class distribution within the mini-batch.

Original: Settings in more detail are provided in Appendix.
Revised: Settings in more detail are provided in Appendix.

Original: Baseline setting In terms of the regularization, we used three  settings, called ‘baseline’, ‘baseline+ l 2 ’, and ‘E’ for a reference.
Revised: Baseline setting In terms of the regularization, we used three reference settings, called ‘baseline’, ‘baseline+ l 2 ’, and ‘E’ for a reference.

Original: First, in ‘baseline’, we examine neural networks without regularization.
Revised: First, in ‘baseline’, we examine neural networks without regularization.

Original: Then, in ‘baseline+ l 2 ’, we adopt l 2 -norm minimization regularization (which is equivalent to a weight decay in SGD setting (Zhang et al., 2019)) via λ f (cid:80) k (cid:107) w k (cid:107) , where w k ∈ W and λ f > 0 in addition to ‘baseline’.
Revised: Then, in ‘baseline+ l 2 ’, we adopt l 2 -norm minimization regularization (which is equivalent to a weight decay in SGD setting (Zhang et al., 2019)) via λ f (cid:80) k (cid:107) w k (cid:107) , where w k ∈ W and λ f > 0 in addition to ‘baseline’.

Original: Lastly, in ‘E’, energy minimization (Liu et al., 2018), diversity promoting regularization without hierarchical information, via λ e (cid:80) i (cid:54) =
Revised: Lastly, in ‘E’, energy minimization (Liu et al., 2018), diversity promoting regularization without hierarchical information, via λ e (cid:80) i (cid:54) =

Original: j d ( w i , w j ) , where d ( · , · ) is a pairwise distance and λ e > 0 , is added to baseline+ l 2 .
Revised: j d ( w i , w j ) , where d ( · , · ) is a pairwise distance and λ e > 0 , is added to baseline+ l 2 .

Original: When the proposed hierarchical regularization (Eq. 4) is applied on the FC layer, we named it as ‘H’.
Revised: When the proposed hierarchical regularization (Eq. 5) is applied on the FC layer, we named it as ‘H’.

Original: Hence, the regularization is incrementally adopted to the baseline: l 2 +‘E’+‘H’.
Revised: Hence, the regularization is incrementally adopted to the baseline: l 2 +‘E’+‘H’.

Original: In terms of a distance metric, we compared the proposed metric with other continuous metrics such as Euclidean distance with the unit-length projection (‘N-euclidean2’, (cid:80) i (cid:54) =
Revised: In terms of a distance metric, we compared the proposed metric with other continuous metrics such as Euclidean distance with the unit-length projection (‘N-euclidean2’, (cid:80) i (cid:54) =

Original: j (cid:107) w i (cid:107) w i (cid:107) − w i (cid:107) w i (cid:107) (cid:107) − 2 ) 8 , angular distance (‘angular2’, (cid:80) i (cid:54)=
Revised: j (cid:107) w i (cid:107) w i (cid:107) − w i (cid:107) w i (cid:107) (cid:107) − 2 ) 8 , angular distance (‘angular2’, (cid:80) [Equation] (cid:54)

Original: j arccos ( w i · w j (cid:107) w i (cid:107)(cid:107) w j (cid:107) ) − 2 ) where ‘2’ is from Riesz s -energy shown higher accuracy, and cosine distance (‘cosine’, 1 − (cid:80) i (cid:54) =
Revised: [Equation] arccos ( [Equation]

Original: j w i · w j (cid:107) w i (cid:107)(cid:107) w j (cid:107) ).
Revised: (cid:107) w [Equation] (cid:107)(cid:107) w j (cid:107) )

Revised: − 2 ) where ‘2’ is from Riesz s -energy shown higher accuracy, and cosine similarity (‘cosine’, (cid:80) i (cid:54) =

Revised: j

Revised: [Equation]

Original: 4. R ESULTS
Revised: 4. R ESULTS

Original: Low resolution object classiﬁcation In Table 

Revised: Low resolution object classiﬁcation In Table

Original: , we showed test accuracy ( % ) using different metrics with energy minimization regularization without hierarchical regularization 9 along a weight decay ( l 2 -norm) using CIFAR-10 and resnet-20.
Revised: , we showed test accuracy ( % ) using different metrics with energy minimization regularization without hierarchical regularization 9 along a weight decay ( l 2 -norm) using CIFAR-10 and resnet-20.

Original: ‘baseline’ method is examined without or with the weight decay.
Revised: ‘baseline’ method is examined without or with the weight decay ( l 2 ).

Original: The discrete angular metric based regularization ( D ter h (ternary code), D bin h (binary code), D cos( HM ) , and D (cid:48) HM cos ) on the baseline can improve the generalization performance in term of test accuracy compared to the other continuous metrics such as N-euclidean2, angular2, and cosine.
Revised: The discrete angular metric based regularization ( D ter h (ternary code), D bin h (binary code), D cos( HM ) , and D (cid:48) HM cos ) on the baseline can improve the generalization performance in term of test accuracy compared to the other continuous metrics such as N-euclidean2, angular2, and cosine.

Original: Due to the unit-length projection in N-euclidean2, their performance is comparable to that other angular metrics.
Revised: Due to the unit-length projection in N-euclidean2, their performance is comparable to that other angular metrics.

Original: D (cid:48) denotes a distance used D (cid:48)
Revised: D (cid:48) denotes a distance used D (cid:48)

Original: h from footnote 4 in Section 3.2.
Revised: h from footnote 4 in Section 3.2.

Original: The regularization terms are applied over all layers except the Batch Normalization (BN) layers.
Revised: The regularization terms are applied over all layers except the Batch Normalization (BN) layers.

Original: As explained in (van Laarhoven, 2017; Zhang et al., 2019), l 2 -norm minimization based regularization (weight decay) showed much improvement due to their effective learning rate in SGD.
Revised: As explained in (van Laarhoven, 2017; Zhang et al., 2019), l 2 -norm minimization based regularization (weight decay) showed much improvement due to their effective learning rate in SGD.

Original: As shown in Table 2, the regularization with proposed metrics shows better accuracy performance than that of the baseline on both resnet-20 and resnet-110 using CIFAR-100 dataset.
Revised: As shown in Table 2, the regularization with proposed metrics shows better accuracy performance than that of the baseline on both resnet-20 and resnet-110 using CIFAR-100 dataset.

Original: Comparing to the baseline+ l 2 , with energy minimization ‘E’ regularization, metrics such as D ter h , D cos( AM ) , and
Revised: Comparing to the baseline+ l 2 , with energy minimization ‘E’ regularization, metrics such as D ter h , D cos( AM ) , and

Original: D (cid:48) cos( HM ) showed better performance than other metrics.
Revised: D (cid:48) cos( HM ) showed better performance than other metrics.

Original: If the proposed hierarchical ‘H’ regularization is applied, the generalization performance is improved further in most metric cases on both
Revised: If the proposed hierarchical ‘H’ regularization is applied, the generalization performance is improved further in most metric cases on both

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: resnet-20 and resnet-110.
Revised: resnet-20 and resnet-110.

Original: As the binary metric shows a better performance than that ternary, we adopt binary discretization for proposed angular metrics ( D • , D cos( • ) , and D • cos ) in experiments.
Revised: As the binary metric shows a better performance than that ternary, we adopt binary discretization for proposed angular metrics ( D • , D cos( • ) , and D • cos ) in experiments.

Original: Fine-grained visual categorization In this experiment
Revised: Fine-grained visual categorization In this experiment

Original: , we used two large size image datasets with a ﬁne-grained category.
Revised: , we used two large size image datasets with a ﬁne-grained category.

Original: One is with birds (CUB200) and another is with cars (Cars) that focus on single species of objects.
Revised: One is with birds (CUB200) and another is with cars (Cars) that focus on single species of objects.

Original: A hierarchy of CUB200 is deﬁned by the academic expert on birds whereas that of Cars is categorized manually based on a name of cars by a non-expert.
Revised: A hierarchy of CUB200 is deﬁned by the academic expert on birds whereas that of Cars is categorized manually based on a name of cars by a non-expert.

Original: The rate between the number of superclass (parent) per subclass of CUB200 (0.35) is much larger than that of Cars (0.0459) (as shown in Table 7 at Appendix).
Revised: The rate between the number of superclass (parent) per subclass of CUB200 (0.35) is much larger than that of Cars (0.0459) (as shown in Table 7 at Appendix).

Original: That rate of Cars is even smaller than that of
Revised: That rate of Cars is even smaller than that of

Original: CIFAR-100.
Revised: CIFAR-100.

Original: As shown in Table 3, the proposed hierarchical regularization signiﬁcantly improve the test accuracy over all metrics on both Resnet-18 and Resnet-50.
Revised: As shown in Table 3, the proposed hierarchical regularization signiﬁcantly improve the test accuracy over all metrics on both Resnet-18 and Resnet-50.

Original: Compared to CUB200, as shown in Table 4, the improvement of the proposed method is not that signiﬁcant using Cars dataset.
Revised: Compared to CUB200, as shown in Table 4, the improvement of the proposed method is not that signiﬁcant for Cars dataset.

Original: This might be because CUB200 dataset has not only better the hierarchical categorization by the expert but also more dirversitiﬁed pairs between the superclass and the subclass.
Revised: This is because CUB200 dataset has not only better the hierarchical categorization by the expert but also more diversitiﬁed pairs between the superclass and the subclass.

Original: Ablation study In Table 5, we showed howthe metrics affect the generalization performance using resnet-20 and CIFAR-100 dataset..
Revised: Ablation study In Table 5, we showed how metrics affect the generalization performance using resnet-20 and CIFAR-100 dataset.

Original: The proposed method  showed signiﬁcantly improved performance compared to the baseline+ l 2 .
Revised: The proposed method with different proposed metrics showed signiﬁcantly improved performance compared to the baseline+ l 2 .

Original: Individual averaging settings (AM, GM, and HM) show different improvement patterns.
Revised: Individual averaging settings (AM, GM, and HM) show different improvement patterns.

Original: In Table 6, we showed how heterogeneous metrics between convolutional (conv.)
Revised: In Table 6, we showed how heterogeneous metrics between convolutional (conv.)

Original: layers (with the energy minimization, ‘E’) and fully connected (FC) layers (with the hierarchical ‘H’ regularization, ‘H’) affect the generalization performance using resnet-20 and CIFAR-100 dataset.
Revised: layers (with the energy minimization, ‘E’) and fully connected (FC) layers (with the hierarchical ‘H’ regularization, ‘H’) affect the generalization performance using resnet-20 and CIFAR-100 dataset.

Original: The cases applying hierarchical regularization showed better performance than that the baseline applying only energy minimization regularization.
Revised: The cases applying hierarchical regularization showed better performance than that the baseline applying only energy minimization regularization.

Original: In this experiment, a combination GM and HM shows a better improvement than that other combinations.
Revised: In this experiment, a combination GM and HM shows a better improvement than that other combinations.

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  R ELATED WORKS
Revised:  R ELATED WORKS

Original: Promoting of diversity on  embedding space or model parameters is  widely adopted concept in machine learning related area to improve the generalization performance (Cogswell et al., 2016;
Revised: Promoting of diversity on an embedding space or model parameters is an widely adopted strategy in machine learning related area to improve the generalization performance (Cogswell et al., 2016;

Original: Yang et al., 2019; Li et al., 2012; Ratzlaff & Fuxin, 2019; Xie et al., 2017b; 2018; 2017a;
Revised: Yang et al., 2019; Li et al., 2012; Ratzlaff & Fuxin, 2019; Xie et al., 2017b; 2018; 2017a;

Original: Liu et al., 2018).
Revised: Liu et al., 2018).

Original: The diversity exists at a variety of levels such as in a feature level (Cogswell et al., 2016; Xie et al., 2018), in a projection parameter level (Xie et al., 2017a; Liu et al., 2018), in a model ensemble level (Zhou et al., 2018; Ratzlaff & Fuxin, 2019), in a latent space model level (Ratzlaff & Fuxin, 2019; Liu et al., 2018), or in a generative model level (Yang et al., 2019; Ratzlaff & Fuxin, 2019).
Revised: This diversity based strategy can be applied in a variety of levels such as in a feature level (Cogswell et al., 2016; Xie et al., 2018), in a projection parameter level (Xie et al., 2017a; Liu et al., 2018), in a model ensemble level (Zhou et al., 2018; Ratzlaff & Fuxin, 2019), in a latent space model level (Ratzlaff & Fuxin, 2019; Liu et al., 2018), and in a generative model level (Yang et al., 2019; Ratzlaff & Fuxin, 2019).

Original: Among the above approaches, enlarging pairwise distance between features (Xie et al., 2018) requires computational efforts due to their covariance matrix.
Revised: Among these approaches, regularization methods require additional optimization effort. Enlarging pairwise distance between features (Xie et al., 2018) requires computational efforts due to their covariance matrix.

Original: In (Xie et al., 2017b),to optimize the solution, unit-eigenvalue is utilized via singular value decomposition.
Revised: In (Xie et al., 2017b), unit-eigenvalue is utilized via singular value decomposition.

Original: To optimize a direction and magnitude of the parameter vector alternatively, an alternating direction method of multipliers (ADMM)
Revised: To optimize a direction and magnitude of the parameter vector alternatively, an alternating direction method of multipliers (ADMM)

Original: (Xie et al., 2017a) is used.
Revised: (Xie et al., 2017a) is used.

Original: In terms of learning on hyperspherical space, (Liu et al., 2017) proposed that hyperspherical convolution (SphereCov) replaces the traditional inner-product based convolution in order to conduct learning angular representation on hyperspheres.
Revised: In terms of learning in hyperspherical space, (Liu et al., 2017) proposed that hyperspherical convolution (SphereCov) replaces the traditional inner-product based convolution in order to conduct learning angular representation on hyperspheres.

Original: To maximize distances between parameters, Minimum Hyperspherical Energy (Liu et al., 2018), is proposed to regularization methods to make parameters equidistributed globally.
Revised: In hyperspherical space, regularization via promoting diversity of parameters has been proposed (Liu et al., 2018) based on Minimum Hyperspherical Energy.

Original: In terms of hierarchical learning, (Nickel & Kiela, 2017) and (Ganea et al., 2018) proposed to apply hyperbolic spaces to embed the data in deep neural networks.
Revised: In terms of hierarchical learning, (Nickel & Kiela, 2017; Ganea et al., 2018) proposed to apply hyperbolic spaces to embed the data in deep neural networks.

Original: They showed that learning representations in the hyperbolic space is effective to the data preserving a latent hierarchy compared to non-Euclidean space.
Revised: They showed that learning representations in the hyperbolic space is effective to the data preserving a latent hierarchy compared to non-Euclidean space.

Original: Different from these works for representation learning, our proposed method is focused on regularization learning using explicit (predeﬁned) hierarchy information.
Revised: Different from these works for representation learning, our proposed method is focused on regularization learning using explicit (predeﬁned) hierarchy information.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We proposed the regularization method, which maximize pairwise distance between parameters preserving hierarchical structure.
Revised: We proposed the regularization method, which maximize pairwise distance between parameters preserving hierarchical structure.

Original: To deﬁne a hierarchical parameter space, we reformulated the topology space with multiple hypersphere spaces.
Revised: To deﬁne a hierarchical parameter space, we reformulated the topology space with multiple hypersphere spaces.

Original: On each hypersphere, a projection is parameterized by the surface parameter with the center parameter.
Revised: In each hypersphere, a projection is parameterized by the surface parameter with the center parameter.

Original: By imposing maximum a pairwise angular distance between the projection parameters, diversity of parameters preserving semantic structure was promoted.
Revised: By imposing maximum a pairwise angular distance between the projection parameters, diversity of parameters preserving semantic structure was promoted.

Original: For the angular distance metric in multiple separated spaces, we proposed a discrete metric integrated with a continuous metric.
Revised: For the angular distance metric in multiple separated spaces, we proposed a discrete metric integrated with a continuous metric.

Original: Extensive experiments using publicly available datasets, the deep neural network with our proposed regularization method showed superior classiﬁcation performance, especially when the number of super-classes is large.
Revised: Extensive experiments using publicly available datasets (CIFAR-10, CIFAR-100, CUB200-2011, and Stanford Cars), the deep neural network with our proposed regularization method showed superior classiﬁcation performance, especially when the number of super-classes is large.

Original: For further exploration in future, our proposed method can be combined with hierarchical representation learning such as hyperbolic (or Poincar ´ e) embeddings.
Revised: For further exploration in future, our proposed method can be combined with hierarchical representation learning such as hyperbolic (or Poincar´e) embeddings (Nickel & Kiela, 2017; Ganea et al., 2018).

Original: A PPENDIX
Revised: A PPENDIX

Original: Dataset acquisition details CIFAR-100 dataset provides their labels.
Revised: Dataset acquisition details CIFAR-100 dataset provides a pair of superclass-subclass labels.

Original: In CUB200 dataset, the pairs can be extracted from their ﬁlename.
Revised: In CUB200 dataset, the pairs can be extracted from their ﬁlename.

Original: In Cars dataset, we parsed each ﬁne label to one of nine coarse vehicle types, such as “Sedan”, “SUV”, “Van” and etc., following (Krause et al., 2013a).
Revised: In Cars dataset, we parsed each subclass label to one of nine superclass vehicle types, such as “Sedan”, “SUV”, “Van” and etc., following (Krause et al., 2013a).

Original: [Table]
Revised: [Table]

Original: Deep neural network models and training details First, resnet-20 (0.29M) and resnet-110 (1.
Revised: Deep neural network models and training details First, resnet-20 (0.29M) and resnet-110 (1.

Original: M), which include a combination of Basic blocks with output channels [16 , 30 , 64] are adopted for light models.
Revised: M), which include a combination of Basic blocks with output channels [16 , 30 , 64] are adopted for light models.

Original: An input dimensionality of the fully connected (FC) layer (a classier) is 64 for both resnet-20 and resnet-110.
Revised: An input dimensionality of the fully connected (FC) layer (a classier) is 64 for both resnet-20 and resnet-110.

Original: Second, heavy models are adopted such as Resnet-18 (11.28M 10 ) and
Revised: Second, heavy models are adopted such as Resnet-18 (11.28M 10 ) and

Original: Resnet-50 (23.91M) which consists of the basic blocks (Resnet-18) or the bottleneck blocks with 
Revised: Resnet-50 (23.91M) which consists of the basic blocks (Resnet-18) or the bottleneck blocks with output channels [64, 128, 256, 512] in Conv. layers.

Revised: An input dimensionality of the FC layer isfor Resnet-18 and 2048 for Resnet-50.

Original: The networks are optimized with SGD for both light and heavy models: we ﬁxed 1) the weight initialization with Random-Seed number 0 in pytorch, 2) learning rate schedule [0.1, 0.01, 0.001], 3) with momentum 0.9, 4) regularization: l 2 -norm minimization with λ f = 0 .
Revised: Networks are optimized with SGD for both light and heavy models: we ﬁxed 1) the weight initialization with Random-Seed number ‘0’ in pytorch, 2) learning rate schedule [0.1, 0.01, 0.001], 3) with momentum 0.9, 4) regularization: l 2 -norm minimization with λ f = 0 .

Original: 0005 , 5)orthogonalization λ o = 0 . 0001 , energy minimization λ
Revised: 0005 , 5) energy minimization λ

Original: e = { 0 . 1 , 1 } , and hierarchical minimization λ l = 0 .
Revised: e = { 0 . 1 , 1 } , and hierarchical minimization λ l = 0 .

Original: 1 × { 1 , 5 } .
Revised: 1 ×{ 1 , 5 } .

Original: All the regularization is not applied to the parameters of Batch Normalization (BN) layers, since BN is also a regularization function.
Revised: All the regularization is not applied to the parameters of Batch Normalization (BN) layers, since BN is also a regularization function.

Original: A bias term in the FC layer is not used.
Revised: A bias term in the FC layer is not used.

Original: The images in training and test, images are resized to 256 size.
Revised: The images in training and test, images are resized to 256 size.

Original: The images is cropped with 224 × 224 size at random location in training and at center location in test.
Revised: The images is cropped with 224 × 224 size at random location in training and at center location in test.

Original: Horizontal ﬂipping is applied in training.
Revised: Horizontal ﬂipping is applied in training.

Original: The light models are trained from scratch without the pretrained weights forepochs.
Revised: The light models are trained from scratch without the pretrained weights for 300 epochs.

Original: The heavy model is trained using pretrained model provided by pytorch library 11 withepochs.
Revised: The heavy model is trained using pretrained model provided by pytorch library 11 with 100 epochs.

Original: The experiments are conduced using GPU “NVIDIA TESLA P40”.
Revised: The experiments are conduced using GPU “NVIDIA TESLA P40”.

Original: [Equation]
Revised: [Equation]

Original: [Equation][Equation] [Equation] [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: (  w ( l,l − 1) c,g i , w ( l,l − 1) c,g j ) + d ( w ( l,l − 1) c,g j , w ( l,l − 1) c,g k ) . . 
Revised: ( l,l − 1) c,g k ) = d ( w ( l,l − 1) c,g i , w ( l,l − 1) c,g j ) + d ( w ( l,l − 1) c,g j , w ( l,l − 1) c,g k ) . . .

Original: corresponds to Eq (6)
Revised: corresponds to Eq (7)

Original: Constraints in Eq.
Revised: Constraints in Eq.

Original: 3 C l := (cid:80) k λ k C l,k can be given as follows:
Revised: 4 C l := (cid:80) k λ k C l,k can be given as follows:

Original: Constraint 1 ( C 1 ): This constraint describes that a radius of an inner sphere must be smaller than that of its outer sphere.
Revised: Constraint 1 ( C 1 ): This constraint describes that a radius of an inner sphere must be smaller than that of its outer sphere.

Original: {+r ( l − 1) − r ( l ) ≥ 
+}
Revised: {+r ( l − 1) − r ( l ) ≥ 
+}

Revised: ⇒ (cid:107) w ( l − 1) − w ( l )

Original: [Equation]
Revised: [Equation]

Revised: w (

Revised: [Equation]

Revised: −

Original:  Constraint 2 ( C 2 ): This constraint describes that a center of an inner sphere must be located in its outer sphere.
Revised: [Equation] ) s [Equation] Constraint 2 ( C 2 ): This constraint describes that a center of an inner sphere must be located within its outer sphere.

Revised: r ( l − 1) − ( (cid:107) w ( l,l − 1) c

Revised: [Equation]

Revised: l

Original: [Equation]
Revised: [Equation]

Revised: w ( l − 1) c (cid:107) − ( (cid:107) w ( l − 1) c − w ( l ) c (cid:107) + (cid:107) w ( l ) s − w ( l ) c (cid:107) ) ≥ 0 .

Original: Constraint 3 ( C 3 ): This constraint describes that a margin between spheres must be larger than zero.
Revised: Constraint 3 ( C 3 ): This constraint describes that a margin between spheres must be larger than zero.

Original: [Equation]{+(cid:107) w ( l,l − 1) c (cid:107) (2 − 2 cos θ ) 0 . 5 − 2 r ( l ) ≥ 
 ⇒ (cid:107) w ( l ) c (cid:107)+}
Revised: {+(cid:107) w ( l,l − 1) c (cid:107) (2 − 2 cos θ ) 0 . 5 − 2 r ( l ) ≥ 
 ⇒ (cid:107) w ( l ) c (cid:107)+}

Original: [Table] [Table]
Revised: [Table] [Table]