Original: D EEP H IERARCHICAL -H YPERSPHERICAL L EARNING (DH 2 L)
Revised: D EEP H IERARCHICAL -H YPERSPHERICAL L EARNING (DH 2 L)

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Regularization is known to be an inexpensive and reasonable solution to alleviate over-ﬁtting problems of inference models, including deep neural networks.
Revised: Regularization is known to be an inexpensive and reasonable solution to alleviate over-ﬁtting problems arising in inference models, including deep neural networks.

Original: In this paper, we propose a hierarchical regularization which preserves the semantic structure of a sample distribution.
Revised: In this paper, we propose a hierarchical regularization which preserves the semantic structure of a sample distribution.

Original: At the same time, this regularization promotes diversity by imposing distance between parameter vectors enlarged within semantic structures.
Revised: At the same time, this regularization promotes diversity by imposing distance between parameter vectors enlarged within semantic structures.

Original: To generate evenly distributed parameters, we constrain them to lie on hierarchical hyperspheres .Evenly distributed parameters are considered to be less redundant.
Revised: To generate evenly distributed parameters which is considered less redundant, we constrain them to lie on hierarchical hyperspheres .

Original: To deﬁne  hierarchical parameter space, we propose to reformulate the topology space with multiple hypersphere space.
Revised: To deﬁne a hierarchical parameter space, we propose to reformulate the topology space with multiple hyperspheres.

Original: On each hypersphere space, the projectionparameter is deﬁned by two individual parameters.
Revised: On each hypersphere, a projection is parameterized by two individual parameters.

Original: Since maximizing  groupwise  pairwise distance between points on hypersphere is nontrivial (generalized Thomson problem), we propose a new discrete metric integrated with continuous angle metric.
Revised: Since maximizing a groupwise and pairwise distance between points on hypersphere is nontrivial (generalized Thomson problem), we propose a new discrete metric integrated with continuous angular metric.

Original: Extensive experiments on  publicly available datasets (CIFAR-10, CIFAR-100, CUB200-2011, and Stanford Cars),our proposed method shows improved generalization performance, especially when the number of super-classes is larger.
Revised: The proposed method shows improved generalization performance on extensive experiments using publicly available datasets (CIFAR-10, CIFAR-100, CUB2002011, and Stanford Cars), especially when the number of super-classes is large.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Diversity promoting learning has been widely adopted via enlarging pairwise distances (Xie et al., 2018; 2017a;
Revised: Diversity promoting learning has been widely adopted via enlarging pairwise distances (Xie et al., 2018; 2017a;

Original: Liu et al., 2018), increasing orthogonality (Xie et al., 2018), reducing covariance between parameters (Xie et al., 2017b), or reducing correlation on feature (Cogswell et al., 2016) to improve generalization performance.
Revised: Liu et al., 2018), increasing orthogonality (Xie et al., 2018), reducing covariance between parameters (Xie et al., 2017b), or reducing correlation on feature (Cogswell et al., 2016) to improve generalization performance.

Original: Among them, diversity promoting regularization (Xie et al., 2017a) (Xie et al., 2017b) by enforcing large diversity between projection parameters achieves a reasonable performance without modifying the model structure.
Revised: Among them, diversity promoting regularization (Xie et al., 2017a;b; Liu et al., 2018) by enforcing large diversity between projection parameters achieves a reasonable performance without modifying the model structure.

Original: Optimizing the objective function with a covariance matrix in these methods is nontrivial.
Revised: Optimizing the objective function with a covariance matrix in (Xie et al., 2017a;b) is nontrivial.

Original: The diversity promoting regularization via minimizing energy of parameters of deep neural networks has been proposed (Liu et al., 2018).
Revised: In different circumstances, the diversity promoting regularization via minimizing energy of parameters of deep neural networks has been proposed (Liu et al., 2018).

Original: By minimizing a pairwise distance between parameters on hypersphere with the known metrics, they achieved the improved generalization performance.
Revised: By minimizing a pairwise distance between parameters on hypersphere with the known metric, they achieved the improved generalization performance.

Original: Following an efﬁcient regularization on hypersphere space we explore further this direction with three main concepts (hierarchical and hyperspherical learning with discrete metrics).
Revised: Following an efﬁcient regularization on hypersphere space, we explore further this direction with three main concepts (hierarchical and hyperspherical learning with discrete metrics).

Original: 1) Why hierarchical learning?
Revised: 1) Why hierarchical learning?

Original: Hierarchical inference explains human intelligence .
Revised: Hierarchical inference explains human intelligence .

Revised: In

Original: In (Kurzweil, 2013), it states that “the neocortex contains about300 million verygeneral pattern recognizers, arranged in a hierarchy”.
Revised: (Kurzweil, 2013) it states that “the neocortex contains about million very

Revised: general pattern recognizers, arranged in

Revised: a

Original:  Applying a hierarchy of multiple classes based on semantic taxonomy is a natural choice to devise machine intelligence .
Revised: hierarchy”. Applying a hierarchy of multiple classes based on semantic taxonomy is a natural choice to devise machine intelligence .

Original: Effectiveness of the hierarchical learning can be found in (Verma et al., 2012).
Revised: Effectiveness of the hierarchical learning can be found in (Verma et al., 2012)

Original: 2) Why hyperspherical learning?
Revised: 2) Why hyperspherical learning?

Original: Hypersphere can be represented by a centroid and a radius.
Revised: Hypersphere can be represented by a centroid and a radius.

Original: Due to the denominator in the unit-length normalization ( w (cid:107) w (cid:107) ), the distance deﬁned on the hypersphere converges when the magnitude of w goes inﬁnity while Euclidean distance goes inﬁnity.
Revised: Due to the denominator in the unit-length normalization ( w (cid:107) w (cid:107) ), the distance deﬁned on the hypersphere converges when the magnitude of w goes inﬁnity while Euclidean distance goes inﬁnity.

Original: Due to this bounded property,  hierarchical structure with multiple separated hyperspheres can be deﬁned.
Revised: Due to this bounded property, a hierarchical structure with multiple separated hyperspheres can be deﬁned.

Original: 3) Why discrete metric learning?
Revised: 3) Why discrete metric learning?

Original: If the vector points form discontinuous series with discrete representation (e.g. multi-dimensional binary or ternary), they are isolated from each other with a certain margin.
Revised: If the vector points form discontinuous series with discrete representation (e g. multi-dimensional binary or ternary), they are isolated from each other with a certain margin.

Original: This property may ﬁt with  disconnected manifold or  groupwise space problem.
Revised: This property may ﬁt with a disconnected manifold or a groupwise space problem.

Original: Moreover, to make points to be equidistributed where a pairwise distance is maximized is a nontrivial task.
Revised: Moreover, to make points to be equidistributed where a pairwise distance is maximized is a nontrivial task.

Original: Because of ﬁnite an isolated points, this metric may reduce search efforts to satisfy those constraints using a set of pairwise distance.
Revised: Because of ﬁnite and isolated points, this metric may reduce search efforts to satisfy those constraints using a set of pairwise distance.

Original: In this paper, we propose to apply hierarchical structure to parameter regularization on the multiple groupwise hyperspherical spaces.
Revised: In this paper, we propose to apply hierarchical structure to parameter regularization on the multiple groupwise hyperspherical spaces.

Original: In order to ﬁnd an appropriate metric on this space, we explore a discrete angluar metric.
Revised: In order to ﬁnd an appropriate metric on this space, we explore a discrete angular metric.

Original: We examine the proposed method on extensive experimental setups in terms of datasets and deep network models.
Revised: We examine the proposed method on extensive experimental setups in terms of datasets and deep network models.

Original:  M ULTIPLE SEPARATED HYPERSPHERES
Revised:  M ULTIPLE SEPARATED HYPERSPHERES

Original: Samples observed from the real world may be on disconnected manifold.
Revised: Samples observed from the real world may be on disconnected manifolds.

Original: In other words, disjoint union of those manifold could generate the global manifold (Lee, 2000).
Revised: In other words, disjoint union of those manifolds could generate the global manifold (Lee, 2000).

Original: In this section, we decompose the one space into multiple spaces (manifolds) and re-deﬁne the space in terms of hierarchical point of view.
Revised: In this section, we decompose the one space into multiple spaces (manifolds) and re-deﬁne the space in terms of hierarchical point of view.

Original: 2. D ISCONNECTED MANIFOLD VIA EQUIVALENT RELATIONS
Revised: 2. D ISCONNECTED MANIFOLD VIA EQUIVALENT RELATIONS

Original: Since it is not suitable to measure a pairwise distance between high dimensional vectors which have the hierarchical structure in the same space, we construct another identiﬁcation space which includes isolated spaces, from the original space (via equivalence relation (Tu, 2010)).
Revised: Since it is not suitable to measure a pairwise distance between high dimensional vectors which have the hierarchical structure in the same space, we construct another identiﬁcation space which includes isolated spaces, from the original space (via equivalence relation (Tu, 2010)).

Original: Denote d -sphere S d to be the set of points that satisﬁes S d = {
Revised: Denote d -sphere S d to be the set of points that satisﬁes S d = {

Original: w ∈ R d +1 : (cid:107) w (cid:107) = 1 } .
Revised: w ∈ R d +1 : (cid:107) w (cid:107) = 1 } .

Original: We construct multiple separated hyperspheres using multiple identifying relations.
Revised: We construct multiple separated hyperspheres using multiple identifying relations.

Original: In Figure 1, we use the center vector w c and the surface vector w s to deﬁne a hypershpere space and the projection parameter w .
Revised: In Figure 1, we use a center vector w c and a surface vector w s to deﬁne a hypershpere space and a projection parameter w .

Original: 2. P RIOR DISTRIBUTION AND REGULARIZATION
Revised: 2. P RIOR DISTRIBUTION AND REGULARIZATION

Original: To makethe parameter vectors uniformly distributed on the unit hypersphere, the vectors are sampled from the Gaussian normal distribution (Muller, 1959; Harman & Lacko, 2010).
Revised: To make parameter vectors uniformly distributed on the unit hypersphere, parameters are sampled from the Gaussian normal distribution (Muller, 1959; Harman & Lacko, 2010).

Original: This is because the normal distribution is spherically symmetric (Muller, 1959).
Revised: This is because the normal distribution is spherically symmetric (Muller, 1959).

Original: In a Bayesian point of view, neural networks with Gaussian priors are known to induce an l 2 -norm regularization (Vladimirova et al., 2019).
Revised: In a Bayesian point of view, neural networks with Gaussian priors are known to induce an l 2 -norm regularization (Vladimirova et al., 2019).

Original: From two evidences, we know that enforcing the parameters to have the Gaussian prior is important in hyperspherical learning in neural networks.
Revised: From two evidences, we know that enforcing the parameters to have the Gaussian prior is important in hyperspherical learning in neural networks.

Original: Note that a parameter which is calculated from the difference arithmetic operation with two parameters on the normal Gaussian distribution is on the normal difference distribution .
Revised: Note that a parameter which is calculated from the difference arithmetic operation with two parameters on the normal Gaussian distribution is on the normal difference distribution .

Original:  M ETHOD
Revised:  M ETHOD

Original: In deep neural networks, the objective function J with regularization R in addition to a loss L ,
Revised: In deep neural networks, the objective function J with regularization R in addition to a loss L ,

Original: J R ( W ) =
Revised: J R ( W ) =

Original: L ( x , W ) + λ R ( W ) , is optimized to ﬁnd the optimal W having a near minimum loss L , arg min W J R ( x , W ) , where x ∈ R d 0 denotes an input vector,
Revised: L ( x , W ) + λ R ( W ) , is optimized to ﬁnd the optimal W having a near minimum loss L , arg min W J R ( x , W ) , where x ∈ R d 0 denotes an input vector,

Original: W = {
Revised: W = {

Original: W i ∈ R
Revised: W i ∈ R

Original: d i − 1 × c
Revised: d i − 1 × c

Original: i : W i = { w j ∈
Revised: i : W i = { w j ∈

Original: R d i − 1 } , j = 1 , . . .
Revised: R d i − 1 } , j = 1 , . . .

Original: c i , i = 1 , . . .
Revised: c i , i = 1 , . . .

Original: , L } denotes a set of parameter matrices (i.e. neurons/kernels),
Revised: , L } denotes a set of parameter matrices (i.e. neurons/kernels),

Original: L denotes the number of layers, and λ > 0 is to control the degree of the regularization.
Revised: L denotes the number of layers, and λ > 0 is to control the degree of the regularization.

Original: For a classiﬁcation task, the cross entropy loss is used for the loss function L .
Revised: For a classiﬁcation task, the cross entropy loss is used for the loss function L .

Original: We propose a new regularization formulation R  in Section 3.1.
Revised: We propose a new regularization formulation R preserving a hierarchical parameter structure in Section 3.1.

Original: 3. R EGULARIZATION FOR H IERARCHICAL H YPERSPHERICAL H YPOTHESES
Revised: 3. R EGULARIZATION FOR H IERARCHICAL H YPERSPHERICAL H YPOTHESES

Original: Denote w aprojection parameter vector (an element of W at a single layer) to transform a given input into the embedding space deﬁned in a Euclidean metric space: x ∈ R d +1 (cid:55)→ w T x ∈ R .
Revised: Denote w a parameter vector (called a projection parameter, an element of W at a single layer) to transform a given input into an embedding space deﬁned in a Euclidean metric space: x ∈ R d +1 (cid:55)→ w T x ∈ R .

Original: By the deﬁnition of unit-length projection w (cid:107) w (cid:107) , a new parameter  ˆ w can be deﬁned on d -sphere: S d = { ˆ w ∈ R d +1 : (cid:107) ˆ w (cid:107)
Revised: By the deﬁnition of unit-length projection w (cid:107) w (cid:107) , a new parameter vector ˆ w can be deﬁned on d -sphere: S d = { ˆ w ∈ R d +1 : (cid:107) ˆ w (cid:107)

Original: = 1 } where (cid:107) · (cid:107) denotes l 2 -norm and the center is zero.
Revised: = 1 } where (cid:107) · (cid:107) denotes l 2 -norm and the center is zero.

Original: In other word, theprojection parameter vector ˆ w can be deﬁned by a center point vector w c ∈ R d +1 and a surface vector w s ∈ R d
Revised: In other word the parameter vector ˆ w can be deﬁned by a center point vector w c ∈ R d +1 and a surface vector w s ∈ R d

Original: +1 using an arithmetic operation:
Revised: +1 using an arithmetic operation:

Original: ˆ w := w s − w c .
Revised: ˆ w := w s − w c .

Original: We deﬁne the d -sphere withthe center and surface vector: S d w c = { w s − w c ∈ R d
Revised: We deﬁne the d -sphere with center and surface vectors: S d w c = { w s − w c ∈ R d

Original: +1 : (cid:107) w s − w c (cid:107) = 1 } .
Revised: +1 : (cid:107) w s − w c (cid:107) = 1 } .

Original: For a notation simplicity, we use w instead of ˆ w hereafter.
Revised: For a notation simplicity, we use w instead of ˆ w hereafter.

Original: While we consider a radius equals to 1 for simplicity, the parameter vector can have a radius r > 0 .
Revised: While we consider a radius equals to 1 for simplicity, the parameter vector can have a radius r > 0 .

Original: 3.1. H IERARCHICAL PARAMETERS DERIVED FROM LEVELWISE AND GROUPWISE CENTROID VECTORS
Revised: 3.1. H IERARCHICAL PARAMETERS DERIVED FROM LEVELWISE AND GROUPWISE CENTROID VECTORS

Original: We assume that the hierarchical structure consists of levelwise structure with a notation ( l ) and groupwise structure with a notation g below.
Revised: We assume that the hierarchical structure consists of levelwise structure with a notation ( l ) and groupwise structure with a notation g below.

Original: We explain these two concept to  parameter vectors serially.
Revised: We explain these two terms to deﬁne parameter vectors serially.

Original: Levelwise structure The above parameter vectors on S d w c can be deﬁned with the level-wise notation ( l ) as follows,
Revised: Levelwise structure The above parameter vector on S d w c can be deﬁned with the levelwise notation ( l ) as follows,

Original: [Equation]
Revised: [Equation]

Original: where the parameters are deﬁned on l -th d -sphere, S d w ( l ) c .
Revised: where the parameter is deﬁned on l -th d -sphere, S d w ( l ) c .

Original: In Figure 2, an example is provided in a lower dimension.
Revised: In Figure 2, an example is provided in a lower dimensional space.

Original: In this paper, we deﬁne the hierarchical parameters in a higher dimensional space than that of (b) and (c) in Figure 2.
Revised: In this paper, we deﬁne the hierarchical parameter in a higher dimensional space than that in (b) and (c) in Figure 2.

Original: In a levelwise setting, w ( l ) s and w ( l ) c are additively represented based on the center parameter (centroid) calculated from the previous level: w ( l − 1) c + −−→ ∆ w ( l ) (cid:55)→ w ( l ) c , where w ( l − 1) c
Revised: In a levelwise setting, w ( l ) s and w ( l ) c are additively represented based on the center parameter (centroid) calculated from the previous level: w ( l − 1) c [Equation] c , where w ( l [Equation]

Revised: 1) c =

Original: = (cid:80) l − 1 i −−→ ∆ w ( i ) is the accumulated center vector and −−→ ∆ w ( l ) denotes a newly connected parameter vector from w ( l − 1) c to w ( l ) c .
Revised: [Equation] ∆ w [Equation] ) is the accumulated center vector and −−→ ∆ w ( l ) denotes a newly connected parameter vector from w ( l − 1) c to w ( l ) c .

Original: By denoting −−→ ∆ w ( l ) as w ( l,l − 1) , the center vector at the l -level is deﬁned as, w ( l ) c :=
Revised: By denoting −−→ ∆ w ( l ) as w ( l,l − 1) , the center vector at the l -level is deﬁned as, w ( l ) c :=

Original: w ( l,l − 1) c
Revised: w ( l,l − 1) c

Original: + w ( l − 1) c , and the surface vector w ( l ) s :=
Revised: + w ( l − 1) c , and the surface vector w ( l ) s :=

Original: w
Revised: w

Original: [Equation]
Revised: [Equation]

Original:  + w 
Revised: l,l − 1) s + w [Equation]

Original: [Equation]
Revised: [Equation]

Original: Both the center vector and the surface vector at the current level are based on the center vector at the previous level 1 .
Revised: Both the center vector and the surface vector at the current level are based on the center vector at the previous level 1 .

Original: Hence, eq. (1) is equivalent to
Revised: Hence, Eq. 1 is equivalent to

Original: [Equation]
Revised: [Equation]

Original: Note that we use ( l, l − 1) to denote a connected parameter from the center parameter at the ( l − 1) th level to  ( l ) th level.
Revised: Note that we use ( l, l − 1) to denote a connecting parameter from the center parameter at the ( l − 1) th level to parameters at the ( l ) th level.

Original: Groupwise structure With a group notation g k ,Then the center parameter in eq.
Revised: Groupwise structure With a group notation g k , the center parameter in Eq.

Original: (1) can be rephrased as w ( l,l − 1) c,g k  on S d w ( l,l − 1) c,gk, which is d -sphere of g k group at l -th level, g ( l ) :
Revised: 2 can be rephrased as w ( l,l − 1) c,g k , on S d w ( l,l − 1) c,gk which denotes d -sphere of g k group at l -th level, g ( l ) :

Original: = { g k } | g ( l ) | k
Revised: = { g k } | g ( l ) | k

Original: =1 , g ( l ) ⊆
Revised: =1 . g ( l ) ⊆

Original: G ( l ) is a group set at the l th level, and | · | denotes the cardinality.
Revised: G ( l ) is a group set at the l th level, and | · | denotes the cardinality.

Original: A group g ( l ) at the current level is conditioned on a group at the previous level g ( l − 1) :=
Revised: A group g ( l ) at the l -th level is conditioned on a group at a parent ( l − 1) -th level, g ( l − 1) :

Original: [Equation] | [Equation]
Revised: = { g k (cid:48) } | g ( l − 1) | k (cid:48)

Original: = where g ( l − 1) ⊆
Revised: = where g ( l − 1) ⊆

Original: G ( l − 1) .
Revised: G ( l − 1) .

Original: With their groupwise relation over levels, an adjacency indication2 P ( l,l − 1) ( { G ( l − 1) , G ( l ) } ) ∈ { 0 , 1 } | G ( l − 1) |×| G ( l ) | calculated.
Revised: With their groupwise relation over levels, an adjacency indication P ( l,l − 1) ( { G ( l − 1) , G ( l ) } ) ∈ { 0 , 1 } | G ( l − 1) |×| G ( l ) | is given.

Original: Hence, the parameterprojection vector at the l th-level is determined as: w ( l ) g k ,i :=
Revised: Hence, the parameter vector at the lth level is determined as: w ( l ) g k ,i :=

Original: { w ( l,l − 1) s,g k ,i − w ( l,l − 1) c,g k} on S d w ( l,l − 1) c ,g k where i = 1 , . .
Revised: w ( l,l − 1) s,g k ,i − w ( l,l − 1) c,g k on S d w ( l,l − 1) c,gk where i = 1 , . .

Original: , | g k | , { w ( l,l − 1) s,g k  , w ( l,l − 1) c,g k }  is calculated based on w ( l − 1) c,g ( l − 1) referring totheir group condition, and the adjacency matrix P ( l,l − 1) .
Revised: , | g k | , { w ( l,l − 1) s,g k ,i , w ( l,l − 1) c,g k } ∀ i is calculated based on w ( l − 1) c,g ( l − 1) referring to the adjacency indication.

Original: A representative vector of the group g k at ( l ) level is  w ( l ) c,g k which is equivalent to a mean vector of w ( l ) s,g k ⇒ µ ( w ( l ) s,g k  ) 
Revised: A representative vector of the group g k at the l -th level is the center vector w ( l ) c,g k and it is equivalent to a mean vector of w ( l ) s,g k ,i ∀ i : µ ( w ( l ) s,g k ,i ) =

Original: = 1 | g k | (cid:80)
Revised: 1 | g k | (cid:80)

Original: | g k |  w ( l ) s,g k  .
Revised: | g k | i w ( l ) s,g k ,i 3 .

Original: If the representative vector for the group g k is determined by a certain parameter vector and the center vector at the previous level, then an adjust factor ( (cid:15) ) can be used: w ( l,l − 1) c,g k = w ( l − 1) c,g k (cid:48)

Original: + (cid:15) · w ( l − 1) g k (cid:48) ,i , where w ( l − 1) g k (cid:48) ,i ∈ S

Original: d w ( l − 1) c,g (cid:48) k .

Original: 3.1. H IERARCHICAL REGULARIZATION
Revised: 3.1. H IERARCHICAL REGULARIZATION

Original: In this section, we deﬁne a regularization term of the hierarchical parameter vectors deﬁned above.
Revised: In this section, we deﬁne a regularization term of the hierarchical parameter vectors deﬁned above.

Original: A set of parameters { W ( l,l − 1) s,g k , w ( l,l − 1) c,g k , w ( l − 1) c,g (cid:48)
Revised: A set of parameters { W ( l,l − 1) s,g k [Equation]

Original: k } ∈ W ∀ g k , ∀ g k (cid:48) where W ( l,l − 1) s,g k := { w ( l,l − 1) s,g k ,i } | g k | i
Revised: where W [Equation]

Original: =1 , is an optimizing target of hierarchical regularization term as follows:
Revised: is an optimizing target of hierarchical regularization term as follows:

Original: [Equation]
Revised: [Equation]

Original: where R l works on individual spheres S d w ( l,l − 1) c,gk , λ l∈ R > 0 , and C l aims to apply geometry-aware constraints across spheres.
Revised: where R l works on individual spheres S d w ( l,l − 1) c,gk , λ l > 0 , and C l aims to apply geometry-aware constraints across spheres.

Original: R l consists of two parts of regularization terms with: 1) R l,p for projection parameter vectors in the same group g k on S d w ( l,l − 1) c,gk and 2) R l,c for center parameter vectors across the groups in the same level on S d w ( l − 1) c,g (cid:48)k ,
Revised: R l consists of two parts of regularization terms with: 1) R l,p for projection parameter vectors in the same group g k on S d w ( l,l − 1) c,gk and 2) R l,c for center parameter vectors across the groups in the same level on S d w ( l − 1) c,g (cid:48)

Original: R l ( W ( l,l − 1) s,g k , w ( l,l − 1) c,g k ) : = R l,p ( W ( l,l − 1) s,g k , w ( l,l − 1) c,g k ) + R l,c ( w ( l,l − 1) c,g k) , (4) where R l,p ( W ( l,l − 1) s,g k , w ( l
Revised: k

Original: [Equation]
Revised: [Equation]

Original: k ) :
Revised: where

Original: = [Equation] [Equation]
Revised: R l,p

Original: [Equation]
Revised: [Equation]

Original: (cid:88) { g k ∈ g ( l ) } (cid:88) { i (cid:54) = j ∈ g k } d ( w ( l
Revised: l

Original: [Equation]
Revised: [Equation]

Original: l
Revised: l

Original: [Equation]
Revised: [Equation]

Original:  1) g k [Equation] i ,  w ( l
Revised: l,l − 1) g k ,j ) , (5) R l,c ( w ( l,l − 1) c,g

Original: [Equation]
Revised: [Equation]

Revised: ) := | C | ( | C | − 1) (cid:88)

Original: l − 1) g k
Revised: M d ( w ( l,l − 1) c,g

Original: [Equation]
Revised: [Equation]

Original: j ) ,(5) R l,c ( w ( l,l − 1) c,g k ) :=
Revised: , w ( l,l − 1) c

Original: C ( C − 1) (cid:88)
Revised: [Equation]

Original: M d ( w ( l,l − 1) c,g i , w ( l,l − 1) c,g j ) , (6) where w ( l,l − 1) g k ,i := w ( l,l − 1) s,g k ,i − w ( l,l − 1) c,g k , ( w ( l,l − 1) 
Revised: ) , (6) where w ( l,l − 1) g k ,i := w ( l,l − 1) s,g k ,i − w ( l,l − 1) c,g k , ( w ( l,l − 1) g k ,j is similarly deﬁned),

Original: g k ,i similarly deﬁned), G = |{ i (cid:54) =
Revised: G = { i (cid:54) =

Original: j ∈ g k }| , C = |{ g i (cid:54) =
Revised: j ∈ g k } , C = { g i (cid:54) =

Original: g j ∈ g ( l ) }| , and d ( · , · ) denotes a distance metric between the parameter vectors.
Revised: g j ∈ g ( l ) } , and d ( · , · ) denotes a distance metric between the parameter vectors.

Original: The distance metric is deﬁned in Section 3.2.
Revised: The distance metric is deﬁned in Section 3.2.

Original: When a (mini)batch of inputs is given, the regularization term becomes: E ( R ( W )) =

Original: [Equation] (cid:80) m x R ( W ; m x ) . We explain these parameter vectors and pairwise distances  in Figure 4 in Appendix.
Revised: We explain these parameter vectors and pairwise distances formulating regularization terms in Figure 4 in Appendix.

Original: In addition to the above hierarchical regularization in eq.

Original: (3), the orthogonality promoting term can be applied to the center vector w ( l,l − 1) c,g k

Original: : arg min W ( l,l − 1) c λ o (cid:107) W ( l,l − 1) c T W (l,l − 1) c − I (cid:107) F where W 
Revised: When a mini-batch of inputs ( m x : a set of inputs { x i } ) is given, the regularization term becomes: E ( R ( W )) =

Original: [Equation]

Original: l,

Original: [Equation]
Revised: [Equation]

Original: ) c ∈ R d ×| g k | , (cid:107) · (cid:107) F is the Frobenius norm and λ o > 0 .

Original: The parameters without the hierarchical information can adopt the magnitude ( l 2 -norm) minimization ( arg min w λ f (cid:80) k

Original: (cid:107) w k (cid:107) , where w k ∈ W and λ f > 0 .)

Original: and energy (pairwise distance) minimization ( arg min w (cid:80) i (cid:54) =

Original: j λ e d ( w i , w j ) , where λ e > 0 .).

Original: The constraint term help construct geometry-aware relational parameters between different spheres on the same level and on the across levels.
Revised: The constraint term help construct geometry-aware relational parameters between different spheres on the same level and on the across levels.

Original: Multiple constraints are deﬁned as C l := (cid:80) k λ k C l,k , where C l,k is k th constraint between parameters at l th and ( l − 1) th, and λ k > 0 is a Lagrange multiplier.
Revised: Multiple constraints are deﬁned as C l := (cid:80) k λ k C l,k , where C l,k is k th constraint between parameters at l th and ( l − 1) th, and λ k > 0 is a Lagrange multiplier.

Original: We apply three constraints in a geometric point of view.
Revised: We apply three constraints in a geometric point of view.

Original: The detailed formulation is deﬁned in appendix.
Revised: The detailed formulation is deﬁned in appendix.

Original: 3. D ISCRETE AND CONTINUOUS ANGULAR DISTANCE METRIC
Revised: 3. D ISCRETE AND CONTINUOUS ANGULAR DISTANCE METRIC

Original: Discrete (code) product metric might be a good ﬁt with the above group-wise deﬁnition.
Revised: Discrete (code) product metric might be a good ﬁt with the above group-wise deﬁnition.

Original: We expect that a projected point from the parameters formed in a discrete metric space, are isolated from each other.
Revised: We expect that a projected point from the parameters formed in a discrete metric space, are isolated from each other.

Original: In Figure 3, discrete distance helps a distribution of pairs having the same angle distance to be diversiﬁed.
Revised: In Figure 3, discrete distance helps a distribution of pairs having the same angle distance to be diversiﬁed.

Original: In order to maximize the distance between the parameters, maximization of discrete distance could help the distribution of parameters diverse.
Revised: In order to maximize the distance between the parameters, maximization of discrete distance could help the distribution of parameters diverse.

Original: To diversify a parameter space, the space with sign could be effective to recognize their difference.
Revised: The center vector for the group g k can be determined by a parameter vector at the ( l − 1) − th level using an adjust factor ( (cid:15) , | (cid:15) | < 1 ): w ( l,l − 1) c,g k [Equation] ,i , where w ( l [Equation]

Revised: k

Revised: [Equation]

Original: Using parameter vectors w i and w j on R d
Revised: Using parameter vectors w i and w j in R d

Original: +1 , we deﬁne a discrete distance metric using a sign function as follows:
Revised: +1 , we deﬁne a discrete distance metric using a sign function as follows:

Original: [Equation]
Revised: [Equation]

Original: where sign ( x ) := (cid:26) 1 , if x ≥− 1 , otherwise , − 1 ≤ D h ≤ 1 , and w = { w ( k ) | ∀ k
Revised: where sign ( x ) := (cid:110) 1 , if x ≥ 0 − 1 , [Equation] , − 1 ≤ D h ≤ 1 , and w = { w ( k ) | ∀ k

Original: = 1 , . . .
Revised: = 1 , . . .

Original: , d } ∈ R d +1 } .
Revised: , d } ∈ R d +1 } .

Original: This is a normalized version of
Revised: This is a normalized version of

Original: Hamming distance.
Revised: Hamming distance.

Original: For a ternary discrete, {− 1 , 0 , 1 } is used.
Revised: For a ternary discrete, {− 1 , 0 , 1 } is used.

Original: In order to consider the discrete distance as an angle distance within [0 , 1] , normalized one is deﬁned as D h 01 :
Revised: In order to consider the discrete distance as an angle distance within [0 , 1] , normalized one is deﬁned as D h 01 :

Original: =
Revised: =

Original: − D h
Revised: − D h

Original: +1, 0 ≤ D h 01 ≤ 1 .
Revised: +1, 0 ≤ D h 01 ≤ 1 .

Original: The angle distance based on the above product can be rephrased as θ D h
Revised: The angle distance based on the above product can be rephrased as θ D h

Original: = D h 01where 0 ≤ θ D h ≤ 1 .
Revised: = D h 01where 0 ≤ θ D h ≤ 1 .

Original: As the discrete distance could be limited to approximate the model distribution. We merge the above discrete distance metric with continuous angle distance metric ( θ = 1 π arccos(
Revised: As the discrete distance could underestimate to approximate the model distribution, we merge the above discrete distance metric with continuous angle distance metric ( θ = 1 π arccos

Original: [Equation]
Revised: [Equation]

Original:  i
Revised: w i

Original: [Equation]
Revised: [Equation]

Original: i
Revised: j

Original: [Equation]
Revised: [Equation]

Original:  j (cid:107) ) , 0 ≤ θ ≤ 1 ) into the single metric.
Revised: w i (cid:107)(cid:107) w j (cid:107) ) , 0 ≤ θ ≤ 1 ) into the single metric.

Original: We simply use the deﬁnition of Pythagorean means which consist of the arithmetic mean (AM), the geometric mean (GM), and the harmonic mean (HM).
Revised: We simply use the deﬁnition of Pythagorean means which consist of the arithmetic mean (AM), the geometric mean (GM), and the harmonic mean (HM).

Original: Pythagorean means using the above angle pair is deﬁned as follows:
Revised: Pythagorean means using the above angle pair is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: In the angular distance 4 using { θ D h , θ } , a reversed form 1 − D { θ Dh ,θ } is adopted to maximize an angle in optimization formulation as a form of minimization instead of ( · ) − s where s = 1 , 2 , . . .
Revised: In the angular distance 5 using { θ D h , θ } , a reversed form 1 − D { θ Dh ,θ } is adopted to maximize an angle in optimization formulation as a form of minimization instead of ( · ) − s where s = 1 , 2 , . . .

Original: which is used in Thomson problem that utilizes s -energy (Brauchart & Grabner, 2015).
Revised: which is used in Thomson problem that utilizes s -energy (Brauchart & Grabner, 2015).

Original: The cosine similarity of these angles can be deﬁned as follows:
Revised: The cosine similarity of these angles can be deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: (9) then the cosine similarity functions are normalized with cos( · )+ to have a distance value within [0 , 1] .
Revised: then the cosine similarity functions are normalized with cos( · )+ to have a distance value within [0 , 1] .

Original: Finally, Pythagorean means of each cosine similarity can be calculated as follows:
Revised: Finally, Pythagorean means of each cosine similarity can be calculated as follows:

Original: [Equation]
Revised: [Equation]

Original: The above metric functions deﬁned in (8), (9), and (10) satisfy the metric conditions: non-negativity, symmetry, and triangle inequality.
Revised: The above metric functions deﬁned in (8), (9), and (10) satisfy the metric conditions: non-negativity, symmetry, and triangle inequality.

Original: The distance using the above metric functions between any two points is bounded, because the hypersphere is a compact manifold.
Revised: The distance using the above metric functions between any two points is bounded, because the hypersphere is a compact manifold.

Original: 3. G RADIENTS AND B ACKPROPAGATION
Revised: 3. G RADIENTS AND B ACKPROPAGATION

Original: As the sign function is not differentiable at the value 0, we adopt alternative backpropagation function.
Revised: As the sign function is not differentiable at the value 0, we adopt alternative backpropagation function.

Original: We adopt straight-through estimator (STE) (Bengio et al., 2013) in the backward path of the neural networks for the sign function in the discrete metric.
Revised: We adopt straight-through estimator (STE) (Bengio et al., 2013) in the backward path of the neural networks for the sign function in the discrete metric.

Original: The derivative of the sign function is substituted with 1 | w |≤ 1 in the backward pass, known as the saturated STE.
Revised: The derivative of the sign function is substituted with 1 | w |≤ 1 in the backward pass, known as the saturated STE.

Original: As the derivative of arccos( x ) ( − 1 √ 1 − x 2 ) is undeﬁned at the value x = ± 1 , we apply clamping to the cosine function to have x ∈ [ − 0 .
Revised: As the derivative of arccos( x ) ( − 1 √ 1 − x 2 ) is undeﬁned at the value x = ± 1 , we apply clamping to the cosine function to have x ∈ [ − 0 .

Original: 99 , 0 . 99] 5 where x = cos( θπ ) , 0 ≤ θ ≤ 1 .
Revised: 99 , 0 . 99] 6 where x = cos( θπ ) , 0 ≤ θ ≤ 1 .

Original:  E XPERIMENTS 4. E XPERIMENTAL SETUP
Revised:  E XPERIMENTS 4. E XPERIMENTAL SETUP

Original: Datasets We conductthe experiments using four publicly available datasets including small size images (CIFAR-10 and CIFAR-100) and large size images (CUB200-2011 (Wah et al., 2011) and
Revised: Datasets We conduct experiments using four publicly available datasets including small size images (CIFAR-10 and CIFAR-100) and large size images (CUB200-2011 (Wah et al., 2011) and

Original: Stanford-Cars (Krause et al., 2013b), shortly CUB200 and Cars respectively).
Revised: Stanford-Cars (Krause et al., 2013b), shortly CUB200 and Cars respectively).

Original: CUB200 and Cars datasets are used for a ﬁne-grained visual categorization.
Revised: CIFAR-10 dataset is used to validate effectiveness of the proposed metric without hierarchical information.

Original: The ﬁne-grained visual categorization is challenging due to their high intra-class variances and low inter-class variances.
Revised: Except

Original: CIFAR-10 dataset is used to validate effectiveness of the proposed metric. Except CIFAR-10, we use two-level hierarchy pairs { parent, child } .
Revised: CIFAR-10, we use two-level hierarchy pairs { parent, child } for the proposed hierarchical regularization on three datasets.

Original: In Table 7 at Appendix, statics of datasets in detail is provided.
Revised: In Table 7 at Appendix, statics of datasets in detail is provided.

Original: Deep neural network models and training setting We adopt different size networks along the datasets. We adopt the deep residual network ( resnet )
Revised: Deep neural network models and training setting We used the deep residual neural networks ( resnet )

Original: (He et al., 2016) with smaller amount of parameters (light models, resnet-20 (0.29M) and resenet-110 (1.73 M)) for a small size input ( 32 ×pixels) such in CIFAR-10 and CIFAR-100 so as not to have redundant parameters leading to overﬁtting.
Revised: (He et al., 2016) with different sizes of parameters (weights) dependent on an image size of datasets. For a small size input ( 32 × 32 pixels) such on CIFAR-10 and CIFAR-100, we used resnet with smaller number of parameters (light models, resnet-20 (0.29M) and resnet-110 (1.73 M)) so as not to have redundant parameters leading to over-ﬁtting.

Original:  The original resnet with larger amount of parameters (heavy models, Resnet-18 (11.28M) and Resnet-50 (23.91M)) which is available from the pytorch library for a ﬁne-grained input ( 224 ×pixels) for CUB200 and Cars.
Revised: For a ﬁne-grained input ( 224 × 224 pixels) such on CUB200 and Cars, The original resnet with larger number of parameters (heavy models 7 , Resnet-18 (11.28M) and Resnet-50 (23.91M)).

Original: We applied hieararchical regularization in the FC layer.
Revised: In training with the hierarchical regularization, we assume that the global hierarchical structure is not given.

Revised: Instead, stochastic or partial hierarchical structure is shown within each mini-batch by the hierarchy pairs.

Original: Mini-batches, 512 for light models andfor heavy models, are used in the SGD optimizer.
Revised: Mini-batches, 512 for light models and 256 for heavy models, are used in the SGD optimizer.

Original: In training with the hierarchical regularization, we assume that the global hierarchical structure is not given.
Revised: We applied hierarchical regularization in the FC layer of the resnet.

Original: Instead, stochastic or partial hierarchical structure is given within the given mini-batch and the label pairs.
Revised: Even though

Original: Even though SGD is known as an unbiased estimation,  stochastic hierarchical pairs could affect the overall approximation performance upon a size of class pairs.
Revised: SGD is known as an unbiased estimation, a stochastic hierarchical structure could affect the overall approximation performance upon the class distribution within the minibatch.

Original: Settings in more detail are provided in Appendix.
Revised: Settings in more detail are provided in Appendix.

Original: 4. R ESULTS
Revised: Baseline setting In terms of the regularization, we used three settings, called ‘baseline’, ‘baseline+ l 2 ’, and ‘E’ for a reference. First, in ‘baseline’, we examine neural networks without regularization. Then, in ‘baseline+ l 2 ’, we adopt l 2 -norm minimization regularization (which is equivalent to a weight decay in SGD setting (Zhang et al., 2019)) via λ f (cid:80) k (cid:107) w k (cid:107) , where w k ∈ W and λ f > 0 in addition to ‘baseline’. Lastly, in ‘E’, energy minimization (Liu et al., 2018), diversity promoting regularization without hierarchical information, via λ e (cid:80) i (cid:54) =

Original: Object classiﬁcation The method with pairwise distance based (‘E’nergy) regularization (‘E’ in Table 1) performs better than the baseline as shown in Table 1.
Revised: j d ( w i , w j ) , where d ( · , · ) is a pairwise distance and λ e > 0 , is added to baseline+ l 2 . When the proposed hierarchical regularization (Eq. 4) is applied on the FC layer, we named it as ‘H’.

Original: The discrete angular metric based regularization ( D terh (ternary code), D binh (binary code), D cos( HM ) , and D (cid:48) HM cos ) can improve the generalization performance in term of test accuracy comparedto the  other  metrics such as angular2 ( (cid:80) i (cid:54) =
Revised: Hence, the regularization is incrementally adopted to the baseline: l 2 +‘E’+‘H’. In terms of a distance metric, we compared the proposed metric with other continuous metrics such as Euclidean distance with the unit-length projection (‘N-euclidean2’, (cid:80) i (cid:54) =

Original: j arccos ( w i · w j (cid:107)  w i (cid:107)(cid:107) w j (cid:107)) − 2 ), cosine ( (cid:80) i (cid:54) = j  w i · w j (cid:107) w i (cid:107)(cid:107) w j (cid:107) ), and N-euclidean ( (cid:80) i (cid:54) =
Revised: j (cid:107) w i (cid:107) w i (cid:107) − w i (cid:107) w i (cid:107) (cid:107) − 2 ) 8 , angular distance (‘angular2’, (cid:80) i (cid:54) = j arccos ( w i · w j (cid:107) w i (cid:107)(cid:107) w j (cid:107) ) − 2 ) where ‘2’ is from Riesz s -energy shown higher accuracy, and cosine distance (‘cosine’, 1 − (cid:80) i (cid:54) =

Original: j(cid:107) w i  (cid:107) w i  (cid:107) −
Revised: j w i · w j (cid:107) w i (cid:107)(cid:107) w j (cid:107) ).

Original: [Equation]
Revised: 4. R ESULTS

Original: (cid:107){+Low resolution object classiﬁcation In Table 
 , we showed test accuracy ( % ) using different metrics with energy minimization regularization without hierarchical regularization 9 along a weight decay ( l 2 -norm) using CIFAR-10 and resnet-20.+}
Revised: {+Low resolution object classiﬁcation In Table 
 , we showed test accuracy ( % ) using different metrics with energy minimization regularization without hierarchical regularization 9 along a weight decay ( l 2 -norm) using CIFAR-10 and resnet-20.+}

Original: [Equation]
Revised: ‘baseline’ method is examined without or with the weight decay.

Original: i (cid:107) (cid:107) − 2 ) 6 where a normalized version of Euclidean. ‘2’ is from Riesz s -energy and is set where higher accuracy is shown.
Revised: The discrete angular metric based regularization ( D ter h (ternary code), D bin h (binary code), D cos( HM ) , and D (cid:48) HM cos ) on the baseline can improve the generalization performance in term of test accuracy compared to the other continuous metrics such as N-euclidean2, angular2, and cosine.

Original: Due to the unit-length projection for Euclidean (Neuclidean2), their performance is comparable to that other angular metrics.
Revised: Due to the unit-length projection in N-euclidean2, their performance is comparable to that other angular metrics.

Original: D (cid:48) denotes a distance used D (cid:48)
Revised: D (cid:48) denotes a distance used D (cid:48)

Original: h from footnote 3 in Section 3.2.
Revised: h from footnote 4 in Section 3.2.

Original: The regularization terms are applied over convolutional layers and FC layers.
Revised: The regularization terms are applied over all layers except the Batch Normalization (BN) layers.

Original: As  l 2 -norm minimization based regularization shows much improvement, we use l 2 regularization by default for experiments. As there are many metrics proposed, in the table, more meaningful metrics are shown.
Revised: As explained in (van Laarhoven, 2017; Zhang et al., 2019), l 2 -norm minimization based regularization (weight decay) showed much improvement due to their effective learning rate in SGD.

Original: [Equation]
Revised: As shown in Table 2, the regularization with proposed metrics shows better accuracy performance than that of the baseline on both resnet-20 and resnet-110 using CIFAR-100 dataset. Comparing to the baseline+ l 2 , with energy minimization ‘E’ regularization, metrics such as D ter h , D cos( AM ) , and D (cid:48) cos( HM ) showed better performance than other metrics. If the proposed hierarchical ‘H’ regularization is applied, the generalization performance is improved further in most metric cases on both

Original: [Table] [Table]
Revised: [Table] [Table]

Original: As shown in Table

Original: [Table]
Revised: [Table]

Original: the regularization shows signiﬁcantly better performance than that of the baseline (without regularization) for both resnet-20 and resnet-100 on CIFAR-100 dataset. Comparing to the baseline l 2 , pairwise distance based ‘E’ regularization ( D terh , D cos( AM ) , D (cid:48) cos( HM ) ) shows better performance than other metrics. If the hierarchical ‘H’ regularization is applied, the generalization is improved further in most cases of both resnet-20 and resnet-110. As the binary metric shows a better performance than that ternary, we adopt binary discretization for the proposed discrete angular metrics ( D • , D cos( • ) , D • cos ) in the experiments.
Revised: [Table]

Original: Ablation study We experiment how the metrics affect the generalization performance. As shown in Table 3, the proposed method shows signiﬁcantly improved performance compared to the baseline ( l 2 ). Individual averaging settings (AM, GM, and HM) show different improvement patterns.
Revised: [Table]

Original: We examine to apply different metrics between convolutional layers (pairwise energy ‘E’ regularization) and fully connected layers (with hierarchical ‘H’ regularization) using resnet-20 and CIFAR100 datasets.
Revised: [Table] resnet-20 and resnet-110.

Original: Asshown in Table 4, the cases applying hierarchical regularization show better performance than the baseline applying only pairwise distance based ‘E’ regularization. In this experiment, a combination GM and HM shows a better improvement than that other combinations.
Revised: As the binary metric shows a better performance than that ternary, we adopt binary discretization for proposed angular metrics ( D • , D cos( • ) , and D • cos ) in experiments.

Original: Fine-grained visual categorization In this experiment
Revised: Fine-grained visual categorization In this experiment

Original: , we use two Fine-grained category datasets.
Revised: , we used two large size image datasets with a ﬁne-grained category.

Original: One is about the birds (CUB200) and another is about the cars (Cars) which focus on single species of object.
Revised: One is with birds (CUB200) and another is with cars (Cars) that focus on single species of objects.

Original: Based on species of Birds, pairs of { parent, child } are generated per sample by the academically from the expert. Birds are based on variety of characteristics, whereas the cars are categorizedby manually based on model names by  non-expert.
Revised: A hierarchy of CUB200 is deﬁned by the academic expert on birds whereas that of Cars is categorized manually based on a name of cars by a non-expert.

Original: The rate between the number of superclass (parent) per subclass of CUB200 (0.35) is much larger than that of Cars (0.0459) (as shown in Table 7 at Appendix).
Revised: The rate between the number of superclass (parent) per subclass of CUB200 (0.35) is much larger than that of Cars (0.0459) (as shown in Table 7 at Appendix).

Original: That rate of Cars is  smaller than that of CIFAR-100.
Revised: That rate of Cars is even smaller than that of CIFAR-100.

Original: As shown in Table 5, the proposed hierarchical regularization signiﬁcantly improve the test accuracy along the all metrics for both Resnet-18 and Resnet-50.
Revised: As shown in Table 3, the proposed hierarchical regularization signiﬁcantly improve the test accuracy over all metrics on both Resnet-18 and Resnet-50.

Original: Compared tothe CUB200, as shown in Table 6, the improvement of the proposed method is not that signiﬁcant in Cars dataset.
Revised: Compared to CUB200, as shown in Table 4, the improvement of the proposed method is not that signiﬁcant using Cars dataset.

Original: This might be because CUB200 dataset has more the hierarchical categorization cases of superclasses and subclasses pairs.
Revised: This might be because CUB200 dataset has not only better the hierarchical categorization by the expert but also more dirversitiﬁed pairs between the superclass and the subclass.

Original: [Table]
Revised: Ablation study In Table 5, we showed how the metrics affect the generalization performance using resnet-20 and CIFAR-100 dataset.. The proposed method showed signiﬁcantly improved performance compared to the baseline+ l 2 . Individual averaging settings (AM, GM, and HM) show different improvement patterns.

Original: [Table]
Revised: In Table 6, we showed how heterogeneous metrics between convolutional (conv.)

Original: [Table]
Revised: layers (with the energy minimization, ‘E’) and fully connected (FC) layers (with the hierarchical ‘H’ regularization, ‘H’) affect the generalization performance using resnet-20 and CIFAR-100 dataset. The cases applying hierarchical regularization showed better performance than that the baseline applying only energy minimization regularization. In this experiment, a combination GM and HM shows a better improvement than that other combinations.

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  R ELATED WORKS
Revised:  R ELATED WORKS

Original: Promoting of diversity on embedding space or model parameters is widely adopted concept in machine learning related area to improve the generalization performance (Cogswell et al., 2016), (Yang et al., 2019), (Li et al., 2012), (Ratzlaff & Fuxin, 2019), (Xie et al., 2017b), (Xie et al., 2018), (Xie et al., 2017a), (Liu et al., 2018).
Revised: Promoting of diversity on embedding space or model parameters is widely adopted concept in machine learning related area to improve the generalization performance (Cogswell et al., 2016; Yang et al., 2019; Li et al., 2012; Ratzlaff & Fuxin, 2019; Xie et al., 2017b; 2018; 2017a; Liu et al., 2018).

Original: The diversity exists at a variety of levels such as in  feature level (Cogswell et al., 2016; Xie et al., 2018), in  projection parameter level (Xie et al., 2017a; Liu et al., 2018), in  model ensemble level (Zhou et al., 2018; Ratzlaff & Fuxin, 2019), in  latent space model level (Ratzlaff & Fuxin, 2019; Liu et al., 2018), or in  generative model level (Yang et al., 2019; Ratzlaff & Fuxin, 2019).
Revised: The diversity exists at a variety of levels such as in a feature level (Cogswell et al., 2016; Xie et al., 2018), in a projection parameter level (Xie et al., 2017a; Liu et al., 2018), in a model ensemble level (Zhou et al., 2018; Ratzlaff & Fuxin, 2019), in a latent space model level (Ratzlaff & Fuxin, 2019; Liu et al., 2018), or in a generative model level (Yang et al., 2019; Ratzlaff & Fuxin, 2019).

Original: Throughout the existing work, in other point of views, the authors utilized enlarging pairwise distance between features or parameters (Xie et al., 2018; 2017a;

Original: Liu et al., 2018), increasing orthogonality (Xie et al., 2018), reducing covariance between projection parameters (Xie et al., 2017b), or reducing correlation on feature (Cogswell et al., 2016).

Original: Among the above approaches, enlarging pairwise distance between features  requires computational efforts due to their covariance matrix.
Revised: Among the above approaches, enlarging pairwise distance between features (Xie et al., 2018) requires computational efforts due to their covariance matrix.

Original: To optimize the solution,via singular value decomposition, unit-eigenvalue is utilized in (Xie et al., 2017b).
Revised: In (Xie et al., 2017b), to optimize the solution, unit-eigenvalue is utilized via singular value decomposition.

Original: From the non-convex optimization problem, another stabilization process such as convex relaxation (Xie et al., 2018) is utilized.

Original: To optimize a direction and magnitude of the parameter vector alternatively,they adopts an alternating direction method of multipliers (ADMM)
Revised: To optimize a direction and magnitude of the parameter vector alternatively, an alternating direction method of multipliers (ADMM)

Original: (Xie et al., 2017a).
Revised: (Xie et al., 2017a) is used.

Original: In terms of learning on hyperspherical space, (Liu et al., 2017) proposed that hyperspherical convolution (SphereCov) replaces the traditional inner-product based convolution in order to conduct learning angular representation on hyperspheres.
Revised: In terms of learning on hyperspherical space, (Liu et al., 2017) proposed that hyperspherical convolution (SphereCov) replaces the traditional inner-product based convolution in order to conduct learning angular representation on hyperspheres.

Original: By making magnitude of vectors during innerproduct operation, learning could be more efﬁcient and stable.

Original: To maximize distances between parameters, Minimum Hyperspherical Energy (Liu et al., 2018), is proposed to regularization methods to make parameters equidistributed globally.
Revised: To maximize distances between parameters, Minimum Hyperspherical Energy (Liu et al., 2018), is proposed to regularization methods to make parameters equidistributed globally.

Revised: In terms of hierarchical learning, (Nickel & Kiela, 2017) and (Ganea et al., 2018) proposed to apply hyperbolic spaces to embed the data in deep neural networks.

Revised: They showed that learning representations in the hyperbolic space is effective to the data preserving a latent hierarchy compared to non-Euclidean space.

Revised: Different from these works for representation learning, our proposed method is focused on regularization learning using explicit (predeﬁned) hierarchy information.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We proposed the regularization method, which utilizes pairwise and groupwise relation between parameters.
Revised: We proposed the regularization method, which maximize pairwise distance between parameters preserving hierarchical structure.

Original: To deﬁne a hierarchical parameter space, we reformulated the topology space with multiple hypersphere space.
Revised: To deﬁne a hierarchical parameter space, we reformulated the topology space with multiple hypersphere spaces.

Original: On each hypersphere,  projectionparameter is determined by the surface parameter at the center parameter, which is constructed from that of the previous level.
Revised: On each hypersphere, a projection is parameterized by the surface parameter with the center parameter.

Original: By imposing maximum  pairwise angular distance between the projection parameter vectors, diversity of parameters preserving semantic structure is promoted.
Revised: By imposing maximum a pairwise angular distance between the projection parameters, diversity of parameters preserving semantic structure was promoted.

Original: As the optimization process on hypersphere space is non-trivial, we proposed the discrete metric integrated with  continuous metric.
Revised: For the angular distance metric in multiple separated spaces, we proposed a discrete metric integrated with a continuous metric.

Original: Extensive experiments using publicly available datasets (CIFAR-10, CIFAR-100, CUB200-2011, and Stanford Cars), the deep neural network with our proposed regularization  showed superior classiﬁcation performance, especially when the number of super-classes is larger.
Revised: Extensive experiments using publicly available datasets, the deep neural network with our proposed regularization method showed superior classiﬁcation performance, especially when the number of super-classes is large. For further exploration in future, our proposed method can be combined with hierarchical representation learning such as hyperbolic (or Poincar ´ e) embeddings.

Original: A PPENDIX
Revised: A PPENDIX

Original: Dataset acquisition details CIFAR-100 dataset provides their labels.
Revised: Dataset acquisition details CIFAR-100 dataset provides their labels.

Original: In CUB200 dataset, the pairs can be extracted from their ﬁlename.
Revised: In CUB200 dataset, the pairs can be extracted from their ﬁlename.

Original: In Cars dataset, we parsed each ﬁne label to one of nine coarse vehicle types, such as “Sedan”, “SUV”, “Van” and etc., following (Krause et al., 2013a).
Revised: In Cars dataset, we parsed each ﬁne label to one of nine coarse vehicle types, such as “Sedan”, “SUV”, “Van” and etc., following (Krause et al., 2013a).

Original: [Table]
Revised: [Table]

Original: Deep neural network models and training details First, resnet-20 (0.29M) and resenet-110 (1.
Revised: Deep neural network models and training details First, resnet-20 (0.29M) and resnet-110 (1.

Original: M), which include a combination of Basic blocks with output channels [16 , 30 , 64] are adopted for light models.
Revised: M), which include a combination of Basic blocks with output channels [16 , 30 , 64] are adopted for light models.

Original: An input dimensionality of the fully connected (FC) layer (a classier) is 64 for both resnet-20 and resenet-110.
Revised: An input dimensionality of the fully connected (FC) layer (a classier) is 64 for both resnet-20 and resnet-110.

Original: Second, heavy models are adopted such as Resnet-18 (11.28M 7 ) and
Revised: Second, heavy models are adopted such as Resnet-18 (11.28M 10 ) and

Original: Resnet-50 (23.91M) which consists of the basic blocks (Resnet-18) or the bottelneck blocks withoutput channels [64, 128, 256, 512] in Conv. layers. An input dimensionality of the FC layer isfor Resnet-18 and 2048 for Resnet-50.
Revised: Resnet-50 (23.91M) which consists of the basic blocks (Resnet-18) or the bottleneck blocks with

Original: The networks are optimized with SGD for both light and heavy models: we ﬁxed i) the weight initialization with Random-Seed number 0 in pytorch, ii) learning rate schedule [0.1, 0.01, 0.001], 3) with momentum 0.9, 4) regularization: l 2 -norm minimization with λ f = 0 .
Revised: The networks are optimized with SGD for both light and heavy models: we ﬁxed 1) the weight initialization with Random-Seed number 0 in pytorch, 2) learning rate schedule [0.1, 0.01, 0.001], 3) with momentum 0.9, 4) regularization: l 2 -norm minimization with λ f = 0 .

Original: 0005 , 5) orthogonalization λ
Revised: 0005 , 5) orthogonalization λ

Original: o = 0 . 0001 , energy minimization λ
Revised: o = 0 . 0001 , energy minimization λ

Original: e = { 0 . 1 , 1 } , and hierarchical minimization λ l = 0 .
Revised: e = { 0 . 1 , 1 } , and hierarchical minimization λ l = 0 .

Original: 1 × { 1 , 5 } .
Revised: 1 × { 1 , 5 } .

Original: All the regularization is not applied to the parameters of BatchNorm layers.
Revised: All the regularization is not applied to the parameters of Batch Normalization (BN) layers, since BN is also a regularization function.

Original: A bias term in the FC layer is not used.
Revised: A bias term in the FC layer is not used.

Original: The images in training and test, images are resized to 256 size.
Revised: The images in training and test, images are resized to 256 size.

Original: The images is cropped  random location  and  location tal is applied in training.
Revised: The images is cropped with 224 × 224 size at random location in training and at center location in test. Horizontal ﬂipping is applied in training.

Original: The light models are trained from scratch without the pretrained weights for 300 epochs.
Revised: The light models are trained from scratch without the pretrained weights forepochs.

Original: The heavy model is trained using pretrained model provided by pytorch library 8 with 100 epochs.
Revised: The heavy model is trained using pretrained model provided by pytorch library 11 withepochs.

Original: The experiments are conduced using GPU “NVIDIA TESLA P40”.
Revised: The experiments are conduced using GPU “NVIDIA TESLA P40”.

Original: [Equation]
Revised: [Equation]

Original: [Equation] 
Revised: [Equation] [Equation] [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Revised: ( w ( l,l − 1) c,g i , w ( l,l − 1) c,g j )

Revised: + d ( w ( l,l − 1) c,g j , w ( l,l − 1) c,g k ) . .

Revised: corresponds to Eq (6)

Original: Constraints in Eq.
Revised: Constraints in Eq.

Original: (3) C l := (cid:80) k λ k C l,k can be given as follows:
Revised: 3 C l := (cid:80) k λ k C l,k can be given as follows:

Original: Constraint 1 ( C 1 ): This constraint describes that a radius of an inner sphere must be smaller than that of its outer sphere.
Revised: Constraint 1 ( C 1 ): This constraint describes that a radius of an inner sphere must be smaller than that of its outer sphere.

Original: [Equation]
Revised: [Equation]

Original: Constraint 2 ( C 2 ): This constraint describes that a center of an inner sphere must be located in its outer sphere.
Revised: Constraint 2 ( C 2 ): This constraint describes that a center of an inner sphere must be located in its outer sphere.

Original: [Equation]
Revised: [Equation]

Original: Constraint 3 ( C 3 ): This constraint describes that a margin between spheres must be larger than zero.
Revised: Constraint 3 ( C 3 ): This constraint describes that a margin between spheres must be larger than zero.

Original: [Equation]
Revised: [Equation]

Original: , where

Revised: [Table]

Original: [Equation]
Revised: [Table]