Original: C ONNECTING S PHERE M ANIFOLDS H IERARCHICALLY FOR R EGULARIZATION
Revised: C ONNECTING S PHERE M ANIFOLDS H IERARCHICALLY FOR R EGULARIZATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: This paper considers classiﬁcation problems with hierarchically organized classes.
Revised: This paper considers classiﬁcation problems with hierarchically organized classes.

Original: We force the classiﬁer (hyperplane) of each class to belong to a sphere manifold, whose center is the classiﬁer of its super-class.
Revised: We force the classiﬁer (hyperplane) of each class to belong to a sphere manifold, whose center is the classiﬁer of its super-class.

Original: Then, individual sphere manifolds are connected based on their hierarchical relations.
Revised: Then, individual sphere manifolds are connected based on their hierarchical relations.

Original: Our technique replaces the last layer of a neural network by combining a spherical fully-connected layer with a hierarchical layer.
Revised: Our technique replaces the last layer of a neural network by combining a spherical fully-connected layer with a hierarchical layer.

Original: This regularization is shown to improve the performance of widely used deep neural network architectures (ResNet and DenseNet) on publicly available datasets (CIFAR100, CUB200, Stanford dogs, Stanford cars, and Tiny-ImageNet).
Revised: This regularization is shown to improve the performance of widely used deep neural network architectures (ResNet and DenseNet) on publicly available datasets (CIFAR100, CUB200, Stanford dogs, Stanford cars, and Tiny-ImageNet).

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Applying inductive biases or prior knowledge to inference models is a popular strategy to improve their generalization performance (Battaglia et al., 2018).
Revised: Applying inductive biases or prior knowledge to inference models is a popular strategy to improve their generalization performance (Battaglia et al., 2018).

Original: For example, a hierarchical structure is found based on the similarity or shared characteristics between samples and thus becomes a basic criterion to categorize particular objects.
Revised: For example, a hierarchical structure is found based on the similarity or shared characteristics between samples and thus becomes a basic criterion to categorize particular objects.

Original: The known hierarchical structures provided by the datasets (e.g., ImageNet (Deng et al., 2009) classiﬁed based on the WordNet graph; CIFAR100 (Krizhevsky, 2009) in ten different groups) can help the network identify the similarity between the given samples.
Revised: The known hierarchical structures provided by the datasets (e.g., ImageNet (Deng et al., 2009) classiﬁed based on the WordNet graph; CIFAR100 (Krizhevsky, 2009) in ten different groups) can help the network identify the similarity between the given samples.

Original: In classiﬁcation tasks, the ﬁnal layer of neural networks maps embedding vectors to a discrete target space.
Revised: In classiﬁcation tasks, the ﬁnal layer of neural networks maps embedding vectors to a discrete target space.

Original: However, there is no mechanism forcing similar categories to be distributed close to each other in the embedding.
Revised: However, there is no mechanism forcing similar categories to be distributed close to each other in the embedding.

Original: Instead, we may observe classes to be uniformly distributed after training, as this simpliﬁes the separation by the last fully-connected layer.
Revised: Instead, we may observe classes to be uniformly distributed after training, as this simpliﬁes the separation by the last fully-connected layer.

Original: This behavior is a consequence of seeing the label structure as ‘ﬂat,’ i.e., when we omit to consider the hierarchical relationships between classes (Bilal et al., 2017).
Revised: This behavior is a consequence of seeing the label structure as ‘ﬂat,’ i.e., when we omit to consider the hierarchical relationships between classes (Bilal et al., 2017).

Original: To alleviate this problem, in this study, we force similar classes to be closer in the embedding by forcing their hyperplanes to follow a given hierarchy.
Revised: To alleviate this problem, in this study, we force similar classes to be closer in the embedding by forcing their hyperplanes to follow a given hierarchy.

Original: One way to realize that is by making children nodes dependent on parent nodes and constraining their distance through a regularization term.
Revised: One way to realize that is by making children nodes dependent on parent nodes and constraining their distance through a regularization term.

Original: However, the norm itself does not give a relevant information on the closeness between classiﬁers.
Revised: However, the norm itself does not give a relevant information on the closeness between classiﬁers.

Original: Indeed, two classiﬁers are close if they classify two similar points in the same class.
Revised: Indeed, two classiﬁers are close if they classify two similar points in the same class.

Original: This means similar classiﬁers have to indicate a similar direction.
Revised: This means similar classiﬁers have to indicate a similar direction.

Original: Therefore, we have to focus on the angle between classiﬁers, which can be achieved through spherical constraints.
Revised: Therefore, we have to focus on the angle between classiﬁers, which can be achieved through spherical constraints.

Original: Contributions.
Revised: Contributions.

Original: In this paper, we propose a simple strategy to incorporate hierarchical information in deep neural network architectures with minimal changes to the training procedure, by modifying only the last layer .
Revised: In this paper, we propose a simple strategy to incorporate hierarchical information in deep neural network architectures with minimal changes to the training procedure, by modifying only the last layer .

Original: Given a hierarchical structure in the labels under the form of a tree, we explicitly force the classiﬁers of classes to belong to a sphere, whose center is the classiﬁer of their super-class, recursively until we reach the root (see Figure 2).
Revised: Given a hierarchical structure in the labels under the form of a tree, we explicitly force the classiﬁers of classes to belong to a sphere, whose center is the classiﬁer of their super-class, recursively until we reach the root (see Figure 2).

Original: We introduce the spherical fully-connected layer and the hierarchically connected layer , whose combination implements our technique.
Revised: We introduce the spherical fully-connected layer and the hierarchically connected layer , whose combination implements our technique.

Original: Finally, we investigate the impact of Riemannian optimization instead of simple norm normalization.
Revised: Finally, we investigate the impact of Riemannian optimization instead of simple norm normalization.

Original: By its nature, the proposed technique is quite versatile because the modiﬁcations only affect the structure of last fully-connected layer of the neural network.
Revised: By its nature, the proposed technique is quite versatile because the modiﬁcations only affect the structure of last fully-connected layer of the neural network.

Original: Thus, it can be combined with many other strategies (like spherical CNN from Xie et al.
Revised: Thus, it can be combined with many other strategies (like spherical CNN from Xie et al.

Original: (2017), or other deep neural network architectures).
Revised: (2017), or other deep neural network architectures).

Original: Related works.
Revised: Related works.

Original: Hierarchical structures are well-studied, and their properties can be effectively learned using manifold embedding.
Revised: Hierarchical structures are well-studied, and their properties can be effectively learned using manifold embedding.

Original: The design of the optimal embedding to learn the latent hierarchy is a complex task, and was extensively studied in the past decade.
Revised: The design of the optimal embedding to learn the latent hierarchy is a complex task, and was extensively studied in the past decade.

Original: For example, Word2Vec (Mikolov et al., 2013b;a) and Poincaré embedding (Nickel & Kiela, 2017) showed a remarkable performance in hierarchical representation learning.
Revised: For example, Word2Vec (Mikolov et al., 2013b;a) and Poincaré embedding (Nickel & Kiela, 2017) showed a remarkable performance in hierarchical representation learning.

Original: (Du et al., 2018) forced the representation of sub-classes to “orbit” around the representation of their super-class to ﬁnd similarity based embedding.
Revised: (Du et al., 2018) forced the representation of sub-classes to “orbit” around the representation of their super-class to ﬁnd similarity based embedding.

Original: Recently, using elliptical manifold embedding (Batmanghelich et al., 2016), hyperbolic manifolds (Nickel &
Revised: Recently, using elliptical manifold embedding (Batmanghelich et al., 2016), hyperbolic manifolds (Nickel &

Original: Kiela, 2017; De Sa et al., 2018; Tifrea et al., 2018), and a combination of the two (Gu et al., 2019;
Revised: Kiela, 2017; De Sa et al., 2018; Tifrea et al., 2018), and a combination of the two (Gu et al., 2019;

Original: Bachmann et al., 2019), shown that the latent structure of many data was non-Euclidean (Zhu et al., 2016; Bronstein et al., 2017; Skopek et al., 2019).
Revised: Bachmann et al., 2019), shown that the latent structure of many data was non-Euclidean (Zhu et al., 2016; Bronstein et al., 2017; Skopek et al., 2019).

Original: (Xie et al., 2017) showed that spheres (with angular constraints) in the hidden layers also induce diversity, thus reducing over-ﬁtting in latent space models.
Revised: (Xie et al., 2017) showed that spheres (with angular constraints) in the hidden layers also induce diversity, thus reducing over-ﬁtting in latent space models.

Original: Mixing hierarchical information and structured prediction is not new, especially in text analysis (Koller & Sahami, 1997; McCallum et al., 1998; Weigend et al., 1999;
Revised: Mixing hierarchical information and structured prediction is not new, especially in text analysis (Koller & Sahami, 1997; McCallum et al., 1998; Weigend et al., 1999;

Original: Wang et al., 1999; Dumais &
Revised: Wang et al., 1999; Dumais &

Original: Chen, 2000).
Revised: Chen, 2000).

Original: Partial order structure of the visual-semantic hierarchy is exploited using a simple order pair with max-margin loss function in (Vendrov et al., 2016).
Revised: Partial order structure of the visual-semantic hierarchy is exploited using a simple order pair with max-margin loss function in (Vendrov et al., 2016).

Original: The results of previous studies indicate that exploiting hierarchical information during training gives better and more resilient classiﬁers, in particular when the number of classes is large (Cai & Hofmann, 2004).
Revised: The results of previous studies indicate that exploiting hierarchical information during training gives better and more resilient classiﬁers, in particular when the number of classes is large (Cai & Hofmann, 2004).

Original: For a given hierarchy, it is possible to design structured models incorporating this information to improve the efﬁciency of the classiﬁer.
Revised: For a given hierarchy, it is possible to design structured models incorporating this information to improve the efﬁciency of the classiﬁer.

Original: For instance, for support vector machines (SVMs), the techniques reported in (Cai &
Revised: For instance, for support vector machines (SVMs), the techniques reported in (Cai &

Original: Hofmann, 2004; 2007; Gopal et al., 2012; Sela et al., 2011) use hierarchical regularization, forcing the classiﬁer of a super-class to be close to the classiﬁers of its sub-classes.
Revised: Hofmann, 2004; 2007; Gopal et al., 2012; Sela et al., 2011) use hierarchical regularization, forcing the classiﬁer of a super-class to be close to the classiﬁers of its sub-classes.

Original: However, the intuition is very different in this case, because SVMs do not learn the embedding.
Revised: However, the intuition is very different in this case, because SVMs do not learn the embedding.

Original: In this study, we consider that the hierarchy of the class labels is known.
Revised: In this study, we consider that the hierarchy of the class labels is known.

Original: Moreover, we do not change prior layers of the deep neural network, and only work on the last layer that directly contributed to build hyperplanes for a classiﬁcation purpose.
Revised: Moreover, we do not change prior layers of the deep neural network, and only work on the last layer that directly contributed to build hyperplanes for a classiﬁcation purpose.

Original: Our work is thus orthogonal to those works on embedding learning, but not incompatible.
Revised: Our work is thus orthogonal to those works on embedding learning, but not incompatible.

Original: Comparison with hyperbolic/Poincaré/graph networks.
Revised: Comparison with hyperbolic/Poincaré/graph networks.

Original: Hyperbolic network is a recent technique that shows impressive results for hierarchical representation learning.
Revised: Hyperbolic network is a recent technique that shows impressive results for hierarchical representation learning.

Original: Poincaré networks (Nickel & Kiela, 2017) were originally designed to learn the latent hierarchy of data using low-dimension embedding.
Revised: Poincaré networks (Nickel & Kiela, 2017) were originally designed to learn the latent hierarchy of data using low-dimension embedding.

Original: To alleviate their drawbacks due to a transductive property which cannot be used for unseen graph inference, hyperbolic neural networks equipped set aggregation operations have been proposed (Chami et al., 2019; Liu et al., 2019).
Revised: To alleviate their drawbacks due to a transductive property which cannot be used for unseen graph inference, hyperbolic neural networks equipped set aggregation operations have been proposed (Chami et al., 2019; Liu et al., 2019).

Original: These methods have been mostly focused on learning embedding using a hyperbolic activation function for hierarchical representation.
Revised: These methods have been mostly focused on learning embedding using a hyperbolic activation function for hierarchical representation.

Original: Our technique is orthogonal to these works: First, we assume that the hierarchical structure is not learnt but already known.
Revised: Our technique is orthogonal to these works: First, we assume that the hierarchical structure is not learnt but already known.

Original: Second, our model focuses on generating individual hyperplanes of embedding vectors given by the network architecture.
Revised: Second, our model focuses on generating individual hyperplanes of embedding vectors given by the network architecture.

Original: While spherical geometry has a positive curvature, moreover, that of hyperbolic space has a constant negative curvature.
Revised: While spherical geometry has a positive curvature, moreover, that of hyperbolic space has a constant negative curvature.

Original: However, our technique and hyperbolic networks are not mutually exclusive .
Revised: However, our technique and hyperbolic networks are not mutually exclusive .

Original: Meanwhile focusing on spheres embedded in R d in this study, it is straightforward to consider spheres embedded in hyperbolic spaces.
Revised: Meanwhile focusing on spheres embedded in R d in this study, it is straightforward to consider spheres embedded in hyperbolic spaces.

Original:  H IERARCHICAL R EGULARIZATION 2. D EFINITION AND NOTATIONS
Revised:  H IERARCHICAL R EGULARIZATION 2. D EFINITION AND NOTATIONS

Original: we use the notation n p with p = { 1 , 3 , 2 } .
Revised: we use the notation n p with p = { 1 , 3 , 2 } .

Original: We use curly brackets {} to write a path, and angle brackets (cid:104)·(cid:105) for the concatenation of paths.
Revised: We use curly brackets {} to write a path, and angle brackets (cid:104)·(cid:105) for the concatenation of paths.

Original: We assume we have samples with hierarchically ordered classes.
Revised: We assume we have samples with hierarchically ordered classes.

Original: For instance, apple, banana, and orange are classes that may belong to the super-class “fruits.”
Revised: For instance, apple, banana, and orange are classes that may belong to the super-class “fruits.”

Original: This represents hierarchical relationships with trees, as depicted in Figure 1.
Revised: This represents hierarchical relationships with trees, as depicted in Figure 1.

Original: We identify nodes in the graph through the path taken in the tree.
Revised: We identify nodes in the graph through the path taken in the tree.

Original: To represent the leaf (highlighted in blue in Figure 1), we use the notation n { 1 , 3 , 2 } .
Revised: To represent the leaf (highlighted in blue in Figure 1), we use the notation n { 1 , 3 , 2 } .

Original: This means it is the second child of the super-class n { 1 , 3 } , and recursively, until we reach the root.
Revised: This means it is the second child of the super-class n { 1 , 3 } , and recursively, until we reach the root.

Original: More formally, we identify nodes as n p , where p is the path to the node.
Revised: More formally, we identify nodes as n p , where p is the path to the node.

Original: A path uniquely deﬁnes a node where only one possible path exists.
Revised: A path uniquely deﬁnes a node where only one possible path exists.

Original: Using the concatenation, between the path p and its child i , a new path ˜
Revised: Using the concatenation, between the path p and its child i , a new path ˜

Original: p can be deﬁned as follows,
Revised: p can be deﬁned as follows,

Original: [Equation]
Revised: [Equation]

Original: We denote P the set of all paths in the tree starting from the root, with cardinality |P| .
Revised: We denote P the set of all paths in the tree starting from the root, with cardinality |P| .

Original: Notice that |P| is also the number of nodes in the tree (i.e., number of classes and super-classes).
Revised: Notice that |P| is also the number of nodes in the tree (i.e., number of classes and super-classes).

Original: We distinguish the set P from the set L , the set of paths associated to nodes whose label appears in the dataset.
Revised: We distinguish the set P from the set L , the set of paths associated to nodes whose label appears in the dataset.

Original: Although
Revised: Although

Original: L may equal to P , this is not the case in our experiments.
Revised: L may equal to P , this is not the case in our experiments.

Original: We show an example in Appendix A.
Revised: We show an example in Appendix A.

Original: 2. S IMILARITY BETWEEN OBJECTS AND THEIR REPRESENTATION
Revised: 2. S IMILARITY BETWEEN OBJECTS AND THEIR REPRESENTATION

Original: Let X be the network input (e.g. an image), and φ θ ( X ) be its representation, i.e., the features of X extracted by a deep neural network parameterized by θ .
Revised: Let X be the network input (e.g. an image), and φ θ ( X ) be its representation, i.e., the features of X extracted by a deep neural network parameterized by θ .

Original: We start with the following observation:
Revised: We start with the following observation:

Original: Given a representation, super-class separators should be similar to separators for their sub-classes.
Revised: Given a representation, super-class separators should be similar to separators for their sub-classes.

Original: This assumption implies the following direct consequence.
Revised: This assumption implies the following direct consequence.

Original: All objects whose labels belong to the same super-class have a similar representation.
Revised: All objects whose labels belong to the same super-class have a similar representation.

Original: That is a natural property that we may expect from a good representation.
Revised: That is a natural property that we may expect from a good representation.

Original: For instance, two dogs from different breeds should share more common features than that of a dog shares with an apple.
Revised: For instance, two dogs from different breeds should share more common features than that of a dog shares with an apple.

Original: Therefore, the parameter of the classiﬁers that identify dog’s breed should also be similar.
Revised: Therefore, the parameter of the classiﬁers that identify dog’s breed should also be similar.

Original: Their difference lies in the parameters associated to some speciﬁc features that differentiate breeds of dogs.
Revised: Their difference lies in the parameters associated to some speciﬁc features that differentiate breeds of dogs.

Original: Although this is not necessarily satisﬁed with arbitrary hierarchical classiﬁcation, we observe this in many existing datasets.
Revised: Although this is not necessarily satisﬁed with arbitrary hierarchical classiﬁcation, we observe this in many existing datasets.

Original: For instance, Caltech-UCSD Birds 200 and Stanford dogs are datasets that classify, respectively, birds and dogs in term of their breeds.
Revised: For instance, Caltech-UCSD Birds 200 and Stanford dogs are datasets that classify, respectively, birds and dogs in term of their breeds.

Original: A possible example where this assumption may not be satisﬁed is a dataset whose super-classes are “labels whose ﬁrst letter is « · ».”
Revised: A possible example where this assumption may not be satisﬁed is a dataset whose super-classes are “labels whose ﬁrst letter is « · ».”

Original: 2. H IERARCHICAL REGULARIZATION
Revised: 2. H IERARCHICAL REGULARIZATION

Original: Starting from a simple observation in the previous section, we propose a regularization technique that forces the network to have similar representation for classes along a path p , which implies having similar representation between similar objects.
Revised: Starting from a simple observation in the previous section, we propose a regularization technique that forces the network to have similar representation for classes along a path p , which implies having similar representation between similar objects.

Original: More formally, if we have an optimal classiﬁer w p for the super-class p and a classiﬁer w (cid:104) p,i (cid:105) for the class (cid:104) p, i (cid:105) , we expect that
Revised: More formally, if we have an optimal classiﬁer w p for the super-class p and a classiﬁer w (cid:104) p,i (cid:105) for the class (cid:104) p, i (cid:105) , we expect that

Original: [Equation]
Revised: [Equation]

Original: If this is satisﬁed, separators for objects in the same super-class are also similar because
Revised: If this is satisﬁed, separators for objects in the same super-class are also similar because

Original: [Equation]
Revised: [Equation]

Original: However, the optimal classiﬁer for an arbitrary representation φ θ ( X ) may not satisfy equation 2.
Revised: However, the optimal classiﬁer for an arbitrary representation φ θ ( X ) may not satisfy equation 2.

Original: The naive and direct way to ensure equation 2 is through hierarchical regularization , which forces classiﬁers in the same path to be close to each other.
Revised: The naive and direct way to ensure equation 2 is through hierarchical regularization , which forces classiﬁers in the same path to be close to each other.

Original: 2. H IERARCHICAL LAYER AND HIERARCHICALLY CONNECTED LAYER
Revised: 2. H IERARCHICAL LAYER AND HIERARCHICALLY CONNECTED LAYER

Original: In the previous section, we described the hierarchical regularization technique given a hierarchical structure in the classes.
Revised: In the previous section, we described the hierarchical regularization technique given a hierarchical structure in the classes.

Original: In this section, we show how to conveniently parametrize equation 2.
Revised: In this section, we show how to conveniently parametrize equation 2.

Original: We ﬁrst express the classiﬁer as a sum of vectors δ deﬁned recursively as follows:
Revised: We ﬁrst express the classiﬁer as a sum of vectors δ deﬁned recursively as follows:

Original: [Equation]
Revised: [Equation]

Original: where {} is the root.
Revised: where {} is the root.

Original: It is possible to consider δ {} (cid:54) = 0 , which shifts separating hyper-planes.
Revised: It is possible to consider δ {} (cid:54) = 0 , which shifts separating hyper-planes.

Original: We do not consider this case in this paper.
Revised: We do not consider this case in this paper.

Original: Given equation 4, we have that (cid:107) δ (cid:104) p,i (cid:105) (cid:107) is small in equation 2.
Revised: Given equation 4, we have that (cid:107) δ (cid:104) p,i (cid:105) (cid:107) is small in equation 2.

Original: Finally, it sufﬁces to penalize the norm of δ (cid:104) p,i (cid:105) during the optimization.
Revised: Finally, it sufﬁces to penalize the norm of δ (cid:104) p,i (cid:105) during the optimization.

Original: Notice that, by construction, the number of δ ’s is equal to the number of nodes in the hierarchical tree.
Revised: Notice that, by construction, the number of δ ’s is equal to the number of nodes in the hierarchical tree.

Original: Next, consider the output of CNNs for classiﬁcation,
Revised: Next, consider the output of CNNs for classiﬁcation,

Original: [Equation]
Revised: [Equation]

Original: where θ denotes the parameters of the hidden layers, W = [ w 1 , . .
Revised: where θ denotes the parameters of the hidden layers, W = [ w 1 , . .

Original: . , w |L| ] denotes the last fullyconnected layer, and w i denotes the separator for the class i .
Revised: . , w |L| ] denotes the last fullyconnected layer, and w i denotes the separator for the class i .

Original: For simplicity, we omit potential additional nonlinear functions, such as a softmax, on top of the prediction.
Revised: For simplicity, we omit potential additional nonlinear functions, such as a softmax, on top of the prediction.

Original: We have parametrized w i following the recursive formula in equation 4.
Revised: We have parametrized w i following the recursive formula in equation 4.

Original: To deﬁne the matrix formulation of equation 4, we ﬁrst introduce the Hierarchical layer H which plays an important role.
Revised: To deﬁne the matrix formulation of equation 4, we ﬁrst introduce the Hierarchical layer H which plays an important role.

Original: This hierarchical layer can be identiﬁed to the adjacency matrix of the hierarchical graph.
Revised: This hierarchical layer can be identiﬁed to the adjacency matrix of the hierarchical graph.

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: (Hierarchical layer).
Revised: (Hierarchical layer).

Original: Consider ordering over the sets P and L , i.e., for i = 1 , . .
Revised: Consider ordering over the sets P and L , i.e., for i = 1 , . .

Original: , |P| and j = 1 , . .
Revised: , |P| and j = 1 , . .

Original: . , |L| ,
Revised: . , |L| ,

Original: P = { p 1 , . . .
Revised: P = { p 1 , . . .

Original: , p i , . .
Revised: , p i , . .

Original: , p |P| } and L = { p 1 , . . .
Revised: , p |P| } and L = { p 1 , . . .

Original: , p j , . .
Revised: , p j , . .

Original: . , p |L| } .
Revised: . , p |L| } .

Original: In other words, we associate to all nodes an index.
Revised: In other words, we associate to all nodes an index.

Original: Then, the hierarchical layer H is deﬁned as
Revised: Then, the hierarchical layer H is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where n p i (cid:22) n p j means n p j is a parent of n p i .
Revised: where n p i (cid:22) n p j means n p j is a parent of n p i .

Original: We illustrate an example of H in Appendix A.
Revised: We illustrate an example of H in Appendix A.

Original: The next proposition shows that equation 5 can be written using a simple matrix-matrix multiplication, involving the hierarchical layer.
Revised: The next proposition shows that equation 5 can be written using a simple matrix-matrix multiplication, involving the hierarchical layer.

Original: Proposition 1. Consider a representation φ θ ( · ) , where φ θ ( · ) ∈ R d .
Revised: Proposition 1. Consider a representation φ θ ( · ) , where φ θ ( · ) ∈ R d .

Original: Let W be the matrix of separators
Revised: Let W be the matrix of separators

Original: [Equation]
Revised: [Equation]

Original: where the separators are parametrized as equation 4.
Revised: where the separators are parametrized as equation 4.

Original: Let ∆ be deﬁned as
Revised: Let ∆ be deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where P and L are deﬁned in Section 2.1.
Revised: where P and L are deﬁned in Section 2.1.

Original: Consider the hierarchical layer deﬁned in Deﬁnition 1.
Revised: Consider the hierarchical layer deﬁned in Deﬁnition 1.

Original: Then, the matrix of separators W can be expressed as
Revised: Then, the matrix of separators W can be expressed as

Original: [Equation]
Revised: [Equation]

Original: We can see W = ∆ H as a combination of an augmented fully-connected layer, combined with the hierarchical layer that selects the right columns of ∆ , hence the term hierarchically connected layer .
Revised: We can see W = ∆ H as a combination of an augmented fully-connected layer, combined with the hierarchical layer that selects the right columns of ∆ , hence the term hierarchically connected layer .

Original: The (cid:96) 2 regularization of the δ can be conducted by the parameter weight decay , which is widely used in training of neural networks.
Revised: The (cid:96) 2 regularization of the δ can be conducted by the parameter weight decay , which is widely used in training of neural networks.

Original: The hierarchical layer H is ﬁxed, while ∆ is learnable.
Revised: The hierarchical layer H is ﬁxed, while ∆ is learnable.

Original: This does not affect the complexity of the back-propagation signiﬁcantly, as ∆ H is a simple linear form.
Revised: This does not affect the complexity of the back-propagation signiﬁcantly, as ∆ H is a simple linear form.

Original: The size of the last layer slightly increases, from |L| × d to |P| × d , where d is the dimension of the representation φ θ ( · ) .
Revised: The size of the last layer slightly increases, from |L| × d to |P| × d , where d is the dimension of the representation φ θ ( · ) .

Original: For instance, in the case of Tiny-ImageNet, the number of parameters of the last layer only increases by roughly 36% ; nevertheless, the increased number of parameters of the last layer is still usually negligible in comparison with the total number of parameters for classical network architectures.
Revised: For instance, in the case of Tiny-ImageNet, the number of parameters of the last layer only increases by roughly 36% ; nevertheless, the increased number of parameters of the last layer is still usually negligible in comparison with the total number of parameters for classical network architectures.

Original:  H IERARCHICAL S PHERES
Revised:  H IERARCHICAL S PHERES

Original: The hierarchical ( (cid:96) 2 ) regularization introduced in the previous section induces separated hyper-planes along a path to be close to each other.
Revised: The hierarchical ( (cid:96) 2 ) regularization introduced in the previous section induces separated hyper-planes along a path to be close to each other.

Original: However, this approach has a signiﬁcant drawback.
Revised: However, this approach has a signiﬁcant drawback.

Original: We rewind equation 2, which models the similarity of two separators w p and w (cid:104) p,i (cid:105) .
Revised: We rewind equation 2, which models the similarity of two separators w p and w (cid:104) p,i (cid:105) .

Original: The similarity between separators (individual hyper-planes) should indicate that they point roughly the same direction, i.e.,
Revised: The similarity between separators (individual hyper-planes) should indicate that they point roughly the same direction, i.e.,

Original: [Equation]
Revised: [Equation]

Original: small
Revised: small

Original: [Equation]
Revised: [Equation]

Original: However, this property is not necessarily captured by equation 2.
Revised: However, this property is not necessarily captured by equation 2.

Original: For instance, assume that w p = − w (cid:104) p,i (cid:105) , i
Revised: For instance, assume that w p = − w (cid:104) p,i (cid:105) , i

Original: [Equation]
Revised: [Equation]

Original: e., the separators point in two opposite directions (and thus completely different).
Revised: e., the separators point in two opposite directions (and thus completely different).

Original: Then, equation 2 can be arbitrarily small in the function of (cid:107) w p (cid:107) but not in equation 10:
Revised: Then, equation 2 can be arbitrarily small in the function of (cid:107) w p (cid:107) but not in equation 10:

Original: [Equation]
Revised: [Equation]

Original: This can be avoided, for example, by deploying the regularization parameter (or weight decay) independently for each (cid:107) δ p (cid:107) .
Revised: This can be avoided, for example, by deploying the regularization parameter (or weight decay) independently for each (cid:107) δ p (cid:107) .

Original: However, it is costly in terms of hyper-parameter estimation.
Revised: However, it is costly in terms of hyper-parameter estimation.

Original: In order to enforce the closeness of embedding vectors whose paths are similar, we penalize large norms of δ .
Revised: In order to enforce the closeness of embedding vectors whose paths are similar, we penalize large norms of δ .

Original: We also want to bound it away from zero to avoid the problem of separators that point in different direction may have a small norm.
Revised: We also want to bound it away from zero to avoid the problem of separators that point in different direction may have a small norm.

Original: This naturally leads to a spherical constraint.
Revised: This naturally leads to a spherical constraint.

Original: Indeed, we transform the (cid:96) 2 regularization over δ p by ﬁxing its norm in advance, i.e.,
Revised: Indeed, we transform the (cid:96) 2 regularization over δ p by ﬁxing its norm in advance, i.e.,

Original: [Equation]
Revised: [Equation]

Original: In other words, we deﬁne δ p on a sphere of radius R p .
Revised: In other words, we deﬁne δ p on a sphere of radius R p .

Original: The fully-connected layer ∆ is then constrained on spheres, hence it is named spherical fully-connected layer .
Revised: The fully-connected layer ∆ is then constrained on spheres, hence it is named spherical fully-connected layer .

Original: Hence, we have w (cid:104) p,i (cid:105) constrained on a sphere centered at w p .
Revised: Hence, we have w (cid:104) p,i (cid:105) constrained on a sphere centered at w p .

Original: This constraint prevents the direction of w (cid:104) p,i (cid:105) from being too different from that of w p , while bounding the distance away from zero.
Revised: This constraint prevents the direction of w (cid:104) p,i (cid:105) from being too different from that of w p , while bounding the distance away from zero.

Original: This does not add hyperparameters: instead of weight decay, we have the radius R p of the sphere.
Revised: This does not add hyperparameters: instead of weight decay, we have the radius R p of the sphere.

Original: 3. R ADIUS DECAY W . R . T . PATH LENGTH
Revised: 3. R ADIUS DECAY W . R . T . PATH LENGTH

Original: We allow the radius of the spheres, R p , to be deﬁned as a function of the path.
Revised: We allow the radius of the spheres, R p , to be deﬁned as a function of the path.

Original: In this study, we use a simple strategy called radius decay , where R p decreases w.r.t.
Revised: In this study, we use a simple strategy called radius decay , where R p decreases w.r.t.

Original: the path length:
Revised: the path length:

Original: [Equation]
Revised: [Equation]

Original: (13) where R 0 is the initial radius, γ is the radius decay parameter, and | p | is the length of the path.
Revised: (13) where R 0 is the initial radius, γ is the radius decay parameter, and | p | is the length of the path.

Original: The optimal radius decay can be easily found using cross-validation.
Revised: The optimal radius decay can be easily found using cross-validation.

Original: The radius decay is applied prior to learning (as opposed to weight-decay); then, the radius remains ﬁxed during the optimization.
Revised: The radius decay is applied prior to learning (as opposed to weight-decay); then, the radius remains ﬁxed during the optimization.

Original: As opposed to weight-decay, whose weight are multiplied by some constant smaller than one after each iteration, the radius decay here depends only on the path length, and the radius remains ﬁxed during the optimization process.
Revised: As opposed to weight-decay, whose weight are multiplied by some constant smaller than one after each iteration, the radius decay here depends only on the path length, and the radius remains ﬁxed during the optimization process.

Original: The simplest way to apply the radius decay is by using the following predeﬁned diagonal matrix D ,
Revised: The simplest way to apply the radius decay is by using the following predeﬁned diagonal matrix D ,

Original: [Equation]
Revised: [Equation]

Original: where p i follows the ordering from Deﬁnition 1.
Revised: where p i follows the ordering from Deﬁnition 1.

Original: Finally, the last layer of the neural network reads,
Revised: Finally, the last layer of the neural network reads,

Original: [Equation]
Revised: [Equation]

Original: The only learnable parameter in the last layer is ∆ .
Revised: The only learnable parameter in the last layer is ∆ .

Original: 3. O PTIMIZATION
Revised: 3. O PTIMIZATION

Original: There are several ways to optimize the network in the presence of the spherical fully-connected layer : by introducing the constraint in the model, “naively” by performing normalization after each step, or by using Riemannian optimization algorithms.
Revised: There are several ways to optimize the network in the presence of the spherical fully-connected layer : by introducing the constraint in the model, “naively” by performing normalization after each step, or by using Riemannian optimization algorithms.

Original: For simplicity, we consider the minimization problem,
Revised: For simplicity, we consider the minimization problem,

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where θ are the parameters of the hidden layers, ∆ the spherical fully-connected layer from equation 8, and f the empirical expectation of the loss of the neural network.
Revised: where θ are the parameters of the hidden layers, ∆ the spherical fully-connected layer from equation 8, and f the empirical expectation of the loss of the neural network.

Original: For clarity, we use noiseless gradients, but all results also apply to stochastic ones.
Revised: For clarity, we use noiseless gradients, but all results also apply to stochastic ones.

Original: The superscript · t denotes the t -th iteration.
Revised: The superscript · t denotes the t -th iteration.

Original: 3.2. I NTEGRATION OF THE CONSTRAINT IN THE MODEL
Revised: 3.2. I NTEGRATION OF THE CONSTRAINT IN THE MODEL

Original: We present the simplest way to force the column of ∆ to lie on a sphere, as this does not require a dedicated optimization algorithm.
Revised: We present the simplest way to force the column of ∆ to lie on a sphere, as this does not require a dedicated optimization algorithm.

Original: It is sufﬁcient to normalize the column of ∆ by their norm in the model.
Revised: It is sufﬁcient to normalize the column of ∆ by their norm in the model.

Original: By introducing a dummy variable ˜∆ , which is the normalized version of ∆ , the last layer of the neural network equation 15 reads
Revised: By introducing a dummy variable ˜∆ , which is the normalized version of ∆ , the last layer of the neural network equation 15 reads

Original: [Equation]
Revised: [Equation]

Original: Then, any standard optimization algorithm can be used for the learning phase.
Revised: Then, any standard optimization algorithm can be used for the learning phase.

Original: Technically, ∆ is not constrained on a sphere, but the model will act as if ∆ follows such constraint.
Revised: Technically, ∆ is not constrained on a sphere, but the model will act as if ∆ follows such constraint.

Original: 3.2. O PTIMIZATION OVER SPHERES : R IEMANNIAN ( STOCHASTIC ) GRADIENT DESCENT
Revised: 3.2. O PTIMIZATION OVER SPHERES : R IEMANNIAN ( STOCHASTIC ) GRADIENT DESCENT

Original: The most direct way to optimize over a sphere is to normalize the columns of ∆ by their norm after each iteration.
Revised: The most direct way to optimize over a sphere is to normalize the columns of ∆ by their norm after each iteration.

Original: However, this method has no convergence guarantee, and requires a modiﬁcation in the optimization algorithm.
Revised: However, this method has no convergence guarantee, and requires a modiﬁcation in the optimization algorithm.

Original: Instead, we perform Riemannian gradient descent which we explain only brieﬂy in this manuscript.
Revised: Instead, we perform Riemannian gradient descent which we explain only brieﬂy in this manuscript.

Original: We give the derivation of Riemannian gradient for spheres in Appendix B.
Revised: We give the derivation of Riemannian gradient for spheres in Appendix B.

Original: Riemannian gradient descent involves two steps: ﬁrst, a projection to the tangent space, and then, a retraction to the manifold.
Revised: Riemannian gradient descent involves two steps: ﬁrst, a projection to the tangent space, and then, a retraction to the manifold.

Original: The projection step computes the gradient of the function on the manifold (as opposed to the ambient space R d ), such that its gradient is tangent to the sphere.
Revised: The projection step computes the gradient of the function on the manifold (as opposed to the ambient space R d ), such that its gradient is tangent to the sphere.

Original: Then, the retraction simply maps the new iterate to the sphere.
Revised: Then, the retraction simply maps the new iterate to the sphere.

Original: With this two-step procedure, all directions pointing outside the manifold, (i.e., orthogonal to the manifold, thus irrelevant) are discarded by the projection.
Revised: With this two-step procedure, all directions pointing outside the manifold, (i.e., orthogonal to the manifold, thus irrelevant) are discarded by the projection.

Original: These two steps are summarized below,
Revised: These two steps are summarized below,

Original: [Equation]
Revised: [Equation]

Original: where s t is the projection of the descent direction to the tangent space, and δ t +1 p is the retraction of the gradient descent step with stepsize h .
Revised: where s t is the projection of the descent direction to the tangent space, and δ t +1 p is the retraction of the gradient descent step with stepsize h .

Original: In our numerical experiments, we used the Geoopt optimizer (Kochurov et al., 2020), which implements Riemannian gradient descent on spheres.
Revised: In our numerical experiments, we used the Geoopt optimizer (Kochurov et al., 2020), which implements Riemannian gradient descent on spheres.

Original:  N UMERICAL E XPERIMENTS
Revised:  N UMERICAL E XPERIMENTS

Original: We experimented the proposed method using ﬁve publicly available datasets, namely CIFAR100 (Krizhevsky, 2009), Caltech-UCSD Birds 200 (CUB200) (Welinder et al., 2010), StanfordCars (Cars) (Krause et al., 2013), Stanford-dogs (Dogs) (Khosla et al., 2011), and Tiny-ImageNet (Tiny-ImNet)
Revised: We experimented the proposed method using ﬁve publicly available datasets, namely CIFAR100 (Krizhevsky, 2009), Caltech-UCSD Birds 200 (CUB200) (Welinder et al., 2010), StanfordCars (Cars) (Krause et al., 2013), Stanford-dogs (Dogs) (Khosla et al., 2011), and Tiny-ImageNet (Tiny-ImNet)

Original: (Deng et al., 2009).
Revised: (Deng et al., 2009).

Original: CUB200, Cars, and Dogs datasets are used for ﬁne-grained visual categorization (recognizing bird, dog bleeds, or car models), while CIFAR100 and Tiny-ImNet datasets are used for the classiﬁcation of objects and animals.
Revised: CUB200, Cars, and Dogs datasets are used for ﬁne-grained visual categorization (recognizing bird, dog bleeds, or car models), while CIFAR100 and Tiny-ImNet datasets are used for the classiﬁcation of objects and animals.

Original: Unlike the datasets for object classiﬁcation, the ﬁne-grained visual categorization datasets show low inter-class variances.
Revised: Unlike the datasets for object classiﬁcation, the ﬁne-grained visual categorization datasets show low inter-class variances.

Original: See Appendix C. and C.3 for more details about the dataset and their hierarchy, respectively.
Revised: See Appendix C. and C.3 for more details about the dataset and their hierarchy, respectively.

Original: 4. D EEP NEURAL NETWORK MODELS AND TRAINING SETTING
Revised: 4. D EEP NEURAL NETWORK MODELS AND TRAINING SETTING

Original: We used the deep neural networks ( ResNet (He et al., 2016) and DenseNet (Huang et al., 2017)).
Revised: We used the deep neural networks ( ResNet (He et al., 2016) and DenseNet (Huang et al., 2017)).

Original: The input size of the datasets CUB200, Cars, Dogs, and Tiny-ImNet is 224 × 224 , and 32 × 32 pixels for CIFAR100.
Revised: The input size of the datasets CUB200, Cars, Dogs, and Tiny-ImNet is 224 × 224 , and 32 × 32 pixels for CIFAR100.

Original: Since the input-size of CIFAR100 does not ﬁt to the original ResNet and DenseNet, we used a smaller kernel size (3 instead of 7) at the ﬁrst convolutional layer and a smaller stride ( instead of 2) at the ﬁrst block.
Revised: Since the input-size of CIFAR100 does not ﬁt to the original ResNet and DenseNet, we used a smaller kernel size (3 instead of 7) at the ﬁrst convolutional layer and a smaller stride ( instead of 2) at the ﬁrst block.

Original: Remark: we do not use pretrained networks.
Revised: Remark: we do not use pretrained networks.

Original: All networks are trained from scratch, i.e., we did not use pre-trained models.
Revised: All networks are trained from scratch, i.e., we did not use pre-trained models.

Original: This is because most publicly available pre-trained models used ImageNet for training while Dogs and Tiny-ImNet are parts of ImageNet.
Revised: This is because most publicly available pre-trained models used ImageNet for training while Dogs and Tiny-ImNet are parts of ImageNet.

Original: We used the stochastic gradient descent (SGD) over 300 epochs, with a mini-batch of 64 and a momentum parameter of 0 . 9 for training.
Revised: We used the stochastic gradient descent (SGD) over 300 epochs, with a mini-batch of 64 and a momentum parameter of 0 . 9 for training.

Original: The learning rate schedule is the same for all experiments, starting at 0 .
Revised: The learning rate schedule is the same for all experiments, starting at 0 .

Original: 1 , then decaying by a factor of 10 after 150 , then 255 epochs.
Revised: 1 , then decaying by a factor of 10 after 150 , then 255 epochs.

Original: All tests are conducted using NVIDIA Tesla V100 GPU with the same random seed.
Revised: All tests are conducted using NVIDIA Tesla V100 GPU with the same random seed.

Original: Settings in more detail are provided in the supplementary material.
Revised: Settings in more detail are provided in the supplementary material.

Original: We emphasize that we used the same parameters and learning rate schedule for all scenarios .
Revised: We emphasize that we used the same parameters and learning rate schedule for all scenarios .

Original: Those parameters and schedule were optimized for SGD on plain networks, but are probably sub-optimal for our proposed methods.
Revised: Those parameters and schedule were optimized for SGD on plain networks, but are probably sub-optimal for our proposed methods.

Original: 4. R ESULTS
Revised: 4. R ESULTS

Original: Tables 1 and 2 show a comparison of the results obtained with several baseline methods and our methods.
Revised: Tables 1 and 2 show a comparison of the results obtained with several baseline methods and our methods.

Original: The ﬁrst method, “ Plain ”, is a plain network for subclass classiﬁcation without hierarchical information.
Revised: The ﬁrst method, “ Plain ”, is a plain network for subclass classiﬁcation without hierarchical information.

Original: The second one, “ Multitask ” is simply the plain network with multitask (subclass and super-class classiﬁcation) setting using the hierarchical information.
Revised: The second one, “ Multitask ” is simply the plain network with multitask (subclass and super-class classiﬁcation) setting using the hierarchical information.

Original: The third one, “ Hierarchy ”, uses our parametrization W = ∆ H with the hierarchical layer H , but the columns of ∆ are not constrained on spheres.
Revised: The third one, “ Hierarchy ”, uses our parametrization W = ∆ H with the hierarchical layer H , but the columns of ∆ are not constrained on spheres.

Original: Then, “ +Manifold ” means that ∆ is restricted on a sphere using the normalization technique from Section 3.2.1.
Revised: Then, “ +Manifold ” means that ∆ is restricted on a sphere using the normalization technique from Section 3.2.1.

Original: Finally, “ +Riemann ” means we used Riemannian optimization from
Revised: Finally, “ +Riemann ” means we used Riemannian optimization from

Original: Section 3.2.2.
Revised: Section 3.2.2.

Original: We show the experimental results on ﬁne-grained visual classiﬁcation (Table 1) and general object classiﬁcation (Table 2).
Revised: We show the experimental results on ﬁne-grained visual classiﬁcation (Table 1) and general object classiﬁcation (Table 2).

Original: Note that the multitask strategy in our experiment (and contrary to our regularization technique) does require an additional hyper-parameter that combines the two losses, because we train classiﬁers for super-classes and sub-classes simultaneously
Revised: Note that the multitask strategy in our experiment (and contrary to our regularization technique) does require an additional hyper-parameter that combines the two losses, because we train classiﬁers for super-classes and sub-classes simultaneously

Original: . 4.2. F INE - GRAINED CATEGORIZATION
Revised: . 4.2. F INE - GRAINED CATEGORIZATION

Original: As shown in Table 1, our proposed parameterization signiﬁcantly improves the test accuracy over the baseline networks (ResNet-18/50, DenseNet-121/160).
Revised: As shown in Table 1, our proposed parameterization signiﬁcantly improves the test accuracy over the baseline networks (ResNet-18/50, DenseNet-121/160).

Original: Even the simple hierarchical setting which uses the hierarchical layer only (without spheres) shows superior performance compared to the baseline networks.
Revised: Even the simple hierarchical setting which uses the hierarchical layer only (without spheres) shows superior performance compared to the baseline networks.

Original: Integrating the manifolds with Riemannian SGD further improves the generalization performance.
Revised: Integrating the manifolds with Riemannian SGD further improves the generalization performance.

Original: Surprisingly, the plain network with deeper layers shows degraded performance.
Revised: Surprisingly, the plain network with deeper layers shows degraded performance.

Original: This can be attributed to overﬁtting which does not occur with our regularization technique, where larger networks show better performance, indicating the high efﬁciency of our approach.
Revised: This can be attributed to overﬁtting which does not occur with our regularization technique, where larger networks show better performance, indicating the high efﬁciency of our approach.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: O BJECT CLASSIFICATION
Revised: O BJECT CLASSIFICATION

Original: We show test accuracy ( % ) of our proposed methods with different network models using CIFARand Tiny-ImNet, in Table 2.
Revised: We show test accuracy ( % ) of our proposed methods with different network models using CIFARand Tiny-ImNet, in Table 2.

Original: From the table, it can be seen that the proposed method has better accuracy than the baseline methods.
Revised: From the table, it can be seen that the proposed method has better accuracy than the baseline methods.

Original: Compared to the ﬁne-grained classiﬁcation datasets, the general object classiﬁcation datasets have less similar classes on the same super-class.
Revised: Compared to the ﬁne-grained classiﬁcation datasets, the general object classiﬁcation datasets have less similar classes on the same super-class.

Original: In these datasets, our method achieved relatively small gains.
Revised: In these datasets, our method achieved relatively small gains.

Original: A higher inter-class variance may explain the lower improvement compared to ﬁne-grained categorization.
Revised: A higher inter-class variance may explain the lower improvement compared to ﬁne-grained categorization.

Original: Nevertheless, for Tiny-ImNet, e.g., ResNet-18 (11.28M parameters) with our parametrization achieves better classiﬁcation performance than plain ResNet-50 (23.91M parameters).
Revised: Nevertheless, for Tiny-ImNet, e.g., ResNet-18 (11.28M parameters) with our parametrization achieves better classiﬁcation performance than plain ResNet-50 (23.91M parameters).

Original: The same applies to DenseNet-121 and DenseNet-161.
Revised: The same applies to DenseNet-121 and DenseNet-161.

Original: These results indicate that our regularization technique, which does not introduce new parameters in the embedding layer , can achieve a classiﬁcation performance similar to that of more complex models.
Revised: These results indicate that our regularization technique, which does not introduce new parameters in the embedding layer , can achieve a classiﬁcation performance similar to that of more complex models.

Original: 4. R IEMANNIAN VS . P ROJECTED SGD
Revised: 4. R IEMANNIAN VS . P ROJECTED SGD

Original: Overall, Riemannian SGD showed slightly superior performance compared to projected SGD for ﬁne-grained datasets, although, in most cases, the performance was similar.
Revised: Overall, Riemannian SGD showed slightly superior performance compared to projected SGD for ﬁne-grained datasets, although, in most cases, the performance was similar.

Original: For instance, with the
Revised: For instance, with the

Original: Dogs dataset on Resnet-50, Riemannian SGD shows a performance 4% higher than the projected
Revised: Dogs dataset on Resnet-50, Riemannian SGD shows a performance 4% higher than the projected

Original: SGD.
Revised: SGD.

Original: For object classiﬁcation, Riemannian SGD performs a bit more poorly.
Revised: For object classiﬁcation, Riemannian SGD performs a bit more poorly.

Original: We suspect that, owing to the different radius decay parameters ( 0 . 5 in Table 1 and 0.9 in Table 2), the learning rate of
Revised: We suspect that, owing to the different radius decay parameters ( 0 . 5 in Table 1 and 0.9 in Table 2), the learning rate of

Original: Riemannian SGD should have been changed to a larger value.
Revised: Riemannian SGD should have been changed to a larger value.

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: We presented a simple regularization method for neural networks using a given hierarchical structure of the classes.
Revised: We presented a simple regularization method for neural networks using a given hierarchical structure of the classes.

Original: The method involves of the reformulation of the fully connected layer of the neural network using the hierarchical layer .
Revised: The method involves of the reformulation of the fully connected layer of the neural network using the hierarchical layer .

Original: We further improved the technique using spherical constraints, transforming the last layer into a spherical fully-connected layer .
Revised: We further improved the technique using spherical constraints, transforming the last layer into a spherical fully-connected layer .

Original: Finally, we compared the optimization of the neural network using several strategies.
Revised: Finally, we compared the optimization of the neural network using several strategies.

Original: The reformulation using the hierarchical layer ∆ H and the spherical constraint had a considerable impact on the generalization accuracy of the network.
Revised: The reformulation using the hierarchical layer ∆ H and the spherical constraint had a considerable impact on the generalization accuracy of the network.

Original: The Riemannian optimization had a lower overall impact, showing sometimes signiﬁcant improvement and sometimes similar to its projected counterpart.
Revised: The Riemannian optimization had a lower overall impact, showing sometimes signiﬁcant improvement and sometimes similar to its projected counterpart.

Original: In this paper, we used the proposed regularization technique only on classical architectures.
Revised: In this paper, we used the proposed regularization technique only on classical architectures.

Original: In the future, it would be interesting to use it on other architectures, e.g. Inception and SqueezeNet, for embedding, e.g. Poincaré, and other applications, e.g. Natural Language Processing (NLP).
Revised: In the future, it would be interesting to use it on other architectures, e.g. Inception and SqueezeNet, for embedding, e.g. Poincaré, and other applications, e.g. Natural Language Processing (NLP).

Original: Moreover, in this paper, we used a given hierarchy mostly based on taxonomy designed by experts.
Revised: Moreover, in this paper, we used a given hierarchy mostly based on taxonomy designed by experts.

Original: This hierarchical structure, which is convenient for humans, may not be most convenient for classiﬁcation algorithms.
Revised: This hierarchical structure, which is convenient for humans, may not be most convenient for classiﬁcation algorithms.

Original: A self-supervised algorithm that learns classiﬁcation and the hierarchy may be convenient because we do not need to access a hierarchy and lead to better results (because the structure will be more adapted to the task).
Revised: A self-supervised algorithm that learns classiﬁcation and the hierarchy may be convenient because we do not need to access a hierarchy and lead to better results (because the structure will be more adapted to the task).

Original: A E XAMPLE OF HIERARCHICAL STRUCTURE
Revised: A E XAMPLE OF HIERARCHICAL STRUCTURE

Original: Consider a dataset composed by the following labels: cats , dogs , apple , orange .
Revised: Consider a dataset composed by the following labels: cats , dogs , apple , orange .

Original: These labels can be organized trough a hierarchical structure, with super-classes animal and fruit .
Revised: These labels can be organized trough a hierarchical structure, with super-classes animal and fruit .

Original: In such case, the set P is composed by
Revised: In such case, the set P is composed by

Original: [Equation]
Revised: [Equation]

Original: while the set L is composed by
Revised: while the set L is composed by

Original: [Equation]
Revised: [Equation]

Original: Then, its hierarchical layer reads (labels were added to ease the reading)
Revised: Then, its hierarchical layer reads (labels were added to ease the reading)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: {fruit
Revised: {fruit

Original: [Equation]
Revised: [Equation]

Original: apple
Revised: apple

Original: [Equation]
Revised: [Equation]

Original: {fruit, orange}    {animal, cat}    {animal
Revised: {fruit, orange}    {animal, cat}    {animal

Original: [Equation]
Revised: [Equation]

Original: dog
Revised: dog

Original: [Equation]
Revised: [Equation]

Original: B O PTIMIZATION O VER S PHERES : R IEMANIAN ( STOCHASTIC ) GRADIENT DESCENT
Revised: B O PTIMIZATION O VER S PHERES : R IEMANIAN ( STOCHASTIC ) GRADIENT DESCENT

Original: We quickly recall some elements of optimization on manifolds, see e.g. Boumal (2020); Absil et al.
Revised: We quickly recall some elements of optimization on manifolds, see e.g. Boumal (2020); Absil et al.

Original: For simplicity, we consider the optimization problem
Revised: For simplicity, we consider the optimization problem

Original: [Equation]
Revised: [Equation]

Original: (19) where S d − 1 is the sphere manifold with radius one centered at zero and embedded in R d .
Revised: (19) where S d − 1 is the sphere manifold with radius one centered at zero and embedded in R d .

Original: The generic
Revised: The generic

Original: Riemannian gradient descent with stepsize h reads
Revised: Riemannian gradient descent with stepsize h reads

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: (21) where grad f is the gradient of f on the sphere , which is a vector that belongs to the tangent space
Revised: (21) where grad f is the gradient of f on the sphere , which is a vector that belongs to the tangent space

Original: T x k S d − 1 (plane tangent to the sphere that contains x k ), and R x k is a second-order retraction, i.e., a mapping from the tangent space T x k S d − 1 to the sphere S d − 1 that satisﬁes some smoothness properties.
Revised: T x k S d − 1 (plane tangent to the sphere that contains x k ), and R x k is a second-order retraction, i.e., a mapping from the tangent space T x k S d − 1 to the sphere S d − 1 that satisﬁes some smoothness properties.

Original: The vector s k (that belongs to the tangent sphere) represents the local descent direction.
Revised: The vector s k (that belongs to the tangent sphere) represents the local descent direction.

Original: We illustrate those quantities in Figure 3.
Revised: We illustrate those quantities in Figure 3.

Original: Stochastic Riemannian gradient descent directly follows from (Bonnabel, 2013), replacing the gradient by its stochastic version.
Revised: Stochastic Riemannian gradient descent directly follows from (Bonnabel, 2013), replacing the gradient by its stochastic version.

Original: In the special case of the sphere, we have an explicit formula for the tangent space and its projection, for the Riemannian gradient, and for the retraction:
Revised: In the special case of the sphere, we have an explicit formula for the tangent space and its projection, for the Riemannian gradient, and for the retraction:

Original: [Equation]
Revised: [Equation]

Original: The retraction is not necessarily unique, but this one satisﬁes all requirement to ensure good convergence properties.
Revised: The retraction is not necessarily unique, but this one satisﬁes all requirement to ensure good convergence properties.

Original: The gradient descent algorithm on a sphere thus reads
Revised: The gradient descent algorithm on a sphere thus reads

Original: [Equation]
Revised: [Equation]

Original: In our case, we have a matrix ∆ , whose each column δ p belongs to a sphere.
Revised: In our case, we have a matrix ∆ , whose each column δ p belongs to a sphere.

Original: It sufﬁces to apply the Riemannian gradient descent separately on each δ p .
Revised: It sufﬁces to apply the Riemannian gradient descent separately on each δ p .

Original: For practical reasons, we used the toolbox
Revised: For practical reasons, we used the toolbox

Original: Geoopt (Kochurov et al., 2020; Bécigneul & Ganea, 2018) for numerical optimization.
Revised: Geoopt (Kochurov et al., 2020; Bécigneul & Ganea, 2018) for numerical optimization.

Original: [Table]
Revised: [Table]

Original: C N UMERICAL EXPERIMENTS : SUPPLEMENTARY MATERIALS C. D EEP NEURAL NETWORK MODELS AND TRAINING DETAILS
Revised: C N UMERICAL EXPERIMENTS : SUPPLEMENTARY MATERIALS C. D EEP NEURAL NETWORK MODELS AND TRAINING DETAILS

Original: We used ResNet which consists of the basic blocks or the bottleneck blocks with output channels [64, 128, 256, 512] in Conv. layers.
Revised: We used ResNet which consists of the basic blocks or the bottleneck blocks with output channels [64, 128, 256, 512] in Conv. layers.

Original: A dimensionality of an input vector to the FC layer is 512.
Revised: A dimensionality of an input vector to the FC layer is 512.

Original: We used
Revised: We used

Original: DenseNet which includes hyperparameters such as [“growth rate”, “block conﬁguration”
Revised: DenseNet which includes hyperparameters such as [“growth rate”, “block conﬁguration”

Original: [Table]
Revised: [Table]

Original: and “initial feature dimension”] for ’DenseNet-121’ [32, (6, 12, 24, 16), 64] and ‘DenseNet-161’ [48, (6, 12, 36, 24), 96], respectively.
Revised: and “initial feature dimension”] for ’DenseNet-121’ [32, (6, 12, 24, 16), 64] and ‘DenseNet-161’ [48, (6, 12, 36, 24), 96], respectively.

Original: A dimensionality of an input vector for DenseNets to the FC layer is 64 and 96.
Revised: A dimensionality of an input vector for DenseNets to the FC layer is 64 and 96.

Original: Parameters in our proposed method using ResNet and DenseNet are optimized using the SGD with several settings: we ﬁxed 1) the weight initialization with Random-Seed number ‘0’ in pytorch, 2) learning rate schedule [0.1, 0.01, 0.001], 3) with momentum 0.9, 4) regularization: weight decay with 0 . 0001 .
Revised: Parameters in our proposed method using ResNet and DenseNet are optimized using the SGD with several settings: we ﬁxed 1) the weight initialization with Random-Seed number ‘0’ in pytorch, 2) learning rate schedule [0.1, 0.01, 0.001], 3) with momentum 0.9, 4) regularization: weight decay with 0 . 0001 .

Original: A bias term in the FC layer is not used.
Revised: A bias term in the FC layer is not used.

Original: The images (CUB200, Cars, Dogs, and Tiny-ImNet) in training and test sets are resized to 256 × 256 size.
Revised: The images (CUB200, Cars, Dogs, and Tiny-ImNet) in training and test sets are resized to 256 × 256 size.

Original: Then, the image is cropped with 224 × 224 size at random location in training and at center location in test.
Revised: Then, the image is cropped with 224 × 224 size at random location in training and at center location in test.

Original: Horizontal ﬂipping is applied in training.
Revised: Horizontal ﬂipping is applied in training.

Original: The learning rate decay by 0.1 at [150, 225] epochs from an initial value of 0.1.
Revised: The learning rate decay by 0.1 at [150, 225] epochs from an initial value of 0.1.

Original: The experiments are conduced using GPU “NVIDIA TESLA V100".
Revised: The experiments are conduced using GPU “NVIDIA TESLA V100".

Original: We used one GPU for ResNet-18, and two GPUs for
Revised: We used one GPU for ResNet-18, and two GPUs for

Original: ResNet-50, DenseNet-121, and DenseNet-161.
Revised: ResNet-50, DenseNet-121, and DenseNet-161.

Original: C. D ATASET
Revised: C. D ATASET

Original: We summarize the important information of the previous datasets in Table 3.
Revised: We summarize the important information of the previous datasets in Table 3.

Original: The next section describe how we build the hierarchical tree for each dataset.
Revised: The next section describe how we build the hierarchical tree for each dataset.

Original: C. H IERARCHY FOR DATASETS
Revised: C. H IERARCHY FOR DATASETS

Original: In this section, we describe how we build the hierarchy tree for each dataset.
Revised: In this section, we describe how we build the hierarchy tree for each dataset.

Original: We provide also the ﬁles containing the hierarchy used in the experiments in the folder Hierarchy_files .
Revised: We provide also the ﬁles containing the hierarchy used in the experiments in the folder Hierarchy_files .

Original: Before explaining how we generate the hierarchy, we quickly describe the content of the ﬁles.
Revised: Before explaining how we generate the hierarchy, we quickly describe the content of the ﬁles.

Original: Their name follow the pattern DATASETNAME_child_parent_pairs.txt .
Revised: Their name follow the pattern DATASETNAME_child_parent_pairs.txt .

Original: The ﬁrst line in the ﬁle corresponds to the number of entries.
Revised: The ﬁrst line in the ﬁle corresponds to the number of entries.

Original: Then, the ﬁle is divided into two columns, representing pairs of (child, parent) .
Revised: Then, the ﬁle is divided into two columns, representing pairs of (child, parent) .

Original: This means if the pair ( n 1 , n 2 ) exists in the ﬁle, the node n 2 is the direct parent of the node n 1 .
Revised: This means if the pair ( n 1 , n 2 ) exists in the ﬁle, the node n 2 is the direct parent of the node n 1 .

Original: All labels have been converted into indexes.
Revised: All labels have been converted into indexes.

Original: C.3. C IFAR
Revised: C.3. C IFAR

Original: The hierarchy of Cifar100 is given by the authors.
Revised: The hierarchy of Cifar100 is given by the authors.

Original: C.3. CUB 
Revised: C.3. CUB

Original: We classiﬁed the breed of birds into different groups, in function of the label name.
Revised: We classiﬁed the breed of birds into different groups, in function of the label name.

Original: For instance, the breeds Black_footed_Albatross , Laysan_Albatross and Sooty_Albatross are classiﬁed in the same super-class Albatross .
Revised: For instance, the breeds Black_footed_Albatross , Laysan_Albatross and Sooty_Albatross are classiﬁed in the same super-class Albatross .

Original: C.3. S TANFORD C ARS
Revised: C.3. S TANFORD C ARS

Original: We manually classiﬁed the dataset into nine different super-classes: SUV, Sedan, Coupe, Hatchback,
Revised: We manually classiﬁed the dataset into nine different super-classes: SUV, Sedan, Coupe, Hatchback,

Original: Convertible, Wagon, Pickup, Van and Mini-Van .
Revised: Convertible, Wagon, Pickup, Van and Mini-Van .

Original: In most cases, the super-class name appears in the name of the label.
Revised: In most cases, the super-class name appears in the name of the label.

Original: C.3. S TANFORD D OGS
Revised: C.3. S TANFORD D OGS

Original: The hierarchy is recovered trough the breed presents at the end of the name of each dog specie.
Revised: The hierarchy is recovered trough the breed presents at the end of the name of each dog specie.

Original: For instance, English Setter, Irish Setter , and Gordon Setter are classiﬁed under the class Setter .
Revised: For instance, English Setter, Irish Setter , and Gordon Setter are classiﬁed under the class Setter .

Original: C.3. (T INY ) I MAGENET
Revised: C.3. (T INY ) I MAGENET

Original: The labels of (tiny-)Imagenet are also Wordnet classes.
Revised: The labels of (tiny-)Imagenet are also Wordnet classes.

Original: We used the Wordnet hierarchy to build the ones of (Tiny) Imagenet.
Revised: We used the Wordnet hierarchy to build the ones of (Tiny) Imagenet.

Original: There are also two post-processing steps:
Revised: There are also two post-processing steps:

Original: Wordnet hierarchy is not a tree, which means one node can have more than one ancestor.
Revised: Wordnet hierarchy is not a tree, which means one node can have more than one ancestor.

Original: The choice was systematic: we arbitrarily chose as unique ancestor the ﬁrst one in the sorted list.
Revised: The choice was systematic: we arbitrarily chose as unique ancestor the ﬁrst one in the sorted list.

Original: In the case where a node has one and only one child, the node and its child are merged.
Revised: In the case where a node has one and only one child, the node and its child are merged.

Original: C. G ENERALIZATION PERFORMANCE ALONG DIFFERENT RADIUS DECAY VALUES
Revised: C. G ENERALIZATION PERFORMANCE ALONG DIFFERENT RADIUS DECAY VALUES

Original: In this section, we show in Table 4 how the radius decay affects the test accuracy.
Revised: In this section, we show in Table 4 how the radius decay affects the test accuracy.

Original: In all experiments, we used the Resnet18 architecture with Riemannian gradient descent to optimize the spherical fully-connected layer.
Revised: In all experiments, we used the Resnet18 architecture with Riemannian gradient descent to optimize the spherical fully-connected layer.

Original: Globally, we see that radius decay may inﬂuence the accuracy of the network.
Revised: Globally, we see that radius decay may inﬂuence the accuracy of the network.

Original: However, in most cases, the performance is not very sensitive to this parameter.
Revised: However, in most cases, the performance is not very sensitive to this parameter.

Original: The exception is for tiny-imagenet, where the hierarchy tree has many levels, and thus small values degrade a lot the accuracy.[Table] C
Revised: The exception is for tiny-imagenet, where the hierarchy tree has many levels, and thus small values degrade a lot the accuracy.

Original: [Table]
Revised: [Table]

Original:  L EARNING R ADIUS D ECAY
Revised: C. L EARNING R ADIUS D ECAY

Original: Here
Revised: Here

Original: [Table]
Revised: [Table]

Original: wesimply replace the diagonal matrix D in equation 14 with  learnable parameter matrix which is trained using backpropagation without an additional constraint or a loss function.
Revised: we replace the diagonal matrix D in equation 14 with a learnable parameter matrix which is trained using backpropagation without an additional constraint or a loss function for simplicity.

Original: As shown in 
Revised: As shown in Table

Original: Table 5, this learnable radius is not effective for animproved classiﬁcation performance.
Revised: [Table] this learnable radius is not effective the in terms of an classiﬁcation performance compared to that the predeﬁned radius decay. Proposed parametrization

Original: [Table]
Revised: [Table]

Original: C. L EARNING WITH R ANDOM H IERARCHY
Revised: C. L EARNING WITH R ANDOM H IERARCHY

Original: As shown in Table 6, our proposed methods(Hierarchy, +Manifold, and +Riemann) with a randomly generated hierarchy showed a degraded performance compared to that with a reasonable hierarchical information.
Revised: As shown in Table 6, the methods with a randomly generated hierarchy showed a degraded performance compared to that with a reasonable hierarchical information.

Original: [Table]
Revised: [Table]

Original: C. S UPER - CLASS C LASSIFICATION E FFICIENCY
Revised: C. S UPER - CLASS C LASSIFICATION E FFICIENCY

Original: As shown in Table
Revised: As shown in Table

Original: [Table]
Revised: [Table]

Original: our proposed methods (Hierarchy
Revised: our proposed methods (Hierarchy

Original: [Table]
Revised: [Table]

Original: +Manifold
Revised: +Manifold

Original: [Table]
Revised: [Table]

Original: and +Riemann) outperformedthe test performance of a multitask (multilabel) classiﬁcation method.
Revised: and +Riemann) outperformed a multitask (multilabel) classiﬁcation method in terms of test accuracy performance.

Original: Note that, in the multitask classiﬁcation, a loss function for boosting a superclass categorization performance is used.
Revised: Note that, in the multitask classiﬁcation [Table] a loss function for classiﬁcation using superclasses is used additionally.

Original: [Table]
Revised: [Table]

Original: V ISUALIZATION OF EMBEDDING VECTORS
Revised: V ISUALIZATION OF EMBEDDING VECTORS

Original: In this section, we visualized an embedding vector which is an input of the last classiﬁcation layer.
Revised: In this section, we visualized an embedding vector which is an input of the last classiﬁcation layer.

Original: As suggested by the reviewer, ﬁrst, we show a distribution of two-dimensional vector ( R 2 ).
Revised: As suggested by the reviewer, ﬁrst, we show a distribution of two-dimensional vector ( R 2 ) learned by the networks.

Original: To obtain these vector, we added new layers (i.e. a mapping function R m (cid:55)→ R 2 , m = 512 ) prior to the last 
Revised: To obtain these vectors, we added new layers (i.e. a mapping function R m (cid:55)→ R 2 , m = 512 ) prior to the last FC layer of ResNet-18.

Original: FC layer of ResNet18.
Revised: As two-dimensional vector is not enough to represent a discriminative feature for the ﬁne-grained dataset which have a large number of classes different from MNIST dataset with ten classes with gray level images, we observe multidimensional vectors used in the ResNet.

Original: Second, we use t-SNE, to visualize the high dimensional embedding vector ( R 512 ) of the original ResNet, which is one of popular methods for exploring high-dimensional vector, introduced by van der Maaten and Hinton in 2008.
Revised: Second, we use t-SNE, to visualize the high dimensional embedding vector ( R 512 ) of the original ResNet, which is one of popular methods for exploring high-dimensional vector, introduced by van der Maaten and Hinton.

Original: Even though this method is known to have limitation which is highly dependent on hyperparameters such perplexity values, it is still useful to observe distribution of those high dimensional vectors by using ﬁxed hyperparameters.
Revised: Even though this method is known to have limitation which is highly dependent on hyperparameters such perplexity values, it is still useful to observe distribution of those high dimensional vectors by using ﬁxed hyperparameters.

Original: For a deterministic way of visualization on 2D plane regardless of hyperparameters, ﬁnally, we visualized the embedding vectors using the traditional dimension reduction technique, namely Principal Component Analysis (PCA).
Revised: For a deterministic way of visualization on 2D plane regardless of hyperparameters, ﬁnally, we visualized the embedding vectors using the traditional dimension reduction technique, namely Principal Component Analysis (PCA).

Original: Note that t-SNE and PCA have complementary characteristics (e.g., stochastic vs. deterministic, non-linear vs. linear, capturing local vs. global structures).
Revised: Note that t-SNE and PCA have complementary characteristics (e.g., stochastic vs. deterministic, non-linear vs. linear, capturing local vs. global structures).

Original: 1) Learned two-dimensional representation.
Revised: 1) Learned two-dimensional representation.

Original: We show a distribution of two-dimensional embedding vector which isused for a classiﬁcation.
Revised: We show a distribution of two-dimensional em- bedding vector which is

Revised: used

Original:  As shown in Figure 4a, 4b, 4c,4d, and 4e, embedding vectors of our proposed methods are distributed more closely (Figure 4c,4d, and 4e) with regard to their superclasses.
Revised: for a classiﬁcation As shown in Figure 4, embedding vectors of our proposed methods are distributed more closely (clustered) with regard to their superclasses (Figure 4c, 4d,

Original:  We used Cars dataset since the number of superclasses (nine super-classes) is small enough to show their distribution clearly.
Revised: and 4e). We used Cars dataset since the number of superclasses (nine super-classes) is small enough to show their distribution clearly.

Original: We show the results where the classiﬁcation performances of all compared methods are similar.
Revised: We show the results where the classiﬁcation performances of all compared methods are similar.

Original: We captured the distribution of embedding vectors from an early epoch due to their a slow convergence rate.
Revised: We captured the distribution of embedding vectors from an early epoch due to their a slow convergence rate.

Original: Two-dimensional vector seems too small to classify images of 196 classes which is highly non-linearly distributed.
Revised: Two-dimensional vector seems too small to classify images of 196 classes which is highly non-linearly distributed

Original: [Figure] 2) Observation of clustered structure via t-SNE.
Revised: 2) Observation of locality preserving structure via t-SNE.

Original: We visualize high-dimension embedding vector extracted from ResNet18 which is mapped into two-dimensional space by preserving local pairwise relationship of those vectors, using t-Distributed Stochastic Neighboring Entities (t-SNE).
Revised: We visualize high-dimension embedding vector extracted from ResNet-18 which is mapped into two-dimensional space by preserving local pairwise relationship of those vectors, using t-Distributed Stochastic Neighboring Entities (tSNE).

Original: As in 
Revised: As in Figure

Original: Figure 5a, 5b, 5c,5d, and 5e, embedding vectorsextracted from ResNet18 of our proposed methods are clearly clustered (Figure 5c,5d, and 5e) with regard to their superclasses than that of the baseline methods (Figure 5a and 5b). [Figure]
Revised: embedding vectors of our proposed methods are clearly clustered (Figure 5c,5d, and 5e with regard to their superclasses compared to that of the baseline methods (Figure 5a and 5b

Original: 3) Observation of global structure via PCA.
Revised: 3) Observation of global structure via PCA.

Original: Using PCA, we capture a global structure of high dimensional embedding vectors by projection onto two-dimensional space. 
Revised: Using PCA, we capture a global structure of high dimensional embedding vectors by projection onto two-dimensional space. As

Original: As shown in Figure 6a, 6b, 6c,6d, and 6e, embedding vectors extracted from ResNet18 of our proposed methods have less-Gaussian shapedistribution (Figure 6d and 6e) with regard to their superclasses than that of the baseline methods (Figure 6a and 5b).
Revised: shown in Figure 6, embedding vectors extracted from ResNet- of our proposed methods have less-Gaussian shape

Revised: distribution (

Revised: Figure 6d and 6e)

Revised: with regard to their superclasses than that of the baseline methods

Revised: Figure 6a and 5b

Original: We observe that individual distribution of each classes is similar to that of Figure 4.
Revised: We observe that individual distribution of each classes is similar to that of Figure 4.

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Revised: [Figure]