Original: L EARNING S TRUCTURED R EPRESENTATIONS BY E MBEDDING C LASS H IERARCHY
Revised: L EARNING S TRUCTURED R EPRESENTATIONS BY E MBEDDING C LASS H IERARCHY

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Existing models for learning representations in supervised classification problems are permutation invariant with respect to class labels.
Revised: Existing models for learning representations in supervised classification problems are permutation invariant with respect to class labels.

Original: However, structured knowledge about the classes, such as hierarchical label structures, widely exists in many real-world datasets, e.g., the ImageNet and CIFAR benchmarks.
Revised: However, structured knowledge about the classes, such as hierarchical label structures, widely exists in many real-world datasets, e.g., the ImageNet and CIFAR benchmarks.

Original: How to learn representations that can preserve such structures among the classes remains an open problem.
Revised: How to learn representations that can preserve such structures among the classes remains an open problem.

Original: To approach this problem, given a tree of class hierarchy, we first define a tree metric between any pair of nodes in the tree to be the length of the shortest path connecting them.
Revised: To approach this problem, given a tree of class hierarchy, we first define a tree metric between any pair of nodes in the tree to be the length of the shortest path connecting them.

Original: We then provide a method to learn the hierarchical relationship of class labels by approximately embedding the tree metric in the Euclidean space of features.
Revised: We then provide a method to learn the hierarchical relationship of class labels by approximately embedding the tree metric in the Euclidean space of features.

Original: More concretely, during supervised training, we propose to use the Cophenetic Correlation Coefficient (CPCC) as a regularizer for the crossentropy loss to correlate the tree metric of classes and the Euclidean distance in the class-conditioned representations.
Revised: More concretely, during supervised training, we propose to use the Cophenetic Correlation Coefficient (CPCC) as a regularizer for the crossentropy loss to correlate the tree metric of classes and the Euclidean distance in the class-conditioned representations.

Original: Our proposed regularizer is computationally lightweight and easy to implement.
Revised: Our proposed regularizer is computationally lightweight and easy to implement.

Original: Empirically, we demonstrate that this approach can help to learn more interpretable representations due to the preservation of the tree metric, and leads to betterin-distribution generalization  as well as under sub-population shifts over six real-world datasets.
Revised: Empirically, we demonstrate that this approach can help to learn more interpretable representations due to the preservation of the tree metric, and leads to better generalization in-distribution as well as under sub-population shifts over multiple datasets.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In supervised learning, the cross-entropy loss is often used for classification tasks.
Revised: In supervised learning, the cross-entropy loss is often used for classification tasks.

Original: As a common practice in deep learning, in order to train a model for classification, practitioners build a linear layer over the representation to obtain the logit score of each class.
Revised: As a common practice in deep learning, in order to train a model for classification, practitioners build a linear layer over the representation to obtain the logit score of each class.

Original: A softmax transformation is then applied to convert the logits into a vector belonging to the probability simplex.
Revised: A softmax transformation is then applied to convert the logits into a vector belonging to the probability simplex.

Original: As a result, we can randomly permute the representations of any classes without affecting the performance of the original classification task.
Revised: As a result, we can randomly permute the representations of any classes without affecting the performance of the original classification task.

Original: However, in many real-world datasets, as we move towards fine-grained classification, labels are not independent from each other anymore: ImageNet (Deng et al., 2009) inherits label relationship from WordNet (Fellbaum, 1998), that contains both semantic and lexical connections; iNaturalist (Van Horn et al., 2017) borrows the biological taxonomy so that each image contains seven labels that reflect the morphological characteristic of the organism.
Revised: However, in many real-world datasets, as we move towards fine-grained classification, labels are not independent from each other anymore: ImageNet (Deng et al., 2009) inherits label relationship from WordNet (Fellbaum, 1998), that contains both semantic and lexical connections; iNaturalist (Van Horn et al., 2017) borrows the biological taxonomy so that each image contains seven labels that reflect the morphological characteristic of the organism.

Original: Many existing works (Deng et al., 2014; Yan et al., 2014; Ristin et al., 2015; Guo et al., 2018; Chen et al., 2019) investigated how to leverage this hierarchical information for various purposes, but how to explicitly project this knowledge onto representations remains unexplored.
Revised: Many existing works (Deng et al., 2014; Yan et al., 2014; Ristin et al., 2015; Guo et al., 2018; Chen et al., 2019) investigated how to leverage this hierarchical information for various purposes, but how to explicitly project this knowledge onto representations remains unexplored.

Original: In this paper, we focus on the most common label relationship: tree hierarchy.
Revised: In this paper, we focus on the most common label relationship: tree hierarchy.

Original: As illustrated in
Revised: As illustrated in

Original: Fig.
Revised: Fig.

Original: 1b, given a tree hierarchy of classes, our goal is to learn representations in feature space such that the Euclidean distances between different class centers approximate the distances between these classes in the tree.
Revised: 1b, given a tree hierarchy of classes, our goal is to learn representations in feature space such that the Euclidean distances between different class centers approximate the distances between these classes in the tree.

Original: More concretely, we shall first define a tree metric to be the length of the shortest path connecting two subset of classes in the tree hierarchy.
Revised: More concretely, we shall first define a tree metric to be the length of the shortest path connecting two subset of classes in the tree hierarchy.

Original: Based on this tree metric, we then propose a regularizer, the cophenetic correlation coefficient (CPCC) between sequences of tree metric and Euclidean distance of the feature space, to ensure that the class-conditional representations inherit the tree structure of the classes.
Revised: Based on this tree metric, we then propose a regularizer, the cophenetic correlation coefficient (CPCC) between sequences of tree metric and Euclidean distance of the feature space, to ensure that the class-conditional representations inherit the tree structure of the classes.

Original: Different from the original cross-entropy loss with softmax activation, the proposed CPCC regularizer helps to break the symmetry of permutation invariance among the classes, and thus also improves the interpretability of the learned representations.
Revised: Different from the original cross-entropy loss with softmax activation, the proposed CPCC regularizer helps to break the symmetry of permutation invariance among the classes, and thus also improves the interpretability of the learned representations.

Original: We show that the proposed CPCC regularizer is computationally lightweight with negligible overhead, and can be applied to a wide range of supervised learning paradigms, including empirical risk minimization(ERM), multitask learning, curriculum learning, and a hierarchical learning loss.
Revised: We show that the proposed CPCC regularizer is computationally lightweight with negligible overhead, and can be applied to a wide range of supervised learning paradigms, including standard em- pirical risk minimization and other hierarchical objectives.

Original: For generalization, over six real-world datasets, we demonstrate that our proposed CPCC regularizer leads to significantly better generalization performance  when there is only limited amount of labeled data as well as on unseen tasks with sub-population shifts.
Revised: For generalization, over six real-world datasets, we demonstrate that our proposed CPCC regularizer leads to improved generalization performance on some unseen tasks with sub-population shifts when there is only limited amount of labeled data, and better results on out-of-distribution detection.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: In this section we first introduce the notations used throughout the paper, formulate our learning problem, and then briefly review the CPCC score to quantify the correlation of two sequences.
Revised: In this section we first introduce the notations used throughout the paper, formulate our learning problem, and then briefly review the CPCC score to quantify the correlation of two sequences.

Original: Notations and Setup We shall use X and Y to denote the input and target random variables, living in spaces X and Y , respectively.
Revised: Notations and Setup We shall use X and Y to denote the input and target random variables, living in spaces X and Y , respectively.

Original: In this work, we mainly focus on the supervised classification setting where for each input data point x ∈ X ⊆ R d , there is a ground-truth label y ∈ Y =
Revised: In this work, we mainly focus on the supervised classification setting where for each input data point x ∈ X ⊆ R d , there is a ground-truth label y ∈ Y =

Original: [ k ] := { 1 , . . .
Revised: [ k ] := { 1 , . . .

Original: , k } , where k is the number of output classes.
Revised: , k } , where k is the number of output classes.

Original: We let µ be the joint distribution over ( X, Y ) from where the data is sampled.
Revised: We let µ be the joint distribution over ( X, Y ) from where the data is sampled.

Original: During the learning process, the learner has access to a dataset
Revised: During the learning process, the learner has access to a dataset

Original: D = { ( x j , y j ) } nj =1 of size n sampled from µ .
Revised: D = { ( x j , y j ) } nj =1 of size n sampled from µ .

Original: In the context of representation learning, a learned representation z = f θ ( x ) is obtained by applying a feature encoder f θ : X → Z parametrized by θ to x , where Z ⊆ R p denotes the feature space.
Revised: In the context of representation learning, a learned representation z = f θ ( x ) is obtained by applying a feature encoder f θ : X → Z parametrized by θ to x , where Z ⊆ R p denotes the feature space.

Original: Upon feature vector z , we further apply a linear predictor g : Z → ∆ k , where we use ∆ k to denote the ( k − 1) -dimensional probability simplex.
Revised: Upon feature vector z , we further apply a linear predictor g : Z → ∆ k , where we use ∆ k to denote the ( k − 1) -dimensional probability simplex.

Original: The cross-entropy loss is our objective function.
Revised: The cross-entropy loss is our objective function.

Original: Specifically, let q y ∈
Revised: Specifically, let q y ∈

Original: ∆
Revised: ∆

Original: k be a one-hot vector with the y -th component being 1.
Revised: k be a one-hot vector with the y -th component being 1.

Original: The cross-entropy loss, ℓ CE ( · , · ) between the prediction
Revised: The cross-entropy loss, ℓ CE ( · , · ) between the prediction

Original: g ◦ f ( x ) and the label y is given by ℓ CE ( g ◦ f ( x ) , y ) :
Revised: g ◦ f ( x ) and the label y is given by ℓ CE ( g ◦ f ( x ) , y ) :

Original: =  (cid:80) i ∈ [ k ] q i log( g ( f ( x )) i ) .
Revised: = − (cid:80) i ∈ [ k ] q i log( g ( f ( x )) i ) .

Original: For z, z ′ , ∥ z − z ′
Revised: For z, z ′ , ∥ z − z ′

Original: ∥ 2 denotes the Euclidean distance between them.
Revised: ∥ 2 denotes the Euclidean distance between them.

Original: C LASS H IERARCHY
Revised: C LASS H IERARCHY

Original: In classification problems, the target label Y ∈ [ k ] is treated as a categorical random variable that can take k different nominal values.
Revised: In classification problems, the target label Y ∈ [ k ] is treated as a categorical random variable that can take k different nominal values.

Original: However, there is no particular ordering among these k categories, i.e., for different categories i, j ∈ [ k ] , one can only compare whether i = j or not.
Revised: However, there is no particular ordering among these k categories, i.e., for different categories i, j ∈ [ k ] , one can only compare whether i = j or not.

Original: Formally, letting d H ( i, j ) = 0 if i = j and d H ( i, j ) = 1 otherwise defines a metric d H ( · , · ) over Y .
Revised: Formally, letting d H ( i, j ) = 0 if i = j and d H ( i, j ) = 1 otherwise defines a metric d H ( · , · ) over Y .

Original: However, in many real-world applications the similarity between different classes is not binary.
Revised: However, in many real-world applications the similarity between different classes is not binary.

Original: Consider object classification in ImageNet (Deng et al., 2009) as an example.
Revised: Consider object classification in ImageNet (Deng et al., 2009) as an example.

Original: Intuitively, one would think the distance between the classes corgi and chihuahua to be smaller than that between corgi and panda.
Revised: Intuitively, one would think the distance between the classes corgi and chihuahua to be smaller than that between corgi and panda.

Original: One way to characterize this distance between different classes is through a tree of class hierarchy, also known as a dendrogram.
Revised: One way to characterize this distance between different classes is through a tree of class hierarchy, also known as a dendrogram.

Original: An example is shown in Fig.
Revised: An example is shown in Fig.

Original: 1a.
Revised: 1a.

Original: Formally, let T :
Revised: Formally, let T :

Original: =
Revised: =

Original: ( V, E, d ) be a weighted tree, where V is the set of nodes, E the set of weighted edges in T , and d : V × V → R + specifies the distance between nodes in the tree.
Revised: ( V, E, d ) be a weighted tree, where V is the set of nodes, E the set of weighted edges in T , and d : V × V → R + specifies the distance between nodes in the tree.

Original: Each node v S in T is associated with a subset of class labels S ⊆
Revised: Each node v S in T is associated with a subset of class labels S ⊆

Original: [ k ] , and can be recursively defined as follows:
Revised: [ k ] , and can be recursively defined as follows:

Original: For each class i ∈ [ k ] , there is a corresponding leaf node v i ∈ V .
Revised: For each class i ∈ [ k ] , there is a corresponding leaf node v i ∈ V .

Original: Conversely, each leaf node v i ∈ V is identified with a single class label i ∈ [ k ] .
Revised: Conversely, each leaf node v i ∈ V is identified with a single class label i ∈ [ k ] .

Original: For some S ⊆
Revised: For some S ⊆

Original: [ k ] , if v S ∈ V is not a leaf node in T , then its children form a partition of S .
Revised: [ k ] , if v S ∈ V is not a leaf node in T , then its children form a partition of S .

Original: In other words, if v S 1 , . . .
Revised: In other words, if v S 1 , . . .

Original: , v S c are the children of v S , then ∀ i ̸
Revised: , v S c are the children of v S , then ∀ i ̸

Original: = j , S i ∩ S j = ∅ and ∪ i ∈
Revised: = j , S i ∩ S j = ∅ and ∪ i ∈

Original: [ c ] S i = S .
Revised: [ c ] S i = S .

Original: The root node of T is v [ k ] .
Revised: The root node of T is v [ k ] .

Original: At a colloquial level, the tree T specifies a hierarchy of class labels that represents the structured knowledge among them.
Revised: At a colloquial level, the tree T specifies a hierarchy of class labels that represents the structured knowledge among them.

Original: For example, as shown in Fig.
Revised: For example, as shown in Fig.

Original: 1a of the MNIST dataset, the two children of the root node correspond to the odd and even numbers, respectively.
Revised: 1a of the MNIST dataset, the two children of the root node correspond to the odd and even numbers, respectively.

Original: Accordingly, the distance between digits 1 and 3 is smaller than that between digits 1 and 2.
Revised: Accordingly, the distance between digits 1 and 3 is smaller than that between digits 1 and 2.

Original: 2. C OPHENETIC C ORRELATION C OEFFICIENT (CPCC)
Revised: 2. C OPHENETIC C ORRELATION C OEFFICIENT (CPCC)

Original: In the context of clustering, Sokal & Rohlf (1962) introduced the cophenetic correlation coefficient (CPCC) to evaluate the correspondence between two dendrograms.
Revised: In the context of clustering, Sokal & Rohlf (1962) introduced the cophenetic correlation coefficient (CPCC) to evaluate the correspondence between two dendrograms.

Original: The CPCC is the Pearson’s correlation coefficient between two sequences of pairwise distances.
Revised: The CPCC is the Pearson’s correlation coefficient between two sequences of pairwise distances.

Original: For a class hierarchy T and a node v ∈ V , the depth dt ( v ) of v is the length of the shortest path from v to the root of T .
Revised: For a class hierarchy T and a node v ∈ V , the depth dt ( v ) of v is the length of the shortest path from v to the root of T .

Original: In the original applications of CPCC, the “dendrogrammatic ” ground-truth distance t ( v i , v j ) between a
Revised: In the original applications of CPCC, the “dendrogrammatic ” ground-truth distance t ( v i , v j ) between a

Original: [Figure]
Revised: [Figure]

Original: pair of nodes v i , v j in T is defined as follows:
Revised: pair of nodes v i , v j in T is defined as follows:

Original: [Equation]
Revised: [Equation]

Original: where LCA ( v i , v j ) is the least common ancestor (LCA) of v i and v j .
Revised: where LCA ( v i , v j ) is the least common ancestor (LCA) of v i and v j .

Original: In Fig.
Revised: In Fig.

Original: 1a, the LCA ofand 3 is “odd” while the LCA of 1 and 0 is the root node.
Revised: 1a, the LCA ofand 3 is “odd” while the LCA of 1 and 0 is the root node.

Original: As an example of ground-truth distance, let us consider the class hierarchy tree T and clustering of classes given in Fig.
Revised: As an example of ground-truth distance, let us consider the class hierarchy tree T and clustering of classes given in Fig.

Original: 1b.
Revised: 1b.

Original: In Fig.
Revised: In Fig.

Original: 1b, t ( △ a , △ b ) = 1 since they share an LCA △ at L1, and t ( △ a , □ c ) = 2 as they go up 2 levels to meet at the the root node, which is their LCA.
Revised: 1b, t ( △ a , △ b ) = 1 since they share an LCA △ at L1, and t ( △ a , □ c ) = 2 as they go up 2 levels to meet at the the root node, which is their LCA.

Original: Now consider a dataset D .
Revised: Now consider a dataset D .

Original: For a node v i ∈ T , since v i corresponds to a subset of classes, we use
Revised: For a node v i ∈ T , since v i corresponds to a subset of classes, we use

Original: D i ⊆
Revised: D i ⊆

Original: D to denote the subset of data points whose class label belongs to v i .
Revised: D to denote the subset of data points whose class label belongs to v i .

Original: The pairwise distance between D i and D j is then defined as the Euclidean distance between the center of D i and D j : ρ
Revised: The pairwise distance between D i and D j is then defined as the Euclidean distance between the center of D i and D j : ρ

Original: [Equation]
Revised: [Equation]

Original: i ,
Revised: i ,

Original: [Equation]
Revised: [Equation]

Original: where n i = |D
Revised: where n i = |D

Original: i | and n j = |D
Revised: i | and n j = |D

Original: j | are the number of points in each cluster.
Revised: j | are the number of points in each cluster.

Original: Then, the CPCC score CPCC ( t, ρ ) , between distances t and ρ is defined as:
Revised: Then, the CPCC score CPCC ( t, ρ ) , between distances t and ρ is defined as:

Original: [Equation]
Revised: [Equation]

Original: where t := 2 (cid:80) i<j t ( v i , v j ) /
Revised: where t := 2 (cid:80) i<j t ( v i , v j ) /

Original: k ( k − 1) , ρ := 2 (cid:80)
Revised: k ( k − 1) , ρ := 2 (cid:80)

Original: i<
Revised: i<

Original: j ρ ( v i , v j ) /k
Revised: j ρ ( v i , v j ) /k

Original: ( k − 1) are the averages of all the pairwise distances for t and ρ .
Revised: ( k − 1) are the averages of all the pairwise distances for t and ρ .

Original:  O UR M ETHOD
Revised:  O UR M ETHOD

Original: In this section we first define an alternative tree metric used to measure the distance between two nodes in a tree.
Revised: In this section we first define an alternative tree metric used to measure the distance between two nodes in a tree.

Original: Then, we proceed to discuss our method that uses the proposed tree metric to learn structured representations when a class hierarchy T is available during supervised learning.
Revised: Then, we proceed to discuss our method that uses the proposed tree metric to learn structured representations when a class hierarchy T is available during supervised learning.

Original: 3. T REE M ETRIC
Revised: 3. T REE M ETRIC

Original: Given a tree T =
Revised: Given a tree T =

Original: ( V, E, d ) forming a class hierarchy, wefirst formally define the tree metric d T between any pair of nodes as follows.
Revised: ( V, E, d ) forming a class hierarchy, we formally define the tree metric d T as:

Original: Definition 3.1 (Tree Metric) .
Revised: Definition 3.1 (Tree Metric) .

Original: The tree metric d T ( v, v ′ ) for any pair of nodes v, v ′ ∈ V is the weighted length of the shortest path in T connecting v and v ′ .
Revised: The tree metric d T ( v, v ′ ) for any pair of nodes v, v ′ ∈ V is the weighted length of the shortest path in T connecting v and v ′ .

Original: Proposition 3.1.
Revised: Proposition 3.1.

Original: For any undirected weighted graph G , d T ( · , · ) is a well-defined metric over Y , if all edge weights of G are positive and G is connected.
Revised: For any undirected weighted graph G , d T ( · , · ) is a well-defined metric over Y , if all edge weights of G are positive and G is connected.

Original: Our main motivation to consider the tree metric d T( · , · ) instead of t( · , · ) as defined in Section 2.2 is two-folds.
Revised: Our main motivation to use the tree metric d T instead of t as defined in Section 2.2 is two-folds.

Original: First, while each edge in the class hierarchy T corresponds to a subset relationship, there are other kinds of structured relationships between classes that go beyond the subset relationship.
Revised: First, while each edge in the class hierarchy T corresponds to a subset relationship, there are other kinds of structured relationships between classes that go beyond the subset relationship.

Original: For example, the benchmark dataset MetaShift (Liang & Zou, 2021) also contains a graph to describe the relationship between different subsets of classes.
Revised: For example, the benchmark dataset MetaShift (Liang & Zou, 2021) also contains a graph to describe the relationship between different subsets of classes.

Original: However, in MetaShift the weighted edges do not correspond to the subset relationship as in the case of a class hierarchy, but rather to a similarity/discrepancy measure between them.
Revised: However, in MetaShift the weighted edges do not correspond to the subset relationship as in the case of a class hierarchy, but rather to a similarity/discrepancy measure between them.

Original: In this case, the relationship between classes corresponds to an undirected and weighted graph, where the notion of LCA does not apply any more.
Revised: In this case, the relationship between classes corresponds to an undirected and weighted graph, where the notion of LCA does not apply any more.

Original: In fact, even in the case of a tree, the LCA is subject to change depending on which node in the tree is chosen as the root node.
Revised: In fact, even in the case of a tree, the LCA is subject to change depending on which node in the tree is chosen as the root node.

Original: However, the tree metric d T is invariant to rotations of the tree, and applies to both trees and general graphs as shown in Proposition 3.1.
Revised: However, the tree metric d T is invariant to rotations of the tree, and applies to both trees and general graphs as shown in Proposition 3.1.

Original: Second, the definition of t ( · , · ) does not account for the weights of edges in T .
Revised: Second, the definition of t ( · , · ) does not account for the weights of edges in T .

Original: This implies that all the fine-grained classes under a given super-class are the same.
Revised: This implies that all the fine-grained classes under a given super-class are the same.

Original: However, in many applications, different fine-grained classes may have different proportions or importance, for a given super-class.
Revised: However, in many applications, different fine-grained classes may have different proportions or importance, for a given super-class.

Original: In these cases, the tree metric is more adapted since it also takes into account the edge weights.
Revised: In these cases, the tree metric is more adapted since it also takes into account the edge weights.

Original: Nevertheless, when T is an unweighted class hierarchy tree, we have the following relationship between the proposed tree metric d T ( · , · ) and t ( · , · ) :
Revised: Nevertheless, when T is an unweighted class hierarchy tree, we have the following relationship between the proposed tree metric d T ( · , · ) and t ( · , · ) :

Original: Proposition 3.2.
Revised: Proposition 3.2.

Original: Let T be an unweighted tree with a fixed root node.
Revised: Let T be an unweighted tree with a fixed root node.

Original: Then for any pair of nodes v i , v j ∈ V , d T ( v i , v j ) =
Revised: Then for any pair of nodes v i , v j ∈ V , d T ( v i , v j ) =

Original: 2 t ( v i , v j ) − | dt ( v i ) − dt ( v j ) | .
Revised: 2 t ( v i , v j ) − | dt ( v i ) − dt ( v j ) | .

Original: The proof is deferred to App.
Revised: The proof is deferred to App.

Original: A.
Revised: A.

Original: In particular, Proposition 3.2 states that if the depths of v i and v j are the same in T , then our tree metric reduces to twice of t ( v i , v j ) .
Revised: In particular, Proposition 3.2 states that if the depths of v i and v j are the same in T , then our tree metric reduces to twice of t ( v i , v j ) .

Original: Multiplying a variable by a constant does not affect its correlation with others.
Revised: Multiplying a variable by a constant does not affect its correlation with others.

Original: Henceforth, in what follows, we propose to use the tree metric d T ( · , · ) in replacement of t ( · , · ) to compute CPCC.
Revised: Henceforth, in what follows, we propose to use the tree metric d T ( · , · ) in replacement of t ( · , · ) to compute CPCC.

Original: 3. S TRUCTURED R EPRESENTATIONS BY E MBEDDING THE T REE M ETRIC
Revised: 3. S TRUCTURED R EPRESENTATIONS BY E MBEDDING THE T REE M ETRIC

Original: Now that we have defined our tree metric d T ( · , · ) , we are interested in learning representations f θ ( · ) , by optimizing the model parameter θ , such that the Euclidean distance between the representations of any pair of class centers i, j ∈ [ k ] approximates d T ( v i , v j ) in the tree T .
Revised: Now that we have defined our tree metric d T ( · , · ) , we are interested in learning representations f θ ( · ) , by optimizing the model parameter θ , such that the Euclidean distance between the representations of any pair of class centers i, j ∈ [ k ] approximates d T ( v i , v j ) in the tree T .

Original: Consider a dataset
Revised: Consider a dataset

Original: D = { ( x i , y i ) } ni
Revised: D = { ( x i , y i ) } ni

Original: =1 of size n for classification problem over ∆ k .
Revised: =1 of size n for classification problem over ∆ k .

Original: We propose to use the CPCC between ρ Z ( · , · ) and d T ( · , · ) as a regularizer to the cross-entropy loss, resulting in the loss function:
Revised: We propose to use the CPCC between ρ Z ( · , · ) and d T ( · , · ) as a regularizer to the cross-entropy loss, resulting in the loss function:

Original: [Equation]
Revised: [Equation]

Original: where λ > 0 is the coefficient used to control the strength of the regularizer.
Revised: where λ > 0 is the regularization strength ( λ = 1 in the experiments).

Original: The Euclidean distance ρ Z is computed in feature space.
Revised: The Euclidean distance ρ Z is computed in feature space.

Original: Concretely, we first apply the encoder f
Revised: Concretely, we first apply the encoder f

Original: θ to D and obtain a set of points in Z × Y : D Z := { ( f θ ( x i ) , y i ) } ni
Revised: θ to D and obtain a set of points in Z × Y : D Z := { ( f θ ( x i ) , y i ) } ni

Original: =1 .
Revised: =1 .

Original: Then, we partition D
Revised: Then, we partition D

Original: Z into k subsets according to the ground-truth labels, and consider the same tree structure T on D Z .
Revised: Z into k subsets according to the ground-truth labels, and consider the same tree structure T on D Z .

Original: Note the negative sign before coefficient λ in the above formulation, as we wish to maximize the CPCC score.
Revised: Note the negative sign before coefficient λ in the above formulation, as we wish to maximize the CPCC score.

Original: In practice, at each iteration during training, since stochastic optimization methods are used, we process a batch of inputs instead of the whole data set D .
Revised: In practice, at each iteration during training, since stochastic optimization methods are used, we process a batch of inputs instead of the whole data set D .

Original: For each incoming batch, we track the number of finest classes represented in the batch before any pairwise calculation.
Revised: For each incoming batch, we track the number of finest classes represented in the batch before any pairwise calculation.

Original: When all the inputs in a batch come from the same coarse class, the CPCC score is not well-defined due to thevariance of d T .
Revised: When all the inputs in a batch come from the same coarse class, the CPCC score is not well-defined due to thevariance of d T .

Original: This can happen when the batch size is relatively small.
Revised: This can happen when the batch size is relatively small.

Original: In such cases, we fix the value of the CPCC regularizer to 0 to avoid the numerical division by zero error.
Revised: In such cases, we fix the value of the CPCC regularizer to 0 to avoid the numerical division by zero error.

Original: Time Complexity of the CPCC Regularizer
Revised: Time Complexity of the CPCC Regularizer

Original: The computation of our CPCC regularizer is lightweight.
Revised: The computation of our CPCC regularizer is lightweight.

Original: For a feature space with p dimensions, for each training iteration, there will be at most O ( p min( b 2 , k 2 )) additional computations, where b is the batch size.
Revised: For a feature space with p dimensions, for each training iteration, there will be at most O ( p min( b 2 , k 2 )) additional computations, where b is the batch size.

Original: Such an overhead is often negligible when compared with the computations needed to train a neural network.
Revised: Such an overhead is often negligible when compared with the computations needed to train a neural network.

Revised: In Appendix B we also provide a brief discussion on the convergence of optimizing the above objective function with

Original:  3. T HE B ENEFITS OF S TRUCTURED R EPRESENTATIONS
Revised: SGD. 3. T HE B ENEFITS OF S TRUCTURED R EPRESENTATIONS

Original: In what follows, we describe two potential benefits of learning structured representations with the proposed CPCC regularizer, before providing thorough empirical validation in Section 4.
Revised: In what follows, we describe two potential benefits of learning structured representations with the proposed CPCC regularizer, before providing thorough empirical validation in Section 4.

Original: Interpretability
Revised: Interpretability

Original: As we briefly discussed before, one potential drawback of the representations learned through supervised learning is the lack of interpretability.
Revised: As we briefly discussed before, one potential drawback of the representations learned through supervised learning is the lack of interpretability.

Original: Recent work (Papyan et al., 2020;
Revised: Recent work (Papyan et al., 2020;

Original: Han et al., 2021) have both empirically and theoretically (under certain assumptions) shown that under the cross-entropy loss, when enough training has happened, the learned representations will have reduced variance within each class, and the set of features corresponding to different classes will converge to the so-called simplex equiangular tight frame (ETF).
Revised: Han et al., 2021) have both empirically and theoretically (under certain assumptions) shown that under the cross-entropy loss, when enough training has happened, the learned representations will have reduced variance within each class, and the set of features corresponding to different classes will converge to the so-called simplex equiangular tight frame (ETF).

Original: Yet, the vertices of the simplex
Revised: Yet, the vertices of the simplex

Original: ETF are symmetric (in the sense of being permutation-invariant), hence the class features do not necessarily reflect the similarities/differences between different classes, even in feature space.
Revised: ETF are symmetric (in the sense of being permutation-invariant), hence the class features do not necessarily reflect the similarities/differences between different classes, even in feature space.

Original: By enforcing the Euclidean distances in feature space between different classes to be close to the tree metric through our CPCC regularization, we attempt to break the symmetry in learning the features.
Revised: By enforcing the Euclidean distances in feature space between different classes to be close to the tree metric through our CPCC regularization, we attempt to break the symmetry in learning the features.

Original: This can potentially lead to more interpretable features, as closer classes (in the sense of the tree metric) are closer to each other in feature space.
Revised: This can potentially lead to more interpretable features, as closer classes (in the sense of the tree metric) are closer to each other in feature space.

Original: Generalization Another by-product of structured representations is potentially better generalization both in-distribution when only limited amount of labels is available, or under sub-population shifts (Santurkar et al., 2020).
Revised: Generalization Another by-product of structured representations is potentially better generalization both in-distribution when only limited amount of labels is available, or under sub-population shifts (Santurkar et al., 2020).

Original: To see this, note that the goal of our CPCC regularizer is consistent with classification accuracy: it essentially pushes data from different classes away proportionally to their distance in the tree.
Revised: To see this, note that the goal of our CPCC regularizer is consistent with classification accuracy: it essentially pushes data from different classes away proportionally to their distance in the tree.

Original: Consequently, for sub-population shifts,  future unseen fine-grained classes from the same coarse category will be further away from those under a different coarse category.
Revised: Consequently, for sub-population shifts, if the hierarchy correctly captures coarse-fine relationship, future unseen fine-grained classes from the same coarse category will be further away from those under a different coarse category.

Original: This could help generalize to unseen fine-grained classes in zero or few-shot learning.
Revised: This may help generalize to unseen fine-grained classes in zero or few-shot learning.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we apply our proposed methodon real-world datasets to: (i) study how using CPCC during training affects the representation learnt under various training objectives (Section 4.3), and (ii) see how the learned structured representations can help generalization  (Section 4.4).
Revised: In this section, we apply our proposed method to: (i) study how using CPCC during training affects the representation learnt under various training objectives (Section 4.3), and (ii) see how the learned structured representations can improve generalization and OOD detection (Section 4.4).

Original: 4. D ATA
Revised: 4. D ATA

Original: We conduct our experiments on MNIST (Lecun et al., 1998), CIFAR100 (Krizhevsky, 2009), and
Revised: We conduct our experiments on MNIST (Lecun et al., 1998), CIFAR100 (Krizhevsky, 2009), and

Original: BREEDS (Santurkar et al., 2020).
Revised: BREEDS (Santurkar et al., 2020).

Original: By using this variety of datasets and hierarchies, we get a comprehensive overview of the usefulness of CPCC as a regularizer. 
Revised: By using this variety of datasets and hierarchies, we get a comprehensive overview of the usefulness of CPCC as a regularizer. See Section G for the full hierarchies.

Original: MNIST contains handwritten digits from 0 to 9.
Revised: MNIST contains handwritten digits from 0 to 9.

Original: We define odd and even digits to be two coarse classes (Figure 1a).
Revised: We define odd and even digits to be two coarse classes (Figure 1a).

Original: The digits in the leaves are called fine classes below.
Revised: The digits in the leaves are called fine classes below.

Original: The artificial level based on odd- and even-ness corresponds to concepts that are not visually observable.
Revised: The artificial level based on odd- and even-ness corresponds to concepts that are not visually observable.

Original: CIFAR100 comes with a predefined hierarchy: its coarse level has 20 classes, each containingfine classes ( e.g. , beaver, dolphin, otter, seal and whale belongs to aquatic mammals).
Revised: CIFAR100 comes with a predefined hierarchy: its coarse level has 20 classes, each containing 5 fine classes ( e.g. , beaver, dolphin, otter, seal and whale belongs to aquatic mammals).

Original: While the hierarchies are semantically meaningful, the coarse level labels are not purely defined by visual similarities.
Revised: While the hierarchies are semantically meaningful, the coarse level labels are not purely defined by visual similarities.

Original: For instance, it is hard to tell the size of an animal from its image (making the coarse classes large omnivores/herbivores and mid-size mammals difficult to distinguish).
Revised: For instance, it is hard to tell the size of an animal from its image (making the coarse classes large omnivores/herbivores and mid-size mammals difficult to distinguish).

Original: BREEDS is a benchmark built on ImageNet (Deng et al., 2009).
Revised: BREEDS is a benchmark built on ImageNet (Deng et al., 2009).

Original: It contains a manually calibrated label hierarchy, based solely on shared visual characteristics.
Revised: It contains a manually calibrated label hierarchy, based solely on shared visual characteristics.

Original: Santurkar et al.
Revised: Santurkar et al.

Original: (2020) proposed four tasks: LIVING17 , ENTITY13 , ENTITY30 , and NONLIVING26 .
Revised: (2020) proposed four tasks: LIVING17 , ENTITY13 , ENTITY30 , and NONLIVING26 .

Original: For each task, we consider their leaf nodes, which are ImageNet classes, as our fine level classes, and define the coarse levels to be their “superclasses” at different granularities in the hierarchy.
Revised: For each, we consider the leaf nodes, which are ImageNet classes, as our fine level classes, and define the coarse levels to be their “superclasses” at different depths.

Original: We end up with trees that only contain the root node and two levels of the initial hierarchy, and hide any other intermediate relationships.
Revised: We end up with trees that only contain the root node and two levels of the initial hierarchy, and ignore intermediate relationships for CPCC regularization. New Levels Based on the coarse and fine levels in the hierarchy, we insert a mid level between the coarse and fine ones, as well as a coarser level between the coarse one and the root node. This results in classes verifying k coarser < k coarse < k mid < k fine .

Revised: In MNIST, the mid classes are 1,3, (odd numbers ≤ 5), 7,9 (odd numbers > 5), 0,2,4 (even numbers ≤ 5), and 6,8 (even numbers > 5).

Revised: We do not consider a coarser level (it is trivial to train on the root node).

Original: 4. B ASELINES AND M ETRICS
Revised: In CIFAR100, each coarse level (containing 5 fine classes) is split into arbitrary groups of 2/3 fine classes, creating 40 classes in the mid level. 2 arbitrary coarse classes are merged into 1 coarser label, creating 10 coarser labels. Since BREEDS contains 8 non-root levels in total, and all 4 datasets’ coarse levels have a depth ≥ 2 , we use the original hierarchy and let the mid level be one level above the fine classes, and the coarser level be one level above the coarse classes. Source & Target Split

Revised: We split BREEDS into source ( s ) and target ( t ): s and t have the same coarse labels but fine classes are different, and coarser level is the same but mid level is different for s and t .

Revised: Following this idea, recall we split MNIST/CIFAR’s coarse levels into groups ofand 3.

Revised: We take 60 fine classes as CIFAR s and the rest as CIFAR t , 6 classes as MNIST s , 4 in

Revised: MNIST t , so source and target have different mid/fine but the same coarse/coarser labels.

Revised: Due to this construction, there is only one mid class in each of CIFAR/MNIST s/t ’s coarse class.

Revised: On the other hand, BREEDS ’s coarse classes have many mid children.

Revised: See App.

Revised: G for the full hierarchies.

Revised: 4. B ASELINES AND M ETRICS

Original: As mentioned above, we operate in a fully supervised setting.
Revised: As mentioned above, we operate in a fully supervised setting.

Original: We denote our neural network as a function h : X → ∆ k . h .
Revised: We denote our neural network as a function h : X → ∆ k . h .

Original: =
Revised: =

Original: g ◦ f θ can be decomposed into a feature extractor f θ , and a linear classifier g to which the softmax is applied.
Revised: g ◦ f θ can be decomposed into a feature extractor f θ , and a linear classifier g to which the softmax is applied.

Original: Training is performed using the followingfour objectives, on the fine-coarse hierarchy of MNIST, CIFAR, and BREEDS, with and without CPCC as a regularization (seethe App.
Revised: Training is performed using the following objectives, on the fine-coarse hierarchy of MNIST, CIFAR, and BREEDS, with and without CPCC as a regularization (see App.

Original: D for additional implementation details).
Revised: C, F for more details).

Original: • ERM :
Revised: Our baselines include:

Original: Train using ℓ CE  on  fine classes only, without leveraging any hierarchical information:
Revised: • Flat ℓ CE : training on the fine classes only without leveraging any hierarchical information.

Original: [Equation]

Original: • Multi-task Learning :We jointly train a two-headed network to treat fine and coarse as two separate tasks.
Revised: • Multi-task Learning : jointly training a two-headed network to treat fine and coarse as two separate tasks.

Original: The network then becomes h : X → ∆ k 1 × ∆ k 2 with a shared feature encoder f θ : X → Z and two classifiers g coarse : Z → ∆ k 1 , g fine : Z → ∆ k 2 .

Original: k 1 is the number of coarse classes and k 2 the number of fine classes.

Original: The loss function is the sum of the cross-entropies on the fine and coarse classification tasks, and we simply set the weight of the two parts to 1 : [Equation]
Revised: The loss function is the sum of the cross-entropies on the fine and coarse classification tasks, and we simply set the weight of the two parts to 1 .

Original: • Curriculum Learning :
Revised: • Curriculum Learning :

Original: In the spirit of curriculum learning (Bengio et al., 2009), we first train on the coarse classes using Eq. 3 and use y coarse instead of y fine .
Revised: In the spirit of curriculum learning, we first train on the coarse classes using ℓ CE and use y coarse instead of y fine .

Original: In the second step, we remove the linear classifier and fine tune a new one on the fine level labels with Equation 3 as the loss function.
Revised: In the second step, we remove the linear classifier and fine tune a new one on the fine level labels with ℓ CE as the loss function.

Original: • Sum Loss : We define a hierarchical Sum Loss as [Equation]
Revised: • Sum Loss : We define a hierarchical Sum Loss as (cid:80) ℓ CE ( y coarse , W h ( x ))

Original:  W is a k 1 by k 2 matrix representing the relationships in the label tree: if a fine class i belongs to a coarse class j , then W ji is 1 , otherwise the entry is set to 0 .
Revised: + ℓ CE ( y fine , h ( x )) , W is a k 1 by k 2 matrix representing the relationships in the label tree: if a fine class i belongs to a coarse class j , then W ji is 1 , otherwise the entry is set to 0 .

Original: In other words, the probability of belonging to a given coarse class is simply the sum of the probabilities of its descendants ( i.e. , fine classes) in the tree.
Revised: • HXE : The Hierarchical Cross Entropy (Bertinetto et al., 2020) that replaces the predicted output in ℓ CE with weighted hierarchical class conditional probabilities. Soft : The soft labels objective (Bertinetto et al., 2020) where labels in ℓ CE are derived from a mapping function to encode class node similarity in y . Quad : The Quadruplet Zhang et al.

Original: This way, we use the information from both levels with one head only. Metrics To evaluate the representation structure learnt with the various loss functions, as well as the influence of CPCC, we use (1) silhouette scores (Rousseeuw, 1987) to measure the salience of clustering patterns at the coarse level, (2) CPCC as a metric to measure howis the whole representation structure  similar to the fine-coarse hierarchy, (3) t -SNE (Van der Maaten & Hinton, 2008) for visualization of the learnt embeddings in 2D, (4) a symmetric distance matrix to evaluate the hierarchical structure.
Revised: (2016) multi-task loss which combines ℓ CE with a generalized triplet loss to enforce different margins at different levels of the hierarchy. Metrics To evaluate the representation structure learnt with the various loss functions, as well as the influence of CPCC, we use (i) silhouette scores (Rousseeuw, 1987) to measure the salience of clustering patterns at the coarse level, (ii) CPCC as a metric to measure how the whole representation structure is similar to the fine-coarse hierarchy, (iii) t -SNE (Van der Maaten & Hinton, 2008) for visualization of the learnt embeddings in 2D, and (iv) a symmetric distance matrix to evaluate the hierarchical structure.

Original: Specifically, we calculate the Euclidean distance between the mean representation vectors for each pair of fine classes.
Revised: Specifically, we calculate the Euclidean distance between the mean representation vectors for each pair of fine classes.

Original: The matrix is organized in a way where fine classes from the same coarse class are grouped together, so that the coarse within-cluster distance is shown around the diagonal while other entries present coarse level between-cluster distances.
Revised: The matrix is organized in a way where fine classes from the same coarse class are grouped together, so that the coarse within-cluster distance is shown around the diagonal while other entries present coarse level between-cluster distances.

Original: Though our goal is not the state-of-the-art, we also report test accuracy as a non-hierarchical metric.

Original: Ideally, enforcing a hierarchical representation does not hurt fine level accuracy.

Original: We calculate the coarse accuracy by appropriately marginalizing the fine classes probabilities.

Original: 4. S TRUCTURE OF THE LEARNT REPRESENTATIONS
Revised: 4. S TRUCTURE OF THE LEARNT REPRESENTATIONS

Original: Figures 2and 3 show the effects of training with CPCC, which matches our expectations shown in Fig.
Revised: Figure 2 show the effects of training with CPCC, which matches our expectations shown in Fig.

Original: 1b.
Revised: 1b.

Original: In the distance matrix, we can see that the within-coarse cluster distance is much smaller than the between-coarse cluster distance (corresponding to off-diagonal 5 by 5 blocks).
Revised: In the distance matrix, we can see that the within-coarse cluster distance is much smaller than the between-coarse cluster distance (corresponding to diagonal 5 by 5 blocks).

Original: This fact is verified qualitatively in the t -SNE plots from Fig. 3, where coarse groups tend to be better separated.
Revised: This fact is verified qualitatively in the t -SNE plots, where coarse groups tend to be better separated.

Original: Similar patterns are observed when CPCC is paired with the other loss functions described above.
Revised: Similar patterns are observed when CPCC is paired with the other loss functions described above.

Original: We want to point out that although other setups have a more structured representation to some extent, these are not as perfect as when paired with CPCC (see App.
Revised: We want to point out that although other setups have a more structured representation to some extent, these are not as perfect as when paired with CPCC (see App.

Original: B for the figures).
Revised: D for the figures).

Original: In Table 1, we see that the objectives leveraging the hierarchical information tend to increase the
Revised: In Table 1, we see that the objectives leveraging the hierarchical information tend to increase the

Original: CPCC.
Revised: CPCC.

Original: This is particularly true of the multi-task  setting.
Revised: This is particularly true of the multi-task and soft labels setting.

Original: But,rather obviously, directly optimizing 
Revised: But, directly optimizing the

Original: CPCC score  gives the largest gains,on both  CPCC and silhouette scores.
Revised: CPCC score still gives the largest gains, both on CPCC and silhouette scores.

Revised: 4. G ENERALIZATION ON DATASETS WITH A SHARED HIERARCHY

Revised: Given a representation f θ trained on the fine-coarse hierarchy (4.2), we want to see if structured representations help performance in-hierarchy , i.e. , on the fine and coarse classes the model was trained on; as well as out-of-hierarchy , i.e. , on new levels and/or new classes of the hierarchy.

Revised: In-hierarchy In this setting, we evaluate the models on classes and levels used during training to construct the various objectives and the tree metric ( i.e. , the fine and coarse classes).

Original: Of particular note, adding our CPCC regularizer leads to better test accuracy at bothfine and coarse levels, across objectives and datasets, with gains sometimes exceeding 1%.
Revised: Results can be found in the FineAcc and CoarseAcc columns of Table 1. Adding our CPCC regularizer leads to better test accuracy at both levels, across objectives and datasets, with gains sometimes exceeding 1%.

Original: According to Goyal et al.
Revised: According to Goyal et al.

Original: (2021), such a performance gain  is rarely observed when hierarchical information is leveraged.
Revised: (2021), such a performance gain (especially on fine classes) is rarely observed when hierarchical information is leveraged.

Original: Overall, our findings suggest that when such information is available, using CPCC as a regularizer can prove beneficial.
Revised: Overall, our findings suggest that when such information is available, using CPCC as a regularizer is beneficial.

Original: 4. O NE - SHOT GENERALIZATION TO NEW LEVELS IN THE HIERARCHY
Revised: Out-of-hierarchy Two questions naturally arise in our hierarchical setting.

Original: Given a representation f θ trained following 4.2, we want to see if structured representations help transfer knowledge to other levels of thetree hierarchy. 
Revised: The first one is how well CPCC structured representations generalize to new levels of the hierarchy. To answer this question, we report in Table 1 the accuracy on the new mid and coarser levels defined in Section 4.1, and not used during training.

Original: To that end, we freeze f θ and fine tune a linear classifier g mid or g coarser on a single image from each label in the new levels (one-shot generalization).
Revised: This accuracy is obtained zero-shot, via a simple marginalization ( e.g. , the probability of a mid class is the sum of the probabilities of all fine classes that belong to it).

Original: New levels Based on the coarse and fine levels in the hierarchy, we insert a mid level between the coarse and fine ones, as well as a coarser level between the coarse one and the root node. This results in classes verifying k coarser < k coarse < k mid < k fine .
Revised: There too, adding the CPCC regularization results in performance gains (also see Appendix Table 4).

Original: In MNIST, the mid classes are 1,3, (odd numbers ≤ 5), 7,9 (odd numbers > 5), 0,2,4 (even numbers ≤ 5), and 6,8 (even numbers > 5).
Revised: The second natural question is if CPCC structured representations can generalize to classes unseen in the training hierarchy. Assume a model has learned that cats and dogs are animals. Does knowing the animal concept help it understand giraffes or horses better?

Original: We do not consider a coarser level (it is trivial to train  on the root node).
Revised: To explore this, we train our models on the source split of the mid and fine classes (Sec. 4.1), and evaluate their performance on the target split.

Original: In CIFAR100, each coarse  level (containing 5 fine classes) is split into 2/3, creating 40 classes in the  midlevel.arbitrary coarse classes are merged into 1 coarser label, creating 10 coarser labels. Since BREEDS contains 8 non-root levels in total, and all 4 datasets’ coarse levels have a depth ≥ 2 , we use the original hierarchy and let the mid level be one level above the fine classes, and the coarser level be one level above the coarse classes. See App. E for the full hierarchies in all datasets.
Revised: We can still apply “zero-shot” transfer to coarse and coarser level via marginalization, but fine tuning is necessary to classify the new mid and fine classes: we freeze f θ and fine tune a linear classifier g mid or g fine on a single image from each new target label (one-shot generalization).

Original: Results are shown inthe last two columns of Table 1.
Revised: Results are shown in Table 2.

Original: On the  coarser levels, we see that without CPCC, objectives (multi-task inparticular) that leverage hierarchical information improve performance.
Revised: First, under this subpopulation shift, using CPCC still outperforms the original loss functions on coarse and coarser levels (zero-shot), which is consistent with results in

Original: Adding our CPCC regularizer provides consistent and significant gains, across datasets and objectives.
Revised: Table 1. Second, in one-shot generalization to new mid levels, CPCC gives an often large advantage.

Original: Intuitively, as all fine classes are grouped together within coarse groups, if one data point is randomly selected, then other data points in the same coarse class will readily be assigned the samecoarser label.
Revised: Intuitively, as all fine classes are grouped together within coarse groups, if one data point is randomly selected, then other data points in the same coarse class will readily be assigned the same label.

Original: Without this structure in the representation, generalization will be more difficult as all fine labels are evenly distributed.
Revised: Without this structure in the representation, generalization is more difficult as all fine labels are evenly distributed.

Original: These findings extend to the mid level task, with the exception of ENTITY13 , whereCPCC hurts the multi-task performance. We posit that it is due to the hierarchical structure of that dataset, where each coarse label has many mid level children and each mid label has very few fine level children (see App.
Revised: The only notable exception is ENTITY13 , where each coarse label has many mid level children. When grouping the representations by coarse levels, it becomes difficult to recover the decision boundary of mid classes as it is close to that of other fine classes in the same [Table]

Original: E for the precise hierarchy).
Revised: 34) FlatCPCC 93.56 (0.46) 48.26 (0.51) 84.97 (0.68) 90.52 (0.54) 91.13 (0.61) 93.66 (0.36) ENTITY13 s Flat 34.97 (0.43) 3.12 (0.03) 82.36 (0.38) 84.52 (0.33) 90.43 (0.34) 93.38 (0.33) FlatCPCC 90.98 (0.22) 38.68 (0.17) 83.37 (0.37) 85.53 (0.28) 91.50 (0.09) 94.22 (0.06)

Original: In such a case, when forcing the representation to be grouped by coarse level, it becomes difficult to recover the decision boundary of mid classes as it is close to that of fine classes within the same coarse label.
Revised: ENTITY30 s

Original: On the other hand, when coarse classes have fewer children as in the other datasets, our CPCC regularizer provides a significant boost over the baselines.
Revised: Flat 24.29 (0.30) 3.07 (0.06) 80.81 (0.35) 82.28 (0.35) 86.00 (0.25) 89.27 (0.27) FlatCPCC 73.55 (0.77) 29.43 (0.12) 82.00 (0.31) 83.57 (0.34) 87.60 (0.13) 90.72 (0.28) NONLIVING26 s Flat 26.95 (0.31) 5.90 (0.13) 80.79 (0.36) 83.49 (0.30) 80.79 (0.36) 87.28 (0.25) FlatCPCC 82.20 (0.44) 34.63 (0.31) 82.96 (0.35) 85.88 (0.37) 87.42 (0.36) 89.35 (0.40) [Figure] coarse class.

Revised: Third, CPCC regularization is often harmful to one-shot fine level generalization due to coarse grouping: new fine classes are close together at the coarse level, making them hard to linearly separate.

Revised: We do observe that other hierarchical methods have some advantage compared to the flat cross entropy.

Revised: Interestingly, when coarse classes are closer to the fine leaves ( LIVING and NONLIVING26 compared to ENTITY ), imposing a coarse grouping becomes helpful, and generalization to unseen classes is improved by CPCC.

Revised: OOD Detection Finally, we explore whether OOD detection is improved by CPCC.

Revised: Following Hoffmann et al.

Revised: , we consider CIFAR10 as OOD compared to CIFAR100 (they have disjoint fine classes), and fit a 100-class conditional multivariate Gaussian on the trained CIFAR100 embeddings.

Revised: The OOD score is the probability that the model gives a higher likelihood to the true class of a random CIFAR100 image than to the maximum class likelihood of a random CIFAR10 one.

Revised: Results are in Fig.

Revised: 3: using CPCC significantly boosts the detection score for all objectives.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: There were many works exploiting label hierarchy and we only refer to the most related ones.
Revised: Many works exploit label hierarchy, we only refer to the most related ones.

Original: However, to the best of our knowledge, none of the previous work set learning structured representations as main objective or embedded the tree metric under this context.

Original: Background of Baseline Methods The most basic label hierarchytree contains only two level, coarse and fine. One way to set up baseline models is to treat them as two tasks, which can be  trained jointly or sequentially.
Revised: The simplest label hierarchy contains only two level, coarse and fine, which can be treated as two tasks trained jointly or sequentially.

Original: The former originates from Multi-task Learning (MTL) where part of single network is shared for multiple heads for each task during training (Caruana, 1997).
Revised: The former originates from Multi-task Learning (MTL) where part of single network is shared for multiple heads for each task during training (Caruana, 1997; Zhao et al., 2020;

Original: To deal with hierarchical labels, algorithms were developed to adjust the weight between different levels of supervision in the loss function which is the sum of weighted cross-entropy for each task (Zhao et al., 2020;

Original: Inoue et al., 2020).
Revised: Inoue et al., 2020).

Original: The latter echoes with Curriculum Learning (CL) firstly proposed in Bengio et al.
Revised: The latter echoes with Curriculum Learning (CL) (Bengio et al., 2009), where [Table]

Original: (2009), where it is claimed that pretraining with a easy task will help the convergence and performance on a hard task.
Revised: pretraining with a easy task will help the convergence and performance on a hard task.

Original: We define coarse level classes as the easier task due to the intuition that coarse level contains less classes and more images for each class as recommended in Ahn et al.
Revised: We define coarse level classes following Ahn et al.

Original: ; Peterson et al.
Revised: ; Peterson et al.

Original: ; Wang & Cottrell (2015) to emulate human learning behavior.
Revised: (2018); Wang & Cottrell (2015); Stretcu et al. (2021) to emulate human learning behavior.

Original: Another approach to use multi-level labels, Sum Loss, is to directly modify the loss function by marginalizing fine classes probability for coarse classes.
Revised: Sum Loss directly modifies the loss function by marginalizing fine classes probability for coarse classes.

Original: Su & Maji (2021); Hu et al.
Revised: Su & Maji (2021); Hu et al.

Original: (2018) defined a similar version to address some partially labeled tasks.
Revised: (2018) defined a similar version to address some partially labeled tasks.

Original: Given that training with naive cross-entropy loss tends to separate classes evenly, we hope that including an extra level of supervision via these baseline approaches will not only push fine classes apart, but also present coarse level separation, although as we have shown that all three baselines are not as perfect as using CPCC as  a regularizer regarding the  representation.
Revised: Bertinetto et al. (2020) provided a thorough survey of different types of hierarchical methods, as well as label embedding methods (encoding hierarchical information into labels, the Soft objective), hierarchical losses ( Quad (Zhang et al., 2016), HXE ). None of these use a regularization method, with the exception of group overlapping lasso (Zhao et al., 2011). However, it was introduced for logistic regression, making it hard to to be applied to modern neural networks that use the penultimate layer as its representation.

Original: The most common motivation of hierarchical models is to improve the fine-level accuracy (Deng et al., 2014; Guo et al., 2018; Chen et al., 2019; Yan et al., 2014).
Revised: The most common motivation of hierarchical models is to improve the fine-level accuracy.

Original: Interestingly,the results of accuracy improvement was mixed: while most work claimed to gain performance improvement, Wang & Cottrell (2015) stated that this improvement was limited, and Goyal et al.
Revised: Interestingly, accuracy improvements are often mixed: while most works claimed to gain performance improvement, Wang & Cottrell (2015) stated that this improvement was limited, and Goyal et al.

Original: (2021) claimed most hierarchical models lead to worse performance on non-hierarchical accuracy metrics.
Revised: (2021) claimed most hierarchical models lead to worse performance on non-hierarchical accuracy metrics.

Original: Additionally, using coarse level labels often appears in a[Table] weakly supervised setting, where coarse classes are always available but fine class labels are only accessible for part of data, to reduce annotation cost at a finer level (Taherkhani et al., 2019; Lei et al., 2017; Ristin et al., 2015).
Revised: Additionally, using coarse level labels often appears in a weakly supervised setting, where coarse classes are always available but fine class labels are only accessible for part of data, to reduce annotation cost at a finer level (Taherkhani et al., 2019; Lei et al., 2017; Ristin et al., 2015).

Original: Many other works built hierarchy from dataset (Murdock et al., 2016; Li et al., 2010; Verma et al., 2012; Han et al., 2018; Zheng et al., 2017).
Revised: Other works built hierarchy from dataset (Murdock et al., 2016; Li et al., 2010; Verma et al., 2012; Han et al., 2018; Zheng et al., 2017).

Original: We only name a few since they are very different from our setting where the hierarchy is defined before training.
Revised: We only name a few since they are very different from our setting where the hierarchy is defined before training.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: How to include label relation into representation is an open question.
Revised: How to include label relation into representation is an open question.

Original: In this paper, in the context of tree label hierarchies, we use the cophenetic correlation coefficient as a regularizer to embed this hierarchical relationship into representations, and outperform other baseline methods.
Revised: In this paper, in the context of tree label hierarchies, we use the cophenetic correlation coefficient as a regularizer to embed this hierarchical relationship into representations, and outperform other baseline methods.

Original: CPCC has multiple advantages, including low time complexity, better interpretability, flexibility on any supervised learning paradigms, and it can be applied on any common label relation graphs.
Revised: CPCC has multiple advantages, including low time complexity, better interpretability, flexibility on any supervised learning paradigms, and it can be applied on any common label relation graphs.

Original: We also demonstrate that it leads to better generalization performance on several downstream tasks.
Revised: We also demonstrate that it leads to better generalization performance on several downstream tasks.

Original: All these benefits show that our method provides an interesting solution to this important problem.
Revised: All these benefits show that our method provides an interesting solution to this important problem.

Original: A P ROOF OF P ROPOSITIONS
Revised: A P ROOF OF P ROPOSITIONS

Original: We first restate propositions and then provide the proof.
Revised: We first restate propositions and then provide the proof.

Original: Proposition 3.1.
Revised: Proposition 3.1.

Original: For any undirected weighted graph G , d T ( · , · ) is a well-defined metric over Y , if all edge weights of G are positive and G is connected.
Revised: For any undirected weighted graph G , d T ( · , · ) is a well-defined metric over Y , if all edge weights of G are positive and G is connected.

Original: Proof.
Revised: Proof.

Original: Given v i , v j ∈ V , if d T ( v i , v j ) is a well-defined metric function, it means d T must satisfy the following three properties:
Revised: Given v i , v j ∈ V , if d T ( v i , v j ) is a well-defined metric function, it means d T must satisfy the following three properties:

Original: if v i ̸ = v j , d T ( v i , v j ) > 0 .
Revised: if v i ̸ = v j , d T ( v i , v j ) > 0 .

Original: Otherwise, d T ( v i , v j ) = 0 .
Revised: Otherwise, d T ( v i , v j ) = 0 .

Original: d T ( v i , v j ) =
Revised: d T ( v i , v j ) =

Original: d T ( v j , v i ) .
Revised: d T ( v j , v i ) .

Original: ∀ v k ∈ V, d T ( v i , v j ) ≤ d T ( v i , v k )
Revised: ∀ v k ∈ V, d T ( v i , v j ) ≤ d T ( v i , v k )

Original: + d T ( v k , v j ) .
Revised: + d T ( v k , v j ) .

Original: By definition d T ( v i , v j ) is the distance of the shortest path from v i to v j .
Revised: By definition d T ( v i , v j ) is the distance of the shortest path from v i to v j .

Original: Since G has positive weights, the first statement holds.
Revised: Since G has positive weights, the first statement holds.

Original: There is always a path p from v i to v j because G is connected.
Revised: There is always a path p from v i to v j because G is connected.

Original: Since G is undirected, there is also a path p ′ from v j to v i by backtracking vertices on p .
Revised: Since G is undirected, there is also a path p ′ from v j to v i by backtracking vertices on p .

Original: The sum of edge weights on p is the same as that on p ′ .
Revised: The sum of edge weights on p is the same as that on p ′ .

Original: p ′ must be the shortest path from v j to v i : if there exists a shorter path q from v j to v i , backtracking q will give a shorter path q ′ from v i to v j .
Revised: p ′ must be the shortest path from v j to v i : if there exists a shorter path q from v j to v i , backtracking q will give a shorter path q ′ from v i to v j .

Original: Thus, d T ( v i , v j ) =
Revised: Thus, d T ( v i , v j ) =

Original: d T ( v j , v i ) .
Revised: d T ( v j , v i ) .

Original: If the shortest path p from v i to v j contains v k , d T ( v i , v j ) =
Revised: If the shortest path p from v i to v j contains v k , d T ( v i , v j ) =

Original: d T ( v i , v k )+ d T ( v k , v j ) .
Revised: d T ( v i , v k ) + d T ( v k , v j ) .

Original: If p does not contain v k , if d T ( v i , v j ) >
Revised: If p does not contain v k , if d T ( v i , v j ) >

Original: d T ( v i , v k ) + d T ( v k , v j ) , v i . . .
Revised: d T ( v i , v k ) + d T ( v k , v j ) , v i . . .

Original: v k . . .
Revised: v k . . .

Original: v j is a valid path that is shorter than p .
Revised: v j is a valid path that is shorter than p .

Original: Contradiction.
Revised: Contradiction.

Original: ■
Revised: ■

Original: Proposition 3.2.
Revised: Proposition 3.2.

Original: Let T be an unweighted tree with a fixed root node.
Revised: Let T be an unweighted tree with a fixed root node.

Original: Then for any pair of nodes v i , v j ∈ V , d T ( v i , v j ) =
Revised: Then for any pair of nodes v i , v j ∈ V , d T ( v i , v j ) =

Original: 2 t ( v i , v j ) − | dt ( v i ) − dt ( v j ) | .
Revised: 2 t ( v i , v j ) − | dt ( v i ) − dt ( v j ) | .

Original: Proof.
Revised: Proof.

Original: Since T is an unweighted tree, there is a unique path connecting v i and v j .
Revised: Since T is an unweighted tree, there is a unique path connecting v i and v j .

Original: Let the path p be v i , v 1 , . .
Revised: Let the path p be v i , v 1 , . .

Original: . , v L − 1 , v j of length L .
Revised: . , v L − 1 , v j of length L .

Original: Let v l :=
Revised: Let v l :=

Original: LCA ( v i , v j ) .
Revised: LCA ( v i , v j ) .

Original: By definition of LCA , there exists a path from v l to v
Revised: By definition of LCA , there exists a path from v l to v

Original: i and a path from v l to v j
Revised: i and a path from v l to v j

Original: as well.
Revised: as well.

Original: Because the path p between v i and v j is unique, v l must be on the path p as well.
Revised: Because the path p between v i and v j is unique, v l must be on the path p as well.

Original: Now, since v l is the least common ancestor, the depth of v i , v j can be written as:
Revised: Now, since v l is the least common ancestor, the depth of v i , v j can be written as:

Original: [Equation]
Revised: [Equation]

Original: Hence, we have
Revised: Hence, we have

Original: [Equation]
Revised: [Equation]

Original: where for the second to last equality we use the fact that max { a, b } − | a − b | = min { a, b } for any a, b ∈ R .
Revised: where for the second to last equality we use the fact that max { a, b } − | a − b | = min { a, b } for any a, b ∈ R .

Original: ■
Revised: ■

Original: B A DDITIONAL F IGURES
Revised: B D ISCUSSION ON C ONVERGENCE OF THE O BJECTIVE F UNCTION

Revised: Convergence of SGD Training with CPCC One question to ask about the CPCC regularizer is that, when coupled with the usual cross-entropy loss, can we guarantee that SGD training over the objective function in Eq. 2 will converge? Note that in general, due to the non-convexity of the loss function with deep neural networks, one cannot hope to obtain convergence to the global optimal (if exists).

Revised: Absent strict assumptions on the saddle points of Eq.

Revised: 2, e.g., the strict saddle point property (Lee et al., 2016) that can be hard to verify in practice, in general, it is still unclear that gradient descent will always converge to a local minimum (Murty & Kabadi, 1985).

Revised: So instead, a more realistic convergence criterion that we can hope for is to show that when optimizing the objective function in Eq. 2, SGD will converge to a stationary point.

Revised: To this end, under certain regularity conditions on the data, we will show that SGD will converge to a stationary point of Eq.with probability 1, by invoking a key theorem in recent advances of nonconvex optimization with

Revised: SGD (Patel & Zhang, 2021, Theorem 2).

Revised: Proposition B.1 (informal) .

Revised: For a fixed dataset D = { ( x i , y i ) } ni =1 , assume the objective function

Revised: Eq. 2 is bounded from below for every θ , and that the data is bounded, i.e., ∃ R > 0 such that ∀ i ∈ [ n ] , ∥ x i ∥ 2 ≤ R , then under the Robbins-Monro condition of learning rate scheduling, SGD will converge to a stationary point of Eq.

Revised: 2 with probability 1.

Revised: The proof of the above proposition mainly follows by verifying that the Assumptions 1-4 1 (Patel & Zhang, 2021) indeed hold in our case under the regularity condition of the data D , so we omit it here.

Revised: Essentially, in order for the regularity condition to hold, since the cross-entropy function is always bounded from below, we only need to ensure that the CPCC regularizer is bounded from below along the trajectory of θ .

Revised: Note that since CPCC is a correlation score, it is bounded between [ − 1 , 1] whenever it is well-defined, i.e., the denominator is non-zero, which is the case as long as the distance

Revised: ρ Z ( · , · ) is not a constant over D .

Revised: The latter is true whenever D contains data points from at least two different classes and the feature map given by θ is not degenerate, i.e., at least two class mean vectors are different – a mild condition to hold in practice.

Revised: C F ORMULATION OF O BJECTIVES

Revised: • Flat : Train using ℓ CE on fine classes only, without leveraging any hierarchical information:

Revised: [Equation]

Revised: • Multi-task Learning : We jointly train a two-headed network to treat fine and coarse as two separate tasks.

Revised: The network then becomes h : X → ∆ k 1 × ∆ k 2 with a shared feature encoder f θ : X → Z and two classifiers g coarse : Z → ∆ k 1 , g fine : Z → ∆ k 2 .

Revised: k 1 is the number of coarse classes and k 2 the number of fine classes.

Revised: The loss function is the sum of the cross-entropies on the fine and coarse classification tasks, and we simply set the weight of the two parts to 1 :

Revised: [Equation]

Revised: • Curriculum Learning :

Revised: In the spirit of curriculum learning (Bengio et al., 2009), we first train on the coarse classes using Eq. 3 and use y coarse instead of y fine .

Revised: In the second step, we remove the linear classifier and fine tune a new one on the fine level labels with Equation 3 as the loss function.

Revised: • Sum Loss : We define a hierarchical Sum Loss as

Revised: [Equation]

Revised: W is a k 1 by k 2 matrix representing the relationships in the label tree: if a fine class i belongs to a coarse class j , then W ji is 1 , otherwise the entry is set to 0 .

Revised: In other words, the probability of belonging to a given coarse class is simply the sum of the probabilities of its descendants ( i.e. , fine classes) in the tree.

Revised: This way, we use the information from both levels with one head only.

Revised: • Hierarchical Cross Entropy (Bertinetto et al., 2020) Let h ( · ) be the height of any label y level in the hierarchy, λ ( y level ) =

Revised: exp( − αh ( y level )) .

Revised: In a two-level hierarchy, the HXE loss function is L HXE ( x, y fine , y coarse ) = − (cid:88)

Revised: [Equation]

Revised: We set α as 0 . 4 .

Revised: When α = 0 , HXE is reduced to Eq.

Revised: As α grows up, we emphasize more on the latter term to optimize coarse level prediction.

Revised: • Soft Label (Bertinetto et al., 2020) replaces the one-hot encoding of y fine in Eq.

Revised: 3’s ℓ CE by y softfine .

Revised: To leverage the hierarchy, denote d ( i, j ) as the height of LCA( i, j ) over the height of the tree.

Revised: Let i ∈

Revised: [ k ] , the one-hot vector of y fine becomes a categorically distributed vector where each component is:

Revised: [Equation]

Revised: We only use the two level coarse-fine hierarchy to calculate d ( i, j ) and set β = 10 .

Revised: When β → ∞ , y softfine is the same as one-hot y fine .

Revised: • Quadruplet Loss (Zhang

Revised: et

Revised: al., 2016) is a multi-task objective L

Revised: = λ

Revised: s L Flat

Revised: + (1 − λ s )

Revised: L Quad

Revised: where L Quad is:

Revised: [Equation]

Revised: In Eq.

Revised: 8, N is the number of valid quadruplets in the mini-batch, D ( · , · ) is the squared Euclidean distance of l 2 normalized features.

Revised: For each incoming mini-batch, we set all images as anchor r i , p

Revised: + i has the same fine label with r i , p − i has the same coarse bot not fine label with r i , and n i and r i have different coarse labels.

Revised: We set λ s = 0 .

Revised: 8 , m 1 = 0 .

Revised: 25 , m 2 = 0 .

Revised: 15 in our experiments.

Revised: Margin m 1 is set to be bigger than m 2 to ensure same fine labels are grouped together, then fine labels within one coarse class are close, and finally classes not in the same coarse class are far away, i.e.,

Revised: D ( r i , p + i ) + m 1 < D ( r i , p − i ) + m 2 < D ( r i , n i ) .

Revised: Once we found D ( r i , p + i ) , we pick the hardest

Revised: D ( r i , p − i ) and then D ( r i , n i ) to accelerate convergence.

Revised: D A DDITIONAL F IGURES

Original: We include t -SNE visualization of representations and distance matrices for the remaining objectives (Multi-task, Curriculum, Sum Loss) of Table 1 in this section.
Revised: We include t -SNE visualization of representations and distance matrices for the remaining objectives (Multi-task, Curriculum, Sum Loss) of Table 1 in this section (Figures 5, 6 and 7 for the t -SNE, and Figures 8, 9 and 10 for the matrices).

Revised: We also include the distance matrix for using CPCC on three layers in Figure 4, showing that CPCC can generalize to deeper hierarchies without sacrificing too much classification accuracy.

Original: C A DDITIONAL E XPERIMENTS
Revised: E A DDITIONAL E XPERIMENTS

Revised: In this section, we include remaining experiments of Table 1 to Table 4.

Original: In this section, we discuss two extra settings where CPCC can be applied.
Revised: Furthermore, we discuss an extra setting where CPCC can be applied.

Original:  Santurkar et al.
Revised: CPCC for sub-population shift Santurkar et al.

Original: (2020) proposed BREEDS benchmark for sub-population shift.
Revised: (2020) proposed BREEDS benchmark for subpopulation shift.

Original: Compared to 4.3, the main difference between two tasks is that BREEDS was only trained on coarse level, while 4.3 is trained on fine level for all objectives without CPCC.
Revised: Compared to 4.3, the main difference between two tasks is that BREEDS was only trained on coarse level, while 4.3 is trained on fine level for all objectives without CPCC.

Original: From the definition of sub-population shift, if we train on the train split of source dataset, there will be a big drop of performance of test performance on source set and target set.
Revised: From the definition of sub-population shift, if we train on the train split of source dataset, there will be a big drop of performance of test performance on source set and target set.

Original: Since models are always trained on coarse labels, fine tuning is unnecessary.
Revised: Since models are always trained on coarse labels , fine tuning is unnecessary.

Original: In Table 2, we show the performance gain of using CPCC on the ERM method.
Revised: In Table 3, we show the performance gain of using CPCC on the Flat method.

Original: Since BREEDS contains more than two levels of hierarchy, we present the result of using CPCC on coarse-fine classes, coarse-coarser classes as examples.
Revised: Since BREEDS contains more than two levels of hierarchy, we present the result of using CPCC on coarse-fine classes, coarse-coarser classes as examples.

Original: The result shows CPCC alleviates the sub-population shift problem by using multiple labels in hierarchy, as shown in Mukherjee et al.
Revised: The result shows CPCC alleviates the sub-population shift problem by using multiple labels in hierarchy, as shown in Mukherjee et al.

Original: (2022) who used other methods.
Revised: (2022) who used other methods.

Original: [Table]
Revised: [Table]

Original: CPCC

Original: [Table]

Original: [Table]

Original: [Table]

Original: C. G ENERALIZATION TO NEW CLASSES IN THE HIERARCHY

Original: Assume kids have learned cat and dog to be animals.

Original: Does knowing the animal concept help them understand giraffe, horse, tiger, and other animals better?

Original: Since BREEDS divides coarse label into source and target sets, we are able to straightforwardly investigate this question.

Original: Instead of evaluating on coarse classes, we take the representations in 4.3 which is learned on the source train test, fine tune a linear classifier on the target train test, and finally evaluate on target validation set.

Original: Thus, we can test the generalizability to new classes in the same level of the hierarchy.

Original: Similar to the other generalization task, we also use the one-shot setting during fine tuning.

Original: Compared to generalizing to new levels, even if we assume the hierarchy is well-defined, it is a much harder task as we must learn on new classes whose decision boundary might be hard to infer from seen labels, let alone when the number of labels is limited.

Original: Thus, we can see a very low test accuracy.

Original: Yet, using CPCC is still beneficial: when we have a limited train set, leveraging hierarchical information can be a good hint for classification.

Original: Objective

Original: LIVING

Original: ENTITY

Original: ENTITY

Original: [Table]

Original: However, from some preliminary trials, this minor advantage disappears as we use the full target data during training.

Original: Additionally, the result is sensitive to the definition of hierarchy: if we perform this task on CIFAR100 by creating an artificial source target split, using CPCC reduces performance, which might be due to the fact that not all coarse classes in CIFAR100 are defined based on visual characteristics (such as vehicles 1 and vehicles 2).

Original: D I MPLEMENTATION D ETAILS
Revised: F I MPLEMENTATION D ETAILS

Original: In this section, we describe the training details for our experiments.
Revised: In this section, we describe the training details for our experiments.

Original: We want to highlight again that we use these parameters to show CPCC’s advantage over other methods in controlled settings, but not to reproduce SOTA results.
Revised: We want to highlight again that we use these parameters to show CPCC’s advantage over other methods in controlled settings, but not to reproduce SOTA results.

Original: All experiments were trained on NVIDIA RTX A6000.
Revised: All experiments were trained on NVIDIA RTX A6000.

Original: Experiments in Sections 4.3 and 4.4 have the same training setting, except that the latter’s learning rate starts from 1/10 of the first task’s initial learning rate for fine tuning tasks.
Revised: Experiments in Sections 4.3 and 4.4 have the same training setting, except for some optimizer setting differences. For a fair comparison, we searched hyperparameters for 4.4 to achieve the best performance for all Flat methods, and apply the same setting to other objectives with or without CPCC.

Original: Compared with other approaches, curriculum learning in 4.3 contains two stages.
Revised: Compared with other approaches, curriculum learning in 4.3 contains two stages.

Original: Empirically, we find that a few first stage training iterations on coarse labels give a better initialization of the second stage on fine labels.
Revised: Empirically, we find that a few first stage training iterations on coarse labels give a better initialization of the second stage on fine labels.

Original: The epochs of the first stage do not have a huge impact on the final performance, but less epochs on the second stage can harm the result.
Revised: The epochs of the first stage do not have a huge impact on the final performance, but less epochs on the second stage can harm the result.

Original: Therefore, we train more epochs on curriculum learning settings so that the fine accuracy for all without CPCC objectives are similar.
Revised: Therefore, we train more epochs on curriculum learning settings so that the fine accuracy for all without CPCC objectives are similar.

Original: We spend approximately the same number of epochs on fine labels for all experiments, and add some extra iterations for curriculum learning to train on coarse labels.
Revised: We spend approximately the same number of epochs on fine labels for all experiments, and add some extra iterations for curriculum learning to train on coarse labels.

Original: Training epochs are the same for generalization experiments, given the fact that all representations have similar fine and coarse accuracy.
Revised: Training epochs are the same for generalization experiments, given the fact that all representations have similar fine and coarse accuracy.

Original: MNIST The feature encoder is composed of these layers sequentially: a 5 × 5 convolutional layer, 2 × 2 max pooling layer, a 5 × 5 convolutional layer, 2 × 2 max pooling layer, fully connected layer with output 120 units, and a fully connected layer with output 50 units.
Revised: MNIST The feature encoder is composed of these layers sequentially: a 5 × 5 convolutional layer, 2 × 2 max pooling layer, a 5 × 5 convolutional layer, 2 × 2 max pooling layer, fully connected layer with output 120 units, and a fully connected layer with output 50 units.

Original: We set batch size as 256, epochs as 70, learning rate start from 0.05, divided by 5 every 25 steps, with momentum of 0.9, weight decay 5 × 10 − 4 of SGD optimizer.
Revised: We set batch size as 256, epochs as 70, learning rate start from 0.05, divided by 5 every 25 steps, with momentum of 0.9, weight decay 5 × 10 − 4 of SGD optimizer.

Original: No data augmentation is applied.
Revised: No data augmentation is applied.

Original: For curriculum learning, we train 18 epochs on coarse labels and 72 epochs on fine labels.
Revised: For curriculum learning, we train 18 epochs on coarse labels and 72 epochs on fine labels.

Revised: We train one-shot transfer on mid level for 70 epochs, learning rate from 0.005 divided by 10 every 25 steps; fine level 70 epochs learning rate from 0.1 divided by 10 every 25 steps.

Original: CIFAR100 We train ResNet18 from scratch.
Revised: CIFAR100 We train ResNet18 from scratch.

Original: Following the original ResNet (He et al., 2015) paper, the first convolution layer has a 3 × 3 kernel size, and the max pooling layer is not included.
Revised: Following the original ResNet (He et al., 2015) paper, the first convolution layer has a 3 × 3 kernel size, and the max pooling layer is not included.

Original: We set batch size as 128, epochs as 200, learning rate starts from 0.1, divided by 5 every 60 steps, with momentum of 0.9, weight decay 5 × 10 − 4 of SGD optimizer.
Revised: We set batch size as 128, epochs as 200, learning rate starts from 0.1, divided by 5 every 60 steps, with momentum of 0.9, weight decay 5 × 10 − 4 of SGD optimizer.

Original: All images are normalized by
Revised: All images are normalized by

Original: CIFAR100’s mean and standard deviation.
Revised: CIFAR100’s mean and standard deviation.

Original: The following data augmentations are applied on train set sequentially: images are zero padded by 4 pixels and cropped into original size; images are flipped horizontally by the chance of 50 % ; images are randomly rotated by − 15 to 15 degrees.
Revised: The following data augmentations are applied on train set sequentially: images are zero padded by 4 pixels and cropped into original size; images are flipped horizontally by the chance of 50 % ; images are randomly rotated by − 15 to 15 degrees.

Original: For curriculum learning, we train 50 epochs on coarse labels and 200 epochs on fine labels.
Revised: For curriculum learning, we train 50 epochs on coarse labels and 200 epochs on fine labels.

Revised: We train one-shot transfer on mid and fine level for 100 epochs, learning rate from 0.05 divided byevery 60 steps.

Original: BREEDS We train ResNet18 from scratch.
Revised: BREEDS We train ResNet18 from scratch.

Original: The first convolution layer has a 7 × 7 kernel size.
Revised: The first convolution layer has a 7 × 7 kernel size.

Original: We follow BREEDS training setting: we set batch size as 128, initial learning rate as 0.1, weight decay 10 − 4 of SGD optimizer.
Revised: We follow BREEDS training setting: we set batch size as 128, initial learning rate as 0.1, weight decay 10 − 4 of SGD optimizer.

Original: On ENTITY13 and ENTITY30 , we train 300 epochs, learning rate divided by 10 every 100 steps; on LIVING17 and NONLIVING26 , we train 450 epochs, learning rate divided by 10 every 150 steps.
Revised: On ENTITY13 and ENTITY30 , we train 300 epochs, learning rate divided by 10 every 100 steps; on LIVING17 and NONLIVING26 , we train 450 epochs, learning rate divided by 10 every 150 steps.

Original: For curriculum learning, on ENTITY13 and ENTITY30 , we train 75 epochs on coarse labels and 300 epochs on fine labels; on LIVING17 and
Revised: For curriculum learning, on ENTITY13 and ENTITY30 , we train 75 epochs on coarse labels and 300 epochs on fine labels; on LIVING17 and

Original: NONLIVING26 , we train 108 epochs on coarse labels and 432 epochs on fine labels.
Revised: NONLIVING26 , we train 108 epochs on coarse labels and 432 epochs on fine labels.

Original: Train set is augmented by a RandomResizedCrop into 224 × 224 , flipped horizontally by the chance of% , and randomly change the brightness, contrast, saturation by 0.9 to 1.1 (ColorJitter).
Revised: Train set is augmented by a RandomResizedCrop into 224 × 224 , flipped horizontally by the chance of% , and randomly change the brightness, contrast, saturation by 0.9 to 1.1 (ColorJitter).

Original: Test set is first resized into 256 × 256 and CenterCropped into 224 × 224 . 
Revised: Test set is first resized into 256 × 256 and CenterCropped into 224 × 224 . train , we train

Original: E D ATASET H IERARCHIES
Revised: G D ATASET H IERARCHIES

Original: We visualize 4 levels of hierarchy (coarser, coarse, mid, fine) of MNIST, CIFAR, and BREEDS  in Fig 10, and provide more details for Section 4.1 and 4.4 about how we constructed the hierarchy based on original labels.
Revised: We visualize 4 levels of hierarchy (coarser, coarse, mid, fine) of MNIST, CIFAR, and BREEDS s in Fig 11, and provide more details for Section 4.1 and 4.4 about how we constructed the hierarchy based on original labels.

Original: We can notice in BREEDS hierarchies, there are some repetitive labels 1 .
Revised: We can notice in BREEDS hierarchies, there are some repetitive labels 2 .

Original: One case is that the parent node is a dummy node which shares the same name as its child.
Revised: One case is that the parent node is a dummy node which shares the same name as its child.

Original: For example, dummy 52 and n02773037 (WordNet ID) are both called “bags”.
Revised: For example, dummy 52 and n02773037 (WordNet ID) are both called “bags”.

Original: Santurkar et al.
Revised: Santurkar et al.

Original: (2020) inserted dummy nodes into the ImageNet hierarchy to adjust the granularity of each level.
Revised: (2020) inserted dummy nodes into the ImageNet hierarchy to adjust the granularity of each level.

Original: Since we create mid labels by backtracking the fine classes by 1 level, the other case of repetitive labels happens when a coarse label coincides with the mid label ( i.e. , if the coarse label is exactly one level higher than the fine label).
Revised: Since we create mid labels by backtracking the fine classes by 1 level, the other case of repetitive labels happens when a coarse label coincides with the mid label ( i.e. , if the coarse label is exactly one level higher than the fine label).

Original: Unlike on MNIST and CIFAR, we do not want to create some artificial new classes on BREEDS.
Revised: Unlike on MNIST and CIFAR, we do not want to create some artificial new classes on BREEDS.

Original: Therefore, we can also treat repetitive nodes as dummy nodes as in the first case, which only affects mid class results.
Revised: Therefore, we can also treat repetitive nodes as dummy nodes as in the first case, which only affects mid class results.

Original: [Figure] [Table] 
Revised: [Table] [Figure]

Revised: eveneven

Original: [Table]
Revised: [Figure]

Revised: ≤

Original: [Table]
Revised: [Figure]

Revised: even and >

Original:  [Table] [Table]
Revised: [Figure] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: boatfireboatyawlspeedboatsailboat
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: warship
Revised: warship

Original: [Table]
Revised: [Table]

Original: war
Revised: war

Original: [Table]
Revised: [Table]

Original: aircraftcarrier furnishing furniture, piece of furniture
Revised: aircraft

Original: [Table]
Revised: [Table]

Original:  article of furniture baby  bed 
Revised: , attack aircraft carrier furnishing furniture, piece of furniture, article of furniture baby bed, baby’s bed bassinet bedfour-poster cabinetfile,filecabinet, filing cabinet screenshojifirescreen, fireguard mosquito net seattoilet seat chairfolding chair chest of drawers, chest, bureau, dresser chiffonier, commode wall unit wardrobe, closet, press

Original: [Table]
Revised: [Table]

Original: baby’s bed bassinet bedfour-poster cabinetfile,filecabinet, filing cabinet screenshojifirescreen, fireguard mosquito net seattoilet seat chairfolding chair chest of drawers, chest, bureau, dresser chiffonier, commode wall unit wardrobe, closet, press
Revised: h [Table] hierarchy

Original: [Table]
Revised: [Table]

Original: part 
Revised: part 4)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: garmentgarmentgarmentaproncoatacademic gown, academic robe, judge’s robe fur coat skirtminiskirt, mini birdpasserine, passeriform bird passerine, passeriform bird brambling, Fringilla montifringilla goldfinch, Carduelis carduelis water ouzel, dipper titmouse, tit chickadee aquatic bird aquatic bird albatross, mollymawk crane stork white stork, Ciconia ciconia sandpiperred
Revised: garmentgarmentgarmentaproncoatacademic gown, academic robe, judge’s robe fur coat skirtminiskirt, mini birdpasserine, passeriform bird passerine, passeriform bird brambling, Fringilla montifringilla goldfinch, Carduelis carduelis water ouzel, dipper titmouse, tit chickadee aquatic bird aquatic bird albatross, mollymawk crane stork white stork, Ciconia ciconia sandpiperred

Original: [Table]
Revised: [Table]

Original: alpinareptile
Revised: alpina

Original: [Table]
Revised: [Table]

Original: reptilian
Revised: reptilian

Original: [Table]
Revised: [Table]

Original: (i) ENTITY30 hierarchy (part 1).
Revised: (i) ENTITY30 hierarchy (part 1).

Original: [Table]
Revised: [Table]

Original: cliphair slide capshower
Revised: cliphair slide capshower

Original: [Table]
Revised: [Table]

Original: helmetpickelhaube neckwearbraceneck brace scarffeather
Revised: helmetpickelhaube neckwearbraceneck brace scarffeather

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: neckwear bib footwear, legwear footwear, legwear clog, geta, patten, sabot tights, leotards maillot shoeLoaferrunning shoe (j) ENTITY30 hierarchy (part 2).
Revised: bib footwear, legwear footwear, legwear clog, geta, patten, sabot tights, [Table] maillot shoeLoaferrunning shoe (j) ENTITY30 hierarchy (part 2).

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: potpotteapotDutchoven
Revised: potpotteapotDutchoven

Original: [Table] [Table]
Revised: [Table] [Table]