Original: C ONTRASTIVE H IERARCHICAL C LUSTERING
Revised: C ONTRASTIVE H IERARCHICAL C LUSTERING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep clustering has been dominated by flat clustering models, which split a dataset into a predefined number of groups.
Revised: Deep clustering has been dominated by flat clustering models, which split a dataset into a predefined number of groups.

Original: Although recent methods achieve extremely high similarity with the ground truth on popular benchmarks, the information contained in the flat partition is limited.
Revised: Although recent methods achieve extremely high similarity with the ground truth on popular benchmarks, the information contained in the flat partition is limited.

Original: In this paper, we introduce CoHiClust, a Contrastive Hierarchical Clustering model based on deep neural networks, which can be applied to large-scale image data.
Revised: In this paper, we introduce CoHiClust, a Contrastive Hierarchical Clustering model based on deep neural networks, which can be applied to large-scale image data.

Original: By employing a self-supervised learning approach, CoHiClust distills the base network into a binary tree without access to any labeled data.
Revised: By employing a self-supervised learning approach, CoHiClust distills the base network into a binary tree without access to any labeled data.

Original: The hierarchical clustering structure can be used to analyze the relationship between clusters as well as to measure the similarity between data points.
Revised: The hierarchical clustering structure can be used to analyze the relationship between clusters as well as to measure the similarity between data points.

Original: Experiments performed on typical image benchmarks demonstrate that CoHiClust generates a reasonable structure of clusters, which is consistent with our intuition and image semantics.
Revised: Experiments performed on typical image benchmarks demonstrate that CoHiClust generates a reasonable structure of clusters, which is consistent with our intuition and image semantics.

Original: Moreover, by applying the proposed pruning strategy, we can restrict the hierarchy to the requested number of clusters (leaf nodes) and obtain the clustering accuracy comparable to the state-of-the-art flat clustering baselines.
Revised: Moreover, by applying the proposed pruning strategy, we can restrict the hierarchy to the requested number of clusters (leaf nodes) and obtain the clustering accuracy outperforming existing hierarchical baselines.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Clustering, a fundamental branch of unsupervised learning, is often one of the first steps in data analysis, which finds applications in anomaly detection Barai & Dey (2017), personalized recommendations Zhang et al.
Revised: Clustering, a fundamental branch of unsupervised learning, is often one of the first steps in data analysis, which finds applications in anomaly detection Barai & Dey (2017), personalized recommendations Zhang et al.

Original: or bioinformatics Lakhani et al.
Revised: or bioinformatics Lakhani et al.

Original: Since it does not use any information about class labels, representation learning becomes an integral part of deep clustering methods.
Revised: Since it does not use any information about class labels, representation learning becomes an integral part of deep clustering methods.

Original: Initial approaches use representations taken from pre-trained models
Revised: Initial approaches use representations taken from pre-trained models

Original: Gu ´ erin et al.
Revised: Gu ´ erin et al.

Original: Naumov et al.
Revised: Naumov et al.

Original: or employ auto-encoders in joint training of the representation and the clustering model Guo et al.
Revised: or employ auto-encoders in joint training of the representation and the clustering model Guo et al.

Original: (2017a)
Revised: (2017a)

Original: ; Mautz et al.
Revised: ; Mautz et al.

Original: More recent works designed to image data frequently follow the self-supervised learning principle, where representation is trained on pairs of similar images automatically generated by data augmentations Li et al.
Revised: More recent works designed to image data frequently follow the self-supervised learning principle, where representation is trained on pairs of similar images automatically generated by data augmentations Li et al.

Original: (2021b)
Revised: (2021b)

Original: ; Dang et al.
Revised: ; Dang et al.

Original: Since augmentations used for image data are class invariant, the latter techniques often obtain very high similarity with the ground truth classes.
Revised: Since augmentations used for image data are class invariant, the latter techniques often obtain very high similarity with the ground truth classes.

Original: However, we should be careful when comparing clustering techniques only by inspecting their accuracy with ground truth classes because the primary goal of clustering is to deliver information about data and not to perform classification.
Revised: However, we should be careful when comparing clustering techniques only by inspecting their accuracy with ground truth classes because the primary goal of clustering is to deliver information about data and not to perform classification.

Original: Most works in the area of deep clustering focus on producing flat partitions with a predefined number of groups.
Revised: Most works in the area of deep clustering focus on producing flat partitions with a predefined number of groups.

Original: Although hierarchical clustering gains notable attention in classical machine learning and has been frequently applied in real-life problems Zou et al.
Revised: Although hierarchical clustering gained notable attention in classical machine learning and has been frequently applied in real-life problems Zou et al.

Original: (2020); ´
Revised: (2020); ´

Original: Smieja et al.
Revised: Smieja et al.

Original: (2014), its role has been drastically marginalized in the era of deep learning.
Revised: (2014), its role has been drastically marginalized in the era of deep learning.

Original: In the case of hierarchical clustering, the exact number of clusters does not have to be specified because we can inspect the partition at various tree levels.
Revised: In the case of hierarchical clustering, the exact number of clusters does not have to be specified because we can inspect the partition at various tree levels.

Original: Moreover, we can analyze the clusters’ relationships, e.g. by finding super-clusters or measuring the distance between groups in the hierarchy.
Revised: Moreover, we can analyze the clusters’ relationships, e.g. by finding superclusters or measuring the distance between groups in the hierarchy.

Original: The above advantages make hierarchical clustering an excellent tool for analyzing complex data.
Revised: The above advantages make hierarchical clustering an excellent tool for analyzing complex data.

Revised: However, in order to take full advantage of hierarchical clustering, it is necessary to create an appropriate image representation, which is possible thanks to the use of deep neural networks.

Original: To the best of our knowledge, DeepECT
Revised: To the best of our knowledge, DeepECT

Original: Mautz et al.
Revised: Mautz et al.

Original: (2019; 2020) is the only hierarchical clustering model trained jointly with the neural network, which however has not been examined to large-scale image datasets.
Revised: (2019; 2020) is the only hierarchical clustering model trained jointly with the neural network. Nevertheless, this method has not been examined to large image datasets, which appear in practical applications.

Original: To fill this gap, we introduce CoHiClust ( Co ntrastive Hi erarchical Clust ering), which creates a hierarchy of clusters and can be applied to large-scale data.
Revised: To fill this gap, we introduce CoHiClust ( Co ntrastive Hi erarchical Clust ering), which creates a hierarchy of clusters and can be applied to large image data.

Original: CoHiClust uses a neural network to generate a high-level representation of data, which is next distilled into the tree hierarchy by applying the projection head, see Figure 2.
Revised: CoHiClust uses a neural network to generate a high-level representation of data, which is next distilled into the tree hierarchy by applying the projection head, see Figure 2.

Original: The whole framework is trained jointly in an end-to-end manner without labels using our novel contrastive loss and automatically generated data augmentations following the self-supervised learning principle.
Revised: The whole framework is trained jointly in an end- to-end manner without labels using our novel contrastive loss and automatically generated data augmentations following the self-supervised learning principle.

Original: The constructed hierarchy uses the structure of a binary tree, where the sequence of decisions made by internal nodes determines the final assignment to clusters (leaf nodes).
Revised: The constructed hierarchy uses the structure of a binary tree, where the sequence of decisions made by internal nodes determines the final assignment to clusters (leaf nodes).

Original: In consequence, similar examples are processed longer by the same path than dissimilar ones.
Revised: In consequence, similar examples are processed longer by the same path than dissimilar ones.

Original: By inspecting the number of edges needed to connect two clusters (leaves), we obtain a natural similarity measure between data points.
Revised: By inspecting the number of edges needed to connect two clusters (leaves), we obtain a natural similarity measure between data points.

Original: Although CoHiClust assumes a pre-defined tree structure with a fixed height, we introduce a pruning mechanism, which removes the least informative leaf nodes until the requested number of leaves is obtained.
Revised: Although CoHiClust assumes a pre-defined tree structure with a fixed height, we introduce a pruning mechanism, which removes the least informative leaf nodes until the requested number of leaves is obtained.

Original: In contrast to typical pruning strategies or tree cuts, where neighboring leaves are only merged to ancestor node, we allow for reassigning data points between clusters by finetuning the whole model, which further improves topology of the tree.
Revised: In contrast to typical pruning strategies or tree cuts, where neighboring leaves are only merged to ancestor node, we allow for reassigning data points between clusters by finetuning the whole model, which further improves topology of the tree.

Original: The proposed model has been examined on various image datasets and compared with both hierarchical and flat clustering baselines.
Revised: The proposed model has been examined on various image datasets and compared with both hierarchical and flat clustering baselines.

Original: By analyzing the constructed hierarchies, we show that CoHiClust generates a structure of clusters that is consistent with our intuition and image semantics, see Figures 1.
Revised: By analyzing the constructed hierarchies, we show that CoHiClust generates a structure of clusters that is consistent with our intuition and image semantics, see Figures 1 for the illustration and discussion.

Original: Our analysis is supported by a quantitative study, which shows that CoHiClust gives higher similarity with ground truth partition than available hierarchical baselines, see Tables 1 and 2.
Revised: Our analysis is supported by a quantitative study, which shows that CoHiClust gives higher similarity with ground truth partition than available hierarchical baselines, see Tables 1, 2 and 3.

Original: Moreover, it is among the three best-performing methods when compared to the flat clustering models, see Table 3.
Revised: Moreover, it is among the three best-performing methods when compared to the flat clustering models, see Table 3.

Original: Our main contributions are summarized as follows:
Revised: Our main contributions are summarized as follows:

Original: • We introduce a hierarchical clustering model CoHiClust, which converts the base neural network into a binary tree.
Revised: • We introduce a hierarchical clustering model CoHiClust, which converts the base neural network into a binary tree.

Original: The model is trained effectively with no supervision using our novel hierarchical contrastive loss applied to self-generated data augmentations.
Revised: The model is trained effectively with no supervision using our novel hierarchical contrastive loss applied to self-generated data augmentations.

Original: • We implement a pruning strategy, which not only leads to creating a fixed number of leaves but also improves the constructed hierarchy.
Revised: • We implement a pruning strategy, which not only leads to creating a fixed number of leaves but also improves the constructed hierarchy.

Original: • Our experimental analysis shows that CoHiClust builds hierarchies based on the welldefined and intuitive patterns retrieved from data.
Revised: • Our experimental analysis shows that CoHiClust builds hierarchies based on the welldefined and intuitive patterns retrieved from data.

Original: • Since CoHiClust is the first deep hierarchical clustering model applied to large-scale image datasets
Revised: • Since CoHiClust is the first deep hierarchical clustering model applied to large-scale image datasets

Original: , we also deliver a new benchmark, which can be used to compare hierarchical clustering methods.
Revised: , we also deliver a new benchmark, which can be used to compare hierarchical clustering methods.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: In this section, we briefly introduce some recent developments in three related topics, i.e., contrastive learning, deep clustering, and hierarchical methods.
Revised: In this section, we briefly introduce some recent developments in three related topics, i.e., contrastive learning, deep clustering, and hierarchical methods.

Original: Contrastive Learning
Revised: Contrastive Learning

Original: The basic idea of contrastive learning is to learn such a feature space, in which similar pairs stay close to each other while dissimilar ones are far apart Chopra et al.
Revised: The basic idea of contrastive learning is to learn such a feature space, in which similar pairs stay close to each other while dissimilar ones are far apart Chopra et al.

Original: In early works, training was performed in a supervised way on pairs or triplets labeled manually

Original: Hadsell et al.

Original: ; Schroff et al.

Original: It waslater observed that in selected domains, such as computer vision, positive  pairs can be generated automatically using adversarial perturbations Miyato et al.
Revised: In recent works, it was observed that in selected domains, such as computer vision, positive (similar) pairs can be generated automatically using adversarial perturbations Miyato et al.

Original: (2018) or data augmentation He et al.
Revised: (2018) or data augmentation He et al.

Original: (2020), giving the rise of a new field called self-supervised learning Chen et al.
Revised: (2020), giving the rise of a new field called self-supervised learning Chen et al.

Original: Fine-tuning a simple classifierusing a small amount of labeled data onthe representation created in a self-supervised way allows for obtaining the accuracy comparable to a fully supervised setting with a large labeled dataset.
Revised: Fine-tuning a simple classifier on self-supervised representation allows for obtaining the accuracy comparable to a fully supervised setting.

Original: SimCLR He
Revised: SimCLR He

Original: et al.
Revised: et al.

Original: (2020) applies NT-Xent loss to maximize the agreement between differently augmented views of the same sample.
Revised: (2020) applies NT-Xent loss to maximize the agreement between differently augmented views of the same sample.

Original: Barlow Twins Zbontar et al.
Revised: Barlow Twins Zbontar et al.

Original: (2021) learns to make the cross-correlation matrix between two distorted versions of the same samples close to the identity.
Revised: (2021) learns to make the cross-correlation matrix between two distorted versions of the same samples close to the identity.

Original: BYOL Grill et al.
Revised: BYOL Grill et al.

Original: (2020) claims to achieve new state-of-the-art results without using negative samples.
Revised: (2020) claims to achieve new state-of-theart results without using negative samples.

Original: Other works use memory banks to reduce the cost of computing the embeddings of negative samples in every batch Wu et al.
Revised: Other works use memory banks to reduce the cost of computing the embeddings of negative samples in every batch Wu et al.

Original:  He et al.
Revised: ; He et al.

Original: Deep clustering A primary focus in deep embedded clustering has merely been on flat clustering objectives with the actual number of clusters known a priori.
Revised: Deep clustering A primary focus in deep embedded clustering has merely been on flat clustering objectives with the actual number of clusters known a priori.

Original: DEC Xie et al.
Revised: DEC Xie et al.

Original: (2016) is one of the first works, which combines the auto-encoder loss with a clustering objective to jointly learn the representation and perform clustering.
Revised: (2016) is one of the first works, which combines the auto-encoder loss with a clustering objective to jointly learn the representation and perform clustering.

Original: This idea was further explored with some improvements in
Revised: This idea was further explored with some improvements in

Original: IDEC Guo et al.
Revised: IDEC Guo et al.

Original: (2017a), JULE Yang et al.
Revised: (2017a), JULE Yang et al.

Original: (2016) and DCEC
Revised: (2016) and DCEC

Original: Guo et al.
Revised: Guo et al.

Original: (2017b).
Revised: (2017b).

Original: IMSAT Hu
Revised: IMSAT Hu

Original: et al.
Revised: et al.

Original: (2017) and IIC Ji et al.
Revised: (2017) and IIC Ji et al.

Original: (2019) use perturbations to generate pairs of similar examples and apply information-theoretic objectives for training.
Revised: (2019) use perturbations to generate pairs of similar examples and apply information-theoretic objectives for training.

Original: PICA Huang et al.
Revised: PICA Huang et al.

Original: (2020) maximizes the global partition confidence of the clustering solution to find the most semantically plausible data separation.
Revised: (2020) maximizes the global partition confidence of the clustering solution to find the most semantically plausible data separation.

Original: Following the progress in self-supervised learning, CC Li et al.
Revised: Following the progress in self-supervised learning, CC Li et al.

Original: (2021b)
Revised: (2021b)

Original: and DCSC Zhang et al.
Revised: and DCSC Zhang et al.

Original: (2022) perform contrastive learning by generating pairs of positive and negative instances through data augmentations.
Revised: (2022) perform contrastive learning by generating pairs of positive and negative instances through data augmentations.

Original: Hierarchical methods Hierarchical clustering algorithms are a well-established area within classical data mining Murtagh & Contreras (2012), but they were rarely studied in deep learning.
Revised: Hierarchical methods Hierarchical clustering algorithms are a well-established area within classical data mining Murtagh & Contreras (2012), but they were rarely studied in deep learning.

Original: DeepECT Mautz et al.
Revised: DeepECT Mautz et al.

Original: (2019; 2020) is the only method, which jointly learns the deep representation using auto-encoder architecture and performs hierarchical clustering in a top-down manner.
Revised: (2019; 2020) is the only method, which jointly learns the deep representation using auto-encoder architecture and performs hierarchical clustering in a top-down manner.

Original: Unfortunately, there is not comparative study conducted on large image data.
Revised: Unfortunately, there is not comparative study conducted on large image data.

Original: The experimental study of objectivebased hierarchical clustering methods performed on the embedding vectors from pre-trained deep learning models is presented in Naumov et al.
Revised: The experimental study of objective-based hierarchical clustering methods performed on the embedding vectors from pretrained deep learning models is presented in Naumov et al.

Original: In the case of classification, there is a growing interest in deep hierarchical methods, which in our opinion should also be reflected in the area of unsupervised learning.
Revised: In the case of classification, there is a growing interest in deep hierarchical methods, which in our opinion should also be reflected in the area of unsupervised learning.

Original: SDT Frosst & Hinton (2017) is one of the first models that distills the base neural networks into a soft decision tree.
Revised: SDT Frosst & Hinton (2017) is one of the first models that distills the base neural networks into a soft decision tree.

Original: More advanced methods automatically generate deep networks with a tree structure in a multi-step or an end-to-end manner Tanno et al.
Revised: More advanced methods automatically generate deep networks with a tree structure in a multi-step or an end-to-end manner Tanno et al.

Original: ; Alaniz et al.
Revised: Alaniz et al.

Original: ; Wan et al.
Revised: ; Wan et al.

Original:  C O H I C LUST MODEL
Revised:  C O H I C LUST MODEL

Original: The proposed CoHiClust builds a hierarchy of clusters based on the output of the base neural network.
Revised: The proposed CoHiClust builds a hierarchy of clusters based on the output of the base neural network.

Original: There are four key components of CoHiClust:
Revised: There are four key components of CoHiClust:

Original: • The base neural network, which generates representation used by the tree.
Revised: • The base neural network, which generates representation used by the tree.

Original: • The tree model, which assigns data points to clusters by a sequence of decisions.
Revised: • The tree model, which assigns data points to clusters by a sequence of decisions.

Original: • The regularized contrastive loss, which allows for training the whole framework.
Revised: • The regularized contrastive loss, which allows for training the whole framework.

Original: • The pruning strategy, which restricts the tree to the requested number of leaves.
Revised: • The pruning strategy, which restricts the tree to the requested number of leaves.

Original: We discuss the above components in detail in the following parts.
Revised: We discuss the above components in detail in the following parts.

Original: Tree hierarchy We use a soft binary decision tree to create a hierarchical structure, where leaves play the role of clusters (similar to Frosst & Hinton (2017)).
Revised: Tree hierarchy We use a soft binary decision tree to create a hierarchical structure, where leaves play the role of clusters (similar to Frosst & Hinton (2017)).

Original: In contrast to hard decision trees, every internal node defines the probability of taking a left/right branch.
Revised: In contrast to hard decision trees, every internal node defines the probability of taking a left/right branch.

Original: The final assignment of the input examples to clusters involves partial decisions made by the internal nodes.
Revised: The final assignment of the input examples to clusters involves partial decisions made by the internal nodes.

Original: Aggregating these decisions induces the posterior probability over leaves.
Revised: Aggregating these decisions induces the posterior probability over leaves.

Original: Let us consider a complete binary tree with T levels, where the root is located at the level 0 and the leaves are represented at the level T .
Revised: Let us consider a complete binary tree with T levels, where the root is located at the level 0 and the leaves are represented at the level T .

Original: This gives us 2 t nodes at the level t denoted by tuples ( t, i ) , for i = 0 , 1 , . . .
Revised: This gives us 2 t nodes at the level t denoted by tuples ( t, i ) , for i = 0 , 1 , . . .

Original: , 2 t − 1 , see Figure 2.
Revised: , 2 t − 1 , see Figure 2.

Original: The path going from the root to the node ( t, i ) is given by the sequence of binary decisions y =
Revised: The path going from the root to the node ( t, i ) is given by the sequence of binary decisions y =

Original: ( y 1 , . . .
Revised: ( y 1 , . . .

Original: , y t ) ∈ { 0 , 1 } t made by the internal nodes, where y s =(
Revised: , y t ) ∈ { 0 , 1 } t made by the internal nodes, where y s =(

Original: y s = 1 ) means that we take the left (right) branch being in the node at the level s .
Revised: y s = 1 ) means that we take the left (right) branch being in the node at the level s .

Original: Observe that we can retrieve the index j of the node at the level s from y by taking j = b s ( y )
Revised: Observe that we can retrieve the index j of the node at the level s from y by taking j = b s ( y )

Original: =
Revised: =

Original: (cid:80) sm
Revised: (cid:80) sm

Original: =1 y m 2 s − m .
Revised: =1 y m 2 s − m .

Original: In other words, the first s bits of y are a binary representation of the number j .
Revised: In other words, the first s bits of y are a binary representation of the number j .

Original: We consider the path induced by the sequence of decisions y
Revised: We consider the path induced by the sequence of decisions y

Original: =
Revised: =

Original: ( y 1 , . . .
Revised: ( y 1 , . . .

Original: , y t ) ∈ { 0 , 1 } t , which goes from the root to the node ( t, i ) , where i =
Revised: , y t ) ∈ { 0 , 1 } t , which goes from the root to the node ( t, i ) , where i =

Original: b t ( y ) .
Revised: b t ( y ) .

Original: We want to calculate the probability P it ( x ) that the input example x ∈ R D reaches node ( t, i ) .
Revised: We want to calculate the probability P i t ( x ) that the input example x ∈ R D reaches node ( t, i ) .

Original: If p b s ( y ) s ( x ) is the probability of going from the parent node ( s − 1 , b s − 1 ( y )) to its descendant ( s,
Revised: If p b s ( y ) s ( x ) is the probability of going from the parent node ( s − 1 , b s − 1 ( y )) to its descendant ( s,

Original: b s ( y )) , then
Revised: b s ( y )) , then

Original: [Equation]
Revised: [Equation]

Original: Observe that P t ( x ) =
Revised: Observe that P t ( x ) =

Original: [ P 0 t ( x ) , P 1 t ( x ) , . . .
Revised: [ P 0 t ( x ) , P 1 t ( x ) , . . .

Original: , P 2 t − 1 t ( x )] defines a proper probability distribution, i.e.[Equation]
Revised: , P 2 t − 1 t ( x )] defines a proper probability distribution, i.e.

Original:  t [Equation] = [Equation]
Revised: (cid:80) 2 t − 1 j =0 P

Original: P jt ( x ) = 1 .
Revised: jt ( x ) = 1 .

Original: In consequence, the probability distribution over clusters (leaves) equals P T ( x ) , see Figure 2.
Revised: In consequence, the probability distribution over clusters (leaves) equals P T ( x ) , see Figure 2.

Original: Tree generation To generate our tree model, we need to parameterize the probabilities p
Revised: Tree generation To generate our tree model, we need to parameterize the probabilities p

Original: it ( x ) of taking the left/right branch in every internal node.
Revised: it ( x ) of taking the left/right branch in every internal node.

Original: To this end, we employ a neural network g : R D
Revised: To this end, we employ a neural network g : R D

Original: → R N with an additional projection head π : R N → R K , where K = 2 T − 1 and T is the height of the tree.
Revised: → R N with an additional projection head π : R N → R K , where K = 2 T − 1 and T is the height of the tree.

Original: The number K of the output neurons equals the number of internal tree nodes.
Revised: The number K of the output neurons equals the number of internal tree nodes.

Original: The neural network g is responsible for extracting high-level information from data.
Revised: The neural network g is responsible for extracting high-level information from data.

Original: It can be instantiated by a typical architecture, such as ResNet, and is used to generate embeddings z
Revised: It can be instantiated by a typical architecture, such as ResNet, and is used to generate embeddings z

Original: = g ( x ) of the input data.
Revised: = g ( x ) of the input data.

Original: We do not use pre-trained networks but train the whole model end-to-end using the proposed hierarchical contrastive loss.
Revised: We do not use pre-trained networks but train the whole model end-to-end using the proposed hierarchical contrastive loss.

Original: This allows for generating the representation, which is suited to the underlying clustering task.
Revised: This allows for generating the representation, which is suited to the underlying clustering task.

Original: The projection head π operates on the embeddings z and parameterizes decisions made by the internal tree nodes.
Revised: The projection head π operates on the embeddings z and parameterizes decisions made by the internal tree nodes.

Original: In our case, π is a single-layer network with the output dimension equal to the number of internal nodes.
Revised: In our case, π is a single-layer network with the output dimension equal to the number of internal nodes.

Original: To model binary decisions of the internal nodes, we apply the sigmoid function σ .
Revised: To model binary decisions of the internal nodes, we apply the sigmoid function σ .

Original: In consequence, the projection head is given by 
Revised: In consequence, the projection head is given by π ( z ) =

Original: [Equation]
Revised: [ σ ( w T 1 z + b 1 ) , . . .

Original:  where w n ∈ R N and b n ∈ R are trainable parameters of π .
Revised: , σ ( w T K z + b K )] , where w n ∈ R N and b n ∈ R are trainable parameters of π .

Original: By interpreting the output neurons of π as the internal nodes of the decision tree, we get the probabilities of left edges in the nodes:
Revised: By interpreting the output neurons of π as the internal nodes of the decision tree, we get the probabilities of left edges in the nodes:

Original: [Equation]
Revised: [Equation]

Original: n [Equation] Note that p 2 i − 1 t +1 ( x ) = 1 − p 2 it +1 ( x ) always corresponds to the probability of right edge.
Revised: Note that p 2 i − 1 t +1 ( x ) = 1 − p 2 it +1 ( x ) always corresponds to the probability of right edge.

Original: Contrastive hierarchical loss To train CoHiClust, we introduce the hierarchical contrastive loss function designed for trees.
Revised: Contrastive hierarchical loss To train CoHiClust, we introduce the hierarchical contrastive loss function designed for trees.

Original: Our idea relies on maximizing the likelihood that similar data points will follow the same path.
Revised: Our idea relies on maximizing the likelihood that similar data points will follow the same path.

Original: The more similar data points, the longer they should be routed through the same nodes.
Revised: The more similar data points, the longer they should be routed through the same nodes.

Original: Since we work in the unsupervised setting, we use a self-supervised approach and generate similar images by data augmentations.
Revised: Since we work in the unsupervised setting, we use a self-supervised approach and generate similar images by data augmentations.

Original: Let us consider two data points x 1 , x 2 and their posterior probabilities P t ( x 1 ) , P t ( x 2 ) at the level t .
Revised: Let us consider two data points x 1 , x 2 and their posterior probabilities P t ( x 1 ) , P t ( x 2 ) at the level t .

Original: The probability that x 1 and x 2 reach the same node at this level is given by the scalar product P t ( x 1 ) ·
Revised: The probability that x 1 and x 2 reach the same node at this level is given by the scalar product P t ( x 1 ) ·

Original: P t ( x 2 ) =
Revised: P t ( x 2 ) =

Original: (cid:80) 2 t − 1 i =

Revised: (cid:80) 2 t − 1 i =

Original: P it ( x 1 ) P it ( x 2 ) .
Revised: P it ( x 1 ) P it ( x 2 ) .

Original: This term is maximal if both probabilities are identical one-hot vectors.
Revised: This term is maximal if both probabilities are identical one-hot vectors.

Original: In a training phase, we do not want to force hard splits in the nodes (binary probabilities) because in this way the model quickly finds a local minimum by assigning data points to fixed leaves with high confidence.
Revised: In a training phase, we do not want to force hard splits in the nodes (binary probabilities) because in this way the model quickly finds a local minimum by assigning data points to fixed leaves with high confidence.

Original: Instead of sticking to hard assignments in a few training epochs, we want to let the model explore possible solutions.
Revised: Instead of sticking to hard assignments in a few training epochs, we want to let the model explore possible solutions.

Original: To this end, we take the square root before applying the scalar product, which corresponds to the Bhattacharyya coefficient Bhattacharyya (1946):
Revised: To this end, we take the square root before applying the scalar product, which corresponds to the Bhattacharyya coefficient Bhattacharyya (1946):

Original: [Equation]
Revised: [Equation]

Original: Observe that s t ( x 1 , x 2 ) =
Revised: Observe that s t ( x 1 , x 2 ) =

Original: 1 , if only P t ( x 1 ) =
Revised: 1 , if only P t ( x 1 ) =

Original: P t ( x 2 ) (probabilities do not have to binarize), which leads to the exploration of possible paths.
Revised: P t ( x 2 ) (probabilities do not have to binarize), which leads to the exploration of possible paths.

Original: By aggregating the similarity scores over all tree levels, we arrive at our final similarity function s ( x 1 , x 2 ) =
Revised: By aggregating the similarity scores over all tree levels, we arrive at our final similarity function s ( x 1 , x 2 ) =

Original: (cid:80) T − 1 t =

Revised: (cid:80) T − 1 t =

Original: s t ( x 1 , x 2 ) .
Revised: s t ( x 1 , x 2 ) .

Original: In a training phase, we take a minibatch { x j } Nj =1 of N examples and generate its augmented view { ˜ x j } Nj =1 .
Revised: In a training phase, we take a minibatch { x j } Nj =1 of N examples and generate its augmented view { ˜ x j } Nj =1 .

Original: Every pair ( x j , ˜ x j ) is considered positive, which means that we will maximize their similarity score.
Revised: Every pair ( x j , ˜ x j ) is considered positive, which means that we will maximize their similarity score.

Original: In consequence, we encourage the model to assign them to the same leaf node.
Revised: In consequence, we encourage the model to assign them to the same leaf node.

Original: To avoid degenerate solutions, where all points end up in the same leaf, we treat all other pairs negative and minimize the similarity scores for them.
Revised: To avoid degenerate solutions, where all points end up in the same leaf, we treat all other pairs negative and minimize the similarity scores for them.

Original: Finally, the proposed hierarchical contrastive loss is given by:
Revised: Finally, the proposed hierarchical contrastive loss is given by:

Original: [Equation]
Revised: [Equation]

Original: By minimizing the above loss, we maximize the likelihood that similar data points follow the same path (second term) and minimize the likelihood that dissimilar ones are grouped together.
Revised: By minimizing the above loss, we maximize the likelihood that similar data points follow the same path (second term) and minimize the likelihood that dissimilar ones are grouped together.

Original: Regularization Final cluster assignments are induced by aggregating several binary decisions made by the internal tree nodes.
Revised: Regularization Final cluster assignments are induced by aggregating several binary decisions made by the internal tree nodes.

Original: In practice, the base neural network may not train all nodes and, in consequence, use only a few leaves for clustering.
Revised: In practice, the base neural network may not train all nodes and, in consequence, use only a few leaves for clustering.

Original: While selecting the number of clusters in flat clustering is desirable, here we would like to create a hierarchy, which is not restricted to a given number of leaves.
Revised: While selecting the number of clusters in flat clustering is desirable, here we would like to create a hierarchy, which is not restricted to a given number of leaves.

Original: To enable end-to-end training of the base neural network with the arbitrary number of leaves, we consider two regularization strategies.
Revised: To enable end-to-end training of the base neural network with the arbitrary number of leaves, we consider two regularization strategies.

Original: The first regularization  explicitly encourages the model to use both left and right sub-trees equally Frosst & Hinton (2017).
Revised: The first regularization (dubbed R 1 ) explicitly encourages the model to use both left and right subtrees equally Frosst & Hinton (2017).

Original: We realize this postulate by minimizing the cross-entropy between the desired distribution [0 . 5 , 0 . 5] and the actual distribution for choosing theleft or right path in a given node. [Table]
Revised: We realize this postulate by minimizing the cross-entropy between the desired distribution [0 . 5 , 0 . 5] and the actual distribution for choosing the

Original: [Table]
Revised: left

Original:  The second regularization  does not directly influence the routing in the tree but focuses on improving the output representation of the base network g .
Revised: or right path in a given node The second regularization (dubbed R 2 ) does not directly influence the routing in the tree but focuses on improving the output representation of the base network g .

Original: For this purpose, we use NT-Xent loss Chen et al.
Revised: For this purpose, we use NT-Xent loss Chen et al.

Original: (2020) on the embedding space
Revised: (2020) on the embedding space

Original: z
Revised: z

Original: = g ( x ) .
Revised: = g ( x ) .

Original: With the NT-Xent loss, we maximize the cosine similarity on the embedding space for all positive pairs and minimize the cosine similarity on the embedding space for all negative pairs.
Revised: With the NT-Xent loss, we maximize the cosine similarity on the embedding space for all positive pairs and minimize the cosine similarity on the embedding space for all negative pairs.

Original: Taking together the contrastive loss CoHiLoss with two regularization functions R 1 (for entropy) and R 2 (for NT-Xent) [Table] we arrive at our final loss:
Revised: Taking together the contrastive loss CoHiLoss with two regularization functions R 1 (for entropy) and R 2 (for NT-Xent), we arrive at our final loss:

Original: [Equation]
Revised: [Equation]

Original: where β 1 , β 2 are the user-defined hyperparameters.
Revised: where β 1 , β 2 are the hyperparameters defining the importance of regularization terms R 1 and R 2 , respectively.

Original: Instead of selecting their values in the validation procedure (which can be problematic in unsupervised learning), we set β 1 proportional to the depth of the tree β 1 = 2 − T and β 2 = 1 . 
Revised: To generate a full hierarchy (complete tree with the assumed height), we set β 1 proportional to the depth of the tree β 1 = 2 − T and β 2 = 1 . In appendix, we show that CoHiClust can also detect the number of clusters automatically by putting β 1 = 0 (see appendix for the ablation study).

Original: Pruning The proposed model builds a complete tree with 2 T leaves.
Revised: Pruning The proposed model builds a complete tree with 2 T leaves.

Original: Although such a structure is useful for analyzing the hierarchy of clusters, in some cases we are interested in creating a tree with the requested number of groups, which is not equal to the power of 2.
Revised: Although such a structure is useful for analyzing the hierarchy of clusters, in some cases we are interested in creating a tree with the requested number of groups.

Original: For this purpose, we introduce a pruning strategy that reduces the least significant leaf nodes.
Revised: For this purpose, we introduce a pruning strategy that reduces the least significant leaf nodes.

Original: In contrast to typical pruning approaches, which focus solely on deleting nodes, we further improve the hierarchy by training CoHiClust in the pruning phase.

Original: We start with calculating the probability of leaves P iT = 1 | X | (cid:80) x ∈ X P iT ( x ) , which describes the expected fraction of data points assigned to a given leaf.
Revised: We start with calculating the probability of leaves P iT = 1 | X | (cid:80) x ∈ X P iT ( x ) , which describes the expected fraction of data points assigned to a given leaf.

Original: Assuming that the importance of the cluster is related to the average number of assigned examples, we reduce a leaf with the lowest probability.
Revised: Assuming that the importance of the cluster is related to the average number of assigned examples, we reduce a leaf with the lowest probability.

Original: After removing the leaf, we finetune the whole model using CoHiClust loss (1).
Revised: After removing the leaf, we fine-tune the whole model using CoHiClust loss (2).

Original: If we want to reduce more leaves, we perform leaf pruning and model retraining iteratively until the requested number of leaves is obtained.
Revised: If we want to reduce more leaves, we perform leaf pruning and model retraining iteratively until the requested number of leaves is obtained.

Original: The resulting tree is binary, but can have an arbitrary structure and does not have to be complete.
Revised: The resulting tree is binary, but can have an arbitrary structure and does not have to be complete.

Original: Consequently, pruning is also a way of improving the topology of the tree model.
Revised: Consequently, pruning is also a way of improving the topology of the tree model.

Revised: Alternatively, we could also perform the pruning strategy without applying fine-tuning step.

Revised: This potentially saves the computational resources, but does not adjust hierarchy to the modified number of groups.

Revised: In appendix, we show that this approach generates partition of lower quality than the model with fine-tuning.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original:  We evaluate our method with several datasets of increasing difficulty.
Revised: [Table] [Table] We evaluate our method with several datasets of increasing difficulty.

Original: We show that CoHiClust outperforms hierarchical clustering models and is among the three best-performing flat clustering methods in terms of similarity with the ground truth classes.
Revised: We show that CoHiClust outperforms hierarchical clustering models and is among the three best-performing flat clustering methods in terms of similarity with the ground truth classes.

Original: In addition, we analyze the constructed hierarchies, which in our opinion are equally important in practical use-cases.
Revised: In addition, we analyze the constructed hierarchies, which in our opinion are equally important in practical use-cases.

Original: Our experiments demonstrate that CoHiClust reveals novel patterns and relations between data points beyond those contained in ground truth classes.
Revised: Our experiments demonstrate that CoHiClust reveals novel patterns and relations between data points beyond those contained in ground truth classes.

Original: Comparison with hierarchical clustering methods First, we compare CoHiClust with hierarchical baselines.
Revised: Comparison with hierarchical clustering methods First, we compare CoHiClust with hierarchical baselines.

Original: To the best of our knowledge, DeepECT Mautz et al.
Revised: To the best of our knowledge, DeepECT Mautz et al.

Original: (2019) is the only hierarchical clustering method based on deep neural networks.
Revised: (2019) is the only hierarchical clustering method based on deep neural networks.

Original: Following their experimental setup, we also consider classical hierarchical algorithms evaluated on the latent representation created by the autoencoder and
Revised: Following their experimental setup, we also consider classical hierarchical algorithms evaluated on the latent representation created by the autoencoder and

Original: IDEC Guo et al.
Revised: IDEC Guo et al.

Original: (2017a).
Revised: (2017a).

Original: We report the results on two popular image datasets 1 :
Revised: We report the results on two popular image datasets 1 :

Original: MNIST and F-MNIST.
Revised: MNIST and F-MNIST.

Original: [Figure] We run CoHiClust with 16 leaves (4 tree levels) using ResNet18 as a base neural network.
Revised: We run CoHiClust with 16 leaves (4 tree levels) using ResNet18 as a base neural network.

Original: The initial model is trained for 50 epochs on MNIST and 100 epochs on F-MNIST using a minibatch size of 256 .
Revised: The initial model is trained for 50 epochs on MNIST and 100 epochs on F-MNIST using a minibatch size of 256 .

Original: Next, we perform pruning with an additional 50 and 100 epochs of fine-tuning on MNIST and F-MNIST, respectively, to end up with 10 clusters, which equals the true number of classes for considered datasets.
Revised: Next, we perform pruning with an additional 50 and 100 epochs of fine-tuning on MNIST and F-MNIST, respectively, to end up with 10 clusters, which equals the true number of classes for considered datasets.

Original: To evaluate the cluster hierarchies against ground truth flat partition, we use dendrogram purity (DP) Kobren et al.
Revised: To evaluate the cluster hierarchies against ground truth flat partition, we use dendrogram purity (DP) Kobren et al.

Original: ; Yang et al.
Revised: ; Yang et al.

Original: (2019),which attains its maximum value of 1 if and only if all data points from the same class are assigned to some pure sub-tree.
Revised: (2019),which attains its maximum value of 1 if and only if all data points from the same class are assigned to some pure sub-tree.

Original: The results summarized in Table 1 demonstrate that CoHiClust outperforms all baselines on both
Revised: The results summarized in Table 1 demonstrate that CoHiClust outperforms all baselines on both

Original: MNIST and F-MNIST datasets.
Revised: MNIST and F-MNIST datasets.

Original: Interestingly, DeepECT benefits from data augmentation in the case of MNIST, while on F-MNIST it deteriorates its performance.
Revised: Interestingly, DeepECT benefits from data augmentation in the case of MNIST, while on F-MNIST it deteriorates its performance.

Original: All methods except CoHiClust and
Revised: All methods except CoHiClust and

Original: DeepECT failed completely to create a hierarchy recovering true classes, which confirms that there is a lack of powerful hierarchical clustering methods based on neural networks.
Revised: DeepECT failed completely to create a hierarchy recovering true classes, which confirms that there is a lack of powerful hierarchical clustering methods based on neural networks.

Original: To further evaluate the performance on MNIST and F-MNIST, we measure the normalized mutual information (NMI) and the clustering accuracy (ACC) between the ground truth partition and the flat clustering generated from the tree hierarchy.
Revised: To further evaluate the performance on MNIST and F-MNIST, we measure the normalized mutual information (NMI) and the clustering accuracy (ACC) between the ground truth partition and the flat clustering generated from the tree hierarchy.

Original: We exclude the combinations of hierarchical and embedding methods from the comparison due to their poor performance.
Revised: We exclude the combinations of hierarchical and embedding methods from the comparison due to their poor performance.

Original: Instead, we consider two flat clustering methods: IDEC and k-means evaluated on the latent representation generated by the autoencoder.
Revised: Instead, we consider two flat clustering methods: IDEC and k-means evaluated on the latent representation generated by the autoencoder.

Original: It can be seen from Table 2 that CoHiClust outperforms baseline methods across all datasets and metrics.
Revised: It can be seen from Table 2 that CoHiClust outperforms baseline methods across all datasets and metrics.

Original: The disproportion between the results obtained on MNIST and F-MNIST demonstrates that recovering true classes of clothes is a significantly more challenging task than recognizing the hand- written digits.
Revised: The disproportion between the results obtained on MNIST and F-MNIST demonstrates that recovering true classes of clothes is a significantly more challenging task than recognizing the hand-written digits.

Original: Analyzing the clustering hierarchies Although comparing constructed clustering with the ground truth partition is a widely-used evaluation measure in the literature, it is even more important from a practical point of view to visualize the results and perform their qualitative assessment.
Revised: Analyzing the clustering hierarchies Although comparing constructed clustering with the ground truth partition is a widely-used evaluation measure in the literature, it is even more important from a practical point of view to visualize the results and perform their qualitative assessment.

Original: Figure 1 presents the clustering tree constructed for F-MNIST (see appendix for the hierarchy of
Revised: Figure 1 presents the clustering tree constructed for F-MNIST (see appendix for the hierarchy of

Original: MNIST).
Revised: MNIST).

Original: Images in nodes show the means calculated over data points assigned to the respective nodes.
Revised: Images in nodes show the means calculated over data points assigned to the respective nodes.

Original: For every leaf, we also present sample images classified to this cluster and the percentage of images from dominant classes in the group (with a fraction higher than 15%).
Revised: For every leaf, we also present sample images classified to this cluster and the percentage of images from dominant classes in the group (with a fraction higher than 15%).

Original: Visual inspection shows that neighboring leaves contain images with similar patterns, see description in Figure 1 for detailed findings.
Revised: Visual inspection shows that neighboring leaves contain images with similar patterns, see description in Figure 1 for detailed findings.

Original: The hierarchy also allows us to define the distance d ( a, b ) between two examples a, b using the number of edges that connect them.
Revised: The hierarchy also allows us to define the distance d ( a, b ) between two examples a, b using the number of edges that connect them.

Original: We use the average of this score to calculate the similarity between ground truth classes A and B : [Equation] where Z is the number of all pairs.
Revised: We use the average of this score to calculate the similarity between ground truth classes A and B given by d ( A, B ) = 1 Z (cid:80) a ∈ A (cid:80) b ∈ B d ( a, b ) , where Z is the number of all pairs.

Original: The above distance is small if examples from classes A and B are located in the nearby leaf nodes (on average).
Revised: The above distance is small if examples from classes A and B are located in the nearby leaf nodes (on average).

Original: The obtained distance matrix presented in Figure 3 (left) confirms
Revised: The obtained distance matrix presented in Figure 4 (left) confirms

Original: that clusters discovered by
Revised: that clusters discovered by

Original: CoHiClust reveal many different patterns than the information encoded in the ground truth classes.
Revised: CoHiClust reveal many different patterns than the information encoded in the ground truth classes.

Original: While clothes are well separated from other items (number of edges greater than 6), we cannot distinguish particular clothes items from each other (distance between 3 and 4).
Revised: While clothes are well separated from other items (number of edges greater than 6), we cannot distinguish particular clothes items from each other (distance between 3 and 4).

Original: As can be seen in
Revised: As can be seen in

Original: Figure 1, CoHiClust generated the hierarchy based on the sleeve length or the presence of designs.
Revised: Figure 1, CoHiClust generated the hierarchy based on the sleeve length or the presence of designs.

Original: From this point of view, we can say that the results obtained by CoHiClust are complementary to the ground truth classification.
Revised: From this point of view, we can say that the results obtained by CoHiClust are complementary to the ground truth classification.

Original: However, we can also observe some similarities with ground truth classification: it is evident that the elements of the following 5 classes have not been spread between clusters: ”trousers”, ”sandals”, ”sneakers”, ”bags” and ”ankle boots” (we have d ( A, A ) < 0 for these classes).
Revised: However, we can also observe some similarities with ground truth classification: it is evident that the elements of the following 5 classes have not been spread between clusters: ”trousers”, ”sandals”, ”sneakers”, ”bags” and ”ankle boots” (we have d ( A, A ) < 0 for these classes).

Original: Evaluation on large-scale datasets In addition to comparing CoHiClust with hierarchical baselines on gray images, we perform the evaluation on large-scale datasets of color images: CIFAR-10,
Revised: Evaluation on large image datasets In addition to comparing CoHiClust with hierarchical baselines on gray images, we perform the evaluation on large datasets of color images: CIFAR-10,

Original: STL-10, ImageNet-Dogs, and ImageNet-10.
Revised: STL-10, ImageNet-Dogs, and ImageNet-10.

Original: Since none of the previous hierarchical methods have been examined on these datasets, we use  the benchmark including flat clustering methods reported in
Revised: Since none of the previous hierarchical methods have been examined on these datasets, we use typical agglomerative hierarchical algorithms applied either to raw image data or to the representation generated by our model.

Original:  Li et al.
Revised: Additionally, to compare with a more diverse set of methods, we use the benchmark including flat clustering methods reported in Li et al.

Original: (2021a).
Revised: (2021a).

Original: It is expected that flat clustering methods will perform better because they directly[Table] focus on partitioning data into a given number of groups.
Revised: It is expected that flat clustering methods will perform better because they directly focus on partitioning data into a given number of groups.

Original: Hierarchical models, such as CoHiClust, build a hierarchy with the number of leaves exceeding the number of classes, which  makes it difficult to obtain so high a resemblance with the ground truth as flat models.
Revised: Hierarchical models, such as CoHiClust, build a hierarchy with the number of leaves exceeding the number of classes, which [Figure] makes it difficult to obtain so high a resemblance with the ground truth as flat models.

Original: Moreover, the optimization of hierarchy is a more difficult and challenging process because final assignments rely on aggregating several binary decisions made by the internal nodes.
Revised: Moreover, the optimization of hierarchy is a more difficult and challenging process because final assignments rely on aggregating several binary decisions made by the internal nodes.

Original: Our model uses ResNet34 as a base network, which better suits the complexity of the current images and coincides with the architecture used in Li et al.
Revised: Our model uses ResNet34 as a base network, which better suits the complexity of the current images and coincides with the architecture used in Li et al.

Original: (2021a).
Revised: (2021a).

Original: We consider a tree with 16 leaves ( levels), which is next pruned to obtain the number of clusters equal to the number of ground truth classes.
Revised: We consider a tree with 16 leaves ( levels), which is next pruned to obtain the number of clusters equal to the number of ground truth classes.

Original: We train the initial complete tree with the batch size of 256 for 1000 epochs and use 100 epochs after pruning every leaf.
Revised: We train the initial complete tree with the batch size of 256 for 1000 epochs and use 100 epochs after pruning every leaf.

Original: To measure the similarity of the constructed partition with the ground truth, we apply three widely-used clustering metrics: normalized mutual information (NMI), clustering accuracy (ACC), and adjusted rand index (ARI).
Revised: To measure the similarity of the constructed partition with the ground truth, we apply three widely-used clustering metrics: normalized mutual information (NMI), clustering accuracy (ACC) [Table] and adjusted rand index (ARI).

Original: In the appendix, we also show DP of the hierarchies generated by CoHiClust.
Revised: In the appendix, we also show DP of the hierarchies generated by CoHiClust.

Original:  The results presented in Table 3 show that CoHiClust usually is  among the three best-performing methods.
Revised: [Table] The results presented in Table 3 show that CoHiClust outperforms hierarchical baselines with a large margin (bottom rows). Moreover, it is usually among the three best-performing flat methods (top rows).

Original: It gives very good results, compared to the baselines, on the subset of the ImageNet dataset achieving 0.742 of NMI.
Revised: It gives very good results, compared to the baselines, on the subset of the ImageNet dataset achieving 0.742 of NMI.

Original: Nevertheless, one should keep in mind that CoHiClust is the only hierarchical method in this comparison, which makes it more challenging to construct flat partition, which resembles the ground truth classes.
Revised: Nevertheless, one should keep in mind that CoHiClust is the only hierarchical method in this comparison, which makes it more challenging to construct flat partition, which resembles the ground truth classes.

Original: To better analyze the results returned by CoHiClust, we plot the constructed clustering hierarchies and distance matrices calculated for ground truth classes.
Revised: To better analyze the results returned by CoHiClust, we plot the constructed clustering hierarchies and distance matrices calculated for ground truth classes.

Original: Figure 4 presents and discusses the tree hierarchy constructed for ImageNet-10 (remaining hierarchies together with distance matrices are presented in the appendix).
Revised: Figure 3 presents and discusses the tree hierarchy constructed for ImageNet-10 (remaining hierarchies together with distance matrices are presented in the appendix).

Original: Supporting the analysis with the use of a distance matrix, we can observe that examples of 5 out of 10 classes have not been spread across different clusters (we have d ( A, A ) ≤ 1 ).
Revised: Supporting the analysis with the use of a distance matrix, we can observe that examples of 5 out of 10 classes have not been spread across different clusters (we have d ( A, A ) ≤ 1 ).

Original: Airliner and airship classes are the most semantically similar.
Revised: Airliner and airship classes are the most semantically similar.

Original: The class king penguin is the most distinct from the classes representing the means of transport with an average distance greater than 7.
Revised: The class king penguin is the most distinct from the classes representing the means of transport with an average distance greater than 7.

Original: Overall, we can see a very high similarity between classes related to the means of transport.
Revised: Overall, we can see a very high similarity between classes related to the means of transport.

Original: The distance between other classes is not as small, which suggests that there are no evident patterns that connect e.g. king penguin, Maltese dog, or orange.
Revised: The distance between other classes is not as small, which suggests that there are no evident patterns that connect e.g. king penguin, Maltese dog, or orange.

Original:  C ONCLUSION 
Revised:  C ONCLUSION AND FUTURE DIRECTIONS

Original: We proposed a contrastive hierarchical clustering model CoHiClust, which suits well to clustering of large-scale image databases.
Revised: We proposed a contrastive hierarchical clustering model CoHiClust, which suits well to clustering of large-scale image databases.

Original: The hierarchical structure constructed by CoHiClust delivers significantly more information about data than typical flat clustering models.
Revised: The hierarchical structure constructed by CoHiClust delivers significantly more information about data than typical flat clustering models.

Original: In particular, we can inspect the similarity between selected groups by measuring their distance in the hierarchy tree and, in consequence, find super-clusters.
Revised: In particular, we can inspect the similarity between selected groups by measuring their distance in the hierarchy tree and, in consequence, find super-clusters.

Original: Experimental analysis performed on typical clustering benchmarks confirms that the produced partitions have high similarity with ground truth classes.
Revised: Experimental analysis performed on typical clustering benchmarks confirms that the produced partitions have high similarity with ground truth classes.

Original: At the same time CoHiClust allows for discovering important patterns which have not been encoded in the class labels.
Revised: At the same time CoHiClust allows for discovering important patterns which have not been encoded in the class labels.

Revised: In future, we plan to apply the constructed model to other unsupervised problems such as anomaly detection.

Revised: Moreover, we will work on constructing clustering hierarchies which are not restricted to a predefined topology.

Revised: In particular, inspired by Tanno et al.

Revised: ; Struski et al.

Revised: (2021), we will focus on models, which automatically adapt the hierarchy structure to a given dataset.

Original:  A S UPPLEMENTARY EXPERIMENTAL RESULTS
Revised: Finally, it is an open issue whether it is possible to extend flat clustering models Li et al. (2021a) ; Hua to the hierarchical ones, which could further improve the clustering metrics. A S UPPLEMENTARY QUALITATIVE RESULTS

Original: In Figures 5, 6, 7 and 8 we illustrate the tree hierarchies constructed by CoHiClust for the remaining datasets.
Revised: In Figures 5, 6, 7 and 8 we illustrate the tree hierarchies constructed by CoHiClust for the remaining datasets.

Original: Distance matrices for ground true classes generated from these hierarchies are shown in
Revised: Distance matrices for ground true classes generated from these hierarchies are shown in

Original: Figure 9.
Revised: Figure 9.

Original: B A BLATION STUDY
Revised: B A NALYSIS OF THE C O H I C LUST MODEL

Revised: In this section

Revised: [Table]

Revised: we analyze selected properties of CoHiClust including the choice of backbone network, the form of the loss function, techniques for reducing the number of leaves and possible training strategies.

Revised: If not stated otherwise, the experiments were run on CIFAR-10 dataset, which we consider the most representative.

Revised: B. R ELIANCE ON BACKBONE NETWORK

Original: In Table
Revised: In Table

Original: [Table]
Revised: [Table]

Original: we show how the selection of the architecture for the base network g influences the performance of CoHiClust.
Revised: we show how the selection of the architecture for the base network g influences the performance of CoHiClust.

Original: As can be seen, the difference in the results is marginal.
Revised: As can be seen, the difference in the results is marginal.

Original: [Table]
Revised: [Table]

Revised: [Figure]

Revised: B. A NALYSIS OF LOSS FUNCTION

Revised: Next, we explain the influence of particular components of CoHiClust loss function.

Revised: Namely, we remove one or both regularization terms from the loss function and report the results.

Revised: As shown in

Revised: Table 5, R 2 regularization (NT-XENT) has a notable impact on the metrics, since it improves the representation of data.

Revised: However, the obtained model has a tendency to build smaller tree and reduce clusters during training (third row).

Revised: The influence of R 1 (entropy) on the quality of clustering is marginal, but this term is crucial for balancing decisions made by the internal nodes and preventing from clusters reduction (second row).

Revised: Combining both regularization terms allows us to build a complete hierarchy with satisfactory metrics.

Revised: [Table]

Revised: Following the above analysis, we observe that CoHiClust can automatically detect the number of clusters if we eliminate the entropy regularization.

Revised: More precisely, R 2 is essential for delivering appropriate representation of data, while R 1 is only used to maintain the requested number clusters.

Revised: If we are interested in generating hierarchy of the size automatically adjusted to a given dataset, then we have to put β 1 = 0 and β 2 = 1 .

Revised: B. A LTERNATIVE DEFINITIONS OF THE LOSS FUNCTION

Revised: Let us recall that the proposed CoHiLoss contains the square root to enable the model to explore possible tree paths, see equation 1.

Revised: If we do not use the square root, we encourage the model to binarize decisions in nodes, which can lead to stacking in local minima at the initial phase of training.

Revised: The results in Table 6 show that the model with square root obtains better metrics than the one without the square root.

Revised: It partially confirms our initial hypothesis that using the square root influences positively on the exploration of the tree

Revised: [Table]

Revised: [Table]

Revised: B. C OMPARING TECHNIQUES FOR REDUCING THE NUMBER OF LEAVES

Revised: Complete hierarchy is a useful tool to analyze the relationships between data.

Revised: To generate the hierarchy with smaller number of leaves we have the following options:

Revised: • Eliminate entropy regularization from the loss function ( β 1 = 0 ).

Revised: In this case, the model automatically detects the number of groups.

Revised: • Reduce the least significant leaves simultaneously with fine-tuning the model.

Revised: Fine-tuning is performed during pruning and after obtaining the final model (our default strategy).

Revised: • Delete the least significant leaves without fine-tuning.

Revised: This corresponds to one of typical pruning strategies used in decision trees.

Revised: Table 7 shows the clustering metrics for the obtained models evaluated on CIFAR-10.

Revised: As can be seen, pruning combined with fine-tuning gives the best results (third row), improving significantly the clustering structure delivered by the complete hierarchy (second row).

Revised: The model without finetuning (fourth row) also gives higher scores than the base model without pruning, however the gain in clustering metrics is not so high.

Revised: This shows that fine-tuning the pruned model is an essential step leading in adjusting the hierarchy to a given number of groups.

Revised: It is notable that the model without entropy regularization ( β 1 = 0 ) is able to reasonably well detect the number of clusters as well as to produce satisfactory results.

Revised: It obtains better clustering metrics than the model with pruning but without fine-tuning.

Revised: [Table]

Revised: CoHiClust ( β 1 = 0 )

Revised: [Table]

Revised: CoHiClust (w/o pruning)

Revised: [Table]

Revised: CoHiClust (pruning and fine-tuning)

Revised: [Table]

Revised: CoHiClust (pruning only)

Revised: [Table]

Revised: B. E ND - TO - END TRAINING VS . PRE - TRAINED REPRESENTATIONS

Original: Next, we analyze whether end-to-end training is crucial for the performance of CoHiClust.
Revised: Finally, we examine whether end-to-end training is crucial for the performance of CoHiClust.

Original: For this purpose, we first pretrain the base network g using NT-Xent loss, which corresponds to the typical SimCLR model Chen et al.
Revised: For this purpose, we first pretrain the base network g using NT-Xent loss, which corresponds to the typical SimCLR model Chen et al.

Original: Next, we add the projection head π to g and train CoHiClust using CoHiLoss + R 2 loss.
Revised: Next, we add the projection head π to g and train CoHiClust using CoHiLoss + R 2 loss.

Original: As can be seen in Table 5 the model trained in an end-to-end manner obtains significantly better scores.[Figure]
Revised: As can be seen in Table 8 the model trained in an end-to-end manner obtains significantly better scores.

Original: [Table]
Revised: [Table]

Original: C D ENDROGRAM PURITY FOR DATASETS OF COLOR IMAGES
Revised: C D ENDROGRAM PURITY FOR DATASETS OF COLOR IMAGES

Original: In Table
Revised: In Table

Original: [Table]
Revised: [Table]

Original: we present dendrogram purity obtained by CoHiClust for datasets of color images.
Revised: we present dendrogram purity obtained by CoHiClust for datasets of color images.

Original: This benchmark can be used to compare other hierarchical clustering models.
Revised: This benchmark can be used to compare other hierarchical clustering models

Original: [Table] 
Revised: [Table] [Table]