An Empirical Study on Clustering Pretrained Embeddings: Is Deep Strictly Better?
Abstract
Recent research in clustering face embeddings has found that unsupervised, shallow, heuristic-based methods—including k -means and hierarchical agglomerative clustering—underperform supervised, deep, inductive methods.
While the reported improvements are indeed impressive, experiments are mostly limited to face datasets, where the clustered embeddings are highly discriminative or wellseparated by class (Recall@1 above 90% and often near ceiling), and the experimental methodology seemingly favors the deep methods.
We conduct an empirical study of 14 clustering methods on two popular non-face datasets—Cars196 and
Stanford Online Products—and obtain robust, but contentious ﬁndings.
Notably, deep methods are surprisingly fragile for embeddings with more uncertainty, where they underperform the shallow, heuristic-based methods.
We believe our benchmarks broaden the scope of supervised clustering methods beyond the face domain and can serve as a foundation on which these methods could be improved.
Clustering—the process by which a set of items are semantically-partitioned into ﬁnite groups— has been employed in many domains of computing including machine learning, graphics, information retrieval, and bioinformatics.
It has numerous applications including interpretability, compression, visualization, outlier detector, and zero-shot classiﬁcation.
While clustering methods have traditionally ingested raw features (e.g., pixels) as input, an alternative explored particularly in the face-veriﬁcation literature is to use deep embeddings .
Deep embeddings are latent-feature vectors discovered by neural networks [11, 22, 28, 29, 34, 35, 41, 44, 43, 45, 46].
They form a natural space for clustering because they are learned with loss functions that encourage grouping of semantically-similar inputs [4, 6, 7, 13, 37, 42, 38] and explicit representation of task-relevant features [30, 31].
Face veriﬁcation systems rely on large backbone networks that produce highly-discriminative embeddings of face images.
The term discriminative refers to the network’s ability to separate embeddings that belong to the same identity or class from those that do not, as measured by a distance function.
Improvements to these systems come largely through increasing the available labeled data, but with an expensive annotation cost.
To overcome the annotation challenge, recent work has successfully used clustering to pseudo-label [15, 41, 44, 43, 46, 34, 28].
Since the crux of the approach is the clustering step, as improved clustering leads to better pseudo-labels and thus improved embedding quality, research has shifted from unsupervised, shallow, heuristic-based clustering methods [3, 5, 8–10, 16, 17, 21–23, 25, 27, 29, 33, 35, 36, 40] to supervised, deep, inductive methods [11, 15, 28, 34, 41, 44, 43, 45, 46], with claimed improvements upward of 20% over k -means [25] and hierarchical agglomerative clustering (HAC) [36], for example.
Cars
[Table] m n mn
[Table]
n
[Table]
One might, and arguably should, expect the deep methods to generalize beyond the face domain to other visual domains, as an innumerable number of research articles attest to inter-domain generalization. Additionally, the prior work on clustering pretrained embeddings has begun to expand beyond the face domain to other datasets (e.g., DeepFashion [24]) showing similar, impressive results. However, the prior work has limitations.
First, by focusing primarily on face datasets, the clustering methods operate on embeddings that are highly discriminative and not representative of the many embedding spaces that could beneﬁt from clustering.
Deng et al.
[7] show that both veriﬁcation and Recall@1 [26] performance of the embeddings are well-above 90% and can even approach ceiling performance across a range of face datasets, including MegaFace [18].
We also conﬁrm that the embeddings for DeepFashion [24]—one of the only non-face datasets explored—exhibit Recall@ performance above 90%.
Second, common face datasets (e.g., MS-Celeb1M [12] and MegaFace [18]) are no longer publicly available, with progress reliant on using shared embeddings extracted from pretrained backbones.
Third, similar to Musgrave et al.
[26], we ﬁnd methodological choices that may implicitly favor the recent, deep methods; these choices include the lack of a validation split and monitoring test performance for hyperparameter tuning and early stopping.
For the reasons above, we empirically studied methods for clustering pretrained embeddings.
We focused on outlining an end-to-end, reproducible pipeline, including dataset curation, backbone training, and clustering.
We then benchmarked 14 clustering methods—including two state-of-the-art, supervised, deep methods, GCN-VE [43] and STAR-FC [34]—on two popular, non-face datasets:
Cars196 [20] and Stanford Online Products (SOP) [39].
We speciﬁcally chose to benchmark against
Cars196 and SOP because they are: (1) popular in the embedding-learning literature [26, 32, 4], (2) extend beyond the face domain, and (3) produce embeddings that are signiﬁcantly less discriminative.
Our results falsify conclusions that deep clustering methods strictly outperform shallow methods.
On Cars196 and SOP, we show that the state-of-the-art, deep methods underperform methods such as k -means and HAC.
We posit a relationship between Recall@1 performance of the embeddings and the beneﬁts provided by deep clustering methods, where the improvements shown with deep methods only manifest for embedding spaces exhibiting strong retrieval performance.
We afﬁrm an additional, but unrelated conclusion that (cid:96) 2 -normalization of embeddings prior to clustering leads to a stark improvement across all heuristic-based methods, exploiting the geometrical properties of embeddings learned with softmax cross-entropy [32].
Methodology
We consider two datasets for our experiments: Cars196 [20] and Stanford Online Products (SOP) [39].
Both datasets are partitioned into a backbone train split, used to train the backbone, a clustering train split, used only to train the deep clustering methods, a validation split, used for hyperparameter tuning and early stopping for the backbone and deep clustering methods, and a test split, used to ﬁt the unsupervised clustering methods and evaluate all clustering methods.
All splits have disjoint sets of classes.
Table 1 has per-split statistics on the number of instances, number of classes, and approximate number of instances per class for both datasets.
We use a ResNet50 [14] for the backbone with an additional fully-connected layer added after global average pooling and prior to the classiﬁcation head, which produces a 256D embedding.
The model is trained with cosine softmax cross-entropy—based on ﬁndings from Scott et al.
[32]—using the backbone-train split of each dataset, with Recall@1 monitored on the validation split for early stopping and hyperparameter tuning.
A hyperparameter search was performed for each of the datasets.
Additional details on training the backbone, including speciﬁcs on the architecture, data augmentation, and hyperparameters are included in Appendix E.1.
The clustering methodology varies between the supervised, deep methods (e.g., GCN-VE and
STAR-FC) and the unsupervised, shallow methods (e.g., k -means), discussed below.
Appendix C
H a r m o n i c M e a n o f
F P a n d
F B
[Figure]
-Normalized Embeddings
[Figure]
a r m o n i c M e a n o f
F P a n d
F B
[Figure]
Unsupervised
Supervised
[Table]
enumerates and details both classes of methods.
Appendix E.2 contains additional methodological details, architecture details for GCN-VE and STAR-FC, and an enumeration of all hyperparameters and their tuned values across all clustering methods.
For supervised clustering, we compute the 256D embeddings for the clustering-train, validation, and test splits.
Methods are trained using the clustering-train embeddings from each dataset with the loss monitored on the validation split for early stopping and hyperparameter tuning.
A hyperparameter search was performed for each of the datasets and we report results using the set of hyperparameter values associated with maximal clustering performance on the validation split.
The model is then applied, inductively, on the test embeddings and ﬁnal performance is reported.
In contrast, the unsupervised methods operate directly on the test embeddings.
A hyperparameter search was also conducted for the unsupervised methods, however, we report results using the set of values associated with maximal clustering performance on the test split.
We admit that optimizing for test performance appears to provide an unfair advantage to the unsupervised methods, however, the supervised methods: (1) have access to a split of data unavailable to the unsupervised methods, (2) have thousands of learnable parameters, and (3) have 12 and 16 hyperparameters for STAR-FC and
GCN-VE, respectively, compared to at-most 4 hyperparameters for the unsupervised methods.
Experimental Results
Using the methodology outlined in Section 1, we conduct extensive experiments on Cars196 and
Stanford Online Products (SOP) across 14 clustering methods.
We measure clustering performance on the test split of both datasets.
The clustering results focus on two metrics: Pairwise ( F P ) [35] and BCubed ( F B ) [1] F-scores.
Because the metrics have different emphases—speciﬁcally,
F P emphasizes ﬁdelity of larger clusters and F B emphasizes ﬁdelity of clusters proportional to their size—we report their harmonic mean when not reporting both, individually.
Figure 1 shows
[Table]
clustering performance for Cars196 (top) and SOP (bottom) across 12 unsupervised methods and the 2 supervised methods.
Appendix B contains tabulated results used to construct the ﬁgure, as well as additional metrics such as normalized mutual information.
There are two key takeaways: (1) using (cid:96) 2normalized embeddings (blue bars) consistently improves clustering performance, sometimes upward of 30%, and (2) the supervised, deep methods are outperformed by unsupervised, shallow methods and are not even among the top-3 performers, contrary to past work reporting consistent state-of-the-art performance.
Among the unsupervised methods, there is no clear best-performing method, however,
HAC performs strongly on both datasets.
As a point of comparison, HAC outperforms GCN-VE and
STAR-FC by 16% and 6% for Cars196, respectively, and 3% and 1% for SOP, respectively.
We posit a relationship between the embedding’s discriminability, measured by Recall@1, and the likelihood of deep clustering methods outperforming baselines.
Table 2 contains Recall@1 for the clustering-train, validation, and test splits of both datasets for unnormalized and (cid:96) 2 -normalized embeddings.
Consistent with Scott et al.
[32], we ﬁnd that (cid:96) 2 -normalization strictly improves discrimination, as it removes intra-class variance.
We emphasize that past conclusions on the deep methods are based on results from embeddings with Recall@1 greater than 0.9.
Cars196 and SOP are more challenging benchmarks due to data limitations and thus exhibit Recall@1 less than 0.8.
The clustering results on Cars196 and SOP highlight a downside of the deep methods, particularly that they become fragile in the presence of uncertainty (i.e., less discriminability) in the embedding space.
Due to the unexpected results for GCN-VE, we further investigated the method to verify our ﬁndings.
We compared our reimplementation of GCN-VE to the open-source version in Table 3.
We conﬁrm that our reimplementation, within reasonable variance, matches the implementation from Yang et al.
Additionally, Appendix D contains an ablation study of GCN-VE, where we ﬁnd that one can recover the majority of GCN-VE’s perform without any deep components.
Conclusions
Many of the innovations in supervised methods for clustering pretrained embeddings are tied to face veriﬁcation.
The methods were benchmarked primarily on face datasets and motivated by enlarging them via pseudo-labeling the vast amounts of unlabeled face data.
Common face datasets (e.g., MS-Celeb1M [12] and MegaFace [18]) are no longer publicly available, with progress reliant on using shared embeddings extracted from pretrained backbones.
Additionally, the impressive improvements of supervised methods are bound to face embeddings, shown to have veriﬁcation and
Recall@1 performance well above 90% and in some cases near ceiling [7].
Finally, results of recent work may implicitly favor the novel, supervised methods based on methodological choices such as the lack of a validation split and hyperparameter tuning based on test performance.
Our goals in conducting an empirical study on clustering pretrained embeddings are to: (1) broaden the scope of supervised methods beyond faces, (2) present an end-to-end pipeline including backbone training along with results for benchmarking future methods, and (3) benchmark the robustness of supervised, deep methods on embeddings with less discriminability.
We do so by presenting benchmarks on 14 clustering methods using two common embedding-learning datasets: Cars196 and
SOP.
Cars196 and SOP are not only from diverse visual domains, but have very different dataset statistics (Table 1) and embedding discriminability (Table 2) compared to common face datasets.
Our clustering results on Cars196 and SOP exemplify the fragility of deep clustering methods, where they underperform unsupervised, shallow methods such as HAC despite having thousands of learnable parameters and three times more hyperparameters.
We attribute the surprising results to (1) our careful, systematic methodology, which corrects for past methodological choices such as not using a validation split and using the test split for hyperparameter tuning and early stopping, and (2) exploring datasets from diverse visual domains with both dataset statistics and embedding discriminability that differs signiﬁcantly from the common face veriﬁcation benchmarks.
Additionally, we encourage practitioners working on embedding clustering to consider (cid:96) 2 -normalization, as we ﬁnd robust improvements to clustering, corroborating results from Scott et al.
A Compute Resources
Each backbone was trained using a single NVIDIA Tesla T4, 16 CPUs, and 64 GB of RAM on
Google Cloud Platform.
Code was implemented using PyTorch v1.6.0 and Python 3.8.2 on Ubuntu 18.04.
For clustering, all methods had access to 16 CPUs, 96 GB of RAM, and a single NVIDIA Tesla T4, when appropriate.
Code was implemented with PyTorch v1.6.0, scikit-learn v0.24.2, FAISS v1.7.1, and Python 3.8.2 on Ubuntu 18.04.
Additionally, we note that both HAC and spectral clustering could be made faster by computing the nearest-neighbor subgraph on GPU instead of CPU.
B Cars196 and SOP Tabulated Results
The tables below contains tabulated clustering results for Cars196 and SOP.
The F P and F B values are used to compute the harmonic means for (cid:96) 2 -normalized embeddings presented in Figure 1.
Cars
Adj
[Table]
NMI
AMI
F
[Table]
F
[Table]
Time to
Number of
Index
Cluster (s)
Clusters
[Table]
MeanShift 0 .0 .0 .0 .0 .
ARO 0 .0 .0 .0 .0 .
DP k -Means 0 .0 .0 .0 .0 .
DP vMF k -Means 0 .0 .0 .0 .0 .
k -Means 0 .0 .0 .0 .0 .
Spherical k -Means 0 .0 .0 .0 .0 .
GMM 0 .0 .0 .0 .0 .
vMF-MM 0 .0 .0 .0 .0 .
HAC 0 .0 .0 .0 .0 .
CDP 0 .0 .0 .0 .0 . 1 ,
Spectral 0 .0 .0 .0 .0 .
GCN-VE 0 .0 .0 .0 .
STAR-FC 0 .0 .0 .0 .
SOP
Adj.
Rand
NMI
AMI
F P
F B
Time to
Number of
Index
Cluster (s)
Clusters
DBSCAN 0 .0 .0 .0 .0 .
MeanShift 0 .0 .0 .0 .0 .
ARO 0 .0 .0 .0 .0 .0 .9 ,
DP k -Means 0 .0 .0 .0 .0 .
DP vMF k -Means 0 .0 .0 .0 .0 .
11 ,k -Means 0 .0 .0 .0 .0 .
Spherical k -Means 0 .0 .0 .0 .0 .
HAC 0 .0 .0 .0 .0 .
CDP 0 .0 .0 .0 .0 .
GCN-VE 0 .
[Table]
C Related Work C.1 Unsupervised Clustering
Clustering is normally characterized as an unsupervised learning task where methods leverage heuristics based on assumptions about the data-generation process or the structure of the data.
Adopting the terminology of Jain [17], we discuss four common classes of heuristics: partition-based, density-based, hierarchical, and graph-theoretic.
Methods using partition-based heuristics [9, 16, 21, 25, 33, 40] have a representation of clusters and decide which data points belong to each cluster, in many cases, using a similarity or distance function.
The most popular of these methods is k -means clustering [25] which estimates empirical cluster centers minimizing within-cluster variance.
Other variants of k -means make assumptions about the geometry of the data, e.g., spherical k -means [16], make assumptions about the scale of the data, e.g., mini-batch k -means [33], or explore alternative formulations that remove the need for a predeﬁned number of clusters, e.g., Dirichlet-process (DP) k -means [21, 40].
Density-based methods assume that clusters represent high-density regions in the feature space and are separated from other clusters by low-density regions.
Popular density-based methods attempt to estimate cluster densities via expectation-maximization, e.g., the Gaussian mixture model (GMM) & the von Mises–Fisher mixture model (vMF-MM) [3], variational inference [10], Parzen windows, e.g., DBSCAN [8] & deep density clustering [22], and kernel density estimation, e.g., MeanShift [5].
Hierarchical methods form clusters by starting either with a singleton cluster containing all data points and iteratively splitting it (i.e., top-down or divisive) or with a cluster for each data point and iteratively merging them (i.e., bottom-up or agglomerative).
The split or merge is decided greedily using a linkage rule until a criteria is satisﬁed such as the total number of clusters.
We experiment with hierarchical agglomerative clustering (HAC) [36] and several popular linkage rules (single, complete, average, and Ward linkage), as well as approximate rank-order (ARO) [29] based on the rank-order linkage metric [47].
The ﬁnal common class of heuristics use graph theoretics.
These methods represent data points as a graph with edges weighted by pairwise similarities.
One of the most popular methods, and the one we employ in our experiments, is spectral clustering [27].
Spectral clustering performs k -means clustering on the eigenvectors of the normalized Laplacian matrix constructed from the graph’s afﬁnity matrix.
C.2 Supervised Clustering
Recent research, instead of relying on heuristics, has proposed deep, supervised methods that inductively cluster.
These methods rely on supervision that indicates which data points belong to the same cluster.
By inductive , we refer to methods that learn a function, for example a function that predicts if two data points should be clustered together, which can then be applied to unseen data.
This is in contrast to the unsupervised clustering methods which lack generalizability from one dataset to another.
Generally, the idea is to leverage local and/or global structure in the feature space to learn what points belong to the same cluster or what pairs of points should be linked.
Clusters can then be formed using simple algorithms such as connected-component labeling.
Consensus-driven propagation (CDP)
[46] uses an ensemble of backbone networks to produce statistics across many k -nearest-neighbor afﬁnity graphs and then trains a mediator network to predict links between data points from which clusters are formed.
Wang et al.
[41] improves link prediction by capturing structure in local afﬁnity graphs directly with graph convolutional networks (GCNs) [19].
A series of GCN-methods followed that incorporate both local and global structure via density information [11] and multi-scale views [44], for example.
Alternatives beyond GCNs have been proposed such as using self-attention via transformers [28, 45] and learning an agglomerative clustering policy with inverse reinforcement learning [15].
Our experiments include results from CDP, as well as two state-of-the-art GCN approaches: GCN-VE [43] and STAR-FC [34].
GCN-VE is a supervised approach that trains two GCNs.
The ﬁrst, GCN-V, estimates a conﬁdence for each data point based on a supervised density measure.
The second,
GCN-E, constructs subgraphs based on the estimated conﬁdences and predicts pairwise links.
The links are used in a tree-deduction algorithm to construct ﬁnal clusters.
STAR-FC trains a single GCN to predict pairwise links, but uses a structure-preserving sampling strategy to train the GCN with both local and global graph structure.
During inference, links are predicted followed by graph parsing and reﬁnement steps to construct clusters.
Consistent with past work, we do not train an ensemble and mediator for CDP, but rather use just the unsupervised label-propagation step to form clusters.
Note that this removes all supervised components of CDP, which is why we classify it as an unsupervised method in Section 2.
D GCN-VE Ablation Study
[Table]
Due to the unexpected results of GCN-VE on Cars196 and SOP, we conducted an ablation study of the method to better understand which components contributed most to the ﬁnal performance.
In general, GCN-VE works by: (1) estimating a conﬁdence for each embedding via the GCN-V graph convolutional network, (2) constructing subgraphs based on the estimated conﬁdences, (3) estimating which edges in the subgraphs are valid connections via the GCN-E graph convolutional network, and (4) running a tree-deduction algorithm to form the ﬁnal clusters.
We consider three ablated variants of GCN-VE, described below.
All that is needed to form clusters is to run tree deduction on a k -nearest-neighbors subgraph with edges pruned by a distance threshold.
We refer to this ﬁrst ablation as Tree Deduction .
Based on empirical analysis from Scott et al.
[32], the embedding (cid:96) 2 -norm is a measure of conﬁdence, thus, we can replace the GCN-V network with the (cid:96) 2 -norm of each embedding.
Additionally, we can replace the GCN-E network simply by considering edges valid if they are connected to higher-conﬁdence embeddings within a distance threshold, and run tree deduction as is.
We refer to this second ablation as Tree Deduction + (cid:96) 2 -norm .
The third ablation, Tree Deduction + (cid:96) 2 -norm + GCN-E , reintroduces GCN-E, but still uses (cid:96) 2 -norm instead of the GCN-V network.
Table 5 measures the clustering performance of the three ablations against GCN-VE for Cars and SOP.
Interestingly, one can recover the majority, if not all, of GCN-VE performance with the
Tree Deduction + (cid:96) 2 -norm variant, corroborating results from Scott et al.
[32] on (cid:96) 2 -norm conveying embedding conﬁdence.
Note that the Tree Deduction + (cid:96) 2 -norm variant has no deep components.
Additionally, the critical component of GCN-VE is not the GCN-E network, but GCN-V, as GCN-E adds no performance gain on top of Tree Deduction + (cid:96) 2 -norm.
E Experimental Details E.1 Backbone
The experimental details mimic Scott et al.
[32] unless explicitly noted.
For completeness, we recount the details below.
E.1.1 Architecture
The backbone network architecture is a ResNet50 [14].
The network is initialized using weights pretrained on ImageNet and all batch-normalization parameters are frozen.
We remove the head of the architecture and add two fully-connected layers, with no activations, directly following the global-average-pooling layer.
The ﬁrst fully-connected layer maps from 2048 units to 256 units, and the second fully-connected layer maps from 256 units to Y units, where Y is the number of classes in the backbone-training split of the dataset.
The embedding dimensionality is thus 256 for all datasets.
For Cars196 and SOP, Y = 49 and Y = 5658 , respectively.
E.1.2 Dataset Augmentation
Cars196.
Data augmentation during training includes: (1) resizing images to 256 × 256, (2) random jittering of brightness, contrast, saturation, and hue with factors in [0.7, 1.3], [0.7, 1.3], [0.7, 1.3], and and 100% of the input size, (4) resizing the ﬁnal cropped images to 224 × 224, and (5) random horizontal ﬂipping and z -score normalization.
Data augmentation during validation and testing includes: (1) resizing images to 256 × 256, (2) center cropping to 224 × 224, and (3) z -score normalization.
The mean and standard deviation for z -score normalization are the same values used for ImageNet.
We match Boudiaf et al.
[4] and sample batches randomly.
SOP.
Data augmentation during training includes: (1) resizing images to 256 × 256, (2) cropping at a random location with random size between 16% and 100% of the input size with the aspect ratio randomly selected in [0.75, 1.33], (3) resizing the ﬁnal cropped images to 224 × 224, and (4) random horizontal ﬂipping and z -score normalization.
Data augmentation during validation and testing includes: (1) resizing images to 256 × 256, (2) center cropping to 224 × 224, and (3) z -score normalization.
The mean and standard deviation for z -score normalization are the same values used for ImageNet.
We match Boudiaf et al.
[4] and sample batches randomly.
E.1.3 Backbone Network Hyperparameters
Models are trained with SGD and either standard or Nesterov momentum.
Based on results from
Scott et al.
[32], we use cosine softmax cross-entropy as the loss.
The validation split’s Recall@1 is monitored throughout training and if it does not improve for 15 epochs, the learning rate is cut in half.
If the validation split’s Recall@1 does not improve for 35 epochs, model training terminates early.
The model parameters are saved for the epoch resulting in the highest validation Recall@1.
The inverse-temperature, β , is parameterized as β = exp( τ ) where τ ∈ R is an unconstrained network parameter learned automatically via gradient descent.
Unlike Scott et al.
[32], τ is optimized using the same learning rate as all other network parameters.
The remaining hyperparameters for training the backbone network are:
• Learning rate • (cid:96) 2 weight decay • Momentum • Nesterov momentum • Initial value of τ
Hyperparameters were optimized with a grid search.
The table below contains the hyperparameter values used for the model achieving the best validation Recall@1 for each dataset.
[Table]
E.
For clustering, the inputs are the 256D embeddings from the penultimate output of the trained backbone network.
The embeddings are (cid:96) 2 -normalized unless explicitly noted otherwise.
To produce the embeddings for clustering, the images were augmented using the testing augmentation for each dataset, which is: (1) resizing images to 256 × 256, (2) center cropping to 224 × 224, and (3) z -score normalization.
E.2.1 GCN-VE Details
We match Yang et al.
[43] and use a one-layer graph convolutional network (GCN) [19] with mean aggregation [44] and ReLU activation for GCN-V, and a similar four-layer graph convolutional network for GCN-E. GCN-V maps the 512D output of the GCN through a fully-connected layer to 512 units, then through a PReLU activation, followed by a ﬁnal fully-connected layer to a single output unit.
The model is trained with mean-squared-error to match an empirically-estimated conﬁdence based on the density of an embedding’s neighborhood.
GCN-E has identical structure to GCN-V except that it contains a four-layer GCN, the GCN output is 256D, and the ﬁrst fully-connected layer has 256 units.
GCN-E is trained with softmax cross-entropy to predict the likelihood that a pair of nearby embeddings belong to the same cluster.
The GCN layers are initialized using
Xavier-uniform for the weights and zero for the biases.
The fully-connected layers are initialized using Kaiming-uniform for both weights and biases.
Both GCN-V and GCN-E are trained with SGD and standard momentum.
If the validation split’s loss does not decrease for 100 epochs, the learning rate is cut in half, and if it does not decrease for 250 epochs, model training terminates early.
The model parameters are saved for the epoch resulting in the lowest validation loss.
E.2.2 STAR-FC Details
We match Shen et al.
[34] and use a one-layer graph convolutional network (GCN) [19] with mean aggregation [44] and ReLU activation.
The GCN output is 1024D which is then passed through a fully-connected layer to 512 units, then through a PReLU activation, followed by a ﬁnal fullyconnected layer to a single output unit.
The model is trained with softmax cross-entropy to predict the likelihood that a pair of embeddings belong to the same cluster.
The GCN layers are initialized using
Xavier-uniform for the weights and zero for the biases.
The fully-connected layers are initialized using Kaiming-uniform for both weights and biases.
STAR-FC is trained with SGD and standard momentum.
If the validation split’s loss does not decrease for 15 epochs, the learning rate is cut in half, and if it does not decrease for 35 epochs, model training terminates early.
The model parameters are saved for the epoch resulting in the lowest validation loss.
E.2.3 Clustering Hyperparameters
For each clustering method, we list the hyperparameters and values for each dataset.
All hyperparameters were optimized with a grid search. k -Means with Random Initialization [25, 33].
The only hyperparameters are k , the number of clusters, and the minibatch size.
A minibatch size of -1 indicates the batch size is equal to the full dataset size.
[Table]
[Table]
Spherical k -Means with Random Initialization [16].
The only hyperparameter is k , the number of clusters.
[Table]
Spherical -Means with -Means+
[Table]
Dirichlet-Process (DP) k -Means [21].
The hyperparameters are:
• λ , the cluster penalty • Initialization strategy, either ‘Global Centroid’ or ‘Random’ • Whether to do an online EM update
[Table]
Dirichlet-Process (DP) von Mises–Fisher (vMF) k
-Means [40].
The hyperparameters are:
• λ , the cluster penalty • Initialization strategy, either ‘Global Centroid’ or ‘Random’ • Whether to do an online EM update
[Table]
Gaussian Mixture Model (GMM).
The hyperparameters are:
• k , the number of clusters • Initialization strategy, either ‘k-Means’ or ‘Random’ • Type of covariance, either ‘Full’ or ‘Diagonal’
GMM was run only on Cars196 due to runtime inefﬁciencies
[Table]
von Mises–Fisher Mixture Model (vMF-MM) [10, 3].
The hyperparameters are:
• k , the number of clusters • Initialization strategy, one of ‘k-Means++’ or ‘Random’, ‘Random-Class’, or ‘RandomOrthonormal’ • Posterior type, either ‘Hard’ or ‘Soft’
vMF-MM was run only on Cars196 due to runtime inefﬁciencies.
[Table]
Hierarchical Agglomerative Clustering (HAC) [36].
The hyperparameters are:
• k , the number of clusters • Afﬁnity metric, one of ‘Euclidean’ or ‘Cosine’ • Linkage criterion, one of ‘Ward’, ‘Complete’, ‘Average’, or ‘Single’ • Number of neighbors for computing the nearest-neighbor afﬁnity subgraph
[Table]
metric
[Table]
Approximate Rank Order (ARO) [29].
The hyperparameters are:
• Number of neighbors for computing the nearest-neighbors subgraph • Threshold on rank-order distances • Distance metric, one of ‘Euclidean’ or ‘Cosine’
Note that for (cid:96) 2 -normalized embeddings, both Euclidean and cosine distance metrics produce identical results because the rank-order is unchanged.
[Table]
Density-Based Spatial Clustering of Applications with Noise (DBSCAN) [8].
The hyperparameters are:
• (cid:15) , the maximum distance between two embeddings in the same neighborhood • Minimum number of samples in neighborhood for core point • Distance metric, either ‘Euclidean’ or ‘Cosine’ • Whether a sparse matrix is used for nearest-neighbor subgraph
[Table]
MeanShift [5].
The hyperparameters are:
• Bandwidth for RBF kernel • Minimum bin frequency • Whether to cluster all embeddings including orphans Bandwidth
[Table]
Spectral Clustering [27].
The hyperparameters are:
• k , the number of clusters • Method for constructing the afﬁnity matrix • Number of neighbors for computing the nearest-neighbor afﬁnity subgraph • Number of components for the spectral embedding
Spectral clustering was run only on Cars196 due to runtime inefﬁciencies.
[Table]
Consensus-Driven Propagation (CDP)
The hyperparameters are:
• Number of neighbors for computing the nearest-neighbor subgraph
• Threshold on cosine similarity in the nearest-neighbor subgraph for pruning edges • Threshold step size • Max cluster size
[Table]
Tree Deduction [43].
The hyperparameters are the number of neighbors for computing the nearestneighbor subgraph and a threshold on cosine similarity for pruning edges during tree deduction.
[Table]
Tree Deduction with Embedding (cid:96) 2 -Norm Conﬁdence.
The hyperparameters are the number of neighbors for computing the nearest-neighbor subgraph and a threshold on cosine similarity for pruning edges during tree deduction.
[Table]
Embedding (cid:96) 2 -Norm Conﬁdence + GCN-E + Tree Deduction.
The hyperparameters are:
• Number of neighbors for computing the adjacency matrix (N) • Threshold on cosine similarity for pruning edges in the adjacency matrix ( τ 1 ) • Ignore ratio (IR) • Threshold on cosine similarity for pruning edges during tree deduction ( τ 2 ) • Number of units in the hidden layers of the network (H) • Learning rate (LR) • Momentum (MOM) • (cid:96) 2 weight decay ( (cid:96) 2 )
• Dropout probability (D)
N τ
[Table]
IR
[Table]
H
[Table]
D
[Table]
GCN-VE [43].
GCN-VE has two sub-networks, GCN-V and GCN-E. We present the hyperparameters associated with each sub-network independently.
The hyperparameters for GCN-V are:
• Number of neighbors for computing the adjacency matrix (N) • Threshold on cosine similarity for pruning edges in the adjacency matrix ( τ ) • Number of units in the hidden layers of the network (H) • Learning rate (LR) • Momentum (MOM) • (cid:96) 2 weight decay ( (cid:96) 2 ) •
Dropout probability (D)
[Table]
The hyperparameters for GCN-E are:
• Number of neighbors for computing the adjacency matrix (N) • Threshold on cosine similarity for pruning edges in the adjacency matrix ( τ 1 ) • Ignore ratio (IR) • Threshold on cosine similarity for pruning edges during tree deduction ( τ 2 ) • Number of units in the hidden layers of the network (H) • Learning rate (LR) • Momentum (MOM) • (cid:96) 2 weight decay ( (cid:96) 2 )
• Dropout probability (D)
[Table]
[Table]
The hyperparameters are:
• Number of neighbors for computing the adjacency matrix (N) • Threshold on cosine similarity for pruning edges in the adjacency matrix ( τ ) • Number of seed clusters (SC) • Number of nearest clusters (NC) • Number of random clusters (RC)
• Random node proportion (RNP) •
Prune threshold (P) • Intimacy threshold (I) • Number of units in the hidden layers of the network (H) • Learning rate (LR)
• Momentum (MOM) • (cid:96) 2 weight decay ( (cid:96) 2 )
[Table]