Original: U NICOM : U NIVERSAL AND C OMPACT R EPRESENTATION L EARNING FOR I MAGE R ETRIEVALAnonymous authors
Revised: U NICOM : U NIVERSAL AND C OMPACT R EPRESENTATION L EARNING FOR I MAGE R ETRIEVAL

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Modern image retrieval methods typically rely on fine-tuning pre-trained encoders to extract image-level descriptors.
Revised: Modern image retrieval methods typically rely on fine-tuning pre-trained encoders to extract image-level descriptors.

Original: However, the most widely used models are pre-trained on ImageNet-1K with limited classes.
Revised: However, the most widely used models are pre-trained on ImageNet-1K with limited classes.

Original: The pre-trained feature representation is therefore not universal enough to generalize well to the diverse open-world classes.
Revised: The pre-trained feature representation is therefore not universal enough to generalize well to the diverse open-world classes.

Original: In this paper, we first cluster the large-scale LAION 

Revised: In this paper, we first cluster the large-scale LAION

Original: M dataset into one million pseudo classes based on the joint textual and visual features extracted by the CLIP model.
Revised: M dataset into one million pseudo classes based on the joint textual and visual features extracted by the CLIP model.

Original: Due to the confusion of label granularity, the automatically clustered dataset inevitably contains heavy inter-class conflict.
Revised: Due to the confusion of label granularity, the automatically clustered dataset inevitably contains heavy inter-class conflict.

Original: To alleviate such conflict, we randomly select partial inter-class prototypes to construct the marginbased softmax loss.
Revised: To alleviate such conflict, we randomly select partial inter-class prototypes to construct the margin-based softmax loss.

Original: To further enhance the low-dimensional feature representation, we randomly select partial feature dimensions when calculating the similarities between embeddings and class-wise prototypes.
Revised: To further enhance the low-dimensional feature representation, we randomly select partial feature dimensions when calculating the similarities between embeddings and class-wise prototypes.

Original: The dual random partial selections are with respect to the class dimension and the feature dimension of the prototype matrix,respectively, making the classification conflict-robust and the feature embedding compact.
Revised: The dual random partial selections are with respect to the class dimension and the feature dimension of the prototype matrix, making the classification conflict-robust and the feature embedding compact.

Original: Our method  outperforms state-of-the-art unsupervised and supervised image retrieval approaches on multiple benchmarks with substantial improvement under different dimension constraints.
Revised: Our method significantly outperforms stateof-the-art unsupervised and supervised image retrieval approaches on multiple benchmarks.

Original: Pre-processed data, training code, and pre-trained models will be released to reproduce our results.
Revised: The code and pre-trained models are released to facilitate future research https://github.com/deepglint/unicom .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Modern image retrieval methods (Lim et al., 2022; Roth et al., 2022; Kim et al., 2022; Ermolov et al., 2022; Patel et al., 2022) can be roughly decomposed into two major components: (1) the encoder (e.g., Convolutional Neural Networks (Szegedy et al., 2015;
Revised: Modern image retrieval methods (Lim et al., 2022; Roth et al., 2022; Kim et al., 2022; Ermolov et al., 2022; Patel et al., 2022) can be roughly decomposed into two major components: (1) the encoder (e.g., Convolutional Neural Networks (Szegedy et al., 2015;

Original: He et al., 2016) or Vision Transformer (Touvron et al., 2021; Dosovitskiy et al., 2021)) mapping the image to its compact representation and (2) the loss function (Musgrave et al., 2020) grouping the representations of similar objects while pushing away representations of dissimilar objects in the embedding space.
Revised: He et al., 2016) or Vision Transformer (Touvron et al., 2021; Dosovitskiy et al., 2021)) mapping the image to its compact representation and (2) the loss function (Musgrave et al., 2020) grouping the representations of similar objects while pushing away representations of dissimilar objects in the embedding space.

Original: To train the encoder, networks pre-trained on crowd-labeled datasets (e.g., ImageNet (Deng et al., 2009)) are widely used for fine-tuning (Wang et al., 2019; Kim et al., 2021).
Revised: To train the encoder, networks pre-trained on crowd-labeled datasets (e.g., ImageNet (Deng et al., 2009)) are widely used for fine-tuning (Wang et al., 2019; Kim et al., 2021).

Original: However, ImageNet only contains 1, pre-defined object classes.
Revised: However, ImageNet only contains 1, pre-defined object classes.

Original: The feature representation learned from ImageNet is not universal enough to generalize to diverse open-world objects.
Revised: The feature representation learned from ImageNet is not universal enough to generalize to diverse open-world objects.

Original: Even though fully supervised pre-training can benefit from a strong semantic learning signal for each training example, supervised learning is not scalable because manual annotation of large-scale training data is time-consuming, costly, and even infeasible.
Revised: Even though fully supervised pre-training can benefit from a strong semantic learning signal for each training example, supervised learning is not scalable because manual annotation of large-scale training data is time-consuming, costly, and even infeasible.

Original: By contrast, self-supervised pre-training methods (He et al., 2020; 2022; Radford et al., 2021;
Revised: By contrast, self-supervised pre-training methods (He et al., 2020; 2022; Radford et al., 2021;

Original: Jia et al., 2021) can be easily scaled to billions of unlabeled examples by designing an appropriate pretext task, such as solving jigsaw puzzles (Noroozi & Favaro, 2016), invariant mapping (Chen & He, 2021), and image-text matching (Radford et al., 2021;
Revised: Jia et al., 2021) can be easily scaled to billions of unlabeled examples by designing an appropriate pretext task, such as solving jigsaw puzzles (Noroozi & Favaro, 2016), invariant mapping (Chen & He, 2021), and image-text matching (Radford et al., 2021;

Original: Jia et al., 2021).
Revised: Jia et al., 2021).

Original: Among them, CLIP (Radford et al., 2021) has recently demonstrated success across various downstream tasks (e.g., image retrieval and classification) due to superior feature representation empowered by image-text contrastive learning.
Revised: Among them, CLIP (Radford et al., 2021) has recently demonstrated success across various downstream tasks (e.g., image retrieval and classification) due to superior feature representation empowered by image-text contrastive learning.

Original: Specifically, CLIP aligns the visual and textual signals of each instance into a unified semantic space by cross-modal instancediscrimination.
Revised: Specifically, CLIP aligns the visual and textual signals of each instance into a unified semantic space by cross-modal instance

Original:  Nevertheless, the instance discrimination method used by CLIP can hardly encode the semantic structure of training data, because instance-wise contrastive learning always treats two samples as a negative pair if they are from different instances, regardless of their semantic similarity.
Revised: [Figure] discrimination. Nevertheless, the instance discrimination method used by CLIP can hardly encode the semantic structure of training data, because instance-wise contrastive learning always treats two samples as a negative pair if they are from different instances, regardless of their semantic similarity.

Original: When thousands of instances are selected into the training batch to form the contrastive loss, negative pairs that share similar semantics will be undesirably pushed apart in the embedding space.
Revised: When thousands of instances are selected into the training batch to form the contrastive loss, negative pairs that share similar semantics will be undesirably pushed apart in the embedding space.

Original: [Figure] To handle the limitations of instance discrimination, cluster discrimination has been proposed for deep unsupervised learning through jointly learning image embeddings and cluster assignments.
Revised: To handle the limitations of instance discrimination, cluster discrimination has been proposed for deep unsupervised learning through jointly learning image embeddings and cluster assignments.

Original: Learning representations with clusters will pull similar instances together, which is beneficial for capturing semantic structures in data.
Revised: Learning representations with clusters will pull similar instances together, which is beneficial for capturing semantic structures in data.

Original: DeepCluster (Caron et al., 2018) performs iterative clustering by k-means and classification by cross-entropy loss, while SeLa (Asano et al., 2020) proposes to solve an optimal transport problem for balanced assignment.
Revised: DeepCluster (Caron et al., 2018) performs iterative clustering by k-means and classification by cross-entropy loss, while SeLa (Asano et al., 2020) proposes to solve an optimal transport problem for balanced assignment.

Original: However, both DeepCluster and SeLa need labels to be assigned offline in a batch mode with representations of all instances.
Revised: However, both DeepCluster and SeLa need labels to be assigned offline in a batch mode with representations of all instances.

Original: To reduce the cost of batch mode clustering, ODC (Zhan et al., 2020), SwAV (Caron et al., 2020), and CoKe (Qian et al., 2022) apply online clustering to avoid the multiple iterations over the entire dataset.
Revised: To reduce the cost of batch mode clustering, ODC (Zhan et al., 2020), SwAV (Caron et al., 2020), and CoKe (Qian et al., 2022) apply online clustering to avoid the multiple iterations over the entire dataset.

Original: Despite improved efficiency, the online clustering method still suffers from the collapsing problem (i.e., a dominant cluster includes instances from multiple classes or most of the instances).
Revised: Despite improved efficiency, the online clustering method still suffers from the collapsing problem (i.e., a dominant cluster includes instances from multiple classes or most of the instances).

Original: In this paper, we aim at boosting the semantic embedding power of the CLIP model by introducing a novel cluster discrimination approach.
Revised: In this paper, we aim at boosting the semantic embedding power of the CLIP model by introducing a novel cluster discrimination approach.

Original: We first conduct one step of off-line clustering by using the image and text features from a pre-trained CLIP model (Radford et al., 2021).
Revised: We first conduct one step of off-line clustering by using the image and text features from a pre-trained CLIP model (Radford et al., 2021).

Original: Due to the limited discrimination power of the CLIP model, the pseudo clusters contain heavy inter-class conflict.
Revised: Due to the limited discrimination power of the CLIP model, the pseudo clusters contain heavy inter-class conflict.

Original: Instead of optimizing the cluster assignment (Qian et al., 2022), we focus on how to robustly train a classifier on the automatically clustered large-scale data.
Revised: Instead of optimizing the cluster assignment (Qian et al., 2022), we focus on how to robustly train a classifier on the automatically clustered large-scale data.

Original: More specifically, we explore two random selections on the prototype matrix
Revised: More specifically, we explore two random selections on the prototype matrix

Original: W ∈ R d × k when preparing the classification loss (as illustrated in Fig. 1(b)).
Revised: W ∈ R d × k when preparing the classification loss (as illustrated in Fig. 1(b)).

Original: The first one is with respect to the class dimension ( k ), and only part of negative prototypes are selected for inter-class comparisons, which helps alleviate inter-class conflict.
Revised: The first one is with respect to the class dimension ( k ), and only part of negative prototypes are selected for inter-class comparisons, which helps alleviate inter-class conflict.

Original: The second one is with respect to the feature dimension ( d ), and only part of features are randomly selected to construct the classification loss,  making feature representation compact for the efficient image retrieval task.
Revised: The second one is with respect to the feature dimension ( d ), and only part of features are randomly selected to construct the classification loss, enhancing the representation power of each neuron and making feature representation compact for the efficient image retrieval task.

Original: Concurrent with our work, partial selection mechanisms along class and feature are separately proposed in (An et al., 2021) and (Xu et al., 2022) to accelerate model training and enhance locally distinguishable features.
Revised: Concurrent with our work, partial selection mechanisms along class and feature are separately proposed in (An et al., 2021; 2022) and (Xu et al., 2022) to accelerate model training and enhance locally distinguishable features.

Original: Both of their experiments are conducted on cleaned face recognition datasets.
Revised: Both of their experiments are conducted on cleaned face recognition datasets.

Original: By contrast, we target at universal and compact representation learning from automatically clustered large-scale data.
Revised: By contrast, we target at universal and compact representation learning from automatically clustered large-scale data.

Original: The main contributions of our paper are the following
Revised: The main contributions of our paper are the following:

Original: : • We propose a novel cluster discrimination method for universal and compact representation learning.
Revised: • We propose a novel cluster discrimination method for universal and compact representation learning.

Original: In the clustering step, we employ both image and text features from the pre-trained CLIP model.
Revised: In the clustering step, we employ both image and text features from the pre-trained CLIP model.

Original: In the discrimination step, we explore two random selections along class and feature, which can potentially alleviate inter-class conflict and improve the feature compactness, respectively.
Revised: In the discrimination step, we explore two random selections along class and feature, which can potentially alleviate inter-class conflict and improve the feature compactness, respectively.

Original: • For both zero-shot learning tasks (e.g., linear probe and unsupervised image retrieval) and transfer learning tasks (e.g.,  supervised image retrieval), the proposed random negative prototype selection for conflict-robust cluster discrimination can significantly boost the representation power of the instance discrimination based model (e.g., CLIP). • For dimension constrained image retrieval, the proposed random feature selection can effectively enhance the representation power of each neuron, and the sub-features are able to achieve state-of-the-art performance across multiple benchmarks.
Revised: • For both zero-shot learning tasks (e g., linear probe and unsupervised image retrieval) and transfer learning tasks (e.g., ImageNet-1K classification and supervised image retrieval), the proposed random negative prototype selection for conflict-robust cluster discrimination can significantly boost the representation power compared to the instance discrimination based model (e.g., CLIP)

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Visual Model Pre-training.
Revised: Visual Model Pre-training.

Original: Model pre-training for visual recognition can be categorized into three main groups: (1) supervised pre-training on datasets with manually annotated class labels (e.g.,
Revised: Model pre-training for visual recognition can be categorized into three main groups: (1) supervised pre-training on datasets with manually annotated class labels (e.g.,

Original: ImageNet-1K/-21K (Deng et al., 2009) and JFT-300M/-3B (Dosovitskiy et al., 2021; Zhai et al., 2022)), (2) weakly-supervised pre-training by using hashtag (Mahajan et al., 2018; Singh et al., 2022) or text descriptions (Radford et al., 2021;
Revised: ImageNet-1K/-21K (Deng et al., 2009) and JFT-300M/-3B (Dosovitskiy et al., 2021; Zhai et al., 2022)), (2) weakly-supervised pre-training by using hashtag (Mahajan et al., 2018; Singh et al., 2022) or text descriptions (Radford et al., 2021;

Original: Jia et al., 2021), and (3) unsupervised pre-training
Revised: Jia et al., 2021), and (3) unsupervised pre-training

Original: (Chen et al., 2020;
Revised: (Chen et al., 2020;

Original: He et al., 2020; Caron et al., 2018).
Revised: He et al., 2020; Caron et al., 2018).

Original: Since supervised pre-training relies on expensive manual annotations, we focus on annotation-free pre-training which has the advantages of being easily scaled to billions of training images and being able to learn universal feature representations for downstream tasks.
Revised: Since supervised pre-training relies on expensive manual annotations, we focus on annotation-free pre-training which has the advantages of being easily scaled to billions of training images and being able to learn universal feature representations for downstream tasks.

Original: Instance and Cluster Discrimination.
Revised: Instance and Cluster Discrimination.

Original: Instance discrimination (Chen et al., 2020;
Revised: Instance discrimination (Chen et al., 2020;

Original: He et al., 2020;
Revised: He et al., 2020;

Original: Radford et al., 2021) is realized with a contrastive loss which targets at pulling closer samples from the same instance while pushing away samples from different instances.
Revised: Radford et al., 2021) is realized with a contrastive loss which targets at pulling closer samples from the same instance while pushing away samples from different instances.

Original: Despite the impressive performance, instance-wise contrastive learning can not capture the semantic information from the training data because it is trained to ignore the similarity between different instances.
Revised: Despite the impressive performance, instance-wise contrastive learning can not capture the semantic information from the training data because it is trained to ignore the similarity between different instances.

Original: Cluster discrimination (Caron et al., 2018; Zhan et al., 2020; Li et al., 2020a) is processed with iterative steps: the clustering step to assign pseudo class labels for each sample, and then the classification step to map each sample to its assigned label.
Revised: Cluster discrimination (Caron et al., 2018; Zhan et al., 2020; Li et al., 2020a) is processed with iterative steps: the clustering step to assign pseudo class labels for each sample, and then the classification step to map each sample to its assigned label.

Original: Since one cluster has more than one instance, learning representations with clusters will gather similar instances together, which can explore potential semantic structures in data.
Revised: Since one cluster has more than one instance, learning representations with clusters will gather similar instances together, which can explore potential semantic structures in data.

Original: As a representative work, DeepCluster (Caron et al., 2018) adopts a standard k-means for clustering, but it contains degenerate solutions.
Revised: As a representative work, DeepCluster (Caron et al., 2018) adopts a standard k-means for clustering, but it contains degenerate solutions.

Original: To this end, recent research work (Asano et al., 2020; Caron et al., 2020; Qian et al., 2022) focuses on improving the label assignment during clustering but employs a standard cross-entropy loss during discrimination.
Revised: To this end, recent research work (Asano et al., 2020; Caron et al., 2020; Qian et al., 2022) focuses on improving the label assignment during clustering but employs a standard cross-entropy loss during discrimination.

Original: In this paper, we only employ one step of off-line clustering but design a robust classifier to achieve good feature representation when training on the automatically clustered large-scale data.
Revised: In this paper, we only employ one step of off-line clustering but design a robust classifier to achieve good feature representation when training on the automatically clustered large-scale data.

Original: Image Retrieval.
Revised: Image Retrieval.

Original: Image retrieval task typically relies on fine-tuning pre-trained visual models (Szegedy et al., 2015;
Revised: Image retrieval task typically relies on fine-tuning pre-trained visual models (Szegedy et al., 2015;

Original: He et al., 2016; Dosovitskiy et al., 2021) and can be divided into two learning categories: supervised and unsupervised metric learning.
Revised: He et al., 2016; Dosovitskiy et al., 2021) and can be divided into two learning categories: supervised and unsupervised metric learning.

Original: For supervised metric learning, pair-wise loss (Hadsell et al., 2006; Schroff et al., 2015; Sohn, 2016) and cross-entropy loss (Zhai & Wu, 2019;
Revised: For supervised metric learning, pair-wise loss (Hadsell et al., 2006; Schroff et al., 2015; Sohn, 2016) and cross-entropy loss (Zhai & Wu, 2019;

Original: Deng et al., 2019; Sun et al., 2020; Qian et al., 2019) are extensively studied and recent bench-marking results (Musgrave et al., 2020) indicate that the margin-based softmax loss (e.g., ArcFace (Deng et al., 2019)) can achieve state-of-the-art performance.
Revised: Deng et al., 2019; Sun et al., 2020; Qian et al., 2019) are extensively studied and recent bench-marking results (Musgrave et al., 2020) indicate that the margin-based softmax loss (e.g., ArcFace (Deng et al., 2019)) can achieve state-of-the-art performance.

Original: For unsupervised metric learning, pseudo labeling methods are employed to discover pseudo classes by applying k-means clustering (Kan et al., 2021;
Revised: For unsupervised metric learning, pseudo labeling methods are employed to discover pseudo classes by applying k-means clustering (Kan et al., 2021;

Original: Li et al., 2020b), hierarchical clustering (Yan et al., 2021), random walk (Iscen et al., 2018), and class-equivalence relations (Kim et al., 2022) to unlabeled training data.
Revised: Li et al., 2020b), hierarchical clustering (Yan et al., 2021), random walk (Iscen et al., 2018), and class-equivalence relations (Kim et al., 2022) to unlabeled training data.

Original: In this paper, we focus on universal and compact feature embedding for both unsupervised and supervised image retrieval task.
Revised: In this paper, we focus on universal and compact feature embedding for both unsupervised and supervised image retrieval task.

Original:  M ETHODOLOGY 3. P RELIMINARIES OF I NSTANCE AND C LUSTER D ISCRIMINATION
Revised:  M ETHODOLOGY 3. P RELIMINARIES OF I NSTANCE AND C LUSTER D ISCRIMINATION

Original: Given a training set X = { x 1 , x 2 , ..., x n } including n images, feature representation learning aims at learning a function f that maps images X to embeddings E
Revised: Given a training set X = { x 1 , x 2 , ..., x n } including n images, feature representation learning aims at learning a function f that maps images X to embeddings E

Original: = { e 1 , e 2 , ..., e n } with e i = f
Revised: = { e 1 , e 2 , ..., e n } with e i = f

Original: ( x i ) , such that embeddings can describe the similarities between different images.
Revised: ( x i ) , such that embeddings can describe the similarities between different images.

Original: Instance discrimination achieves this objective by optimizing a contrastive loss function defined as:
Revised: Instance discrimination achieves this objective by optimizing a contrastive loss function defined as:

Original: [Equation]
Revised: [Equation]

Revised: [Figure]

Original: where e i and e ′ i are positive embeddings of the instance i , and e ′ j consists of one positive embedding of i and its m negative embeddings from other instances.
Revised: where e i and e ′ i are positive embeddings of the instance i , and e ′ j consists of one positive embedding of i and its m negative embeddings from other instances.

Original: By contrast, cluster discrimination for representation learning consists of two main phases: clustering and discrimination.
Revised: By contrast, cluster discrimination for representation learning consists of two main phases: clustering and discrimination.

Original: The clustering step assigns each instance a pseudo class label that will be subsequently used as supervision to train a classifier in the discrimination step.
Revised: The clustering step assigns each instance a pseudo class label that will be subsequently used as supervision to train a classifier in the discrimination step.

Original: Following this, automatic clustering on the features e i = f
Revised: Following this, automatic clustering on the features e i = f

Original: ( x i ) is first performed to obtain k clusters and the centroid w i is viewed as the prototype of i -th cluster.
Revised: ( x i ) is first performed to obtain k clusters and the centroid w i is viewed as the prototype of i -th cluster.

Original: Then, the training data { x i } n i =1 are partitioned into k[Figure] classes represented by prototypes W = { w i } ki =1 .
Revised: Then, the training data { x i } ni =1 are partitioned into k classes represented by prototypes W = { w i } ki =1 .

Original: With pseudo labels and centroids obtained from the clustering step, cluster discrimination can be implemented by optimizing a standard softmax classification loss as:
Revised: With pseudo labels and centroids obtained from the clustering step, cluster discrimination can be implemented by optimizing a standard softmax classification loss as:

Original: [Equation]
Revised: [Equation]

Original: where e i is the embedding of the image x i and x i belongs to the class represented by w i .
Revised: where e i is the embedding of the image x i and x i belongs to the class represented by w i .

Original: By comparing Eq. 1 and Eq. 2, we can observe the difference that instance discrimination employs an augmented feature e ′ i to calculate the similarities while cluster discrimination uses a prototype w i .
Revised: By comparing Eq. 1 and Eq. 2, we can observe the difference that instance discrimination employs an augmented feature e ′ i to calculate the similarities while cluster discrimination uses a prototype w i .

Original: 3. M ULTIMODAL C LUSTERING
Revised: 3. M ULTIMODAL C LUSTERING

Original: In this paper, we focus on the standard clustering algorithm, k -means, which takes a set of vectors as input and clusters them into k distinct groups based on the nearest neighbor criterion.
Revised: In this paper, we focus on the standard clustering algorithm, k -means, which takes a set of vectors as input and clusters them into k distinct groups based on the nearest neighbor criterion.

Original: To seek a better representation, we combined the image and text features produced by the pre-trained CLIP model (Radford et al., 2021) due to their mutual complementary nature.
Revised: To seek a better representation, we combined the image and text features produced by the pre-trained CLIP model (Radford et al., 2021) due to their mutual complementary nature.

Original: The clustering step jointly learns a d × k centroid matrix W and the cluster assignments
Revised: The clustering step jointly learns a d × k centroid matrix W and the cluster assignments

Original: y i of each image x
Revised: y i of each image x

Original: i by solving the following problem:
Revised: i by solving the following problem:

Original: [Equation]
Revised: [Equation]

Original: where f ( x i ) is the image feature embedding by the image encoder f and f ′ ( x ′ i ) is the text feature embedding by the text encoder f ′ , Φ is a feature ensemble function,
Revised: where f ( x i ) is the image feature embedding by the image encoder f and f ′ ( x ′ i ) is the text feature embedding by the text encoder f ′ , Φ is a feature ensemble function,

Original: W ∈ R d × k is the centroid matrix, y i in { 0 , 1 } k is a single label assignment constrained by y ⊤ i 1 k = 1 , and 1 k is 1-vector of size k .
Revised: W ∈ R d × k is the centroid matrix, y i in { 0 , 1 } k is a single label assignment constrained by y ⊤ i 1 k = 1 , and 1 k is 1-vector of size k .

Original: In this work, we employ the simplest feature ensemble function, that is, averaging the image and text features, as the CLIP model provides an aligned visual-textual representation.
Revised: In this work, we employ the simplest feature ensemble function, that is, averaging the image and text features, as the CLIP model provides an aligned visual-textual representation.

Original: Considering that iterative clustering and discrimination are time-consuming, we only employ one step of off-line clustering in this work.
Revised: Considering that iterative clustering and discrimination are time-consuming, we only employ one step of off-line clustering in this work.

Original: Aided by the efficient feature quantization (Johnson et al., 2019), the large-scale LAION 400M dataset can be clustered within 10 minutes using the embedded image and text features.
Revised: Aided by the efficient feature quantization (Johnson et al., 2019), the large-scale LAION 400M dataset can be clustered within 10 minutes using the embedded image and text features.

Original: The only hyper-parameter we consider here is the cluster number k .
Revised: The only hyper-parameter we consider here is the cluster number k .

Original: Even though the clustering step is straightforward, the automatically clustered large-scale dataset inevitably  confronts inter-class conflict due to multi-label signals in one image and specific definition of class granularity (as illustrated in Fig. 3).
Revised: Even though the clustering step is straightforward, the automatically clustered large-scale dataset inevitably [Figure] confronts inter-class conflict due to multi-label signals in one image and specific definition of class granularity (as illustrated in Fig. 3).

Original: For instance, the bird pair in Fig.
Revised: For instance, the bird pair in Fig.

Original: 3(b) is clustered into wedding pair, which will be conflicted with another specific category of bird.
Revised: 3(b) is clustered into wedding pair, which will be conflicted with another specific category of bird.

Original: In addition, the close-up capture of wedding flowers in Fig. 3(a) also exists in the class of wedding place, where flowers are the most popular decoration.
Revised: In addition, the close-up capture of wedding flowers in Fig. 3(a) also exists in the class of wedding place, where flowers are the most popular decoration.

Original: 3. CONFLICT - ROBUST AND R EPRESENTATION - COMPACT D ISCRIMINATION
Revised: 3. CONFLICT - ROBUST AND R EPRESENTATION - COMPACT D ISCRIMINATION

Original: After the clustering step, we can employ a vanilla classification loss (e.g., softmax) to learn the feature representation.
Revised: After the clustering step, we can employ a vanilla classification loss (e.g., softmax) to learn the feature representation.

Original: For the softmax loss given in Eq. 2, the derivatives to a class-wise prototype w j ∈ R d[Figure] and a sample embedding feature e i ∈ R
Revised: For the softmax loss given in Eq. 2, the derivatives to a class-wise prototype w j ∈ R d and a sample embedding feature e i ∈ R

Original: d are:
Revised: d are:

Original: [Equation]
Revised: [Equation]

Original: where b is the batch size, k is the class number, and p ij =
Revised: where b is the batch size, k is the class number, and p ij =

Original: e w Tj e i / (cid:80) k
Revised: e w Tj e i / (cid:80) kl

Original: [Equation] =1 e w Tl e i is the normalized probability of the sample embedding e i belonging to the prototype w j , 1 ( · ) is the indicator function which is 1 when the statement is true and 0{+
+} otherwise.
Revised: =1 e w Tl e i is the normalized probability of the sample embedding e i belonging to the prototype w j , 1 ( · ) is the indicator function which is 1 when the statement is true and {+
+} otherwise.

Original: In Eq.
Revised: In Eq.

Original: 4, the derivative of the prototype is a “weighted sum” over sample features from the mini-batch, and the derivative of the sample feature is a “weighted sum” over the prototypes of all classes.
Revised: 4, the derivative of the prototype is a “weighted sum” over sample features from the mini-batch, and the derivative of the sample feature is a “weighted sum” over the prototypes of all classes.

Original: If conflicted classes exist, wrong gradient signals from these conflicted negative prototypes will affect the update of model parameters.
Revised: If conflicted classes exist, wrong gradient signals from these conflicted negative prototypes will affect the update of model parameters.

Original: To this end, we propose a random negative prototype selection to efficiently construct a negative prototype subset from the entire negative prototypes.
Revised: To this end, we propose a random negative prototype selection to efficiently construct a negative prototype subset from the entire negative prototypes.

Original: Therefore, the derivative to a sample embedding feature is:
Revised: Therefore, the derivative to a sample embedding feature is:

Original: [Equation]
Revised: [Equation]

Original: where p + and w + denote the probability and prototype of the positive class, p − j and w − j refer to negative probabilities and prototypes, S is a subset of all negative classes and one positive class, | S | =
Revised: where p + and w + denote the probability and prototype of the positive class, p − j and w − j refer to negative probabilities and prototypes, S is a subset of all negative classes and one positive class, | S | =

Original: k ∗ r 1 , and r 1 ∈ [0 , 1] is the sampling ratio.
Revised: k ∗ r 1 , and r 1 ∈ [0 , 1] is the sampling ratio.

Original: Even though all class-wise prototypes are still maintained throughout the whole training process, only positive prototypes and a subset of negative prototypes are selected and updated in each iteration.
Revised: Even though all class-wise prototypes are still maintained throughout the whole training process, only positive prototypes and a subset of negative prototypes are selected and updated in each iteration.

Original: Therefore, the inter-class conflict will be reduced as the possibility of sampling a conflict negative prototype is directly decreased by r 1 .
Revised: Therefore, the inter-class conflict will be reduced as the possibility of sampling a conflict negative prototype is directly decreased by r 1 .

Original: To achieve a compact representation for efficient image retrieval, previous methods (Babenko &
Revised: To achieve a compact representation for efficient image retrieval, previous methods (Babenko &

Original: Lempitsky, 2015; Tolias et al., 2016) adopt Principal Component Analysis (PCA) on an independent set for dimension reduction.
Revised: Lempitsky, 2015; Tolias et al., 2016) adopt Principal Component Analysis (PCA) on an independent set for dimension reduction.

Original: To reduce the descriptor dimension to d ′ , only eigenvectors corresponding to d ′ largest eigenvalues are retained.
Revised: To reduce the descriptor dimension to d ′ , only eigenvectors corresponding to d ′ largest eigenvalues are retained.

Original: However, the representation produced by PCA is sub-optimal because it is a post-processing step detached from the target task.
Revised: However, the representation produced by PCA is sub-optimal because it is a post-processing step detached from the target task.

Original: To this end, we propose a feature approximation strategy by randomly selectingd ′ subspace features to construct the classification loss:
Revised: To this end, we propose a feature approximation strategy by randomly selecting subspace features to construct the classification loss:

Original: [Equation]
Revised: [Equation]

Original:  where Γ t ∈ { 0 , 1 } d is a random binary vector at the iteration step of t , the non-zero element ratio of Γ t is r 2 ∈ [0 , 1] , ⊙ denotes element-wise product.
Revised: 6) where Γ t ∈ { 0 , 1 } d is a random binary vector at the iteration step of t , the non-zero element ratio of Γ t is r 2 ∈ [0 , 1] , ⊙ denotes element-wise product.

Original: Different from the well-known regularization technique, Dropout (Srivastava et al., 2014), our random feature selection Γ t is same for all training samples within the mini-batch.
Revised: Different from the well-known regularization technique, Dropout (Srivastava et al., 2014), our random feature selection Γ t is same for all training samples within the mini-batch.

Original: Γ t is applied to both feature e i and prototypes w j , thus the dimension of derivatives in Eq.
Revised: Γ t is applied to both feature e i and prototypes w j , thus the dimension of derivatives in Eq.

Original: 4 decreases to d ′ .
Revised: 4 decreases to d ′ .

Original: By contrast, Dropout is independently applied to each individual feature e i within the mini-batch by setting a specific ratio r 3 ∈ [0 , 1] of neurons to 0 and enlarging the rest neurons by 1 / (1 − r 3 ) .
Revised: By contrast, Dropout is independently applied to each individual feature e i within the mini-batch by setting a specific ratio r 3 ∈ [0 , 1] of neurons to 0 and enlarging the rest neurons by 1 / (1 − r 3 ) .

Original: The dimension of derivatives in Eq.
Revised: The dimension of derivatives in Eq.

Original: 4 is still d .
Revised: 4 is still d .

Original: Therefore, the sub-feature optimization in the proposed random feature selection can not be completed by directly calling the Dropout function.
Revised: Therefore, the sub-feature optimization in the proposed random feature selection can not be completed by directly calling the Dropout function.

Original: Since the binary vector varies at different iterations, different sub-features are trained and sub-gradients are calculated.
Revised: Since the binary vector varies at different iterations, different sub-features are trained and sub-gradients are calculated.

Original: This leads to a solution that each embedding neuron contains the similarity representation power.
Revised: This leads to a solution that each embedding neuron contains the similarity representation power.

Original: The schematic of the proposed method is in Fig.
Revised: The schematic of the proposed method is in Fig.

Original: 2(b).
Revised: 2(b).

Original: As shown, the prototype matrix W is maintained in the memory at the dimension of d × k during the whole training process, but only part of the classes ( k ′ =
Revised: As shown, the prototype matrix W is maintained in the memory at the dimension of d × k during the whole training process, but only part of the classes ( k ′ =

Original: k ∗ r 1 ) and features ( d ′ =
Revised: k ∗ r 1 ) and features ( d ′ =

Original: d ∗ r 2 ) are randomly selected to construct the softmax loss.
Revised: d ∗ r 2 ) are randomly selected to construct the softmax loss.

Original: The first random selection along the class dimension is for conflict-robust learning to achieve universal representation and the second random selection along the feature dimension is for feature compression required by efficient retrieval.
Revised: The first random selection along the class dimension is for conflict-robust learning to achieve universal representation and the second random selection along the feature dimension is for feature compression required by efficient retrieval.

Original: Therefore, we name our method UNIversal and COMpact (UNICOM) representation learning.
Revised: Therefore, we name our method UNIversal and COMpact (UNICOM) representation learning.

Original:  E XPERIMENTS 4. I MPLEMENTATION D ETAILS
Revised:  E XPERIMENTS 4. I MPLEMENTATION D ETAILS

Original: Unless otherwise specified, all ViT models in our experiments follow the same architecture designs in
Revised: Unless otherwise specified, all ViT models in our experiments follow the same architecture designs in

Original: CLIP, and are trained from scratch for 32 epochs on the automatically clustered LAION 400M dataset (Section 3.2) with cluster number k = 1 M .
Revised: CLIP, and are trained from scratch for 32 epochs on the automatically clustered LAION 400M dataset (Section 3.2) with cluster number k = 1 M .

Original: During training, we randomly crop and horizontally flip each image to get the input image with 224 × 224 resolution.
Revised: During training, we randomly crop and horizontally flip each image to get the input image with 224 × 224 resolution.

Original: We set the random class sampling ratio r 1 as 0 .
Revised: We set the random class sampling ratio r 1 as 0 .

Original: 1without random feature selection in the pre-training step.
Revised: 1 in the pre-training step.

Original: The training is conducted on  NVIDIA V100 GPUs across 16 nodes.
Revised: The training is conducted on 128 NVIDIA V100 GPUs acrossnodes.

Original: To save memory and scale up the batch size, mixed-precision and gradient checkpoint are used.
Revised: To save memory and scale up the batch size, mixed-precision and gradient checkpoint are used.

Original: We use AdamW (Loshchilov & Hutter, 2018) as the optimizer with an initial learning rate of 0 . 001 , and a weight decay of 0 . 05 .
Revised: We use AdamW (Loshchilov & Hutter, 2018) as the optimizer with an initial learning rate of 0 . 001 , and a weight decay of 0 . 05 .

Original: We employ margin-based softmax loss (Deng et al., 2019) for both pre-training and image retrieval tasks.
Revised: We employ margin-based softmax loss, ArcFace (Deng et al., 2019; 2020), for both pre-training and image retrieval tasks.

Original: The margin value is set to 0 .and
Revised: The margin value is set to 0 .

Original:  the feature scale is set to 64 .
Revised: 3 and the feature scale is set to 64 .

Original: To achieve compact representation for unsupervised retrieval, we fine-tune models on LAION 400M dataset for one epoch with the proposed random feature selection ( r 2 = 0 . 5 ).

Original: For supervised retrieval, we follow the data-split settings of the baseline methods (Patel et al., 2022; Ermolov et al., 2022) to fine-tune models.
Revised: For supervised retrieval, we follow the data-split settings of the baseline methods (Patel et al., 2022; Ermolov et al., 2022) to fine-tune models.

Original: 4. C OMPARISONS ON F EATURE R EPRESENTATION L EARNING
Revised: 4. C OMPARISONS ON F EATURE R EPRESENTATION L EARNING

Original: In this section, we first compare the performance of the proposed method and other baseline models (i.e., CLIP and OPEN-CLIP) on the linear probe and unsupervised image retrieval.
Revised: In this section, we first compare the performance of the proposed method and other baseline models (i.e., CLIP and OPEN-CLIP) on the linear probe and unsupervised image retrieval.

Original: Specifically, after the training on the automatically clustered 1M classes, we fix the backbones of our models.
Revised: Specifically, after the training on the automatically clustered 1M classes, we fix the backbones of our models.

Original: For the linear probe task, we learn an additional FC layer for classification on each test set.
Revised: For the linear probe task, we learn an additional FC layer for classification on each test set.

Original: For unsupervised image retrieval, we directly use the embedding features for testing.
Revised: For unsupervised image retrieval, we directly use the embedding features for testing.

Original: Then, we fine-tune the pre-trained models for supervised image retrieval on each image retrieval dataset.
Revised: Then, we fine-tune the pre-trained models for supervised image retrieval on each image retrieval dataset.

Original: Linear Probe.
Revised: Linear Probe.

Original: Following the same evaluation setting as CLIP (Radford et al., 2021), we freeze the pre-trained models on LAION 400M dataset and only fine-tune the last linear classification layer.
Revised: Following the same evaluation setting as CLIP (Radford et al., 2021), we freeze the pre-trained models on LAION 400M dataset and only fine-tune the last linear classification layer.

Original: We report the linear probing performance over 13 datasets in Tab. 1.
Revised: We report the linear probing performance over 13 datasets in Tab. 1.

Original: The proposed conflict-robust cluster discrimination method significantly outperforms the CLIP and OPEN-CLIP (Ilharco et al., 2021) models.
Revised: The proposed conflict-robust cluster discrimination method significantly outperforms the CLIP and OPEN-CLIP (Ilharco et al., 2021) models.

Original: Notably, our ViT B/32, ViT B/16, and ViT L/14 models surpass counterparts of OPEN-CLIP by 3.6%, 2.7% and 1.4% on average with the same training data, indicating that the proposed cluster discrimination can enhance the representation power over instance discrimination.
Revised: Notably, our ViT B/32, ViT B/16, and ViT L/14 models surpass counterparts of OPEN-CLIP by 3.6%, 2.7% and 1.4% on average with the same training data, indicating that the proposed cluster discrimination can enhance the representation power over instance discrimination.

Original: Unsupervised Image Retrieval.
Revised: Unsupervised Image Retrieval.

Original: In Tab. 2, we compare the performance of unsupervised image retrieval by directly using the pre-trained models for feature embedding.
Revised: In Tab. 2, we compare the performance of unsupervised image retrieval by directly using the pre-trained models for feature embedding.

Original: The GLDv2 (Weyand et al., 2020) employs mean Average Precision@100 (mAP@100) as the evaluation metric, while other datasets use Recall@1.
Revised: The GLDv2 (Weyand et al., 2020) employs mean Average Precision@100 (mAP@100) as the evaluation metric, while other datasets use Recall@1.

Original: Our ViT L/14 model achieves 69.6% average result across 7 image retrieval datasets, surpassing the OPEN-CLIP counterpart by 7.3% and even outperforming the larger
Revised: Our ViT L/14 model achieves 69.9% average result across 7 image retrieval datasets, surpassing the OPEN-CLIP counterpart by 7.5% and even outperforming the larger

Original: OPEN-CLIP model ViT H/14 by 5 . 4% .
Revised: OPEN-CLIP model ViT H/14 by 5 . 4% .

Original: The reason behind this significant improvement is that the proposed cluster discrimination can capture the semantic structure in data, which is crucial for the image retrieval task.
Revised: The reason behind this significant improvement is that the proposed cluster discrimination can capture the semantic structure in data, which is crucial for the image retrieval task.

Original: At the bottom of Tab. 2, we further explore unsupervised image retrieval  under  dimension constraints (e.g., 128-D and 64-D).
Revised: In Fig. 1(a), we compare our method with the state-of-the-art unsupervised image retrieval approach, STML (Kim et al., 2022), under different dimension constraints on the [Table] [Table] the same training data.

Revised: [Table]

Revised: CUB dataset.

Original: We set the random feature selection ratio ras 0 .
Revised: We set the random feature selection ratio r 2 as 0 .

Original: 5 ( d ′ = 256) for one additional  epoch of training.
Revised: 75 for one additional training epoch on the LAION 400M dataset.

Original: Then, we select the first 128-D and 64-D features for testing across different datasets.
Revised: Then, we select the first 256-D, 128-D, 64-D, and 32-D features for testing.

Original: Our ViT L/14 model constrained to 128-D and 64-D still surpasses the OPEN-CLIP ViT L/14 (768-D) model by 5.7% and 3.2%, indicating the superiority of the proposed random feature selection for compact feature representation learning.

Original: Here, we have not included the results from STML (Kim et al., 2022) for comparison as STML employs an ImageNet-1K pre-trained GoogleNet (Szegedy et al., 2015) and then explores unsupervised training on each retrieval dataset.
Revised: STML employs an ImageNet-1K pre-trained GoogleNet (Szegedy et al., 2015) and then explores unsupervised training on the CUB dataset.

Original: By contrast, our ViT based model is only pre-trained on the automatically clustered LAION 400M dataset without any further training on each image retrieval dataset.
Revised: Even though our ViT-based model is only trained on the automatically clustered LAION 400M dataset without any further training on the image retrieval dataset, our method outperforms STML (Kim et al., 2022) by a large margin across different test dimensions, indicating the superiority of the proposed random feature selection for compact feature representation learning.

Original: In
Revised: Fine-tune for ImageNet-1K Classification.

Original: Fig.
Revised: In Tab. 3, we compare our method with state-of-the-art supervised and weakly supervised pre-training (Dosovitskiy et al., 2021; Zhai et al., 2022; Singh et al., 2022) in transfer-learning experiments on ImageNet-1k. Our models consistently outperform OPENCLIP models in the Top-1 accuracy, showing the superiority of the proposed cluster discrimination. For ViT B/16, our pre-training achieves 85.9%, surpassing both the supervised pre-training on IN-21K (84.0%) and the weakly supervised pre-training on IG 3.6B (85.3%). In addition, our ViT L/ obtains 88.3%, outperforming ViT L/16 pre-trained on JFT 300M (87.8%) and ViT L/16 pre-trained on IG 3.6B (88.1%).

Original: 1(a), we show that Unicom can outperform STML by a large margin on the CUB dataset.
Revised: The overall results in ImageNet-1K classification task show that our models are

Original: [Table] [Table]
Revised: [Table] [Table]

Original: using thesame training data. For the proposed Unicom, the random feature selection ratio r 2 is set to 0 .
Revised: very competitive as they can achieve better or comparable accuracy even though the training data used by the competitors are much larger (e

Original: [Table]
Revised: [Table]

Original:  and the models are fine-tuned for additional one epoch on the LAION 400M dataset [Table] [Table] [Table]
Revised: g., JFT 3B and IG 3.6B)

Original: [Table]
Revised: [Table]

Original: Fine-tune for Supervised Image Retrieval.
Revised: Fine-tune for Supervised Image Retrieval.

Original: In Tab. 3 and 4, we compare the proposed Unicom with the latest  methods (Brown et al., 2020; Patel et al., 2022; Ermolov et al., 2022) trained with vision transformer.
Revised: In Tab. [Table] we compare the proposed approach with the latest image retrieval methods (Patel et al., 2022; Ermolov et al., 2022) trained with vision transformer.

Original: During fine-tuning of our models, the random negative class selection ratio r 1 is set to 1 . 

Revised: During fine-tuning of our models, the random negative class selection ratio r 1 is set to 1 .

Original: as the training data is clean and the random feature selection ratio r 2 is set to 0 . 5 to achieve compact feature embeddings.
Revised: as the training data is clean.

Original: Under different dimension constraints, Unicom consistently surpassesthe RA@K  on iNaturalist, SOP,  and Cars datasets, and outperforms Hyp-ViT  on CUB, Cars and In-shop datasets.On VehicleID and SOP, Unicom also achieves comparable performance compared to RA@K and Hyp-ViT if it does not rank first in specific tracks.
Revised: Under different computing regimes, the proposed method consistently surpasses RA@K (Patel et al., 2022) on the SOP, iNaturalist, and VehicleID datasets and outperforms Hyp-ViT (Ermolov et al., 2022) on the CUB and In-shop datasets.

Original: 4. A BLATION S TUDY
Revised: 4. A BLATION S TUDY

Original: Encoder for Clustering.
Revised: Encoder for Clustering.

Original: In Tab. 5, we compare the results of linear probe and unsupervised image retrieval under image-based clustering and text-based clustering by using the pre-trained CLIP and
Revised: In Tab. 5, we compare the results of linear probe and unsupervised image retrieval under image-based clustering and text-based clustering by using the pre-trained CLIP and

Original: OPEN-CLIP models.
Revised: OPEN-CLIP models.

Original: As we can see, the text encoder is more powerful than the image encoder, and image and text signals are complementary as the joint clustering significantly outperforms each individual.
Revised: As we can see, the text encoder is more powerful than the image encoder, and image and text signals are complementary as the joint clustering significantly outperforms each individual.

Original: By referring to Tab. 1 and Tab. 2, the OPEN-CLIP ViT B/32 model achieves 82 .
Revised: By referring to Tab. 1 and Tab. 2, the OPEN-CLIP ViT B/32 model achieves 82 .

Original: 1% and 55 .
Revised: 1% and 55 .

Original: 0% average results on the linear probe and unsupervised image retrieval tasks, while the proposed cluster discrimination method ( r 1 = 0 . 1 ) significantly boosts the performance to 84 .
Revised: 2% average results on the linear probe and unsupervised image retrieval tasks, while the proposed cluster discrimination method ( r 1 = 0 . 1 ) significantly boosts the performance to 84 .

Original: 1% and 60 .
Revised: 1% and 61 .

Original: 9% by using the OPEN-CLIP image and text models for clustering.
Revised: 1% by using the OPEN-CLIP image and text models for clustering.

Original: By using the CLIP image and text models for clustering, the performance can further increase to 85 .
Revised: By using the CLIP image and text models for clustering, the performance can further increase to 85 .

Original: 7% and 62 .
Revised: 7% and 62 .

Original: 2% on the linear probe and unsupervised image retrieval tasks.
Revised: 8% on the linear probe and unsupervised image retrieval tasks.

Original: Therefore, we choose the CLIP models for clustering.
Revised: Therefore, we choose the CLIP models for clustering.

Original: Cluster Number.
Revised: Cluster Number.

Original: In Tab. 5, we also compare the performance under different cluster numbers, i.e., 100K, 1M, and 10M, by using the CLIP image and text models.
Revised: In Tab. 5, we also compare the performance under different cluster numbers, i.e., 100K, 1M, and 10M, by using the CLIP image and text models.

Original: As can be seen, the best results can be achieved when the cluster number is set as 1 million, with the average image number per class being  around 400.
Revised: As can be seen, the best results can be achieved when the cluster number is set as 1 million, with the average image number per class being [Table] [Table] around 400.

Original: The cluster number needs to be balanced between the intra-class noises and inter-class noises.
Revised: The cluster number needs to be balanced between the intra-class noises and inter-class noises.

Original: Too small cluster numbers will incur heavy intra-class noise, which dramatically decreases the performance of the pre-trained classification model.
Revised: Too small cluster numbers will incur heavy intra-class noise, which dramatically decreases the performance of the pre-trained classification model.

Original: Besides, too many clusters will increase the computation and storage burden on the FC layer.
Revised: Besides, too many clusters will increase the computation and storage burden on the FC layer.

Original: Most important, the over-decomposition will increase the inter-class noise ratio and undermine the discriminative power of the pre-trained model.
Revised: Most important, the over-decomposition will increase the inter-class noise ratio and undermine the discriminative power of the pre-trained model.

Original: Random Class Selection.
Revised: Random Class Selection.

Original: In Tab. 6, we train ViT B/32 models under different inter-class sampling ratios.
Revised: In Tab. 6, we train ViT B/32 models under different inter-class sampling ratios.

Original: The basic margin-based softmax loss ( r 1 = 1 . 0 ) only achieves 77 .
Revised: The basic margin-based softmax loss ( r 1 = 1 . 0 ) only achieves 77 .

Original: 9% on the linear probe task as it can hardly adapt to the heavy inter-class conflict in the automatically clustered dataset.
Revised: 9% on the linear probe task as it can hardly adapt to the heavy inter-class conflict in the automatically clustered dataset.

Original: When the sampling ratio is decreased from 1 . 0 to 0 .
Revised: When the sampling ratio is decreased from 1 . 0 to 0 .

Original: 3 and 0 .
Revised: 3 and 0 .

Original: 1 , our method exhibits consistently better performance than the baseline, indicating random inter-class sampling is beneficial for the model’s robustness.
Revised: 1 , our method exhibits consistently better performance than the baseline, indicating random inter-class sampling is beneficial for the model’s robustness.

Original: When r 1 is set to 0 .
Revised: When r 1 is set to 0 .

Original: 05 , there is a slight performance drop because the inter-class interaction is insufficient during training.
Revised: 05 , there is a slight performance drop because the inter-class interaction is insufficient during training.

Original: Therefore, we choose the random negative class selection ratio as 0 . 1 , obtaining 85 .
Revised: Therefore, we choose the random negative class selection ratio as 0 . 1 , obtaining 85 .

Original: 7% and 62 .
Revised: 7% and 62 .

Original: 2% on the linear probe and unsupervised image retrieval tasks.[Table] [Table]
Revised: 8% on the linear probe and unsupervised image retrieval tasks.

Original: Random Feature Selection.
Revised: Random Feature Selection.

Original: In Tab. 6, we compare the performance of the proposed Unicom under different random feature selection ratios ( r 2 ) on the task of dimension-constrained unsupervised image retrieval.
Revised: In Tab. 6, we compare the performance of the proposed Unicom under different random feature selection ratios ( r 2 ) on the task of dimension-constrained unsupervised image retrieval.

Original: Here, we also include Dropout at different drop ratios ( r 3 ) for comparison.
Revised: Here, we also include Dropout at different drop ratios ( r 3 ) for comparison.

Original: From the results, we can have the following observations: (1) both random feature selection and Dropout can not improve linear probe and unsupervised image retrieval at a full dimension of 512 as the LAION 400M dataset is large enough and regularization is not necessary for the final classification layer, (2) there is obvious performance drop when the random feature selection ratio is too small (e.g., r 2 = 0 . 125 ), and (3) the proposed random feature selection ( r 2 = 0 . 5 ) can improve 0 .9% and 1 .
Revised: From the results, we can have the following observations: (1) both random feature selection and Dropout can not improve linear probe and unsupervised image retrieval at a full dimension of 512 as the LAION 400M dataset is large enough and regularization is not necessary for the final classification layer, (2) there is slight performance drop when the random feature selection ratio is decreasing, and (3) the proposed random feature selection ( r 2 = 0 . 75 ) can improve 0 .

Original: 2% for 128-D and 64-D unsupervised image retrieval, while Dropout can not improve dimension-constrained unsupervised image retrieval.
Revised: 3% for 256-D unsupervised image retrieval, while Dropout can not improve dimension-constrained unsupervised image retrieval.

Original: Even though Dropout enforces partial features for classification, the global randomization within the mini-batch makes the optimization involve all feature dimensions.
Revised: Even though Dropout enforces partial features for classification, the global randomization within the mini-batch makes the optimization involve all feature dimensions.

Original: By contrast, the proposed random feature selection is fixed within the mini-batch thus it can benefit from optimization in a sub-feature space.In Tab. 7, we compare the proposed random feature selection with Adaptive Pooling, PCA, and
Revised: By contrast, the proposed random feature selection is fixed within the mini-batch thus it can benefit from optimization in a sub-feature space.

Original: FC under supervised image retrieval.

Original: As we can see, PCA slightly outperforms Adaptive Pooling by 0 .

Original: 1% , and additionally trained FC surpasses PCA by 0 .

Original: When the random feature selection ratio r 2 decreases from 1 . 0 to 0 .

Original: during the training of the proposed Unicom, the test accuracy under low dimension constraint (e.g., 64-D and 32-D) first increases and then decreases, indicating appropriate random feature selection can enhance compact feature representation during training.

Original: When the random feature selection ratio is set to 0 .

Original: 5 , the proposed Unicom consistently surpasses the FC baseline across different dimensions.

Original: Please note that the FC projectors need to be trained under each dimension constraint while our Unicom is only trained once and tested across different dimensions.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: This paper introduces Unicom, a simple yet effective framework for universal and compact feature embedding.
Revised: This paper introduces Unicom, a simple yet effective framework for universal and compact feature embedding.

Original: Given the automatically clustered large-scale data, we employ one random negative class selection to improve the robustness under the heavy inter-class conflict and another random feature selection to improve the compactness of the feature representation.
Revised: Given the automatically clustered large-scale data, we employ one random negative class selection to improve the robustness under the heavy inter-class conflict and another random feature selection to improve the compactness of the feature representation.

Original: For both unsupervised and supervised image retrieval on different datasets, the proposed Unicom achieves state-of-the-art performance under different dimension constraints, confirming that cluster discrimination is beneficial to explore the semantic structure within the large-scale training data.
Revised: For both unsupervised and supervised image retrieval on different datasets, the proposed Unicom achieves state-of-the-art performance, confirming that cluster discrimination is beneficial to explore the semantic structure within the large-scale training data.

Original: A A PPENDIX A. 
Revised: A A PPENDIX A. M ODEL A RCHITECTURES

Original: M ODEL A RCHITECTURES

Original: We follow the same architecture design as CLIP.
Revised: We follow the same architecture design as CLIP.

Original: Tab. 8 describes the details of architectures.
Revised: Tab. 7 describes the details of architectures.

Original: A. V ISUALIZATION OF P SEUDO C LUSTERS AND D ATA D ISTRIBUTION
Revised: A. V ISUALIZATION OF P SEUDO C LUSTERS AND D ATA D ISTRIBUTION

Original: In Fig.
Revised: In Fig.

Original: 4, we show the data distribution under different settings of class number k .
Revised: we show the data distribution under different settings of class number k .

Original: In Fig.
Revised: In Fig.

Original: 5, we show some exemplar classes from the proposed automatic clustering.
Revised: we show some exemplar classes from the proposed automatic clustering.

Original: As we can see, there are some fine-grained classes, such as the top with love icons and the top with cartoons.
Revised: As we can see, there are some fine-grained classes, such as the top with love icons and the top with cartoons.

Original: Even though such clustering is reasonable and explainable, there is class confusion if we classify these samples from other views, such as color and targeting customer age. A. L INEAR P ROBE D ATASETS
Revised: Even though such clustering is reasonable and explainable, there is class confusion if we classify these samples from other views, such as color and targeting customer age

Original: We use 13 image classification datasets to prove the effectiveness of our method.

Original: These datasets include CIFAR10(Krizhevsky et al., 2009), CIFAR100(Krizhevsky et al., 2009), Caltech101(Fei-Fei et al., 2004), Stanford Cars(Krause et al., 2013a), Oxford Flowers(Nilsback & Zisserman, 2008),

Original: Food-101(Bossard et al., 2014), Birdsnap(Berg et al., 2014), SUN397(Xiao et al., 2010), Describable Textures(Cimpoi et al., 2014), FGVC Aircraft(Maji et al., 2013), Oxford-IIIT Pets(Parkhi et al., 2012), EuroSAT(Helber et al., 2019), ImageNet-1k(Russakovsky et al., 2015).
Revised: Model

Original: Details on each dataset and the corresponding evaluation metrics are provided in Tab. 9.

Original: [Table] [Figure] 
Revised: [Figure] [Table]

Original: The training and evaluation of image retrieval experiments on seven widely used datasets, namely CUB-200-2011(CUB) (Welinder et al., 2010), Stanford Cars(Cars196) (Krause et al., 2013b), Stanford Online Products(SOP)
Revised: Input

Original: (Oh Song et al., 2016), In-shop Clothes Retrieval(In-Shop) (Liu et al., 2016b), iNaturalist (Van Horn et al., 2018), VehicleID (Liu et al., 2016a), and Google Landmarks dataset (GLDv2) (Weyand et al., 2020). The number of examples and classes can be found in Tab. 10.
Revised: [Table]

Original: In this paper, the CLIP model is only used for the clustering step and our models are trained from scratch. In Fig. 6, we compare the training loss curves between fine-tuning and training from scratch.
Revised: resolution layers

Original: For fine-tuning, the backbone is initialized from the CLIP model (ViT-B/12), and the classifier (FC layer) is randomly initialized. The fine-tuning strategy can converge faster than training from scratch, but the final loss value is higher. In Tab. 11, we also find that training from scratch outperforms fine-tuning from the CLIP model by 0 . 7% on the task of linear probe.
Revised: [Table]

Original: [Figure] [Table] [Table] 
Revised: [Table] [Figure]

Original: VehicleID Train (Liu et al., 2016a) 110, 13, VehicleID Test (Liu et al., 2016a) 40, 4, GLDv2 Train (Weyand et al., 2020) 1,580, 81, GLDv2 Test (Weyand et al., 2020) 762, 1,
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: C
Revised: C

Original: I F
Revised: I F

Original: A
Revised: A

Original: R
Revised: R

Original: C
Revised: C

Original: I F
Revised: I F

Original: A
Revised: A

Original: R
Revised: R

Original: C a lt ec h
Revised: C a lt ec h

Original: C a r s
Revised: C a r s

Original: F l o w e r s Food
Revised: F l o w e r s F

Original: [Table]
Revised: [Table]

Original: i r d s n a p
Revised: i r d s n a p

Original: S
Revised: S

Original: [Table]
Revised: [Table]

Original: i r c ra f t
Revised: i r c r

Original: [Table]
Revised: [Table]

Original: [Table] C LUSTER D ISCRIMINATION A PPLIED TO D IFFERENT L ARGE P RE - TRAINED M ODELS In Tab.
Revised: t

Original: [Table]
Revised: [Table]

Original: we show that our ViT B/32 model trained by cluster discrimination significantly outperforms the baseline CLIP ViT B/32 model on the tasks of the linear probe and unsupervised image retrieval. Here, we also consider another large pre-trained model, Masked Autoencoders (MAE) (He et al., 2022), which is trained on ImageNet (Deng et al., 2009) by masking random patches of the input image and reconstructing the missing pixels.
Revised: [Table]

Original: We employ the largest model from MAE (ViT-H) for experiments. The MAE ViT-H model achieves 76 . 1% and 34 . 9% on the tasks of the linear probe and unsupervised image retrieval.
Revised: T RAINING FROM S CRATCH VS . F INE - TUNING FROM THE CLIP M ODEL

Original: As the MAE model istrained on ImageNet (Deng et al., 2009) with only 1,000 classes, the CLIP model shows superior performance than the MAE model in universal feature representation.
Revised: In this paper, the CLIP model is only used for the clustering step and our models are trained from scratch.

Original: After applying the MAE ViT-H model for automatic clustering on the LAION 400M dataset, we train a ViT B/32 model by the proposed cluster discrimination. As given in Tab. 12, the proposed cluster discrimination can significantly boost the performance on both linear probe and unsupervised image retrieval even though ViT B/32 is much smaller than ViT-H. Considering the performance gap between MAE and CLIP models on image retrieval, we focus on applying cluster discrimination to the CLIP model in this paper.
Revised: In Fig.

Original: In Tab. 13, we compare two discarding strategies to alleviate inter-class conflict. The proposed random negative prototype selection achieves 85 . 7% and 62 .
Revised: 6, we compare the training loss curves between fine-tuning and training from scratch.

Original: 2% on the tasks of the linear probe andunsupervised image retrieval when the random selection ratio is set as 0 . 1 . We also try to use a threshold ( τ ) to discard suspicious negative prototypes. As the threshold increases from 0 . 4 to 0 .
Revised: For fine-tuning, the backbone is initialized from the CLIP model (ViT-B/12), and the classifier (FC layer) is randomly initialized.

Original: 8 , the performance raises first and then drops. When the threshold is set to 0 . 6 , the online hard negative prototype discarding method obtains the best results of 84 . 4% and 61 .
Revised: The fine-tuning strategy can converge faster than training from scratch, but the final loss value is higher.

Original: 0% on the tasks of the linear probe and unsupervised image retrieval.

Original: Considering the effectiveness of the proposed random negative class selection, we have not dived deep into discarding strategies based on thresholding.

Original: Fine-tune for ImageNet-1K Classification. In Tab. 14, we compare our method with state-of-the-art supervised and weakly supervised pre-training (Dosovitskiy et al., 2021; Zhai et al., 2022; Singh et al., 2022) in transfer-learning experiments on ImageNet-1k. Our models consistently outperform OPENCLIP models in the Top-1 accuracy, showing superiority of the proposed cluster discrimination.
Revised: In Tab. 8, we also find that training from scratch outperforms fine-tuning from the CLIP model by 0 .

Original: For ViT B/16, our pre-training achieves 85.9%, surpassing both the supervised pre-training onIN-21K (84.0%) and theweakly supervised pre-training on IG 3.6B (85.3%). In addition, our ViT L/ obtains 88.3%, outperforming ViT L/16 pre-trained on JFT 300M (87.8%) and ViT L/16 pre-trained on IG 3.6B (88.1%). The overall results in ImageNet-1K classification task show that our models are very competitive as they can achieve better or comparable accuracy even though the training data used by the competitors are much larger (e.g., JFT 3B and IG 3.6B).
Revised: 7% on the task of linear probe.

Original: [Table] [Table] Comparisons on COCO Detection and Segmentation.
Revised: A. C OMPARISONS ON COCO D ETECTION AND S EGMENTATION

Original: Following the experiment setting in (Li et al., 2022), we use Mask R-CNN (He et al., 2017) for bounding-box object detection and instance segmentation.
Revised: Following the experiment setting in (Li et al., 2022), we use Mask R-CNN (He et al., 2017) for bounding-box object detection and instance segmentation.

Original: We fine-tune models on the COCO (Lin et al., 2014)
Revised: We fine-tune models on the COCO (Lin et al., 2014)

Original: train2017 split and evaluate on the val2017 split.
Revised: train2017 split and evaluate on the val2017 split.

Original: In Tab. 15, our method outperforms both OPEN-CLIP and supervised pre-training in all metrics, demonstrating the effectiveness of the proposed cluster discrimination.
Revised: In Tab. 9, our method outperforms both OPEN-CLIP and supervised pre-training in all metrics, demonstrating the effectiveness of the proposed cluster discrimination.

Original: Comparisons on Google Landmarks Dataset v2.
Revised: We use 13 image classification datasets to prove the effectiveness of our method. These datasets include CIFAR10(Krizhevsky & Hinton, 2009), CIFAR100(Krizhevsky & Hinton, 2009), Caltech101(Fei-Fei et al., 2004), Stanford Cars(Krause et al., 2013a), Oxford Flowers(Nilsback & Zisserman, 2008), Food-101(Bossard et al., 2014), Birdsnap(Berg et al., 2014), SUN397(Xiao et al., 2010), Describable Textures(Cimpoi et al., 2014), FGVC Aircraft(Maji et al., 2013), Oxford-IIIT

Original: We also compare our model with state-of-the-art methods on the instance-level large-scale image retrieval task of the Google Landmarks dataset v (Weyand et al., 2020) (referred to as GLDv2).
Revised: Pets(Parkhi et al., 2012), EuroSAT(Helber et al., 2019), ImageNet-1k(Russakovsky et al., 2015). Details on each dataset and the corresponding evaluation metrics are provided in Tab. 10.

Original: We evaluate on the standard test set, which contains 1,129 query images and 761,757 gallery images. The performance is evaluated using a mean Average
Revised: The training and evaluation of image retrieval experiments on seven widely used datasets, namely

Original: Precision@100 (mAP@100) metric. For the training of the proposed method, the input image is augmented with random cropping/aspect ratio distortion and resized to 512 × 512 following (Lee et al., 2022). We fine-tune our pre-trained models (e.g., ViT B/32 and ViT B/16) on the training set of GLDv2.
Revised: CUB-200-2011(CUB) (Welinder et al., 2010), Stanford Cars(Cars196) (Krause et al., 2013b), Stanford

Original: Random class selection and random feature selection are not used here as the training set is clean and there is no dimension limitation. Tab. 16 shows a comparison between the results of the proposed models and state-of-theart image retrieval methods on GLDv2. The proposed method ViT B/32 outperforms CVNet-Global
Revised: Online Products(SOP)

Original: Object detection and instance segmentation on COCO. We evaluate bounding-box AP (AP bb ) and mask AP (AP mk ) on val
Revised: (Oh Song et al., 2016), In-shop Clothes Retrieval(In-Shop)

Original: [Table] [Table] [Table] (Lee
Revised: (Liu et al., 2016b), iNaturalist (Van Horn et al., 2018), VehicleID

Original: [Table] al [Table] 2022) by 0 .
Revised: (Liu et al., 2016a), and Google Landmarks dataset (GLDv2) (Weyand et al., 2020).

Original: 1% on the private test set with less computation cost and more compact feature representation. In addition, our ViT B/16 model achieves 35 . 7% , surpassing CVNet-Global by 3 . 2% .
Revised: The number of examples and classes can be found in Tab. 11.

Original: [Table]
Revised: [Table]

Original: Google Universal Image Embedding competition is part of the Instance-Level Recognition workshop at ECCV 2022, which aims to create universal image representations that work across many visual domains. The evaluation dataset of this competition is kept private, containing images of apparel & accessories, packaged goods, landmarks, furniture & home decor, storefronts, dishes, artwork, toys, memes, illustrations, and cars.
Revised: [Table]

Original: Labels are defined generally at the instance level, for example, the same T-shirt, the same building, the same painting, and the same dish. In this competition, participants need to submit models to be evaluated by the test server. The submitted models take an image as an input and return a float vector (i.e., the image embedding) as the output.
Revised: VehicleID Train (Liu et al., 2016a) 110, 13,

Original: The embedding dimensionality is constrained to 64 , and the submissions are evaluated according to the mean Precision@5 metric. To prove the proposed Unicom can provide universal embeddings, we compared it with CLIP and OPEN-CLIP models in this competition.
Revised: VehicleID Test (Liu et al., 2016a) 40, 4,

Original: As shown in Tab. 17, our models significantly improve the score. Our ViT L/14-336 model achieves 56.2%, surpassing the CLIP counterpart by 11.8% and even outperforming the larger OPEN-CLIP model ViT g/14 by 3.8%. This improvement confirms that cluster discrimination can explore the semantic structure within the data, showing superiority over instance discrimination on the image retrieval task. We further explore the proposed random feature selection for unsupervised image retrieval under dimension constraints.
Revised: GLDv2 Train (Weyand et al., 2020) 1,580, 81,

Original: We set the random feature selection ratio as 0.5 ( d ′ = 256 ) for one additional epoch of training on the automatically clustered LAION-400M dataset, and we further improve the score of ViT L/14-336 by 2.7%.
Revised: GLDv2 Test (Weyand et al., 2020) 762, 1,