Paper: Multi-label Cluster Discrimination for Visual Representation Learning (ICLR-2024)

Reviewer comment (Writing):
This manuscript requires further refinement in terms of writing to facilitate reader comprehension, particularly by providing detailed explanations for the mathematical symbols used in the text, thus reducing reading barriers.

Author response:
**A1:**  In the revised version, we add detailed explanations for all mathematical symbols used in the equations and text.

Reviewer comment (Experiments):
In section $3.2, the authors claim efficient parallel computation and scalability of the model training process. However, is there quantitative data to support this point?

Author response:
**A2:**  In the revised Appendix A.1, we add a detailed explanation regarding the gradient calculation and we also include time cost comparison on the classification layer.

As we can see, the proposed method enables an elegant separation of positive and negative gradient calculation, which can decrease the communication frequency of calling the allreduce operation. 
To compare the time cost on the classification layer, we train UNICOM, MLC, and MLCD on the LAION-400M dataset with one million classes.
We use ViT-B/32 as the backbone, and the final embedding dimension is 512.
The batch size is set as $32,800$ and the A100 GPU number is 80 in total distributed across 10 computation nodes.
The time cost of the forward and backward steps regarding the backbone is $409$ms.
For MLC and MLCD, the positive class number is set as 8. 
For the single-label classification method, UNICOM, the time cost on the classification layer is $75$ms. 
For MLC and MLCD, the time cost on the classification layer is $138$ms and $82$ms. The proposed multi-label cluster discrimination method slightly increases the time cost by 9.3% on the classification layer compared to the single-label cluster discrimination method, UNICOM. Compared to the widely used multi-label classification method, the proposed method obviously decreases the time cost by 45.6% on the classification layer. Therefore, the proposed decomposition of contrastive loss ensures efficient parallel computation across multiple GPUs from different computation nodes with minimal communication overhead. 

With the aid of efficient feature quantization (Johnson et al., 2019), it only takes around 10 minutes to complete the offline clustering step on the large-scale LAION-400M dataset. For multi-label training, there is a slight increase of 9.3% in the time cost of the classification layer compared to the single-label cluster discrimination method.

Reviewer comment (Evaluation):
Does the incorporation of clustering significantly improve the training time?

Author response:
**A2:**  In the revised Appendix A.1, we add a detailed explanation regarding the gradient calculation and we also include time cost comparison on the classification layer.

As we can see, the proposed method enables an elegant separation of positive and negative gradient calculation, which can decrease the communication frequency of calling the allreduce operation. 
To compare the time cost on the classification layer, we train UNICOM, MLC, and MLCD on the LAION-400M dataset with one million classes.
We use ViT-B/32 as the backbone, and the final embedding dimension is 512.
The batch size is set as $32,800$ and the A100 GPU number is 80 in total distributed across 10 computation nodes.
The time cost of the forward and backward steps regarding the backbone is $409$ms.
For MLC and MLCD, the positive class number is set as 8. 
For the single-label classification method, UNICOM, the time cost on the classification layer is $75$ms. 
For MLC and MLCD, the time cost on the classification layer is $138$ms and $82$ms. The proposed multi-label cluster discrimination method slightly increases the time cost by 9.3% on the classification layer compared to the single-label cluster discrimination method, UNICOM. Compared to the widely used multi-label classification method, the proposed method obviously decreases the time cost by 45.6% on the classification layer. Therefore, the proposed decomposition of contrastive loss ensures efficient parallel computation across multiple GPUs from different computation nodes with minimal communication overhead. 

With the aid of efficient feature quantization (Johnson et al., 2019), it only takes around 10 minutes to complete the offline clustering step on the large-scale LAION-400M dataset. For multi-label training, there is a slight increase of 9.3% in the time cost of the classification layer compared to the single-label cluster discrimination method.

Reviewer comment (Reproducibility):
In section $3.2, you employ some distribution training techniques but details are not provided, which hinders the reproducibility of the work.

Author response:
**A3:**  In the revised Appendix A.1, we add a detailed explanation regarding the gradient calculation and the distribution training. In the supplementary material, we put our training code, which is visible to the public now and can be used for reproducing our method.

Reviewer comment (Reproducibility):
The performance reported in the CLIP paper differs from your reproduced version in Tab 1 and Tab 2, which may influence the validation of improvement of your model. Have you checked the implementation and settings?

Author response:
**A1:** Even though CLIP has generously open-sourced the models, they have not provided the evaluation tools. 
To this end, we contacted the UNICOM [1] authors to get their evaluation toolkit and all test datasets for the task of linear probe. In UNICOM, the authors have developed a GPU-accelerated logistic regression algorithm, facilitating batch-wise assessment of linear probe performance. In this paper, the performance of CLIP is also reported by using this evaluation toolkit.

For the task of zero-shot classification, we adopted the same experimental settings as FLIP [2] and referred to the FLIP paper to report the zero-shot results.  

[1] An, Xiang, et al. Unicom: Universal and Compact Representation Learning for Image Retrieval. ICLR 2023.

[2] Li, Yanghao, et al. Scaling language-image pre-training via masking. CVPR 2023.

Reviewer comment (Experiments):
my key concern on the high-level idea is whether the top-k closest clusters to an image can really reveal what objects/attributes (will use “concepts” for clarity hereafter) are involved in it. At the cluster level, samples of the same clusters are likely to share more nearest clusters (in a global picture) but the concepts involved in each independent image are almost random. Is it possible that the multiple labels assigned to the same images provide models with additional knowledge about the co-occurrence/relevance of different concepts (cluster-to-cluster relationships) rather than actually telling models what is involved in images (sample-to-cluster relationships)? It will be interesting to see more exploration and analysis of why the multi-label clustering idea is beneficial. One simple verification can be pre-training on a dataset with known non-overlapping class structure, eg ImageNet, and see if the multi-label clustering still benefits.

Author response:
**A1:** We added corresponding experiments and visualization in the revised version.

 |   CASE   | DATA | 0.5K | 1K   | 2K   | 4K   | 8K   | 20K |
 | ------   | ------ | ------ | ------ | ------ | ------ | ------ | ------ |
 |   UNICOM | IN1K | 42.1 | 58.4 | 61.5 | 62.8 | 62.4 | 61.5|
 |   MLCD   | IN1K | 63.2 | 67.2 | 68.2 | **69.9** | 69.7 | 69.0|

In Tab. 7 (a), we compare the proposed multi-label cluster discrimination and the single-label cluster discrimination on ImageNet with the clustered class number ranging from 0.5K to 20K. The clustering step is conducted by using the features predicted by the CLIP model. In the discrimination step, both MLCD and UNICOM employ the negative class center sampling with a ratio of $0.1$, and the positive number for MLCD is set as $8$.
Pre-training is executed on the ImageNet training dataset by 100 epochs. The model backbone is ResNet-50. The evaluation is undertaken using a linear probe on the ImageNet validation set.
As we can see, the proposed multi-label learning significantly surpasses UNICOM and achieves the best performance of 69.9% when the class number is 4K, which is four times of the true class number of ImageNet. In Fig. 5 of the Appendix, we visualize the top three labels for our training samples. When training with multiple labels, our method can not only capture the label correlations (e.g., paintbrush and oil painting, shield and armour, ladle and frying pan) but also learn complementary visual signals (e.g., different breeds of dogs, different locations of figs) to improve visual representation learning.

Reviewer comment (Experiments):
The modifications made to triplet loss make sense to me but their effects are unclear. How will the proposed model perform if all its designs are kept unchanged except for replacing L’_MLCD (Eq.6) with L_MLC

Author response:
**A2:** In the revised version, we add the experiments to verify the effectiveness of the proposed MLCD.

|CASE | DATA | Finetune | Linear Probe | Zero Shot|
| ------   | ------ | ------ | ------ | ------ | 
|MLC  | LAION-400M| 80.9 | 76.9 | 63.9 |
|MLCD | LAION-400M| 81.2 | 78.1 | 64.5 | 

Tab. 7 (a), we compare the performance of the vanilla MLC and the proposed MLCD on the ImageNet validation dataset. Pre-training is executed on the LAION-400M dataset by 32 epochs. The model backbone is ViT-B/32. 
Both MLC and MLCD employ the negative class center sampling with a ratio of $0.1$. MLCD outperforms MLC in all three settings: fine-tuning, linear classification, and zero-shot, confirming the effectiveness of the two additional optimization targets. In Appendix. A.1, we compare their gradient calculation and time cost on the classification layer. The proposed contrastive loss decomposition can significantly decrease the communication cost, facilitating distributed training on large-scale training data.

Reviewer comment (Presentation):
What are the blue and green cells standing for in the grids pointing to the text “contrastive loss”?

Author response:
**A3:**  For Fig. 1(a), blue cells stand for the similarity scores of positive image-text pairs. The similarities in the blue cells need to be increased during training.
For Fig. 1(b), there are two training samples in the minibatch.
The blue and green cells stand for the positive scores in their corresponding one-hot target logits.
For Fig. 1(c), there are also two training samples in the minibatch.
However, each sample has three positive labels. Thus, the target logits contain three positive similarity scores that need to be increased during training.

Reviewer comment (Presentation):
Whilst Fig.2 is the first figure being referred to, Fig.1 is simply mentioned as the illustration of visual representation learning but it lacks further explanation/discussion.

Author response:
**A4:** In the revised introduction section, we refer to Fig. 1 first when we introduce instance discrimination methods and cluster discrimination methods. In the revised preliminary section, we add detailed explanations of Fig. 1 (a) and Fig. 1 (b). In Section 3.2, we also refer to Fig. 1 (c) when we introduce our method.

Reviewer comment (Theory):
In Eq.1, I assume the pairwise similarity is cosine similarity if following CLIP, but without normalisation of features, it is just an inner product. So I’m wondering if it is a mistake or my misunderstanding.

Author response:
**A5:** We employed the normalization to all embedding features, and the pair-wise similarity is the cosine similarity. In the revised version, we added the keyword of normalization to features around Eq. (1).

Reviewer comment (Writing):
In Eq.1, the index in the cumulative sum starts from 0 to k while that in Eq.2 is from 1 to k, is this deliberate and why?

Author response:
**A6:** In the revised version, all indices in the cumulative sum start from 1 to k. In Eq. (1), $e_j'$ contains one positive text representation for $i$ and $(k-1)$ negative text representations sourced from different instances.

Reviewer comment (Writing):
The exponential function is denoted as exp and e at the same time in Eq.3, which makes the equation really confusing when the feature representations are also denoted as e_i.

Author response:
**A7:** In the revised version, we use exp() to denote the exponential function in all equations to avoid confusion with our feature embedding symbol.

Reviewer comment (Experiments):
The ablation studies are a bit unclear to me. For example, when investigating the effects of sample ratio, the best linear probe performance is obtained when the sample ratio is set to 0.1, and the best result is 75.2. However, the linear probe performance of the proposed model shown in Table 5 is 84.6.

Author response:
**A8:** The performance gap between Tab.5 and Tab. 6 is due to different backbones and epochs used for training.
In Tab. 1 - 5, the model backbone used is ViT-L/14. The epoch number is 32 (default epoch number).
To accelerate the hyper-parameter search, we use ViT-B/32 as the backbone and 5 epochs. 
In the revised version, we also add the backbone information to the captions of Tab. 4, 5, and 6 to avoid confusion. Besides, we also add the epoch number into the caption of Tab. 6.

Reviewer comment (Evaluation):
Although the proposed method yielded impressive performance, it is also crucial for me to figure out the underlying reasons for the effectiveness. So further evidence and discussions about this will be helpful.

Author response:
First of all, great thanks for your quick reply.

The performance increase of the proposed multi-label cluster discrimination method comes from the complementary supervision signals. It is a united story covering all cases. 
For an image with multiple objects, our method can avoid false negative supervision signals. Even on the single-label dataset, ImageNet, multiple objects exist in a large number of images.
For an image including only one object, our method can improve vision representation by learning advanced concepts, e.g., attributes from different views. 

We revised the appendix section regarding multi-label learning on ImageNet.

In Fig. 5, we visualize the top 3 labels given to some selected training samples. We cluster the original ImageNet training dataset into 4K classes by using the pre-trained CLIP model. As depicted in Fig. 5, these multiple positive labels exhibit complementary visual signals (e.g., different activities on the snow, different breeds of dogs, different locations of figs, and different objects in the room). For each of the automatically clustered 4K classes, we match the dominant class to the original 1K classes. In the single-label cluster discrimination method, only one label is used for training, and the recall of the correct label is 76.2%. By contrast, in the proposed multi-label cluster discrimination, the top 8 labels are used for training, and the recall of the correct label is 93.5%. Even though using multiple labels introduces some incorrect labels,  the proposed multi-label cluster discrimination method can learn complementary visual signals to improve visual representation learning. 

In Fig. 6, we also visualize some interesting labels given to the automatically clustered samples. As we can see, multiple positive labels sometimes show abstract correlations. For example, there is the concept of "colorful'' between "paintbrush'' and "oil painting'' in the 1st row, the concept of "shining'' between "shield'' and "armour'' in the 2nd row, and the concept of "with a handle'' between "ladle'' and "frying pan'' in the 3rd row. The concepts of "colorful'', "shining'' and "with a handle'' are class descriptions from different views. Learning from these complementary concepts can also increase vision representation learning.

Reviewer comment (Novelty):
The novelty and originality of this work are limited. It seems like the method proposed in this paper incorporates several techniques introduced in the literature. It does not offer sufficient technical inspirations for the readers to follow.

Author response:
**A1:**  In the instance discrimination methods (e.g., CLIP), each image-text pair represents one unique class. CLIP can provide richer forms of labels for a single image, e.g., objects, scenes, actions, and relations, at multiple levels of granularity. However, negative pairs that share similar semantics will be undesirably pushed apart in the embedding space. 

In the cluster discrimination methods (e.g., UNICOM), visually similar instances are pulled together by the classification step. However, UNICOM only defines a single pseudo-label for each image.

To the best of our knowledge, we are the first to propose multi-label learning on the large-scale dataset (LAION-400M).
The proposed multi-label cluster discrimination approach can not only capture the semantic structures in the data but also support the learning of multiple granularity of labels for a single image. In addition, we introduce a novel decomposition of contrastive loss, to 
avoid ambiguity during the optimization of $(s_j-s_i)$ as well as decrease the communication cost during distributed training.

Reviewer comment (Novelty):
With respect to the limited technical novelty of this work and overall moderate improvement (I see in Table 1 and Table 2), it may not seem to be worthwhile using such huge computing resources (80 NVIDIA A100 GPUs), especially considering that visual-language pre-training field has already achieved remarkable performance.

Author response:
**A2:** In this paper, we mainly compared our method with the latest state-of-the-art methods (e.g., UNICOM [1] and FLIP [2]). In the task of linear probe, our method outperforms UNICOM by 1.3% (Tab. 1). In the task of zero-shot classification, our method surpasses FLIP by 1.5% (Tab. 2). Vision representation learning is one of the most competitive research topics, and our improvement compared to this year's publications is indeed significant. 

[1] An, Xiang, et al. Unicom: Universal and Compact Representation Learning for Image Retrieval. ICLR 2023.

[2] Li, Yanghao, et al. Scaling language-image pre-training via masking. CVPR 2023.

Reviewer comment (Writing):
According to my understanding, as proposed method is developed upon the feature embedding from the pre-trained CLIP model and it does not involve any textual information in the proposed multiple label loss. If this is correct, this paper should make this more clear.

Author response:
**A3:** In this paper, we focus on improving visual representation by introducing multi-label cluster discrimination. Our method does not involve text learning. In Fig. 1, we compare our method with CLIP and UNICOM. CLIP consists of the image encoder and text encoder, while UNICOM and the proposed method only contain the image encoder. In Section 4.1, we mention how to train an additional text encoder following Locked-image Tuning (LiT)[3], which teaches a text model to read out good representations from a locked image model for zero-shot classification and image-text retrieval tasks.

"To assess the performance of zero-shot classification and zero-shot image-text retrieval tasks, we employ contrastive learning to train a text encoder from scratch for 32 epochs with a frozen visual encoder following LiT (Zhai et al., 2022b). The structure of the text encoder is also identical to CLIP. "

[3]Zhai, Xiaohua, et al. Lit: Zero-shot transfer with locked-image text tuning. CVPR 2022.

Reviewer comment (Experiments):
In Equation 5, two new items are further introduced into the multi-label loss. This is regarded as one of the key contributions by this paper. However, its efficacy does not seem to be clearly verified in the ablation study. This needs to be addressed.

Author response:
**A4:** 
In the revised version, we add the experiments to verify the effectiveness of the proposed MLCD.

|CASE | DATA | Finetune | Linear Probe | Zero Shot|
| ------   | ------  | ------        | ------               | ------        | 
|MLC    | LAION-400M     | 80.9 | 76.9 | 63.9 |
|MLCD | LAION-400M     | 81.2 | 78.1 | 64.5 | 

Tab. 7 (a), we compare the performance of the vanilla MLC and the proposed MLCD on the ImageNet validation dataset. Pre-training is executed on the LAION-400M dataset by 32 epochs. The model backbone is ViT-B/32. 
Both MLC and MLCD employ the negative class center sampling with a ratio of $0.1$. MLCD outperforms MLC in all three settings: fine-tuning, linear classification, and zero-shot, confirming the effectiveness of the two additional optimization targets. In Appendix. A.1, we compare their gradient calculation and time cost on the classification layer. The proposed contrastive loss decomposition can significantly decrease the communication cost, facilitating distributed training on large-scale training data.

Reviewer comment (Experiments):
As one of the main contributions of this paper, the authors claim that the modification of optimization loss can elegantly separate the positive class labels and negative class labels, resulting in promotion of the distributed training on large-scale training data. Please explain and experimentally demonstrate how this modification can facilitate the distributed learning more clearly. For example, in Subsection Distributed Multi-label Classification of Section 3.2 MULTI-LABEL CLUSTER DISCRIMINATION, The first sentence “Eq. 6 is able to distribute the weights associated with one million class centers across all GPUs with minimal communication overhead.” Why?

Author response:
**A1:** 
In the revised version (Appendix A.1), we add a detailed explanation regarding the gradient calculation and we also include a time cost comparison on the classification layer.

As we can see, the proposed method enables an elegant separation of positive and negative gradient calculation, which can decrease the communication frequency of calling the allreduce operation. 
To compare the time cost on the classification layer, we train UNICOM, MLC, and MLCD on the LAION-400M dataset with one million classes.
We use ViT-B/32 as the backbone, and the final embedding dimension is 512.
The batch size is set as $32,800$ and the A100 GPU number is 80 in total distributed across 10 computation nodes.
The time cost of the forward and backward steps regarding the backbone is $409$ms.
For MLC and MLCD, the positive class number is set as 8. 
For the single-label classification method, UNICOM, the time cost on the classification layer is $75$ms. 
For MLC and MLCD, the time cost on the classification layer is $138$ms and $82$ms. The proposed multi-label cluster discrimination method slightly increases the time cost by 9.3% on the classification layer compared to the single-label cluster discrimination method, UNICOM. Compared to the widely used multi-label classification method, the proposed method obviously decreases the time cost by 45.6% on the classification layer. Therefore, the proposed decomposition of contrastive loss ensures efficient parallel computation across multiple GPUs from different computation nodes with minimal communication overhead.

Reviewer comment (Reproducibility):
Some technical details are missing, e.g., in Section 4.1, the authors should explicitly point out the number of classes (k) and number of positive centers (l) they use when pre-training the model on LAION-400M dataset.

Author response:
**A2:** In the revised version, we add these technical details in Section 4.1.
"In the following experiments, unless otherwise specified, the model used is ViT-L/14, the number of classes ($k$) is one million, and the number of positive labels ($l$) assigned to each image is $8$."

Reviewer comment (Presentation):
In Table 3, the best results consist of both the proposed method in this paper and the FLIP (i.e., 89.1%). Notably, only the results of the proposed methodology have been highlighted.

Author response:
**A3:**   We fixed this in the revision, and we also checked other tables.

Reviewer comment (Presentation):
In Section 4.6, the meaning of the y-axis of the charts should be provided to improve the clarity of the paper.

Author response:
**A4:** In Fig. 3, we visualize per-dataset differences in the performance. The Y-axis shows the performance difference. Most of the evaluation metrics are accuracy as shown in Tab. 10 (Appendix). Green bars indicate our model outperforms the baselines, while the orange bars depict our model is surpassed by the baselines. Our model outperforms UNICOM on 23 datasets on the task of linear probe and surpasses FLIP on 15 datasets on the task of zero-shot classification. In the revised version, we added a detailed explanation in the caption of Fig. 3.