Paper: Data De-Duplication and Semantic Enhancement for Contrastive Language-Image Pre-training (ICLR-2024)

Reviewer comment (Writing):
The biggest concern for this paper is the key intuition and motivation of the proposed method. The data de-duplication is leveraged to reduce the training samples, which is useful for training efficiency. However, the other three tricks mostly focus on augmentation, thus introducing more data for training. This mix-up makes the readers follow the key contribution of this paper.

Author response:
> **Q1: The key intuition and motivation of the proposed method. Explanation of the contradiction between the effectiveness of ($\text{D}^3$) and the other three modules e.g. text augmentation, DCTM, and MSTM.**

The key intuition and motivation of the proposed method is that construct a semantically diverse multimodal dataset and present an efficient training algorithm for better representation learning.

The generated texts are completed by one-time processing, which can be reused to train other models, the time does not require double-counting. The efficient training algorithm is proposed to utilize semantically enriched datasets. We also provide the specific calculation time for each module.

Specifically, as Q4 of review Fx4U, we counted the time for all modules. It takes about 175 hours to rewrite one text from LLaMA for the entire Laion400M dataset on 32 A100 GPU machines, and it takes only 75 hours to generate texts from LLaVA with batch inference. Training baseline CLIP on LAION-400M with large ViT-L/14 almost takes 550 hours on 32 A100 GPU machine. Training with half of the clustered augmented data can achieve comparable performance but its training time is reduced by half. Clustering time is negligible as compared to text generating, it only takes a few hours on one A100 GPU machine. The total training cost has decreased a little overall (550 vs 525) on 32 A100 GPU machines.

Reviewer comment (Experiments):
The first approach, i.e., data de-duplication, is also limited by the pre-trained vision model. If we use another model rather than DINO, the sampled images could be different, which may lead to different conclusions.

Author response:
> **Q3: The effect of the pre-trained model e.g. DINO in data de-duplication.**

###### 						Tab.7: Ablation study for clustering pre-trained model on ImageNet(IN) datasets for Zero-shot image classification (%), 																							all experiments are used 50% clustered data.

| Evaluation Type | Clustering Pre-trained Model | IN |
| :-: | :-: | :-: |
| Zero-shot | Dino | 34.6 |
| Zero-shot | MAE | 34.2 |


As per your suggestion, we explored the use of different models to extract image features for clustering e.g. MAE and Dino.

As shown in the above table, there is no big gap in classification performance between using pre-trained model MAE to extract image features and using Dino to extract image features for clustering. As long as with a well-pre-trained model, the image feature for clustering has little impact, all of which can produce better clustering results.

Reviewer comment (Writing):
There are many notational errors (for example, N or B for the number of pairs?) in the descriptions of Sec. 3.1. Please carefully revise them.

Author response:
> **Q4: Corrected notational errors.**


Thanks for your kind suggestion, we have modified the paper in detail.

Reviewer comment (Theory):
Eqn.1 seems not right. Normally the NCE contrastive loss only holds one positive label. But for this approach, there could be at least three positive labels. Maybe some theoretical analysis helps address this concern.

Author response:
> **Q5: Normally the NCE contrastive loss only holds one positive label. But for this approach, there could be at least three positive labels. Analysis of Eqn.1.**


The NCE contrastive loss only holds one positive label. According to SE in DS-CLIP, one image has $M+1$ multiple generated texts represented as in Eqn.1, which constructs $M+1$ image-text pairs. In Eqn.1, for each image corresponding to $M+1$ multiple texts, we calculate the mean of contrastive loss of $M+1$ image-text pairs during training. In all, Eqn.1 can be interpreted as the mean of multiple NCE contrastive losses.

Reviewer comment (Novelty):
The core contributions of this paper are the D3 and SE modules, which belong to the data augmentation and data cleaning essentially. The clustering, re-sampling, and text re-generation are all very common strategies in recent work, e.g., BLIP [1], and BLIP-2 [2]. Hence the technical contribution is weak.

Author response:
**BLIP[1]** proposes an efficient utilization method of web noisy data. It is first trained with noisy data, and then BLIP is used to generate a series of subtitles through a pre-trained Captioner, moreover, the generated subtitles are filtered through a pre-trained Filter, deleting the noisy subtitles from the original web and synthetic texts to get clean data. Finally, BLIP is trained again with clean data. The re-sampling in BLIP is filtered and text re-generation is iteratively updated as model training. 

**In contrast to the BLIP**, our method uses the rich knowledge of LLM/VLLM to enhance the text data and rewrite the data semantically and structurally without repeated model training. We pay more attention to constructing a more diverse and semantic dataset from the web noisy dataset rather than the training strategy. The re-sampling is without filtering data and keeps the diversity of the dataset at each training epoch. Additionally, we conducted experiments comparing the training approaches of initializing the model with the backbone of the VLLM itself and training the multimodal representation model using the data generated by the LLM and VLLM. We find that utilizing the augmented data from a well-trained LLM and VLLM yields superior performance, further confirming the importance of LLM and VLLM in data construction. We hope our augmented dataset will help further research and pay more attention to the data construction by LLM and VLLM.

**BLIP-2[2]** proposes an efficient pre-trained algorithm through training Q-Former to align a visual model and a Large Language Model (LLM) of freezing parameters. The Q-Former connects an image encoder of a visual model and an LLM encoder. In BLIP-2, the data is not extended or augmented from source, and the training parameters are reduced from the model perspective for efficient training.

**The essential difference with our method** is that we start from the misaligned between image and text, and use the rich knowledge of LLM/VLLM to enrich text representation. We focus on the problem of data noise from the source rather than the training strategy. We utilize Data De-Duplication ($\text{D}^3$) based on augmented diverse data to achieve superior performance with fewer computing resources. $\text{D}^3$ is different from data cleaning, we do not filter data during training but sample a certain percentage of data points from each pre-clustered center. The text re-generated and sampling strategy are quite different from BLIP[1] and BLIP-2[2]. Extensive experiments show that the constructed dataset is helpful in improving the performance of the current modal e.g. DeCLIP, LaCLIP. Please refer to the question 6 of review YX2M.

**Reference:**

[1] Li J, Li D, Xiong C, et al. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation[C]//International Conference on Machine Learning. PMLR, 2022: 12888-12900.

[2] Li J, Li D, Savarese S, et al. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models[J]. arXiv preprint arXiv:2301.12597, 2023.

Reviewer comment (Experiments):
The CLIP is a famous multi-modal pretraining model. However, this paper only contains pure vision-understanding tasks and lacks sufficient experiments on various multi-modal tasks and datasets, e.g., image-text matching, video-text retrieval, and image captioning.

Author response:
Thanks for your suggestion. We added more experiments on image-text matching, video-text retrieval, and image captioning.

**For the image-text matching task**, we conducted image-text retrieval on COCO and Flickr30K datasets, as shown in Sec.4.3 in the paper. 

In order to verify that the image-language model (DS-CLIP) has a strong generalization ability for video-language tasks, we test DS-CLIP on a 1k test split of MSR-VTT for **video-text retrieval tasks**. The evaluation indicator of R@1 is the same as image-text retrieval. For video-text retrieval, zero-shot DS-CLIP outperforms baseline CLIP as shown in the following table.

**For the image captioning task**, we consider the COCO dataset with BLEU@4 and CIDEr indicator. We use image encoder VIT-B/16 of CLIP and DS-CLIP to initialize the image encoder of coca[1] for fair comparison following the original code training on the COCO caption trainset. The result is shown in the following table, we can find out DS-CLIP far exceeds baseline CLIP by 1.7% of BLEU@4 and 4.2% of CIDEr.

###### 		Tab.5: Performance of DS-CLIP on downstream tasks (Video-to-Text(V2T) Retrieval, Image Caption, and Image-to-Text(I2T) Retrieval) (%).

| Method | V2T Retrieval |  | Image Caption |  | I2T Retrieval |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | MSR-VTT |  | COCO |  | COCO |  | Flickr30K |  |
|  | V2T/R@1 | T2V/R@1 | BLEU@4 | CIDEr | I2T/R@1 | T2I/R@1 | I2T/R@1 | T2I/R@1 |
| CLIP | 9.1 | 9.1 | 33.7 | 108.3 | 20.4 | 33.3 | 37.8 | 57.3 |
| DS-CLIP | 15.1 (+6.0) | 15.1 (+6.0) | 35.4 (+1.7) | 112.5 (+4.2) | 30.7 (+10.3) | 45.5 (+12.2) | 52.3 (+14.5) | 75.3 (+18.0) |


**Reference:**

[1] Yu J, Wang Z, Vasudevan V, et al. Coca: Contrastive captioners are image-text foundation models[J]. arXiv preprint arXiv:2205.01917, 2022.

Reviewer comment (Evaluation):
This paper has claimed that the DS-CLIP only needs half the training time compared with traditional CLIP. But the D3 and SE modules also need large computational costs, e.g., the clustering operation, and the inference process of LLM/VLLM. It is necessary to report related time costs since the extra data augment is an important part of the proposed method.

Author response:
As shown in Sec4.2, **''DS-CLIP is scalable for training on large datasets and models''**, we report the inference time of SE.

It takes about 175 hours to rewrite one text from LLaMA for the entire Laion400M dataset on 32 A100 GPU machines, and it takes only 75 hours to generate texts from LLaVA with batch inference on 32 A100 GPU machines. Training baseline CLIP on LAION-400M with large ViT-L/14 takes 550 hours on 32 A100 GPU machines. Training with half of clustered augmented data can achieve comparable performance but its training time is reduced by half. Clustering time is negligible as compared to text generating, it only takes a few hours on one A100 GPU machine. The total training cost has decreased a little overall(550 vs 525) on 32 A100 GPU machines.

In addition, the generated texts are completed by one-time processing, which can be reused to train other models, the time does not require double-counting.

Reviewer comment (Experiments):
The Data De-Duplication (D3) relies on the clustering algorithm to converge unlabeled data. However, the K-means is not a good choice for large amounts of data, in which the runtime and memory cost are non-negligible with multiple iterations. Have you tried any other clustering algorithms, e.g., spectral clustering? Besides, why not cluster the texts?

Author response:
Thanks for your advice. We use spectral clustering instead of K-means to pre-process the YFCC15M dataset, keeping other parameters are same. The clustering type has little impact on the performance of DS-CLIP as shown in the following table, the results are almost the same(34.6% vs 34.5%).

###### Tab.6: Ablation study for Clustering type and feature on ImageNet(IN) datasets for Zero-shot image classification (%), all experiments are used 																													50% clustered data.

| Evaluation Type | Clustering Feature | Clustering Type | IN |
| :-: | :-: | :-: | :-: |
| Zero-shot | Image | K-means | 34.6 |
| Zero-shot | Image | Spectral | 34.5 |
| Zero-shot | Text | K-means | 32.8 |


> **Q6: Using the clustering text in our experiments.**


We report the results of using the text clustering in Appendix.D, the model with the clustering feature of the image performs better than the text clustering(34.6% vs 32.8%), which is also shown in the above table.

Reviewer comment (Novelty):
In SE: The effectiveness of synthetic captions from VLP models for mitigating noisy image-text alignment has already been demonstrated by BLIP. Therefore, it is somewhat straightforward that more descriptive captions from recent LLaVA models would be effective. Furthermore, as the authors themselves pointed out, the concept of using LLM-generated captions has already been proposed in LaCLIP. Moreover, the effectiveness of using both LLaVA and LLaMA is unclear. See question 2.

Author response:
Thank you for your insightful advice and valuable questions, we will respond to your concerns point by point.

> **Q1: Limitation of the novelty of the proposed methods. Different contributions of our method compared with BLIP and LaCLIP.**

**BLIP[1]** proposes an efficient utilization method of noisy web data. It is first trained with noisy data, and then BLIP is used to generate a series of subtitles through a pre-trained Captioner, next, the generated subtitles are filtered through a pre-trained Filter, deleting the noisy subtitles from the original web and synthetic texts to get clean data. Finally, BLIP is trained again with clean data. 

**In contrast to the BLIP**, our method uses the rich knowledge of LLM/VLLM to enhance the text data and semantically and structurally rewrite the data without repeated model training. We pay more attention to constructing a more diverse and semantic dataset from the web noise dataset rather than the training strategy. Additionally, we conducted experiments comparing the training approaches of initializing the model with the backbone of the VLLM itself and training the multimodal representation model using the data generated by the LLM and VLLM. We find that utilizing the augmented data from a well-trained LLM and VLLM yields superior performance, further confirming the importance of LLM and VLLM in data construction.

**LaCLIP[2]** rewrites and changes the structure of text from the original text using LLaMA and chatbots but maintains semantic information of the text. Text rewritten in LaCLIP only ensures the semantics associated with the corresponding original text but it does not address the misalignment problem of image and text.

**We can supplement the shortcomings of LaCLIP** from the aspects of text augmented via VLLM. We combine our generated data with LaCLIP, and the average mAP on 10 image classification datasets is improved by 0.6% with our augmented data. Please refer to question 6 in the review YX2M.

**Reference:**

[1] Li J, Li D, Xiong C, et al. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation[C]//International Conference on Machine Learning. PMLR, 2022: 12888-12900.

[2] Fan L, Krishnan D, Isola P, et al. Improving CLIP Training with Language Rewrites[J]. arXiv preprint arXiv:2305.20088, 2023.

Reviewer comment (Experiments):
In DCTM: Previous works like OSCAR [1], ALBEF [2], and BLIP have empirically shown that diverse captions (one image with multiple captions) from sources like COCO and Flickr are effective in enhancing performance. These works treat each image-caption pair as unique; for instance, if one image comes with five captions as in the COCO setting, they construct five distinct pairs. The difference in the current approach is the use of diverse captions with a multi-positive contrastive loss. However, it remains unclear where the benefits of this approach specifically originate from. See question 1.

Author response:
> **Q3: The benefits from DCTM come from the data augmentations (use multiple captions) or from multi-positive contrastive loss.**

In our experiment, the data augmentations (use multiple captions) and multi-positive contrastive loss are all helpful in improving DS-CLIP performance. 

As shown in Tab.1, Experiment ID 4 represents that one text is sampled from multiple generating captions with the standard text-image contrastive loss. The performance of DS-CLIP is improved by 3.3% compared with without augmented captions as Experiment ID 3. Furthermore, we utilize multi-positive contrastive loss in DCTM for training as Eqn.1, Experiment ID 5 in Tab. 1 shows that multi-positive contrastive loss used in DCTM brings +0.6% improvement in the accuracy of classification. Therefore, using multiple captions and multi-positive contrastive loss in DCTM could implicitly help the model to improve the data diversity and align the image and text embeddings more efficaciously. These analyses are represented in Sec4.1 in the paper.

Reviewer comment (Novelty):
In MSTM: The utility of uni-modal contrastive losses in improving performance has already been showcased by ERNIE-VIL 2.0 [3].

Author response:
> **Q5: Limitation of MSTM: Uni-modal contrastive loss is showcased by ERNIE-VIL 2.0 [4].**

**ERNIE-VIL 2.0** proposed multi-view learning with image-to-text, text-to-image, image-to-image, and text-to-text contrastive loss, which is single and one-to-one self-supervision loss in-mode.

**Different from ERNIE-VIL 2.0**, MSTM is a multi-positive contrastive loss. We naturally utilize multi-positive contrastive loss to the pre-train model because of the generated multiple captions.

In particular, one image has $M+1$ multiple texts and each image with one corresponding text calculates a standard self-supervision loss, then the mean of $M+1$ Multiple Self-supervision Losses is the final loss

The purpose of MSTM is to enhance uni-modal representation significantly, and the performance of DS-CLIP with MSTM is greatly promoted as Experiment ID 6 in Tab.1 in the paper.

**Reference:**

[4] Shan B, Yin W, Sun Y, et al. ERNIE-ViL 2.0: Multi-view Contrastive Learning for Image-Text Pre-training[J]. arXiv preprint arXiv:2209.15270, 2022.

Reviewer comment (Experiments):
The benefits from DCTM comes from the data augmentations (use multiple captions) or from multi-positive contrastive loss? Moreover, what is the difference between multi-positive contrastive loss and supervised contrastive loss [4]? What is the advantage of using multi-positive contrastive loss?

Author response:
> **Q3: The benefits from DCTM come from the data augmentations (use multiple captions) or from multi-positive contrastive loss.**

In our experiment, the data augmentations (use multiple captions) and multi-positive contrastive loss are all helpful in improving DS-CLIP performance. 

As shown in Tab.1, Experiment ID 4 represents that one text is sampled from multiple generating captions with the standard text-image contrastive loss. The performance of DS-CLIP is improved by 3.3% compared with without augmented captions as Experiment ID 3. Furthermore, we utilize multi-positive contrastive loss in DCTM for training as Eqn.1, Experiment ID 5 in Tab. 1 shows that multi-positive contrastive loss used in DCTM brings +0.6% improvement in the accuracy of classification. Therefore, using multiple captions and multi-positive contrastive loss in DCTM could implicitly help the model to improve the data diversity and align the image and text embeddings more efficaciously. These analyses are represented in Sec4.1 in the paper.

> **Q4: The difference between multi-positive contrastive loss and supervised contrastive loss, and the advantage of using multi-positive contrastive loss.**


- **Supervised contrastive loss**[3] introduces a **fully supervised** setting into contrastive loss. It pulled samples with the same class together in embedding space and pushed apart clusters of samples from different classes combined with contrastive loss. 

- **Multi-positive contrastive loss** leverages the additional and richer supervision from the language modality to align the formation of image-text embeddings, which allows each image to be paired with all of the diverse texts describing its content. Training with multi-positive contrastive loss in our method is without requiring the label data. The main consideration is that of the alignment between image and text in cross-mode, in addition, we utilize MSTM to significantly enhance uni-modal representation. 

Through data clustering operation in DS-CLIP, each sample has a clustered label, and the supervised contrastive loss can also be used in DS-CLIP, the performance of DS-CLIP with combining supervised contrastive loss is further improved by 0.5% as shown in the following table. Multi-positive contrastive loss and Supervised Contrastive loss can be used together.

The advantage of multi-positive contrastive loss could implicitly help the model to align the image and text embeddings more efficaciously with diverse potential information of data. In future work, we will consider supervised contrastive loss in our method.

###### 								Tab.4: Ablation study with different contrastive losses on ImageNet(IN) for zero-shot image classification (%)

| DSTM | supervised contrastive loss | Top-1 |
| :-: | :-: | :-: |
| ✓ | ✗ | 38.5 |
| ✗ | ✓ | 38.2 |
| ✓ | ✓ | 39.0 |

**Reference:**

[3] Khosla P, Teterwak P, Wang C, et al. Supervised contrastive learning[J]. Advances in neural information processing systems, 2020, 33: 18661-18673.

Reviewer comment (Experiments):
Does it have to use both LLaVA and LLaMA? In table 2 (c), the gap between LLaVA only and LLaVA/LLaMA seems very marginal. Isn't it possible to use LLaVA only to generate diverse captions with proper prompts?

Author response:
> **Q2: Explanation of the effectiveness of using both LLaVA and LLaMA. Verify that it is possible to use LLaVA only to generate diverse captions with proper prompts.**

Thanks for your suggestion. The pre-trained dataset for DS-CLIP is YFCC15M in our experiments, whose text description is too short, and the generated texts according to LLaMA have fewer variations. Using LLaMA to generate text has less improvement on YFCC15M.

1. We also pre-train DS-CLIP based on ViTB/16 on CC3M with SE as shown in the following table. The text description of CC3M is longer and more abundant. The improvement is respectively 4.7% and 5.8% based on LLaMA and LLaVA. On the basis of LLaVA, the performance continues to increase by 1.6% combined with LLaMA.

2. We design several proper prompts for LLaVA to generate texts such as ''Can you explain this image in detail?'', ''Tell me in detail what is main object description in this picture.'' and ''What happened in the picture?'' Using these proper prompts to generate three texts for each image with LLaVA on the CC3M dataset, we conduct an experiment named LLaVA+ in the following table. The performance improves by 0.4% from 21.6% to 22.0% compared with text augmented from LLaVA, but it is still lower than text augmentation from LLaVA and LLaMA. The generated text from LLaVA has already described the image content in as much detail as possible, with more prompts to generate texts has fewer improvements.

###### 								Tab.3: Ablation study with text augmentation on ImageNet(IN) for zero-shot image classification (%)

| LLaMA | LLaVA | LLaVA+ | Top-1 |
| :-: | :-: | :-: | :-: |
| ✗ | ✗ | ✗ | 15.8 |
| ✓ | ✗ | ✗ | 20.5 |
| ✗ | ✓ | ✗ | 21.6 |
| ✓ | ✓ | ✗ | 23.2 |
| ✗ | ✗ | ✓ | 22.0 |

Reviewer comment (Presentation):
In Figure 3 and Figure 5, it seems that the boundaries are still indistinguishable.

Author response:
> **Q6: Boundaries are still indistinguishable in Figure 3 and Figure 5.**

The performance of DS-CLIP has significantly improved compared with baseline CLIP on various tasks. t-SNE is utilized to measure the quality of features learned. Figure 3 and Figure 5 show all classification datasets, we have a significant improvement and clear boundary for class 3 and class 5 in Figure 3.

However, such as ImageNetv2, pets, dogs, and DTD, which have a limited number of samples per class in the test set, making it difficult to generate more obvious visualizations. Other datasets contain enough samples per class (e.g. SUN397, Food101, CIFAR10, CIFAR100), which still have better boundaries and more distinct clusters compared to baseline CLIP. The results on the evaluation indicators are more intuitive to verify the effectiveness of DS-CLIP.

Reviewer comment (Writing):
There are several abbreviations introduced in the abstract and introduction, e.g., D^3, SE, DS-CLIP, DCTM, and MSTM. Additionally, those abbreviations seem to have a hierarchical structure, where DS-CLIP is for SE and D^3, SE is for DCTM and MSTM, which is really confusing.

Author response:
I'm sorry that these abbreviations have confused you.

In this abstract and introduction, **DS-CLIP** is a simple and efficient training algorithm for multimodel representation learning.

**Data De-Duplication ($\text{D}^3$)** is proposed to reduce training costs without lossing data diversity.

Specifically, we uniformly sample a certain percentage of data points from each pre-clustered center, producing a small subset of the dataset. Therefore, the duplicate data were reduced according to the above re-sampling at each training epoch.

**Semantic Enhancement (SE)** is proposed to address the issue of image-text misalignment and enrich textual diversity. Different from LaCLIP only with generated texts from LLM, we augment texts from the Large Language Model (LLM) and Visual Large Language Model (VLLM).

With generated multiple texts, we utilize the Diverse Captions Training Mechanism (DCTM) and Modality Self-enhancement Training Mechanism (MSTM) to learn a superior representation model.

**DCTM** is proposed to use all the generated captions simultaneously, this is shown to improve the performance in experiments. Concretely, it calculates the mean of the contrastive loss for the image to each corresponding text, and standard contrastive loss for the original text to the original image as shown in Eqn (1) and Eqn (2).

**MSTM** leverages self-supervision within each modality for self-enhancement, the purpose is the same with DCTM. Similar to DCTM, MSTM calculates the mean of the self-supervision loss for the multiple text pairs, and standard self-supervision loss for the image and its augmented image as shown in Eqn (3) and Eqn (4).

We add more explanation and remodify the logic of these representations of motivation. The revised motivation, core idea, and contributions about these abbreviations are shown in General Response.

Reviewer comment (Writing):
Several dataset abbreviations are introduced in Sec. 4.1. However, those abbreviations are used in Sec. 4.3. You'd better define them when used.

Author response:
We appreciate your constructive advice. We move these dataset abbreviations from Sec4.1 into Sec4.3 in the revised paper as follows. ''Evaluations are conducted on 10 widely used visual recognition benchmarks including ImageNet (IN), ImageNetV2 (INV2), CIFAR10 (C10), CIFAR100 (C100), Caltech101 (Cal101), Oxford Pets, SUN397 (S397), Food101 (F101), DTD and Stanford Dogs.''

Reviewer comment (Experiments):
The ablation results in Tab. 1 are hard to read. What's your default setting and final setting for those experiments?

Author response:
Thank you for your suggestions.

(1) We conduct extensive experiments related to our proposed $\text{D}^3$, SE, and training mechanisms DCTM and MSTM with different settings in Tab.1. The pre-trained dataset is YFCC15M for ViT-B/32 and ViT-B/16, and Laion400M is used for ViT-L/14. 

Tab.1 consists of **three parts** with the model architecture ViT-B/32, ViT-B/16, and ViT-L/14. 

**The first part** is designed to verify the effect of each component of the proposed module. 

**The second and third parts** are presented to prove that generated text from well-pre-trained VLLM is more helpful than training initialization based on the image encoder of VLLM in the aspect of the different model architecture. Specifically, the second part employs the equivalent parameters image encoder of BLIP[1] to initialize the image encoder ViT-B/16, and the third part employs the large image encoder of LLaVA to initialize the image encoder ViT-L/14. 

**In addition**, we combine three parts to prove that DS-CLIP is scalable for training on large datasets and models. We readjusted Tab.1 with clearer representations and added two experiments on ViT-L/14 aligned with Part 2.

The following analysis is modified in Sec4.2. 

(2) In particular, the baseline is the CLIP of Experiment ID 1 in Tab.1, which uses 100% data to pre-train the CLIP without $\text{D}^3$, SE, and training mechanisms DCTM and MSTM. The default setting is DS-CLIP of Experiment ID 6 in Tab.1 sampling 50% training data according to the pre-clustered prototype with SE and training mechanism DCTM and MSTM, which is used for image classification tasks as our default setting. For other downstream tasks, we use VIT-B/16 as Experiment ID 13 in Tab.1 to initialize the image encoder of CLIP for fair comparison following the original code of MIMdet, MMSegmentation, and CoCa, and other parameters are maintained.

The other default setting of the hyper-parameter clustered number $K$ is equal to 10000, using the image feature for clustering and uniform sampling type.

We will change it in the revised paper.

###### Tab.1: Ablation experiments on zero-shot ImageNet classification with our proposed components. All 100%, Random 50%, and Clustered 50% represent training with all data, randomly selecting 50% data for training and sampling 50% data according to the pre-clustered prototype during the training epoch. "--" is no training. The default setting of hyper-parameter clustered number K is equal to 10000, using the image feature for clustering and uniform sampling type.

| ID | Method | Init. of Image Enc. | $\text{D}^3$ | SE/LLaVA | SE/LLaMA | DCTM | MSTM | ImageNet Top-1 |
| --: | --- | --- | --- | --- | --- | --- | --- | --- |
| Dataset: YFCC15M; Model Architecture: ViT-B/32 |  |  |  |  |  |  |  |  |
| 1 | CLIP | ViT rand. | All 100% | ✗ | ✗ | ✗ | ✗ | 37.7 |
| 2 | CLIP | ViT rand. | Random 50% | ✗ | ✗ | ✗ | ✗ | 32.5 |
| 3 | DS-CLIP | ViT rand. | Clustered 50% | ✗ | ✗ | ✗ | ✗ | 34.6 |
| 4 | DS-CLIP | ViT rand. | Clustered 50% | ✓ | ✓ | ✗ | ✗ | 37.9 |
| 5 | DS-CLIP | ViT rand. | Clustered 50% | ✓ | ✓ | ✓ | ✗ | 38.5 |
| 6 | DS-CLIP | ViT rand. | Clustered 50% | ✓ | ✓ | ✓ | ✓ | 39.3 |
| 7 | DS-CLIP | ViT rand. | All 100% | ✓ | ✓ | ✗ | ✗ | 41.7 |
| 8 | DS-CLIP | ViT rand. | All 100% | ✓ | ✓ | ✓ | ✓ | **42.1** |
| Dataset: YFCC15M; Model Architecture: ViT-B/16 |  |  |  |  |  |  |  |  |
| 9 | CLIP | ViT rand. | All 100% | ✗ | ✗ | ✗ | ✗ | 43.4 |
| 10 | CLIP | BLIP init. | -- | ✗ | ✗ | ✗ | ✗ | 43.8 |
| 11 | CLIP | BLIP init. | All 100% | ✗ | ✗ | ✗ | ✗ | 44.3 |
| 12 | DS-CLIP | ViT rand. | All 100% | ✓ | ✗ | ✗ | ✗ | 45.7 |
| 13 | DS-CLIP | ViT rand. | Clustered 50% | ✓ | ✓ | ✓ | ✓ | 45.4 |
| 14 | DS-CLIP | ViT rand. | All 100% | ✓ | ✓ | ✓ | ✓ | **47.7** |
| Dataset: Laion400M; Model Architecture: ViT-L/14 |  |  |  |  |  |  |  |  |
| 15 | CLIP | ViT rand. | All 100% | ✗ | ✗ | ✗ | ✗ | 75.0 |
| 16 | CLIP | LLaVA init. | -- | ✗ | ✗ | ✗ | ✗ | 75.3 |
| 17 | CLIP | LLaVA init. | All 100% | ✗ | ✗ | ✗ | ✗ | 76.0 |
| 18 | DS-CLIP | ViT rand. | All 100% | ✓ | ✗ | ✗ | ✗ | 76.7 |
| 19 | DS-CLIP | ViT rand. | Clustered 50% | ✓ | ✓ | ✓ | ✓ | 76.1 |
| 20 | DS-CLIP | ViT rand. | All 100% | ✓ | ✓ | ✓ | ✓ | **78.7** |

Reviewer comment (Novelty):
While there are several techniques introduced in this paper, many of them are already proposed in prior arts. DCTM has been proposed in LaCLIP (Fan et al.). MSTM was introduced in DeCLIP (Li et al.). I can't find the main contribution of this paper. If those techniques are not your contribution, do not claim it. What's your main point and how does your main contribution affect the final performance?

Author response:
(1) Multimodal datasets are a critical component in recent breakthroughs such as CLIP and GPT4, but their design does not receive the same research attention as model architectures or training algorithms. 

In this paper, we **focus on** constructing semantically diverse multimodal datasets and achieving the first publicly available high-quality image-text dataset with diverse semantic captions . In particular, we utilize LLM and VLLM to augment a more diverse dataset. Although the LLM-generated data is presented in LaCLIP, it overlooks the importance of the visual LLM(VLLM) and still exists unmatched image-text pair in the organized data.

Additionally, we provide evidence to **support** that training on our newly constructed dataset leads to a model that surpassed the original VLLM model, further confirming the importance of LLM and VLLM in data construction. Please refer to the experiment 16 and 17 in Table 1 for details. 

The well-processed dataset will be released for further research. The motivation, idea, and contribution of DS-CLIP are re-summarised. Please refer to General Response.

(2) Each proposed module including $\text{D}^3$, SE, and training mechanism DCTM and MSTM has important effects. As presented in Sec4.1, when employing a 50% sampling ratio, the model trained with pre-clustered prototype sampling in **$\text{D}^3$** has a higher 2.1% compared with the model trained with random sampling. Combining **SE** with CLIP (Experiment ID 7) during training improves by 4% compared to baseline CLIP (Experiment ID 1). With training strategy **DCTM** and **MSTM**, the performance of DS-CLIP further improves by 1.4% (Experiment ID 4 vs Experiment ID 6). More analyses are presented in Sec4.1 in the revised paper.