Paper: Effective pruning of web-scale datasets based on complexity of concept clusters (ICLR-2024)

Reviewer comment (Experiments):
We cannot determine if the current data is effective for training ViT-B-16 and ViT-L-14 models.

Author response:
We have evaluated this question in our paper and would like to point the reviewer to Table 2 where we show that DBP outperforms CLIP-score filtering for different model sizes.
We also include the Table below for convenience.
| | Dataset | CLIP-S/32  | CLIP-B/32 | CLIP-L/14 |
|--|-|--|---|---|
| Method/Model | Size    | (63M params.) | (151M params.) | (428M params.) |
|  |   | IN top1 acc   | IN top1 acc    | IN top1 acc    |
| CLIP score   | 30M     | 32.32  | 38.07   | 47.61  |
| DB-Pruning   | 30M     | **39.04**   |**43.41** | **53.21**  |

Reviewer comment (Evaluation):
The majority of the experimental evaluation metric is Imagenet zero-shot top1. Is there any bias in the algorithm towards Imagenet data?

Author response:
We thank the reviewer for their comment and refer them to the newly added Tables 9 and 10 in the Appendix; we also included Table 10 in our general comment to all reviewers. In Table 9, we find that we outperform the best baseline released by DataComp on 35 out of 38 tasks. T-MARS have unfortunately neither released the numbers on all test sets nor the trained models which makes it impossible to compare to their results. In Table 10, we show results on the 38 tasks from DataComp for the models trained on LAION-CAT-440M, as well as on filtered versions of this dataset.

Reviewer comment (Evaluation):
Detailed results for each dataset in VTAB are not provided.

Author response:
We thank the reviewer for their comment and refer them to the newly added Tables 9 and 10 in the Appendix; we also included Table 10 in our general comment to all reviewers. In Table 9, we find that we outperform the best baseline released by DataComp on 35 out of 38 tasks. T-MARS have unfortunately neither released the numbers on all test sets nor the trained models which makes it impossible to compare to their results. In Table 10, we show results on the 38 tasks from DataComp for the models trained on LAION-CAT-440M, as well as on filtered versions of this dataset.

Reviewer comment (Experiments):
Does the robustness of model training change when the dataset size is reduced? Please provide robustness evaluation results.

Author response:
Thank you for raising this important concern. We added the following paragraph to our Results section:

The performance on retrieval and ImageNet distribution shifts is relatively lower compared to ImageNet zero-shot accuracy (Fig. 3). This trend is consistent across different baselines for retrieval tasks and we hypothesize that retrieval and ImageNet dist. shift tasks need relatively longer training (in Fig. 3 we reduce the number of training iterations/cost seen to ≤ 55.4%). To study this behavior, we measure the performance gains obtained with longer training. We increase the number of iterations for training on the 166M dataset from 41.6% (Fig. 3) to 69% and measure the difference in performance on each of the validation tasks as illustrated in Fig 4(right). We observe that among the four tasks (ImageNet, ImageNet dist. shift, VTAB, retrieval), the ImageNet dist. shift and retrieval tasks benefit the most from longer training. Therefore, we conclude that the observed performance drops on ImageNet dist. shifts and retrieval tasks can be attributed to shorter training.

Reviewer comment (Novelty):
The difference between DBP and SSP-Pruning is the choice of how many examples are taken from each cluster. It seems that innovation has some shortcomings.

Author response:
We would like to stress that scaling SSP-Pruning to LAION was highly untrivial in itself because 1) it required an initial step of deduplication as well as 2) the correct choice of the encoder to calculate the image / caption embeddings. We also believe that our idea to use cluster statistics to gauge the complexity of different concepts is novel and has not been proposed for dataset pruning.

Reviewer comment (Evaluation):
In Fig.3, 222M performs worse zero-shot accuracy than 166M, please give a reasonable explanation.

Author response:
Let’s recall that data quality is different from data quantity. In this work, we reduce the dataset size while maintaining and/or improving the quality of the data by balancing the data clusters and removing easy examples. This increases the marginal information gain the model gets from every training batch. 
We pruning a dataset we need to balance between the dataset size and the dataset quality. The 222M dataset (lower quality) and the 166M (higher quality) are good examples of this balance for the ImageNet classification task. We notice that same performance behavior when we look at the LAION-440M (lower quality) dataset and the filtered datasets (higher quality) in Fig 3. 

Some previous works like, [1] show theoretically and practically that balancing the data by “removing” examples from the majority groups/classes can result in a better worst group/ class performance and a better model even though the dataset size gets reduced. We added a discussion on this point to our Results section.

[1] Arjovsky et al. “Throwing away data improves worst-class error in imbalanced classification”, ICML 2023

Reviewer comment (Experiments):
Why weren't models of the same size, such as CLIP ViT-L-14, OPENCLIP LAION400M ViT-L-14, EVA ViT-L-14, and DINOv2 ViT-L-14, used for clustering comparisons?

Author response:
We agree with the reviewer that using models with the same size can be a good way to compare between models. In this work, we want to show that the quality of the clustering embeddings doesn’t depend on the model size only. For example, the Sentence BERT model is a very small text transformer (22M param.) that has been used by [2] for clustering, and we find that it performs better than much bigger models like CLIP or BLIP which have been trained using multimodal datasets. 

[2] Tirumala el.al.: “D4: Improving LLM pretraining via document de-duplication and diversification.”

Reviewer comment (Novelty):
The biggest concern I have is about the technical novelty. The main idea is based on SSP-Pruning. Therefore, the technical contribution is very trivial.

Author response:
We would like to stress that scaling SSP-Pruning to LAION was highly untrivial in itself because 1) it required an initial step of deduplication as well as 2) the correct choice of the encoder to calculate the image / caption embeddings. We also believe that our idea to use cluster statistics to gauge the complexity of different concepts is novel and has not been proposed for dataset pruning.

Reviewer comment (Experiments):
This work only reports the results on image classification. How about the zero-shot results of CLIP in retrieval tasks, such as COCO/Flickr30k image-to-text and text-to image retrieval? The zero-shot retrieval performance is also very important to measure the quality of pre-training and the effectiveness of the proposed method. The paper mentions about the retrieval task and shows some results in Figure 3 but I am not sure the implementation details: which dataset do they use? It would be helpful to report the results on standard benchmark (COCO/Flickr30k).

Author response:
We thank the reviewer for their comment and refer them to the newly added Tables 9 and 10 in the Appendix; we also included Table 10 in our general comment to all reviewers. In Table 9, we find that we outperform the best baseline released by DataComp on 35 out of 38 tasks. T-MARS have unfortunately neither released the numbers on all test sets nor the trained models which makes it impossible to compare to their results. In Table 10, we show results on the 38 tasks from DataComp for the models trained on LAION-CAT-440M, as well as on filtered versions of this dataset.

Reviewer comment (Evaluation):
The method involves CLIP-score filtering to curate a new dataset to train CLIP. I am unsure if such involvement is fair, as the filtering itself is similar to distilling some knowledge from a well-trained CLIP. Then, the new CLIP is somehow learning from the well-trained CLIP on large-scale datasets. Therefore, it may not be fair to claim the data efficiency for the proposed method.

Author response:
We thank the reviewer for raising this important concern. CLIP-score filtering has become a standard step in almost all data selection frameworks. For example, DataComp is becoming an established benchmark for testing data pruning methods at scale, and uses CLIP-score filtering as part of most of their baselines. Thus, we are comparing methods which all use CLIP-score filtering in some form. In our Table showing DataComp results and in our Figure 1 showing results on LAION, all baselines have also used CLIP-score filtering. In that sense, our approach is more efficient with respect to other methods which also use CLIP-score filtering.

Reviewer comment (Presentation):
The related work section contains excessive content, and there is a considerable amount of duplication. Besides, it is better to introduce “coreset selection[1-3]”, as it is closely related to your work.

Author response:
We thank the reviewer for their suggestions and apologise for missing these important papers. We added them to our related work section which now has a new paragraph titled “Data curation in supervised learning” where we included several other works in addition to the reviewer’s suggestions. We also removed redundancy between Introduction and Related Work.

Reviewer comment (Experiments):
Missing some baselines for comparison, including random selection and other data pruning methods [3-4].

Author response:
We thank the reviewer for their suggestions.
* Random selection: We excluded the random pruning baseline to save compute because it has been proven to be very weak. For example, [1] shows that random pruning consistently lags behind the SemDeDup baseline across various dataset sizes and evaluation tasks. Random selection has been tested by the DataComp creators on the “small” scale where it underperformed all other baselines (Table 12, https://arxiv.org/pdf/2304.14108.pdf). [1] Abbas et.al. 2023 https://arxiv.org/abs/2303.09540
* Comparison to [3]: 
    * **The task considered in [3] is not compatible with ours.** In [3], the authors tackle the problem of semi-supervised learning and select the coreset of the unlabeled data resulting in minimum labeled set loss when trained upon in a semi-supervised manner. In contrast, we consider the fully unsupervised setting where we train a CLIP model on a pre-chosen coreset. 
    * Further, **The dataset and model scales of [3] differ strongly from ours.** In [3], the authors perform experiments on small-scale datasets such as CIFAR10 and SVHN using WideResNets and LeNet architectures, while we perform experiments on web-scale datasets using transformer architectures. Sorscher et al. showed that methods which perform well on small-scale datasets do not scale to ImageNet, and we found scaling SSP-Prototypes up to LAION / DataComp highly untrivial. We hypothesize that scaling [3] to DataComp to enable a fair comparison will be highly involved, as well as it is not clear that the small-scale datasets are sufficiently large to train transformer models on.
* Missing comparison to [4]: While the setting of [4] is comparable to ours since the authors of [4] also study CLIP training on large-scale datasets, **their choice of datasets is different from ours**. Our base datasets are either LAION-CAT-440M (which is a filtered version of LAION-2B) or DataComp Medium, while [4] conduct CLIP experiments on CC3M. To enable a fair comparison, we would either need to scale [4] to DataComp or implement our method on CC3M; neither of these options is trivial and cannot be done during the short rebuttal period. If the reviewer would like to see results of DBP on CC3M, we would be happy to run these experiments until the camera ready version.

Reviewer comment (Writing):
The description in the methods section is very confusing.

Author response:
We rewrote and restructured the Methods section to improve clarity. In particular, we rewrote the paragraphs on deduplication and CLIP-score filtering. Following the suggestion of reviewer u5qD, we moved details on the QP program to the Appendix.

Reviewer comment (Experiments):
The tasks are relatively simple (ImageNet zero-hot). Could we compare DBP with other baselines on different tasks?

Author response:
We thank the reviewer for their comment and refer them to the newly added Tables 9 and 10 in the Appendix; we included Table 10 in our general comment to all reviewers. In Table 9, we find that we outperform the best baseline released by DataComp on 35 out of 38 tasks. T-MARS have unfortunately neither released the numbers on all test sets nor the trained models which makes it impossible to compare to their results. In Table 10, we show results on the 38 tasks from DataComp for the models trained on LAION-CAT-440M, as well as on filtered versions of this dataset.

Reviewer comment (Reproducibility):
In the methods section, the filtering pipeline includes deduplication, CLIP score filtering, and Density-Based Self-Supervised Prototypes pruning. However, there is no description about CLIP score filtering. The description about deduplication does not provide me with any useful information.

Author response:
We rewrote and restructured the Methods section to improve clarity. In particular, we rewrote the paragraphs on deduplication and CLIP-score filtering. Following the suggestion of reviewer u5qD, we moved details on the QP program to the Appendix.