Paper: Data Filtering Networks (ICLR-2024)

Reviewer comment (Reproducibility):
The HQITP-350M dataset is the key to the success of the DFN, so it is disappointing that it cannot be publicly accessible by the research community.

Author response:
Thank you for your review. We are happy that you find our work interesting and the results impressive. We hope to address your remaining concerns.

To address your questions and concerns about HQITP we provide a copy of our general response from above.
We provide some clarity about the HQITP-350M dataset and what we omitted from the paper.

* While we cannot release HQITP-350M for legal purposes, we have shown it is possible to build a DFN better than the existing OAI-CLIP models using only publicly available data. While we only did this up to the “large” DataComp scale in the original submission, we extend this to the XL scale below and compare it to the previous state of the art filtering network (OAI-CLIP-B/32). Our DFN with none of the additional modifications (fine-tuning/open-ai-init/augmentation) from section 4 in the paper outperforms the OAI DFN.
* Because HQITP-350M dataset would be similar in nature to licensed stock image datasets, so we don’t have access to the exact labeling decision behind each image caption. We also note we will release the dataset induced by our best performing DFN which is sufficient for the community to train strong models.
* In table 2 (above), We provide an additional ablation below where we control the number of examples from HQITP used to train the DFN and measure its performance on the datacomp benchmark (at the medium scale). We show that one can achieve most of the gains with a fraction of the of the full dataset. We note this experiment was done with an earlier version of HQITP that only contained 135M examples.

Reviewer comment (Experiments):
It seems like the improvement in performance over other DFNs is primarily in ImageNet (and distribution shift) Zero-Shot as well as Retrieval. It seems like without "finetuning" on MS COCO training set, Flickr30k training set, and ImageNet 1K training set with ImageNet templates as captions, these performance improvements nearly disappear. While the main paper does ablate over fine-tuning, it doesn't highlight just how significant fine-tuning on these particular datasets is (primarily because these are the datasets we wish to evaluate on).

Author response:
## Overfitting to ImageNet/COCO/FLICKR
We understand that the reviewer is concerned about potential overfitting to ImageNet because our filtering network trains on ImageNet + Coco + Flickr (while our data pool is de-duplicated against these datasets). To address this we first provide a full results table to provide an in depth analysis of results and provide ablations at a higher compute scale that show even less signs of overfitting to ImageNet/COCO/Flickr. We also add that while fine-tuning does help performance on a set of key benchmarks, our non-finetuned DFN consistently outperforms the best and most widely used DFN today (the OAI-CLIP-ViT-B//32).
We apologize for only providing averages of our evaluation which could cloud some of the takeaways. We provide a full results table (Table 3 and 4 above) for our best performing and comparable external models on the full evaluation suite. We will update our final manuscript to have links to these full results tables. We would like to highlight three points from this full results analysis
1. Performance on several of the VTAB tasks (Cifar-10, Caltech-101, STL-10)  are close to saturated so making large improvements across all the tasks can be difficult, and even a 0.5% average improvement can be meaningful. We do agree that performance on a few high quality benchmarks may be more indicative of performance improvements.
2. Our benchmark contains 2 additional high quality test sets specifically for highlighting biases in traditional imagenet style models DollarStreet and GeODE, this is accounted for in the average calculation but not VTAB calculation.
3. We note the retrieval benchmark average is an average of 3 tasks (Flickr, Coco and WinoGAVIL). WinoGAVIL was a retrieval benchmark released in 2022 containing images from google images and de-duplicated from our training set.

With the in-depth results table in mind, we provide 3 additional models that provide evidence that overfitting to ImageNet/CoCO/Flickr is not a significant concern.
1. First we scaled up our ViT-H/14 method further with 1 epoch of high resolution fine-tuning and compare our results across all categories to a state of the art CLIP model that performs inference at a higher resolution: SiGLIP (particularly the ViT-SO400M-14-SigLIP variant). We show that not only does our approach out-perform SiGLIP on ImageNet (84.4 vs 83.1) and average across the 38 benchmarks (71.0 vs 69.4), our performance gap on VTAB is greater than that on ImageNet or Retrieval (68.5 vs 64.6). We show strong performance on DollarStreet and GeODE, slightly under-performing SiGLIP on the former, and over-performing on the latter. We additionally note that this is all with the SigLIP model using a more efficient architecture [2] and being trained using the same compute budget.
2. We agree our ViT-B/16 results seem to indicate we are “weakening” the model somehow by training the DFN on ImageNet, we show that at larger scale this phenomena reverses. We provide results of a ViT-L at XL datacomp-scale induced by the DFN without the fine-tuning on ImageNet/COCO/FLICKR. We see that the *average* performance of the model is higher for the finetuned-DFN model compared to the “base” one (that does not finetune on IN). In addition we see improvements in both WinoGAVIL (+3pp) , DollarStreet(+2pp) and a slight decrease on GeoDE (-0.7pp). 
3. Finally we take our XL ViT-L model and continue training it for 3x the number of iterations (39B examples seen) and we see that while we get a slight performance win on ImageNet (+0.7pp), we get a larger gain on other aspects of our benchmark such as  +1.2pp on average, +0.9pp on VTAB, +1.5pp on Geode and a slight performance degradation on DollarStreet (-0.5pp).

Reviewer comment (Novelty):
Personally, I don't find the conclusion that training a DFN on high-quality image-caption datasets allows the induced subsets to be higher quality very surprising or interesting. I do think a fine-grained study into specific properties of the pretraining data (e.g. similarity with downstream tasks' data, diversity of data, data imbalance) can make this work interesting and help further our understanding of DFNs. In its current form, apart from the observation on ImageNet accuracy of DFN and that of the model trained on the induced dataset, I don't see what the contribution of this work is towards understanding DFNs.

Author response:
While some may not find these results surprising, we believe this is the first work that trains a CLIP model specifically for the purpose of filtering data for large-scale dataset creation, whereas all prior work relies on using an OpenAI CLIP model to filter the data.

In the paper we provided analysis for understanding DFNs beyond showing that better ImageNet CLIP models do not result in better filtering models. We demonstrated that data quality is more important for training filtering models than for training standard models. Adding model filtered data from a raw pool is a noisy process that can still include bad training points, and these points negatively affect a model used for filtering much more than it affects standard models. For additional details, see Figure 4 and Section 3.3 in the paper.

Reviewer comment (Evaluation):
While I recognize the improvement in accuracy on ImageNet, Retrieval etc., the importance of fine-tuning the DFN on the evaluation datasets leads me to believe that this approach may not generalize to other downstream tasks (as evidenced by the < 1% improvement on VTAB).

Author response:
## Overfitting to ImageNet/COCO/FLICKR
We understand that the reviewer is concerned about potential overfitting to ImageNet because our filtering network trains on ImageNet + Coco + Flickr (while our data pool is de-duplicated against these datasets). To address this we first provide a full results table to provide an in depth analysis of results and provide ablations at a higher compute scale that show even less signs of overfitting to ImageNet/COCO/Flickr. We also add that while fine-tuning does help performance on a set of key benchmarks, our non-finetuned DFN consistently outperforms the best and most widely used DFN today (the OAI-CLIP-ViT-B//32).
We apologize for only providing averages of our evaluation which could cloud some of the takeaways. We provide a full results table (Table 3 and 4 above) for our best performing and comparable external models on the full evaluation suite. We will update our final manuscript to have links to these full results tables. We would like to highlight three points from this full results analysis
1. Performance on several of the VTAB tasks (Cifar-10, Caltech-101, STL-10)  are close to saturated so making large improvements across all the tasks can be difficult, and even a 0.5% average improvement can be meaningful. We do agree that performance on a few high quality benchmarks may be more indicative of performance improvements.
2. Our benchmark contains 2 additional high quality test sets specifically for highlighting biases in traditional imagenet style models DollarStreet and GeODE, this is accounted for in the average calculation but not VTAB calculation.
3. We note the retrieval benchmark average is an average of 3 tasks (Flickr, Coco and WinoGAVIL). WinoGAVIL was a retrieval benchmark released in 2022 containing images from google images and de-duplicated from our training set.

With the in-depth results table in mind, we provide 3 additional models that provide evidence that overfitting to ImageNet/CoCO/Flickr is not a significant concern.
1. First we scaled up our ViT-H/14 method further with 1 epoch of high resolution fine-tuning and compare our results across all categories to a state of the art CLIP model that performs inference at a higher resolution: SiGLIP (particularly the ViT-SO400M-14-SigLIP variant). We show that not only does our approach out-perform SiGLIP on ImageNet (84.4 vs 83.1) and average across the 38 benchmarks (71.0 vs 69.4), our performance gap on VTAB is greater than that on ImageNet or Retrieval (68.5 vs 64.6). We show strong performance on DollarStreet and GeODE, slightly under-performing SiGLIP on the former, and over-performing on the latter. We additionally note that this is all with the SigLIP model using a more efficient architecture [2] and being trained using the same compute budget.
2. We agree our ViT-B/16 results seem to indicate we are “weakening” the model somehow by training the DFN on ImageNet, we show that at larger scale this phenomena reverses. We provide results of a ViT-L at XL datacomp-scale induced by the DFN without the fine-tuning on ImageNet/COCO/FLICKR. We see that the *average* performance of the model is higher for the finetuned-DFN model compared to the “base” one (that does not finetune on IN). In addition we see improvements in both WinoGAVIL (+3pp) , DollarStreet(+2pp) and a slight decrease on GeoDE (-0.7pp). 
3. Finally we take our XL ViT-L model and continue training it for 3x the number of iterations (39B examples seen) and we see that while we get a slight performance win on ImageNet (+0.7pp), we get a larger gain on other aspects of our benchmark such as  +1.2pp on average, +0.9pp on VTAB, +1.5pp on Geode and a slight performance degradation on DollarStreet (-0.5pp).

Reviewer comment (Novelty):
The proposed approach is still a heuristic for filtering data, and is limited to pointwise filtering for the sake of parallelizability.

Author response:
We agree that pointwise filtering is limited, but is computationally important due to parallelizability; however, this approach can be combined with other dataset filtering approaches, such as restricting keyword counts in the text (e.g. MetaCLIP).

Reviewer comment (Presentation):
Can you highlight the cost of filtering in e.g. Figure 1?

Author response:
While the cost of filtering is not reflected in Figure 1, DFN, DC-1B, and LAION all rely on CLIP filtering to filter the dataset. DFN and LAION use a ViT B/32 model, which is cheaper (both training and inference) than the ViT-L/14 used in DC-1B. Additionally, the DFN we train for filtering is trained for less samples seen than the OpenAI models used to filter DC-1B and LAION (5B vs 12.8B). Finally, the cost of training a ViT-B/32 (4.4 GMAC at 224px) is much smaller than that of training the best models, such as a ViT-L/14 (81.1 GMAC at 224px) and a ViT-H/14 (167.4 GMAC at 224px). So the total cost of training our best filtering model is 22e9 GMACs, while the total cost of training our ViT-L and ViT-H models are 1e12 GMACs and  6.5e12 GMACS respectively. This corresponds to a filtering overhead of 2% and 0.3% respectively. Unfortunately, we are unable to directly compare with approaches used in proprietary datasets like OAI-WIT-400M and WebLI, as their creation processes are not public knowledge.