Paper: Your CLIP Model Might Be Undertrained (ICLR-2024)

Reviewer comment (Experiments):
While the paper makes interesting observations, the paper is missing a lot of discussion and potential for extending the observations given the unused page limit. A few unanswered, immediate and simple questions: How many cycles are optimal? How long should each cycle be? Is there a relation between the number of samples and the length of the LR cycles? Figure 5 only presents results with one schedule.

Author response:
We thank the reviewer for their suggestions. We have indeed experimented with some of them.
 
*Re- number of extra cycles:* We have found in our investigation that additional cycles do not provide any further performance boost.

*Re- length of cycle:* We investigated in Figure 3 how changing the cycle duration affects performance. We can see that a small cycle of 3 epochs (on CC12M) was enough to get most of the gain.

*Re- relation with dataset size:* We have observed that for small datasets, 10 epochs was enough for both CC3M and CC12M (a dataset with 4x more samples). In particular, by applying a cycle with 10 epochs, we improve the ImageNet zero-shor accuracy of a ResNet-50 CLIP model by 4% when trained on CC3M (Table 7), and by 11% when trained on CC12M (Table 2).

Reviewer comment (Experiments):
Results in Figure 4 and 5 are limited to only one architecture and are not shown to hold for other architectures. Would these results hold for ViT-L/14?

Author response:
Our experiments on smaller datasets with several models suggest that the phenomenon is model-agnostic. For example, as we can see in Table 2, the ImageNet zero-shot accuracy of several CLIP models improves when applying our cycle: ResNet-50 by 11.3%, ViT-B-32 by 7.83% and ViT-B-16 by 8.4%. This improvement translates to other downstream tasks. We expect the same results with a ViT-L/14.

Reviewer comment (Experiments):
The final conclusion is that “CLIP models trained on large datasets are less likely to be undertrained.”. This is based on only one LR schedule and one model that does not provide definitive evidence for the conclusion.

Author response:
We agree with the reviewer that more analysis needs to be done at large scale to confirm. Unfortunately, such an analysis is beyond our compute capabilities, especially given the rebuttal period length. Our results from a single run on LAION-400M indicate only a little improvement by adding the extra training cycle, and as such, we think that applying our method to other large models would lead to the same results. We are happy to add more large-scale experiments for the final version if the reviewer thinks it would strengthen the paper.

Reviewer comment (Presentation):
Where are Figure 2 and Table 1?

Author response:
We thank the reviewer for pointing this out. We have a unified numbering system for figures and tables. We will fix that.

Reviewer comment (Novelty):
In my view, this paper presents a effective but somewhat hyperparameter training technique which is more of a engineering trick rather than pure novel contribution. In other words, this paper says that one should use a altered version of multi-cycle LR scheduler instead of a single-cyle LR schedule to improve CLIP performance.

Author response:
We fully agree with the reviewer that the method is a simple and straightforward baseline for CLIP. In fact, we have re-named the paper and positioned the method accordingly. Given its competitive performance, we believe it provides a better (and fast to compute) baseline for CLIP on smaller datasets.

Reviewer comment (Experiments):
This work shows performance comparison on zero-shot tasks, but it will be great to see how the learned embeddings provide benefits for adaptation tasks like linear probing or its use on downstream tasks which uses CLIP features in their framework.

Author response:
We thank the reviewer for these suggestions, and will incorporate the suggested experiments in our final paper. Sadly, we were not able to finish running all these experiments by the time of the rebuttal deadline, but we have some preliminary results that we believe are quite promising. Please refer to **[GP4]** for more zero-shot results and **[GP5]** for linear probing results.

Reviewer comment (Experiments):
Although competitive to prior methods trained on CC3M and CC12M, it is unclear how this technique performs when combined on pre-trained models of prior methods. For example, does this technique shows complementary effect when plugged on CyCLIP pretrained model? (using same CyCLIP objective functions even in the proposed technique)

Author response:
We thank the authors for their suggestion. We have indeed experimented a bit with this idea and found out that applying the extra training cycle on top of other approaches leads to the same results as applying the extra training cycle by itself. Although we were not able to run conclusive enough experiments to update the draft of our paper, we are happy to include these results in the final version of the paper.

Reviewer comment (Experiments):
Similar results with-and without this technique on large scale CLIP models like CLIP-LAION400M shows that this trick is only valid for small scale models with less than 50% baseline accuracy.

Author response:
We agree with the reviewer that the finding does not translate to large datasets. However, we believe that the smaller data regime is definitely interesting. Training CLIP models on large datasets is expensive and only large companies and a very limited number of academic labs afford it. Most academic labs however cannot train CLIP models except on smaller datasets. As such, when such works propose methods to improve the performance of CLIP models, it is necessary to consider better baselines to validate the effectiveness of the proposed method, and to increase our confidence that it would work on large datasets.
 
We believe our approach can provide such a baseline given its simplicity.

Reviewer comment (Presentation):
There are missing tables in the paper. For example, I cannot see Table 1 anywhere in the paper.

Author response:
We thank the reviewer for pointing this out. We were using a unified numbering system for figures and tables (so Figure 1, Table 2, Figure 3, etc.), but will change it for the final version.

Reviewer comment (Novelty):
My main concern is that this paper mainly shows a training trick, which poses questions for novelty of this work.

Author response:
We fully agree with the reviewer that the method is a simple and straightforward baseline for CLIP. In fact, we have re-named the paper and positioned the method accordingly. Given its competitive performance, we believe it provides a better (and fast to compute) baseline for CLIP on smaller datasets.

Reviewer comment (Theory):
There is not much analysis / theory on why such behavior exists (or not) on different scales of datasets.

Author response:
We refer the reviewer to **[GP2]** above—although more theory CLIP models are known to require a lot of training data to achieve a good performance. We believe that when the training data is limited, it is easy for the model to be stuck in a local optimum. As the dataset size increases, the sheer number of training data points leads to a better generalization.

Reviewer comment (Experiments):
It would be interesting to see how the proposed schedule helps when we apply to other approaches in Table 7.

Author response:
We thank the authors for their suggestion. We have indeed experimented a bit with this idea and found out that applying the extra training cycle on top of other approaches leads to the same results as applying the extra training cycle by itself. Although we were not able to run conclusive enough experiments to update the draft of our paper, we are happy to include these results in the final version of the paper.

Reviewer comment (Experiments):
Why is the proposed observation of under-training only present with the smaller dataset e.g. CC3M but not with the larger ones like LAION-400M? It'd be very helpful if the authors can provide some insights here, because this seems to be the core contribution of the paper. For example, does the noise level in the image-text dataset affect the fitting behavior in some way?

Author response:
We refer the reviewer to the general points **[GP2]** above. In short, additional experimentation rules out factors such as label noise or model capacity as the underlying causes of undertraining. These results suggest that the root of undertraining might simply be the number of data points (e.g., the model is more likely to be stuck in some local optimum).

Reviewer comment (Experiments):
Many existing CLIP recipes rely on high-resolution finetuning after the low-resolution pretraining (e.g.ALIGN, CoCa), which is essentially doing the same finetuning with the extra cycle as shown in Figure 1 (right). Would the proposed approach still benefit models that are already trained with high-res finetuning?

Author response:
Good observation! As far as we know, fine-tuning on higher resolution images is usually employed with large datasets (where under-training is not an issue). At small scale, we suspect that it would provide similar benefits, provided that the learning rate scheduler and optimizer state are reset.

Reviewer comment (Experiments):
For ablation, I think it'd be cleaner to compare with a baseline that is trained on the same number of epochs so that the only difference is the learning schedule (instead of adding additional training epochs).

Author response:
We thank the reviewer for their recommendation. We have trained several baseline models, on different number of epochs, ranging from 30 to 200, on CC3M and CC12M. We have observed that all these models have almost the same downstream performance at the end of their training.

Reviewer comment (Writing):
The paper writing can use some improvement to expand on the introduction, method, analysis, and related work (still plenty of space left).

Author response:
We thank the author for their suggestion. We have indeed significantly revised the paper to include more details.