Paper: Label-efficient Training of Small Task-specific Models by Leveraging Vision Foundation Models (ICLR-2024)

Reviewer comment (Experiments):
The paper could benefit from a comparative analysis with a few other knowledge transfer approaches. This would help provide a clearer picture of how the proposed method stands against existing techniques.

Author response:
**Additional comparisons:**
ImageNet pretraining and CLIP pretraining are the most popular and widely-used pretraining approaches in the context of transfer learning. Hence, we used them for comparison in this paper. To be more comprehensive, we performed additional experiments comparing the proposed approach with an alternative transfer learning strategy: self-supervised pretraining followed by finetuning. Please see the results here: https://openreview.net/forum?id=HnVtsfyvap&noteId=7vBTW2DhMA

Our main goal is to establish “Adapt VFM → Transfer knowledge across architectures → Finetune small model” as a highly effective strategy for training small task-specific models. In this work, we used supervised finetuning strategy for adapting VFMs and feature/task-prediction matching for transferring knowledge across architectures, and showed that we already outperform other widely-used approaches. More sophisticated finetuning/distillation approaches can easily be incorporated into our general strategy to further improve the performance.

Reviewer comment (Evaluation):
The paper mentions that the target models may inherit the biases of the foundation models. It would be beneficial to discuss potential ways to address this issue.

Author response:
**Biases:**
A VFM pretrained on web data may have its own inherent biases that may be useful/harmful for a particular target task. The proposed approach advocates for adapting pretrained VFMs to target task/domain before transferring their knowledge to small models. This strategy gives us an opportunity to enhance/suppress desired/irrelevant knowledge in VFMs by using appropriate target domain data. Using a well-curated and balanced target domain set for knowledge transfer can also help us mitigate some of the biases. This is an interesting line of research for followup works.

Reviewer comment (Reproducibility):
Lack of detail on fine-tuning VFMs: The paper mentions using labeled target task data to fine-tune VFMs but doesn't provide much detail on this process. More information on how VFMs are fine-tuned could help readers better understand the complete methodology.

Author response:
**Details of finetuning VFMs:**
The process of finetuning VFMs is similar to the process of finetuning small models. We attach a randomly initialized task-specific head to a pretrained VFM encoder, and finetune the entire model. The details of task-specific heads are described in Appendix A.1. The optimizer,  learning rates, number of epochs and image augmentations used for finetuning VFMs are same as the ones used for finetuning small models. Please see Appendix A.2 for details. We will make these details clearer in the final draft. Please let us know if there are any specific details that we may have missed.

Reviewer comment (Novelty):
Most of the findings in the paper are obvious like task oriented or task agnostic knowledge transfer would lead to better performance than model pre-trained on CLIP or Imagenet because we are creating transfer set similar to target dataset.

Author response:
**Obvious findings:**
As far as we know, there is no existing work that concretely establishes the findings presented in this paper about leveraging VFMs for training mobile models on a new task with limited labeled data. Especially, we show that

1. Knowledge transfer (both task-oriented and task-agnostic) from a VFM (even with generic CC3M transfer set) is a better pre-training strategy for small models when compared to supervised ImageNet or web-scale CLIP pretraining. 
2. Task-oriented knowledge transfer from VFMs is better than task agnostic transfer. 

The reviewer seems to be under the impression that proposed knowledge transfer outperforms ImageNet/CLIP pretraining because we are curating transfer sets. We would like to emphasize that knowledge transfer outperforms ImageNet/CLIP pretraining even when using the generic CC3M transfer set. A curated transfer set just makes the performance gap even bigger. We strongly believe that these non-trivial findings corroborated by our thorough experimental validation would be very useful to the community, especially given that VFMs are becoming larger and difficult to deploy as time progresses.

Reviewer comment (Experiments):
Transfer set creation process is mostly heuristic based and it's one of the important factors to get good performance on the target task.

Author response:
**Transfer set creation:**
The main focus of this paper is on leveraging VFMs to effectively pretrain small task-specific models, not the specific strategy used for curating transfer set. Our results on ADE20K dataset show that a simple image retrieval approach can also be effective in curating good transfer sets. We agree that a  more sophisticated curation process could further improve the results. However, that is not the main focus of this paper and we plan to explore this direction in our followup works.

Reviewer comment (Evaluation):
Based on the figure 5 (left) if we use random transfer set the performance of the model on target task is similar to models pretrained on Imagenet or CLIP based pretraining.

Author response:
**Random transfer set performance:**
Random transfer set performance in Fig. 5(a) is not similar to ImageNet/CLIP pretraining as claimed by the reviewer. It indeed outperforms ImageNet/CLIP pretraining. Below are the ADE20K segmentation performances when using 4800 labeled images:

| Fig. 5(a) - Random subset of YFCC (154K images) | 35.7 |
|---|---|
| Table 3 - ImageNet pretraining | 33.1 |
| Table 3 - CLIP pretraining | 27.6 |

Reviewer comment (Novelty):
Lacking Novelty: In essence, the proposed approach is knowledge distillation. Under the setup the authors consider, the proposed approach is the most straightforward approach one could consider. While the investigation of the use of not in-domain transfer dataset is interesting, the finding — when sufficient in amount, in-domain transfer set is better — is somewhat expected (e.g prior work[1] on open-set semi-supervised learning has shown that out-of-domain example can hurt performance of semi-supervised algorithm). It would be more interesting if the authors investigated the use of data from different domains (medical, satellite, clipart, etc) for distillation and proposed a strategy for selecting the right data for distillation. The current submission does contain some investigation on selecting the right data for distillation (section 3.5), but the investigation is not thorough enough to convince the reviewer that their strategy is applicable for different target tasks and with data from different domains.

Author response:
**Novelty:**
As far as we know, there is no existing work that concretely establishes the findings presented in this paper about leveraging VFMs for training small models on a new task with limited labeled data. Especially, we show that

1. Knowledge transfer (both task-oriented and task-agnostic) from a VFM (even with generic CC3M transfer set) is a better pre-training strategy for small models when compared to ImageNet/CLIP pretraining. 
2. Task-oriented knowledge transfer from VFMs is better than task-agnostic transfer. 

We strongly believe that our findings corroborated by thorough experimental validation would be very useful to the community, especially given that VFMs are becoming larger and difficult to deploy as time progresses. 

The main focus of this paper is on leveraging VFMs to effectively pretrain small task-specific models, not the specific strategy used for curating the transfer set. Please note that knowledge transfer is better than ImageNet/CLIP pretraining even when generic CC3M transfer set is used. So, it is not necessary to use task-related transfer set to outperform ImageNet/CLIP pretraining. Using a task-related transfer set just makes the performance gap bigger. Our results on ADE20K show that a simple image retrieval approach can be effective in curating good transfer sets. While a more sophisticated curation process could further improve the results and **transfer set curation** is an interesting topic worth exploring in further detail, it is not the main focus of this work. However, we agree that it would be an excellent topic for followup works.

We appreciate the reviewer's effort in bringing [1] to our attention. We will include related discussion in our revised version. Similar to our work, [1] focuses on learning a small task-specific model with limited labeled data, and shows that knowledge distillation from the right teacher model (the focus of their work) is better than ImageNet pretraining. However, there are many differences between our work and [1], and our work conveys several important and novel messages that are missing in [1]:

**Comparison with CLIP**  [1] does not compare with CLIP training. We show that knowledge transfer from VFMs outperforms CLIP training. We would like to respectfully disagree with the reviewer comment "It is not surprising that similar results would hold for CLIP". None of the existing works show how effective web-scale CLIP pretraining is for transfer learning in the context of small models. In fact, one would have expected that due to the scale of data, CLIP pretraining may outperform ImageNet pretraining and may even be competitive with knowledge transfer. **Our experiments surprisingly show the opposite trends**.

**Works without target domain unlabeled data** [1] assumes the availability of target domain unlabeled dataset. In contrast, we show that even when target-task unlabeled data is unavailable, knowledge transfer from VFMs using generic internet imagery is highly effective. We think **this is interesting and beyond expectation**. Note that HAM10K data is quite different from typical internet imagery and still knowledge transfer using CC3M dataset outperforms other popular pretraining strategies.

**Experiments with VFMs**
Note that [1] does not experiment with VFMs. Their teacher models are relatively small CNNs trained on small-scale classification datasets. So, the utility of [1] is unclear in the context of large scale VFMs trained on web-scale datasets. Different from [1], we experiment with VFMs and present many interesting results.

 **Transfer set curation** We a present a simple and effective transfer set curation method using retrieval, and further demonstrate the importance of a balanced curation with respect to seed query set (Figure 5). In contrast, [1] assumes that a large target domain unlabeled dataset is always available.

**Knowledge transfer formulation** [1] assumes that source models are classification models. They use a distillation loss that matches the target and source model predictions in the label space of source model (see Eq. (3) in [1]).  Unfortunately, most of the existing VFMs (CLIP, DINOV2, SAM, etc.) are not supervised classification models, and the formulation of [1] is not directly applicable to them. One could extend [1] to VFMs by using feature matching instead of source label matching as the loss function. Then, it becomes similar to our task-agnostic transfer. As shown in our results, task-agnostic transfer is inferior to task-oriented transfer.

**Works well for segmentation task** [1] only experimented with classification tasks. We show the effectiveness of knowledge transfer from VFMs for semantic segmentation task.

**Comparison with SSL** [1] does not compare with the DINO, which is a popular SSL pretraining approach. Our new results show that knowledge transfer from VFMs is significantly better than DINO pretraining.

Reviewer comment (Experiments):
Problematic Baselines: One would consider using the task-agnostic approach if fine-tuning the pre-trained models could be an issue (perhaps due to compute-constraint) but since the authors only consider compute-constraint during inference (instead of training), it seems unnatural to consider task-agnostic distillation as a baseline. In addition, the CLIP-pretrain baseline is trained on an internal dataset. Without knowing the distribution/statistics of the data and how the dataset compares to CC3M or DataComp-1B, it is difficult to understand whether the CLIP-pretrain baseline is worse because contrastive pretraining is not the right approach or the data used for training is flawed.

Author response:
**Baselines:**
Sorry to say that we did not understand why task-agnostic distillation is “unnatural”.  Task-agnostic distillation is a valid approach to leverage VFMs, and hence we compared it with the proposed task-oriented knowledge transfer in a fair manner using same amount of labeled and unlabeled data.

**CLIP dataset:**
We use the 1.1B image-text pairs used in the following papers: “STAIR: Learning Sparse Text and Image Representation in Grounded Tokens” and “RangeAugment: Efficient Online Augmentation with Range Learning”.
ViT-B/16 and ViT-H/16 CLIP models trained on this dataset achieve 72.8% and 78.6% ImageNet zero-shot accuracy which are comparable to OpenCLIP models trained on DataComp-1B. These results confirm that the dataset we use is a high quality dataset.

**Training compute** Note that our work focuses on inference time compute constraints and hence on small task-specific models. While experimenting under a fixed training budget is also an interesting problem, that is not the focus of this work. That said, finetuning the VFMs we used in this work is not as costly as the reviewer might be thinking. The key thing to notice here is that **VFMs are fine-tuned only on a small labeled dataset**. The actual compute used for finetuning VFM is a small fraction of the compute used for distillation in our experiments. For example, finetuning OpenLCIP-ViT-L on Places365 dataset with 50 images/class for 200 epochs takes around 16 A100 GPUh, and distilling frozen OpenCLIP model on full unlabeled Places dataset for 200 epochs takes around 370 A100 GPUh. Also, as VFMs become larger and larger in the future, one could use efficient fine-tuning approaches, such as LoRa[2], which have been shown to be quite effective in the NLP community.

Reviewer comment (Experiments):
Have the authors tried distilling from the patch tokens from CLIP?

Author response:
**Distilling patch tokens from CLIP:**
We tried this in our early experiments but did not observe promising results. DINOV2 patch features are significantly better than OpenCLIP patch features for segmentation. This is because DINOV2 has been explicitly trained to produce good patch features while OpenCLIP was trained only with a loss on CLS features. Below are the performances of OpenCLIP-ViT-L/14 and DINOV2-ViT-L/14 models on the ADE20K segmentation dataset with Deeplab-V3 segmentation head:

| Labeled images | 1200 | 2401 | 4802 | 9605 |
|---|---|---|---|---|
| OpenCLIP | 31.82 | 36.69 | 40.22 | 42.85 | 
| DINOV2 |  39.32 | 42.76 | 46.35 | 49.42 |

DINOV2 outperforms OpenCLIP by a significant margin. Hence, we did not further explore the direction of distilling inferior CLIP patch features. The following paper also observed that patch features of CLIP models trained with CLS token are not good for dense predictions tasks: “Perceptual Grouping in Contrastive Vision-Language Models”.

Reviewer comment (Experiments):
Additional baseline: CLIP is a weakly supervised pre-training approach, the reviewer recommends adding self-supervised pre-training approaches such as DINO.

Author response:
**Comparison with DINO:** Please see https://openreview.net/forum?id=HnVtsfyvap&noteId=7vBTW2DhMA

**Comparison with SSL** [1] does not compare with the DINO, which is a popular SSL pretraining approach. Our new results show that knowledge transfer from VFMs is significantly better than DINO pretraining.

Reviewer comment (Experiments):
Transfer set: The use of not in-domain transfer set is worth more explanation. Right now, there is not much explanation/description on the use of not in-domain transfer set (e.g. why an in-domain transfer set is difficult to obtain).

Author response:
**Not in-domain transfer set:**
If we have large enough task-related data, we should always use that as transfer set. However, the amount of readily available task-related data depends on the domain/task. For example, the number of images in AD20K dataset is around 20K which is not large enough as demonstrated by our experiments. We get better results by using not in-domain CC3M transfer set instead of the limited in-domain ADE20K dataset (see Fig. 4). We will make this more clear in the final draft.

**Works without target domain unlabeled data** [1] assumes the availability of target domain unlabeled dataset. In contrast, we show that even when target-task unlabeled data is unavailable, knowledge transfer from VFMs using generic internet imagery is highly effective. We think **this is interesting and beyond expectation**. Note that HAM10K data is quite different from typical internet imagery and still knowledge transfer using CC3M dataset outperforms other popular pretraining strategies.

Reviewer comment (Reproducibility):
Distillation mechanism: Distilling patch features is important for the segmentation task. However, there are only a few sentences describing the process. The reviewer recommends expanding the section on loss function either via using equations (the use of contrastive loss for distillation is uncommon) or by providing a figure explaining the process.

Author response:
**Distillation loss:**
We would like to clarify that we do not use contrastive loss for distilling patch features. As mentioned in Sec 3.1 (towards the end of the paragraph on loss function), we use standard cosine similarly loss for patch feature distillation. Contrastive loss is used when distilling global image features following the popular contrastive representation distillation approach. Yonglong Tian, Dilip Krishnan, Phillip Isola, “Contrastive Representation Distillation”, ICLR 2020. We will describe the loss functions in further detail in the final version.