Paper: Don't Pre-train, Teach Your Small Model (ICLR-2024)

Reviewer comment (Experiments):
Baselines considered for experiments seems weaker. I would have liked to see the comparison between student model which is pretrained and fully fine tuned vs DPT (2x).

Author response:
A fully finetuned student would be an interesting data point. Intuitively, we believe that it would achieve better performance, but would be the most expensive method on our cost plots. In addition, this would really limit the student to being good on only one task due to the backbone distortion mentioned in section 3.1. However, if you believe that this data would significantly strengthen the main intention of our paper, we can provide it.

Reviewer comment (Experiments):
Datasets considered in the paper are such that we can easily use generative model to create more data. The approach might not give good performance in cases where we don't get good images using generative models.

Author response:
*Sometimes* data generation is a key step. Importantly, we find that the synthetic data component is the most significant in the data scarce scenarios. As we can see (Table 3), the datasets with plenty of samples (CIFAR-10, CIFAR-100, CUB-2011) are able to surpass their pre-trained counterparts without adding any synthetic data (DPT (0x)). Overall, synthetic data just boosts performance at test time, which is an observation known to the community for a while now. If it is the case that the synthetic data is not as easily generated, then our method would take a small hit, but by no means would it be failing as a result. Echoing what we said in the Novelty section above, the synthetic data should be viewed as a free benefit of work in pre-trained deep generative models. If it is there and works; great. If not, one still has the teacher and distillation algorithm that can give significant gains regardless.

Reviewer comment (Novelty):
From a practical standpoint where do you think this method might be helpful? Because of the models considered in the paper have pre-trained counterparts available off-the-shelf i.e cost of pretraining is already paid and you only need to fine tune these models to downstream tasks.

Author response:
From a practical standpoint, we believe it’s useful for those who want to use a small model that *isn’t* available in any of these hubs. Granted, we used small models that were already trained on ImageNet in our illustrative examples, but that was mainly for our convenience and research efficiency. However, the future of model hubs and licenses is unknown, but our method enables practitioners to lower their reliance on hubs for all of their model needs.

Reviewer comment (Experiments):
In this case synthetic data generation is a key step in boosting performance. In some datasets generative models might not be able to do good job in adding more synthetic data. Would the approach usefulness fail here?

Author response:
*Sometimes* data generation is a key step. Importantly, we find that the synthetic data component is the most significant in the data scarce scenarios. As we can see (Table 3), the datasets with plenty of samples (CIFAR-10, CIFAR-100, CUB-2011) are able to surpass their pre-trained counterparts without adding any synthetic data (DPT (0x)). Overall, synthetic data just boosts performance at test time, which is an observation known to the community for a while now. If it is the case that the synthetic data is not as easily generated, then our method would take a small hit, but by no means would it be failing as a result. Echoing what we said in the Novelty section above, the synthetic data should be viewed as a free benefit of work in pre-trained deep generative models. If it is there and works; great. If not, one still has the teacher and distillation algorithm that can give significant gains regardless.

Reviewer comment (Novelty):
The proposed distillation objective appears very derivative of contrastive distillation proposed in Tian et al. 2020, and it would be important if the authors provided a stronger differentiation here.

Author response:
5. (3) With regards to the distillation algorithm itself, as we mentioned in the paper, our choice of the alignment/uniformity metric was purely illustrative. We appreciate the work of CRD in bridging contrastive learning with KD, but their work was done pre- the modern boom in contrastive learning brought on by SimCLR, MoCo, CLIP, etc. Therefore, we formulate KD as a contrastive objective with the modern notation and considerations, massaging it into an InfoNCE-type form that SimCLR and MoCo build off of. From here, any recent development (most of which took place after the publication of CRD), could be used, and which we assume would also be successful. The A/U metric was chosen since one of the main themes of the paper was efficiency; if we’re going to minimize the training time for the small model, let’s try a method that doesn’t include all the bells and whistles such as large batch sizes or momentum-updated models.

Reviewer comment (Experiments):
Training merely a linear probe on top of a teacher is insufficient exploration into teacher finetuning to claim sufficient adaptation to a target task at hand, particularly when distribution shifts are larger.

Author response:
7. (6) Our reasons for linear probing as opposed to full finetuning come from theoretical and practical considerations. While full finetuning is certainly more robust to distribution shifts, it has been found to lead to a distortion of the backbone which will probably harm future finetuning to other tasks. We wish to leave our teacher “unharmed” in the case where we wish to use the foundation for multiple student tasks. From a practical perspective, most foundations that are finetuned for a task in the modern era are linearly probed for this exact reason. In addition, full finetuning incurs a *much* higher cost on the practitioners end as they now have to train the teacher end-to-end on each task. With the increasing size of foundations, this would be a significant cost for the intended users of our method. This is fundamentally even closer to the “vanilla KD” regime that you bring up in your first point; one minor improvement we have is that by taking an off-the-shelf pre-trained one, we only need to train one linear layer and our teacher is ready to provide big improvements.

Reviewer comment (Writing):
There are issues with formulations used in this paper, starting with the title itself, which is contradictory to what is actually proposed, as the introduced setup does leverage pretraining both to teach and for synthetic data generation.

Author response:
8. (7) This sounds like a grammatical issue, which we don’t agree is contradictory to the message of our paper. We believe it is reasonable for readers to infer that the “Pre-train” in the title refers to the object of the phrase, “small model”, as opposed to being a declaration of not pre-training anything ever. This should also be clear after reading the abstract and/or introduction as well. For how it fits in the broader context of other methods, see above “Comparison to NAS, Pruning, Quantization”.

Reviewer comment (Novelty):
Absence of crucial references and an insufficient comparison with state-of-the-art methods in knowledge distillation. While the author has touched upon various studies in the domain of knowledge distillation, the discussion appears to be incomplete. For example, works such as [1, 2] that delve into efficient (pre-)training via knowledge distillation techniques, have already revealed that knowledge distillation can be used as an effective technique to boost the training efficiency for tasks including image classification, object detection and semantic segmentation. In a closely related domain, active knowledge distillation can, to a degree, achieve the objectives set by the authors, making it imperative to discuss and compare with studies like [3].

Author response:
We added references to these works, thank you for pointing them out. However, we do not believe a comparison to them is appropriate. KDEP is fundamentally a pre-training optimization, whereas we want to completely avoid pre-training of any kind. KCD takes an E-M approach to the choice of the transfer dataset each iteration, whereas we propose to augment it at the beginning. Both of these are great ideas, and there is no reason why one cannot apply them to our formulation as well (i.e. running KCD on a synthetically-augmented transfer set, or doing KDEP to warm up the student, although this loses the spirit of our approach). We believe both these approaches tackle different aspects of the training procedure that do not intersect with our work. TrustAL is an interesting approach, though their work concerns the language domain. More importantly, we do not see how the problem that they tackle, example forgetting, is related to our work that justifies a comparison. Sure, they use a teacher model to help minimize forgotten knowledge, but that is fundamentally a different use of a teacher model compared to ours. Again, there is an opportunity here to utilize their method in tandem with ours in an active learning setup.

Reviewer comment (Experiments):
The experimental results presented in this paper are underwhelming. In light of the benchmarks set by leading methods in the realm of knowledge distillation, the performance depicted in this paper feels somewhat middling. For instance, when employing ResNet-18 as the student and ResNet-50 as the teacher, [1] manages to attain 82.22% in a mere 16 GPU hours. In contrast, this paper's proposed technique takes over 24 GPU hours to reach a modest 75.8%. While there might be nuances that make a direct comparison slightly skewed, it remains necessary for the authors to validate their approach in comparable scenarios and surpass at least one of the aforementioned studies. One experiment that might be needed is to merge the real and synthetic data and then apply the proposed method of this paper to see whether the proposed method can surpass the previous works in knowledge distillation.

Author response:
KDEP is a great work that proposes a well-justified and effective optimization to pre-training. We particularly enjoyed their observation that entire foundation datasets may not even be necessary to pre-train! However, we maintain the position that a direct comparison to their work is unnecessary. In addition, the “16 hours” claimed is not a comparable data point without accounting for differences in hardware used. Their paper states that their timing numbers were done on 4 V100 GPU’s, whereas we used 1 P100, an older and slower architecture. Even if we ignore the supremacy of the Volta (V) architecture over Pascal (P), this translates to KDEP taking around 64 hours to achieve that 82%, which is 3x longer than our method; however, they do achieve a higher accuracy at this high cost. We believe that the desired “comparable scenario” is addressed in Section 4.3, where we fix the same teacher-student, use the same transfer dataset, but compare to popular and effective distillation algorithms: vanilla KD, CRD, and SRRL. There, we see that our method is competitive, but no method comes out to be the outstanding winner. Since our main contribution is the overall recipe (see Novelty response above), we are not looking to show purely the supremacy of our contrastive-based distillation algorithm, but that we can formulate KD as a contrastive objective, and any algorithm (like the Align/Uniform we choose) can be competitive to SoTA. Also, our experiments do indeed combine the real and synthetic datasets. DPT (1x) combines the real dataset with an equally sized synthetic one, while DPT (2x) combines the real with a synthetic one twice the size. If there is anywhere in the paper that we can clarify this better, please let us know.

Reviewer comment (Experiments):
The claim of general applicability might benefit from a more diverse testing. Relying on only 6 computer vision datasets and a handful of teacher-student architectures narrows the perspective.

Author response:
We welcome suggestions as to other datasets or pairings you believe would strengthen our case. Granted, we chose 1 modality, but in this modality we attempted to use datasets that reflect different realities (data is abundant vs data is scarce). For the models, we carefully chose pairings to exemplify how flexible our method is. One conv-based and one transformer-based teacher have no problems working with small models that have different design approaches.

Reviewer comment (Evaluation):
It would be helpful to have clarity on the number of experiments behind each reported measurement and the inclusion of standard deviations.

Author response:
We only did 1 experiment for each. More experiments are always better, but given our time and resources, we do not believe we can run every experiment multiple times by the deadline. However, a future version of this paper would definitely have them.

Reviewer comment (Evaluation):
"small models" and cost considerations need to be clearly defined. The paper seems not to account for certain factors that contribute to the total cost, such as training the existing pre-trained models.

Author response:
If by “pre-training cost”, you are referring to the cost of pre-training the teacher and generative models, we chose to omit those since those costs are not absorbed by the intended practitioners of this method. They can assume that these models already exist and are available for the using immediately. We added a clarification about this in our updated version.

Reviewer comment (Experiments):
While synthetic samples do enhance performance, it remains unclear if baselines are also exposed to them during training, raising questions about fairness in comparisons.

Author response:
The baselines were not exposed to the synthetic samples. The only scenarios that were were DPT (1x) and DPT (2x). We updated the wording in our results section to clarify that only the transfer dataset is augmented. We can certainly provide those if desired, but does the reviewer believe that these would lead to a different or stronger conclusion than is currently provided?