Paper: Efficient Transfer Learning from Arbitrary Pre-Trained Models (ICLR-2024)

Reviewer comment (Presentation):
I understand the attempt of providing a justification of the regularization loss using information theoretical bonds, but the way the authors arrive to the final form of the regularization they use, which is just a kernel version of the L2, eq. 9, is in my opinion unnecessarily involved and might create confusion. I suggest moving the text from eq 2 to eq 9 to the appendix.

Author response:
**On arriving at the final objective.** We included a detailed derivation of the objective starting with the bound on the mutual information to highlight the conceptual difference between AFT and KD. While the final objective is indeed a kernelized version of the L2 objective in Eq 9 (now Eq 4), it is fundamentally different from the similar-looking L2 objective in Eq 10 (now Eq 5) which KD optimizes. However, we appreciate the suggestion for a more concise presentation and have compressed the derivation.

Reviewer comment (Theory):
It is not very clear why this form of R should help avoiding redundancy: the sigmoid function can set to zero irrelevant features, but if several features are simultaneously relevant (but correlated), I expect the solution will not be sparse, but it will contain weights contributions from all. I suspect that this might lead to overfitting in data-scarce scenarios.

Author response:
Thank you for responding! We’d like to clarify our answer to this question. If multiple features are simultaneously relevant but correlated, the weights learned by AFT indeed may not be sparse, nor have we claimed that they would be. 

However, assigning non-zero weights to correlated features should not harm AFT’s performance. In fact, we can show that if non-zero weights $\rho_a$ and $\rho_b$ are assigned to two duplicated features $\psi_a$ and $\psi_b$ for $a\neq b,$ the effect is exactly identical to having assigned a sparse $\rho'$ where $\rho'_a = \sqrt{\rho^2_a + \rho^2_b}$, $\rho'_b = 0$, and $\rho'_i = \rho_i$ for $i\notin\{a,b\}.$ 

To prove this, it suffices to show that the target kernel $K_{\rho\Psi}(x,x') = (\rho \psi(x))^\top (\rho \psi(x'))$ in Eq 6 constructed by AFT with non-sparse weights $\rho$ is identical to the kernel constructed by AFT with sparse weights $\rho':$ $K_{\rho'\Psi}(x,x') = (\rho' \psi(x))^\top (\rho' \psi(x')).$ 

This follows from direct calculation: $K_{\rho\Psi}(x,x') = (\rho \psi(x))^\top (\rho \psi(x')) = \sum_{i} \rho^2_i \psi_i(x)\psi_i(x') = \sum_{i\notin\{a,b\}} \rho^2_i \psi_i(x)\psi_i(x') + \rho^2_a \psi_a(x)\psi_a(x') + \rho^2_b \psi_b(x)\psi_b(x') = \sum_{i\notin\{a,b\}} \rho^2_i \psi_i(x)\psi_i(x') + (\rho^2_a + \rho^2_b) \psi_a(x)\psi_a(x') $ 

By definition of $\rho'$, the above is equal to $\sum_{i\notin\{a,b\}} \rho^2_i \psi_i(x)\psi_i(x') + \rho'^2_a \psi_a(x)\psi_a(x') = (\rho' \psi(x))^\top (\rho' \psi(x')) = K_{\rho'\Psi}(x,x').$ The argument can be straightforwardly extended to an arbitrary number of duplicated features. Therefore, due to using the kernelized objective, assigning non-zero weights to duplicated features is automatically equivalent to assigning sparse weights in AFT.

We hope you find this answer satisfactory and will consider revising your score!

Reviewer comment (Experiments):
Given the topic of the article I would have expected a comparison with LORA (https://arxiv.org/abs/2106.), where the features for the downstream task are selected by multiplying the original features by a low rank matrix before downstream fine tuning.

Author response:
**On LoRA.** LoRA is a method to efficiently fine-tune the original pre-trained model, but does not transfer to a smaller model and therefore still requires running expensive inferences with the original pre-trained model. As a result, LoRA is not a viable alternative to AFT or KD as a method to efficiently transfer from larger models into smaller ones. On the other hand, LoRA is entirely complementary to AFT as it can be used together during AFT fine-tuning to further reduce training cost.

Reviewer comment (Experiments):
Since the focus of the paper is transfer learning, which typically happens towards data-scarce tasks, I would have liked to see if the procedure is robust with respect to aggressive decimation of the target task. What happens if one attempts to use ~100 examples for category, as typical in clinical image analysis applications?

Author response:
**On data scarcity.** Many of our datasets are already very small in size: Oxford flowers (1K), Oxford pets (3.6K), RTE (2.5K), MRPC (3.7K), and AFT leads to significant gains on these datasets (Figure 2(b) and Figure 3(b)). Furthermore, in Figure 5(b), we present an additional experiment where we subsample the training data down to 10 examples per class on CIFAR-100, and 100 examples per class on BoolQ. We show that AFT continues to outperform competing methods even in extreme data-scarce settings.

Reviewer comment (Writing):
The last paragraph of page 4 is not very clear. The variational parameters are the components of the vector s, which, via the sigmoidal function, set the weight of the corresponding psi component in the rhoPsi kernel?

Author response:
**On clarifying our notations**
The $d_\psi \times d_\psi$ matrix $\rho$ is parameterized by the $d_\psi$ dimensional vector $s$ via $\rho = \mathrm{diag}(\sigma(s)),$ i.e. $\rho_{ii} = \sigma(s_i),$ where $\sigma$ is the sigmoid function. We have clarified this in the text.

Reviewer comment (Experiments):
When they present the setting on page 3, the labels seem to be linear regression labels, while the experiments are on classification datasets. Please clarify.

Author response:
We did not assume that the labels $Y$ are linear regression labels. The equation $Y=W \Phi$ states that the output of the model is a linear transformation of its last layer features $\Phi,$ which holds for all neural networks and in both regression and classification tasks (where $Y$ would be the logits). Similarly, Algorithm 1 is applicable to both regression and classification tasks and we have updated the text to show that the loss $L$ is any loss function specified for the downstream task.

Reviewer comment (Novelty):
Some related works are missing in this paper, including those improving standard transfer learning and those considering transfer learning from multiple pre-trained models. For example, [2] also proposes to match pre-trained features and downstream features during transfer learning. [3] and [4] also consider transfer learning from multiple pre-trained models and propose to use features or knowledge distillation from pre-trained models. More related works in these two topics should be discussed in the paper. In experiments, some of these more advanced transfer learning methods should be compared, instead of only comparing AFT with standard transfer learning or knowledge distillation.

Author response:
**On comparison with related work.** Following your suggestion, we have added B-Tuning, the method proposed in [4], as a stronger baseline throughout our experiments. In Figure 2(a) and 3(a), we show that AFT significantly outperforms B-Tuning on average across all 6 vision datasets and 8 language datasets. Similar to KD, we find B-Tuning benefits considerably less than AFT from better pre-trained models, as shown in Figure 2(c) and Figure 3(c). We also added a discussion of other related works to the paper.

Reviewer comment (Experiments):
It seems that in this paper, the pre-trained models are stronger than downstream models. Figures 2(c) and 3(c) also show that transfer learning by directly using pre-trained models leads to better results than AFT. This makes the problem setting in the experiments less convincing, especially considering that the linear probe from pre-trained models is also efficient.

Author response:
**On AFR vs linear probe on pre-trained features.**
Simply using a linear model with the pre-trained features is not a viable alternative, as it requires running expensive inference with the original pre-trained model, the very problem that AFT (and KD) aims to solve by transferring to much smaller models. For example, the linear probe accuracy of ViT-L CLIP roughly matches AFT accuracy when transferred to ViT-S on CIFAR-100. But ViT-L CLIP has 428M parameters, 20 times larger than ViT-S. We have added this discussion to the paper.

Reviewer comment (Experiments):
Transfer learning from multiple models is interesting, but currently, the number of models in the experiments is still small, and the improvements by using more pre-trained models are not clear from the results.

Author response:
**On whether more pre-trained models should always help.** 
While AFT enables transfer from multiple pre-trained models, it does not follow that transfer from more pre-trained models will always improve performance. Whether transfer from multiple pre-trained models improves performance depends on whether the pre-trained models provide complementary features, and an ideal method should benefit from more pre-trained models to the extent that they provide complementary features.

We have demonstrated that AFT achieves exactly this: as shown in Figure 2(d) and Figure 3(c), unlike other competing methods, AFT performance directly reflects the performance of the pre-trained features on the downstream task, measured by the linear probe accuracy, for both transfer from single and multiple models. For example, combining features from DINO + CLIP or BiT + DINO + CLIP both achieves higher linear probe accuracy than individual models, and AFT indeed achieves better performance by transfer from the two combinations than from individual models. Conversely, AFT does not improve performance by combining pre-trained models if their features are not complementary and do not improve linear probe accuracy in the first place, as we find in the NLP experiments (Figure 3(c)). We note that our findings agree with the results in [1], which also finds that transfer from more pre-trained models does not always improve performance and recommends using only a small number of pre-trained models in practice.

Reviewer comment (Presentation):
What are the exact results before normalization in Figure 2(b)?

Author response:
**On results before normalization.** We've added a table of the unnormalized results to Appendix A.4.

Reviewer comment (Experiments):
Could the kernel method in Section 3.2 still improve the performance if the downstream datasets have more training data? It would be better to have more experiments on more datasets or situations to validate the efficacy of such a design.

Author response:
**On using larger downstream datasets.** We've evaluated on 5 new downstream datasets that are larger than any dataset we previously used, including QQP (364K), MNLI (393K), QNLI (105K), Food101 (76K), and SST-2 (67.3K). AFT is the best performing method for all downstream models on 4 of these 5 datasets, verifying that AFT is effective even when the downstream datasets are large.

Reviewer comment (Evaluation):
The proposed method promises to distill features from **any** set of models to a given model once the downstream task is known. The paper is positioned as a generic method that could be applied to any set of models (possibly containing architectures different to the downstream one). However, while the presented theory to justify the method is sound and generic, the empirical results do not seem to support the claim. For example, in Figure 1 (right) and Figure 2 (b) adding convolutional features to a ViT based downstream model seems to reduce the performance of the model. This seems to suggest that the proposed method is not strong enough to reject some features that will lead to a worse downstream model. If this is the case, the current algorithm should be coupled with model selection techniques to pick the best features that are more likely to help (see [1] and references therein). Can the authors comment on this more?

Author response:
**On rejecting less useful features.** In Figure 2(c) (previously Figure 2(b)), transferring convolutional features from a BiT into a ViT downstream model did in fact improve its performance compared to the ViT only standard transfer learning baseline (STL). It's true that transferring from BiT + DINO + CLIP is slightly worse than DINO + CLIP only. However, AFT is successful at upweighting the more useful features even in this case since transfer from BiT + DINO + CLIP is much better than from BiT alone. Indeed, we verify this in Figure 4(a) by visualizing the learned $\rho,$ which are much smaller for BiT features than for DINO and CLIP features when transferring from all three models. To further test AFT's ability to downweight useless features, we run a new experiment where we append random noise features to DINO features and find that AFT learns to significantly downweights the noise features (Figure 4b), and thereby maintains its performance as we increase the number of noise features (Figure 4c) to up to 2048, whereas KD performance quickly degrades.

Reviewer comment (Experiments):
The previous limitation gets even worse when the set of conditioning models gets larger since the signal-to-noise ratio drops, making extracting the relevant information for the downstream task even harder. It is suggested that the authors consider comparing with explicit sparsity-inducing methods as the ones proposed in [2] and the references therein.

Author response:
**On rejecting less useful features.** In Figure 2(c) (previously Figure 2(b)), transferring convolutional features from a BiT into a ViT downstream model did in fact improve its performance compared to the ViT only standard transfer learning baseline (STL). It's true that transferring from BiT + DINO + CLIP is slightly worse than DINO + CLIP only. However, AFT is successful at upweighting the more useful features even in this case since transfer from BiT + DINO + CLIP is much better than from BiT alone. Indeed, we verify this in Figure 4(a) by visualizing the learned $\rho,$ which are much smaller for BiT features than for DINO and CLIP features when transferring from all three models. To further test AFT's ability to downweight useless features, we run a new experiment where we append random noise features to DINO features and find that AFT learns to significantly downweights the noise features (Figure 4b), and thereby maintains its performance as we increase the number of noise features (Figure 4c) to up to 2048, whereas KD performance quickly degrades.

Reviewer comment (Experiments):
The final algorithm is optimizing theta and rho jointly. However, one would expect \rho being optimized more often than \theta. Typically, this is done with bi-level optimization techniques or by simply rewriting \rho in closed form for each given \theta. Did the authors try those more natural alternatives? If \rho is not optimized fast enough, the most likely trajectory induced by SGD will be around a stationary point of \rho, which leads to a maximally insensitive/uninformative \rho that will be reasonably good on average for many possible \theta, however not optimal for any in particular.

Author response:
**On optimizing $\rho$ more often than $\theta$.** By using the kernel formulation and only learning a diagonal $\rho,$ we have considerably reduced the difficulty of optimizing $\rho.$ Instead of optimizing a dense $d_\psi \times d_\psi$ matrix, we only optimize a $d_\psi$ dimensional vector. Furthermore, the bound in Eq 2 is valid even if $\rho$ is not optimal for a given $\theta.$ Therefore, to maximize the number of updates to $\theta$ given a fixed compute budget, we optimize them jointly, exactly analogous to how the encoder and decoder are jointly optimized in VAEs. However, following your suggestion, we added an ablation experiment where we update $\rho$ 5 times per $\theta$ update, and find that it does not improve performance even with the number of updates to $\theta$ fixed (Figure 5(a)).

Reviewer comment (Theory):
Why should invariance under orthogonal transformation be of help in the practical optimization objective? Can the authors prove how the optimization landscape will change and get easier to optimize? As of now, this intuitive fact is left to the ablation studies and only supported by empirical observations.

Author response:
**On the benefit of invariance under orthogonal transformation.** Invariance of the kernel under orthogonal transformation of the features is helpful because it reduces the number of parameters to learn. By only learning a diagonal $\rho$ with $d_\psi$  parameters, we can effectively search for all transformations $\rho' = U\rho$ for any orthogonal matrix $U$ without having to learn $d_\psi^2$ parameters.

Reviewer comment (Experiments):
Why not use a different kernel than the linear one? This could make the optimization space much smoother (e.g., by choosing a Gaussian kernel).

Author response:
**On using a different kernel.** We have added an ablation experiment where we use a Gaussian (RBF) kernel, and find that it achieves similar but slightly worse performance (Figure 5(a)).

Reviewer comment (Experiments):
Visual evaluation on CIFAR100 is quite limited. To increase the impact of the paper on the community, it is suggested that the authors extend the evaluation to other datasets as the ones used in [1].

Author response:
**On further evaluation.**
Following your suggestion, we include 2 additional vision datasets and 6 additional language datasets from the GLUE benchmark, bringing the total number of datasets to 15. We also report the averaged results in Figure 2(a) and Figure 3(a), showing that AFT significantly outperforms competing methods on average.

Reviewer comment (Experiments):
Make the scatter plots with linear probe accuracy vs. test accuracy on the same scale. Is the proposed method worse than directly using a linear classifier on the concatenated features?

Author response:
**On AFT performance vs linear probe performance.** We plotted the linear probe accuracy vs test accuracy on different scales for the convenience of visualization because the former has a much larger range than the latter. It’s true that for sufficiently large and strong pre-trained models, linear probe can be better than AFT (and all other methods). However, the linear probe is expensive to deploy, as it requires inference with the original pre-trained model, the very problem that AFT aims to solve, and is therefore not a viable alternative. For example, the linear probe accuracy of ViT-L CLIP roughly matches AFT accuracy when transferred to ViT-S on CIFAR-100. But ViT-L CLIP has 428M parameters, 20 times larger than ViT-S. We have added this discussion to the paper.

Reviewer comment (Writing):
There is a missing $\prime$ in the first kernel definition, the definition of applying the kernel function to vector is undefined in equation (9), $X$ and $X^{\prime}$ should be the same according to Algorithm 1 but not mentioned in the text.

Author response:
**On clarifying our definitions.**
(1) Thank you for catching the typo in the kernel definition. We have fixed it. In Eq 9 (now Eq 6), $k_\Phi(X, X')$ follows the same definition given in the paragraph directly above, namely $\phi(X)\phi(X'),$ where $X$ and $X'$ each denote a single input.

Reviewer comment (Theory):
Why the $\rho$ in Section 3.2 does not downsample the feature to the shape of the downstream features ($d_{\phi}$)?

Author response:
**On clarifying our definitions.**
(2) Under the kernel formulation, $\rho$ no longer needs to map the pre-trained features to the shape of the downstream features because the kernel enables comparison between features of different shapes. We have added a sentence to clarify this.

Reviewer comment (Reproducibility):
In Algorithm 1, the definition of $\hat{L}(\theta)$ is missing and $\hat{Y}_{batch}$ is not used.

Author response:
**On clarifying our definitions.**
(4) We updated the text to clarify that the mini-batch loss $\hat{L}(\theta)$ is computed as $L(\theta, Y_{\mathrm{batch}}, \hat{Y}_{\mathrm{batch}})$ where $L$ is some given loss function on the downstream task (e.g. cross-entropy loss).

Reviewer comment (Evaluation):
The evaluation is conducted only on small subsets of benchmarks. Using more datasets and reporting the average results would make the results more convincing (like datasets used in few-shot experiments in CLIP, GLUE, SuperGLUE, Winogrande, etc).

Author response:
**On further evaluation.** Following your suggestion, we now include 2 additional vision datasets used in CLIP evaluation (DTD and Food-101) and 6 additional language datasets from the GLUE benchmark, bringing the total number of datasets to 15. We also report the averaged results in Figure 2(a) and Figure 3(a), showing that AFT significantly outperforms competing methods on average.

Reviewer comment (Theory):
Why choose Eq (7) as the starting point to develop AFT rather than equation (8), as in Figure 4, the results of AFT w/o kernel (optimizing Eq 7 only) are not better than STL (maybe KD either).

Author response:
**On choosing the starting point for AFT.**
As mentioned above, Eq (7) and Eq (8) (now Eq (4) and Eq (5)) are fundamentally different despite their similar forms. The former is conceptually more well-suited for transfer learning and allows selectively transferring the most relevant features. While the ablation study shows that directly optimizing Eq (7) (AFT w/o kernel) is not effective, we show that its kernelized version (AFT) is the best performing method. By contrast, kernelizing Eq (8) is similar to AFT w/o $\rho$ (not allowing weighting the pre-trained features), which we find is less effective.