Paper: Project and Probe: Sample-Efficient Adaptation by Interpolating Orthogonal Features (ICLR-2024)

Reviewer comment (Theory):
One major limitation of this work is the fact that the project and probe processes are considered solely in the linear model regime due to the pre-trained feature extraction. Also, the theoretical analysis was conducted with a linear model. How would it be extended to large-scale problems?

Author response:
There are two key reasons for focusing on the linear setting. One is that the pre-trained feature extractors are quite good, and training linear heads on top of pre-trained features has been shown to be a powerful approach for a wide range of problems. Furthermore, such a pipeline is computationally cheap to run, as the optimization is linear and does not require fine-tuning the whole backbone.The second is that we are considering the low data regime. With only a small number of datapoints from the target domain, a full feature extractor will overfit and not be able to generalize to new test points. As such, the pre-trained feature extraction and linear project and probe head are key to the method, which is why all the analysis is conducted with both of these. We find that our method does work on challenging real-world tasks, such as FMoW satellite images.

Reviewer comment (Experiments):
The authors only compare with projection-based baselines. I think comparisons with recent general unsupervised domain adaptation methods are needed.

Author response:
The problem setting we are studying is different from unsupervised domain adaptation. Whereas unlabeled DA uses unlabeled target data, our setting uses a small labeled dataset from the target distribution; we give a full description in Section 3. Because such methods do not effectively use labeled target domain data, we do not directly compare to them for fairness.

Instead, we have added two additional points of comparison that use both a small amount of labeled target domain data like Pro^2 and additionally use unlabeled data: Pagliardini et al. [47] and Lee et al [27], shown below. While both methods perform well, Pro^2 at least matches the performance with any target data size and sometimes even outperforms the other two methods, especially with few datapoints (2) on Waterbirds and more datapoints (32) on CelebA, despite not using any additional unlabeled data, which may be difficult to obtain.

| Waterbirds (Balanced) |           | Target Train Data Size (per label) |           |
|:---------------------:|:---------:|:----------------------------------:|:---------:|
|        Method         |     2     |                 8                 |    32     |
|         D-BAT         | 90.1 (0.5)|             91.5 (0.4)             | 92.3 (0.2)|
|         DivDis        | 89.5 (0.8)|             91.1 (0.5)             | 93.0 (0.2)|
|         Pro^2         | 91.8 (0.2)|             91.7 (0.4)             | 92.7 (0.3)|

| CelebA (Balanced)    |           | Target Train Data Size (per label) |           |
|:--------------------:|:---------:|:----------------------------------:|:---------:|
|       Method         |     2     |                 8                 |    32     |
|       D-BAT          | 76.7 (0.5)|             78.4 (0.7)             | 78.5 (0.7)|
|       DivDis         | 76.2 (0.7)|             77.8 (0.5)             | 77.7 (1.0)|
|       Pro^2          | 76.5 (1.4)|             78.3 (0.9)             | 80.8 (0.9)|

Reviewer comment (Novelty):
While learning diverse/orthogonal features is novel in the context of domain adaptation, there is an active line of research that explores this idea in the standard supervised learning setting, such as [1-7]. I think these methods should be discussed in the related work.

Author response:
Thank you for the comment and references. We have added these references and some discussion in the related work section. Prior works have explored learning diverse or orthogonal features in standard supervised learning settings and we show how diversification can lead to more sample-efficient adaptation to distribution shifts.

Reviewer comment (Experiments):
There are approaches that exploit Linear Discriminant Analysis (LDA) and Quadratic Discriminant Analysis (QDA) for the adaptation of the head of a pretrained model with success (Shysheya et al. 2022). Can the authors comment on the differences between their method and these methods? While the paper covers the relation with respect to LDA, it does not seem to mention the relation with QDA. Adding those baselines to the empirical evaluation may be beneficial.

Author response:
Both LDA and QDA operate in the setting without distribution shift, and are typically used to learn Bayes optimal predictors on in-distribution data samples under assumptions on the class conditional distributions being Gaussians. On the contrary, our method extracts linear classifiers for few shot adaptation on out-of-distribution data. This requires extracting more directions than just the optimal classifiers on in-distribution source data, which we accomplish with our “project” step. Nevertheless, in Corollary 9, we show that Pro^2 with dimensionality d = 1 provably recovers the LDA direction in a shifted homoscedastic Gaussian model. Similarly, under heteroscedastic Gaussian model, minimizing cross entropy loss with a quadratic kernel (feature extractor), will recover the QDA classifier for d = 1. We also show that using higher values of d is critical in adapting to higher degrees of distribution shift. In this sense, the projection of Pro^2 can be seen as a generalization of LDA/QDA. We compare to a wide range of state-of-the-art baselines (e.g. DFR, Teney et al., D-BAT, DivDis, etc) and include a comparison between our method and replacing the projection step with PCA in the Appendix.

Reviewer comment (Evaluation):
Would the authors be able to provide a detailed analysis of the complexity for the proposed method (e.g. FLOPs and/or MACs)? Understanding complexity is crucial as it essentially represents the computational budget. How does the method's time complexity stand in comparison to leading fine-tuning approaches, such as BiT by Kolesnikov et al. (2020), which adapt the entire model body or FiT (Shysheya et al. 2022), which adapt a subset of the body parameters?

Author response:
Our training involves at most two linear layers on top of cached feature vectors and is therefore generally much more computationally efficient than methods that adapt the entire model body or a large subset. The optimization problem is linear, so we were able to perform 30k runs within 24 hours using four standard CPUs and no GPUs.