Paper: A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation (ICLR-2024)

Reviewer comment (Experiments):
The training-free metric-based classifier has been well-studied in the few-shot learning literature. The authors are encouraged to compare GDA with other classifiers, such as the Neareast-Centroid Classifier with Euclidean distance [1] and Mahalanobis distance [2].

Author response:
**[A2].** Thank you for the advice. We report the results for Euclidean distance[1] and Mahalanobis distance[2] below. All experiments utilize the RN50 CLIP on 16-shot datasets, and we provide the corresponding 95 confidence intervals. As the results from both evaluation methods are consistent, we continue to use the evaluation method outlined in our paper. For a fair comparison, we also integrate both methods with the zero-shot classifier.

The results are shown below. Despite Mahalanobis distance incorporating covariance, it estimates a covariance matrix for each category, introducing significant noise in few-shot settings. Furthermore, it lacks the use of robust methods for precision matrix estimation, resulting in a precision matrix with substantial noise. Thus, it achieves similar performance with the Euclidean method. As indicated in Table 6 of our paper, the precision matrix estimation significantly influences the final results.

|Method|Pets|Flo|FGVC|DTD|EuroSAT|Cars|Food|SUN|Cal|UCF|IN|Avg|
|-|-|-|-|-|-|-|-|-|-|-|-|-|
|CLIP|85.77|66.14|17.28|42.32|37.56|55.61|77.31|58.52|86.29|61.46|58.18|58.77|
|Euclidean [1]|86.76+-0.89|86.21+-0.34|26.26+-1.22|60.84+-0.54|74.19+-1.42|67.24+-0.04|78.01+-0.07|68.04+-0.54|90.70+-0.34|74.03+-0.32|61.75+-0.05|70.37+-0.52|
|Mahalanobis [2]|86.77+-0.92|86.32+-0.36|26.43+-1.23|61.19+-0.78|74.44+-1.37|67.45+-0.14|78.01+-0.07|68.21+-0.54|90.80+-0.28|74.16+-0.35|61.77+-0.04|70.50+-0.55|
|ours|**88.81+-0.36**|**95.72+-0.41**|**40.61+-0.38**|**66.51+-0.17**|**86.12+-0.28**|**75.12+-0.41**|**79.05+-0.11**|**70.70+-0.13**|**92.55+-0.26**|**77.53+-0.64**|**63.82+-0.13**|**76.05+-0.30**|

[1] Jake Snell, et al, "Prototypical Networks for Few-shot Learning." In NeurIPS 2017.
[2] Bateni, Peyman, et al. "Improved Few-Shot Visual Classification." In CVPR 2020.

Reviewer comment (Evaluation):
The KNN method for Base-to-Novel generalization requires that the base classes should be very similar to novel classes. The method is expected to perform badly on dataset generalization settings.

Author response:
**[A3].** Yes, this is a limitation of the approach. The KNN method is based on the observation that similar classes share similar statistical information [1]. Thus, our method may fail in the case when new classes are fairly dissimilar from base classes. Moreover, as mentioned in our paper, we opted for straightforward modifications to the method to demonstrate its generalizability across diverse settings, rather than developing a complex, yet perfect, solution for each scenario. On the one hand, complex solutions compromise the simplicity of our method; on the other hand, if there are substantial differences in methods across diverse scenarios, it impacts the evaluation of our approach's generalization ability.

[1] Yang, Shuo, et al. "Free lunch for few-shot learning: Distribution calibration." In ICLR 2021

Reviewer comment (Novelty):
Though simple yet effective, the overall novelty is limited. The proposed method simply treats the pre-trained CLIP model as a frozen feature extractor, and incorporates the idea of Gaussian Discriminant Analysis (GDA) to estimate the classifier without training for the downstream tasks. The key contribution might simply be the validation of the strong feature extraction capability of the pre-trained CLIP model.

Author response:
**[A1].** We are sorry we didn't showcase this well in the paper and caused your confusion. Below, we elaborate on the novelty of our paper.

**(Ⅰ) We identify a simple but hard-to-beat training-free classifier.** First of all, all existing efficient fine-tuning methods for CLIP (e.g., CoOp/APE/Tip-Adapter/CLIP-Adapter, etc.) have frozen the feature extractor of the CLIP model. Therefore, we do not simply validate the powerful feature extraction capability of CLIP. **Our first contribution is revealing that there exists a simple but effective classifier that performs on par with or even better than previous complicated methods.** Previous methods either require additional learnable modules, such as learnable prompts, and Adapters, or need extra cache modules to store training data. However, we point out that there exists a simple yet effective training-free classifier (i.e., GDA) that has been overlooked in the existing literature. Without the need for additional optimization and design, it surpasses all training-free methods and achieves results on par with or better than training-required methods. This facilitates rapid model transfer and prompts a reconsideration of the effectiveness of these complex modules, inspiring the future design of more efficient fine-tuning methods. 

**(Ⅱ) We extend its applicability to various scenarios.** **Our second contribution is that we further extend our method to other domains with minimal modification, including imbalanced learning, base-to-new generalization, and unsupervised learning.** Our method also achieves superior results in these areas, demonstrating its wide-ranging effectiveness.

Reviewer comment (Evaluation):
It lacks of more insightful analyses. While the paper introduces the integration of Gaussian Discriminant Analysis (GDA) into the CLIP features, it falls short in providing a comprehensive and detailed analysis of the underlying mechanisms and the reasoning behind the method's effectiveness.

Author response:
**[A2].** We attribute the superiority of GDA to its Gaussian assumption. On the one hand, this assumption narrows down the parameter search space, reducing the risk of model overfitting. On the other hand, it introduces covariance into the score calculation (i.e., the $\mu_i^T\Sigma^{-1}x$ term in Eq.5), measuring the anisotropy of space. Previous methods, in contrast, only optimized the similarity between features and corresponding class weights ($w_i^Tx$), overlooking the importance of features in different dimensions. For a comparison, we set the covariance in our method to the identity matrix. The results, as shown in the table below, indicate a significant decline in method performance, yielding results similar to Tip-Adapter.

|Method|Pets|Flo|FGVC|DTD|EuroSAT|Cars|Food|SUN|Cal|UCF|IN|Avg|
|-|-|-|-|-|-|-|-|-|-|-|-|-|
|Tip-Adapter|88.14|89.89|29.76|60.93|70.54|66.77|77.83|66.85|90.18|70.58|62.01|70.32|
|Ours (cov=I)|86.76|86.21|26.26|60.83|74.19|67.24|78.01|68.04|90.70|74.03|61.75|70.37|
|Ours|**88.81**|**95.72**|**40.61**|**66.51**|**86.12**|**75.12**|**79.05**|**70.70**|**92.55**|**77.53**|**63.82**|**76.05**|

Reviewer comment (Experiments):
The effectiveness of the proposed method is dependent on the hyperparameter α, which controls the trade-off in the classifier ensemble. However, the paper lacks detailed evaluations regarding the sensitivity of this parameter.

Author response:
**[A3].** The following table analyses the sensitivity of the hyperparameter $\alpha$, which has been updated in the supplementary materials of the paper. From the results, it can be observed that our approach is relatively stable with respect to $\alpha$. Except for $\alpha=1$, the average performance across 11 datasets remains around 75.5. Even when all $\alpha$ is set to 10, the model only experiences a marginal drop of 0.7 points, still outperforming CoOp (73.42) and Tip-Adapter (70.32).

|Method|Pets|Flo|FGVC|DTD|EuroSAT|Cars|Food|SUN|Cal|UCF|IN|Avg|
|-|-|-|-|-|-|-|-|-|-|-|-|-|
|Ours (alpha=1)|88.81|89.32|31.10|67.65|85.87|67.54|79.05|66.73|92.55|77.53|61.79|73.45|
|Ours (alpha=3)|87.35|95.18|38.20|67.42|85.86|74.07|77.80|70.15|92.63|78.53|63.19|75.49|
|Ours (alpha=5)|86.45|95.67|39.72|67.18|85.85|75.45|76.58|70.81|90.13|78.24|63.73|75.44|
|Ours (alpha=7)|85.71|95.90|40.36|67.00|85.82|75.82|75.76|70.97|91.97|77.84|63.91|75.55|
|Ours (alpha=9)|85.36|95.99|40.63|66.86|85.80|75.62|75.17|70.79|91.94|77.59|63.88|75.42|
|Ours (alpha=10)|85.26|95.72|40.69|66.51|86.12|75.12|74.94|70.70|91.90|77.53|63.82|75.30|
|Ours (paper)|88.81|95.72|40.61|66.51|86.12|75.12|79.05|70.70|92.55|77.53|63.82|76.05|

Reviewer comment (Novelty):
What is the key novelty of the paper?

Author response:
**[A1].** We are sorry we didn't showcase this well in the paper and caused your confusion. Below, we elaborate on the novelty of our paper.

**(Ⅰ) We identify a simple but hard-to-beat training-free classifier.** First of all, all existing efficient fine-tuning methods for CLIP (e.g., CoOp/APE/Tip-Adapter/CLIP-Adapter, etc.) have frozen the feature extractor of the CLIP model. Therefore, we do not simply validate the powerful feature extraction capability of CLIP. **Our first contribution is revealing that there exists a simple but effective classifier that performs on par with or even better than previous complicated methods.** Previous methods either require additional learnable modules, such as learnable prompts, and Adapters, or need extra cache modules to store training data. However, we point out that there exists a simple yet effective training-free classifier (i.e., GDA) that has been overlooked in the existing literature. Without the need for additional optimization and design, it surpasses all training-free methods and achieves results on par with or better than training-required methods. This facilitates rapid model transfer and prompts a reconsideration of the effectiveness of these complex modules, inspiring the future design of more efficient fine-tuning methods. 

**(Ⅱ) We extend its applicability to various scenarios.** **Our second contribution is that we further extend our method to other domains with minimal modification, including imbalanced learning, base-to-new generalization, and unsupervised learning.** Our method also achieves superior results in these areas, demonstrating its wide-ranging effectiveness.

Reviewer comment (Writing):
What are the main insights?

Author response:
**[A2].** We attribute the superiority of GDA to its Gaussian assumption. On the one hand, this assumption narrows down the parameter search space, reducing the risk of model overfitting. On the other hand, it introduces covariance into the score calculation (i.e., the $\mu_i^T\Sigma^{-1}x$ term in Eq.5), measuring the anisotropy of space. Previous methods, in contrast, only optimized the similarity between features and corresponding class weights ($w_i^Tx$), overlooking the importance of features in different dimensions. For a comparison, we set the covariance in our method to the identity matrix. The results, as shown in the table below, indicate a significant decline in method performance, yielding results similar to Tip-Adapter.

|Method|Pets|Flo|FGVC|DTD|EuroSAT|Cars|Food|SUN|Cal|UCF|IN|Avg|
|-|-|-|-|-|-|-|-|-|-|-|-|-|
|Tip-Adapter|88.14|89.89|29.76|60.93|70.54|66.77|77.83|66.85|90.18|70.58|62.01|70.32|
|Ours (cov=I)|86.76|86.21|26.26|60.83|74.19|67.24|78.01|68.04|90.70|74.03|61.75|70.37|
|Ours|**88.81**|**95.72**|**40.61**|**66.51**|**86.12**|**75.12**|**79.05**|**70.70**|**92.55**|**77.53**|**63.82**|**76.05**|

Reviewer comment (Experiments):
How sensitive is the method w.r.t. the hyperparamter α?

Author response:
**[A3].** The following table analyses the sensitivity of the hyperparameter $\alpha$, which has been updated in the supplementary materials of the paper. From the results, it can be observed that our approach is relatively stable with respect to $\alpha$. Except for $\alpha=1$, the average performance across 11 datasets remains around 75.5. Even when all $\alpha$ is set to 10, the model only experiences a marginal drop of 0.7 points, still outperforming CoOp (73.42) and Tip-Adapter (70.32).

|Method|Pets|Flo|FGVC|DTD|EuroSAT|Cars|Food|SUN|Cal|UCF|IN|Avg|
|-|-|-|-|-|-|-|-|-|-|-|-|-|
|Ours (alpha=1)|88.81|89.32|31.10|67.65|85.87|67.54|79.05|66.73|92.55|77.53|61.79|73.45|
|Ours (alpha=3)|87.35|95.18|38.20|67.42|85.86|74.07|77.80|70.15|92.63|78.53|63.19|75.49|
|Ours (alpha=5)|86.45|95.67|39.72|67.18|85.85|75.45|76.58|70.81|90.13|78.24|63.73|75.44|
|Ours (alpha=7)|85.71|95.90|40.36|67.00|85.82|75.82|75.76|70.97|91.97|77.84|63.91|75.55|
|Ours (alpha=9)|85.36|95.99|40.63|66.86|85.80|75.62|75.17|70.79|91.94|77.59|63.88|75.42|
|Ours (alpha=10)|85.26|95.72|40.69|66.51|86.12|75.12|74.94|70.70|91.90|77.53|63.82|75.30|
|Ours (paper)|88.81|95.72|40.61|66.51|86.12|75.12|79.05|70.70|92.55|77.53|63.82|76.05|

Reviewer comment (Theory):
For completeness, provide reference or formulations on how the numerator of eq. (2) is achieved under the assumption of Gaussian distribution.

Author response:
**[A1].** Thank you for the advice. We update the calculation process of Equation (2) in the Appendix. Assuming that the features of different classes follow the Gaussian distribution with identical covariance, i.e., $(X|Y=i)\sim \mathcal{N}(\mu_i, \Sigma)$ for $i=1,2,.., K$. The probability of class $i$ is $p(x|y=i) = \frac{1}{(2\pi)^\frac{d}{2}|\Sigma|^{\frac{1}{2}}}\exp(-\frac{(x - \mu_i)^T\Sigma^{-1}(x - \mu_i)}{2})$. Then the classification probability can be expressed as follows:

$$ 
\begin{aligned}
p(y=i|x) 
    & = \frac{p(x|y=i)p(y=i)}{\sum_{j=1}^{K}p(x|y=j)p(y=j)} \quad \text{(Bayesian formula)} \\\\
    & = \frac{\frac{1}{(2\pi)^\frac{d}{2}|\Sigma|^{\frac{1}{2}}}\exp(-\frac{(x - \mu_i)^T\Sigma^{-1}(x - \mu_i)}{2})p(y=i)}{\sum_{j=1}^{K}\frac{1}{(2\pi)^\frac{d}{2}|\Sigma|^{\frac{1}{2}}}\exp(-\frac{(x - \mu_j)^T\Sigma^{-1}(x - \mu_j)}{2})p(y=j)} \\\\
    & = 
    \frac{
    {{\frac{1}{(2\pi)^\frac{d}{2}|\Sigma|^{\frac{1}{2}}}}}
    \exp
    ({{{-\frac{x^T\Sigma^{-1}x}{2}}}} + \mu_i\Sigma^{-1}x - \frac{\mu_i^T\Sigma^{-1}\mu_i}{2})p(y=i)}
    {\sum_{j=1}^{K}{{{\frac{1}{(2\pi)^\frac{d}{2}|\Sigma|^{\frac{1}{2}}}}}}\exp({{{-\frac{x^T\Sigma^{-1}x}{2}}}} + \mu_j\Sigma^{-1}x - \frac{\mu_j^T\Sigma^{-1}\mu_j}{2})p(y=j)}  \\\\
    & = \frac{\exp(\mu_i^T\Sigma^{-1}x - \frac{1}{2}\mu_i^T\Sigma^{-1}\mu_i + \log p_i)}{\sum_{j=1}^K\exp(\mu_j^T\Sigma^{-1}x - \frac{1}{2}\mu_j^T\Sigma^{-1}\mu_j + \log p_j)} \quad \text{(denoted $p_i = p(y=i)$)}
\end{aligned}
$$

Reviewer comment (Experiments):
Do all the datasets contain both validation and test set? How the hyperparameter $\alpha$ is set for datasets without validation set (e.g. ImageNet)?

Author response:
**[A2].** All datasets include both validation and test sets except ImageNet. Due to this sole exception and for a fair comparison, we align with previous methods (e.g., Tip-Adapter, APE, etc.) and thus validate $\alpha$ on the test set.

Reviewer comment (Reproducibility):
Please report the $\alpha$ value that is used for obtaining the results either in captions or appendix.

Author response:
**[A3].** The $\alpha$ value of 16-shot datasets is attached below, and we have updated it in the appendix.
||Pets|Flo|FGVC|DTD|EuroSAT|Cars|Food|SUN|Cal|UCF|IN|
|-|-|-|-|-|-|-|-|-|-|-|-|
|$\alpha$|1.00|10.00|100.00|10.00|10.00|10.00|1.00|10.00|1.00|10.00|10.00|

Reviewer comment (Experiments):
GDA constructing a linear classifier for CLIP. Discuss why GDA is superior to linear probe? Does linear probe performance depend on the training batchsize? Would increasing batchsize improve linear probe performance?

Author response:
**[A4].** GDA is superior to linear classifiers mainly because it explicitly makes assumptions about the data distribution. This confers two key advantages. Firstly, it narrows down the parameter search space, mitigating the risk of overfitting to specific local optima. Moreover, optimal parameters within this subspace can be directly obtained through statistics, eliminating the need for gradient optimization. Secondly, under the distribution assumption of GDA, the covariance of the data is explicitly introduced into the score calculation (i.e., the $\mu_i^T\Sigma^{-1}x$ term in Eq.5), while linear classifiers only optimize the similarity between features and corresponding class weights. We posit that the inclusion of covariance, the second-order statistic, is pivotal in this context. We employ the empirical Bayes ridge-type estimator for precision matrix estimation, which is specifically designed for scenarios where the sample size is smaller than the dimension. The results in Table 6 provide an ablation of different covariance matrix estimators.

Reviewer comment (Experiments):
Similarly, how the batchsize effects the results of Tip-Adapter-F?

Author response:
**[A5].** The impact of batch size on Linear Probe and Tip-Adapter-F is shown in the table below. All experiments utilize RN50 CLIP on 16-shot datasets. As indicated in the table, smaller batch sizes result in better performance for the Linear Probe. This may be attributed to the fact that in larger batch sizes, the model is more prone to overfitting, while in smaller batch sizes, the model is more likely to escape local optima due to increased randomness. Conversely, for Tip-Adapter-F, a larger batch size yields better results, possibly because it integrates the zero-shot classifier, serving a regularization role. Despite this, there is still a gap between their results and our method.

||LP(bs=256)|LP(bs=128)|LP(bs=64)|Tip-F(bs=256)|Tip-F(bs=128)|Tip-F(bs=64)|Ours|
|-|-|-|-|-|-|-|-|
|Pets|74.87|76.37|77.41|88.90|89.23|88.84|88.81|
|Flowers|92.02|93.30|94.23|92.34|91.66|90.19|95.72|
|FGVC|31.17|32.40|33.29|33.37|32.77|31.41|40.61|

Reviewer comment (Experiments):
How does the split to base and new classes is made for each dataset?

Author response:
**[A6].** We adopt the standard evaluation protocol proposed by CoCoOp. For each dataset, we split the classes equally into two groups, one as base classes and the other as the new classes. For example, ImageNet has 1,000 classes. We use the class id in (0-499) as the base classes and (500-999) as the novel classes.

Reviewer comment (Experiments):
I am skeptical regarding the dataset sampling using KNN for estimating the statistics for new classes. Please discuss the scenarios with new classes that are less similar or dissimilar to the training dataset and how such dataset sampling affect the results.

Author response:
**[A7].** Yes, this is a limitation of the approach. The KNN method is based on the observation that similar classes share similar statistical information [1]. Thus, our method may fail in the case when new classes are fairly dissimilar with base classes. Nevertheless, as mentioned in our paper, we opted for straightforward modifications to the method to demonstrate its generalizability across diverse settings, rather than developing a complex, yet perfect, solution for each scenario. On the one hand, complex solutions compromise the simplicity of our method; on the other hand, if there are substantial differences in methods across diverse scenarios, it impacts the evaluation of our approach's generalization ability.

[1] Yang, Shuo, et al. "Free lunch for few-shot learning: Distribution calibration." In ICLR 2021

Reviewer comment (Experiments):
Prior work KgCoop [a] is missing in the base-to-new generalization comparisons.

Author response:
**[A8].** Thank you for pointing out this. We add the comparison into the draft and detailed comparison in Appendix. Average results over 11 datasets are presented below.

||train|base|new|H|
|-|-|-|-|-|
|CLIP|x|69.34|74.22|71.70|
|CoOp|✓|82.69|63.22|71.66|
|CoCoOp|✓|80.47|71.69|75.83|
|KgCoOp|✓|80.73|73.60|77.00|
|Ours|x|**83.96**|**74.53**|**78.72**|

Reviewer comment (Evaluation):
Please report the time required for preparing the classifier (computing mean and precision matrix) in Table 7 under Train time.

Author response:
**[A9].** Thanks for the advice. We have updated it in the draft. The process is quite fast, it takes about 1.6s on the 16-shot ImageNet and about 3.6s on the full ImageNet dataset.