Paper: Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD Generalization (ICLR-2024)

Reviewer comment (Novelty):
[$\textbf{Unconvincing Statement}$] The work mentions that it is the first to unveil the pitfalls of finetuning VLMs by prompt learning can cause overfitting on base classes, resulting in poor performance on novel classes. However, CoCoOp “Conditional Prompt Learning for Vision-Language Models, CVPR 2022” has already observed this and proposed conditional prompt learning to address it.

Author response:
**Unconvincing statement: CoCoOp already observes overfitting with a method proposed to address it.**

We humbly argue that we are the first to provide a ``comprehensive’’ study of overfitting across different finetuning baselines, datasets and OOD settings. Hence in the revised paper, we will tone down the point about being the first and instead focus on our contribution of being comprehensive. Algorithm-wise, instead of using input-conditional prompts in CoCoOp that implicitly tackles overfitting, we propose OGEN to directly reduce overfitting via unknown-aware optimization with OOD feature synthesis and adaptive model distillation. Our benefits are validated by the gains when applying OGEN on top of CoCoOp (Table 1 & 2).

Reviewer comment (Theory):
[$\textbf{Unclear Model Design}$] This work uses known image and text features as K and V, while unknown text features as Q to generate unknown image features. The rationale behind this is not clear. It would be nice to explain this in more details.

Author response:
**Unclear model design: explain the rationale behind the attention method used to generate unknown image features.**

Please refer to Section 3.2 (3rd paragraph) for the reasoning. At high level, to generate unknown image features from just the class name (using its text features), we found it's hard to directly learn the text-to-image feature mapping. This is backed by our ablations in Table 4 - see the poorly performing ``no extrapolation'' baseline. Then we choose to condition the feature generation process on some knowledge about known classes (i.e. the image/text features of kNN known classes), arriving at an extrapolation solution which is easier. The remainder of Section 3.2 introduces two extrapolation schemes, both naturally implemented by cross-attention. The attention mechanism is similar to measuring similarities between the unknown text (query) and kNN known text (key) features first, and then using such text similarities to extrapolate unknown image features from kNN known image features (value). We will clarify more on such rationale in the revised paper.

Reviewer comment (Experiments):
[$\textbf{Small Performance Gains}$] The results in Table 1 show the improvements from adding the proposed method are rather limited. Moreover, the performance is much worse than some SOTA methods, e.g., “MaPLe: Multi-modal Prompt Learning, CVPR 2023”, “Improving Zero-Shot Generalization for CLIP with Synthesized Prompts, ICCV 2023” and “Self-regulating Prompts: Foundational Model Adaptation without Forgetting, ICCV 2023”. It would be nice to see the performance of these SOTA methods by adding the proposed components.

Author response:
**Small performance gains in Table 1, and results of OGEN+SOTA methods.**

Our observations from Table 1 are that 1) OGEN achieves notable improvements on new classes, 2) while its base class accuracy remains competitive. This is the desired behavior to us since we focus more on the new class generalization, although this may make the gains in the Harmonic mean of base and new accuracies seem smaller. Table 2 is another example where OGEN achieves convincing gains on new datasets without hurting the source ImageNet performance. For SOTA results, please refer to our “Response to common concern: comparing with the SOTA”.

Reviewer comment (Experiments):
My main concern with this paper would be the results. In particular, they are not state-of-the-art (see [Maple] and [Clipood]). Furthermore, the reported CoOp performance seems low. With some tuning, CoOp can be much better, e.g. [KgCoOp] reported a harmonic mean of 74.6 for CoOp on average, compared to 71.7 reported by the authors. From personal experiments, I know that simply finetuning both encoders along with the prompt with cross-entropy can achieve much better results on these benchmarks (80.3 % HM on the base-to-novel benchmark, average of 11 datasets). However, the authors seem to focus on just prompt tuning, so it might be ok.

Author response:
Thanks for the positive feedback on our work and recognition of its novelty. To address concerns about SOTA results, we include more comparisons in our “Response to common concern: comparing with the SOTA”, where our method is found to consistently improve more recent methods, including the suggested Maple and KgCoOp.

Reviewer comment (Evaluation):
I'm incline to think that the method is interesting enough for acceptance, even though, in my opinion, the results are not state-of-the-art.

Author response:
Thanks for the positive feedback on our work and recognition of its novelty. To address concerns about SOTA results, we include more comparisons in our “Response to common concern: comparing with the SOTA”, where our method is found to consistently improve more recent methods, including the suggested Maple and KgCoOp.

Reviewer comment (Presentation):
In Tab 3 and Tab 4, the proposed class-conditional feature generator slightly decreases the performance of the base classes. In the appendix Fig 5, there are some explanations regarding the performance increase in the new classes and performance variation in the base classes. These discussions need to move to the main script.

Author response:
**Move Appendix Fig. 5 to the main paper, poor paper writing, and missing objective function in Sec 3.3.**

We appreciate the reviewer for highlighting these presentation issues. We will fix them as suggested, and try to identify and rewrite sections that are dense or difficult to follow as well. If you have other suggestions on areas that could be improved to help us understand where to focus, it would be really helpful.

Reviewer comment (Theory):
The first step of OGEN, novel class extrapolation, is problematic. It is unreasonable to utilize base features to extrapolate novel features, since there are usually large conceptual gaps between base and novel classes. The authors provide a special case, that is “cat, bear->raccoon”. But in CIFAR-10, for example, the “ship” class is not conceptually close to any other classes.

Author response:
**Class extrapolation is unreasonable when there are large conceptual gaps between base and novel classes.**

We agree the quality of class extrapolation highly depends on the semantic similarity between base and novel classes. We do observe faithfully extrapolated class examples when the semantic gap is small (Fig. 3 for example), but less faithful ones otherwise.

Among the many downstream datasets we have experimented on, most of them demonstrate shared regularities between classes (unlike CIFAR10), which gives rise to good extrapolation quality. Examples include datasets with a large number of conceptually close classes like FGVCAircraft and StanfordCars, as well as datasets with only a few but pattern-sharing classes like EuroSAT (satellite images) and DTD (texture images). The well-extrapolated OOD data on these datasets yield consistent, and sometimes large, gains on novel class performance, see Fig. 6(a).

On the other hand, there are datasets UCF101 and Caltech101 that have relatively large semantic gap between classes. Their data size is also small (13k and 9k respectively). Both of the two factors will negatively impact the class extrapolation quality. However, we still observe from Fig. 6(a) convincing gains on novel classes for UCF101 and Caltech101. The reason is twofold: 1) the abundance of extrapolated class examples alleviates overfitting on small datasets by scaling up the negative data, 2) extrapolated data from known classes, despite being noisy, can still serve as ``semi-hard'' negatives to regularize decision boundaries. One natural direction for improvement is to enrich the data used for extrapolation, e.g. by combining a living memory of diverse web data with the few training classes on downstream datasets.

Reviewer comment (Novelty):
Another contribution of this paper, claimed by authors, is Adaptive Local Mean Teacher (ALMT), which is viewed as just a trivial trick of hyperparameter tuning. The difference between ALMT and conventional MT is just modifying the sliding window size. The novelty is quite low, and the performance improvement brought by ALMT over “No distillation” is insignificant (less than 1%), as shown in Table 6.

Author response:
**Limited novelty and performance gains of ALMT.**

In response, ALMT is not the main innovation of this paper, and we view ALMT as a surprisingly simple add-on to reduce overfitting. Note behind the adaptive window size in ALMT lies a well-grounded rationale, i.e. to reduce both overfitting and underfitting during model distillation (see texts before Eq. 7). Such MT adaptation does not require extensive hyperparameter tuning to gain improvements, which is non-trivial.

Performance-wise, Tables 3&6 verify that, ALMT in its simple form, can indeed improve both base (ID) and new (OOD) class performance. Notably, the improvement on base class is larger ($ \approx$ 1\%), which is particularly useful in our case since our OOD feature generator may slightly degrade the base class performance when improving OOD generalization. Overall, ALMT serves as an optional booster that helps to strike a good balance between the base and new class performance. We will add such discussions in the revised paper, with ALMT's role to be toned down accordingly.

Reviewer comment (Writing):
The paper writing is quite poor and hard to follow. The objective function is missing in Sec 3.3.

Author response:
**Move Appendix Fig. 5 to the main paper, poor paper writing, and missing objective function in Sec 3.3.**

We appreciate the reviewer for highlighting these presentation issues. We will fix them as suggested, and try to identify and rewrite sections that are dense or difficult to follow as well. If you have other suggestions on areas that could be improved to help us understand where to focus, it would be really helpful.