Paper: Text Descriptions are Compressive and Invariant Representations for Visual Learning (ICLR-2024)

Reviewer comment (Theory):
The theoretical analysis is not well-aligned with the proposed method:  
- The major conclusion that could be derived from sec 3.3 is that features from CLIP's joint VL space are more compressed and invariant to visual variations (compared with vision features).  
- This is good and helps us understand CLIP itself, but does not explain why using descriptions (no matter w/ or w/o selection) is better than using other texts (e.g., the default templates), which is the foundation of the proposed method.  
- From fig. 2, $I(H_{avd}; A)$ is almost identical to $I(H_{cp}; A)$, indicating descriptions do not introduce more invariance than template ensemble, which raises concern on why descriptions are needed in this work. $I(H_{avd}; Y)$ is higher than $I(H_{cp}; Y)$, which should indicate better predictions, yet in tab. 1 the gain of ZS-AVD over ZS is just marginal.

Author response:
It is actually a favorable result that $I(H_{avd}, A)$ is almost identical to $I(H_{cp}, A)$. One can think of $H_{cp}$ as a postprocessing of $H_{avd}$ (since the former is a subset of the latter). This result suggests that even if $H_{avd}$ has more features, these extra features do not harm invariance. The difference between $I(H_{avd}, Y)$ and $I(H_{cp}, Y)$ is reflected in the newly revised Figure 3. There the L1 class prompts perform far worse than AVD.

Reviewer comment (Experiments):
The experiments also do not support this work (description selection)'s superiority over other template designs:  
- In tab. 2 & 3, SLR's superiority over FT & LP under the few-shot setting is expected, since both WISE-FT's $W_\text{learned}$ and LP's classifier are trained from scratch given only very few samples, thus could not match the performance of classifiers derived from CLIP's text encoder.  
- Yet this does not help understand SLR's strength over other classifiers. For instance:  
  a) what if we drop the selection process and just follow Menon & Vondric's average ensemble,  
  b) what about WaffleCLIP [1]'s random descriptors,  
  c) how much is SLR's improvement over CLIP's default templates, and  
  d) how do k-NN classifiers (use average pooling of the labeled few-shot samples to form classifiers, try both vision features and VL features) perform?

Author response:
2.1 Even though SLR’s features are derived from text encoders, its coefficients are not regularized to be close to ZS-AVD. Instead, we only enforce sparsity. Therefore, the model can still be wrong in arbitrary ways. The results suggest that the discrete text space has a stronger implicit bias than the image embedding space.  
2.2 the average ensemble corresponds to ZS-VD or ZS-AVD. Notice that without WISE-FT, SLR doesn’t really incorporate the strong language prior. Since only sparsity is enforced, the coefficients can still overfit the few-shot data in almost arbitrary ways. One can of course overcome this issue by regularizing the difference between the SLR coefficients and the zeroshot weights, which is almost what WISE-FT does. Compared to regularization during SLR training, WISE-FT is post hoc. So we only need to train the model once. We choose to train this way because it helps us understand the implicit bias of each space (image embedding space vs the discrete text space that is congruent to natural language), rather than the effect of the zeroshot prior.  

We add WaffleCLIP zeroshot performance in Table 12, and we notice that its performance tends to decrease as you add more random tokens. On the other hand, our descriptors have length around 8-10 tokens. This suggests that the most important factor is to perturb your basis around the class prompt in a controlled way, which is more easily achieved by using language congruent descriptions. We also compare WaffleCLIP+WISE-FT to SLR+WISE-FT. The result in figure 15 also suggests that SLR has a better performance than WaffleCLIP.

Reviewer comment (Experiments):
One possible cause of ZS-VD's much inferior performance than ZS in tab.1 is the use of class templates. If the templates of ZS are also applied to ZS-VD, the difference in their performance should be marginal (refer to WaffleCLIP). The reviewer suggests the authors give it a try.

Author response:
In Table 12, we also tried to average the templates for ZS-VD, the results are listed in column ZS-VD$^2$. We see a slight improvement.  

We also want to remark that the gap between WaffleCLIP and our ZS baseline is because we use the hand-selected 7 templates later released in the OpenAI CLIP github repo, and the original WaffleCLIP paper randomly selects 30 out of the original 80 templates for ensembling.

Reviewer comment (Experiments):
It will be good to have an overall comparison with different methods and an ablation study of the proposed method. I am thinking mainly of using the same base model (e.g. ViT-B/16), maybe just some standard few-shot k (e.g. k=16). A table with all datasets as columns and different methods (ZS, LP, MLP probing, FullFinetunning, CoOp, Wise) and ablations (ZS-AVD, SLR-AVD etc.).

Author response:
We added the comparison in Table 13 in the appendix for k=4, since this is the k that we have used for every model.

Reviewer comment (Experiments):
It will be good to see the performance of *all* models presented using different numbers of shots: 1, 2, 4, 8, 16, 32. At the moment some ablations contain only up to 4 or 16 shots.

Author response:
We have added the comparison to CoOp on 32 shots, see figure 15 in the appendix. For Full FT, we are limited by our resources.

Reviewer comment (Experiments):
The paper focuses on few-shot learning and this is an important area. To see the tradeoffs of the proposed method and the baselines, it will be interesting to also see the performance using larger training sets. For example, use k from 64, 256, 1024, etc. This way we can see at which scale of training samples the proposed method is more beneficial.

Author response:
See Fgure 16 in the appendix for larger k. We tried k=64, 256, 1024. L1 AVD still outperforms linear probing on most datasets.

Reviewer comment (Experiments):
Comparison with CoOP does not seem to be fair. “Since CoOp injects ‘classname’ to the prompt during inference directly, this enforces a very strong prior. For a fair comparison, we also inject a strong prior by interpolating our learned linear head”. It is not clear what this means, and why a direct comparison is not fair. Why is the ‘classname’ injection a strong prior for CoOP? Isn’t SLR-AVD also using the same classname to produce the class prompts (CP)? Combining the proposed method with Wise and comparing to plain CoOp seems unfair.

Author response:
The reason behind this statement is that CoOp optimizes a continuous prompt like “X X X X {classname}” for each class, and only the “X X X X” part is optimized in the continuous space; the “{classname}” is always fixed. On the other hand, the coefficients of SLR-AVD are not regularized until it incorporates WISE-FT. To prevent confusion, we change Table 5 so that CoOp also incorporates WISE-FT, and SLR-AVD still outperforms CoOp, especially when the amount of data increases. 

W5.2. CoOp comparison now is done on ViT-B/16.

Reviewer comment (Experiments):
The comparison with CoOP is made using Resnet-50, which gives poorer performance for CLIP. Comparison using the ViT models should also be made.

Author response:
CoOp comparison now is done on ViT-B/16.

Reviewer comment (Experiments):
It is not clear how hyperparameter selection is done. Hyperparameter and model selection are crucial for domain generalization, thus this should be made more clear. For Wise models, alpha seems to be selected optimally, using validation OOD data.

Author response:
This is one drawback that we inherit from WISE-FT. The nice property of WISE-FT is that it’s fully post hoc. So in reality, even if the $\alpha$ is not optimally picked initially, it can always be easily updated. The original WISE-FT method suggests a heuristic of choosing $\alpha=0.5$. However, their original setting has abundant ID data, and 0.5 fails in our case. Our heuristic suggests that picking $\alpha=0.05k$ seems to perform pretty well.

Reviewer comment (Experiments):
In Figure 4 top, it seems like WISE-FT+LP, which finetunes only the last linear layer, is compared against WISE-FT+SLR-AVD, which finetunes the entire model. Is this correct, or is WISE-FT+SLR-AVD finetunning only the linear classifier? Both should either update the linear layer or full-finetuning. Also, what is the difference between WISE-FT+LP and WISE-SLR?

Author response:
WISE-FT+SLR-AVD only finetunes the last layer. WISE-SLR is the method that first finds a sparsity pattern using SLR-AVD. Then we fixed the pattern and finetuned the whole model. WISE-FT+LP is linear probing with image embeddings +WISE-FT on only the last linear layer.

Reviewer comment (Presentation):
Minor: Figure 4 should be improved, e.g. use consistent symbols, especially for start and end points. Show the optimal checkpoint in the figure.

Author response:
We updated the figures. As for optimal checkpoint, it is a little ambiguous since sometimes the optimal ID acc and OOD acc are achieved by slightly different $\alpha$, so we decide not to include it.

Reviewer comment (Reproducibility):
What is the number of learnable parameters of SLR-AVD, how does it compare to linear probing?

Author response:
The total learnable parameters in SLR-AVD are 6804\*1000. The average number of non-zero entries for each class after learning is $447, 248, 182, 177, 173$, and $135$ for $k=1,2,4,8,16,32$. The numbers are rounded to the nearest integers. So the total number of “used” parameters is the aforementioned numbers * 1000 (for each class). Linear probing optimizes 512\*1000 parameters. So SLR does in the end lead to much less parameters. Also, as the training data increases, SLR-AVD can eliminate more irrelevant descriptors.

Reviewer comment (Experiments):
The benefit of automatically generating prompts for obtaining zero-shot classifiers is not very clear and significant. Table-1 compares the proposed method of generating textual descriptions from GPT-3 (ZS-AVD) against using only manually defined class prompts (ZS), and we see marginal improvements (max a few decimal points). It would be nice to see the impact of the number and diversity of generated prompts into performance. Section 4.1 mentions that "...class names are probably one of the strongest prompts... One can certainly try to improve ZS-VD results by more carefully prompting GPT-3, or gathering descriptors from different data sources/search engines." This contradicts with the motivation of the paper, no?

Author response:
The main motivation is that we want to use multiple features to learn a classifier in the presence of few shot data. If we only have one prompt for each class, then there is no candidate to learn from (except using the vanilla image embeddings). An important finding is that the visual descriptions perturb the original class prompt in a specific way such that the perturbed features have nice information theoretic properties. See the newly added Table 12 in the appendix for a comparison to WaffleCLIP, which appends random words to the class prompts. Our visual descriptors have length around 8-10 tokens, and WaffleCLIP’s performance decreases as the random token length increases. WaffleCLIP only does well when a couple words are added. This small perturbation is beneficial even if the words are irrelevant. However, when more words (8-10) are added, WaffleCLIP performance decreases, while our method can pick useful descriptors of this length that constructively enhance the original class prompt.  
We also have a small scale experiments on CIFAR10 in the appendix, paragraph "Choosing $\gamma$ and LLM prompting". There we found the most important factor is to generate more diverse descriptors by setting the frequency penalty.

Reviewer comment (Experiments):
The benefit of using $\ell_1$-regularized logistic regression technique is not very clear either. It would be nice to see if simple $\ell_2$ regularization performs the same, or what kind of prompts are selected for certain classes using different regularization criteria. Also, a simple k-NN based approach can be applied as a baseline both with soft or hard assignment. On the other hand, similar logistic regression can be trained also for ZS (where there can be multiple manually defined class prompts). Maybe the impact is only due to learning weights ($W$) which connect the two modalities (image and text) using some regularization technique. Section-5 mentions that "Applying sparse logistic regression then successfully selects the important features, which turn out to be intuitive" but we don't see any evidence, right?

Author response:
See revised figure 3. L1 consistently leads to better performance than L2. Sparsely chosen class prompts perform worse than AVD on most datasets. These results still hold when combining with WISE-FT, see figure 15 in the appendix. All results indicate that choosing features with L1 is important.

Reviewer comment (Writing):
1st sentence of Introduction: "Self-supervised vision-language models (VLMs) like CLIP..." — is there any reference claiming CLIP to be self-supervised?

Author response:
-"1st sentence of Introduction: "Self-supervised vision-language models (VLMs) like CLIP..." is there any reference claiming CLIP to be self-supervised?” 
We changed this to “natural language supervised.”

Reviewer comment (Writing):
2nd sentence of Section-2: "WLOG..." — what is WLOG?

Author response:
-"2nd sentence of Section-2: "WLOG..." what is WLOG?”
This stands for “without loss of generality.” We modified the text to clarify this term.

Reviewer comment (Presentation):
Missing closing parenthesis in Figure-1.

Author response:
-"Missing closing paranthesis in Figure-1”
Noted - we fixed this in the paper.

Reviewer comment (Presentation):
4th paragraph of Section-3.1 ("Denote M ...") is very confusing. It would be nice to explain all $U$ and $W$ in a diagram/visualization.

Author response:
-"4th paragraph of Section-3.1 ("Denote M ...") is very confusing. It would be nice to explain all $U$ and $W$ in a diagram/visualization. "
We added a visualization of $U, W$ at the end of the appendix

Reviewer comment (Presentation):
Figure-3 caption "the x-axis represents..." (not y).

Author response:
-"Figure-3 caption "the x-axis represents..." (not y)” and “what is the label for y-axis?”
The y-axis refers to test accuracy; we have modified the text accordingly.

Reviewer comment (Presentation):
Figure-3 — what is the label for y-axis?

Author response:
-"Figure-3 caption "the x-axis represents..." (not y)” and “what is the label for y-axis?”
The y-axis refers to test accuracy; we have modified the text accordingly.

Reviewer comment (Reproducibility):
The training and inference process is not clear.

Author response:
1.1 The model is trained by the normal multiclass cross entropy loss + L1 regularization.

1.3 The particular algorithm we use to perform cross-entropy loss minimization with L1 regularization is a mini-batch invariant of SAGA, which is a first-order method, detailed in [1]. The appendix paragraph “hyperparameter” gives a brief introduction to the method used.

Reviewer comment (Reproducibility):
It is not specified what loss function is used to train the model.

Author response:
1.1 The model is trained by the normal multiclass cross entropy loss + L1 regularization.

Reviewer comment (Evaluation):
The experiments show limited performance gain in zero-shot classification.

Author response:
2 While the improvement observed is relatively modest, we believe the exploration of learning with descriptive features is still noteworthy. The application of information theoretic results in this context is itself valuable, as it provides a useful framework for analyzing these features. Furthermore, the MI-based analysis offers a convincing explanation for the strong performance and robustness of both AVD and the zeroshot baseline, which is an interesting aspect on its own.

Reviewer comment (Evaluation):
The results in Table 1 indicate that ZS-VD performs worse than ZS, and ZS-AVD only provides marginal improvement over ZS, i.e., 0.74 (IN), 0.74 (IN-V2), 0.13 (IN-R), 0.23 (IN-A), 0.27 (ObjectNet).

Author response:
2 While the improvement observed is relatively modest, we believe the exploration of learning with descriptive features is still noteworthy. The application of information theoretic results in this context is itself valuable, as it provides a useful framework for analyzing these features. Furthermore, the MI-based analysis offers a convincing explanation for the strong performance and robustness of both AVD and the zeroshot baseline, which is an interesting aspect on its own.

Reviewer comment (Presentation):
In Figure 3, it would be interesting to show the performance of the ZS-VD model.

Author response:
3 The results are added, see the updated figure 3. VD performs worse than AVD but better than class prompts, suggesting the descriptors are useful.

Reviewer comment (Experiments):
In section 4.2, it is unclear which dataset is the OOD test set.

Author response:
4 The OOD datasets are ImageNet-A (an adversarial dataset), ImageNet-R (only contains arts and paintings etc), ImageNet-Sketch (only contains sketch), ImageNet-V2 (natural distribution shift), and ObjectNet.

Reviewer comment (Presentation):
Figure 4 is unclear and the sub-caption in the bottom-right corner is blocked.

Author response:
Figure 4 has been updated according to the comments.