Paper: Understanding prompt engineering may not require rethinking generalization (ICLR-2024)

Reviewer comment (Theory):
The validity of almost all derivations in the manuscript hinges on the assumption that there is no data contamination and that the encoder was not trained on the data it is asked to infer (i.e., the setting is truly zero-shot). This assumption cannot be fully proved or disproved, partially due to the lack of full transparency and public knowledge about the training data content.

Author response:
Thank you for your continued support for this work!

> data contamination

Since our last discussion, there has been a new work that shows removing imagenet-like data from LAION does not significantly affect the performance of CLIP [1], so we believe that this offers a partial answer to the question. Among other things, the paper showed that even if there is (almost) no data contamination, CLIP would still work well, and we can expect the bound to hold. 
However, we agree that studying data contamination in pretraining datasets will continue to be an important problem moving forward. 

**Reference**

[1] Does CLIP's Generalization Performance Mainly Stem from High Train-Test Similarity? Mayilvahanan et al.

Reviewer comment (Experiments):
The proposed bounds are only evaluated on image classification tasks, but prompting and prompt engineering are equally if not more common in NLP for text classification and other tasks.

Author response:
> The proposed bounds are only evaluated on image classification tasks

This is a good point. This choice was in line with the existing literature on generalization bounds in deep learning. Extending this methodology to NLP is indeed an interesting future direction, though it would require careful consideration of specific protocols since there is not a well-established one.

Reviewer comment (Evaluation):
The abstract claims that "the bound is remarkably suitable for model selection: the models with the best bound typically have the best test performance." Based on Figure 3, it seems like the bound is no more useful than just looking at the training error. But it's hard to tell because the points in the plot are not paired. Are there examples where using the PAC-Bayes bound leads to a different and better model selection choice than just using the training error?

Author response:
> Based on Figure 3, it seems like the bound is no more useful than just looking at the training error.

This might be a misunderstanding. We do observe that, especially in regions of low training error, lower PAC-Bayes bounds (blue dots) often align with lower test errors. Conversely, low training errors (orange dots) can correspond to higher test errors. This indicates that our PAC-Bayes bound is a more reliable indicator of generalization than training error alone. We are currently revising the figure's caption to make this point clearer.

Reviewer comment (Evaluation):
As far as I can tell, the analysis doesn't actually suggest a useful practical algorithm. Based on Figures 3 and 4, if I want to find the best prompt, it seems like my best bet is still to just pick the one that minimizes the training error (using whatever search method I come up with) and not bother with SRM.

Author response:
> As far as I can tell, the analysis doesn't actually suggest a useful practical algorithm. Based on Figures 3 and 4, if I want to find the best prompt, it seems like my best bet is still to just pick the one that minimizes the training error

As we mentioned above, Figure 3 in fact shows that prompts that have low training error can actually have high test error (e.g., negative correlation) whereas PAC-Bayes bounds correlate well with generalization.

Reviewer comment (Evaluation):
linear probe significantly better than using the greedy prompt search as shown in Tab. 3, which calls the performance of the greedy method into question and makes it unclear whether one should even use prompt engineering in such cases then despite the tighter bounds

Author response:
> linear probe significantly better than using the greedy prompt search as shown in Tab. 3

The goal of the paper is not to argue the superiority of prompt engineering over finetuning. The experiment's goal is to highlight that prompt engineering has a much smaller hypothesis space than even linear probing so they naturally generalize well. In most applications of pretrained models, finetuning with gradients would generally outperform prompting, but prompting has the benefit of being extremely simple and intuitive (i.e., requiring only language-based descriptions of classes), and can work even in the absence of an explicit finetuning set or parameter update (though such a finetuning set would, of course, be needed to compute our actual bounds).  Indeed, we feel it is largely for these reasons that prompt engineering is such a dominant approach in practice.  Further, although we highlight prompt search as a method (which again would require a finetuning set), the most common usage of prompt engineering is usually in this more manual setting.

Reviewer comment (Theory):
introduction & abstract dwell on the issue of classical deep learning PAC Bayes bounds, which are usually vacuous, unless conditioned on data. However, pretraining on a massive data set is rather similar to conditioning on data so I find this a bit inconsistent. I suppose it would be good to define data-dependent clearly if it applies to standard bounds but not the proposed one. Especially considering the likely case of data leakage into clip embeddings, claiming that the proposed method is not data-dependent seems inappropriate.

Author response:
> introduction & abstract dwell on the issue of classical deep learning PAC Bayes bounds…

“Data dependent” generally refers to the fact that the prior is dependent on the data from the task of interest (e.g., CIFAR10) [1]. Concretely, one would partition the training set into two disjoint sets and use one set to define the prior and the other set to learn the model. This definition does not preclude using other data (unless of course there is data leakage). A recent work [2] showed that CLIP still works well on Imagenet even if one removes instances similar to ImageNet, so we believe that data leakage may not be a significant issue for CLIP’s performance. We will revise the text to make it clearer what we mean by data-dependent (currently still revising).

Reviewer comment (Novelty):
title claims "understanding" but the paper did not really provide a new understanding of prompt engineering, I would argue, and the authors seem to agree with this as can be seen in the 2nd paragraph of the conclusion. The 3rd paragraph of the conclusion is much more clear and honest given the results.

Author response:
> title claims "understanding" but the paper did not really provide a new understanding of prompt engineering…

The core insight we hope to highlight is that prompt engineering operates in an extremely “small” hypothesis space by the standard of deep learning, which comes with good generalization properties out of the box. We believe that this is an interesting, if not valuable, observation that is not well-known currently in the community. The goal of the conclusion is to highlight that understanding prompt engineering requires understanding something else beyond generalization which is in line with our title.

Reviewer comment (Novelty):
data-dependent PAC-Bayes bounds would learn a prior on parameters based on a small pretraining dataset. This is extremely similar to the approach of pretraining a vision-language model. Would you disagree with this or see this differently?

Author response:
> This is extremely similar to the approach of pretraining a vision-language model. Would you disagree with this or see this differently?

This is a great question. They are similar in the sense they both leverage some data that are not the training data but there are some subtle differences. In the data-dependent PAC-Bayes literature, this pre-training data is chosen to be from the same distribution as the actual data (e.g., partitioning the training data and using the small subset to learn the prior). This ensures that the bound is tight because the weights would not move “far” from the prior since the two partitions are from the same distribution.
This is generally not the case for VLM models where it is trained on a lot of data that are not from the same distribution and not with the same objective. This difference may seem subtle but it is crucial in practice and in theory since we have a large amount of data from different tasks but not necessarily from the same task.