Paper: Enhancing Transfer Learning with Flexible Nonparametric Posterior Sampling (ICLR-2024)

Reviewer comment (Theory):
The reviewer feels confused about the authors' claim that "the NPL posterior is robust to model misspecification through the adoption of a nonparametric model and gives asymptotically superior predictions to the regular Bayesian posterior on $\theta$".

Author response:
Thank you for your constructive and positive comment with an important question.

__Q1 Is the NPL posterior robust to model misspecification compared to the regular Bayesian posterior on $\theta$?__

Thank you to the reviewer for posing an important and insightful question. You are indeed correct that the parametric posterior and NPL posterior target the same parameter, that is $\theta^*$ which minimizes the KL divergence between the true $F^*$ and the model $F_\theta$ [Walker (2013)]. 

However, we highlight that NPL elicits a nonparametric prior, which induces a 
posterior distributions on $\theta^*$ that has superior asymptotic properties to the regular Bayesian posterior when the model is misspecified. Intuitively, the reason for this subpar performance for parametric Bayes is that the posterior is computed assuming there exists $\theta^*$ such that $F_{\theta^*} = F^*$. On the other hand, NPL does not make this assumption (i.e. model misspecification is acknowledged), and updates the posterior distribution $\pi_n(F)$ in a nonparametric fashion. This in turn leads to more robust posterior inferences on $\theta^*$, as well as asymptotically superior predictions. This improvement in prediction is indeed observed practically as well [Fong et al. (2019)].

We now outline what we mean specifically by superior asymptotic posteriors and predictions.  Lyddon et al. (2018) show that the Bayesian bootstrap posterior (which has the same limit as NPL) asymptotically has the sandwich covariance matrix, which is known to be robust. On the other hand, the parametric posterior does not obtain this variance asymptotically.

For prediction, we are interested in the posterior predictive density,
$p_n(y) = \int f_\theta(y) \pi_n(\theta)$, where $\pi_n$ is either the Bayesian or the NPL posterior. Theorem 1 of Lyddon et al. (2018) shows that asymptotically, the KL divergence between $F^*$ and $P_n$ is smaller for the NPL posterior compared to the Bayesian posterior. This asymptotic improvement is indeed due to the robust sandwich covariance matrix [Muller (2013)].

Thank you again for highlighting this important point. We will add a detailed discussion of how NPL addresses model misspecification in the revision paper. And if there are no additional questions or uncertainties, we kindly request you to contemplate revisiting your evaluation to ensure it accurately reflects the situation.
 
__References__

[1] Walker, S. G. (2013). Bayesian inference with misspecified models. Journal of statistical planning and inference, 143(10), 1621-1633.

[2] Lyddon, S. P., Holmes, C. C., & Walker, S. G. (2019). General Bayesian updating and the loss-likelihood bootstrap. Biometrika, 106(2), 465-478.


[3] Lyddon, S., Walker, S., & Holmes, C. C. (2018). Nonparametric learning from Bayesian models with randomized objective functions. Advances in neural information processing systems, 31.

[4] Müller, U. K. (2013). Risk of Bayesian inference in misspecified models, and the sandwich covariance matrix. Econometrica, 81(5), 1805-1849.

[5] Fong, E., Lyddon, S., & Holmes, C. (2019, May). Scalable nonparametric sampling from multimodal posteriors with the posterior bootstrap. In International Conference on Machine Learning (pp. 1952-1962). PMLR.

Reviewer comment (Theory):
The reviewer notes that both methods assume a parameterized model with parameters $\theta$, meaning the assumed parameterized model remains susceptible to possible discrepancies with the true data distribution, which could result in model misspecification. Therefore, if this understanding is correct, NPL may not inherently address the issue of model misspecification.

Author response:
Thank you for your constructive and positive comment with an important question.

__Q1 Is the NPL posterior robust to model misspecification compared to the regular Bayesian posterior on $\theta$?__

Thank you to the reviewer for posing an important and insightful question. You are indeed correct that the parametric posterior and NPL posterior target the same parameter, that is $\theta^*$ which minimizes the KL divergence between the true $F^*$ and the model $F_\theta$ [Walker (2013)]. 

However, we highlight that NPL elicits a nonparametric prior, which induces a 
posterior distributions on $\theta^*$ that has superior asymptotic properties to the regular Bayesian posterior when the model is misspecified. Intuitively, the reason for this subpar performance for parametric Bayes is that the posterior is computed assuming there exists $\theta^*$ such that $F_{\theta^*} = F^*$. On the other hand, NPL does not make this assumption (i.e. model misspecification is acknowledged), and updates the posterior distribution $\pi_n(F)$ in a nonparametric fashion. This in turn leads to more robust posterior inferences on $\theta^*$, as well as asymptotically superior predictions. This improvement in prediction is indeed observed practically as well [Fong et al. (2019)].

We now outline what we mean specifically by superior asymptotic posteriors and predictions.  Lyddon et al. (2018) show that the Bayesian bootstrap posterior (which has the same limit as NPL) asymptotically has the sandwich covariance matrix, which is known to be robust. On the other hand, the parametric posterior does not obtain this variance asymptotically.

For prediction, we are interested in the posterior predictive density,
$p_n(y) = \int f_\theta(y) \pi_n(\theta)$, where $\pi_n$ is either the Bayesian or the NPL posterior. Theorem 1 of Lyddon et al. (2018) shows that asymptotically, the KL divergence between $F^*$ and $P_n$ is smaller for the NPL posterior compared to the Bayesian posterior. This asymptotic improvement is indeed due to the robust sandwich covariance matrix [Muller (2013)].

Thank you again for highlighting this important point. We will add a detailed discussion of how NPL addresses model misspecification in the revision paper. And if there are no additional questions or uncertainties, we kindly request you to contemplate revisiting your evaluation to ensure it accurately reflects the situation.
 
__References__

[1] Walker, S. G. (2013). Bayesian inference with misspecified models. Journal of statistical planning and inference, 143(10), 1621-1633.

[2] Lyddon, S. P., Holmes, C. C., & Walker, S. G. (2019). General Bayesian updating and the loss-likelihood bootstrap. Biometrika, 106(2), 465-478.


[3] Lyddon, S., Walker, S., & Holmes, C. C. (2018). Nonparametric learning from Bayesian models with randomized objective functions. Advances in neural information processing systems, 31.

[4] Müller, U. K. (2013). Risk of Bayesian inference in misspecified models, and the sandwich covariance matrix. Econometrica, 81(5), 1805-1849.

[5] Fong, E., Lyddon, S., & Holmes, C. (2019, May). Scalable nonparametric sampling from multimodal posteriors with the posterior bootstrap. In International Conference on Machine Learning (pp. 1952-1962). PMLR.

Reviewer comment (Evaluation):
A detailed discussion of limitations is lacking. A possible aspect could be the computation cost. A careful discussion of pros and cons could be very helpful to the community.

Author response:
As you pointed out, the apparent downside of the suggested NPTL approach is the extra training cost. For instance, applying the proposed NPTL algorithm involves additional computational expenses to obtain $f_{\text{probed}}$ through linear probing for the provided downstream data. Nevertheless, the computational cost associated with linear probing is minimal in contrast to training the complete network. We will add a discussion on such limitations regarding computational costs in the next version of the paper.

Reviewer comment (Experiments):
Following the experiments of [1], could the proposed method also use self-supervised pretrained models and include experiments about segmentation?

Author response:
Thank you for posing an important question about constructing pre-trained models. Since the informative base measure $F_\pi(x, y)$ could be any probability measure representing our initial assumption about $F_0$ based on our prior knowledge of the actual data distribution, we have the flexibility to employ linear probed self-supervised pre-trained models as our model $f_{\text{probed}}. Indeed, our preliminary experimental results on Caltech101, utilizing ResNet-50-SimCLR as the pre-trained model, are as follows: Ensemble + L2SP attains ACC=0.875 and NLL=0.441, whereas NPTL attains ACC=0.879 and NLL=0.410. We believe that integrating the outcomes of self-supervised pre-training will undoubtedly enhance the paper, and we are grateful for constructive feedback. Additional results regarding self-supervised pre-training will be available in the camera-ready version.

__Q1 (2) Experiments on semantic segmentation__

Understanding semantic segmentation as pixel-level classification would simplify the implementation of the proposed NPTL in semantic segmentation experiments. More precisely, we first obtain $f_{\text{probed}}$ by exclusively training additional modules for semantic segmentation tasks while keeping the fixed (self-supervised) pre-trained backbone network. Subsequently, we proceed to train the entire semantic segmentation model, incorporating the proposed NPTL term.

In this manner, there is no obstacle to the conceptual application of our NPTL methodology to semantic segmentation tasks. We are of the opinion that providing additional experimental results for semantic segmentation, as per your suggestion, will enhance the emphasis on our contribution. Nevertheless, due to practical limitations like restricted computational resources and time constraints, we regretfully acknowledge that we may be unable to provide additional experiments on semantic segmentation during the rebuttal period. Even if the results are not ready by then, we will be committed to incorporating them into the camera-ready version. We earnestly request your understanding and consideration of this matter for your assessment.

Reviewer comment (Theory):
Access to the posterior distribution given pre-trained models. In general, I see the idea, but it is not entirely clear to me how we can get posterior samples from any model that has been pre-trained without a particular prior before. Are we in the MAP solution? similarly to the Laplace approximation. Are there conditions or requirements for pre-training the models?

Author response:
Thank you for pointing out an important question. As we commented on the question from reviewer izux, there is no restriction to constructing pre-trained models. Since the informative base measure $F_\pi(x, y)$ could be __any probability measure representing our initial assumption about $F_0$ based on our prior knowledge of the actual data distribution__, we have the flexibility to employ any type of pre-trained models that contain information of generating process of the upstream dataset (MAP solution from supervised tasks or solution from self-supervised).

Reviewer comment (Evaluation):
I appreciate the details and the sincere comments on the heuristics used and so on. However, I do not get a good feeling on the scalability and the computational cost. Is really the methods playing a role given huge models with large number of parameters? or is it just bc the pre-trained models are doing still well on similar vision tasks. In that regard, I'm not entirely convinced by the empirical results.

Author response:
As you pointed out, there is a trend in modern transfer learning to assume large pre-trained models on extensive upstream data. Consequently, it would be essential to assess the scalability of transfer learning approaches. To this end, we conducted experiments employing the sizable ViT-B/16 and RoBERTa-Base models (Specifically, ResNet-50 has approximately 23 million parameters, whereas ViT-B/16 and RoBERTa-Base have 86 million and 123 million parameters, respectively). We believe that our experiments using ViT and RoBERTa adequately validate the scalability of the proposed NPTL methodology.

Your question brings up an intriguing point. Indeed, a well-pretrained model tends to perform effectively across related tasks. Yet, if the prior distribution isn't suitable, it can significantly impact the performance of the Bayesian model averaging (BMA) with posterior samples. To demonstrate this, we showcase the BMA performance using the zero-mean isotropic Gaussian prior, denoted as $\mathcal{N}(0,\sigma^2 * I)$. The empirical results in Table R.1. and Table R.2. distinctly illustrate how an unsuitable prior distribution can markedly degrade performance.

__Table R.1.__ Additional BMA accuracy performance of (a) SGHMC with zero-mean isotropic Gaussian prior and (b) NPTL with ResNet-20x4.
|     | CIFAR-10 | CIFAR-100 | Food-101 | Dogs-120 |
| :-  | :-:          | :-:          | :-:          | :-:          |
| (a) | 0.872 ± 0.000 | 0.625 ± 0.001 | 0.550 ± 0.001 | 0.235 ± 0.001 |
| (b) | 0.964 ± 0.001 | 0.818 ± 0.000 | 0.644 ± 0.002 | 0.634 ± 0.001 |

__Table R.2.__ Additional BMA negative log-likelihood performance of (a) SGHMC with zero-mean isotropic Gaussian and (b) NPTL with ResNet-20x4.
|     | CIFAR-10 | CIFAR-100 | Food-101 | Dogs-120 |
| :-  | :-:          | :-:          | :-:          | :-:          |
| (a) | 0.381 ± 0.001 | 1.321 ± 0.001 | 1.733 ± 0.003 | 3.071 ± 0.002 |
| (b) | 0.102 ± 0.001 | 0.606 ± 0.002 | 1.347 ± 0.001 | 1.297 ± 0.003 |

Reviewer comment (Reproducibility):
For the linear prob model, my understanding is that $\phi^*$ is from the pre-trained model, how do we get $W^*$, is it obtained by only fitting the last fully connected layer on the downstream task?

Author response:
Your understanding is correct. We specifically fit the last fully-connected layer for the downstream task, ensuring that our $(\phi^\ast, W^\ast)$ retains the essence of the upstream data-generating process in $\phi^\ast$ and transfers the knowledge from the upstream through $W^\ast$.

Reviewer comment (Reproducibility):
In step 7 of algorithm 1, how is $\theta$ initialized, randomly or by $(\phi^*, W^*)$, i.e. is the pre-trained model only used to create pseudo samples for the prior, or is it also used to initialize parameter as well. Looking at the training setting, e.g. ResNet-50, with a cosine learning rate decay schedule, it looks like the parameters are trained from scratch, otherwise, I imagine the large initial learning rate would drive parameters away from the initialization.

Author response:
Our proposed posterior sampling method is tailored for transfer learning scenarios, and we initiated the model parameters as $\mathbf{\theta} = (\phi^\ast, W)$, where $\phi^\ast$ represents a pre-trained feature extractor parameter, and $W$ is the randomly initialized task-specific head parameter. The linear-probed head parameter $W^\ast$ is solely utilized in constructing the informative base measure for NPTL (although it could potentially be initialized with $W^\ast$, we assert that using the same random initialization for all methods offers a more equitable basis for experimental comparison).

As you pointed out, when fine-tuning from a pre-trained solution, it is crucial to be cautious about avoiding substantial deviations from it. In our ResNet-50 experiments, we opted for a learning rate of 0.01, smaller than the conventional value used in from-scratch training (e.g. 0.1 in practice). Consequently, we did not encounter situations where the model significantly deviated from the pre-trained initialization. Additionally, when testing larger learning rate values such as 0.1 and 0.03, we observed a significant performance decline in line with your concern, indicating a departure from the pre-trained initialization.

Reviewer comment (Experiments):
For the SGHMC baseline, how is the prior specified and how does the pre-trained model help the SGHMC method? And how does the pre-trained model help the ensemble baseline?

Author response:
For the SGHMC baseline, we employed two distinct weight prior distributions as outlined below:
1. **L2SP:** Gaussian prior with a mean set as the pre-trained weight and a variance of $\sigma^2 * I$, where $I$ represents the identity matrix.
2. **PTYL:** Gaussian prior with a mean derived from the pre-trained weight, further trained on the upstream dataset (acknowledged as challenging to utilize, as emphasized in the paper). The variance is a non-diagonal matrix $\Sigma$ constructed with empirical covariance using weights gathered during additional training on the upstream dataset.

Considering the context of posterior sampling in the transfer learning scenario, both the ensemble baselines and the SGHMC method adopt the same initialization approach. Specifically, we initialize our method and all the baselines with pre-trained feature extractor parameters $\phi$ and randomly initialized task-specific parameters $W$.

Reviewer comment (Theory):
In section 2.1 paragraph 1, the zero mean isotropic Gaussian is called a non-informative prior. If I remember correctly, the non-informative prior usually refers to something else.

Author response:
Thank you for bringing this confusion to our attention. We will seek more suitable language and make the necessary corrections.

Reviewer comment (Theory):
For the Bayesian inference, one criterion of setting prior can be ensuring posterior concentration, e.g. in the sense of theorem 2.1 in [1]. Can the authors comment a bit on how the proposed prior related to those concepts?

Author response:
Thank you for highlighting the importance of posterior concentration. As NPL is based on the Dirichlet Process, we can leverage standard results. In fact, more can be said than the posterior contraction rate - for example, Theorem 12.2 of Ghosal & Van der Vaart (2017) gives a Bernstein-von Mises theorem for the Dirichlet process.