We exhibit a strong link between frequentist PAC-Bayesian bounds and the Bayesian marginal likelihood. That is, for the negative log-likelihood loss function, we show that the minimization of PAC-Bayesian generalization bounds maximizes the Bayesian marginal likelihood. This provides an alternative explanation to the Bayesian Occam's razor criteria, under the assumption that the data is generated by a i.i.d. distribution. Moreover, as the negative log-likelihood is an unbounded loss function, we motivate and propose a PAC-Bayesian theorem tailored for the sub-Gamma loss family, and we show that our approach is sound on classical Bayesian linear regression tasks.
PAC-Bayesian bounds for the NLL loss function are intimately related to learning Bayesian inference REF .
PAC-Bayesian Theory Meets Bayesian Inference
Advances in Neural Information Processing Systems 29 (NIPS 2016), p. 1884-1892
Computer Science Mathematics