Paper: A Bias-Variance-Covariance Decomposition of Kernel Scores for Generative Models (ICLR-2024)

Reviewer comment (Evaluation):
Can you explain the reason why this method works best on question answering datasets?

Author response:
Kernel scores are loss functions giving good values when the prediction is close to the ground truth and bad values when not. In Figure 4 and 6, the Pearson correlation between the kernel entropy (our uncertainty measure) and the loss (kernel score) shows that kernel entropy is very effective in predicting the loss. 
The nlg results in Figure 7 confirm that this holds for the loss used to evaluate question answering datasets.
Part of the explanation for our good results is that the embedder transforms the generated answers into a semantically meaningful vector space, in which the kernel then compares the similarities. Vector embeddings combined with the cosine similarity have a long history in the NLP domain, which underlines their effectiveness (Camacho-Collados et al, 2018).

**References:**

Jose Camacho-Collados and Mohammad Taher Pilehvar. From word to sense embeddings: A survey on vector representations of meaning. Journal of Artificial Intelligence Research, 63:743–788, 2018.

Reviewer comment (Theory):
I’m wondering how to generalize the theory to others except the kernel scores.

Author response:
We were able to derive the bias-variance-covariance decomposition by using a quadratic form in the definition of kernel scores. This quadratic form is in general not present for other losses, which prohibits deriving a covariance term.

Reviewer comment (Presentation):
In figure 4, what are the points being plotted? The 20 different models?

Author response:
* Q: “In figure 4, what are the points being plotted? The 20 different models?”

A: Each point in Figure 4 corresponds to a digit.

Reviewer comment (Experiments):
Why is only a single model used to compute kernel entropy? Are there any advantages of using multiple models to compute the entropy?

Author response:
* Q: “Why is only a single model used to compute kernel entropy? Are there any advantages of using multiple models to compute the entropy?”

A: We compute the kernel entropy based on single models to stay as truthful as possible to practical constraints: Generative models are usually computationally very expensive and closed-source access is almost always provided only to single models. It is likely that the average kernel entropy of multiple models gives even better results because this improves the estimation.

Reviewer comment (Evaluation):
How is the bias calculated in the mode-collapse experiment of section 5.1? Is the original distribution used to compute the bias, or the modified distribution with reduced frequency of class 0 used?

Author response:
* Q: “How is the bias calculated in the mode-collapse experiment of section 5.1? Is the original distribution used to compute the bias, or the modified distribution with reduced frequency of class 0 used?”

A: The MMD, bias, and variance values in Figure 5 are only computed with respect to digit ‘0’ (which has reduced frequency in the training data). 
The other digits in the mode-collapse experiment have similar lines as in Figure 3 (no reduced frequency in the training data).

Reviewer comment (Experiments):
Why use RBF kernel for images, but Laplace kernel for audio?

Author response:
The RBF kernel is a common kernel for low-to-medium dimensional data, such as low resolution images like MNIST (Schölkopf, 1997).
However, the RBF kernel does not scale well to higher dimensions (Binkowski et al, 2018).
This is problematic since audio instances are represented by vectors of length 100,000 or more in our experiments (c.f. Figure 10 Appendix C.2.2).
Consequently, we use the Laplacian kernel in the main paper but also add an evaluation with the RBF kernel in Figure 11 in Appendix C.2.2.
As expected, the results are more erratic compared to the Laplacian kernel but show a similar trend.
Specifically, the kernel score and kernel entropy increase and decrease at the same training iterations, however the absolute Pearson correlation between them is only between 0.8 and 0.9.

Reviewer comment (Experiments):
Why do we see an entropy increase as training progresses for audio, but not for images? Could it be an artifact of the choice to use a different kernel for images vs audio?

Author response:
The discrepancy of the kernel entropy behavior between the image and audio experiments can be explained via the architecture and task setup. For image generation, the initial diffusion model already generates volatile and noisy samples. This leads to a roughly constant kernel entropy throughout training. In contrast, the audio model initially generates very short audio instances (c.f. Figure 10 Appendix C.2.2).
This means that most of the entries in the audio vector are zeros, which results in a small kernel entropy. Consequently, the kernel entropy increases as soon as the model learns to produce longer audio instances.

Reviewer comment (Experiments):
In the language model experiment, does the superiority of using kernel entropy for predicting correct answers depend on the choice of kernel? How are the AUC scores for Laplace kernel?

Author response:
We compare the AUROC of the kernel entropy for various choices of kernels in Figure 14 in Appendix C.2.3.
The differences between RBF and polynomial kernel as well as cosine similarity are marginal, while the Laplacian kernel performs worse.
This is in line with common practice in NLP, where cosine similarity is the most often used kernel to compare text embeddings (Camacho-Collados & Pilehvar, 2018).

Reviewer comment (Novelty):
The authors claim that one of the advantages of their uncertainty estimator is that it includes a covariance term. I would hope to see a more detailed discussion on this, for example, how that covariance term characterizes uncertainty that cannot be done by prior work.

Author response:
In previous work, the bias-variance decomposition is often used to motivate ensemble approaches. Specifically, it is assumed that the individual ensemble members are independent, which then reduces the variance term and also the generalization error. However, it is not clear how negative the effect on the generalization error is when this assumption is violated.
Our variance-covariance decomposition allows us to quantify the negative impact of correlated ensemble members via the covariance term. In Figure 3 we use the normalized covariances (=correlations) for better interpretability. As we can see, the correlations between epochs 20 to 40 are very high, which means that creating an ensemble based on iterative gradient steps, like stochastic weight averaging Gaussian (Maddox et al, 2019), strongly violates the independence assumption. Specifically, we can now quantify how much this correlation/covariance impacts the generalization error. This underlines the practical relevance of our theory.

**References:**

Maddox, W. J., Izmailov, P., Garipov, T., Vetrov, D. P., & Wilson, A. G. (2019). A simple baseline for bayesian uncertainty in deep learning. Advances in neural information processing systems, 32.

Reviewer comment (Experiments):
"This includes the discovery that mode collapse of underrepresented minority groups is expressed purely in the bias." made as a claim in the Introduction should be probably revisited. As far as I understand, the authors conclude this from their results on a single model and a single experiment, which is not enough to make a general statement.

Author response:
We agree with the reviewer that the provided evidence is not sufficient to back our strong claims about mode collapse. We now soften our claims/formulations and clarify that our bias-variance-covariance decomposition can be used as a tool to diagnose mode collapse or overfitting and elucidate its nature in terms of bias and variance. We now also provide additional experiments to showcase the practical utility and repeat the same mode-collapse experiment of Figure 5 with other digits reduced (digit ‘2’ and ‘3’, c.f. Figure 9 Appendix C.2.1). Based on our bias-variance decomposition, we discovered that mode collapse does not appear in these cases. However, the lack of training data is still expressed in an increased bias term. Consequently, we are still convinced that our decomposition is a useful tool to gain insights into the fitting behavior of generative models.

Reviewer comment (Experiments):
While the paper is largely motivated by the need for evaluating predictive uncertainty for generative models, the only assessment of the approach for this purpose is made for LLMs on QA task (Section 5.3). The authors claim that their proposed uncertainty measure is applicable to a large range of generative models, and that this is an advantage over previously proposed methods, so it would be important to test its effectiveness in assessing predictive uncertainty for other generative tasks (e.g. image generation).

Author response:
We now clarify how the experiments for image and audio are connected to the natural language generation (nlg) ones in terms of uncertainty estimation. In general, a meaningful measure of uncertainty is able to predict the loss for a given prediction.
The literature formulates nlg uncertainty estimation as a binary classification problem with a binary loss (Kadavath et al, 2022; Kuhn et al, 2023), i.e. a generation is either correct or wrong. Consequently, it makes sense to use the AUROC as a summary measure of how well an uncertainty measure predicts the correctness (i.e. is correlated to the loss). For image and audio generation we use a kernel-based loss function (kernel score), which has a continuous range of how good a prediction is. Consequently, we cannot use the AUROC anymore. Instead, we use the Pearson correlation (range between -1 and 1) to assess how well an uncertainty measure can predict the loss. Absolute values close to 1 indicate a very good uncertainty estimate.
In Figure 4 and 6, we can see that kernel entropy is very predictive of the loss (absolut Pearson correlation >0.9), making it an excellent measure of uncertainty in these cases. The nlg experiments in Figure 7 confirm that kernel entropy also gives state-of-the-art results for the binary loss function proposed by Kuhn et al (2023).

**References:**

Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022.

Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations, 2023.