Paper: Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models (ICLR-2024)

Reviewer comment (Writing):
The paper mentions several times that uncertainty is not the same as confidence. Confidence scores are defined based on the token probabilities, which are not available in black-box inference settings. The paper however isn't very clear in how uncertainty and confidence are not the same thing, with multiple sentences referring to "uncertainty/confidence" as though they are the same, and section 4.2. not written in a way that helps clarify the working definitions of how these differ either. This is frustrating for the reader.

Author response:
Apologize for this confusion. 
To clarify, in short, $U$ is a property of the model's perceived posterior and depends only on $x$ (see L3-5 in Section 3.1, and Eq 1).
$C$ depends on both $x$ and the answer $\mathbf{s}$ (L2-L3 of Section 3.2 and Eq 2).
We have updated Section 3.2 to further clarify this.

> Confidence scores are defined based on the token probabilities, which are not available in black-box inference settings

Just want to clarify that Eq(1) and Eq(2) are only one possible uncertainty/confidence (that are however mainstream).
Exact definition is just anything taking the form of $U(x)$ and $C(x,\mathbf{s})$.
We hope the updated section 3.2 and the example helps clarify this.

Reviewer comment (Experiments):
The main limitation is only using question answering to test the models here. There are a spectrum of NLG tasks, and I agree with the authors that caring about uncertainty is less likely for completely open generation. However there are tasks like data-to-text (ie knowledge grounded) NLG, where the outputs need to be trusted, and it remains an open issue whether these methods reported here are transferable to tasks other than QA. The QA task is also interesting in that it is extracting answers from the LLMs weights, rather than using the LLM as a general tool for e.g. the data-to-text task, where the info is given as structural inputs.

Author response:
(We assume "data-to-text" means tasks like captioning. Please let us know if this is a misunderstanding.)
Other NLP applications like data-to-text are definitely something we plan to explore next.
This paper mostly just follows existing UQ literature and uses QA as it is the easiest to perform evaluation on it. 
Evaluation in general is still quite hard.
In fact, we have seen very recent explorations on using model-predicted likelihoood as confidence scores on X-ray report generation task (but they also judge the quality of the report with ROUGE).

We believe that a lot of the ideas are transferrable and we would like note that CoQA is to some extent "data-to-text" except that the data is in textual form as well.
Our methods do not assume that the input has to be text, as we are only comparing the generations' similarities.
If the input is data, we think there are two possible solutions: 
We skip the data (image or audio) and directly use Jaccard or NLI model on the question and answers, OR;
We fine-tune a domain-specific NLI model by training a "bridging" module that "translates" the other modalities to embeddings that the NLI model understands.

Reviewer comment (Evaluation):
How is the issue of model calibration different?

Author response:
(See end of Section 2 for a discussion, and a newly added Appendix C.5 for calibration experiments.)
The UQ we focus on tries to find a measure of uncertainty or confidence that is *discriminative* of what's uncertain and what's not. 
In other words, we care about whether the *ranking* of our UQ measure could effectively predict the reliability of the answers.
This is also why most literature uses AUROC as an evaluation metric.
Calibration cares about the discrepancy between the predicted probability and the frequency.
(Such "discriminative-ness" is often referred to as "sharpness" and is considered orthogonal to calibration [3].)
In the extreme case - if we know the model's accuracy is 0.6 on average, and we keep predicting a answer's confidence is 0.6 regardless of the answer, we get something that is prefectly calibrated, yet this constant confidence is not useful because it doesn't tell us which answer is more reliable than others (i.e. not "sharp").

[3] Gneiting, T., Balabdaoui, F., and Raftery, A. E. Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 69 (2):243–268, 2007.

Reviewer comment (Experiments):
Do you think these results transfer to other tasks such as data-to-text? Or do you think there may be different results beyond QA tasks?

Author response:
(We assume "data-to-text" means tasks like captioning. Please let us know if this is a misunderstanding.)
Other NLP applications like data-to-text are definitely something we plan to explore next.
This paper mostly just follows existing UQ literature and uses QA as it is the easiest to perform evaluation on it. 
Evaluation in general is still quite hard.
In fact, we have seen very recent explorations on using model-predicted likelihoood as confidence scores on X-ray report generation task (but they also judge the quality of the report with ROUGE).

We believe that a lot of the ideas are transferrable and we would like note that CoQA is to some extent "data-to-text" except that the data is in textual form as well.
Our methods do not assume that the input has to be text, as we are only comparing the generations' similarities.
If the input is data, we think there are two possible solutions: 
We skip the data (image or audio) and directly use Jaccard or NLI model on the question and answers, OR;
We fine-tune a domain-specific NLI model by training a "bridging" module that "translates" the other modalities to embeddings that the NLI model understands.

Reviewer comment (Evaluation):
It might be helpful to measure expected or adaptive calibration error to show this method outputs confidence scores which are better in comparison to other methods.

Author response:
Thank you for the suggestion.
We tried to distinguish the focus of our paper at the end of Section 2.
Just like in our baseline papers, we focus on the uncertainty estimation, and we care more about the *ranking* of such measures - that is, whether a high confidence means highly reliable prediction or accuracy.
All uncertainty or confidence estimates could be conformalized or calibrated to bear concrete meaning in the frequentist sense, by using a calibration dataset.
There is a large collection of calibration literature that we can leverage here. 
In repsonse to your suggestion, we have also run additional experiments using histogram binning to calibrate all methods, and report the ACE in the Appendidx C.3.

Reviewer comment (Experiments):
Relevant work: https://arxiv.org/pdf/2306.13063.pdf, might be helpful to include comparison between their method and your proposed method.

Author response:
Thank you for the suggestion and we have included this reference to our revision for completeness.
After reading the paper, we found that this method is more suitable for questions with exact answers (e.g. multiple choice or arithmetics) as the consistency-based component requires comparing exact matches of predictions, which is quite hard to achieve for examples like Figure 6 in the Appendix.
The "verbalized confidence" is similar to the P(true) baseline we tried.

Reviewer comment (Evaluation):
As pointed out in the weakness, for confidence scores measuring ECE or ACE might be helpful in showcasing the effectiveness of the approach.

Author response:
Thank you for the suggestion.
We tried to distinguish the focus of our paper at the end of Section 2.
Just like in our baseline papers, we focus on the uncertainty estimation, and we care more about the *ranking* of such measures - that is, whether a high confidence means highly reliable prediction or accuracy.
All uncertainty or confidence estimates could be conformalized or calibrated to bear concrete meaning in the frequentist sense, by using a calibration dataset.
There is a large collection of calibration literature that we can leverage here. 
In repsonse to your suggestion, we have also run additional experiments using histogram binning to calibrate all methods, and report the ACE in the Appendidx C.3.

Reviewer comment (Experiments):
In cases where we have access to model embeddings, I think same method of clustering could be applied where we use similarity between embeddings instead of entailment prob/Jaccard similarity. Would this method give better confidence or uncertainty estimates in those settings as well?

Author response:
Thank you for the question.
While we did not perform the experiments ourselves (and is not a black-box method), we noticed that our baseline paper's (Kuhn et al. 2023) official implementation tried this idea.
As it was not used nor reported in their paper, it is probably not performing well.

Reviewer comment (Writing):
There is a discussion regarding uncertainty vs. model confidence and why it matters, but the differentiation is not clear—Section 3.2 does not even differentiate the two in detail, where the expectation is that the discussion should be there because of the subsection title.

Author response:
We apologize for the confusion here.
The structure was that 3.1 defines uncertainty and 3.2 defines confidence (which is different).
$U$ is a property of the model's perceived posterior and depends only on $x$ (see L3-5 in Section 3.1, and Eq 1).
$C$ depends on both $x$ and the answer $\mathbf{s}$ (L2-L3 of Section 3.2 and Eq 2).
These are commonly used terms in two somewhat disconnected domains and we have made updates to Section 3.2 to emphasize this distinction.

Reviewer comment (Evaluation):
There are many variations/ablations regarding the methods used, i.e., similarity metrics and also uncertainty/confidence measurement, but there is little analysis on which perform best, why, and what is the recommendation for future work if they want to quantify uncertainty/model confidence.

Author response:
Thank you for the constructive question/suggestion.
In section 5.3, under *Uncertainty Measures*, our conclusion was that the similarity measure (Section 4.1) seems more important than variants proposed in Section 4.2.
We hypothesize that "entail" works better because generally the output space is huge, and for two generations, "not contradicting" is likely not a sufficient indicator of low-uncertainty (as they can both be bad random responses).
We have updated section 5.3 and Appendix B.7 with examples to reflect this hypothesis.

Reviewer comment (Evaluation):
There is concern regarding evaluation with a GPT model. Although the authors perform human verification on a subset, the subset is pretty small (33 samples per dataset). Many of these datasets are factual, thus human evaluation might not be as tricky as creative writing tasks, so having more human verification on the results would make the claim stronger. There is also no explanation about inter-annotator agreement on the verification.

Author response:
We would like to clarify that the number of evaluations is *99* per dataset instead of 33 (33 per (model, dataset) pair  * 3 models)
The number of total human evaluations is chosen basing on (Kuhn et al. 2023), which evaluated 200 answers for two datasets (we have 297 on three).
We tried our best to verify the reliability of GPT evaluation, but the verification process is actually quite slow, as for CoQA we need to read the passage first before judging the answers, and even for TriviaQA or NQ, sometimes seemingly unrelated answers are just two names of the same person and require careful research.
We agree that a larger-scale human evaluation with mulitple annotators is more desirable, but the main focus of this paper is UQ, and we think the topic of automatic evaluation is an important research area by itself.
Existing UQ literature almost always use purely lexical measures like ROUGE-based automatic evaluation, and we believe the GPT evaluation is better (please refer to Appendix B.3 for a discussion).


To address your concern, we increased the number of annotations to 200 per dataset (50 per (model, dataset) * 4 models) and the new accuracies are 89.4/96.5/93 for coqa/trivia/nq.
Using the same annotations, the ROUGE-L based metric's accuracy is 81.5/93.5/85. 
If we construct a confidence interval we could reject that "GPT and ROUGE are equally good" (our estimate is that further shrinking the confidence interval by half requires about 20 hours of annotation time).
We found that llama2 (newly added) tends to generate different expressions of the same answer, which is sometimes judged incorrect by GPT.
Excluding llama2, the GPT accuracies are 90.6/98/94 for coqa/trivia/nq.

Reviewer comment (Theory):
What is the reasoning for \(a_{NLI, contra}\)? If you use a standard NLI model which has three classes, Eq. 4 (right) would take into account the entailment and neutral class (since it is \(1 - p_{contra}\)), is that correct?

Author response:
> What is the reasoning for a_{NLI, contra}? If you use a standard NLI model which has three classes, Eq 4 (right) would take into account the entailment and neutral class (since it is 1-p_{contra}, is that correct?

$a_{NLI, contra}$ accounts for both neutral and entailment. 
We could consider the difference between $a_{NLI, contra}$ and $a_{NLI, entail}$ as whether we think "neutral" is closer to “similar” or “dissimilar”.

Reviewer comment (Evaluation):
It is unclear why uncertainty is used for expected accuracy and model confidence for individual accuracy.

Author response:
> Unclear why uncertainty is used for expected accuracy and model confidence for individual accuracy.

The setting of uncertainty + individual accuracy was in Table 11 (now 10), and we discuss in Appendix C.2 why we choose not to include it in the main text. 
Essentially, since $U$ does not depend on a particular generation (i.e. it is a property of the model's posterior, for a random generation), we think it should be used to predict the (estimated) expected accuracy (which also depends on the posterior/a random generation).

Reviewer comment (Evaluation):
Provide examples on when the uncertainty and confidence estimation align/do not align with the prediction.

Author response:
> Provide examples on when the uncertainty and confidence estimation aligns/doesn't align with the prediction

We've added more examples (e.g. Fig 5-7) in the Appendix in the updated pdf, but we are not entirely sure with what you mean by aligning with the prediction.
Could you check if the new examples address your concerns, or clarify your suggestion? Thank you!

Reviewer comment (Evaluation):
Provide more in-depth analysis on why a particular method (similarity metric/quantification method) outperforms the other methods used in the experiment.

Author response:
Thank you for the constructive question/suggestion.
In section 5.3, under *Uncertainty Measures*, our conclusion was that the similarity measure (Section 4.1) seems more important than variants proposed in Section 4.2.
We hypothesize that "entail" works better because generally the output space is huge, and for two generations, "not contradicting" is likely not a sufficient indicator of low-uncertainty (as they can both be bad random responses).
We have updated section 5.3 and Appendix B.7 with examples to reflect this hypothesis.

Reviewer comment (Evaluation):
Some more analysis should be conducted, e.g., the impact of similarity functions and intuitions of why one similarity function is more preferred than the other.

Author response:
Thank you for this constructive suggestion.
We had some discussion in Appendix B.7.
To clarify this further, we have updated section 5.3 and Appendix B.7 with some examples to reflect our hypothesis. Newly added text is reflected in blue.

Reviewer comment (Evaluation):
Some quantitative examples would be great, such as what kind of questions are evaluated as uncertain as the proposed metric vs the baselines, etc.

Author response:
Thank you for the suggestion.
The only two black-box baselines are NumSet nad LexiSim.
NumSet is discrete, so our method (EigV in particular) has an advantage over it as it's more granular.
LexiSim is purely lexical based, and we have some discussion of why meaning-based measures are more useful in Appendix right before B.4.
We have also updated the appendix B.7 to include discussion and example of why the proposed measures are better than the baselines.

Reviewer comment (Theory):
Is there any reason for the definition of the degree matrix?

Author response:
The degree matrix (Eq.7) was used because it seems like quite a straightforward measure of ``dispersion'', as $C_{deg}$ is just the average distance to other nodes.

Reviewer comment (Experiments):
Have you tried other similarity measures, such as cosine similarity, using existing word embeddings?

Author response:
This is an interesting idea and seems related to the Jaccard similarity.
We just finished running some experiments using the largest Glove embedding (glove.840B.300d) where an answer's embedding is computed as the average embedding of its words, and it performs significantly worse than Jaccard.
While we haven't fully explored this idea due to limited time, we did notice that the baseline paper (Kuhn et al. 2023) seems to have tried the sentence embeddings in their code release but didn't use it in the paper, and we assumed that sentence embedding-based method didn't work well.