Paper: Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs (ICLR-2024)

Reviewer comment (Novelty):
The high level observations enumerated in the introduction are interesting but also seem very preliminary, and difficult to operationalize or build upon.

Author response:
Thanks for the helpful feedback. To guide practitioners on how to utilize our findings, we have revised the paper to include a dedicated discussion in Section 6.

**TL;DR: we highlight simple yet effective strategies and recommend the practitioners use “Top-K prompt + Self-Random sampling + Avg-Conf/Pair-Rank aggregation” as the best method so far due to their stable performance.**

The recommendation is made by balancing efficiency, simplicity, and effectiveness through multiple benchmark datasets: 
- Top-K outperforms all other methods on GPT-3.5 and is comparable to the top-performing method Self-Probing on GPT4. Compared to Self-Probing which requires two inference phases, the Top-K prompt is chosen for the balance between effectiveness and efficiency.
- As shown in Sec 5.3, ensemble methods (e.g., $M=5$) are consistently more effective than verbalized confidence ($M=1$) in eliciting a model's confidence. Regarding the sampling strategies, Self-Random is selected for being more straightforward and commonly used, since the performance difference of different sampling strategies is minimal.
- For aggregation, strategies based on both answers and verbalized confidences (e.g., Avg-Conf and Pair-Rank) outperform *aggregation based on answers only (e.g., consistency)*. Then we recommend Pair-Rank and Avg-Conf for different downstream tasks according to their relatively good performance on different metrics. For example, for tasks that prioritize the exact confidence values, like calculating expected risk, Pair-Rank is recommended, while Avg-Conf is better suited for tasks related to failure prediction, e.g., factual error detection.

Reviewer comment (Theory):
Is there a reason to believe that the model's certainty _should_ be correlated with model correctness? Or to understand a priori (e.g., based on training data) when it might be more or less likely to be so?

Author response:
This is indeed an excellent question! In fact, the correlation depends on the definition of uncertainty, e.g., data/model uncertainty[1]. Here to answer the reviewer’s question, we clarify the following subquestions:

1. What is our definition of model certainty? 
2. Does this defined model certainty reflect model correctness?

In line with established research [1,2,3,4,6], we define the 'ground-truth' certainty of a model's prediction $\mathbf{x}\_t$ as the corresponding probability from the data distribution $P_{data}(\mathbf{x}\_t|\mathbf{x}\_{1:t-1})$. This probability is highly correlated with model correctness, that is, a lower probability implies that the prediction is less likely to be correct. Therefore, we believe that **the model’s certainty should be correlated with model correctness**, which is also practically useful, e.g., help humans identify potential factual errors of LLMs.  

3. Can our proposed uncertainty scores capture the model correctness?

We suspect the reviewer's question might also arise from doubts about whether our proposed consistency or verbalized confidence can capture model correctness, e.g., the models can be confident but wrong. **We would like to clarify that our methods are designed with the potential to capture correctness**, although issues such as computational constraints can affect their efficacy:

- For consistency, sampling more queries theoretically allows us to approximate the model's internal distribution, thus capturing correctness. The issues making this method ineffective can be the model's poor calibration [4] or the computational constraints. For example, the limited number of sampled queries might lead to inaccurate estimates.

- For verbalized confidence, we note that humans are able to verbalize their uncertainty, e.g., giving insight as to whether our answers and reasonings are correct or not. So it is reasonable to expect LLMs to have also learned this ability, or to learn it at some point in the future. LLM’s current inaccuracy at this task points to an important research gap and might be explained by the training data being affected by the observed overconfident tendency of human uncertainty expression[5]. 

----

- [1] Malinin, Andrey, and Mark Gales. "Predictive uncertainty estimation via prior networks." NeurIPS (2018).
- [2] Guo, Chuan, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. "On calibration of modern neural networks." ICML 2017.
- [3] Liu, Jeremiah, et al. "Simple and principled uncertainty estimation with deterministic deep learning via distance awareness." NeurIPS (2020)
- [4] Kuhn, Lorenz, Yarin Gal, and Sebastian Farquhar. "Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation." ICLR (2023).
- [5] Garthwaite, Paul H., Joseph B. Kadane, and Anthony O'Hagan. "Statistical methods for eliciting probability distributions." Journal of the American statistical Association (2005)
- [6] Jiang, Zhengbao, et al. "How can we know when language models know? on the calibration of language models for question answering." ACL (2021).

Reviewer comment (Experiments):
Minor variations in prompt wording can have significant effects on performance. How was the specific prompt text chosen for each of the prompt strategies? Was it optimized or experimented with?

Author response:
Thanks for the question. Indeed, the prompts (see Appendix Sec D for all the used prompts) are selected through iterative refinement to avoid potential ambiguity and bias, and we are happy to share some insights from this refinement process. 

Initially, we started with a straightforward prompt like "Provide your confidence in this answer." However, we quickly discovered that the model's responses varied widely, ranging from textual descriptors like "confidence: high," to statements such as "I am very confident," and even disclaimers like "as AI language models, I cannot provide confidence."
 
In addition, we observed that providing examples in the prompts can insert bias into the distribution of outputs. For instance, saying "ONLY the option letter, e.g., C" would inadvertently increase the frequency of 'C' as a selected answer in weaker models like Vicuna. 

Consequently, we refined our prompts further by removing such examples to reduce bias. These improvements were made step by step, informed by careful observation and analysis of the output results. While we cannot guarantee that the current prompts are the optimal versions, they have been designed to minimize the interference of irrelevant factors as much as possible.