Factual Confidence of LLM s: on Reliability and Robustness of Current Estimators (2024)

Abstract: Large Language Models (LLMs) tend to be unreliable on fact-based answers.To address this problem, NLP researchers have proposed a range of techniques to estimate LLM’s confidence over facts. However, due to the lack of a systematic comparison, it is not clear how the different methods compare to one other.To fill this gap, we present a rigorous survey and empirical comparison of estimators of factual confidence.We define an experimental framework allowing for fair comparison, covering both fact-verification and QA. Our experiments across a series of LLMs indicate that trained hidden-state probes provide the most reliable confidence estimates; albeit at the expense of requiring access to weights and supervision data. We also conduct a deeper assessment of the methods, in which we measure the consistency of model behavior under meaning-preserving variations in the input. We find that the factual confidence of LLMs is often unstable across semantically equivalent inputs, suggesting there is much room for improvement for the stability of models’ parametric knowledge.

## introduction a major problem of large language models (llms) is that they do not always generate truthful information. models can hallucinate by convincingly reporting information that is actually false or they are not confident about, or provide factual answers only when prompted in a certain way @xcite @xcite @xcite . this behavior can be severely harmful, especially given the current explosion of llm usage: a lack of truthfulness can lead to spread of misinformation and breaches to user trust @xcite * work conducted during an internship at aws ai labs. 2021; @xcite . having a reliable estimate of the model's confidence over a fact-the degree to which it is expected to have accurate factual knowledge with respect to an input-is key for mitigating this problem @xcite . survey of the current state-of-the-art for estimating the factual confidence in llms, and performing a systematic empirical comparison of the reliability and robustness of the existing methods. we introduce an experimental framework, shown in figure 1 , enabling a fair comparison between methods across models and datasets. we adopt two distinct formulations for measuring factual confidence: (i) the probability of a statement to be true, noted p (true)-fact-verification @xcite , and (ii) the probability of yielding a truthful answer to a query, noted p (i know)-question answering @xcite . additionally, we categorize the existing methods into five groups: trained probes, sequence probability, verbalization, surrogate token probability, and consistency-based. our experiments across eight publicly available llms indicate that prompting-based methods are less reliable than supervised-probing, although the latter requires training data and access to models' weights. for instruction-tuned llms, verbalization and consistency-based methods are viable alternatives. further, we argue that all methods for estimating factual confidence can ultimately lead to misleading conclusions if only tested on a single way of asserting a fact: an llm may seem to know a fact given an input, but then contradict itself given an alternative phrasing of that fact @xcite @xcite @xcite . in our experiments, we find evidence of such instability, suggesting that llms do not always encode facts based on abstractions over diverse input variations. in summary, this paper provides the following contributions: ## factual confidence: survey of methods based on a review of the research area, we identify five groups of existing methods to estimate factual confidence, which we discuss in the following subsections. in ## robustness to linguistic variations in this section, we apply meaning-preserving linguistic variations to each input in order to: (i) assess the robustness of methods. the expectation is that if a method is robust, it should produce equally reliable estimates (equally high auprc scores) across different input formulations (section 6.1); ## discussion & conclusion in this paper, we compare existing methods to estimate llms factual confidence. obtaining reliable estimates can benefit llms applications, by anticipating potential hallucinations and limiting the non-factual information output by a model @xcite . however, if not reliable, such estimates can be counterproductive, as they would introduce errors and negatively affect user-model interactions. our experiments across eight llms demonstrate that the trained probe method is the most reliable estimator of llm factual confidence. it works well for both fact-verification (p (t)) and question answering (p (ik)) across all 8 tested models, indicating that its reliability is likely to generalize to other llms. moreover, we show that it generalizes to out-of-domain data: (i) when a model trained on t-rex is applied to an unseen dataset (popqa, section 5.3), and (ii) in a cross-lingual transfer setting (section 6.1). we must note, though, that the fine-tuning nature of the method clearly puts it at an advantage over the other zero-shot probing methods. moreover, applying the trained probe method comes with strong requirements: (i) access to model weights-not always provided by proprietary llms, and (ii) need of supervised data. if these requirements cannot be met, but the model is instruction-tuned @xcite we recommend estimating p (t) with verbalized confidence or surrogate probabilities. the other methods under study, especially if applied to non-instructiontuned llms, are not consistently reliable. our results highlight the need for more research to develop reliable estimators that can be applied to black-box models, with inaccessible internal representations. we expect the reliability gap between methods like verbalized confidence and trained probe to get smaller with increasingly powerful llms, especially in their ability to follow instructions. however, strong results of trained probe indicate that hidden states contain signal about factual confidence, and it is unclear whether this can ever be fully leveraged by the prompting approaches. besides the comparison among methods, we also provide insights on the stability of factual knowledge in llms @xcite @xcite . we show that the factual confidence of an llm is not always consistent under meaning-preserving variations of the input (paraphrases and translations). while the model may sometimes be sure that a fact is true or false, or that it knows the answer to a question, it may actually behave differently if we reformulate the statement or question. an interesting direction for future research is the exploration of training methods that teach an llm to better disentangle facts from the diversity of forms they can be stated in, and ultimately exhibit better and more consistent factual knowledge. this would also contribute to increasing llms resistance to adversarial attacks @xcite , mitigating the generation of misinformation due to an incorrect sensitivity to input changes.