Calibrating Large Language Models Using Their Generations Only (2024)

Abstract: As large language models (LLMs) are increasingly deployed in user-facing applications, building trust and maintaining safety by accurately quantifying a model’s confidence in its prediction becomes even more important. However, finding effective ways to calibrate LLMs—especially when the only interface to the models is their generated text—remains a challenge. We propose APRICOT (Auxiliary prediction of confidence targets): A method to set confidence targets and train an additional model that predicts an LLM’s confidence based on its textual input and output alone. This approach has several advantages: It is conceptually simple, does not require access to the target model beyond its output, does not interfere with the language generation, and has a multitude of potential usages, for instance by verbalizing the predicted confidence or using it to re-prompting the LLM to accurately reflecting its uncertainty. We show how our approach performs competitively in terms of calibration error for white-box and black-box LLMs on closed-book question-answering to detect incorrect LLM answers.

## introduction when given a case description of "a man superglued his face to a piano and he says it's making it hard to get a full night of sleep", a recently released medical llm was found to list unrelated potential causes in its diagnosis, including narcolepsy, sleep apnea and others. 1 this, of course, ignores the seemingly obvious reason for the patient's complaints. while humorous, this example illustrates the pitfalls of practical llm applications: despite often looking convincing on the surfaceespecially to non-experts-model responses can be wrong or unreliable, leading to potentially harmful outcomes or a loss of trust in the system, foregoing its benefits. indeed, consistent behavior (imagine 1 https://x.com/spiantado/status/ 1620459270180569090 (last accessed @xcite . what is the largest eu country by population? it's germany. e.g. reliably indicating a lack of confidence for unsure responses) has been argued as one way to build trust in automated systems @xcite , while misleading predictions have been empirically shown to lead to a loss of trust that can be hard to recover from @xcite . we introduce apricot , a method to use an auxiliary model to infer a llm's confidence in an open-ended question-answering setting. the auxiliary model does so based on the given input to and generated output text from the llm alone. the model is trained by using these two parts as input and predicting calibration targets. the latter are obtained without access to the llm's sequence likelihoods or internal states by clustering input representations produced by an additional embedding model, and thus only require black-box access. this is especially relevant since an increasing number of llm providers safeguard their model behind black-box apis. this approach is conceptually straightforward, easy to optimize, and opens up a large number of possible applications, for instance, verbalizing uncertainty-a recently popular way to communicate uncertainty for llms @xcite )-or adjusting a model's response with linguistic markers of confidence. our contributions are as follows: we propose to obtain calibration targets without requiring any additional information about llm internals or question metadata. we show that using auxiliary models on the target llm's input and output is sufficient to predict a useful notion of confidence. we also perform additional studies to identify which parts of the llm's output are most useful to predict confidence. all the code is openly available. 2 2 related work trustworthiness in ml systems. trustworthiness has been identified as a key challenge to exploit the potential applications of automated systems @xcite . this trait is seen as especially under risk in the presence of out-of-distribution examples or distributional shift @xcite . as illustrated by the introductory example, addressing these problems is paramount in high-stakes domains such as healthcare @xcite @xcite , analyzing asylum cases @xcite , or legal deliberations @xcite . @xcite argue that trust in automated systems can for instance be built extrinsically, e.g. through consistent and predictable behavior. one such a way is uncertainty quantification, i.e. by supplying a user with scores reflecting the reliability of a model prediction @xcite @xcite . however, the lab experiments of @xcite also demonstrate the flip side of this: when exposing human participants to unreliable confidence estimates, they measure a decrease in trust and participant outcomes alike. therefore, a method like ours can help to build trust in llms and pave the way for reaping their benefits. uncertainty quantification for llms. while methods for predictive uncertainty quantification have already been explored in nlp for classification @xcite @xcite , regression @xcite @xcite and language generation tasks @xcite , their application to llms has posed novel challenges. besides the different kinds of uncertainty due to the nature of language itself @xcite , llms are usually too large for bayesian methods and are more expensive to be finetuned (unless one resorts to low-rank approximations; yang et al., 2023). furthermore, the generation process is often shielded behind black-box apis for commercial models, only leaving access to the predicted probabilities, or-in the worst case-the generated text. existing approaches operate on the model's confidence and improve it through (re-)calibration @xcite @xcite , the ensembling of prompts @xcite , asking the model to rate its own uncertainty @xcite @xcite or analyzing its use of linguistic markers @xcite , computing the entropy over sets of generations with similar meaning @xcite or comparing the textual similarity of generations for the same input @xcite . the most similar work to ours comes from @xcite , who train a calibrator on the target model's hidden states to predict whether the answer is correct or incorrect. they then finetune the target model using control tokens that are based on the calibrator's prediction and indicate the desired confidence level. in comparison, our method has multiple advantages: we only assume access to the input question and text outputs, not requiring white-box model access or finetuning of the target model. we further demonstrate in our experiments that using more fine-grained targets than the simple binary yields better calibration overall. predicting properties of generated text. instead of trying to obtain a piece of information of interest-e.g. truthfulness or predictive confidence-from the original model, other works have investigated whether this can be done so through secondary (neural) models. for instance, @xcite ; @xcite fit density estimators on internal model representations. in nlp, @xcite demonstrated that wrong statements by llms can be detected by collecting yes / no answers for a set of questions, storing them in a vector and fitting a logistic regression model. in general, using secondary neural models to predict properties of the generated text also has connections to other tasks such as translation quality estimation @xcite @xcite @xcite , toxicity classification (maslej-krešňáková et al., 2020) or fine-grained reward modeling @xcite . estimating the confidence of a llm can be challenging, since their size rules out many traditional techniques that require finetuning or access to model parameters. in this light, using the likelihood of the generated sequence might seem like an appealing alternative; however, it might not actually reflect the reliability of the model's answer and often cannot be retrieved when using blackbox models, where the only output is the generated text. verbalized uncertainty, i.e. prompting the llm to express its uncertainty in words, can be a solution when the model is powerful enough. but as we later show in section 4, the generated confidence expressions are not very diverse, and results are not always consistent, meaning that the model does not always generate a desired confidence self-assessment. as we illustrate in in figure 2 we give an overview of apricot , which consists of three main steps: firstly, we prompt the target llm to generate training data for our auxiliary model (section 3.1). secondly, we set calibration targets in a way that does not require access to the target llm beyond its generated outputs (section 3.2). lastly, we train the auxiliary calibrator to predict the target llm's confidence for a given question (section 3.3). thereby, we contribute two parts that are agnostic to the llm in question: the creation of calibration targets and their prediction through the auxiliary model. note that we will use the terms auxiliary model or calibrator interchangeably in the following sections. (a) default prompting. (b) chain-of-though prompting. (c) prompting with verbalized confidence. ## experiments we now demonstrate how apricot provides a simple yet effective solution to calibrate llms. before assessing the quality of the unsupervised clustering to determine calibration targets from section 3, we first introduce the dataset and models. datasets. we employ triviaqa @xcite , a common (closed-book) questionanswering dataset. open-ended question answering is an ideal testbed for natural language generation tasks, since it is comparatively easy to check whether an answer is correct or not, so calibration has an intuitive interpretation. to preprocess triviaqa, we create a training set of 12k examples and choose another 1.5k samples as a validation and test split, respectively. 5 secondly, we run experiments on coqa @xcite , a conversational question-answering dataset in which the model is quizzed about the information in a passage of text. we treat the dataset as an open-book dataset, where the model is shown the passage and then asked one of the corresponding questions at a time. we extract a subset of the dataset to match the split sizes of triviaqa. ## discussion despite the difficulty of predicting the llm's confidence from its generated text alone, our experiments have shown that apricot can be used to produce reasonable scores even under these strict constraints. we showed in the past sections that the auxiliary model can be finetuned to learn from multiple signals. on the one hand, the auxiliary calibrator learns a mapping from a latent category of question to the expected difficulty for a target llm. on the other hand, including the answer given through cot prompting and including the llm's own assessment of its uncertainty helped to further improve results. while sometimes beaten in terms of calibration error, our method consistently outperforms our baselines in misprediction auroc, meaning that it can provide the best signal to detect wrong llm answers. compared to other approaches, this yields some desirable properties: apricot is available when sequence likelihood is not; it is more reliable than verbalized uncertainty; and it only needs a light finetuning once, adding negligible inference overhead. compared to other methods such as @xcite ; @xcite , it also does not require more generations for the same input, reducing the more expensive llm inference costs. ## conclusion we presented apricot , a general method to obtain confidence scores from any language model on the input and text output alone. we showed that it is possible to compute calibration targets through the clustering of question embeddings. through the subsequent finetuning of a smaller language model, we then outperform other methods to distinguish incorrect from correct answers with competitive calibration scores, on different models and datasets. while we only presented a first, more fundamental version of this approach in this work, it lends itself naturally to a whole body of research that aims to improve the calibration of pretrained language models @xcite @xcite . lastly, future studies might also investigate the uncertainty of the auxiliary model itself and use techniques such as conformal prediction @xcite @xcite to produce estimates of llm confidence intervals.