C on U : Conformal Uncertainty in Large Language Models with Correctness Coverage Guarantees (2024)

Abstract: Uncertainty quantification (UQ) in natural language generation (NLG) tasks remains an open challenge, exacerbated by the closed-source nature of the latest large language models (LLMs). This study investigates applying conformal prediction (CP), which can transform any heuristic uncertainty notion into rigorous prediction sets, to black-box LLMs in open-ended NLG tasks. We introduce a novel uncertainty measure based on self-consistency theory, and then develop a conformal uncertainty criterion by integrating the uncertainty condition aligned with correctness into the CP algorithm. Empirical evaluations indicate that our uncertainty measure outperforms prior state-of-the-art methods. Furthermore, we achieve strict control over the correctness coverage rate utilizing 7 popular LLMs on 4 free-form NLG datasets, spanning general-purpose and medical scenarios. Additionally, the calibrated prediction sets with small size further highlights the efficiency of our method in providing trustworthy guarantees for practical open-ended NLG applications.

## introduction despite advancements in various natural language generation (nlg) tasks @xcite @xcite , large language models (llms) are proven to hallucinate facts and confidently generate textual information that is not correct or grounded in reality @xcite . factually incorrect answers can confuse and mislead users, resulting in erroneous conclusions and ultimately undermining the trustworthiness of llmsbased high-stakes applications. uncertainty quantification (uq) provides valuable insights into the reliability of model responses, facilitating risk assessment and hallucination detection @xcite . however, it demands investigating black-box uncertainty measures with the proliferation of llms served via apis @xcite , which only allows textual inputs and outputs. conformal prediction (cp) @xcite @xcite is known for providing a model-agnostic and statistically rigorous uncertainty estimation. cp was primarily employed in classification @xcite and regression tasks @xcite . for nlg tasks, cp is first adapted to the multiple-choice question-answering (mcqa) setting, where the acceptable response is selected from a fixed set of options @xcite , limiting its applications in real-world open-ended nlg tasks. conformal language modeling @xcite relies on the model likelihoods and calibrates a stopping rule to sample prediction sets from the infinite output space until users are confident that the set covers at least one response satisfied. lofreecp @xcite studies cp for api-only llms without logit access by leveraging uncertainty information from diverse sources. our study explores adapting cp for general nlg applications. the nonconformity score (ns) in cp serves as a criterion for calibrating prediction sets, which provide coverage guarantees by selecting a set of possible labels that satisfy the ns threshold (angelopoulos and bates, 2021). since typical logits-based ns may encounter miscalibration, we aim to integrate black-box uq into the definition of ns, by closely aligning it with the uncertainty condition of the correct answers and devising a conformal uncertainty criterion, while it is more reliable to analyze the uncertainty within llms' true output space. then, we employ the uncertainty criterion, concluded from a small amount of independent and identically distributed (i.i.d.) calibration data, to construct prediction sets by selecting generations sharing a similar uncertainty condition from the unbounded output space on test samples. typically, there are two goals of cp: (1) the calibrated prediction set contains the correct answer with at least a user-specified probability; and (2) the average set size should be small, demonstrating the prediction efficiency of our method. the first challenge is uq for black-box llms. our solution is inspired by an intuitive observation: if a language model generates more semantically diverse outputs for the same prompt, the uncertainty is likely higher @xcite @xcite . regardless of the model's capability to tackle the current problem, the confidence score that the model assigns to a generation can be represented by its frequency within the output space. we approximate the model's output distribution by sampling multiple answers to the same question. then, we perform semantic clustering on the sampled generations, and propose to measure the uncertainty of each generation by combining two factors: the frequency of occurrence of the semantic meaning it conveys, and the consistency between its semantic and other semantic clusters augmented by their individual frequency. based on the measure, we define the ns as the uncertainty of the generation. to this end, the generation meets the correctness criterion and is semantically most similar to the reference answer in the calibration set. we then calculate the quantile q of nss for all calibration samples, based on the user-specified upper bound of error rate α. next, we utilize the conformal uncertainty criterion (i.e., the uncertainty threshold q) to construct a prediction set for each test sample by selecting generations that satisfy the uncertainty conditions strictly associated with correctness from the candidate generations. additionally, for black-box uq, we propose employing the most frequent generation or semantic (i.e., the model's most confident answer) as a more trustworthy reference object for the query and leveraging it to measure the overall uncertainty of the current uq process. we term this measure conu, as it employs the same approach as the conformal uncertainty criterion. extensive experimental results exhibit that conu generally outperforms prior state-of-the-art methods and verify the strict correctness coverage guarantees. specifically, the prediction sets calibrated by the conformal uncertainty criterion always encompass the correct answers under various userspecified error rates. furthermore, the average prediction set size is small, highlighting the prediction efficiency of our approach. to our knowledge, this is the first method in the literature to strictly link the ns with the uncertainty condition aligned with correctness via black-box uq, thereby developing a more robust conformal uncertainty criterion, which provides rigorous correctness coverage guarantees in practical open-ended nlg tasks, and its unique inspiration in benchmarking uq in llms through cp generates independent interest * . in summary, our major contributions are listed as follows: ## method our method investigates two key issues: (1) how to estimate the uncertainty in black-box llms when we can only access the output texts; and (2) how to provide rigorous guarantees on the error rate in open-ended nlg tasks. we first devise a black-box uncertainty measure grounded in self-consistency to provide the trustworthiness notion of model responses. furthermore, we utilize the split cp technique to convert the heuristic approximation into a statistically rigorous one, thereby ensuring a more robust and systematic assessment of uncertainty. ## conclusion in this work, we introduce conu tailored for blackbox uq in open-ended nlg tasks. relying on cp which can transform any heuristic approximation into a statistically rigorous uncertainty notion, we develop a robust conformal uncertainty criterion to provide reliable guarantees of correctness coverage under various user-specified error rates. we achieve strict control of the coverage rate across 7 practical llms on 4 free-from nlg datasets. furthermore, the small average uncertainty set size underscores the efficiency of our methods. utilizing these calibrated prediction sets, we perform selective prediction and obtain remarkable improvements in model accuracy. we envisage that our conformal uncertainty criterion can provide new strategies for principled uq in open-ended nlg tasks.