API Is Enough: Conformal Prediction for Large Language Models Without Logit-Access (2024)

Abstract: This study aims to address the pervasive challenge of quantifying uncertainty in large language models (LLMs) with black-box API access. Conformal Prediction (CP), known for its model-agnostic and distribution-free features, is a desired approach for various LLMs and data distributions. However, existing CP methods for LLMs typically assume access to the logits, which are unavailable for some API-only LLMs. In addition, logits are known to be miscalibrated, potentially leading to degraded CP performance. To tackle these challenges, we introduce a novel CP method that (1) is tailored for API-only LLMs without logit-access; (2) minimizes the size of prediction sets; and (3) ensures a statistical guarantee of the user-defined coverage. The core idea of this approach is to formulate nonconformity measures using both coarse-grained (i.e., sample frequency) and fine-grained uncertainty notions (e.g., semantic similarity). Experimental results on both close-ended and open-ended Question Answering tasks show our approach can mostly outperform the logit-based CP baselines.

## introduction large language models (llms) have made significant advancements @xcite @xcite , highlighting the research potential of natural language generation @xcite . however, they often generate information that is not accurate, factual, or grounded in reality, referred to as "hallucination" @xcite . therefore, it is crucial to quantify llm uncertainty to ensure responsible responses. however, uncertainty quantification (uq) for llms is challenging due to the complex data distributions and inner model mechanism, as well as the often limited access to logit information. a potential solution is to use conformal prediction (cp) @xcite bates, * *corresponding authors. 2021; @xcite , which is known for being model-agnostic and distributionfree, and with rigorous coverage guarantees. given a user-defined error rate α, cp provides a guaranteed coverage rate for prediction sets/intervals. it measures the uncertainty from a model prediction using nonconformity score functions, e.g., 1f (x) y @xcite , where f (x) y is the softmax score for the true label y . most of the existing cps for llms rely on the access to model logits to measure nonconformity scores. for instance, @xcite define nonconformity scores as softmax scores for logits of different options in the multi-choice question answering (mcq) task and @xcite apply the conformal risk control framework @xcite , an extension of cp, to llms by utilizing model-based log probability. however, for some api-only llms like bard @xcite , logit-access is almost impossible for end users. even though the logits are available (e.g., gpt 4v (openai, 2023)), they are known to be miscalibrated and can lead to degraded performance of cp w.r.t. estimating the prediction sets or intervals @xcite , e.g., a large set size (i.e., low efficiency). to enable cp without logit-access, a straightforward way is to calculate the frequency of each response via sampling and approximate model-based probabilities. however, we theoretically prove that this approach is extremely computationally expensive (lemma 3.1). as nonconformity scores typically measure the level of uncertainty, cp depends on the ranking of the nonconformity measures rather than their actual values @xcite . therefore, we propose to sample responses for a certain number of times (e.g., 30) for each input and then utilize the frequency of each response as a coarse-grained uncertainty notion. this approach reduces the overall sampling costs and eliminates the dependence on the logits. ## preliminaries of conformal prediction conformal prediction (cp) @xcite is a model-agnostic method offering distribution-free uncertainty quantification, which produces prediction sets/intervals containing ground-truth labels with a desired error rate α. one of the widely used cp methods is split cp. formally, let (x, y ) be a sample, where x represents features and y represents the outcome. we denote the calibration set as (x i , y i ) @xmath0,...,n and the test set as (x test , y test ). cp presents the following nesting property: theorem 2.1 (conformal coverage guarantee). suppose (x i , y i ) @xmath1,...,n and (x test , y test ) are independent and identically distributed (i.i.d.). c 1-α (x test ) is a set-valued mapping satisfying the nesting property in eq. 1. then the following holds: ## methodology our method considers two pivotal challenges arising from the llms without logit-access: how to approximate the logit information of llms; and how to further improve cp efficiency, i.e., small prediction sets. we propose the logit-free conformal prediction for llms (lofreecp), where its nonconformity measures consist of three notions: frequency, representing coarse-grained uncertainty; ne, representing prompt-wise fine-grained uncertainty; and ss, representing response-wise finegrained uncertainty. ## conclusion we study the critical problem of cp for api-only llms without logit-access. we propose a novel solution to define the nonconformity score function by leveraging uncertainty information from diverse sources. in particular, under a limited sampling budget, we first use the response frequency as the coarse-grained proxy of uncertainty levels. we then propose two fine-grained uncertainty notions (ne and ss) to further distinguish uncertainty at a nuanced level. our proposed approach does not rely on model logits and can alleviate the known miscalibration issue when using logits. experiments demonstrate the superior performance of our approach compared to logit-based and logit-free baselines. our work opens up a new avenue to uncertainty estimation in llms without logit-access.