P areto Optimal Learning for Estimating Large Language Model Errors (2024)

Abstract: Large Language Models (LLMs) have shown impressive abilities in many applications. When a concrete and precise answer is desired, it is important to have a quantitative estimation of the potential error rate. However, this can be challenging due to the text-in-text-out nature of the generative models. We present a method based on Pareto optimization that generates a risk score to estimate the probability of error in an LLM response by integrating multiple sources of information. We prove theoretically that the error estimator optimized in our framework aligns with the LLM and the information sources in an Pareto optimal manner. Experimental results show that the risk scores estimated by our method are well correlated with the true LLM error rate, thus facilitating error correction. By dynamically combining with prompting strategies such as self-verification and information retrieval, we demonstrate the proposed method can be utilized to increase the performance of an LLM, surpassing state-of-the-art task specific model.

## introduction large language models (llms) have evolved to become impressively powerful in recent developments @xcite , with generative pretrained transformer (gpt) models showing increasingly effective capabilities. the evolution from gpt-3 @xcite to @xcite , along with the emergence of other llms such as palm @xcite and llama @xcite , has marked a significant leap in natural language understanding and problem-solving abilities. the generative nature of these models has led to their widespread adoption in numerous application fields. despite their advanced capabilities, llms are capable of generating incorrect results @xcite , an issue that is particularly problematic in applications where precision and dependability are critical, like the biomedical and healthcare fields @xcite . existing approaches for improving the correctness of llms include prompt engineering @xcite , retrieval methods @xcite , domain-specific tuning @xcite among many others @xcite . while existing methods show varying degrees of improvement on different tasks, in general there lacks a systematic way to efficiently quantify the likelihood of errors in llm outputs. one strategy involves querying the llm in various ways to estimate the answer's correctness (e.g., @xcite . however, these approaches are computationally expensive and biased by the llm itself. in many llm applications, including disease prediction @xcite , medical diagnosis @xcite , and question answering (qa) @xcite , a concrete and precise answer is desired. for these mission critical tasks, a quantitative error estimation or confidence level for the response is equally important as giving a correct answer itself. however, the text-in text-out nature of the generative language models makes it challenging to estimate the error probability of the answer quantitatively. although some llms provide internal probability scores for the generated tokens, they are poorly calibrated to the true error rate, particularly after applying reinforcement learning with human feedback (rlhf) @xcite . our goal in this paper is to address this issue by establishing a systematic way to quantitatively estimate the likelihood of error in llm answer. we approach this through training an estimator model h via multi-objective optimization, leveraging extensive research in pareto optimization @xcite . given the optimized model h and any llm response, we can then directly estimate the llm response error rate which we refer to as the pareto optimal learning assessed risk (polar) score (sec-tion 3.2). this framework leverages structured information retrieval from other information sources such as knowledge bases. we introduce a novel approach that trains a pareto-optimal probabilistic model h to simultaneously optimize on llm and align with the external information sources. our key contributions are as follows: ## related work several heuristics have been proposed to reduce error and estimate the confidence level of an llm response. @xcite used self-consistency to infer the reliability of the answer. @xcite proposed selfcheckgpt as a blackbox method to detect hallucination. the chain-ofthought method @xcite has also been used to indicate potential errors. these approaches are not able to provide a quantitative error estimation that is calibrated to the true error rate, and susceptible to the issue that the model's selfassessment of confidence is inherently biased. the quality of the results is also highly dependent on the prompting strategy. estimation of uncertainty in llm outputs has been studied in @xcite ; @xcite ; @xcite , however, uncertainty does dot directly reflect the confidence score estimating error probability. producing confidence scores that are well correlated with a model's error rate has been well established in traditional machine learning. the study of model calibration dates back to the seminal work of platt scaling @xcite , where a logistic calibration model is fitted on top of the original model output. various techniques have been developed afterwards for model calibration, including isotonic regression @xcite , temperature scaling @xcite , and bayesian binning @xcite . for llm, a contextual calibration method for llms was proposed by @xcite , which adjusts the class balance by taking an ensemble of llm queries with content-free input. these methods rely on annotated calibration data and access to model's output probabilities that are not always available. the problem of aggregating multiple sources of information or supervision sources is studied extensively in programmatic weak supervision @xcite @xcite @xcite . notable works include distant supervision @xcite , data programming @xcite and metal, also known as snorkel, @xcite . most of these works weigh the supervision sources across all examples and combine the multiple sources into a single label per example. this approach have shown success but also exhibits significant limitations when applied to identifying llm errors, primarily due to the weighting dilemma, where if the weight assigned to the llm is too low, the aggregated result can be noisy, and if the llm weight is too high, the output is dominated by the llm, making detecting llm error difficult. @xcite mitigates the weighting problem with instance-dependent weighting, but the expectation maximization procedure demonstrates significant learning variance. in this work we present a framework to systematically estimate error of an llm output by simultaneously aligning to multiple information sources while circumventing the weighting dilemma through pareto optimization. ## methodology our error estimation framework for llm responses is a two-step process. in the first step, we iterate through a corpus of input instances to collect the corresponding llm responses, while dynamically retrieving heuristic answers from other information sources. in this process, a probabilistic function h is learned that fits the multiple sources in a pareto optimal manner. in the second step, the optimized h model is used to estimate the error rate of the llm response on any new input instance, which we refer to as the polar score. after the error estimation step, we also provide an optional third step that strategically re-prompt the llm based on the polar score, and leverage the information retrieved from the information sources in an rag manner @xcite . an overview of the framework is shown in figure 1 . ## ablations external sources of information to illustrate the importance of these additional sources, we examine the performance without them by removing all λ j for @xmath0, • • • , m, and only take λ 0 := λ as the llm itself in the framework. note that in that sense, solving equation (3) becomes a singleobjective learning problem, which essentially is equivalent to distilling a small model h from λ. we show the error estimation comparison for gpt-4 in ## conclusion we presented a novel framework for llm error estimation using pareto optimal learning. the error estimator learned in our framework aligns with the llm and other information sources pareto optimally. we showed experimentally that the proposed polar score is well calibrated with the llm error rate evaluated on ground truth, ensuring reliable error estimation. we proposed two polar-assisted dynamic prompting strategies, and showed that polar-assisted rag enhances gpt-4's performance, surpassing state-of-the-art task-specific model. this development marks a substantial advancement in the application of llm, providing an effective method to both estimate and reduce llm errors.