When to Trust LLM s: Aligning Confidence with Response Quality (2024)

Abstract: Despite the success of large language models (LLMs) in natural language generation, much evidence shows that LLMs may produce incorrect or nonsensical text. This limitation highlights the importance of discerning when to trust LLMs, especially in safety-critical domains. Existing methods often express reliability by confidence level, however, their effectiveness is limited by the lack of objective guidance. To address this, we propose CONfidence-Quality-ORDer-preserving alignment approach (CONQORD), which leverages reinforcement learning guided by a tailored dual-component reward function. This function integrates quality reward and order-preserving alignment reward functions. Specifically, the order-preserving reward incentivizes the model to verbalize greater confidence for responses of higher quality to align the order of confidence and quality. Experiments demonstrate that CONQORD significantly improves the alignment performance between confidence and response accuracy, without causing over-cautious. Furthermore, the aligned confidence provided by CONQORD informs when to trust LLMs, and acts as a determinant for initiating the retrieval process of external knowledge. Aligning confidence with response quality ensures more transparent and reliable responses, providing better trustworthiness.

## introduction large language models (llms) have excelled in natural language understanding and generation @xcite . however, mounting evidence indicates that llms generate incorrect or nonsensical text, including fabricated citations or incorrect medical information, risking errors in critical applications @xcite @xcite @xcite . the urgent question is: * corresponding author when can we trust llms? addressing this concern is essential to prevent the uncritical acceptance of misleading information and to guide decisions on when to rely on llms versus when to seek external knowledge. recently, researchers have explored prompting llms to output calibrated confidence alongside text @xcite for determining llms' reliability. the essence of confidence calibration is to ensure that the confidence expressed corresponds with the correctness of the response, which is critical for the model's transparency and trustworthiness. while classification tasks benefit from ground truth labels to calibrate predicted probabil-ities against actual correctness @xcite , generative tasks confront the challenge of calibration without clear ground truths. current calibration methods for text generation rely on heuristics that consistency among multiple responses @xcite or the top-k responses facilitate calibration @xcite . however, these assumptions often bear little relevance to the intrinsic quality of the responses. therefore, these strategies frequently result in a misalignment between the expressed confidence and the actual quality of the response, shown in figure 1 . the critical issue is the absence of a gold standard for confidence that directly reflects response quality, leaving models incapable of guiding confidence levels aligned with response quality. in this study, we explore a reinforcement learning (rl) framework to tackle this challenge, designing reward functions that align confidence levels with response quality. this framework takes advantage of the adaptability afforded by the diverse reward functions in rl to bridge the gap between confidence and response quality @xcite . a direct reward approach involves rewarding language models for well-aligned response-confidence pairs while penalizing misaligned ones, where the reward model is fine-tuned on the constructed training data. however, this method can inadvertently encourage language models to take shortcuts. specifically, such a strategy may inadvertently encourage the generation of lower-quality responses paired with correspondingly reduced confidence (empirically demonstrated in section 3). to address this issue, it is imperative to devise a reward function that encourages the generation of accurate and well-aligned responses. we propose confidence-quality-order-preserving alignment approach, called conqord (sounds as concord), designing a dual-component reward strategy focusing on response quality and confidence alignment. the reward model comprises:: (i) a quality reward model that rates the response quality. (ii) an order-preserving alignment reward model encourages an ordinal relationship consistency between confidence and quality rating, while penalizing ordinal discrepancies. this order-preserving reward fosters careful self-calibration and adaptability to various contexts. the order-preserving nature of the reward minimizes the impact of outliers. by integrating the quality reward model with the alignment reward function, we apply the proximal pol-icy optimization (ppo) algorithm to harmonize verbalized confidence with response quality, thus preventing the model from becoming cautious. we conduct experiments using four foundational models, including llama-2 7b, zephyr 7b, mistral 7b, and llama-2 13b, across two datasets: nq and truthfulqa. experimental results demonstrate that our conqord substantially improves the alignment performance between confidence levels and the quality of responses without inducing excessive caution. moreover, we evaluate the practicality of conqord's calibrated confidence in adaptive retrieval task @xcite , where confidence scores are used to guide the activation of external knowledge. our experiments confirm that conqord's confidence alignment reliably dictates the trustworthiness of llm outputs. conqord contributes to making the model-generated responses not only more transparent but also more reliable, through its refined confidence calibration. ## related works confidence, or uncertainty, refers to the degree of certainty or assurance that accompanies a prediction or decision made by a model @xcite . the calibration of confidence is essential for the reliability of machine learning systems, as it ensures that predicted probabilities match the true likelihood of outcomes as closely as possible @xcite . while in traditional classification tasks, this involves aligning predicted probabilities with actual ground truth labels, the task becomes more challenging for generative models due to the inherently ambiguous nature of the ground truth @xcite . confidence elicitation in llms confidence elicitation in llms aims to gauge the certainty of responses without modifying the model or accessing its internals @xcite . mielke et al. @xcite proposed an external calibrator, which requires access to the model's internals, often impractical. lin et al. @xcite introduced verbalized confidence, prompting models to declare their certainty. yet, they focused on fine-tuned models and did not explore zero-shot scenarios. @xcite examined confidence in prompt design but did not provide explicit confidence measures to users. confidence calibration in llms calibration in llms is an emerging new research direction. @xcite investigate verbalized approaches to calibrate the confidence of llms. they direct llms to produce the top k predictions for a query, where each prediction is paired with a distinct probability value that reflects the model's confidence in the accuracy of that prediction. xiong et al. @xcite proposed a hybrid method by combining verbalized numbers and consistency-based scores for benchmarking. limitations and challanges prior methods depend on heuristic presumptions that assume consistency across multiple samples or posit that recalling top-k samples aids in confidence calibration. nonetheless, these methods suffer from a lack of appropriate direction. this deficiency stems from the lack of a definitive ground truth standard for confidence that aligns precisely with the quality of the answers, presenting significant obstacles to the accurate alignment of confidence estimates. ## confidence alignment via rl we adopt a reinforcement learning (rl) framework to tackle the challenge of lacking a groundtruth standard for confidence assessment. different from supervised fine-tuning (sft), which depends on labeled data, rl offers a more adaptable solution by allowing any indicator as a reward. we follow previous studies @xcite , regard text language generation as a markov decision process (mdp) while the remaining elements of mdp are listed in appendix a. in the rest of this section, we elaborate on the reward strategy. ## confidence-quality-orderpreserving alignment approach in this section, we propose confidence-quality-order-preserving alignment approach via reinforcement learning, namely conqord. ## experiments in this section, we evaluate the alignment performance of our proposed conqord on benchmark datasets. further hyperparameter analysis and case studies are also provided. ## conclusion and future work in this paper, we propose a confidence-qualityorder-preserving alignment approach (con-qord), which marks a significant step forward in the domain of confidence alignment for llms. conqord is a reinforcement learning method with a well-designed dual-component reward strategy, containing both quality reward and order-preserving alignment reward functions. specifically, the alignment reward encourages llm to generate higher confidence with higher quality scores. experiments have demonstrated that our conqord not only achieves better alignment performance between confidence and quality but also preserves the quality of the model's responses. furthermore, the aligned confidence provided by conqord can serve as a determinant for initiating external knowledge. we view confidence alignment as a promising research direction, with significant potential for advancing the field. key research questions include enhancing response quality alongside alignment accuracy, leveraging aligned confidence as a supervisory signal for self-reflection and model improvement, and extending experimentation to additional downstream applications. we are enthusiastic about exploring directions further and plan to conduct additional investigations in the future.