Enhancing Healthcare LLM Trust with Atypical Presentations Recalibration (2024)

Abstract: Black-box large language models (LLMs) are increasingly deployed in various environments, making it essential for these models to effectively convey their confidence and uncertainty, especially in high-stakes settings. However, these models often exhibit overconfidence, leading to potential risks and misjudgments. Existing techniques for eliciting and calibrating LLM confidence have primarily focused on general reasoning datasets, yielding only modest improvements. Accurate calibration is crucial for informed decision-making and preventing adverse outcomes but remains challenging due to the complexity and variability of tasks these models perform. In this work, we investigate the miscalibration behavior of black-box LLMs within the healthcare setting. We propose a novel method, Atypical Presentations Recalibration, which leverages atypical presentations to adjust the model’s confidence estimates. Our approach significantly improves calibration, reducing calibration errors by approximately 60% on three medical question answering datasets and outperforming existing methods such as vanilla verbalized confidence, CoT verbalized confidence and others. Additionally, we provide an in-depth analysis of the role of atypicality within the recalibration framework.

## introduction despite recent successes and innovations in large language models (llms), their translational value in high-stakes environments, such as healthcare, has not been fully realized. this is primarily due to concerns about the trustworthiness and transparency of these models, stemming from their complex architecture and black-box nature. recent studies @xcite @xcite @xcite figure 1 : a physician diagnoses a patient who returned from a camping trip, presenting a combination of common symptoms and signs like fever and headaches. however, by recognizing rashes an atypical symptom, the physician ultimately identifies the condition as an allergy. previous work on confidence elicitation and calibration of large language models (llms) has mainly focused on general reasoning and general knowledge datasets for tasks such as logical reasoning, commonsense reasoning, mathematical reasoning, and scientific knowledge @xcite @xcite @xcite . few studies have investigated tasks that require expert knowledge, and these have shown considerable room for improvement. moreover, with the success of many closed-source llms, such as gpt-3.5 and gpt-4, which do not allow access to token-likelihoods and text embeddings, it has become prevalent to develop tailored methods for eliciting confidence estimates. however, most approaches developed consist of general prompting and sampling strategies without using domain-specific characteristics. traditionally, clinicians are taught to recognize and diagnose typical presentations of common illnesses based on patient demographics, symptoms and signs, test results, and other standard indicators @xcite . however, the frequent occurrence of atypical presentations is often overlooked @xcite . failing to identify atypical presentations can result in worse outcomes, missed diagnoses, and lost opportunities for treating common conditions. thus, awareness of atypical presentations in clinical practice is fundamental to providing high-quality care and making informed decisions. figure 1 depicts a simplistic example of how atypicality plays a role in diagnosis. incorporating the concept of atypicality has been shown to improve uncertainty quantification and model performance for discriminative neural networks and white-box large language models @xcite . this underscores the importance of leveraging atypical presentations to enhance the calibration of llms, particularly in high-stakes environments like healthcare. our study aims to address these gaps by first investigating the miscalibration of black-box llms when answering medical questions using non-logitbased uncertainty quantification methods. we begin by testing various baseline methods to benchmark the calibration of these models across a range of medical question-answering datasets. this benchmarking provides a comprehensive understanding of the current state of calibration in llms within the healthcare domain and highlights the limitations of existing approaches. next, we propose a new recalibration framework based on the concept of atypicality, termed atypical presentations recalibration. this method leverages atypical presentations to adjust the model's confidence estimates, making them more accurate and reliable. under this framework, we construct two distinct atypicality-aware prompting strategies for the llms, encouraging them to consider and reason over atypical cases explicitly. we then compare the performance and calibration of these strategies against the baseline methods to evaluate their effectiveness. finally, our empirical results reveal several key findings. first, black-box llms often fail to provide calibrated confidence estimates when answering medical questions and tend to remain overconfident. second, our proposed atypical presentations aware recalibration method significantly improves calibration, reducing calibration errors by approximately 60% on three medical question answering datasets and consistently outperforming existing baseline methods across all datasets. third, we observe that atypicality interacts in a complex manner with both performance and calibration, suggesting that considering atypical presentations is crucial for developing more accurate and trustworthy llms in healthcare settings. finally, we show that our framework is generalizable to open-source models with noteworthy improvements in confidence calibration. ## method in this section, we describe the methods used to elicit confidence from large language models (llms) as well as our recalibration methods. calibration in our settings refers to the alignment between the confidence estimates and the true likelihood of outcomes @xcite . our experiments are based on the framework described by @xcite , which divides the approaches into three main components: prompting, sampling, and aggregation, and uses it as baselines. in their framework, they leverage common prompting strategies such as vanilla prompting and chain-of-thoughts while also leveraging the stochasticity of llms. in contrast, we propose an approach, atypical presentation recalibration, that retrieves atypicality scores and use them as a recalibration method in order to have more accurate confidence estimates. our framework is mainly divided into two parts: atypicality prompting and atypicality recalibration. we explain how each of the three components are applied to our tasks and how we integrate atypicality to develop hybrid methods that combine these elements. ## conclusion in our study, we have demonstrated that llms remain miscalibrated and overconfident in the medical domain. our results indicate that incorporating the notion of atypicality when eliciting llm confidence leads to significant gains in calibration and some improvement in failure prediction for medical qa tasks. this finding opens the door to further investigate the calibration of llms in other high-stakes domains. additionally, it motivates the development of methods that leverage important domain-specific notions and adapting our method for white-box llms. we hope that our work can inspire others to tackle these challenges and to develop methods for more trustworthy, explainable and transparent models, which are becoming increasingly urgent. limitations this study present a first effort into assessing black-box llms calibration and the use of atypicality in the healthcare domain. several aspects of the study can further be improved for a better assessment. while we restricted ourselves to three modical question-answering datasets, we can expand it to more datasets with questions that are more open-ended or even different tasks such as clinical notes summarization which could also benefit a lot from having trusted confidence estimates. while our method generalizes to both closed and open source models like gpt4 and llama, it does not leverage information that we can potentially get from open source models such as the internal representations, token embeddings and tokenlikelihoods. an intereting future work could focus on how atypicality is represented internally in a llm, and if we can potentially read and control its representations of atypicality for better calibration. interpretability approaches such as mechanistic interpretability or representation engineering are interesting directions to consider. morever, our approach is still dependent on a prompt, and since llms are quite sensitive to how we prompt them, there could be even more optimal prompts for retrieving atypicality scores. lastly, the notion of atypicality is not only seen and leverage in healthcare, but it is also present in other domains such as law. adapting our methodology for other domains could further improve llms calibration performance.