Can LLM s Correct Physicians, Yet? Investigating Effective Interaction Methods in the Medical Domain (2024)

Abstract: We explore the potential of Large Language Models (LLMs) to assist and potentially correct physicians in medical decision-making tasks. We evaluate several LLMs, including Meditron, Llama2, and Mistral, to analyze the ability of these models to interact effectively with physicians across different scenarios. We consider questions from PubMedQA and several tasks, ranging from binary (yes/no) responses to long answer generation, where the answer of the model is produced after an interaction with a physician. Our findings suggest that prompt design significantly influences the downstream accuracy of LLMs and that LLMs can provide valuable feedback to physicians, challenging incorrect diagnoses and contributing to more accurate decision-making. For example, when the physician is accurate 38% of the time, Mistral can produce the correct answer, improving accuracy up to 74% depending on the prompt being used, while Llama2 and Meditron models exhibit greater sensitivity to prompt choice. Our analysis also uncovers the challenges of ensuring that LLM-generated suggestions are pertinent and useful, emphasizing the need for further research in this area.

## introduction recent advancements demonstrate large language models' (llms) effectiveness in medical ai applications, notably in diagnosis and clinical support systems @xcite . studies reveal their proficiency in answering diverse medical inquiries with high precision @xcite @xcite @xcite @xcite @xcite , emphasizing the importance of tailored prompt design @xcite , and advanced prompting techniques for complex tasks @xcite . despite their potential, there are still challenges in deploying llms in the clinical domain @xcite @xcite . furthermore, existing works evaluate the quality of the standalone llm, while we are interested in the setting where the llm is supporting a human decision-maker. in many high-stakes medical scenarios, human experts (e.g., physicians) are responsible for making final decisions, and they can seek assistance from ai agents: understanding how ai systems and experts can interact is essential for ensuring their practical utility and reliability. we aim to bridge this gap by analyzing the accuracy of llms in medical and clinical tasks when interacting with a domain expert (i.e., a physician). for the sake of simplicity, we consider the setting where the llm is asked to answer a question after a domain expert verbalizes their opinion. we examine whether llms avoid challenging expert inputs, potentially affecting response quality. through empirical tests, we assess llms' ability to rectify expert errors while maintaining collaboration, analyzing the impact of expert performance and prompt design on optimizing the performance in clinical decision-making. our study presents two main contributions. first, we introduce a binary pubmedqa @xcite dataset featuring plausible correct and incorrect explanations generated by gpt4. second, we highlight the importance of prompt design in enhancing llm interactions with medical experts, showing its influence on llms' ability to correct physician errors, explain medical reasoning, adapt to physician input, and ultimately improve llm performance. ## experimental setup we ran an experimental evaluation aimed at answering the following research questions: q1: can llms correct physicians when needed? q2: can llms explain the reasons behind their answers? q3: can llms correct physicians when they provide arguments for their answers? q4: can llms fed with physician answers outperform both themselves and physicians? ## limitations a limitation of our study is the use of gpt-4 to simulate plausible correct and incorrect responses to the questions, to complement the ground-truth ones contained in the pubmedqa dataset. this choice is justified by recent findings @xcite highlighting the effectiveness of llms as generative reasoners capable of modeling user behavior and simulating their opinions/preferences in human-llm interactions. nonetheless, real-world experiments involving interactions with physicians should be planned to corroborate and strengthen the results found in this paper. a second limitation is that this work is not providing solutions to the problems being raised. indeed, the main goal of the work is raising awareness on the limitations of current open-source llms for medical decision support. we hope that these insights will encourage further research aimed to address these limitations.