P lug M ed: Improving Specificity in Patient-Centered Medical Dialogue Generation using In-Context Learning (2023)

Abstract: The patient-centered medical dialogue systems strive to offer diagnostic interpretation services to users who are less knowledgeable about medical knowledge, through emphasizing the importance of providing responses specific to the patients. It is difficult for the large language models (LLMs) to guarantee the specificity of responses in spite of its promising performance even in some tasks in medical field. Inspired by in-context learning, we propose PlugMed, a Plug-and-Play Medical Dialogue System, for addressing this challenge. PlugMed is equipped with two modules, the prompt generation (PG) module and the response ranking (RR) module, to enhances LLMs’ dialogue strategies for improving the specificity of the dialogue. The PG module is designed to stimulate the imitative ability of LLMs by providing them with real dialogues from similar patients as prompts. The RR module incorporates fine-tuned small model as response filter to enable the selection of appropriate responses generated by LLMs. Furthermore, we introduce a new evaluation method based on matching both user’s intent and high-frequency medical term to effectively assess the specificity of the responses. We conduct experimental evaluations on three medical dialogue datasets, and the results, including both automatic and human evaluation, demonstrate the effectiveness of our approach.

## introduction as a key task in health conversational assistants, medical dialogue generation aims to automatically generate informative responses to the users. it's better for such dialogues to be medical knowledgeable, patient-specific and context-aware, as the patients may be less knowledgeable about medical knowledge and their dialogue contexts may vary. prior studies @xcite have emphasized the critical role of domain knowledge in medical dialogue generation, drinking honey water on an empty stomach will indeed stimulate the secretion of gastric acid and cause stomach discomfort. it is recommended that you drink some warm water first, and then drink honey water after half an hour, which can reduce the irritation to the stomach. ## automatic evaluation metrics we observe that previous studies @xcite employ the metrics for open domain dialogue tasks that often cannot effectively measure system performance in task-oriented settings @xcite . from the results given in figure (3), we observe that the llm exhibits overconfidence. to judge whether our approach alleviates the problem, it becomes crucial to evaluate the dialogue actions taken by the llm. it is equally important to evaluate the dialogue content generated by llm. hence, we introduce two metrics that consider both intent and the usage of high-frequency medical terms. ## experiments ## conclusion in this paper, we use in-context learning to develop a patient-centered medical dialogue model. to this end, we introduce a prompt generation module capable of generating llm input from both global and local views. additionally, we construct a response ranking module using a supervised trained small model to filter the llm output. experimental results indicate that the responses generated by plugmed exhibit a greater inclusion of comprehensive medical terms and plugmed yields more accurate dialogue intents than other large language model based methods.