Benchmarking Large Language Models on Communicative Medical Coaching: A Dataset and a Novel System (2024)

Abstract: Traditional applications of natural language processing (NLP) in healthcare have predominantly focused on patient-centered services, enhancing patient interactions and care delivery, such as through medical dialogue systems. However, the potential of NLP to benefit inexperienced doctors, particularly in areas such as communicative medical coaching, remains largely unexplored. We introduce “ChatCoach”, a human-AI cooperative framework designed to assist medical learners in practicing their communication skills during patient consultations. ChatCoach differentiates itself from conventional dialogue systems by offering a simulated environment where medical learners can practice dialogues with a patient agent, while a coach agent provides immediate, structured feedback. This is facilitated by our proposed Generalized Chain-of-Thought (GCoT) approach, which fosters the generation of structured feedback and enhances the utilization of external knowledge sources. Additionally, we have developed a dataset specifically for evaluating Large Language Models (LLMs) within the ChatCoach framework on communicative medical coaching tasks. Our empirical results validate the effectiveness of ChatCoach.

## introduction the advent of natural language processing (nlp) has significantly impacted the healthcare domain, carving pathways for numerous applications that enhance both patient-centered services and healthcare operations. these applications encompass medical dialogue systems, automated medical coding, clinical decision support, and information extraction from electronic health records, among others @xcite . despite the strides made in these areas, there remains largely untapped potential of nlp in aiding the professional development of early-stage medical learners and early-career practitioners. a critical aspect of this professional development revolves around enhancing communication skills, especially in the context of medical consultations. a wealth of research highlights the critical importance of effective communication in medical practice. @xcite found a strong consensus among medical students on the need to refine communication skills for better medical practice, with a significant proportion showing marked improvements after training. various other studies @xcite @xcite have consistently shown that proficient communication skills are key to increasing patient satisfaction, enhancing diagnostic precision, and fostering stronger doctor-patient relationships. despite this acknowledgment, the area of communicative medical coaching, particularly through leveraging advanced language language models (llms), remains relatively unexplored. addressing this gap, we introduce chatcoach, a novel human-ai cooperative framework devised to enhance communicative proficiency among medical learners. unlike traditional dialogue systems focused on patient engagement, chatcoach transitions the focus towards the professional development of medical practitioners. this approach fosters a dynamic environment where learners can engage in realistic dialogues, receive immediate feedback, and refine their understanding of medical terminologies. chatcoach provides a simulated, realistic environment for medical learners to practice their communication skills during patient consultations. the architecture of chatcoach (shown in fig. 1(a) ) includes a patient agent simulating realworld doctor-patient interactions and a coach agent providing real-time feedback on learners' termino-logical usage. a major challenge in this direction is the absence of publicly available data for communicative medical coaching, largely due to the sensitive nature of healthcare information and the substantial costs of data collection and annotation. to overcome this challenge, we devised a multi-agent data generation framework (shown in fig. 1(b) ) using external resources to produce training data for fine-tuning an open-source llm. this framework employs llmbased agents, including patient, coach, and doctor agents, which interact by querying and retrieving information from two sources: a medical dialogue database and a medical knowledge database. additionally, we compiled a human-annotated testing dataset to assess llms' capabilities in communicative medical coaching. our contributions are threefold: ## generalized chain-of-thought (gcot) communicative medical coaching poses a unique challenge for large language models (llms), characterized by its open-ended, knowledgeintensive reasoning demands. the feedback generated must be real-time and easily understandable by medical practitioners. additionally, the reasoning process requires the utilization of external knowledge databases. traditional prompting methods, such as zero-shot cot often fall short in generating structured feedback and effectively incorporating external knowledge (refer to fig. 3 for an example of coach feedback generated by prompting-based approaches). here, we introduce the generalized chain-of-thought (gcot) approach. gcot improves upon cot by embedding generalizable variables within reasoning paths. these variables are elements shared across various data samples' reasoning steps, facilitating the creation of structured feedback and seamless external knowledge integration. gcot adopts a two-step process aimed at utilizing generalizable variables for prompt generation: ## constructing the chatcoach dataset: a multi-agent approach for generating domain-specific conversational data the development of novel nlp applications, especially in specialized fields such as medical coaching, is hindered by the scarcity of domain-specific conversational datasets. to bridge this gap, we introduce a novel multi-agent data generation framework (illustrated in fig. 1(b) ) leveraging external resources to synthetically produce training data for fine-tuning open-source llms. our framework integrates three types of llmbased agents -patient, coach, and doctor -each designed to simulate real-world roles within medical dialogues. these agents interact through querying and retrieving information from two primary sources: a medical dialogue database and a medical knowledge database. specifically, the patient agent simulates patient inquiries; the doctor agent generates medical responses, potentially including common errors; and the coach agent offers correc- ## experiments we assess the medical coaching capabilities of both an open-sourced llm and a close-sourced llm using the proposed chatcoach frameworks on two tasks specified in sec. 5, namely detection and correction of misuse of the medical terminology. the generated coach feedbacks are evaluated based on both automatic and human evaluation metrics. ## conclusion this work introduces chatcoach, a new human-ai cooperative framework for communicative medical coaching. at the core of our approach is the generalized chain-of-thought (gcot), a strategy that significantly improves feedback structuring and the integration of external knowledge. we developed the first benchmark dataset designed to evaluate the medical coaching capabilities of large language models (llms) within the chatcoach framework. through a series of automatic and human evaluations, we demonstrate chatcoach's effectiveness in tackling two key tasks in communicative medical coaching, showcasing its potential to enhance medical education through ai.