Integrating Physician Diagnostic Logic into Large Language Models: Preference Learning from Process Feedback (2024)

Abstract: The utilization of large language models for medical dialogue generation has attracted considerable attention due to its potential to enhance response richness and coherence. While previous studies have made strides in optimizing model performance, there is a pressing need to bolster the model’s capacity for diagnostic logic to ensure patient safety. In response to this need, we propose an approach termed preference learning from process feedback (PLPF), which involves integrating the doctor’s diagnostic logic into LLMs. PLPF encompasses three key components: rule modeling, preference data generation, and preference alignment. These components collectively serve to train the model to adhere to the diagnostic process. Our experimental results, utilizing Standardized Patient Testing, demonstrate that PLPF enhances the diagnostic accuracy of the baseline model in medical conversations by 17.6%, surpassing the performance of traditional approaches. Moreover, PLPF exhibits effectiveness in both multi-round and single-round dialogue tasks, thereby highlighting its potential in improving medical dialogue generation. Our dataset is available at https://github.com/Chengfeng-Dou/SpTesting.

## introduction the use of large language models (llms) @xcite has recently exploded in the field of medical dialogue generation. however, training robust medical dialogue models is crucially based on high-quality training data @xcite . as a result, considerable efforts have been made to generate extensive training data sets to fine-tune these models. furthermore, certain studies have made notable progress, such as the application of reinforcement learning from human feedback (rhlf) to guide models in generating user-friendly responses @xcite . despite the significant advancements in prior research on rlhf, current open-source medical llms remain inadequately user-directed. they frequently oscillate between two problematic behaviors: 1) precipitously rendering a diagnosis without adequate patient data collection, and 2) becoming ensnared in an endless loop of data gathering without progressing to a definitive diagnosis. this issue appears to originate from the inherent discordance among the preference data provided by annotators. in practical settings, physicians' diagnostic approaches typically bifurcate into: 1) an initial comprehensive collection of patient information followed by a diagnosis, and 2) an early presentation of a conjectural diagnosis, which is then substantiated through progressive data accumulation. it is the simultaneous learning from these two divergent preferences that likely precipitates the disarray in the model's conversational strategy. we believe that standardising the medical dialogue process is the key to solving the above problems. our proposed approach, known as preference learning from process feedback (plpf), focuses on ensuring the rationality of the conversation flow, which sets it apart from traditional methods. the core idea is to represent the doctor's diagnostic logic using a flowchart and employ preference learning to train the model to avoid generating responses that deviate from the established process. in particular, we have developed a flowchart based on the doctor's expertise, as depicted in fig. 1 . this flowchart effectively outlines the objectives the physician must achieve and the constraints that must be followed during the diagnostic process, while also illustrating the dependencies between these objectives. to utilize the flowchart for guiding model training, we have established explicit rules for each activity, decision, and constraint outlined in the flowchart. the state of a dialogue in the flowchart can be determined by evaluating whether the dialogue conforms to these rules. based on these established rules, our approach consists of three phases: rule modeling, preference data generation, and preference alignment. initially, we have developed a rule evaluation model (rem) to automatically assess whether a conversation adheres to a specific rule. building on this, we have devised a method to assign scores to conversations that take into account rule dependencies. these scores are then used for the generation of preference data. we achieve this using an innovative one-shot learning-based approach to predict dialogue trajectories and leveraging rem to appropriately rank these trajectories. subsequently, we employ the direct preference optimization (dpo) algorithm @xcite to train models based on the acquired preference data. we used standardized patient testing, a widely accepted method in the medical field, to assess our approach. to achieve this, we built the chinese standardized patient test (cspt) dataset. furthermore, we employed a retrieval-augmented generation technique to create a patient simulator for interactive testing with llm. the results of our experiments indicate that our approach improves the diagnostic precision of the baseline model in medical conversations by 17.6%. we also tested our approach on three public datasets to assess its performance in both multi-round and single-round conversations. the results show that our approach effectively enhances the model's understanding of physician expressions. in summary, our work provides the following contributions. ## conclusion in this work, we have introduced an innovative approach termed preference learning from process feedback (plpf), which integrates the diagnostic logic of healthcare professionals into the llm. plpf encompasses rule modeling, preference data generation, and preference alignment to train the model to adhere to the diagnostic process. our experimental findings, using standardized patient testing, reveal that plpf enhances the diagnostic accuracy of the baseline model in medical conversations by 17.6%. furthermore, plpf exhibits efficacy in both multi-round and single-round dialogue tasks, underscoring its potential for advancing medical dialogue generation.