H uatuo GPT , Towards Taming Language Model to Be a Doctor (2023)

Abstract: In this paper, we present HuatuoGPT, a Large Language Model (LLM) for medical consultation. The core recipe of HuatuoGPT is to leverage both distilled data from **ChatGPT** and real-world data from **doctors** in the supervised fine-tuning stage. This is not only because purely using **ChatGPT**-distilled data might cause ‘model collapse’, but also because real-world data from **doctors** would be complementary to **ChatGPT**-distilled data. The responses from ChatGPT are usually detailed, well-presented, fluent, and instruction-followed, but it cannot perform like a doctor in many aspects, e.g. for interactive diagnosis. Therefore, the extra doctors’ data could tame a distilled language model to perform like doctors. To synergize the strengths of both data sources, we introduce RLMF (Reinforcement Learning from Mixed Feedback) where a reward model is trained to align the language model with the merits that both sources (ChatGPT and doctors) bring. Experimental results (in GPT-4 evaluation, human evaluation, and medical benchmark datasets) demonstrate that HuatuoGPT achieves state-of-the-art results in performing medical consultation among open-source LLMs. It is worth noting that by using additional real-world data and RLMF, the distilled language model (i.e., HuatuoGPT) outperforms its teacher model (i.e., ChatGPT) in most cases.

## introduction medicine stands as a paramount pillar in human existence, with its effectiveness heavily relying on the expertise and experience of professionals. yet, the advent of large language models (llms) like chatgpt heralds a transformative era for such experience-driven domains @xcite . llms learn and mimic human language by leveraging retrospective data and generating prospective output. considering the confluence of experiencedriven foundations in both llms and medicine, we believe that llms hold significant potential to improve medical consultation and diagnostic support. learning from chatgpt? recent studies @xcite @xcite show distilling from chatgpt could equip a language model with fluent chat capability in some common scenarios. however, chatgpt, and even gpt-4, exhibit relatively poorer performance in vertical domains (such as medicine) compared to domain experts. one contributing factor to this phenomenon is that annotators for chatgpt did not reach the expert level and lack of proficiency in medical knowledge that medical doctors have. therefore it is problematic to purely distill a medical llm from a poor teacher (i.e., chatgpt), as popular general democratized llms did @xcite . moreover, purely using chatgpt-distilled data might cause 'model collapse' @xcite and therefore limits the generalization of medical llms. learning from doctors? on the other hand, when a model only learns from real-world medical diagnosis dialogues @xcite @xcite , it produces overly concise responses and lacks the necessary level of detail due to the time constraints doctors face during the diagnostic process, leading to a condensed presentation of information. compared to the responses from chatgpt that are detailed and well-presented (e.g., with well-organized paragraphs, tables and bullets), doctors' responses are not friendly enough in terms of patience, level of detail, and logical organization. philosophy of huatuogpt to overcome the above issues, the core recipe of this work is to leverage both real-world data from doctors and figure 1 : example of chatgpt responses (left) and doctor responses (right), where texts are translated from chinese to english. questions raised by doctors are in blue, and medical diagnoses are underlined. note that chatgpt usually does not raise questions in response to patients or provide medical diagnoses like doctors. in the supervised fine-tuning (sft) stage, we leverage both real-world data from doctors and distilled data from chatgpt. the distilled data from chatgpt is used to tame language models to follow medical instructions and talk fluently. the additional real-world medical data not only inject medical knowledge into language models but also tame the language models to perform medical diagnoses like a doctor. moreover, the two sources of data are aligned with each other by role-enhanced promoting and chatgpt polishing respectively. after the sft stage, to further leverage the strengths of chatgpt and doctors and meanwhile mitigate their weaknesses, we propose reinforcement learning with mixed feedback (rlmf), inspired by rlhf @xcite ) and 1 to commemorate the renowned chinese physician hua tuo ( https://en.wikipedia.org/wiki/hua_tuo ). @xcite . it is used to reward generated responses that are not only patient-friendly (learned from chatgpt with better presentation quality, detailed, instruction-following abilities, and fluent chat), but also doctor-like (learned from doctors with professional and interactive diagnosis). evaluation we propose a comprehensive evaluation method that includes both manual and automatic evaluations for medical llms, covering single-turn and multi-turn medical consultations. the experimental results show that our hu-atuogpt outperforms existing open-source medical llms and chatgpt in automatic and manual evaluation. more impressively, our model surpasses the performance of gpt-3.5-turbo in terms of automatic evaluation and doctors' evaluation. moreover, huatuogpt achieves state-of-the-art (sota) zero-shot performance in several medical benchmarks such as cmedqa @xcite , webmedqa @xcite , and hu-atuo26m @xcite datasets. we will opensource our training data, code, huatuogpt model, and the reward model at https://github.com/ freedomintelligence/huatuogpt . ## motivation as shown in figure 1 , the responses from chatgpt and doctors are different but complementary. ## methodology following the above motivation, we propose to combine the strengths of both distilled data (from chatgpt) and real-world data (from doctors) to tame the medical llm to perform like a doctor, as illustrated in ## experiments in this section, we first introduce the baselines and then present the evaluation protocol and results including pairwise evaluation by gpt-4/doctors on single-turn questions and multi-turn conversations. considering the small scale of pairwise evaluation, we also evaluate our model on the large-scale medical qa benchmark to validate huatuogpt. ## related work the medical language model has always been a concern for researchers. the early models were mainly based on the gpt-2 series models to continue pretraining in the domain, such as biomedlm foot_4 and biogpt @xcite . recently, the success of distilled from chatgpt @xcite has stimulated some efforts to fine-tune large-scale language models using medical-related instruction data. chatdoctor @xcite ) is a medical model based on llama @xcite trained on the real-world instruction data (healthcaremagic-100k and icliniq-10k) between patient and physician and the distilled instruction data from chatgpt (genmedgpt-5k and disease database). based on the dataset of chatdoctor, medalpaca @xcite added more data points across a diverse range of tasks, including openly curated medical data transformed into q/a pairs with chatgpt and a collection of established nlp tasks in the medical domain. different from the previous work, visual med-alpaca foot_5 is only trained on the distilled instruction dataset while the 54k samples are filtered and edited by human experts. recently, med-palm2 @xcite was published, which is based on palm2 and finetuned in multimedqa for expert-level medical question answering. in chinese, bentsao @xcite ) is a knowledge-enhanced chinese medical llm trained on over 8k instructions generated from cmekg by chatgpt without real-world data. medicalgpt-zh @xcite is a chinese medical model based on chatglm-6b lora with 16-bit instruction fine-tuning. the training dataset was obtained from chinese medical knowledge question-and-answer pairs and clinical guideline texts from 28 medical departments. doctor-glm @xcite is a chinese medical chatglm-based llm trained on multiple medical datasets. in addition to the distilled instruction data from chatdoctor through translation, they incorporate some chinese medical conversation data into the training dataset. see app. h for more details on medical llms. ## conclusion this paper presents huatuogpt, a medical large language model, which leverages hybrid data and incorporates reinforcement learning from mixed feedback with chatgpt and doctors. gathered merits distilled data from chatgpt and real-world data from doctors, huatuogpt is able to generate patient-friendly and doctor-like responses. by refining its responses based on mixed feedback, the model can improve its conversational abilities while maintaining the reliability necessary for medical applications. the experimental results of automatic and manual evaluations indicate that our model surpasses the existing medical open-source llms and even outperforms chatgpt in most cases. as we continue to explore and develop this approach, we believe that further research in this area holds significant potential for advancing the field of ai in medicine.