C hi M ed- GPT : A C hinese Medical Large Language Model with Full Training Regime and Better Alignment to Human Preferences (2024)

Abstract: Recently, the increasing demand for superior medical services has highlighted the discrepancies in the medical infrastructure. With big data, especially texts, forming the foundation of medical services, there is an exigent need for effective natural language processing (NLP) solutions tailored to the healthcare domain. Conventional approaches leveraging pre-trained models present promising results in this domain and current large language models (LLMs) offer advanced foundation for medical text processing. However, most medical LLMs are trained only with supervised fine-tuning (SFT), even though it efficiently empowers LLMs to understand and respond to medical instructions but is ineffective in learning domain knowledge and aligning with human preference. In this work, we propose ChiMed-GPT, a new benchmark LLM designed explicitly for Chinese medical domain, and undergoes a comprehensive training regime with pre-training, SFT, and RLHF. Evaluations on tasks including information extraction, question answering, and dialogue generation demonstrate ChiMed-GPT’s superior performance over general domain LLMs. Furthermore, we analyze possible biases through prompting ChiMed-GPT to perform attitude scales regarding discrimination of patients, so as to contribute to further responsible development of LLMs in the medical domain.

## introduction medical service is one of the cornerstones of societal welfare, instrumental in advancing social development and elevating the contentment of its members. with the public's growing expectations for better health care, the burgeoning demand for medical services juxtaposed against a limited medical workforce, intensifies the imbalance between healthcare availability and peoples' requests. this mismatch underscores the challenges that current medical infrastructure is required to meet societal needs, thus highlighting the importance of advancing medical intelligence so that healthcare services could be provided sufficiently and automatically. to improve medical intelligence, natural language processing (nlp) technologies are of great significance to efficiently process text data, which are the primary medium for information processing. among widely used nlp techniques, for years that pre-trained models serve as the foundation and are utilized in various nlp tasks and achieve state-of-the-art performance @xcite @xcite @xcite @xcite @xcite . however, such approaches with pre-trained models heavily depended on the pretraining and fine-tuning paradigm, where a model requires considerable labeled medical data and is normally trained on them and applied to particular tasks. in doing so, pre-trained models are bound to given tasks and hard to transfer to new scenarios. recently, the emergence of large language models (llms), such as gpt-3.5 and gpt-4 (openai, 2023) that are accessible via online api service and alpaca @xcite , vicuna @xcite , and ziya @xcite whose model weights are open sourced, lead to a paradigm shift, offering unified solutions to a wide range of tasks. however, most well-performed llms are trained in the general domain, facing challenges when they are applied to specialized fields, such as the medical domain, owing to the knowledge gap among domains. for example, gpt-4 很抱歉听到您的不适。您描述的症状可能是肠胃道感染或其他肠道问题。简要地说：对孩子的影响：轻度的腹 泻通常不会直接伤害到胎儿，但持续的腹泻可能导致脱水，这对您和孩子都不好。脱水可能会导致宫缩，这可 能早产。应该怎么办：马上咨询医生或者去医院。确保饮用足够的水分以防止脱水。避免摄入任何可能导致腹 泻的食物或饮料。建议您尽快咨询医生以得到专业的医疗建议。 i'm sorry to hear about your discomfort. the symptoms you described might be related to a gastrointestinal infection or other intestinal issues. in brief: impact on the child: mild diarrhea usually won't directly harm the fetus, but persistent diarrhea might lead to dehydration, which isn't good for both you and the child. dehydration can cause contractions, which might result in preterm labor. what to do: consult a doctor immediately or go to the hospital. ensure you drink enough fluids to prevent dehydration. avoid consuming any foods or drinks that might cause diarrhea. it's recommended that you seek medical advice from a professional as soon as possible. chimed-gpt 考虑是腹泻现象，容易诱发子宫收缩出现早产症状，可以在医生的指导下服用黄连素，思密达等治疗，同时注 意多喝水，饮食要清淡，禁食生冷刺激食物。 it is considered to be diarrhea, which can easily induce uterine contractions and cause symptoms of premature labor. you can take berberine, smecta and other treatments under the guidance of a doctor. at the same time, pay attention to drinking more water, eating a light diet, and fasting away from raw and cold stimulating foods. to address these challenges, in this paper, we propose chimed-gpt, a new benchmark llm for chinese medical text processing. following the convention in existing studies to train domainspecific llms, we continually train a general domain llm, ziya-13b-v2 @xcite , on large medical data, and perform a full training regime including (continue) pre-training, sft, and rlhf. as a distinctive feature, we use data augmentation to produce high-quality human preference data for reword model training and employ rejection sampling fine-tuning to learn from the data, which is proved to be more efficient @xcite than standard proximal policy optimization (ppo). the data to train chimed-gpt are extracted from multiple resources ranging from medical articles to real-world interactions between patients and doctors, which allows our llm to effectively align with medical knowledge and generate appropriate responses for patients' inquires rather than difficult-to-understand text produced by other llms. in addition, we train chimed-gpt on safety data containing desired and appropriate responses (e.g., those refuse-to-answer instances) to deal with the scenario when llms are prompted with toxic instructions. particularly, the context length of chimed-gpt is set to 4,096, which exceeds the context size of existing medical llm, so that provides better text processing ability to medical application. we evaluate chimed-gpt on 2 the chimed-gpt chimed-gpt is built on ziya-13b-v2 @xcite , whose architecture follows the standard transformer @xcite decoder with 13 billion parameters and is pre-trained on 600 billion chinese and english tokens, guaranteeing a superior chinese text processing capability. particularly, ziya-13b-v2 also extends the context length to 4,096 tokens, which ensures to meet our requirement for better context processing with chimed-gpt. the overall architecture and training procedure of chimed-gpt are illustrated in figure 1, which consists of three stages, namely, pretraining, sft, and rlhf, where training details (i.e., datasets and implementation) for each stage are illustrated in the following subsections. ## results and analysis following existing studies, we evaluate chimed-gpt under zero-and few-shot settings, and report the results on three tasks, namely, information ex- ## bias analysis llms have been shown to reproduce and amplify biases present in the training data and to generate potentially toxic or offensive content @xcite @xcite @xcite . particularly for medical applications, llms are expected to be carefully examined on potential biases, which are critical in affecting psychological state of patients when they are occurred along with the text produced from llms. therefore, it becomes essential to thoroughly investigate and understand any biases that may manifest in these models when they are utilized to assist healthcare professionals or directly provide information to patients. in evaluating potential bias of llms, we choose mental disease statements as test cases to assess if appropriate responses are provided towards particular inputs. 12 in doing so, we utilize two scales, namely, community attitudes toward the mentally ill (cami) @xcite and mental illness: clinicians' attitudes (mica) @xcite , whose details are elaborated as follows: input 假设你是一名医生，以下对心理疾病患者说法你是 否同意。请在"完全不同意"、"不同意"、"稍微不同 意"、"稍微同意"、"同意"、"完全同意"中选择 患有严重精神疾病的人往往很危险。 translation assuming you are a doctor, do you agree with the following statements about mental illness patients? please choose among "strongly disagree", "disagree", "somewhat disagree", "somewhat agree", "agree", "strongly agree". people with serious mental illness are often dangerous. for both scales, the response to each statement is translated to a bias score based on the scale guideline (e.g., strongly disagree to strongly agree are mapped into a range of scores), guaranteeing higher scores indicating stronger bias and lower scores suggesting weaker bias. note that official mapping rules for various statements are different, e.g., for statements with bias, agreeing on them leads to high scores and for those without bias, disagreeing corresponds to high scores. the range of bias scores for cami and mica are @xcite and @xcite , respectively. we prompt different llms, including gpt-3.5-turbo, gpt-4, ziya-v1, ziya-v2, barichuan, taiyi, medicalgpt (z), medicalgpt (b), to conduct the scale test in the same way as human participants normally do, comparing with that performed by chimed-gpt. in doing so, we firstly translate scale statements into chinese manually to facilitate llm processing since the original ones are in english. then, we ask llms to produce the level of agreement to the statements in cami and mica, where an example input prompt is presented in ## related work learning good text representation is of great importance in natural language processing @xcite @xcite @xcite recently, llms have achieved remarkable success in this area especially for text generation, such as gpt-4 (openai, 2023), alpaca @xcite , vicuna @xcite , and ziya @xcite . based on this circumstance, there has been a seamless transition of these llms into specialized domains, such as medical, financial, and legal domains @xcite @xcite . among different domains, the medical one attracts much attention in existing studies for its great value in real-world applications, where several medical llms are presented recently, such as biomedlm (venigalla 13 we present more examples in appendix e showing chimed-gpt replies appropriate responses to toxic inputs. et al., 2022), biogpt @xcite , medalpaca @xcite , chatdoctor @xcite , and baize-healthcare @xcite , etc. for chinese medical llms, researchers continually train general domain llms on medical domain qa pairs or dialogues, such as bentsao and medicalgpt trained on chatglm and ziya, respectively, and also further incorporate medical knowledge into llms @xcite . however, most models are primarily trained with sft, which restricts their ability in aligning llms with particular knowledge such as human preferences. in addition, almost all open-sourced llms suffer from context restriction with only 2,048 tokens, so do the medical ones, which prevents them from processing long medical texts. although our approach follows the paradigm of continue training general llms with medical data, it applies a full training regime including pre-training, sft, and rlhf, which allows it to effectively enhance with domain knowledge, understand domain-specific instructions, and align with human preferences. data augmentation and rejection sampling fine-tuning are also utilized in further enhancing chimed-gpt with improved guidance on distinguishing outputs from human and models. moreover, the context length of chimed-gpt is extended to 4,096 with the help of its foundation model, which guarantees its practical value of being utilized in the medical domain through enhanced context processing capability. ## conclusion in this paper, we propose chimed-gpt for chinese medical text processing, which is built upon ziya-13b-v2 and inherited its capability to process extensive context lengths. chimed-gpt is learned through a holistic training framework that seamlessly integrates pre-training, sft, and rlhf stages, and ensured that it not only captures domain-specific knowledge but also adapts to multiple scenarios, outshining existing models that often solely resort to sft. empirical results on typical medical text processing tasks, i.e., information extraction, question answering, and dialogue generation, demonstrate the effectiveness of chimed-gpt, where it outperforms strong baselines and existing studies on different benchmark datasets. further analyses show a relatively low bias in chimed-gpt, which confirms our efforts to develop responsible domain-specific llms. question 我想问如何避免牙龈肿痛？ i would like to ask how to avoid swollen gums? gpt-3.5-turbo 要避免牙龈肿痛，首先要保持良好的口腔卫生习惯，包括每天刷牙两次，使用牙线清洁牙齿间隙，定期洗牙。此 外，避免吃太多粘腻的食物，减少吸烟和饮酒的频率，注意控制牙龈周围的炎症。如果牙刷过硬或刷牙力度太大， 也会导致牙龈炎症，所以刷牙要使用软毛牙刷，并轻柔地刷牙。如果牙龈肿痛无法缓解，最好去找牙医进行检查和 治疗。 to avoid gum swelling and pain, you must first maintain good oral hygiene habits, including brushing twice a day, flossing between teeth, and regular teeth cleaning. in addition, avoid eating too much sticky food, reduce the frequency of smoking and drinking, and pay attention to controlling inflammation around the gums. if your toothbrush is too hard or you brush your teeth too hard, it can also cause gum inflammation, so use a soft-bristled toothbrush and brush your teeth gently. if gum swelling and pain persist, it is best to see a dentist for examination and treatment. gpt-4 1. 每日两次刷牙，使用含氟牙膏。2. 使用牙线清除食物残渣。3. 使用抗菌漱口水。4. 不要吸烟或使用烟草制品。5. 定期看牙医进行检查和洗牙。注意：若牙龈已经肿痛，请咨询牙医。 1. brush your teeth twice a day using fluoride toothpaste. 2. use dental floss to remove food debris. 3. use antibacterial mouthwash. 4. avoid smoking or using tobacco products. 5. regularly visit the dentist for check-ups and cleanings. note: if your gums are already swollen and painful, consult a dentist. ziya-v1 你好！牙龈肿痛是指牙龈出现疼痛，一般是由于牙龈炎或者是牙周炎引起的。 hello! gum swelling refers to pain in the gums, usually due to gingivitis or periodontitis. ziya-v2 可以保持良好的口腔卫生，饭后漱口，睡前刷牙，刷牙时不要过大的刷，以免损伤口腔黏膜。及时治疗感冒，增加 抵抗力，注意休息，避免劳累和熬夜，保持心情舒畅。 you can maintain good oral hygiene, rinse your mouth after meals, brush your teeth before going to bed, and do not brush your teeth too hard to avoid damaging the oral mucosa. treat colds in time to increase resistance, pay attention to rest, avoid fatigue and staying up late, and keep your mood comfortable. baichuan 你好，目前你的情况需要注意是否有龋坏，如果有龋坏需要及时进行修复治疗。 hello, in your current situation, you need to pay attention to whether there are caries. if there are caries, you need to repair and treat them in time. taiyi 牙龈肿痛可能是由于多种原因引起的，以下是一些可能的建议：1. 刷牙和使用牙线：保持口腔清洁是预防牙龈肿痛 的关键。每天刷牙两次，使用牙线和漱口水可以帮助去除牙菌斑和食物残渣，减少牙龈炎和牙龈肿痛的发生。2. 饮 食调整：减少摄入高糖和高脂肪的食物，增加蔬菜和水果的摄入，可以帮助控制口腔细菌的生长，减少牙龈炎和牙 龈肿痛的发生。3. 避免吸烟和饮酒：吸烟和饮酒会刺激牙龈，增加牙龈炎和牙龈肿痛的风险。4. 定期看牙医：定期 看牙医可以帮助发现和治疗牙龈炎和牙龈肿痛，避免病情恶化。如果以上方法不能有效缓解牙龈肿痛，建议及时就 医，寻求专业医生的帮助。 sore gums can be caused by a number of reasons, here are some possible suggestions: 1. brush and floss: keeping your mouth clean is key to preventing sore gums. brushing your teeth twice a day, using dental floss and mouthwash can help remove plaque and food particles and reduce the occurrence of gingivitis and gum swelling. 2. dietary adjustment: reducing the intake of high-sugar and high-fat foods and increasing the intake of vegetables and fruits can help control the growth of oral bacteria and reduce the occurrence of gingivitis and gum swelling. 3. avoid smoking and drinking: smoking and drinking can irritate the gums and increase the risk of gingivitis and gum swelling. 4. visit your dentist regularly: regular visits to your dentist can help detect and treat gingivitis and gum swelling before the condition gets worse. if the above methods cannot effectively relieve gum swelling and pain, it is recommended to seek medical treatment in time and seek professional help.