M ed D ialog: Large-scale Medical Dialogue Datasets (2020)

Abstract: Medical dialogue systems are promising in assisting in telemedicine to increase access to healthcare services, improve the quality of patient care, and reduce medical costs. To facilitate the research and development of medical dialogue systems, we build large-scale medical dialogue datasets – MedDialog, which contain 1) a Chinese dataset with 3.4 million conversations between patients and doctors, 11.3 million utterances, 660.2 million tokens, covering 172 specialties of diseases, and 2) an English dataset with 0.26 million conversations, 0.51 million utterances, 44.53 million tokens, covering 96 specialties of diseases. To our best knowledge, MedDialog is the largest medical dialogue dataset to date. We pretrain several dialogue generation models on the Chinese MedDialog dataset, including Transformer, GPT, BERT-GPT, and compare their performance. It is shown that models trained on MedDialog are able to generate clinically correct and doctor-like medical dialogues. We also study the transferability of models trained on MedDialog to low-resource medical dialogue generation tasks. It is shown that via transfer learning which finetunes the models pretrained on MedDialog, the performance on medical dialogue generation tasks with small datasets can be greatly improved, as shown in human evaluation and automatic evaluation. The datasets and code are available at https://github.com/UCSD-AI4H/Medical-Dialogue-System

## introduction telemedicine refers to the practice of delivering patient care remotely, where doctors provide medical consultations to patients using hipaa compliant video-conferencing tools. as an important complement to traditional face-to-face medicine practiced physically in hospitals and clinics, telemedicine has a number of advantages. first, it increases access to care. for people living in medically under-served communities (e.g., rural areas) that are in shortage of clinicians, telemedicine enables them to receive faster and cheaper care compared with traveling over a long distance to visit a clinician. second, it reduces healthcare costs. in a study foot_0 by jefferson health, it is shown that diverting patients from emergency departments with telemedicine can save more than $1,500 per visit. third, telemedicine can improve the quality of care. the study in @xcite shows that telemedicine patients score lower for depression, anxiety, and stress, and have 38% fewer hospital admissions. other advantages include improving patient engagement and satisfaction, improving provider satisfaction, etc. please refer to @xcite for a more comprehensive review. while telemedicine is promising, it has several limitations. first, it puts additional burden on physicians. in addition to practicing face-toface medicine which already makes physicians very busy, physicians need to provide remote telemedicine consultations, which further increases the risk of physician burnout. second, different from in-hospital patients, the progression of whose medical conditions can be easily tracked by clinicians, remote patients are difficult to track and monitor. to address such problems, there has been increasing research interest in developing artificial intelligence (ai) methods to assist in telemedicine. in particular, medical dialogue systems are being developed to serve as "virtual doctors". these "virtual doctors" are aimed to interact with patients via natural dialogues, asking about the medical conditions and history of patients and providing clinical advice. they can also proactively reach out to patients to ask about the progression of patients' conditions and provide timely interventions. to build medical dialogue systems, a large collection of conversations between patients and doctors is needed as training data. due to data privacy concerns, such data is difficult to obtain. the existing medical dialogue datasets @xcite are limited in size or biased to certain diseases, which cannot adequately serve the purpose of training medical dialogue systems that can achieve doctor-level intelligence and cover many specialities in medicine. to address the limitations of existing datasets, we build large-scale medical dialogue datasets -meddialog -that contain 1) a chinese dataset with 3.4 million conversations between patients and doctors, 11.3 million utterances, 660.2 million tokens, covering 172 specialties of diseases, and 2) an english dataset with 0.26 million conversations, 0.51 million utterances, 44.53 million tokens, covering 96 specialties of diseases. both datasets cover almost all specialities in medicine, ranging from internal medicine to family medicine and covers a wide spectrum of diseases, including cancer, pneumonia, etc. to our best knowledge, they are the largest chinese and english medical dialogue datasets to date. the data is open to the public. each consultation starts with a description of medical conditions and history, followed by the conversation between doctor and patient. in certain consultations, doctors make diagnosis conclusions and give suggestions on treatment. the conversations have multiple turns. on the chinese meddialog (meddialog-cn) dataset, we train several dialogue generation models for the interested community to benchmark with. generating a response given the conversation history can be formulated as a sequence-tosequence (seq2seq) learning problem, where we use the transformer @xcite architecture to perform this task. transformer consists of an encoder which embeds the conversation history and a decoder which generates the response. both the encoder and decoder use self-attention to capture long-range dependency between tokens. in addition to training the transformer on meddialog-cn from scratch, we can pretrain the encoder and decoder on a corpora much larger than meddialog-cn, then finetune them on meddialog-cn. bert-gpt @xcite is a pretrained model where the encoder is pretrained using bert @xcite and the decoder is pretrained using gpt @xcite . besides the seq2seq formulation, dialogue generation can be formulated as a language modeling problem which generates the next token in the response conditioned on the concatenation of the already generated tokens in the response and the conversation history. gpt @xcite is a pretrained language model based on transformer decoder. bert-gpt and gpt are finetuned on meddialog-cn. we perform evaluation of these models using automatic metrics including perplexity, bleu @xcite , dist @xcite , etc. the generated responses are clinically informative, accurate, and human-like. we utilize the models trained on the large-scale meddialog-cn dataset to improve performance in low-resource dialogue generation tasks where the dataset size is small. the study is performed on covid-19 dialogue generation on the coviddialog @xcite dataset, which contains 1,088 dialogues and 9,494 utterances. the small size of this dataset incurs high risk of overfitting, if directly training the large-sized neural models on it. to alleviate this risk, we take the weights of dialogue generation models pretrained on meddialog-cn and finetune the weights on coviddialog. human evaluation and automatic evaluation show that pretraining on meddialog-cn can greatly improve the performance on coviddialog and generate clinically meaningful consultations about covid-19. the major contributions of this paper are: the rest of this paper is organized as follows. section 2 and 3 present the datasets and dialogue generation models (dgms). section 4 gives experimental results of developing dgms on chinese meddialog and studies the transferability of dgms trained on meddialog-cn to other low-resource medical dialogue generation tasks. section 5 reviews related works and section 6 concludes the paper. ## related works there have been several works investigating medical dialogue generation. wei et al. @xcite built a task-oriented dialogue system for automatic diagnosis. the system detects the user intent and slots with values from utterances, tracks dialogue states, and generates responses. xu et al. @xcite developed a medical dialogue system for automatic medical diagnosis that converses with patients to collect additional symptoms beyond their self-reports and automatically makes a diagnosis. this system incorporates a medical knowledge graph into the topic transition in dialogue management. xia et al. @xcite developed a reinforcement learning (rl) based dialogue system for automatic diagnosis. they proposed a policy gradient framework based on the generative adversarial network to optimize the rl model. ## datasets our meddialog consists of a chinese dataset and an english dataset, collected from different sources. ## methods we train several dialogue generation models on the chinese meddialog dataset for the interested research community to benchmark with. during training, given a dialogue containing a sequence of alternating utterances between patient and doctor, we process it into a set of pairs {(s i , t i )} where the target t i is a response from the doctor and the source s i is the concatenation of all utterances (from both patient and doctor) before t i . a dialogue generation model takes s as input and generates t. this problem can be formulated either as a sequence-to-sequence learning problem where the goal is to generate t conditioned on s via an encoder-decoder model, or as a language modeling problem which generates the i-th token t i in t conditioned on the concatenation of the conversation history s and the already generated sequence t 1 , • • • , t i-1 in the response before t i via a language model. ## conclusions and future works to facilitate the research and development of medical dialogue systems that can potentially assist in telemedicine, we build large-scale medical dialogue datasets -meddialog -which contain 1) a chinese dataset with 3.4 million conversations between patients and doctors, 11.3 million utterances, 660.2 million tokens, covering 172 specialties of diseases, and 2) an english dataset with 0.26 million conversations, 0.51 million utterances, 44.53 million tokens, covering 96 specialties of diseases. to our best knowledge, they the largest of their kind. we pretrain transformer, gpt, and bert-gpt on meddialog-cn. the results show that the generated dialogues by these pretrained models are clinically meaningful and human-like. we use transfer learning to apply these pretrained models for low-resource dialogue generation. on a covid-19 dialogue generation task where the dataset is small, human evaluation and automatic evaluation show that models pretrained on meddialog-cn can effectively improve the quality of generated responses. for future work, we will annotate medical entities in our datasets. such annotations can facilitate the development of goal-oriented medical dialog systems.