On the Generation of Medical Dialogs for COVID -19 (2021)

Abstract: Under the pandemic of COVID-19, people experiencing COVID19-related symptoms have a pressing need to consult doctors. Because of the shortage of medical professionals, many people cannot receive online consultations timely. To address this problem, we aim to develop a medical dialog system that can provide COVID19-related consultations. We collected two dialog datasets – CovidDialog – (in English and Chinese respectively) containing conversations between doctors and patients about COVID-19. While the largest of their kind, these two datasets are still relatively small compared with general-domain dialog datasets. Training complex dialog generation models on small datasets bears high risk of overfitting. To alleviate overfitting, we develop a multi-task learning approach, which regularizes the data-deficient dialog generation task with a masked token prediction task. Experiments on the CovidDialog datasets demonstrate the effectiveness of our approach. We perform both human evaluation and automatic evaluation of dialogs generated by our method. Results show that the generated responses are promising in being doctor-like, relevant to conversation history, clinically informative and correct. The code and the data are available at https://github.com/UCSD-AI4H/COVID-Dialogue .

## introduction during the covid-19 pandemic, people who are experiencing symptoms similar to those of covid-19 or were exposed to risk factors have a pressing need to consult doctors. however, medical professionals are highly occupied, who do not have enough bandwidth to provide covid19-related consultations. to address this issue, we aim to develop a covid19-targeted dialog system. we build two medical dialog datasets that contain conversations between doctors and patients, about covid-19 and other pneumonia: (1) an english dataset containing 603 consultations, 1232 utterances, and 90664 tokens (english words); (2) a chinese dataset containing 1088 consultations, 9494 utterances, and 406550 tokens (chinese characters). while the largest of their kind, these two datasets are still relatively small compared with generaldomain dialog datasets. training complex dialog generation models on small datasets bears high risk of overfitting. to alleviate overfitting in covid-19 dialog generation, we develop a multi-task learning approach where a masked-token prediction (mtp) @xcite task is used to regularize the training of dialog generation models. our method performs the mtp task and the dialog generation task simultaneously. the mtp loss serves as a regularization term and is optimized jointly with the dialog generation loss. due to the presence of the mtp task, the dialog generation model is less likely to be biased to the dialog generation task defined on the small-sized training data. we perform experiments on our collected two covid-19 dialog datasets, where the results demonstrate the effectiveness of our approach. we perform human evaluation and automatic evaluation of dialogs generated by our approach. the results show that the generated responses demonstrate high potential to be doctor-like, relevant to patient history, clinically informative and correct. the major contributions of this paper are: ## related works several works have studied data-driven medical dialog generation. @xcite proposed a taskoriented dialog system to make medical diagnosis automatically based on reinforcement learning. the system converses with patients to collect additional symptoms beyond their self-reports. @xcite proposed a knowledge-routed relational dialog system that incorporates medical knowledge graph into topic transition in dialog management. @xcite an automatic diagnosis dialog system based on reinforcement learning. in these works, the neural models are trained from scratch on small-sized medical dialog datasets, which are prone to overfitting. ## datasets we collected two dialog datasets -coviddialog-english and coviddialog-chinese -which contain medical conversations between patients and doctors about covid-19 and other related pneumonia. the statistics of these two datasets are summarized in the english dataset the coviddialog-english dataset contains 603 english consultations about covid-19 and other related pneumonia, having 1,232 utterances. the number of tokens (english words) is 90,664. the average, maximum, and minimum number of utterances in a conversation is 2.0, 17, and 2 respectively. the average, maximum, and minimum number of tokens in an utterance is 49.8, 339, and 2 respectively. the conversations are from 582 patients and 117 doctors. each consultation starts with a short description of the medical conditions of a patient, followed by the conversation between the patient and a doctor. the chinese dataset the coviddialog-chinese dataset contains 1,088 chinese consultations about covid-19 and other related pneumonia, having 9,494 utterances. in this work, we develop models directly on chinese characters without performing word segmentation. each chinese character in the text is treated as a token. the total number of tokens in the dataset is 406,550. the average, maximum, and minimum number of utterances in a conversation is 8.7, 116, and 2 respectively. the average, maximum, and minimum number of tokens in an utterance is 42.8, 2001, and 1 respectively. the conversations are from 935 patients and 352 doctors. each consultation consists of three this description is used as the first utterance from the patient. for both datasets, the dialogs are crawled from openly accessible medical websites whose owners make these dialogs visible to the public. the patients' personal information is de-identified by owners of these websites. we further checked the crawled dialogs to ensure they do not contain private information of patients. besides, we also manually removed borderline sensitive information, such as specific dates and destinations in patients' travel histories. experts in privacy and security domains helped to check the final version of shared data and ensured there is no breach of patient privacy or confidentiality. ## method given a dialog containing a sequence of alternating utterances between patient and doctor, we process it into a set of pairs {(s i , t i )} where the target t i is a response from the doctor and the source s i is the conversation history -the concatenation of all utterances (from both patient and doctor) before t i . a dialog generation model takes s as input and generates t. the model consists of an encoder which encodes s and a decoder which takes the encoding of s as input and generates t. the size of the coviddialog datasets is small. training neural dialog models on these small datasets has high risk of overfitting. the prediction is conducted using a dialog decoder. the other task is masked-token prediction (mtp). in mtp, some percentage of the input tokens are masked at random. the text with masked tokens is fed into the text encoder which learns a latent representation for each token including the masked ones. the task is to predict these masked tokens by feeding the final hidden vectors (produced by the encoder) of the masked tokens into an output softmax operation over the vocabulary. the loss of the mtp task serves as a data-dependent regularizer of the encoder to prevent the encoder from overfitting to the data-deficient dialog generation task. formally, the method solves the following optimization problem: we compare with the following baselines: transformer @xcite , gpt-2 @xcite , unregularized bart @xcite , unregularized bert-gpt @xcite , and task adaptive pretraining (tapt) @xcite . ## conclusions in this paper, we make the first attempt to develop dialog generation models about covid-19. we first collected two datasets -coviddialogs -which contain medical conversations between patients and doctors about covid-19. to alleviate the risk of overfitting, we develop a multi-task learning approach, which uses a masked-token prediction task to regularize the dialog generation model. human evaluation and automatic evaluation results demonstrate the effectiveness of our proposed method in alleviating overfitting and generating clinically meaningful and linguistically high-quality dialogs about covid-19.