M edi TOD : An E nglish Dialogue Dataset for Medical History Taking with Comprehensive Annotations (2024)

Abstract: Medical task-oriented dialogue systems can assist doctors by collecting patient medical history, aiding in diagnosis, or guiding treatment selection, thereby reducing doctor burnout and expanding access to medical services. However, doctor-patient dialogue datasets are not readily available, primarily due to privacy regulations. Moreover, existing datasets lack comprehensive annotations involving medical slots and their different attributes, such as symptoms and their onset, progression, and severity. These comprehensive annotations are crucial for accurate diagnosis. Finally, most existing datasets are non-English, limiting their utility for the larger research community.In response, we introduce MediTOD, a new dataset of doctor-patient dialogues in English for the medical history-taking task. Collaborating with doctors, we devise a questionnaire-based labeling scheme tailored to the medical domain. Then, medical professionals create the dataset with high-quality comprehensive annotations, capturing medical slots and their attributes. We establish benchmarks in supervised and few-shot settings on MediTOD for natural language understanding, policy learning, and natural language generation subtasks, evaluating models from both TOD and biomedical domains. We make MediTOD publicly available for future research.

## introduction medical task-oriented dialogue (tod) systems are gaining importance in modern healthcare by assisting doctors in patient history-taking, diagnosis suggestions, and treatment recommendations, alleviating doctor burnout and extending the reach of medical services @xcite @xcite . recently, medical tod systems have witnessed significant progress, particularly in individual submodules such as natural language understanding (nlu) @xcite , policy learning (pol) @xcite , and natural language generation (nlg) @xcite . most of the existing medical tod datasets contain annotations required for training only one sub-component @xcite @xcite , and only a few include annotations for all sub-components, thereby enabling the construction of a complete dialogue system @xcite . training nlu, pol, and nlg sub-modules requires dialogues to be annotated with intents, slots, dialogue states, and actions. in existing medical tod datasets, slots are primarily represented as key-value pairs. however, this simplistic representation often fails to capture the inherent complexity of the medical domain. for example, in figure 1 , the patient expresses two symptoms (pharyngitis and fever) along with its onset (past four days and last two days). existing annotation schemes would fail to capture the symptom-onset link and only represent four independent key-value pairs. to overcome this problem, we define a new slot schema, named comprehensive medical attribute schema (cmas), that captures the inherent complexity of the slots in the medical domain. it maintains multiple attributes specific to each slot type for better representation. for example in figure 1 , 'onset' is treated as an attribute of the slot 'symptom', establishing a more accurate patient profile. this paper presents meditod, the first publicly available english medical tod dataset annotated in the cmas ontology. meditod comprises dialogues from staged doctor-patient interviews in objective structured clinical examination format @xcite . by leveraging these high-quality dialogues, privacy concerns are mitigated while providing realistic medical scenarios. collaborating closely with doctors, we develop a questionnaire-based annotation framework to collect slots and corresponding attributes. the annotations are further canonicalized, where possible, to precise medical concepts in unified medical language system (umls). through the release of meditod, we aim to provide a valuable resource for advancing research in medical tod systems. our main contributions are as follows. with pre-trained language models (lms), recent approaches showcase remarkable performance on all three tod sub-tasks. @xcite , @xcite achieve state-of-theart performance for understanding user's requirements (nlu). @xcite , @xcite , and @xcite showcase similar trends for system action prediction and response generation tasks. in line with this trend, we benchmark pretrained language models flan-t5 @xcite , biogpt @xcite and pptod @xcite on meditod dataset. medical dialogue systems: many such datasets exist; however, only a few (see notably, remedi and code-mixed are the only datasets that collect low-level attributes in their labels. however, meditod differs from them in several ways. first, while these datasets capture the attributes, they do not link them to appropriate slots. for example, remedi would label the onset in a patient's utterance as (time, onset, past two days) without linking it to the symptom, fever. in contrast, meditod uses cmas to record slots and attributes together through a questionnaire-based annotation framework. second, meditod has canonicalized values for medical labels, such as symptoms and diseases, to ensure meaningful evaluation and to support future research. ## the meditod dataset to advance research in medical dialogue systems, datasets which capture canonicalized, comprehensive annotations must be available publicly. however, existing datasets (listed in to address these gaps, we curate meditod, an english dataset of doctor-patient dialogues for collecting patient medical histories. first, we form the dialogues in meditod by collecting publicly available transcripts of doctor-patient encounters @xcite . to capture the complexity of the slots, we define a comprehensive medical attribute schema (cmas) and develop a questionnaire-based labeling framework to annotate dialogues based on cmas. for each slot type (e.g., symptom, personal medical history), medical professionals vet questions to capture associated attributes (severity, onset, etc.). utilizing this framework, professional annotators then label utterances by answering the questionnaire corresponding to each slot under the doctor's supervision. finally, we canonicalize the slots values to standard medical concepts in the umls vocabulary. 1 ## experimental setup meditod supports all three subtasks in a tod system -natural language understanding (nlu), policy learning (pol), and natural language generation (nlg). nlu involves understanding the information presented in the latest patient utterance. specifically, given the dialogue so far, an nlu model predicts the intent, active slots, and their attributes of the patient utterance. this updates the dialogue state, i.e., the aggregate information the patient reports up to the current turn in the dialogue. pol requires predicting the doctor's next action. an action consists of intents, slots, and associated attributes. a pol model inputs the dialogue history and the dialogue state and predicts the action. finally, nlg involves transforming the doctor's action into natural language. an nlg model predicts the doctor's utterance based on the dialogue history and the action from the pol model. ## results we begin by evaluating the models for the nlu, pol, and nlg tasks on the in-domain test set. next, we assess the top-performing models from the in-domain evaluation on the out-of-domain dataset. ## conclusion and future works in this work, we introduced meditod, a novel english dataset of doctor-patient dialogues for collecting patient medical history. unlike existing medical datasets, meditod uses a novel schema (cmas) to capture attributes such as the onset and duration of symptoms relevant for downstream diagnosis. further, we link values for medical attributes in meditod labels to their precise medical concepts within umls vocabularies. finally, we propose new benchmarks for nlu, pol, and nlg tasks in the medical dialogue domain. our initial experiments with baseline models reveal the challenges inherent in these tasks and underscore the potential for improvement. furthermore, meditod facilitates the exploration of additional research settings such as knowledge grounded tod. canonicalization in meditod allows for seamless integration of umls vocabularies and semantic networks into tod settings, potentially enhancing performance. meditod also opens doors to research in medical dialogue summarization, offering opportunities to distill complex dialogues into concise medical summaries. our annotation portal can aid in creating large-scale medical dialogue datasets for medical specialties beyond pulmonology and musculoskeletal. this expansion will broaden the applicability and relevance of meditod in medical dialogue research. we release the meditod resources at https://github.com/dair-iitd/meditod .