CLIP : A Dataset for Extracting Action Items for Physicians from Hospital Discharge Notes (2021)

Abstract: Continuity of care is crucial to ensuring positive health outcomes for patients discharged from an inpatient hospital setting, and improved information sharing can help. To share information, caregivers write discharge notes containing action items to share with patients and their future caregivers, but these action items are easily lost due to the lengthiness of the documents. In this work, we describe our creation of a dataset of clinical action items annotated over MIMIC-III, the largest publicly available dataset of real clinical notes. This dataset, which we call CLIP, is annotated by physicians and covers 718 documents representing 100K sentences. We describe the task of extracting the action items from these documents as multi-aspect extractive summarization, with each aspect representing a type of action to be taken. We evaluate several machine learning models on this task, and show that the best models exploit in-domain language model pre-training on 59K unannotated documents, and incorporate context from neighboring sentences. We also propose an approach to pre-training data selection that allows us to explore the trade-off between size and domain-specificity of pre-training datasets for this task.

## introduction transitioning patient care from hospitals to primary care providers (pcps) can frequently result in medical errors @xcite . when patients are discharged, they often require further actions to be taken by their pcp, who manages their longterm health, such as reviewing results for lab tests once they are available @xcite . yet pcps often have many patients and little time to review new clinical documents related to a recent hospital stay @xcite , so making this review fast, actionable, and accurate will be beneficial. discharge notes are typically lengthy @xcite and written as free text, so pcps may fail to identify important pending actions, which inadvertently leads patients to poor outcomes. @xcite found that pcps considered the lack of a standardized follow-up section to be a key driver in missing follow-up action items. while discharge notes may include follow-up sections, they are typically aimed at the patient and not curated for pcp use. @xcite found that following up on pending clinical actions is critical for minimizing risk of medical error during care transitions, especially for patients with complex treatment plans. automatic extraction of action items can make physicians more efficient by reducing the high cognitive load and time-consuming burden of using electronic health records @xcite @xcite . to our knowledge, there has been little previous work using machine learning to address this important clinical problem. potential impact successful automatic extraction of action items can have several direct benefits. first, it can improve patient safety by fostering more comprehensive and complete care by pcps. second, it might make physicians more efficient at performing a comprehensive review of action items, which is critical as physicians spend an increasing amount of time interacting with electronic health record (ehr) systems @xcite . further, reviewing and synthesizing lengthy or complicated patient histories places a significant cognitive load on physicians, which has been associated with increased medical error @xcite , so reducing this cognitive load is an area of opportunity. finally, a working system might integrate with ehrs to au- appointment appointments to be made by the pcp, or monitored to ensure the patient attends them. the patient requires a neurology consult at xyz for evaluation. ## related work and datasets clinical information extraction there has been a wealth of previous work on extracting information from clinical notes, much of which also follows an extractive summarization approach. for example, @xcite extracts items such as patient smoking status and obesity comorbidities from discharge notes. @xcite created a hybrid system of regex-based heuristics, neural network models trained on pre-existing datasets, and models such as support vector machines for disease-specific extractive summarization. @xcite developed a pseudo-labelling, semi-supervised approach, using intrinsic correlation between notes, to train extractive summarization models for disease-specific summaries. we differ from these efforts in that we do not aim to generate general-purpose or disease-specific summaries, rather we focus on extracting specific action items from discharge notes to facilitate care transfer. clinical datasets datasets and challenges on the extraction of medication, tests, and procedure mentions in clinical text @xcite @xcite have been released, but without the focus on providing actionable insight to pcps. additionally, multiple datasets @xcite have been introduced for detecting temporal and co-reference relations between parts of a note. while it may be useful for a model to have a good grasp of co-reference and temporal dependencies to understand what constitutes actionable information for a pcp, we choose to optimize directly for the end task, noting recent work demonstrating that modern pre-trained neural networks will identify and exploit such information as needed @xcite . although on different tasks, we note that our dataset of 718 annotated documents is larger than recently released datasets, such as those from the n2c2 shared tasks. for comparison, 500 documents were annotated for adverse drug event extraction @xcite , 150 documents for family history extraction @xcite , and 100 documents for clinical concept normalization @xcite . one of the largest annotated clinical datasets, emrqa, is built on 2,425 clinical notes @xcite . summarization prior summarization work, which we build on, uses pre-trained transformer models to construct sentence representations that are contextualized with the entire document @xcite . @xcite evaluate on three benchmark summarization datasets consisting of news articles. they are shorter, with average document lengths from 400-800 words, whereas mimic-iii discharge notes average over 1,400 words. @xcite evaluate with rouge scores standard in summarization, whereas we take advantage of having ground truth extracted sentences and evaluate with classification metrics, providing a substantially different task. @xcite develop a disease-specific summary dataset, but it is not public and their methods involve combining a mix of outputs from models performing auxiliary tasks such as concept recognition, adverse drug event extraction, and medication change detection, each of which have to be individually developed and maintained. ## clip dataset in this section, we describe the process of creating our clip dataset, short for clinicalfollow-up, and report statistics on the dataset. ## learning to extract action items with a discharge note as input, the task is to output the clinically actionable follow-up items found ## results the main set of results are reported in we next investigate results on each label (see ## discussion our results show that the common regime of finetuning a large pre-trained model is a useful method for our task of extracting clinical action items. additionally, we investigated the trade-off between task-specificity and pre-training data size, and found our task-targeted pre-training method enables one to navigate this trade-off, producing models with comparable performance on the end task that require less data for pre-training. while trading off these concerns may not be needed if effective public models exist for a given task, we believe this technique is useful in scenarios in which users have large, domain-specific, private datasets and specific tasks in mind. this is often the case for healthcare institutions and developers of clinical machine learning software, as privacy concerns tend to preclude data sharing between institutions. from a modeling perspective, there are many possible avenues for future work. taking a structured prediction lens and leveraging sentence-level label dependencies or applying structured prediction models could be helpful, although @xcite note that crf layers did not improve their performance for a sequential sentence classification task. we acknowledge that our sentence classification approach is a simplification of the more general span detection problem, and this approach could bring improved precision by focusing on which parts of sentences matter, which may be important as we found that sentence tokenization was non-trivial for clinical notes. finally, the question of whether such an ap-proach to follow-up workflow augmentation is successful in increasing patient safety, clinician efficiency, or ehr usability is an empirical one. we hope to evaluate in the future whether a highlighted note such as one these models could provide will reduce the time a physician takes to, for example, answer certain questions about a patient's hospital stay. in alignment with recent calls for increased rigor in the evaluation of machine learningderived clinical decision support systems @xcite , future work should include further prospective, controlled evaluation of the generalizability, stability, interpretability, unbiasedness, usability, and efficacy of this approach. we hope that our dataset and initial model development can lay the groundwork for future investigation. ## conclusion we introduce the task of detecting clinical action items from discharge notes to help primary care physicians more quickly and comprehensively identify actionable information, and present the clip dataset, which we will release to the community. given perfect performance, this would reduce the number of sentences a pcp may need to read by 88%. the best model's binary f1 is near 0.9, compared to the human benchmark of 0.93. these models could additionally be used for clinical research. for example, a calibrated model could derive statistics for how often each type of action item is seen for different patient populations, which can provide insight into typical patient or pcp burden after hospital discharge. we evaluated bert-based models that incorporate multi-sentence context, and introduced a novel task-targeted pre-training approach that can reduce pre-training time while maintaining similar performance to models pre-trained on much larger in-domain datasets. the models have promising results, however we anticipate there is still room for improvement, particularly for the rare labels. we encourage the clinical nlp community to further investigate the problem of detecting action items from hospital discharge notes, which can help improve reliably safe transitions of care for the most vulnerable patients.