Characterizing the Value of Information in Medical Notes (2020)

Abstract: Machine learning models depend on the quality of input data. As electronic health records are widely adopted, the amount of data in health care is growing, along with complaints about the quality of medical notes. We use two prediction tasks, readmission prediction and in-hospital mortality prediction, to characterize the value of information in medical notes. We show that as a whole, medical notes only provide additional predictive power over structured information in readmission prediction. We further propose a probing framework to select parts of notes that enable more accurate predictions than using all notes, despite that the selected information leads to a distribution shift from the training data (“all notes”). Finally, we demonstrate that models trained on the selected valuable information achieve even better predictive performance, with only 6.8%of all the tokens for readmission prediction.

## introduction as electronic health records (ehrs) are widely adopted in health care, medicine is increasingly an information science @xcite @xcite : obtaining and analyzing information is critical for the diagnosis, prognosis, treatment, and prevention of disease. although ehrs may increase the accuracy of storing structured information (e.g., lab results), there are growing complaints about unstructured medical notes (henceforth "notes") @xcite @xcite . these complaints can be grouped into two perspectives: consumption and production. on the one hand, information overload poses a critical challenge on the consumption side. that is, the sheer amount of information makes it difficult to glean meaningful information from ehrs, including notes @xcite . on the other hand, from the perspective of production, for every hour spent on patient interaction, physicians have an added one-to-two hours finishing the progress notes and reviewing results among other things, without extra compensation @xcite . the additional work contributes to physician burnout, along with lowquality notes and even errors in the notes. consequently, physicians tend to directly copy large volumes of patient data into notes, but may fail to record information only available through interaction with patients. for instance, they may miss the wheezing breath for the diagnosis of the chronic obstructive pulmonary disease, or fail to have engaging conversations for evaluating signs of depression @xcite . while the nlp community has focused on alleviating the challenges in analyzing information (e.g., information overload), we argue that it is equally important to help caregivers obtain and record valuable information in the first place. we aim to take a first step towards this direction by characterizing the value of information in medical notes computationally. in this work, we define valuable information as information that is useful for evaluating medical conditions and making medical decisions. to do that, we first examine the value of notes as a whole conditioned on structured information. while narrative texts can potentially provide valuable information only accessible through physician-patient interaction, our analysis addresses the typical complaint that notes contain too many direct copies of structured information such as lab results. therefore, a natural question is whether notes provide additional predictive power for medical decisions beyond structured information. by systematically studying two critical tasks, readmission prediction and in-hospital mortality prediction, we demonstrate that notes are valuable for readmission predictions, but not useful for mortality prediction. our results differ from previous studies demonstrating the effectiveness of notes in mortality prediction, partly because @xcite use a limited set of structured variables and thus achieve limited predictive power with structured information alone. we then develop a probing framework to evaluate the prediction performance of parts of notes selected by value functions. we hypothesize that not all components of notes are equally valuable and some parts of notes can provide stronger predictive power than the whole. we find that discharge summaries are especially predictive for readmission, while nursing notes are most valuable for mortality prediction. furthermore, we leverage hypotheses from the medical literature to develop interpretable value functions to identify valuable sentences in notes. similarity with prior notes turns out to be powerful: a mix of most and least similar sentences provide better performance than using all notes, despite containing only a fraction of tokens. building on these findings, we finally demonstrate the power of valuable information beyond the probing framework. we show that classification models trained on the selected valuable information alone provide even better predictive power than using all notes. in other words, our interpretable value functions can effectively filter noisy information in notes and lead to better models. we hope that our work encourages future work in understanding the value of information and ultimately improving the quality of medical information obtained and recorded by caregivers, because information is after all created by people. ## our predictive framework we investigate the value of notes through a predictive framework. we consider two prediction tasks using mimic-iii: foot_0 readmission prediction and mortality prediction. for each task, we examine two questions: 1) does a model trained on both notes and structured information outperform the model with structured information alone? ( §3) 2) using a model trained on all notes, are there interpretable ways to identify parts of notes that are more valuable than all notes? ( §4) 2.1 an overview of mimic-iii mimic-iii is a freely available medical database of de-identified patient records. this dataset includes basic information about patients such as admission details and demographics, which allows us to identify outcomes of interest such as mortality and readmission. it also contains detailed information that characterizes the patients' health history at the hospital, known as events, including laboratory events, charting events, and medical notes. the data derived from these events are elicited while patients are in the hospital. our goal is to characterize the value of such elicited information, in particular, notes, through predictive experiments. next, we break down the information into two categories: structured vs. unstructured. ## finding needles in a haystack: probing for the valuable information the key goal of this work is to identify valuable components within notes, as we hypothesize that not all information in notes is valuable for medical decisions, as measured by the predictive power. to identify valuable components, we leverage an existing machine learning model (e.g., models in fig. 2 ) and hypothesize that the test performance is better if we only use the "valuable" components. formally, assume that we trained a model using all notes, f all . note that s i by definition entails a distribution shift from the data that f all is trained on (s i ), because s i is much shorter than s i . the challenge lies in developing interpretable ways to identify valuable content. we first compare the value of different types of notes in §4.1, which can be seen as trivial value functions based on type of note, and then propose interpretable value functions to zoom in on the content of notes ( §4.2). finally, we show that these valuable components not only provide accurate predictions with a model trained with all the notes, but also allow us to learn a model with better predictive power than that trained with all the notes ( §4.3). in other words, we can effectively remove the noise by focusing on the valuable components. ## conclusion our results confirm the value of medical notes, especially for readmission prediction. we further demonstrate that parts can outperform the whole. for instance, selected sentences from discharge summaries can better predict future readmission than using all notes and structured variables. our work can be viewed as the reverse direction of adversarial nlp @xcite : instead of generating triggers that fool nlp models, we identify valuable information in texts towards enabling humans to generate valuable texts. beyond confirming intuitions that "assessment and plan" in physician notes is valuable, our work highlights the importance of nursing notes. our results also suggest that a possible strategy to improve the value of medical notes is to help caregivers efficiently provide novel content while highlighting important prior information (mixed similarity). substantial future work is required to achieve the long-term goal of improving the notetaking process by nudging caregivers towards obtaining and recording valuable information. in general, the issue of effective information solicitation has been understudied by the nlp community. in addition to model advances, we need to develop human-centered approaches to collect data of better quality from people. as @xcite argued, "as medicine incorporates new technology, its focus should remain on interaction between the sick and healer." we hope that our study will encourage studies to understand the interaction process and the note-taking process, beyond understanding the resulting information as a given. after all, people are at the center of data.