U o G Siephers at “Discharge Me!”: Exploring Ways to Generate Synthetic Patient Notes From Multi-Part Electronic Health Records (2024)

Abstract: This paper presents the UoG Siephers team participation at the Discharge Me! Shared Task on Streamlining Discharge Documentation. For our participation, we investigate appropriately selecting and encoding specific sections of Electronic Health Records (EHR) as input data for sequence-to-sequence models, to generate the discharge instructions and brief hospital course sections of a patient’s EHR. We found that, despite the large volume of disparate information that is often available in EHRs, selectively choosing an appropriate EHR section for training and prompting sequence-to-sequence models resulted in improved generative quality. In particular, we found that using only the history of present illness section of an EHR as input often led to better performance than using multiple EHR sections.

## introduction in the clinical domain, writing notes about patients' health, diagnoses, and treatments is a necessary part of the patient healthcare journey, but it is also time consuming @xcite . the time spent by essential care staff, such as doctors and nurses, writing the notes in electronic health records (ehrs) could be time better spent performing important clinical duties. the discharge me! bionlp acl'24 shared task on streamlining discharge documentation challenged participants to produce a system that can automatically generate: discharge instructions, which contain detailed guidelines provided to a patient upon their discharge from hospital; and brief hospital courses, which summarise the key events, treatments and progress for a patient during their hospital stay @xcite . discharge me! participants were provided a dataset curated from the mimic-iv database @xcite , which contains de-identified patients' ehrs. ehrs are complex collections of, often long and disparate, reports about a patient's stay in hospital, including reports on patient demographics, medical history, laboratory tests and results, instructions for the patient and many more sections. in this work, we investigate several ways of appropriately selecting and encoding specific sections of ehrs as input data for sequence-to-sequence (seq2seq) models to generate the two target sections of the discharge me! shared task, i.e., the discharge instruction and the brief hospital course. in particular, in this work we investigate the following three research questions that guide our experimentation: rq1: what is the effect of using different sections of ehrs as training data for seq2seq models? rq2: can a model that uses multiple ehr sections as input achieve better performance than models trained on single sections of ehrs? rq3: when concatenating multiple ehr sections as input, is it better to concatenate lexically, or concatenate embeddings post-encoding? ## related work most relevant to our work is that of @xcite , who employed various encoderdecoder models with different pre-trained checkpoints @xcite to generate a brief hospital course. hartman and campion attempted to summarise ehr records as short day-by-day summaries so that the ehr summaries would fit within the context limit of seq2seq encoder-decoder models. in our work, instead of summarising the input data to fit the context limit of an encoder-decoder model, we experiment with selectively choosing individual subsections of the ehr records to train seq2seq models. pal et al. @xcite used the nursing report section of ehrs to generate a variety of ehr sections, such as the history of present illness and discharge instructions. the authors showed that seq2seq models, such as t5 and bart, can be effective for this task @xcite . differently from pal et al., we explore using multiple sections of the ehr as input data, and ways to combine ehr sections as input. finally, the work of @xcite used the discharge instructions of historic patients, who had similar symptoms to a new patient, to write the new patient's discharge instruction. they used graphbased reasoning to generate the new discharge instruction based on the historic patients' instructions. differently, we focus on using information that is entirely available in the patients own record and do not rely on the information of other patients. ## methods in this section we describe our approaches for generating discharge instruction and brief hospital course sections of ehrs. using different pre-trained encoder and/or decoder models within seq2seq models has been shown to be an effective way to adapt such models for different tasks @xcite . therefore, in this work, we investigate three approaches for constructing the input data for seq2seq models, such that we can use the models' limited context effectively to model the ehrs sections. for each of our approaches, we deploy encoder-decoder models following the work of @xcite . ## experimental setup in this section, we present the experimental setup to investigate the three research questions that we presented in section 1. ## results this section describes our findings relating to the research questions presented in section 1. concerning rq1, the t5 and roberta-gpt2 models perform better when trained on the hpi input. on the other hand, the bert-gpt2 model and clinicalbert-gpt2 model perform better when trained with rr input. however, the performance increases that are obtained from training on hpi data are notably greater than any performance improvements that are obtained from training on rr data. the t5(hpi) model shows 141% improvement in overall score compared to the t5(rr) model, whereas the bert-gpt2(hpi) model resulted in only a 7% overall score drop compared to its bert-gpt2(rr) counterpart. similarly, roberta-gpt2(hpi) achieves a 29% improvement over roberta-gpt2(rr), while there is only a 4% drop in overall score between clinicalbert-gpt2(rr) and clinicalbert-gpt2(hpi). answering rq1, we find that when training on individual record sections, training on hpi most often leads to better performance compared to models trained on rr. indeed, the gains in overall score from training on hpi compared to rr are notably greater. in answering rq2, we find that training seq2seq models on multiple concatenated sections of ehr with models does not outperform models trained on a single input section of a record. our best performing concatenation model, longformer-gpt-2, outperforms several models trained on single ehr sections. however, both bert-gpt2(rr) and t5(hpi) both outperform the longformer-gpt2 model. this indicates that choosing a single input section for the right model can outperform a model that has access to both sections of the data. specifically, we see that the longformer-gpt2 outperforms the bert-gpt2(rr) model by a small margin in bertscore and meteor. however, the two models perform very similarly, indicating that the additional information in hpi that was available to the longformer-gpt2 model did not improve its performance markedly. thus, training on additional information is not always beneficial. regarding rq3, we find that concatenating the different sections of ehr records lexically, and then using an encoder with a larger context window is a more effective method for this task than encoding the different sections separately as proposed in section 3.3. neither of our fusion models beat the longformer-gpt2 model in overall score, despite fusion-roberta-gpt2-large using a larger decoder model. considering this in the context of the findings of our first and second research question, this may indicate that the decoder model is not able to utilise the separately embedded records sections as effectively as it is able to understand embeddings of a single section of the report. replacing the decoder with a larger model does improve performance, but still the performance is worse than a t5-base model trained on the single hpi section. ## qualitative analysis in this section we investigate the generated record quality for the best performing seq2seq model, t5(hpi). we analyse the ten highest and ten lowest scoring generated discharge instructions and brief hospital course, in terms of their rouge-1 scores. in the highest scoring generated ehr sections, the core ailments of the patients are correctly described. in the generated discharge instructions, the recommended followup treatment is often inaccurate but the structure of the instructions, which all contain subheadings (e.g. "why you were in hospital", "what you should do after leaving"), are usually correctly generated and match the target texts. this improves the overall quality of the generated discharge instructions. the generated brief hospital courses match most of the text of their corresponding target texts exactly. however, they deviate towards the end of the text often adding extended information that is still topically relevant, but not actually part of the true target text. inspecting the lowest scoring generated samples, we find common problems with the generation process for both the discharge instructions and brief hospital courses. while our models are effective at writing structured discharge instructions with specific sub headings, and brief hospital courses that contain a verbose description of the patient's problems, the effectiveness of the generation degrades when the target texts are not in line with these formats. for example, when the target discharge instruction is a short single-line note, such as instructions about avoiding a certain kind of food, or a reminder for the patient to weigh themselves, the models attempts to generate a long instruction with many unnecessary subsections. similarly, our model will attempt to generate a verbose brief hospital course, even when the true target is a list of vital-sign readings. uniquely to the discharge instruction generation, we find that several of the target sections are written in spanish. in such cases, our model still attempts to generate english text, as the input section is always written in english. ## conclusion to conclude, we have found that training a seq2seq model to generate discharge instructions and brief hospital courses using single sections from electronic health records (ehr) as input, outperformed models trained using multiple sections of ehr as input. moreover, choosing which single section to use as input is an important factor that depends on the chosen seq2seq model and that generally, some sections can expect to provide reasonable performance overall compared to others.