W is P er M ed at “Discharge Me!”: Advancing Text Generation in Healthcare with Large Language Models, Dynamic Expert Selection, and Priming Techniques on MIMIC - IV (2024)

Abstract: This study aims to leverage state of the art language models to automate generating the “Brief Hospital Course” and “Discharge Instructions” sections of Discharge Summaries from the MIMIC-IV dataset, reducing clinicians’ administrative workload. We investigate how automation can improve documentation accuracy, alleviate clinician burnout, and enhance operational efficacy in healthcare facilities. This research was conducted within our participation in the Shared Task Discharge Me! at BioNLP @ ACL 2024. Various strategies were employed, including Few-Shot learning, instruction tuning, and Dynamic Expert Selection (DES), to develop models capable of generating the required text sections. Utilizing an additional clinical domain-specific dataset demonstrated substantial potential to enhance clinical language processing. The DES method, which optimizes the selection of text outputs from multiple predictions, proved to be especially effective. It achieved the highest overall score of 0.332 in the competition, surpassing single-model outputs. This finding suggests that advanced deep learning methods in combination with DES can effectively automate parts of electronic health record documentation. These advancements could enhance patient care by freeing clinician time for patient interactions. The integration of text selection strategies represents a promising avenue for further research.

## introduction clinical notes in electronic health records (ehrs) are used by clinicians to document patient progress in free-text format. these notes typically include the patient's experiences, symptoms, findings, diagnoses, and details of procedures and interventions performed. they serve as the foundation for discharge summaries (ds), which contain a section with concise overviews of the entire hospital encounter known as brief hospital course (bhc) @xcite . they are embedded in the ds and are written by senior physicians who are responsible for the patient's overall care. in addition to bhc, ds also includes discharge instructions (di), which are detailed guidelines provided to patients regarding their post-hospital care. these instructions cover the patient's ongoing care, such as medication instructions, follow-up appointments, and any necessary lifestyle adjustments to ensure proper recovery. discharge instructions are designed to facilitate a smooth transition from hospital care to home care and to prevent readmissions. writing such summaries (bhc) and instructions (di) can be time-consuming and tedious. consequently, physicians often spend a big portion of their clinical day dedicated to ehr documentation and desk work @xcite . this paper presents wispermed's contribution to the shared task discharge me! @xcite , which is part of bionlp @ acl 2024. this shared task aims to ease the administrative burden on clinicians by developing automated methods to generate critical sections in ds, specifically the "brief hospital course" and "discharge instruction". automating the creation of these sections has the potential to improve documentation accuracy, reduce clinician burnout, and ultimately optimize the processes in healthcare facilities @xcite by allowing clinicians to allocate more time toward direct patient care. our work focuses on designing and implementing various innovative approaches to overcome this challenge and contribute to the overall goals of the shared task. ## dataset the dataset @xcite provided for this shared task utilizes the mimic-iv (medical information mart for intensive care) database @xcite . mimic-iv is a publicly available database sourced from the ehr of the beth israel deaconess medical center and is accessible on physionet @xcite . the task dataset is divided into four subsets: a training set consisting of 68,785 samples, a validation set containing 14,719 samples, a phase i testing set with 14,702 samples, and a phase ii testing set comprising 10,962 samples. each subset includes ds that are organized into various sections. all records contain two mandatory sections: "brief hospital course" and "discharge instructions". the bhc section typically provides an overview of the patient's treatment and progress during their hospital stay and precedes the di section. these di summarize post-hospitalization care instructions and are positioned at the conclusion of the summary. the challenge organizers provided a regular expression (regex) query to extract these two sections from the ds. the regex query ensures that the relevant information is accurately identified and separated from the rest of the ds content. for the remainder of this paper, any reference to the "discharge summary" (ds) will exclude the target sections, bhc or di. ## evaluation the submissions to the shared task were evaluated using eight metrics, which assess the relevance and factuality of the generated target. these metrics include bilingual evaluation understudy (bleu-4) @xcite , recall-oriented understudy for gisting evaluation (rouge-1, rouge-2, rouge-l) @xcite , bertscore @xcite , metric for evaluation of translation with explicit ordering (meteor) @xcite , alignscore @xcite , and medical concept (medcon) @xcite . the overall score was calculated by averaging the scores across these eight metrics. in addition to these evaluation metrics, readability scoring metrics were also investigated and utilized in some of the developed approaches. after the conclusion of the competition, submissions from the highest-performing teams, determined by the overall score, were evaluated by a panel of clinicians foot_0 . the generated sections were assessed based on their completeness, correctness, readability, and overall comparison to the reference text. these criteria were evaluated on a scale ranging from 1 to 5, where 1 signifies performance that is considerably worse than the reference text, and 5 indicates performance that is considerably better than the reference text. three independent clinicians scored 25 di and 25 bhc texts from each team, using the same ds. ## methods this section describes different approaches to the shared task. licenses for the used models, frameworks, and additional datasets can be found in appendix e. ## results this section describes results of the evaluation of the developed models using the metrics described in section 3. the evaluation done by clinicians can be seen in ## discussion despite being one of the less robust models evaluated, wizardlm-2 exceeded the established baseline, showing its effectiveness in a few-shot learning context. with minimal training examples, the model still produced high-quality texts, according to the metrics, highlighting the potential of few-shot learning in enhancing performance metrics. the performance of instruction-tuned models revealed mixed outcomes. despite being a specialized adaptation of the llama-3-70b-i model tailored for medical contexts, openbiollm-70b underperformed in relation to its base model. this behavior was unexpected, considering its design to enhance relevance and accuracy in clinical applications. conversely, the mistral-7b-i-v0.2 model demonstrated impressive capabilities, outperforming both the larger openbiollm-70b and the llama-8b-i models. this highlights the effectiveness of mistral-7b-i-v0.2 in handling complex medical text generation and summarization tasks despite its smaller size. in contrast to its reputation as one of the most promising state-of-the-art open-source llms, the llama-3 models have been found to be less effective in this challenge. this is on par with the findings from lmsys chatbot arena @xcite where llama-3 models showed the weakest performance compared to other state-of-the-art models on the task of summarization @xcite . using the asclepius dataset for priming substantially improved model performance during the fine-tuning phases. for instance, the llama-8b-i model's score rose from 0.253 to 0.302, and the mistral model's performance increased from 0.289 to 0.307. notably, the mistral-7b-i-v0.2 + asclepius model was the top performer in the challenge, aside from des approaches. this underscores the benefits of further training models with specialized datasets to enhance accuracy and relevance in domain-specific tasks. the reordering of sections within the mimic-sid approach moderately enhanced overall model performance, demonstrating that prioritizing the most relevant sections can be beneficial. however, it is important to note that metrics sensitive to text order, such as meteor and alignscore, experienced a decline. this suggests that while reordering can improve general outcomes by emphasizing key information, it may simultaneously compromise the sequential integrity of the text. therefore, this strategy confirms the utility of structurally optimizing input for task-specific relevance, albeit with some trade-offs in textual coherence. exploration of hyperparameter settings revealed that more complex configurations did not yield superior results. the basic approach, utilizing @xmath0, @xmath1.6, and @xmath2.9, consistently outperformed other tested configurations, including those with greedy decoding and encoder repetition penalties. this emphasizes the efficacy of maintaining simpler hyperparameter settings for stable and high-quality text generation. additional complexity in parameter tuning did not always correlate with improved model performance. the des that relied on the pre-calculated scores had varying effects on the metrics evaluated. using medcon and meteor in combination with cli improved the results, whereas choosing the correlation as weights resulted in no improvement. a possible reason might be that the pre-calculated scores were only calculated on the entire ds and the manual evaluation seen in ## conclusion the research identified several opportunities for future investigation that may enhance the performance and utility of the discussed models. initially, due to the extensive size of the training dataset and the constraints imposed by the context length of input texts, each model was trained for a maximum of only three epochs. therefore, extending the training duration may provide improvements and merits further exploration. moreover, alterations to inference parameters have demonstrated notable effects on model outputs. for example, employing the erp parameter, while maintaining other settings constant resulted in a degradation of performance metrics (from 0.253 to 0.238 overall score). this suggests a systematic evaluation of inference parameters could further enhance model output. additionally, priming the model has substantially improved results. investigating additional datasets for priming purposes could further optimize model performance and expand its applica-bility across diverse textual tasks. this could be a promising direction for future research efforts. further opportunities lie in optimizing section reordering to balance task-specific relevance while maintaining text coherence. the winning approach, evaluated by the automatic and clinicans' evaluation, a des, achieved the highest overall score. this suggests that generating multiple outputs and developing methodologies to select the optimal text may further improve performance. therefore, exploring various des techniques and selection criteria is a field for further research. the clinicians' evaluation added valuable insights, and the automatic scores demonstrated high robustness and strong alignment with the manual assessments. this alignment indicates that the manual evaluation, even on a small subset, effectively validates the reliability of the generated texts. lastly, efforts to enhance the quality of medical machine learning algorithms are ongoing, along with a responsibility to report the environmental impact of the research. in this study, the total energy consumption for training and inference is estimated with 1,552.10 kwh, resulting in 591.35 kg co 2 emission. detailed information is provided in appendix d.