Ixa- M ed at Discharge Me! Retrieval-Assisted Generation for Streamlining Discharge Documentation (2024)

Abstract: In this paper we present our system for the BioNLP ACL’24 “Discharge Me!” task on automating discharge summary section generation. Using Retrieval-Augmented Generation, we combine a Large Language Model (LLM) with external knowledge to guide the generation of the target sections. Our approach generates structured patient summaries from discharge notes using an instructed LLM, retrieves relevant “Brief Hospital Course” and “Discharge Instructions” examples via BM25 and SentenceBERT, and provides this context to a frozen LLM for generation. Our top system using SentenceBERT retrieval achieves an overall score of 0.183, outperforming zero-shot baselines. We analyze performance across different aspects, discussing limitations and future research directions.

## introduction generating detailed clinical notes in electronic health records (ehrs) is a time-consuming task that can lead to clinician burnout and operational inefficiencies in healthcare systems. the bionlp acl'24 shared task, "discharge me!" @xcite , aims to automate the generation of critical discharge summary sections using natural language processing (nlp). while large language models (llms) like @xcite and llama-3 (meta, 2024) have advanced nlp capabilities, they can produce hallucinations when encountering out-of-distribution queries @xcite . the retrieval-augmented generation (rag) framework aims to mitigate hallucinations in large language models (llms) by combining external knowledge retrieval with llm generation @xcite . a naive rag approach involves indexing data into vectors, retrieving relevant vectors for a given query, and providing the retrieved context to a frozen llm. how-ever, this naive implementation often suffers from limitations in retrieval precision, recall, and generation quality. notwithstanding, we evaluate the efficacy of a naive rag framework for the "discharge me!" task. section 3 describes our system methodology and presents our results. section 4 analyzes the limits of our approach and outlines prospective research areas for improvement. ## task description the bionlp acl'24 shared task, 'discharge me!", focuses on streamlining the clinical documentation process by automating the generation of two critical sections in discharge summaries: "brief hospital course" and "discharge instructions". by reducing the time and effort clinicians expend on writing these detailed notes in electronic health records (ehrs), we can alleviate administrative burden, minimize clinician burnout, and ultimately improve operational efficiencies and patient care quality. ## conclusion our work explored a retrieval-augmented generation approach for the "discharge me!" shared task on automating the generation of "brief hospital course" and "discharge instructions" sections. we grounded a large language model with structured patient summaries and retrieved relevant examples from the challenge training data set, aiming to mitigate hallucinations and enhance generation quality. while our grounded approach demonstrated potential in generating coherent summaries, several areas exist for performance improvement. finetuning the pipeline on "brief hospital course" and "discharge instructions" sections could better align generated text with domain-specific language patterns. incorporating constrained decoding or post-processing could improve n-gram overlap with references. optimizing retrieval for stylistic similarity could indirectly benefit n-gram metrics. moreover, metrics like medcon could be improved by retrieving unified medical language system (umls) concept-rich examples or integrating umls databases during retrieval/generation. exploring advanced rag architectures with iterative retrieval and multi-step reasoning could address naive rag limitations.