MLBMIKABR at “Discharge Me!”: Concept Based Clinical Text Description Generation (2024)

Abstract: This paper presents a method called Concept Based Description Generation, aimed at creating summaries (Brief Hospital Course and Discharge Instructions) using source (Discharge and Radiology) texts. We propose a rule-based approach for segmenting both the source and target texts. In the target text, we not only segment the content but also identify the concept of each segment based on text patterns. Our methodology involves creating a combined summarized version of each text segment, extracting important information, and then fine-tuning a Large Language Model (LLM) to generate aspects. Subsequently, we fine-tune a new LLM using a specific aspect, the combined summary, and a list of all aspects to generate detailed descriptions for each task. This approach integrates segmentation, concept identification, summarization, and language modeling to achieve accurate and informative descriptions for medical documentation tasks. Due to lack to time, We could only train on 10000 training data.

## introduction the "discharge me!" @xcite task within the bionlp workshop at the annual meeting of the association for computational linguistics (acl) 2024 aims to automate the generation of "brief hospital course" and "discharge instructions" sections in discharge notes. these notes are derived from a subset of the mimic-iv-note @xcite and mimic-iv-ed @xcite datasets. hosted on codabench, the competition provides defined training, validation, and testing sets comprising 109,168 emergency department admissions. document statistics from the training data ( our approach focused on managing source data size and generating target output systematically to capture crucial information effectively. we aimed to condense lengthy documents while retaining essential details by using a rule-based segmentation method and appropriate prompts for extracting summaries without compromising important information. this strategy allowed us to compress large documents while preserving necessary data for target text generation. we aim to generate the target text in a structured format by creating summaries that describe specific topics related to the task. these topics are referred to as "concepts". a concept can encompass various subjects such as patient instructions, medication details, disease information, etc. an example of such a concept is illustrated in appx. a.1. hence, next part of our approach unfolds in two phases. first, we predict the concepts relevant to summarized text. then, leveraging these concepts, we generate descriptions from the same input text. this process allows us to tailor responses effectively, as the concepts are inherently task-specific, enhancing the accuracy and relevance of our generated content. hepos by @xcite introduces an innovative encoder-decoder attention mechanism for scalable long document summarization, processing ten times more tokens than traditional models. @xcite developed the semantic self-segmentation approach to overcome memory limitations in transformer architectures, particularly beneficial in law domains. @xcite al. (2021 ) proposed a hierarchical propagation layer to enhance reasoning in long document summarization. pang et al. (2023) suggested a hierarchical inference framework improving summarization models' performance on lengthy texts. koh et al. (2022) conducted a survey for evaluating research progress and future directions in long document summarization. efforts focus on training large language models (llms) with medical data, like mimic-iii @xcite . asclepius-r @xcite is a specialized clinical large language model trained on synthetic clinical notes created from publicly available case reports, designed to handle patients' clinical notes while addressing privacy and accessibility challenges. biomistral @xcite is tailored for biomedical text, showing superior performance in question-answering and across languages, supporting research in healthcare. ## methods our pipeline, outlined in figure 1 , consists of four stages and uses three of the six provided data files. the "text" fields from the "radiology" and "discharge" files serve as source documents, while the "discharge_instructions" and "brief_hospital_course" fields from the "dis-charge_target" data are used for target summaries. the first stage involves segmenting documents into distinct segments using predefined rules for both source and target data. in the second stage, we use open large language models (llms) to gen-erate segment-wise summaries, extracting essential information from each segment of the source text and combining these into a condensed version. for the target texts, we determine the concept for each segment by training llms to predict and describe concepts from the summarized text. this process is repeated for both the "discharge instruction" and "brief hospital course" tasks. finally, the extracted concepts and descriptions are combined and presented as the output for each task. ## results and discussion the "discharge me!" dataset comprises training (68,785 samples), validation (14,719 samples), phase i testing (14,702 samples), and phase ii testing (10,962 samples) datasets, all sourced from mimic-iv submodules. it is worth noting that the phase ii testing dataset, set for release on april 12th, 2024, will serve as the final evaluation test set. due to time constraints, we were unable to execute our model on the entire dataset, ultimately utilizing a subset of 10,000 samples for training purposes. during the generation of summaries using the biomistral-7b model @xcite , we set the temperature to 0.0 to ensure determinism in our results. this controlled setting aimed to maintain consistency in the generated outputs during our experiments. we proceeded to fine-tune the same model separately for four distinct tasks: concept generation and subsequent description generation corresponding to each concept, aimed at generating the "brief hospital course" and "discharge instructions" sections. this fine-tuning process involved adopting the instruction tuning method, implemented using the supervised fine-tuning trainer foot_0 . we configured the parameters as follows: @xmath0, @xmath1. to reduce memory size, we utilized 4bit quantization, and for low-rank adaptation @xcite , we set @xmath2, @xmath3, and @xmath4.1. the model underwent training for 5 epochs, employing a training batch size of 4 and an evaluation batch size of 20. these parameter settings were chosen to strike a balance between training efficiency and model performance across the different tasks. in ## conclusion in conclusion, this paper describes our participation in the dischargeme shared task, which involved generating summaries of hospital course and discharge reports using mimic-iv data. our approach included data segmentation, concept identification and description, prompt-based summarization, and training models for concept extraction and description generation. we used pre-trained and fine-tuned large language models (llms) to produce structured, informative summaries. future work will compare different models and prompts and explore advanced data segmentation techniques to improve accuracy and efficiency.