Team: PULSAR at P rob S um 2023: PULSAR : Pre-training with Extracted Healthcare Terms for Summarising Patients’ Problems and Data Augmentation with Black-box Large Language Models (2023)

Abstract: Medical progress notes play a crucial role in documenting a patient’s hospital journey, including his or her condition, treatment plan, and any updates for healthcare providers. Automatic summarisation of a patient’s problems in the form of a “problem list” can aid stakeholders in understanding a patient’s condition, reducing workload and cognitive bias. BioNLP 2023 Shared Task 1A focusses on generating a list of diagnoses and problems from the provider’s progress notes during hospitalisation. In this paper, we introduce our proposed approach to this task, which integrates two complementary components. One component employs large language models (LLMs) for data augmentation; the other is an abstractive summarisation LLM with a novel pre-training objective for generating the patients’ problems summarised as a list. Our approach was ranked second among all submissions to the shared task. The performance of our model on the development and test datasets shows that our approach is more robust on unknown data, with an improvement of up to 3.1 points over the same size of the larger model.

## introduction medical progress notes are used to document a patient's course in a hospital, including their current condition, treatment plan, and any updates to the plan @xcite . automated identification of treated problems from the assessment sections of a progress note in form of a "problem list" can help healthcare stakeholders to gain an accurate understanding of the patient's condition, reducing workload and cognitive bias @xcite . this problem list is then used to outline and pursue a detailed treatment plan. the main contribution of this work is a novel framework for data augmentation and summarisation of diagnoses/problems. in our approach, first, we optimise a domain-specific language model (lm) using a combination of different pre-training objectives, depicted in figure 1 ; this model significantly outperforms the state-of-the-art, even when optimised on a limited number of manually annotated progress notes. second, we instruct large language models (llms) to generate synthetic data, in order to reduce the reliance on large, highquality annotated datasets. finally, we use the generated data to fine-tune the domain-specific lm on the task of problem list generation, given appropriate progress note sections. our approach ranked second among all submissions to the shared task without additional annotated data. the results of our evaluation suggest that our pre-training objectives are aligned with the downstream task of summarisation and can significantly improve performance. ## methodology figure 1 shows the two components of our framework: first we pre-train an encoder-decoder model on mimic-iii progress notes @xcite using three different concept masking pre-training objectives. then we employ data augmentation when fine-tuning our model for the summarisation task. ## experimental setup baselines: we have chosen to adapt t5-base as one of our baselines, similar to the approach taken by @xcite . additionally, we have incorporated various state-of-the-art models such as @xcite , clinicalt5 @xcite and pegasus @xcite . whereas flant5 is an enhanced version of t5 that has been finetuned in a mixture of tasks @xcite and clinicalt5 pre-trained on mimic-iii @xcite . pegasus is an abstractive summarisation model with gap sentences generation and masked language model @xcite as pre-train tasks. evaluation metrics: we calculate rouge @xcite scores on the test set, by comparing the generated summaries to their corresponding references, averaging for all generation samples. for all experiments, the data set was divided into a "train" and a "dev" set with a ratio of 8:2 for training and evaluation, respectively. the results are presented in ## results and analysis pre-training helps: both data augmentation is effective when the data distribution is consistent; it is significantly more helpful for small models when on a random data distribution: the model is capable of discriminating irrelevant information, but longer input lengths may result in decreased performance: we conducted ablation experiments on pulsar-3b to verify the impact of the input text type. in contrast to @xcite 's findings on the small model, the results (pulsar-3b vs. pulsar-3b-a) in ## conclusion this paper contributed to the development of summarising patients' problems. firstly, we proposed a novel task-specific pre-training llm objective. compared with other submissions, we rank 2nd at the official shared task without using additional manually annotated training samples. secondly, we propose a new data augmentation framework and demonstrate its effectiveness in the healthcare domain. in the future, we will explore the applicability of our approach to other domain-specific generative tasks and conduct a deeper analysis of factors that contribute to overall model performance. tianyi zhang, varsha kishore, felix wu, kilian q. weinberger, and yoav artzi. 2020b. bertscore: evaluating text generation with bert. in iclr. openreview.net. yuhao zhang, daisy yi ding, tianpei qian, christopher d. manning, and curtis p. langlotz. 2018. learning to summarize radiology findings. in louhi@emnlp, pages 204-213. association for computational linguistics.