shs-nlp at R ad S um23: Domain-Adaptive Pre-training of Instruction-tuned LLM s for Radiology Report Impression Generation (2023)

Abstract: Instruction-tuned generative large language models (LLMs), such as ChatGPT and Bloomz, possess excellent generalization abilities. However, they face limitations in understanding radiology reports, particularly when generating the IMPRESSIONS section from the FINDINGS section. These models tend to produce either verbose or incomplete IMPRESSIONS, mainly due to insufficient exposure to medical text data during training. We present a system that leverages large-scale medical text data for domain-adaptive pre-training of instruction-tuned LLMs, enhancing their medical knowledge and performance on specific medical tasks. We demonstrate that this system performs better in a zero-shot setting compared to several pretrain-and-finetune adaptation methods on the IMPRESSIONS generation task. Furthermore, it ranks 1st among participating systems in Task 1B: Radiology Report Summarization.

## introduction a radiology report is the primary method for radiologists to communicate medical image interpretations (e.g., x-rays) and conclusions to ordering physicians. these reports typically include several sections @xcite , with the most important ones being the findings and impressions sections. the findings section describes abnormalities and diagnoses, while the impressions section summarizes the findings and highlights major abnormalities and recommendations. a sample report with these sections can be found in the field of radiology has experienced rapid growth in nlp techniques due to the need for efficient and accurate analysis of radiological reports, which often consist of unstructured textual data @xcite . for instance, various efforts have been made to automatically generate findings bifrontal hemorrhagic contusions are once again noted, stable compared to most recent prior, . . . . subarachnoid hemorrhage is once again noted within the left . . . . subdural hematoma is noted overlying the left temporal lobe and to the left . . . . there is no shift of normally midline structures. the ventricles appear unremarkable. a left temporal lobe hemorrhagic contusion remains stable in size . . . . the visualized paranasal sinuses are clear. there is no evidence of acute fracture. plms are trained on vast, diverse corpora, allowing them to capture various linguistic patterns @xcite . despite their strengths, they have limitations and biases affecting out-of-domain tasks. domain-specific plms, like biobert @xcite and radbert @xcite , have been developed for biomedical and clinical nlp. however, these models face constraints, such as radbert's training on small datasets with limited anatomical specialties. the pretrain-and-finetune paradigm for plms has emerged as the prevailing approach to address downstream tasks with substantial training data scarcity @xcite . recent studies (e.g., @xcite ) suggest that additional pretraining on in-domain text, known as specialist pretraining, is more effective in enhancing downstream performance. currently, another approach has surfaced, where instead of finetuning plms to perform downstream tasks, the objectives of downstream tasks are reformulated using textual prompts similar to the original pre-training objectives @xcite . this pretrain-and-prompttune paradigm is commonly referred to as prompttuning. multitask prompted finetuning (also known as instruction tuning) is a type of large-scale pretrainand-prompt-tune paradigm in which large plms (also referred to as llms) are finetuned using datasets representing various nlp tasks, defined by instructions as natural language prompts @xcite . utilizing this approach, scao et al. (2022) endowed their llm, bloom, with the ability to perform multilingual zero-shot instruction-based tasks and labeled it bloomz. we propose an extension to the domain adaptation paradigm beyond the typical method of pretrain-and-finetune for domain-specific tasks. we posit that further pretraining of instructiontuned llms, already subjected to the pretrainand-prompt-tune process, with in-domain text will improve and simplify adaptation. we refer to our approach as general-pretrain-prompt-tune-andspecial-pretrain. in this approach, the model is trained using the same initial lm objective in each of the three training stages (i.e., general pretraining, prompt-tuning, and domain-specialized pretraining), which is a significant advantage. we continued pretraining the instruction-tuned bloomz on mimic-iv to form radbloomz and evaluated this adaptation paradigm on the radiology report summarization task. the proposed system, in a zero-shot setting, demonstrates better performance than pretrain-and-finetune methods and ranks 1st among participating systems in task 1b: radiology report summarization at the bionlp 2023 workshop. overall, our contributions are as follows: ## methods our methods comprise preprocessing, domainadaptive pretraining, fine-tuning, and inference. ## experiments we propose two experimental runs for the summarization task. ## results and discussion we evaluated radbloomz against other systems using rouge for n-gram overlap and f1radgraph for fact overlap. in ## conclusion in this work, we introduce a new domain adaptation paradigm of general-pretrain-prompt-tuneand-special-pretrain, where we further pretrain an instruction-tuned llm (bloomz) on radiology domain text. we use radiology report summarization as the domain-specific task and demonstrate that the new paradigm-based llm model outperforms the standard "pretrain-and-finetune" method, even in a zero-shot setting. the system ranks 1st among participating systems in the hidden-test category for task 1b: radiology report summarization at the bionlp 2023 workshop.