R ad A dapt: Radiology Report Summarization via Lightweight Domain Adaptation of Large Language Models (2023)

Abstract: We systematically investigate lightweight strategies to adapt large language models (LLMs) for the task of radiology report summarization (RRS). Specifically, we focus on domain adaptation via pretraining (on natural language, biomedical text, or clinical text) and via discrete prompting or parameter-efficient fine-tuning. Our results consistently achieve best performance by maximally adapting to the task via pretraining on clinical text and fine-tuning on RRS examples. Importantly, this method fine-tunes a mere 0.32% of parameters throughout the model, in contrast to end-to-end fine-tuning (100% of parameters). Additionally, we study the effect of in-context examples and out-of-distribution (OOD) training before concluding with a radiologist reader study and qualitative analysis. Our findings highlight the importance of domain adaptation in RRS and provide valuable insights toward developing effective natural language processing solutions for clinical tasks.

## introduction radiology reports are comprehensive documents that capture and interpret the results of a radiological imaging examination. reports are often structured into three main sections: (1) a background section that provides general information about the exam and the patient, e.g. medical history (2) a findings section that presents detailed exam analysis and results, and (3) an impression section that concisely summarizes the most salient findings. in a typical workflow, a radiologist first dictates the detailed findings and then distills them into a concise impression. this impression is the most significant part of a radiology report, as it contains crucial information for clinical decisionmaking @xcite . however, perform-* equal contribution ing radiology report summarization (rrs) manually can be labor-intensive and prone to errors @xcite , motivating the importance of automating this task. large language models (llms) have demonstrated remarkable capabilities in natural language understanding and generation, serving as foundation models that can be adapted to various domains and tasks. however, their sheer size, sometimes exceeding 100b parameters, makes training for domain-specific tasks prohibitively expensive in terms of computation and training data. we address this challenge by exploring lightweight strategies for domain adaptation in the context of rrs, culminating in the following contributions: overall, our research presents a comprehensive investigation of lightweight strategies for domain adaptation in the context of rrs, offering insights into the effectiveness of different approaches and highlighting the potential of llms for this task. our findings contribute toward the advancement of applied natural language processing (nlp) to radiology with implications for improving radiologists' workflow and patient care. ## related work in recent years, transformer-based @xcite language models have become ubiquitous in nlp due to their state-of-the-art performance across many tasks including language generation, question answering, and machine translation. transformer models bert @xcite and gpt-2 @xcite established a new paradigm of first training on large amounts of general data and then fine-tuning on domain-specific data, as opposed to direct training on domainspecific data. this has led to training transformers with more parameters on increasingly more data, resulting in llms such as gpt-3 @xcite , palm @xcite , and the "text-to-text transfer transformer," or t5 @xcite . however, end-to-end fine-tuning llms like gpt-3 (175b parameters) requires substantial computational resources, creating a high barrier to entry. as a result, there has been a growing interest in lightweight methods for domain adaptation. one such method is prompting, in which one provides initial text input to the llm so it has context for the given task. performance depends heavily on the provided prompt @xcite , motivating principled prompting methods. various works have pursued this in the form of natural language instructions @xcite or supplying "in-context" examples of desired output @xcite ). an alternative approach consists of parameter-efficient fine-tuning, where one freezes existing model weights and inserts a small number of tunable parameters @xcite @xcite . we focus on the two highest regarded parameter-efficient fine-tuning methods: prefix tuning (li and liang, 2021; @xcite and lora @xcite , discussed further in section 3. in addition to methods for prompting and finetuning, previous work has demonstrated adaptation to the medical domain via pretraining on biomedical or clinical text. consider scifive @xcite , clinical-t5 @xcite , and med-palm @xcite , which leveraged llms for text generation on medical tasks. both scifive (biomedical) and clinical-t5 (clinical) achieved state-of-the art results for their respective domains in tasks such as named entity recognition, natural language inference, and questionanswering. additionally, med-palm achieved success aligning text generation models to clinical tasks via methods such as chain-of-thought prompting, prompt tuning @xcite , and instruction tuning @xcite . within the clinical domain, we focus specifically on the task of rrs. previous work has approached this task with a focus on consistency and factual correctness @xcite @xcite . to the best of our knowledge, this work is the first to leverage lightweight domain adaptation strategies on llms for the rrs task. additionally, most prior work on rrs uses only chest x-rays @xcite from large datasets like mimic-cxr @xcite and @xcite . in contrast, our work employs the mimic-iii dataset @xcite , which contains longer radiology reports from a diverse set of two imaging modalities (mr, ct) and seven anatomies (head, chest, etc.), presenting a more difficult summarization task. ## methods as depicted in figure 1 , we investigate adapting llms to the task of rrs along two axes: (1) models pretrained on various combinations of natural language, biomedical, and clinical text data, as described in section 3.1, and (2) various methods for discrete prompting and parameter-efficient finetuning, as described in section 3.2. c4 c4 c4 pubmed c4 pubmed mimic-iii, iv c4 mimic-iii, iv models instruction prompt tuning datasets used to pretrain methods used for prompting, tuning null summarize the following radiology report: findings: ... impression: prefix findings: ... impression: findings: [example] impression: [example] ... findings: ... impression: in-context [tune parameters prefixed to model] findings: ... impression: prefix tuning lora { , , , , } { , , , , } t5 flan-t5 scifive clin-t5-sci clin-t5 template × zero-shot discrete prompting parameter-efficient fine-tuning few-shot increasing domain adaptation via model pretraining (top) and methods for prompting or tuning (bottom) [tune parameters injected within model] findings: ... impression: ## conclusion our research employs innovative lightweight strategies to adapt llms for the task of rrs. we investigate how domain adaptation-both via model pretraining on relevant data and via methods for discrete prompting and parameter-efficient fine-tuning-affects downstream rrs task performance. we achieve best performance using a model pretrained on clinical text (clin-t5) and subsequently fine-tuned with rrs samples using lora. these compelling results require tuning a mere 0.32% of model parameters. while further validation is required before clinical deployment, we believe our findings contribute to the literature and advance the potential for improved radiologist workflows and patient care.