CSTRL : Context-Driven Sequential Transfer Learning for Abstractive Radiology Report Summarization (2025)

Abstract: A radiology report comprises several sections, including the Findings and Impression of the diagnosis. Automatically generating the Impression from the Findings is crucial for reducing radiologists’ workload and improving diagnostic accuracy. Pretrained models that excel in common abstractive summarization problems encounter challenges when applied to specialized medical domains largely due to the complex terminology and the necessity for accurate clinical context. Such tasks in medical domains demand extracting core information, avoiding context shifts, and maintaining proper flow. Misuse of medical terms can lead to drastic clinical errors. To address these issues, we introduce a sequential transfer learning that ensures key content extraction and coherent summarization. Sequential transfer learning often faces challenges like initial parameter decay and knowledge loss, which we resolve with the Fisher matrix regularization. Using MIMIC-CXR and Open-I datasets, our model, CSTRL — Context-driven Sequential TRansfer Learning — achieved state-of-the-art performance, showing 56.2% improvement in BLEU-1, 40.5% in BLEU-2, 84.3% in BLEU-3, 28.9% in ROUGE-1, 41.0% in ROUGE-2 and 26.5% in ROGUE-3 score over benchmark studies. We also analyze factual consistency scores while preserving the medical context. Our code is publicly available at https://github.com/fahmidahossain/Report_Summarization.

## introduction a radiology report summarizes a radiologist's analysis of imaging data, detailing sections like type of exam, clinical history, technique, findings, and impression @xcite . findings are observations from the examination of diagnosed body parts, classified as normal, abnormal, or potentially abnormal @xcite . impression is a summary of findings, including possible causes @xcite . quality of contextual, informative, and factual correctness of generated impression is subpar due to limited training for writing impression (≤1 hour/year for 86% of radiologists) @xcite . synthesizing impression from the findings is, therefore, crucial in automated radiology report summarization. writing impression from findings falls under the abstractive summarization task that distills key clinical insights from diagnostic data @xcite . extractive summarization selects sentences from the original text to create a summary, while abstractive summarization generates a new paragraph to summarize the content of the original document @xcite . abstractive summarization is more complex than extractive summarization but yields more flexible and concise summaries @xcite . however, systematic evaluation of abstractive summarization across diverse domains is limited @xcite . one challenge of abstractive summarization in the clinical domain is managing the balance between providing concise, clinically relevant information and avoiding excessive technical language. for example, "liver metastases have enlarged" is more straightforward and actionable than "there is redemonstration of multiple liver lesions consistent with metastases, which have increased in size in the interval" @xcite . including irrelevant findings can create ambiguity. for example, "diverticulosis without diverticulitis" @xcite : diverticulosis indicates the presence of diverticula in the colon, which is typically benign, but, without proper context, it may raise concerns about potential complications and suggest that diverticulitis (inflammation of diverticula) could develop. to prioritize core sentences of findings and preserve context, we propose a novel approach combining sequential transfer learning with a knowledge distillation (kd) strategy, which automates the creation of impression from the find-ings of a radiology report. deep learning techniques for abstractive summarization face specific challenges with radiology corpora. a key limitation is the difficulty in evaluating performance on datasets dense with radiologyspecific terminology, along with significant differences in word distributions between general and radiology report domains @xcite . previous works (section 2) have several shortcomings. firstly, there is a lack of a proper methodology for generating impression that focuses on primary observations @xcite . we propose an optimized gap sentence generation (gsg) technique to identify key findings. secondly, subtle differences in terminology can obscure the context and misrepresent the clinical scenario @xcite . in response, we propose a contextual tagging method. thirdly, poor bleu scores of recent studies indicate that the generated impression section often fails to accurately reflect core points from the findings section @xcite . it affects the readability, informativeness, and reliability of reports for clinical use. lastly, reducing dimensionality and computational complexity for real-time production is a challenge @xcite @xcite , for which we propose a kd method. our key contributions in constructing impression through proper clinical methodology with high performance and factual consistency are as follows: to mitigate catastrophic forgetting during this sequential knowledge transfer, our model includes fisher matrix regularization with penalty adjustment. ## related work in recent years, deep learning has significantly improved neural abstractive summarization tasks in nlp @xcite @xcite @xcite . traditional models are primarily trained on general datasets like cnn/daily mail @xcite and gigaword corpus @xcite . the attention-based seq2seq model by @xcite and its extension by @xcite laid the groundwork for these advancements. recently, large-scale pre-trained language models have shown impressive summarization results @xcite @xcite . @xcite employed bert and gpt for summarizing covid-19 research. however, unique word distributions in radiology corpora limit the applicability of these techniques @xcite . @xcite first explored automatic radiology impression generation, followed by @xcite , who introduced an ontologyaware pointer-generator model to enhance summarization quality. a background-augmented pointergenerator network with copy and backgroundguided decoding was proposed in @xcite , and a word graph captured critical words and relations in @xcite . anatomies were extracted, radiographs encoded, and fused with anatomy-enhanced co-attention in @xcite . sotudeh gharebagh et al. (2020) enhanced clinical summarization by augmenting ontological terms, and hu et al. (2022) integrated findings by a unified framework with knowledge via text and graph encoders. karn et al. (2022) proposed a two-step extractive-abstractive method using a differentiable multi-agent actor-critic framework. li et al. (2018) proposed a hybrid retrievalgeneration agent that integrates human knowledge with neural networks for medical report generation. models like openai gpt (chang et al., radiology corpus w 1 w 2 mask(x) w 3 mask(x) w 4 sentence 3 sentence 2 sentence 1 sentence 4 encoder decoder findings sentence scoring & selection (sss) selective sentence masking (ssm) encoder decoder w1 f 1 f l f l f ≈ 1 l l 1 l l x 1 x 2 x 3 x 4 y 1 y 2 y 3 x 1 x 2 x 3 x 4 y 1 y 2 y 3 <@xmath0 figure 1 : workflow of the proposed scheme for sequential transfer learning. firstly, sentences from the radiology corpus are scored with rouge and bleu metrics, and top-scoring sentences are masked. secondly, the t5 model is trained on the gsg task to predict the masked sentences. the fisher information matrix is calculated to identify critical parameters. lastly, the penalty term is adjusted during fine-tuning for summarization. 2021), bert @xcite , elmo @xcite , and xlnet @xcite have improved various nlp tasks through extensive external knowledge. biobert @xcite captures semantic features in the biomedical field, pre-trained on large corpora like pubmed abstracts and pmc articles. recently, @xcite proposed a pre-trained language model, chestxraybert, specifically designed for summarizing chest radiology reports. however, these studies may struggle with effective impression generation methodologies. ## method we adopt a sequential two-step approach to finetune the t5 model @xcite using a dataset on radiology corpus optimized by the gsg methodology @xcite . in the first step, the model is trained on a gsg task, where it predicts masked sentences, thereby gaining a deeper understanding of the crucial statements for crafting coherent summaries in the impression section. in the second step, we fine-tune the model for summarization using the learned weights as initial parameters and employ fisher matrix regularization for effective gsg-trained knowledge transfer to the summarization task. then, the gsg fine-tuned model drives sequential knowledge distillation to achieve dimensionality reduction. ## conclusion this paper proposes cstrl-context-driven sequential transfer learning-a novel fine-tuning framework for generating factually correct and coherent radiology report summaries. cstrl follows a dual-stage process, where it is fine-tuned on both the gsg and summarization tasks. we show in the ablation study that it not only improves summarization quality, as measured by rouge and bleu scores, but also enhances factual correctness. to the best of our knowledge, we first introduce a fusion of methodologies for generating accurate impressions and preserving context within the clinical domain.