Divide and Conquer Radiology Report Generation via Observation Level Fine-grained Pretraining and Prompt Tuning (2024)

Abstract: The automation of radiology report generation (RRG) holds immense potential to alleviate radiologists’ workloads and improve diagnostic accuracy. Despite advancements in image captioning and vision-language pretraining, RRG remains challenging due to the lengthy and complex nature of radiology reports. In this work, we proposes the Divide and Conquer Radiology Report Generation (DCRRG) model, which breaks down full-text radiology reports into concise observation descriptions. This approach enables the model to capture fine-grained representations from each observation through a two-stage process: an encoding stage focusing on observation prediction tasks to learn fine-grained representations, and a decoding stage for integrating these descriptions into cohesive and comprehensive radiology reports. Experimental results on two benchmark datasets demonstrate that DCRRG achieves significant improvements across all evaluated metrics, underscoring its capability to generate semantically coherent and clinically accurate radiology reports.

## introduction radiology images are commonly used to diagnose, monitor, and treat medical conditions in clinical practice @xcite . recently, automatic radiology report generation (rrg) has garnered increasing attention from both the machine learning and medical fields. this technology aims to generate semantically coherent and informative reports to describe the corresponding examination images. such techniques hold significant clinical potential by alleviating the workload of junior radiologists and reducing diagnostic errors through improved interpretation @xcite . notable advancements in artificial intelligence have led researchers to propose various datadriven neural networks for automatic rrg, yielding promising results @xcite @xcite @xcite @xcite . similar to the task of image captioning @xcite , which aims to describe the visual content of images, rrg models typically use an encoder-decoder architecture. in the encoding stage, visual representations of the images are extracted by a vision encoder, usually pretrained on image-label datasets. in the decoding stage, the medical report is generated by a decoder, employing either a transformer @xcite or lstm @xcite architecture with image-text datasets. recently, large-scale vision-language pretraining (vlp), such as clip @xcite , has achieved significant success through contrastive learning on image-text datasets. by jointly training on large-scale image-text pairs, this approach generates transferable representations that support versatile downstream tasks, enhancing the efficacy of vision encoders for rrg. building on these advancements, recent studies @xcite @xcite concentrate on refining vision encoders for rrg using contrastive learning with image-text paired datasets. despite the substantial advancements achieved in this field, rrg remains a difficult task. as is demonstrated in figure 1 , the radiology report comprise 6 sentences, describing 5 observations, among which 3 are positive, 1 is negative, and 1 is uncertain. the lengthy and complex nature of medical reports necessitates both the encoder and decoder be able to handle subtle and fine-grained representations. the accompanying bar chart displays the distribution of each observation, each facing varying degrees of data imbalance. however, since a report usually includes multiple observations, addressing the imbalance of one observation without impacting others presents a significant challenge. in summary, several challenges significantly impact the effectiveness of rrg models: (1) compared to image captioning, radiology reports are notably longer and comprise multiple sentences, each detailing specific medical observations. (2) medical datasets frequently exhibit substantial class imbalances and skewed data distributions. (3) the scarcity of training data particularly limits the performance of data-intensive vlp models. therefore, an ideal rrg model should: (1) capably learn subtle and detailed representations both during the encoding and decoding stages. (2) effectively address data imbalances for individual observations without unintended consequences for others. (3) harness uni-modal data (such as image-only or text-only) alongside image-text pairs for enhanced training effectiveness. noticing that a fundamental distinction between rrg and image captioning lies in the constrained terminology used for describing diseases (observations) in radiology reports, compared to the open vocabulary typically encountered in image captioning tasks. leveraging this distinction allows us to break down the lengthy and complex full-text radiology report into limited observation descriptions. the challenge then shifts to how to effectively learn subtle and nuanced representations from each observation and integrate them cohesively to generate the complete radiology report. in response to this challenge, we propose the divide and conquer radiology report generation (dcrrg) model. this approach focuses on two main stages: (1) encoding stage: the model learns fine-grained representations through observation prediction tasks. each observation is treated independently to capture detailed nuances specific to that observation. (2) decoding stage: the model generates a cohesive description for each indicated observation, integrating them to form the full-text radiology report. specifically, we employ prompts to bridge between individual observations and full reports. in the encoding stage, both the text and image encoders are trained with prompts to predict observation labels. this setup allows us to implement data balancing strategies and integrate uni-modal data effectively. subsequently, the encoders undergo joint fine-tuning to align cross-modal representations via contrastive learning. in the decoding stage, the decoder is trained to generate descriptions for individual observations based on either image or text representations, guided by prompts indicating the specific observation to be generated. prompt tuning further refines the model for efficient fulltext report generation. in summary, our proposed dcrrg model contributes: ## method in this section, we will detail the implementation of our proposed divide and conquer radiology report generation approach (dcrrg). the overall architecture of dcrrg is illustrated in figservation. in step 4, prompt tuning is applied to efficiently adjust the model parameters for full-text report generation. the details of each step is described in the following sections. ## experiments we assess our methods across two dimensions: the generated reports from dcrrg and the learned representations from dcclip. first, we outline two benchmark datasets, the metrics employed, and the evaluation settings. subsequently, we present the primary findings and in-depth analysis of our approach on these datasets. the experimental settings can be found in appendix a. ## conclusion in this study, we introduce dcrrg, a divide and conquer training strategy for radiology report generation. this approach optimizes the feature extraction network individually for each observation, effectively mitigating data imbalance. to enhance the learning of uni-modal representations and facilitate integration between visual and textual modalities, we propose divide-and-conquer contrastive learning (dcclip). this method aims to refine accurate and fine-grained representations, thereby improving performance in text-oriented medical report generation tasks. experimental results on two benchmark datasets show that our approach achieves substantial enhancements across all evaluation metrics.