Fact-Aware Multimodal Retrieval Augmentation for Accurate Medical Radiology Report Generation (2025)

Abstract: Multimodal foundation models hold significant potential for automating radiology report generation, thereby assisting clinicians in diagnosing cardiac diseases. However, generated reports often suffer from serious factual inaccuracy. In this paper, we introduce a fact-aware multimodal retrieval-augmented pipeline in generating accurate radiology reports (FactMM-RAG). We first leverage RadGraph to mine factual report pairs, then integrate factual knowledge to train a universal multimodal retriever. Given a radiology image, our retriever can identify high-quality reference reports to augment multimodal foundation models, thus enhancing the factual completeness and correctness of report generation. Experiments on two benchmark datasets demonstrate that our multimodal retriever significantly outperforms other state-of-the-art retrievers on both language generation and radiology-specific metrics, up to 6.5% and 2% score in F1CheXbert and F1RadGraph. Further analysis indicates that employing our factually-informed training strategy imposes an effective supervision signal, without relying on explicit diagnostic label guidance, and successfully propagate fact-aware capabilities from the multimodal retriever to the multimodal foundation model in radiology report generation.

## introduction within hospitals worldwide, chest radiology serves as a critical technique in identifying cardiac diseases and abnormalities. results of a chest radiograph are typically consolidated in a radiology report, including the source x-ray and a radiologist-produced findings section detailing clinical observations. manually generating these reports, however, can be both time-consuming and potentially inaccessible in under-resourced hospitals @xcite . recent multimodal foundation models have exhibited remarkable capabilities in challenging healthcare tasks, motivating an automation of this process to enhance physicians' efficiency on clinical decision-making and improve patient health outcomes @xcite @xcite @xcite . although prior medical multimodal foundation models have demonstrated promising capabilities on report generation given the radiology image, they still suffer from serious hallucinations by generating factually inaccurate reports @xcite @xcite . factual correctness is especially critical in chest radiology domains, as minute textual differences can drastically invert radiology report meaning and downstream prescribed treatments @xcite @xcite . retrieval-augmented generation (rag) has emerged as a popular paradigm to address this issue by grounding text generation with retrieved relevant knowledge given a query @xcite @xcite . however, developing medical multimodal retrievers remains challenging, requiring retrievers to bridge the gap between symptomatic image semantics and factually-equivalent report text. to capture fine-grained details in chest radiographs and improve the factual completeness of generated reports, we introduce factmm-rag, a fact-aware multimodal retrieval-augmented pipeline for generating accurate radiology reports given a radiology image. by designing a novel report pair-mining procedure incorporating factual knowledge, we develop a fact-aware retriever to augment multimodal foundation models in generating accurate chest x-ray radiology reports. specifically, we first leverage radgraph @xcite to mine factually-oriented report pairs by annotating consistent radiology entities and relations between query and reference reports with certain abnormalities. next, we train a universal multimodal encoding architecture through mined report pairs to conduct multimodal dense retrieval. given an unseen patient's radiology image, our retriever encodes it and searches for the most similar factually-informed reference report from an available report corpus. passing them together into a multimodal foundation model unlocks its fact-aware potential to generate more accurate radiology reports. our experiments reveal that our retriever outperforms all state-of-the-art retrievers in both language generation and clinically relevant metrics on the mimic-cxr and chexpert datasets, achieving up to 6.5% and 2% score in f1chexbert and f1radgraph for final rag evaluation. we also investigate our retriever's fact-aware capability controlled by factual similarity thresholds and confirm that our factually-informed training strategy can impose a useful supervision signal without relying on explicit diagnostic label guidance. further analysis through retrieval evaluation metrics shows that the fact-aware capability of our retriever can be effectively propagated to the multimodal foundation models. lastly, our case study highlights that among reports describing the same symptom from different retrievers, those generated by our model are more accurate and achieve greater factual correctness. our main contributions can be summarized as follows: the rest of this paper is organized as follows. we review related work in in section 2. we discuss the pipeline of factmm-rag in sections 3. section 4 and 5 discuss our experimental setup and results. ## methodology in this section, we present the overall methodology of factmm-rag. we first detail the training procedure of our fact-aware medical multimodal retriever in section 3.1. we then provide the pipeline for retrieval-augmented radiology report generation with our multimodal retriever in section 3.2. the overview is illustrated in figure 1 . ## experimental setup dataset. following @xcite , we use the processed mimic-cxr @xcite to train both retriever and foundation model. this dataset contains 125,417 training radiology image-report pairs, 991 validation pairs, and 1,624 test pairs. they are sourced from the beth israel deaconess medical center. chexpert @xcite is another chest x-ray dataset from stanford health care. since it contains complete finding reports only for a testing dataset containing 1000 pairs, we use it as zero-shot evaluation. evaluation metrics. we evaluate our proposed system using both natural language generation and medically-tailored evaluation metrics. for language fluency measures, we use rouge-l @xcite to evaluate the longest common subsequence overlap between the generated and reference findings, and bertscore @xcite to evaluate non-clinical semantic sentence similarity. for clinical accuracy measures, we use chexbert @xcite to generate the ground-truth diagnostic labels for finding reports. following delbrouck et al. (2023) , we then calculate the f1chexbert @xcite , which is the f1-score for 5 observations (cardiomegaly, edema, consolidation, atelectasis, pleural effusion) by comparing the generated report with the reference report's classifications. beyond using the limited diagnostic labels for evaluation, we also adopt f1radgraph @xcite to measure factual correctness by calculating the overlap in radiological entities and clinical relations between the generated report and the reference report. see appendix a.3 for more details. baselines. we mainly compare our retriever with other baselines under multimodal rag setting. we include the following baselines, clip @xcite is a multimodal retriever pretrained from general-domain image-text pairs; gloria @xcite leverages attention-weighted image regions with contextual words to learn localized and global representations for radiology images and reports; medclip @xcite and cxr-clip @xcite build upon clip and utilize diagnostic labels as training signals for learning radiology image and text representations; biomedclip @xcite extends the radiology-specific dataset and pretrains on a larger magnitude of biomedical data to learn multimodal representations; med-marvel utilizes universal encoder marvel @xcite to conduct contrastive learning on each patient's self image-report pair without further training on factual image-report pairs. we also compare our method with non-rag approach. "no retriever" refers to directly fine-tuning the backbone to generate reports without retrieval augmentation; organ @xcite first creates an observation plan, then feeds the plan and radiographs to generate the report through tree reasoning mechanism. upper-bound results using an oracle in training corpus with top-1 factual similarity to test query report are presented. more details are in appendix a.3. ## evaluation results in this section, we present our experimental results. we first evaluate the overall performance between different retrievers in section 5.1. next, we discuss the ablation studies in section 5.2. we then explore the fact-aware capability of our retriever in section 5.3 and section 5.4. lastly, we show the superiority of our retriever through a case study in section 5.5. ## conclusion in this paper, we aim at improving radiology report generation by introducing a fact-informed medical multimodal retriever for retrieval-augmented generation. in particular, we utilize radgraph to annotate chest radiograph reports and mine clinicallyrelevant pairs. we integrate factual information into a universal multimodal retriever, presenting factmm-rag, a fact-aware multimodal retrievalaugmented radiology report generation pipeline. factmm-rag outperforms all state-of-the-art retrievers evaluated by factual correctness and textual coherence for final report generation in mimic-cxr and chexpert datasets. we further confirm the benefit of our multimodal retriever from the analysis of its fact-aware capability.