Argus: Benchmarking and Enhancing Vision-Language Models for 3 D Radiology Report Generation (2025)

Abstract: Automatic radiology report generation holds significant potential to streamline the labor-intensive process of report writing by radiologists, particularly for 3D radiographs such as CT scans. While CT scans are critical for clinical diagnostics, they remain less explored compared to 2D radiographs. To date, there has been no comprehensive benchmark for 3D radiograph report generation (3DRRG), nor sufficient investigation into the optimal training strategies for Vision Language Models (VLMs) in this context, particularly with respect to vision encoder choices, visual token compression, and model scaling.In this work, we make two three contributions. We curate CT-3DRRG, the largest publicly available 3D CT-report dataset, establishing a robust and diverse benchmark for evaluating VLM performance on 3DRRG. Furthermore, we propose a comprehensive training recipe for building high-performing VLMs for 3DRRG, exploring key factors such as vision encoder pretraining strategies, visual token compression, and the impact of data & model scale. Guided by these findings, we introduce Argus, a state-of-the-art family of VLMs that achieve superior performance across different model sizes and input 3D medical image resolutions, efficiently processing high-resolution 3D images up to 512 × 512 × 256.

## introduction radiology reports are essential for clinical decision making, yet their manual creation is laborintensive and time-consuming @xcite @xcite . this has driven the need for automation in radiology report generation. existing work on this task primarily focuses on 2d images, such as chest x-rays @xcite . however, compared to 2d images, 3d medical images provide more comprehensive diagnostic information and are crucial for identifying life-threatening diseases such as pulmonary opacities and earlystage cancer @xcite . while 3d radiograph report generation (3drrg) has the potential to significantly improve clinical workflows, several challenges remain unresolved in the literature. notably, there is no comprehensive benchmark for 3drrg, and many existing studies lack robust evaluations across multiple datasets and metrics. traditional nlp metrics alone are inadequate for assessing the clinical relevance of generated reports. as a result, new evaluation metrics, such as clinical efficacy measures like green @xcite and ratescore @xcite , are essential for inclusion in the benchmark to evaluate report quality in clinical settings. additionally, high-resolution (hr) 3d images offer more clinical value than low-resolution (lr) images, as certain conditions, such as pulmonary nodules, are harder to detect in lr scans @xcite . however, most studies @xcite downsample hr volumes (e.g., 512 × 512 × 256) to lr (256 × 256 × 32) for report generation, which leads to significant information loss and may overlook critical details. this downsampling is primarily driven by the increased computational cost associated with processing the larger number of visual tokens generated by hr volumes. furthermore, no clear training recipe exists for effectively building vision-language models (vlms) for 3drrg, particularly in terms of selecting and pretraining vision encoders, visual token compression, and optimizing model and data scale. this brings us to a crucial challenge: the lack of large-scale public datasets and a comprehensive benchmark for 3drrg. additionally, training strategies for vlms remain unclear, requiring a systematic analysis of key components. to address this challenge, we make 4 key contributions: ct-report dataset, ct-3drrg, comprising 49,849 ct volumes paired with radiology reports from three publicly available sources: bimcv-r, ct-rate, and inspect @xcite @xcite . vision-language models. the development of vision-language models (vlms) combines computer vision and natural language processing to enhance visual and linguistic capabilities. this integration is crucial for tasks that require both visual perception and language comprehension. models like clip @xcite , flamingo @xcite , and blip @xcite have improved the alignment between these modalities by using extensive image-text data samples, resulting in significant performance gains. llava @xcite further simplifies this approach by using a basic linear projector to align visual features with language space, involving only a few learnable parts and tailored instruction data to fully leverage the model's strong capabilities. despite their effectiveness, these approaches focus mainly on 2d images. to explore the relatively underexplored area of 3d medical imaging, such as ct scans, we extend the llava framework by incorporating a customized 3d vision encoder specifically designed for 3d image variants. our goal is to tackle the challenge of generating accurate radiology reports for 3d medical images. ## argus, the state-of-the-art vlm for 3drrg we leverage insights from our studies to develop argus, a state-of-the-art (sota) vlm for the 3drrg task. to ensure a fair comparison, we evaluate argus alongside three existing baselines: radfm @xcite , m3d @xcite , and ct2rep @xcite , all of which have publicly available code, allowing for direct evaluation. to maintain consistency, we replace the llm backbones of these baselines with llama3.1-instruct (8b and 70b) and llama3.2-instruct-3b, aligning them with our llm selection. we reimplement their training on our ct-3drrg dataset while using the official vision encoder weights from their repositories. additionally, we adhere to their original implementations for token compression strategies, connectors, and training schedules. for the high-resolution setting, we interpolate the positional embeddings of their vision encoders to support a resolution of 512×512×256, enabling them to process high-resolution 3d ct scans. we compare argus across both normal (256 × 256 × 64) all models are trained on the same training set from ct-3drrg and evaluated on three test sets. we report performance using average nlp metrics (bleu1-4, rouge-1,2,l, meteor, cider) and clinical evaluation metrics, including green, rad-graphxl, and ratescore @xcite @xcite , as shown in we further visualize the generated reports from m3d-8b and argus-8b under the normal resolution setting, as shown in figure 7 . the results demonstrate that argus-8b accurately captures key anatomical structures and pathological findings, while m3d-8b exhibits inconsistencies in lesion descriptions and misidentifies certain structures. this highlights the superiority of our method in @xcite , radfm @xcite , and m3d @xcite under both normal and high-resolution settings. argus-3b outperforms other 8b models and even achieves comparable performance with larger-scale models. argus-8b matches the performance of other 70b variants, while at the 70b scale, argus surpasses all competing methods. these results demonstrate the superiority of our approach, highlighting its scalability and effectiveness across resolutions. ## conclusion in this work, we present ct-3drrg, the first and largest dataset comprising 3d volume-report pairs for 3drrg, built entirely from publicly available resources. we critically examine key components of vlms, including vision encoder pre-training, visual token compression, and associated design and training strategies, while also analyzing the impact of scaling both data and model sizes. leveraging these findings, we introduce argus, a family of vlms ranging from 3b to 70b parameters, which achieves superior performance on 3drrg task. thanks to our efficient design, even smaller models such as argus-3b outperform existing methods utilizing larger llms (e.g., 7b), demonstrating the effectiveness and efficiency of our approach. our study provides valuable insights into how these factors influence vlm performance in 3drrg tasks and establishes a strong foundation for future research, fostering the development of more efficient and scalable vlms for 3drrg.