Report2MRI: 3D rectified-flow brain MRI generation conditioned on fused radiology-report embeddings from multiple frozen medical text encoders

Winner of VLM3D @ MICCAI 2026 MRI Report-to-Volume Challenge.

Overview

Report2MRI is a 3D rectified-flow brain MRI (T1w, T2w, FLAIR, or SWI) generation model conditioned on fused free-form report embeddings from multiple frozen medical text encoders.

It builds on NVIDIA's frozen NV-Generate-MR-Brain rectified-flow latent diffusion model and adds trainable zero-initialized cross-attention adapters (11.75 M trainable parameters).

The report is encoded in the style of Report2CT (code). The findings, impression and a short acquisition description (modality, plane and voxel spacing) are each embedded by three frozen medical text encoders. The three embeddings are concatenated, so the adapters receive one 2560-dimensional token per section.

Results

These are the official hidden-test scores (690 of 690 cases scored). The final ranking used the mean rank over the four metrics.

	SSIM ↑	PSNR ↑ (dB)	MSE ↓	FID-2.5D avg ↓	mean rank
This model	0.4437	13.64	0.0447	204.05	1.25 (1st)
2nd place	0.4891	13.49	0.0466	263.51	1.75
3rd place	0.3716	13.00	0.0525	1190.57	3.00

Files in this repository

File	Contents
adapter.pt	Adapter checkpoint (format mrrate_r2v_adapter_v1): the adapter tensors plus the config, text-encoder identity, base-model sha256, scale_factor and validation metrics. Optimizer and RNG state are removed. Loads with torch.load(..., weights_only=True).
adapter.safetensors	The same 74 adapter tensors as a plain safetensors file, without the metadata.
inference_config.json	The exact inference settings of the leaderboard run.
native_spacing_table.json	Slice-thickness distribution per (modality, plane), measured on 575,536 MR-RATE training series. Used for the slice-axis resampling.
inplane_spacing_table.json	In-plane pixel-spacing distribution, measured on the same series. Used for the in-plane spline resampling.

Not included (download from the original sources):

File(s)	Source
diff_unet_3d_rflow-mr-brain_v0.pt	nvidia/NV-Generate-MR-Brain, models/ (NVIDIA Open Model License). Use v0, not v1.
autoencoder_v1.pt	nvidia/NV-Generate-CT, models/ (NVIDIA Open Model License). This is the shared MAISI VAE.
MedEmbed-large-v0.1/	abhinand/MedEmbed-large-v0.1
Bio_ClinicalBERT/	emilyalsentzer/Bio_ClinicalBERT
BiomedVLP-CXR-BERT-specialized/	microsoft/BiomedVLP-CXR-BERT-specialized

The adapter refuses to load on a base UNet whose sha256 differs from the one recorded in the checkpoint.

Model

report ─┬ findings    ┐  each section is encoded separately by three frozen encoders,
        ├ impression  ├─ mean-pooled and concatenated: 1024 (MedEmbed) + 768 (Bio_ClinicalBERT)
        └ acquisition ┘  + 768 (CXR-BERT) = 2560 features per section → context (B, 3, 2560)
                │
          context_proj (trainable): LayerNorm → Linear 2560→2048 → GELU → Linear 2048→512 → LayerNorm
                │
latent ──► NV-Generate-MR-Brain UNet (frozen) + 5 zero-initialised cross-attention adapters (trainable)

• Acquisition token. The acquisition section is the string [MODALITY] <m> [PLANE] <p> [SPACING] x y z, built from the requested series rather than parsed from the report.
• Adapter placement. The adapters sit at the two coarsest encoder levels, the two mirrored decoder levels and the bottleneck. They are zero-initialised, so before training the model reproduces NVIDIA's model exactly.
• Classifier-free guidance. A learned null_context stands in for a missing report, which allows report-level guidance. Modality guidance uses the base model's own class labels.
• Supported modalities. T1w, T2w, FLAIR, and SWI in axial, sagittal, or coronal planes.

License and attribution

• These weights: CC BY-NC-SA 4.0. They are trained on MR-RATE, which is released under CC BY-NC-SA 4.0, so non-commercial use and share-alike apply.
• Code: Apache-2.0 (see the GitHub repository).
• Base model: NV-Generate-MR-Brain and the MAISI VAE from NV-Generate-CT are under the NVIDIA Open Model License and are not redistributed here.
• Dataset: MR-RATE, by Forithmus, University of Zurich, Istanbul Medipol University and NVIDIA (Forithmus/MR-RATE).

Citation

@misc{spasojevic2026report2mri,
  title        = {Report-Conditioned Adapters on a Frozen 3D Brain-MRI Diffusion Prior},
  author       = {Spasojevic, Teodora and Wang, Jiayi and Durugoel, Oemer Faruk and Hagen, Luca and Nuetzel, Felix},
  year         = {2026},
  howpublished = {\url{https://github.com/teodoraspasojevic/Report2MRI}},
  note         = {1st place, VLM3D @ MICCAI 2026, MRI Report-to-Volume}
}