CortexDecoder

Bridge weights for CortexDecoder, trained on NSD Subject 1. Code is on GitHub and the flatmaps and reconstructions are in teooi/cortexdecoder-nsd-flatmaps.

A frozen CortexMAE encoder turns visual-cortex flatmap patches into 200 brain tokens of dimension 768. Two cross-attention bridges map those tokens to image features.

File	Bridge	Output	Parameters
vgg_bridge.safetensors	low-level	7190 VGG16-BN tokens of dimension 512, inverted to an image with a Deep Image Prior	43.7M
clip_bridge.safetensors	semantic	256 OpenCLIP ViT-bigG/14 tokens of dimension 1664, used to condition SDXL-unCLIP	39.3M

checkpoints/ has the original training checkpoints, which also contain the optimizer state. config.json lists the bridge settings.

Loading

from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from cortexdecoder.core.config import Config                     # from the GitHub repo, code/v2
from cortexdecoder.models.bridge import FlatmapSingleHeadBridge

bridge = FlatmapSingleHeadBridge(Config(), brain_dim=768, num_patches=200, target="clip")
bridge.load_state_dict(load_file(hf_hub_download("teooi/cortexdecoder", "clip_bridge.safetensors")))
clip_tokens = bridge(brain_tokens)                               # [B, 200, 768] -> [B, 256, 1664]

The brain tokens come from code/v2/scripts/generate_roi_tokens.py, which runs CortexMAE with only the visual-cortex patches visible.

Results

NSD Subject 1, 998 test images, one diffusion sample per image.

PixCorr	SSIM	Alex(2)	Alex(5)	Incep	CLIP	Eff	SwAV
0.366	0.413	98.1%	99.1%	94.9%	93.5%	0.628	0.360

Citation

@article{imototar2026cortexdecoder,
  title   = {CortexDecoder: fMRI-to-image Reconstruction from Cortical Flatmaps},
  author  = {Imoto-Tar, Teo and Fung, Maxwell and Gurha, Arnav},
  journal = {bioRxiv},
  year    = {2026},
  doi     = {10.64898/2026.10.09.757766}
}