Hugging Face's logo

ffh92r32rm0
/
Amphora_V5.1

Feature Extraction
PyTorch
English
neuroscience
brain-encoding
fmri
multimodal
fsaverage5
bold
algonauts
Model card Files
xet
 Community

🧠 V5 NeuroDiffusion Brain Encoder

v0.1-pretrain — Epoch 41, R ≈ 0.10 on naturalistic held-out data. Full-corpus retrain targeting TRIBE v2 Algonauts 2025 (mean R 0.2146) is in progress. This checkpoint is released for research access and reproducibility.

Model Summary

V5 is a 544M-parameter multimodal brain-encoding model that predicts whole-brain fMRI BOLD responses to naturalistic stimuli (video, audio, text, image) across the full fsaverage5 + subcortical mesh — 28,444 spatial units covering the entire cortical surface and 7 subcortical structures.

Primary output: full-mesh prediction (28,444 voxels) via a per-subject SRM adapter. 56 universal ROIs are derived from these voxels and used only for performance benchmarking — they are not the training target.

Key differentiators:

  • Full mesh output — 20,484 fsaverage5 cortical vertices + 7,960 Harvard-Oxford subcortical voxels
  • 21 feature blocks from 6 foundation model families (SigLIP2, DINOv2-G, CLIP-ViT-bigG, V-JEPA2, Whisper-Large-v3, Qwen3-Embed)
  • 56 universal ROIs used for eval only; 34/56 had ground-truth supervision in v0.1 (22 missing due to incomplete Schaefer-100 atlas coverage — fully fixed for v0.2 via HCP MMP1.0)
  • Zero-shot inference via subject dropout training — no per-subject fine-tuning required
  • Perceiver fusion trunk with reliability-gated block weighting handles missing modalities gracefully

Architecture 

Stimulus → [21 frozen feature extractors] → [Reliability-Gated Fusion]
  → [Perceiver Trunk (1024d, 8 latents)] + MAE + windowed temporal attention
  → [Per-subject SRM adapter → 28,444 voxels]  ← PRIMARY OUTPUT
  → [Universal ROI head (zero-shot) → 56 ROIs]  ← EVAL ONLY

Output space

Component Voxels Source
Cortical (fsaverage5) 20,484 Surface mesh — left+right hemisphere
Subcortical 7,960 Harvard-Oxford atlas in MNI 2mm
Total 28,444

Architecture details

Component Detail
Backbone dim 1024
Temporal mode Attention pooling over video/audio frames per TR
Windowed attention Enabled for naturalistic sequences
Subject adapter Per-subject SRM-style rotation: shared 1024d → 28,444 voxels
Subject dropout 10% (TRIBE-style, enables zero-shot)
Universal ROI head Linear: 1024d → 56 ROIs; zero-shot only; eval metric
Training precision bfloat16

Feature Blocks (v0.1 — 21 active)

Family Blocks Dim
SigLIP2-L siglip2_l7, siglip2_l14, siglip2_l22 1152
DINOv2-G dinov2_g_l10, dinov2_g_l20, dinov2_g_l32 1536
CLIP-ViT-bigG-14 clip_l12, clip_l24, clip_l38 1664
V-JEPA2 vjepa2_l10, vjepa2_l20, vjepa2_l32 1408
Whisper-Large-v3 whisper_l8, whisper_l16, whisper_l24 1280
HSEmotion hsemotion 512
Qwen3-Embed-8B (wired, extraction in progress for naturalistic data) 2560

Missing blocks at inference time are automatically zero-filled; the reliability-gated fusion learns to ignore them.

Training Data

Dataset Sessions Stimulus Type Role
CNeuroMod Friends (S1–6) 152 Naturalistic movie Training
BOLDMoments 509 Short video clips (3s) Training
HCP 7T Movie 61 Naturalistic movie (BBC) Training
NSD 284 Natural scene images (COCO) Training
Cowen-Keltner 40 Emotion video clips Training
THINGS-fMRI2 3 Object images Training
Narratives (ds003020) 345 Spoken narrative audio Holdout
Language fMRI 9 Language reading/listening Holdout

All data reprojected to fsaverage5 + subcortical (28,444 units). 87 training subjects total. Per-session beta series computed via least-squares single-trial GLMs.

Performance

v0.1 — Cowen-Keltner benchmark

⚠ This comparison is not apples-to-apples: V5 is in-sample on CK while TRIBE v2 is OOD. Fair OOD benchmark uses SpaceTop (ds005256) — OOD for both models — in progress.

Model Mean R (23 shared ROIs) V1 V2 V3 STG
V5 v0.1 (in-sample CK) +0.086 +0.107 +0.034 +0.058 +0.081
TRIBE v2 (OOD CK) +0.051 +0.121 +0.215 +0.116 +0.023

V5 v0.1 lacks V-JEPA2 temporal features for naturalistic training sets — V1–V3 is the primary gap. Resolved in v0.2.

ROI supervision in v0.1

Only 34/56 ROIs had ground-truth training signal due to Schaefer-100 atlas gaps:

Status ROIs
Supervised (34) V1, V2, V3, V4, V3A, MT, MST, V7, dlPFC, vlPFC, OFC, ACC, mPFC, FP2, IFG, IFGorb, STG, STS, MTG, PCC, mPFC_dmn, LP_R, HPC_L, HPC_R, AI, dACC, vmPFC, Amygdala_L/R, Caudate_L/R, Putamen_L/R, Thalamus
Unsupervised (22) V3B, LO1, LO2, IPS1–5, SPL1, hIP1–3, FFA-1, FFA-2, PPA, RSC, OFA, EBA, FP1, AG, LP_L, sgACC

In v0.2 all 56 ROIs will be derived from the full mesh betas using HCP MMP1.0 vertex labels, eliminating this gap. The primary training loss (voxel reconstruction on 28,444 units) is unaffected.

Algonauts 2025 targets

Model Mean R (CNeuroMod 4-subj)
TRIBE v2 (winner) 0.2146
MIRAGE (single model) 0.217
V5 v0.1 ~0.10 (pre-release)
V5 v1.0 target >0.2146

Usage

Install 

pip install torch transformers h5py numpy einops timm

Zero-shot full-mesh prediction 

import torch
from algonauts_proxy_eval import load_model

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model, meta, kind = load_model("best_joint_fullcorpus.pt", device)

# Provide whichever feature blocks you have — others auto zero-fill
blocks = {
    "siglip2_l22":  torch.randn(N, 1152, device=device, dtype=torch.bfloat16),
    "clip_l38":     torch.randn(N, 1664, device=device, dtype=torch.bfloat16),
    "vjepa2_l32":   torch.randn(N, 1408, device=device, dtype=torch.bfloat16),
}

with torch.no_grad():
    out = model(blocks, subject_key="zero_shot",
                dataset_key="unknown", force_zero_shot=True)

mesh_pred = out["voxels"]   # (N, 28444) — full cortical+subcortical mesh
roi_pred  = out["z_roi"]    # (N, 56)   — ROI summary for eval

Feature extraction from raw stimuli 

python extract_full_v45.py \
    --h5-dir  /path/to/subject_h5s/ \
    --stimulus-root /path/to/stimuli/ \
    --device cuda \
    --blocks siglip2_l7 siglip2_l14 siglip2_l22 \
             dinov2_g_l10 dinov2_g_l20 dinov2_g_l32 \
             clip_l12 clip_l24 clip_l38 \
             vjepa2_l10 vjepa2_l20 vjepa2_l32 \
             whisper_l8 whisper_l16 whisper_l24

Universal ROI Schema (56 regions — eval only)

Group ROIs
Early visual V1 V2 V3 V4 V3A V3B LO1 LO2 MT MST V7
Dorsal/parietal IPS1 IPS2 IPS3 IPS4 IPS5 SPL1 hIP1 hIP2 hIP3
Ventral visual FFA-1 FFA-2 PPA RSC OFA EBA
Prefrontal dlPFC vlPFC OFC ACC mPFC FP1 FP2
Language IFG IFGorb STG STS MTG AG
Default mode PCC mPFC_dmn LP_L LP_R
Cingulate/insula dACC sgACC vmPFC AI
Medial temporal HPC_L HPC_R
Subcortical Amygdala_L Amygdala_R Caudate_L Caudate_R Putamen_L Putamen_R Thalamus

ROI values are derived post-hoc from the full-mesh output by averaging voxels within each ROI's vertex mask (HCP MMP1.0 for cortical; Harvard-Oxford for subcortical). They are not directly predicted.

Limitations

  • V2/V3 temporal gap (v0.1): V-JEPA2 features not yet extracted for naturalistic training sessions — early retinotopic cortex predictions weaker than TRIBE v2. Resolved in v0.2.
  • 22/56 ROIs unsupervised (v0.1): Schaefer-100 atlas missing FFA/PPA/IPS/EBA vertex labels. ROI head for these is derived via graph propagation only. Resolved in v0.2 via HCP MMP1.0 rebuild.
  • Text modality absent (v0.1): Qwen3 text features not extracted for naturalistic data. Language ROIs driven by Whisper audio only.
  • Zero-shot vs per-subject: Zero-shot mode uses average-subject representation. Per-subject adapters substantially improve within-subject R but require training labels.
  • Canonical HRF assumed: Non-canonical hemodynamics (subcortical, clinical) may reduce accuracy.
  • Training/eval overlap: CNeuroMod Friends and BOLDMoments appear in both V5 and TRIBE v2 training. SpaceTop (ds005256) provides a clean OOD comparison.

Roadmap

Version Changes Target
v0.2 Full vjepa2 extraction → scratch retrain; HCP MMP1.0 vert2roi for all 56 ROIs V2/V3 parity with TRIBE
v0.3 SpaceTop + AOMIC + LPP + HCP-task; NeuroText projector; qwen3 text features +language ROIs
v1.0 Full 2,936-session corpus; fsaverage5 CNeuroMod 4-subj benchmark Beat TRIBE v2 (R > 0.2146)

Citation 

@misc{v5_neurodiffusion_2026,
  title     = {V5 NeuroDiffusion Brain Encoder: Multimodal fMRI Prediction
               across the Full Cortical and Subcortical Mesh},
  author    = {hamcoderfran},
  year      = {2026},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/hamcoderfran/neurodiffusion-v5},
  note      = {v0.1-pretrain checkpoint, epoch 41}
}

Part of the NeuroDiffusion Studio project.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
Feature Extraction
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support