Amphora NeuroText — Text & Audio → Brain ROI Activation
Three lightweight models trained on real naturalistic fMRI data that predict activation across 56 canonical brain ROIs from text or audio stimuli.
All models are trained in fsaverage5 + subcortical common space (28,444 voxels) and output predictions in the HCP MMP1.0 parcellation schema.
Model designed and built by Amphora, open source and free for commercial use.
Models
| File | Input | Val R | Best for |
|---|---|---|---|
text2roi_projector.pt |
Text (Qwen3-Embedding-4B, 2560d) | 0.212 | Semantic, narrative, conceptual stimuli |
text2roi_whisper.pt |
Audio (Whisper-large-v3, 1280d) | 0.413 | Speech, music, naturalistic audio |
text2roi_dual_v2.pt |
Text + Audio | 0.170 | Paired text+audio with modality-specialized heads |
Val R = mean Pearson correlation across 56 ROIs on held-out real fMRI data.
Quick Start
from predict import predict_text, predict_audio, predict_combined, top_rois
# Text → ROI (qwen3 projector)
roi_map = predict_text(
"watching a spider crawl toward me",
"text2roi_projector.pt"
)
print(top_rois(roi_map, n=5))
# [('Amygdala_L', 0.54), ('AI', 0.53), ('dACC', 0.55), ...]
# Audio → ROI (whisper projector)
roi_map = predict_audio("speech_clip.wav", "text2roi_whisper.pt")
# Text + Audio → ROI (dual-tower)
roi_map = predict_combined("narration", "audio.wav", "text2roi_dual_v2.pt")
CLI
python predict.py "remembering your childhood home" --checkpoint text2roi_projector.pt
python predict.py "jazz improvisation" --audio clip.wav --checkpoint text2roi_whisper.pt
python predict.py "a story about loss" --audio story.wav --checkpoint text2roi_dual_v2.pt
Installation
pip install torch transformers librosa numpy
# For brain visualization:
pip install nibabel nilearn mne
56-ROI Schema
Visual: V1 V2 V3 V4 V3A V3B LO1 LO2 MT MST V7 IPS1
FFA-1 FFA-2 PPA RSC OFA EBA
Parietal: IPS2 IPS3 IPS4 IPS5 SPL1 hIP1 hIP2 hIP3
Frontal: dlPFC vlPFC OFC ACC mPFC FP1 FP2
Language: IFG IFGorb STG STS MTG AG
Default mode: PCC mPFC_dmn LP_L LP_R HPC_L HPC_R
Salience: AI dACC sgACC vmPFC
Amygdala_L Amygdala_R
Subcortical: Caudate_L Caudate_R Putamen_L Putamen_R Thalamus
Training Data
All three models are trained exclusively on real fMRI recordings — no synthetic data, no simulated activations.
| Dataset | Subjects | Hours | Modality |
|---|---|---|---|
| Narratives (DS003020) | 345 runs | ~187k TRs | Audio stories |
| LPP (Little Prince) | 45 runs | ~136k TRs | Audio + text |
| HCP-task | 729 subjects | — | Task fMRI |
| AOMIC | — | — | Task fMRI |
| Clinical | 804 subjects | — | Resting / task |
Total: ~893k text-brain pairs (qwen3), ~373k audio-brain pairs (whisper). All preprocessed and projected to fsaverage5 + subcortical 28,444-voxel space.
Architecture
Text projector & Whisper projector (shared MLP class):
Linear(in_dim → 1024) → GELU → Dropout → LayerNorm
→ Linear(1024 → 512) → GELU → Dropout
→ Linear(512 → 56)
- Text projector:
in_dim=2560(Qwen3-Embedding-4B) - Whisper projector:
in_dim=1280(Whisper-large-v3 encoder mean pool)
Dual-tower (modality-specialized heads):
whisper_tower(1280→512) → whisper_head → 6 auditory/salience ROIs
qwen3_tower(2560→512) → qwen3_head → 13 semantic/DMN ROIs
[wt ⊕ qt] → fusion_head → 37 visual/parietal/subcortical ROIs
Training loss: pearson_r + 0.1 × MSE on real fMRI targets.
Augmentation: 8× noisy brain target copies per batch (~6.4M effective pairs/epoch).
Visualization
See the included Colab notebook (text2roi_colab_v2.ipynb) for full visualization
on a glass-brain and fsaverage5 surface using Nilearn/MNE.
Citation
If you use these models, please cite the underlying fMRI datasets:
- Narratives: Nastase et al. (2021). NeuroImage. doi:10.1016/j.neuroimage.2021.118023
- LPP: Dehghani et al. (2017). Language, Cognition and Neuroscience.
- HCP: Van Essen et al. (2013). NeuroImage. doi:10.1016/j.neuroimage.2013.05.041
Notes
- The whisper projector (R=0.413) is the strongest single model; it captures auditory cortex and salience network responses with high fidelity.
- The text projector (R=0.212) generalizes well to novel semantic content; best used for written stimuli, transcripts, or conceptual queries.
- The dual-tower (R=0.170) requires both modalities; useful when you have synchronized audio+transcript and want modality-partitioned predictions.
- Predictions are in z-score space relative to a per-subject resting baseline. Positive values = activation above baseline; negative = suppression.
- Downloads last month
- 17