Cajal-1

A Whole-Cortex Encoding Model of Vision, Audition, and Language for In-Silico Neuroscience

Watch a film. See the whole cortex respond.

Cajal-1 is a brain encoding model. Give it any video with sound and it predicts, every 1.49 seconds, how each of the 20,484 points of the human cortical surface will respond — the same signal an fMRI scanner records. On the films Meta held out to test TRIBE v2, it predicts the real brain activity of the same viewers 16% better than TRIBE v2.

Left: Purkinje neurons of the pigeon cerebellum, drawn by Santiago Ramón y Cajal in 1899. Right: the areas Cajal-1 predicts respond most to faces, places and bodies, from nothing but the video a person is watching.

Try it in five minutes: open demo.ipynb in Google Colab (File → Upload notebook). It downloads the model and a sample film and draws the brain for you.

---

The story

In the 1890s Santiago Ramón y Cajal stained single neurons with silver and drew them by hand. His drawings showed that the brain is made of separate cells wired into circuits, and earned him the 1906 Nobel Prize. A century later fMRI lets us watch the whole living cortex at work, about once a second, while a person watches a film.

Cajal-1 learns the mapping between the two worlds: what is on screen and in the soundtrack, and what the cortex does about it. It was trained on about 3,100 fMRI recordings of people watching TV episodes, films and video clips, viewing photographs and listening to stories. Once trained, it needs no scanner: a video file is enough.

How it works

1. Three foundation models watch and listen. V-JEPA 2 reads motion and scenes, SigLIP reads objects and visual meaning, Whisper reads sound and speech (and, optionally, Qwen3 reads the words being said). One summary per modality is taken every 1.49 s, the rhythm of an fMRI scan.
2. A 12-layer causal transformer (218.8 M parameters) combines them, remembers the last 91 seconds, and learns the slow haemodynamic delay between an event on screen and the blood-flow response it causes.
3. A readout maps that state onto the 20,484 vertices of the fsaverage5 cortical surface: either the average viewer, or one of 41 individual training viewers.

Results

Test set: TRIBE v2's own held-out movie10 segments (The Bourne Supremacy, The Wolf of Wall Street, Hidden Figures, chunks 06 and 10), 40 fMRI runs from the CNeuroMod movie10 dataset. Score: Pearson r between predicted and measured activity per run, averaged over runs, after a 128 s high-pass filter. "Parcel" averages vertices within 1,000 Schaefer parcels first; "vertex" scores every vertex.

	Cajal-1	TRIBE v2	Ratio	Runs where Cajal-1 is ahead
Known viewers, parcel	0.183	0.157	1.16× [1.09, 1.25]	27 / 32
Known viewers, vertex	0.119	0.097	1.24× [1.16, 1.33]	29 / 32
Known viewers, parcel, both using an average-brain readout	0.176	0.157	1.12×
sub-06, parcel	0.180	0.142	1.27× [1.07, 1.45]	7 / 8

Brackets are 95% bootstrap intervals over runs (10,000 resamples). TRIBE v2 was run from its released checkpoint with its own video and audio feature extractors. Both models were scored on the same rows of the same runs by the same code. A three-seed ensemble of Cajal-1 reaches 1.19×; this release is the best single seed.

A note on sub-06: TRIBE v2 never saw this viewer, while Cajal-1 trained on sub-06's other recordings (never these test films). The sub-06 row is therefore a comparison on a viewer Cajal-1 knows, not a test on a new person.

These numbers are reproduced by the packaged code: tests/test_e1_parity.py re-scores all 40 runs and matches the original evaluation to within 0.001 per run (mean 0.1822 vs 0.1822).

What it learned about the brain

The brain has areas that prefer faces, places, bodies and written words. Neuroscientists find them with a "localizer": a sequence of pictures shown one at a time in the scanner. We showed Cajal-1 the same kind of localizer video (396 images, one every 8 seconds) and contrasted its predicted response to each category with the others. No brain data enters these maps; they are entirely the model's.

• Places and bodies are recovered: the strongest predicted responses fall in the parahippocampal place area and the extrastriate body area, where real localizers find them.
• Faces, half: the face-selective superior temporal sulcus is found, the fusiform face area only weakly.
• Words and objects are missed. Cajal-1 learned mostly from films, where isolated written words and objects on a blank background are rare and sound is always present.

This is an exploratory demonstration, not the formal comparison against measured localizer maps, which is still to be run.

What it is useful for

• Experiments before the scanner. Run a localizer, a new stimulus set or a film through the model to see which networks it engages, and refine a design before booking scanner time.
• Comparing stimuli. Ask which of two clips drives face, voice, motion or language areas more, at the scale of thousands of clips.
• Testing what makes a model brain-like. Swap or ablate feature streams and see how the predicted cortex changes.
• Teaching and outreach. Show students the cortex responding to a film, second by second, in colour.
• A starting point. Its shared readout and per-viewer readouts make it a base for fitting new people or new datasets.

It is a research model. It predicts an average viewer's response, not a specific person's, and it is not a diagnostic tool.

Quick start

from huggingface_hub import snapshot_download
import sys
path = snapshot_download("ffh92r32rm0/Cajal-1")
sys.path.insert(0, path)
import cajal

model = cajal.load(path)                       # GPU if available
brain = model.predict_video("my_video.mp4")    # features + prediction in one call

brain.plot()                  # film frames above the predicted cortex, TRIBE v2 style
brain.show(at=42)             # every side of the cortex at 42 s
brain.top_regions(at=42)      # the most active areas then, with plain-language names
brain.movie("brain.mp4")      # the film and the cortex side by side
brain.save("my_video_brain")  # .npz with the (time points, 20484) prediction and times

brain.pred is the raw (time points, 20484) array if you want to analyse it yourself. The lower-level path (cajal.features.extract_features then model.predict) is unchanged, for example to reuse features across runs or to choose a training viewer's readout per call.

Install: pip install -r requirements.txt (or pip install torch "transformers>=4.53" safetensors huggingface_hub nilearn av matplotlib h5py sentencepiece plotly, plus faster-whisper for the optional transcript stream and colorcet for the exact colour map).

The Colab notebook does all of this with a sample film (Tears of Steel, CC BY 3.0) or your own upload, and adds a rotatable 3D view and region time courses. A 3-minute clip takes about 5-10 minutes on a free T4 GPU.

Rendering. cajal.plotting draws the cortex the way the TRIBE v2 figures do: half-inflated fsaverage5 surface, sulcal shading and a fire colour scale, with smooth per-pixel shading. Each view is rasterised once, after which a frame takes tens of milliseconds, so whole-film brain movies are practical on a laptop.

Inputs and outputs

Input. Four feature streams on a 1.49 s grid, one row per fMRI volume. cajal.features computes them from a video file with the exact recipe used in training:

Stream	Model	Per row
vjepa2	V-JEPA 2 ViT-g (facebook/vjepa2-vitg-fpc64-256)	64 frames over the 4 s ending at the row; mean of blocks 20-29
siglip2	SigLIP so400m (google/siglip-so400m-patch14-384)	the frame at the middle of the row; pooled output
whisper_large	Whisper-large-v3 encoder	50 Hz frames averaged over the row
qwen3_embed (optional)	Qwen3-Embedding-4B on a faster-whisper transcript	each word with up to 1,023 words of context; layers 18-26; words averaged per row

Text is off by default because the scores above were measured without it.

Output. (T, 20484) float32: left hemisphere vertices 0-10,241, right hemisphere 10,242-20,483, in fsaverage5 order. Values are z-units: each vertex relative to its own mean and spread over the run, so a prediction says where and when activity rises and falls. Rows with no valid input (past the end of the video) are NaN.

Context. The model uses up to 61 preceding rows (91 s) of context. The first 61 rows of a clip are predicted from the context available, which is less accurate; predict(..., include_warmup=False) returns NaN for them instead.

Readout. subject="average" (default) uses the shared population readout. Readouts learned for 41 individual training viewers or pooled datasets are also available (model.subjects, e.g. "movie10:01").

Model

• 218.8 M parameters. Per-stream projections, a learned haemodynamic response per dataset, and a 12-layer causal transformer trunk (width 1,152, 12 heads, attention reach 61 rows) with rotary positions in seconds.
• A shared low-rank readout to 20,484 vertices, plus a small residual readout per training viewer.
• Trained 36,180 steps with a mean-squared-error loss on z-scored, 128 s high-pass-filtered targets, learning rate 5e-4 with plateau decay, weight decay 0.1, dropout 0.2 and 30% per-modality dropout. This checkpoint is a weighted average of four late checkpoints of one seed.

Training data

About 3,100 fMRI recordings from ten public datasets, all projected to fsaverage5:

Dataset	Stimuli	Recordings
CNeuroMod Friends	TV episodes	1,021
HCP 7T movie	film clips	725
BOLD Moments	3 s video clips	509
Wen et al. 2017	natural video	257
CNeuroMod movie10	feature films (the TRIBE v2 test segments excluded)	251
Natural Scenes Dataset	images	244
BOLD5000	images	54
Natural Object Dataset	images	47
Lebel et al. 2023	spoken stories	9
Cowen-Keltner emotion clips	short emotional videos	64

No run, film or segment from the test set above was used in training.

Limitations

• Category areas. Places and bodies land in the right areas; the fusiform face area is found only weakly, and the visual word-form area and object-selective lateral occipital cortex are not recovered (see above).
• Subcortex. The checkpoint has subcortical output columns, but they were not trained; this release returns cortex only.
• Population. Training viewers were healthy adults. Predictions describe an average viewer, not a specific person, unless a training viewer's readout is chosen.
• Not a diagnostic tool. This is a research model.

Licensing and credits

Before using or redistributing this model, check the terms of:

• the training datasets, including the HCP and CNeuroMod data use terms and the NSD terms of use;
• each feature model (V-JEPA 2, SigLIP, Whisper, Qwen3-Embedding, faster-whisper).

Images in this card (all from Wikimedia Commons):

• Santiago Ramón y Cajal, portrait, 1899 (Clark University; restored): public domain.
• Cajal, drawings of the human cerebral cortex and of Purkinje neurons (1899): public domain.
• MRI scanner photograph (Siemens Biograph mMR 3T) by Jeong Kyu Park, Sung Kyu Kim, Ihn Ho Cho and Eun Jung Kong, CC BY 4.0.
• Tears of Steel frames © Blender Foundation (mango.blender.org), CC BY 3.0. The same film is the notebook's sample.

Brain figures are rendered from Cajal-1's own outputs by tools/make_readme_figures.py.

Testing

Every release file was checked before publication. Results are in tests/results/.

Test	What it checks	Result
test_e1_parity.py (GPU, bf16)	the 40 TRIBE v2 test runs re-scored from stored features with the packaged predictor	mean parcel r 0.1822 vs 0.1822 published; worst run off by 0.0009
test_e1_parity.py (CPU, fp32, from the release files)	the same, loading model.safetensors	0.1822 vs 0.1822; worst run off by 0.0007
test_feature_parity.py	features extracted from the raw film by cajal.features vs the features the model was scored on (30 rows)	cosine 1.0000 (V-JEPA 2), 0.9993 (SigLIP), 1.0000 (Whisper); predicted maps r = 0.9999
test_text_stream.py	transcript + Qwen3 text stream on 2 min of film dialogue	299 words, 55 of 80 rows with speech; runs end to end
test_unit.py	26 checks: clip lengths from 1 row up, window coverage, missing streams, rows past the end of a video, readout choices, input validation, plotting and the renderer, the Prediction object	all pass (Windows; the 25 earlier checks also on Linux)
run_notebook.py	demo.ipynb executed top to bottom (CPU) on 45 s of Tears of Steel: download, features, prediction, mosaic, 3D view, region plot, movie, saved files	v1.1 notebook re-run: PASS. All 9 code cells executed with no errors (30 s clip on a Linux CPU machine; 3 figures, 3D view, movie, saved prediction); output in tests/results/demo_executed.ipynb

Not tested here: a real Colab session. The notebook was executed with the same code on a Linux CPU machine. On a GPU without native bfloat16 (such as Colab's T4), feature extraction runs in float32. That is the precision used in the feature parity test above.

What's next: Cajal-2

Cajal-2 is being prepared on the same architecture with a larger and cleaner corpus: hundreds more Friends episodes of the same viewers, a rebuilt natural-speech dataset covering ten listeners, further recordings of the same people (entering only if they pass registered quality gates), and fixes from a full data-quality audit. Its evaluation plan and validation set are fixed in advance, and it will be scored the same way as Cajal-1.

Files

File	Content
model.safetensors	weights (218.8 M parameters, float32)
norm_stats.safetensors	feature normalisation statistics
config.json	architecture, readout keys, window settings
cajal/	model code, feature extraction, plotting, brain atlases
demo.ipynb	Colab demo
assets/readme/	the figures in this card
tests/	unit tests, the 40-run TRIBE v2 test-set reproduction, feature-extraction checks
tools/	checkpoint export, notebook generation, README figures