encoder-things-meg

These models predict the MEG response to an image as recorded in THINGS-MEG: 271 sensors from 0 to 1.29 s after image onset, sampled at 100 Hz, each sensor z-scored against its pre-stimulus baseline. By default the prediction is the average over the dataset's four participants, the best guess for someone who was not in the dataset; each participant's own prediction is also available. Four variants trade accuracy for speed. All were fitted on the 22,248 THINGS-MEG training images and scored on its 200 test images, which are new images of concepts that also appear in training.

Quick start

pip install torch transformers pillow

from PIL import Image
from transformers import AutoModel

model = AutoModel.from_pretrained(
    "epfl-neuroai/encoder-things-meg", revision="v1", trust_remote_code=True
).to("cuda")

images = [Image.open("dog.jpg"), Image.open("teapot.jpg")]
meg = model(images)  # (2, 271, 130): average over participants
meg_all = model(images, subjects="all")  # (2, 4, 271, 130)
meg_one = model(images, subjects=["P2"])  # (2, 1, 271, 130)

model.subjects  # "P1" ... "P4"
model.sensors  # sensor names, in output order
model.times  # 0.0, 0.01, ..., 1.29 seconds after image onset

This loads the default variant, efficient. Add variant="tiny", "cnn" or "accurate" for another; only the selected variant's weights are downloaded (for accurate, the Qwen weights follow on the first call). The model takes a PIL image or a list of them and returns a float32 tensor; batch_size (default 32) sets how many images go through the GPU at once. For other recordings, model.features(images) returns the readout input, 128 values per image (256 for accurate), on which you can fit your own linear readout.

trust_remote_code=True runs this repository's Python files on your machine: modeling.py, the model in plain PyTorch, and two short transformers wrappers. Review them and pin revision. Without transformers (pip install torch pillow safetensors huggingface-hub):

import importlib.util
from huggingface_hub import snapshot_download

code = snapshot_download("epfl-neuroai/encoder-things-meg", revision="v1", allow_patterns=["*.py"])
spec = importlib.util.spec_from_file_location("encoder_things_meg", f"{code}/modeling.py")
modeling = importlib.util.module_from_spec(spec)
spec.loader.exec_module(modeling)
model = modeling.load("epfl-neuroai/encoder-things-meg", revision="v1", device="cuda")

Variants

variant	image networks	test: own readout	average readout	new participant	download	images/s	GPU memory
efficient (default)	ResNet-50 + ViT-S	0.504	0.363	0.246	0.96 GB	119	1.4 GiB
tiny	ResNet-18 + ViT-S	0.496	0.357	0.246	0.55 GB	127	0.9 GiB
cnn	ResNet-152	0.509	0.367	0.246	1.5 GB	112	1.9 GiB
accurate	ResNet-152 + DINOv2-B + Qwen3-VL-2B vision	0.544	0.394	0.262	4.4 GB + 4.3 GB from Qwen	16	6.7 GiB

The three smaller variants are close to each other; tiny is the smallest and fastest. accurate scores best everywhere, at a seventh to an eighth of the speed, and downloads the Qwen3-VL-2B weights from Qwen's repository at a pinned revision. Speed is end to end from PIL images on one NVIDIA A100 (batch 32).

Scores. Each score is the Pearson correlation between predicted and recorded responses across the 200 test images, per sensor and time point, divided by its noise ceiling, averaged over the points whose ceiling exceeds 0.1 and then over participants. Each participant is scored with their own readout, with the average readout (the default output), or, for "new participant", with the average of the other three. On the THINGS-MEG benchmark leaderboard, the best model without fine-tuning, Qwen3-VL-2B, scores 0.463; that score is a mean over seven sensor groups, so it is not directly comparable to the scores above.

Matching your recordings

The output follows the preprocessing of THINGS-MEG (Hebart et al., 2023), as used by its benchmark:

1. Band-pass the continuous recording at 0.1-40 Hz.
2. Cut epochs from -0.1 to 1.3 s around image onset (THINGS-MEG used a photodiode) and z-score each epoch and sensor against its 100 ms before onset.
3. Resample to 100 Hz and keep 0 to 1.29 s (130 samples). Average each image's repetitions.
4. Keep the sensors you share with model.sensors (CTF axial gradiometers), in the same order. On other MEG systems no sensors correspond; fit a readout on model.features(images) instead.
5. Check for a constant delay between your triggers and the image.

Why the score is r, not R²

The score asks whether, at each sensor and time point, the predictions rise and fall across images with the recordings. Pearson r measures this and ignores offset and scale. Much of the MEG signal is the evoked response that every image produces. Its size, its timing and how it falls on the sensors differ between people, partly because each head sits differently in the helmet. A participant's own readout captures most of this; the average readout, used for someone new, does not, and R² punishes that even when the image-to-image pattern is right.

For efficient on the test images, with the same processing applied to predictions and recordings (here r without the ceiling, to compare with R²):

processing	own readout: r	own readout: R²	new participant: r	new participant: R²
none	0.207	0.060	0.104	-0.29
mean over the test images subtracted	0.207	0.066	0.104	-0.005
mean response to the training images subtracted	0.207	0.056	0.104	-0.015
linear detrend of each epoch	0.168	0.038	0.070	-0.26
1 Hz high-pass of each epoch	0.129	0.026	0.051	-0.19

To compare amplitudes, subtract from each side its own average response, over your images or over a separate set. This removes the shared evoked response and leaves r unchanged; for a new participant it brings R² to about zero, not above it. Detrending or high-passing the epochs does not help here: it removes slow components that differ between images and lowers r.

Other datasets

We applied the models without refitting to NOD-MEG (Zhang et al., 2025): 30 participants, also on a CTF system (269 sensors shared with THINGS-MEG), each shown a different set of ImageNet images once, 57,000 images in all. We used the authors' preprocessed epochs as released (cleaned with ICA and baseline-corrected), resampled to 100 Hz, and compared them with the default output. One constant delay was chosen on half of the data and scored on the other half, and then the halves were swapped. Scores cover 0 to 0.7 s, the length of NOD's epochs.

	efficient	tiny	cnn	accurate
class averages, r / ceiling	0.622	0.621	0.616	0.630
single trials, r (mean of sub-01 to sub-09, who saw 4,000 images each)	0.0378	0.0375	0.0369	0.0382

With every image shown once, single trials are noisy, and single-trial r stays small. Averaging the trials of each of the 1,000 ImageNet classes over all participants (about 57 per class), and the predictions in the same way, gives a signal with a measurable split-half ceiling. The class scores cover the 6,100-6,400 of 18,830 sensor-time points whose ceiling exceeds 0.1. The recordings lagged the predictions by 10-20 ms.

Why DINOv2 and not DINOv3

We evaluated DINOv3 (ViT-B/16 and ViT-L/16) in the same parts of the models, with the same layer search and fine-tuning. It did not help: in ensembles built the same way, before the final refit, scores on held-out training images were unchanged, and on the test images (own readout) DINOv3-B and -L reached 0.537 and 0.539, against 0.537 with DINOv2-B. DINOv2 is released under Apache 2.0 and can be downloaded freely, while DINOv3's weights are distributed under the DINOv3 License, with gated access on Meta's repositories and conditions that would carry over to these models.

How the encoders were built

• MEG data: THINGS-MEG (4 participants; 22,248 training images shown once, 200 test images shown 12 times), the authors' preprocessed epochs, processed as in "Matching your recordings".
• Image networks: adversarially trained ResNets from Gokce & Schrimpf (2025), ViT-S/16 and DINOv2-B from timm, and the Qwen3-VL-2B vision encoder, each cut at an intermediate layer (listed per variant in config.json). The last blocks before that layer were fine-tuned to the training images' MEG with low-rank updates (LoRA).
• Readout: each member's feature map is average-pooled at several grid sizes and compressed to its top 1,024 principal components; for each member and participant, a ridge regression with a penalty per value maps the components to the 35,230 sensor-time values. The members' predictions are combined with fixed weights chosen on held-out training images, and the final models were refitted on all training images. The four participants' readouts were then reduced to a shared basis of rank 128 (256 for accurate, ranks chosen on held-out images). This makes the files 4 to 10 times smaller without lowering the scores.
• Preprocessing: ResNets and ViTs see a 224 x 224 center crop after resizing the shorter side to 256 pixels; Qwen sees a square center crop resized to 500 pixels, then to its 512 x 512 input.
• Export check: in float32 with TF32 off, the released files reproduce the original test predictions, with the shared-basis projection applied, to within 2e-5 of their standard deviation (1.3e-3 for accurate, whose Qwen blocks run in bfloat16).

Limitations

• Fitted on four adults, one lab and one paradigm: images shown for 500 ms, the next one at least 1.3 s after onset. The predicted waveform includes the response to the image's offset, so for other durations it will not match after about 0.5 s. New participants, head positions, systems and paradigms lower accuracy; the average readout is the best starting point.
• Tested on object images only (THINGS, and ImageNet through NOD-MEG), not on scenes, text or close-up faces.
• The ResNets and ViT-S were trained on ImageNet, licensed for non-commercial research; check this for commercial use.

License

Apache 2.0 for the weights and code in this repository. THINGS-MEG is CC0; the THINGS images are not redistributed. ViT-S and DINOv2 (timm) and Qwen3-VL-2B are Apache 2.0; the Qwen weights are downloaded from Qwen's repository, not redistributed here.

Citation

Please cite the benchmark and the ResNet checkpoints:

@inproceedings{gokce2026multimodal_brain_scaling,
  title     = {Multimodal Scaling Laws for Task \& Data-Optimized Models of Visual Cortex},
  author    = {Abdulkadir Gokce and Yingtian Tang and Martin Schrimpf},
  booktitle = {Forty-third International Conference on Machine Learning},
  year      = {2026},
  url       = {https://openreview.net/forum?id=OQ6jQHJPTT}
}

@inproceedings{gokce2025scalingprimatevvs,
  title     = {Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Stream},
  author    = {Abdulkadir Gokce and Martin Schrimpf},
  booktitle = {Forty-second International Conference on Machine Learning},
  year      = {2025},
  url       = {https://openreview.net/forum?id=WxY61MmHYo}
}

Also cite THINGS-MEG (Hebart et al., 2023, eLife 12, e82580), THINGS (Hebart et al., 2019, PLOS ONE 14, e0223792), DINOv2 (Oquab et al., 2024, TMLR), ViT-S (Steiner et al., 2022, TMLR) and Qwen3-VL for accurate (Qwen Team, 2025, arXiv:2511.21631). Transfer dataset: NOD-MEG (Zhang et al., 2025, Scientific Data 12, 857).