han-ning/Brain-OF-Large

Brain-OF Large

Brain-OF: An Omnifunctional Foundation Model for fMRI, EEG and MEG
NeurIPS 2026

Introduction

Brain-OF Large is a foundation model jointly pretrained on fMRI, EEG, and MEG, aiming to learn richer representations through cross-modal synergy. ARNESS maps signals with varying channel/ROI counts and temporal resolutions to a shared latent space. Sparse MoE captures shared and modality-specific patterns, while DINT attention filters irrelevant context and integrates local and global dependencies. Masked Temporal-Frequency Modeling pretrains the model to reconstruct masked signals in the time and frequency domains. Pretrained on 37 public datasets with 5.87 million samples from 32,278 participants, the model supports unimodal and paired multimodal downstream tasks.

Model specifications

Property	Value
Total parameters	331M
Active parameters per token	150M
Backbone layers	24
Patch size	64 time points
Checkpoint format	PyTorch .pth; DDP loading path

Usage

After installing the environment, replace the checkpoint path below and run the example from the BrainOF repository root. It extracts a 512-dimensional embedding from a randomly generated signal:

import numpy as np
import torch
from datasets import DataTransform
from models.transformer import brainof_large_331M_A150M

model = brainof_large_331M_A150M(num_classes=0, layer_scale_type="per_channel")
checkpoint = torch.load(
    "/path/to/brainof_large_331M_A150M.pth", map_location="cpu", weights_only=False
)
weights = {
    k.removeprefix("encoder."): v
    for k, v in checkpoint["model"].items() if k.startswith("encoder.")
}
model.load_state_dict(weights)
model.eval()

# Generate a random signal: 19 channels, 2,000 time points
signal = np.random.default_rng(0).standard_normal((19, 2000)).astype(np.float32)
transform = DataTransform(signal_size=signal.shape, patch_size=64, max_seq_length=2048)
patches, mask, _ = transform(signal)
x = torch.tensor(patches, dtype=torch.float32).unsqueeze(0)
attn_mask = torch.tensor(mask, dtype=torch.bool).unsqueeze(0)

with torch.inference_mode():
    embedding = model(x, attn_mask)
print(embedding.shape)  # torch.Size([1, 512])

For real data, replace the random signal with a preprocessed NumPy array of shape (channels_or_ROIs, time_points). DataTransform normalizes, patches, and pads it. We recommend using at most 2,048 patch tokens per sample and the token count must not exceed max_seq_length (see Input Size and Tokenization).

Model family

Model	Total parameters	Active / token	Layers	Embedding dim.
	47.5M	21.5M	12	256
	331M	150M	24	512
	1.7B	500M	36	768
	9B	1.2B	48	1024

License

This repository and the model weights are licensed under the CC BY-NC 4.0.

Citation

@inproceedings{guo2026brainof,
  title     = {{Brain-OF}: An Omnifunctional Foundation Model for {fMRI}, {EEG} and {MEG}},
  author    = {Guo, Hanning and Bi, Hanwen and Abdellatif, Farah and Galbenus, Andrei and Shah, Jon. N. and Morrison, Abigail and Dammers, Jürgen},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026},
  url       = {https://arxiv.org/abs/2602.23410}
}