--- license: cc-by-nc-nd-4.0 --- # BrainLM model The pretrained model of Brain Language Model (BrainLM) aims to achieve a general understanding of brain dynamics through self-supervised masked prediction. It is introduced in [this paper](https://www.biorxiv.org/content/10.1101/2023.09.12.557460v1) and its code is available at [this repository](https://github.com/vandijklab/BrainLM) ## Model Details ### Model Description We introduce the Brain Language Model (BrainLM), a foundation model for brain activity dynamics trained on 6,700 hours of fMRI recordings. Utilizing self-supervised masked-prediction training, BrainLM demonstrates proficiency in both fine-tuning and zero-shot inference tasks. Fine-tuning allows for the prediction of clinical variables and future brain states. In zero-shot inference, the model identifies functional networks and generates interpretable latent representations of neural activity. Furthermore, we introduce a novel prompting technique, allowing BrainLM to function as an in silico simulator of brain activity responses to perturbations. BrainLM offers a novel framework for the analysis and understanding of large-scale brain activity data, serving as a “lens” through which new data can be more effectively interpreted. - **Developed by:** [van Dijk Lab](https://www.vandijklab.org/) at Yale University - **Model type:** ViTMAE - **License:** [![Preprint License: CC BY-NC-ND 4.0](https://img.shields.io/badge/License-CC_BY--NC--ND_4.0-lightgrey.svg)](https://creativecommons.org/licenses/by-nc-nd/4.0/) ### Model Sources [optional] - **Repository:** https://github.com/vandijklab/BrainLM - **Paper:** https://www.biorxiv.org/content/10.1101/2023.09.12.557460v1 - **Demo [optional]:** [More Information Needed] ## Uses BrainLM is a versatile foundation model for fMRI analysis. It can be used for: - Decoding cognitive variables and mental health biomarkers from brain activity patterns - Predicting future brain states by learning spatiotemporal fMRI dynamics - Discovering intrinsic functional networks in the brain without supervision - Perturbation analysis to simulate the effect of interventions on brain activity ### Out-of-Scope Use Currently, this model has been trained and tested only on fMRI data. There are no guarantees regarding its performance on different modalities of brain recordings. ## Bias, Risks, and Limitations - The model was trained only on healthy adults, so may not generalize to other populations - The fMRI data has limited spatial-temporal resolution and BOLD signals are an indirect measure of neural activity - The model has only been evaluated on reconstruction and simple regression/classification tasks so far - Attention weights provide one method of interpretation but have known limitations ### Recommendations - Downstream applications of the model should undergo careful testing and validation before clinical deployment. - Like any AI system, model predictions should be carefully reviewed by domain experts before informing decision-making. ## How to Get Started with the Model Use the code below to get started with the model. ## Training Details ### Data Data stats: - UK Biobank (UKB): 76,296 recordings (~6450 hours) - Human Connectome Project (HCP): 1002 recordings (~250 hours) Preprocessing Steps: - Motion Correction - Normalization - Temporal Filtering - ICA Denoising Feature Extraction: - Brain Parcellation: AAL-424 atlas is used to divide the brain into 424 regions. - Temporal Resolution: ~1 Hz with 0.735s for UKB and 0.72s for HCP. - Dimensionality: 424-dimensional time series per scan. Data Scaling - Robust scaling was applied, involving the subtraction of the median and division by the interquartile range across subjects for each parcel. Data split: - Training data: 80% of the UKB dataset - Validation data: 10% of the UKB dataset - Test data: 10% of the UKB dataset and HCP dataset ### Training Procedure BrainLM was pretrained on fMRI recordings from the UK Biobank and HCP datasets. Recordings were parcellated, embedded, masked, and reconstructed via a Transformer autoencoder. The model was evaluated on held-out test partitions of both datasets. Objective: Mean squared error loss between original and predicted parcels Pretraining: - 100 epochs - Batch size 512 - Adam optimizer - Masking ratios: 20%, 75% and 90% Downstream training: Fine-tuning on future state prediction and regression/classification clinical variables #### Metrics In this work, we use the following metrics to evaluate the model's performance: - Reconstruction error (MSE between predicted and original parcel timeseries) - Clinical variable regression error (e.g. age, neuroticism scores) - Functional network classification accuracy [More Information Needed] ### Results [More Information Needed] #### Summary ## Model Examination [optional] [More Information Needed] ## Technical Specifications [optional] ### Model Architecture and Objective [More Information Needed] **BibTeX:** ```bibtex @article{ortega2023brainlm, title={BrainLM: A foundation model for brain activity recordings}, author={Ortega Caro, Josue and Oliveira Fonseca, Antonio Henrique and Averill, Christopher and Rizvi, Syed A and Rosati, Matteo and Cross, James L and Mittal, Prateek and Zappala, Emanuele and Levine, Daniel and Dhodapkar, Rahul M and others}, journal={bioRxiv}, pages={2023--09}, year={2023}, publisher={Cold Spring Harbor Laboratory} } ```