Self-Supervised Learning Of Brain Dynamics From Broad Neuroimaging Data
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Self-supervised learning techniques are celebrating immense success in naturallanguage processing (NLP) by enabling models to learn from broad language dataat unprecedented scales.
Here, we aim to leverage the success of these techniquesfor mental state decoding, where researchers aim to identify speciﬁc mental states(e.g. , the experience of anger or joy) from brain activity.
To this end, we devisea set of novel self-supervised learning frameworks for neuroimaging data basedon prominent learning frameworks in NLP.
At their core, these frameworks learnthe dynamics of brain activity by modeling sequences of activity akin to how NLPmodels sequences of text.
We evaluate the frameworks by pre-training models ona broad neuroimaging dataset spanning functional Magnetic Resonance Imagingdata from 11 , 980 experimental runs of 1 , 726 individuals across 34 datasets andsubsequently adapting the pre-trained models to two benchmark mental statedecoding datasets.
The pre-trained models transfer well, generally outperformingbaseline models trained from scratch, while models trained in a learning frameworkbased on causal language modeling clearly outperform the others.
Introduction
In mental state decoding (1), researchers aim to identify certain mental states (e.g., deciding to acceptor reject a gamble or the experience of happiness or fear) from brain activity.
To this end, researcherstrain predictive models to correctly identify (i e., decode) these mental states from measured brainactivity.
At ﬁrst sight, this approach seems straightforward, yet, researchers interested in trainingmental state decoding models are faced with the challenge that individual neuroimaging datasetsare generally of high-dimension and small sample-size, such that individual samples often comprisemany hundred thousand dimensions while each dataset includes only a few hundred samples for eachof tens to hundreds of individuals.
In this setting, predictive models are prone to overﬁtting, therebynot generalizing well to new datasets, which severely limits any generalizable insights that can begained from training these models about the studied mental states and brain activity.
With the advent of open science, neuroimaging research has recently entered a big data era, asindividual researchers more frequently share their collected datasets publicly (2; 3).
In addition,several efforts have been made to standardize the data organization (4) and preprocessing (5) ofneuroimaging data.
Together, these developments open up new possibilities for the application pretraining in neuroimaging at scale by training models on large and diverse sets of public neuroimagingdata so that the resulting models can be adapted well to the datasets of individual researchers, withoutthe need for much new data.
A wealth of empirical evidence has demonstrated that models pre-trained on large neuroimagingdatasets generally achieve better mental state decoding performance on new data, while requiring lesstraining time and data, when compared to models trained from scratch (e.g., 6; 7; 8; 9).
Yet, much of this work has either pre-trained models on large but homogenous datasets, such as data from manyindividuals who all perform the same few tasks at the same few acquisition sites (10; 7), which canlimit the generalizability of the resulting models due to potential systematic biases in homogeneousdatasets (e.g., speciﬁc to the acquisition site or experimental paradigm; 11; 12; 13; 14; 15), or requiredhighly-preprocessed input data (e.g., statistical maps summarizing the measured sequences of brainactivity; 16).
In addition, researchers often use supervised pre-training techniques by training modelsto decode speciﬁc mental states that are assigned to each sample of the data (6; 7; 10; 17). Whilesupervised training can be fruitful within individual neuroimaging datasets, it is difﬁcult to extend tomany datasets, as neuroimaging researchers often do not use standardized labeling schemes for mentalstates when assigning labels to the mental states of their experiments. This lack of standardizationmakes it challenging to determine whether two datasets contain the same or distinct sets of mentalstates (for a detailed discussion of this issue, see 18; 19).
Here, we aim to overcome these limitations by leveraging recent advances in self-supervised learningto pre-train deep learning (DL) models on broad, public neuroimaging data.
In particular, we takeinspiration from natural language processing (NLP), where the application of self-supervised learninghas led to unprecedented breakthroughs in the adaptive capabilities of models trained on public textdata (e.g., 20; 21; 22; 23).
We test whether modeling sequences of brain activity akin to sequencesof text enables the learning of generalizable representations of the dynamics of brain activity.
Tothis end, we represent sequences of brain activity akin to the embedded representation of words insequences of text (24; 25) by describing each time point as a vector in which each dimension reﬂectsthe activity of a speciﬁc functional brain network at this point in time.
Contributions.
We devise novel self-supervised learning frameworks for neuroimaging data basedon prominent learning frameworks in NLP, namely, sequence-to-sequence autoencoding (26), causallanguage modeling (20), and masked language modeling (21).
We pre-train DL models with theseframeworks on a large-scale upstream dataset comprising functional Magnetic Resonance Imaging(fMRI) data from 11 , 980 experimental runs of 1726 individuals in 34 datasets.
To our knowledge,this represents one of the broadest neuroimaging datasets used for pre-training to date, spanning awealth of experimental conditions and a diverse set of acquisition sites.
We evaluate the downstreamadaptation performance of the pre-trained models in two benchmark mental state decoding datasetsand show that models pre-trained in a causal sequence modeling framework clearly outperform theothers, while the pre-trained models generally also outperform baseline models trained from scratch.
We make our training data and pre-trained models publicly available (see Appendix A.4).
Methods overview
2.1 Input data: from brain volumes to brain networks
FMRI data are conventionally represented in four dimensions, such that the measured blood-oxygenlevel-dependent (BOLD) signal is indicated in temporal sequences S
= { V 1 , ..., V t } of 3-dimensionalvolumes V 2 R x ⇥
y ⇥ z that show the BOLD signal intensity for each spatial location of the brain (asindicated by the three spatial dimensions x , y , and z ).
However, due to the strong spatial correlationsof brain activity, the measured BOLD signal is often summarized differently by representing it asa set ⇥ =
{ ✓ 1 , ..., ✓ n } of n brain networks ✓ , each describing the BOLD signal for some subset ofvoxels v x,y,z ⇢ V with correlated activity patterns.
Here, we utilize such a functional parcellation of brain activity to represent the measured BOLDsignal, namely, Dictionaries of Functional Modes (DiFuMo), which were recently proposed by(27) and learned across millions of fMRI volumes.
DiFuMo deﬁnes ⇥ through a sparse dictionarymatrix D 2 R p ⇥ n containing n networks, each deﬁned by p values (one for each voxel v 2 V ,ﬂattened over the three spatial dimensions).
The BOLD signal of volume V t is then described asa linear combination of these networks by the use of weights ↵ t 2 R n , such that V t
= D ↵ t and↵ t = D † V t , where D † indicates the pseudo-inverse of D : D † =
( D T D )  1 D T 2 R n ⇥ p .
We utilize aparcelation with n = 1024 networks, such that the resulting BOLD parcelation X 2 R t ⇥ n describesthe measured BOLD signal of each of these networks at each time point t of the input sequence S .
This representation of the input data is thereby analogous to the embedded representation of sentencesin NLP (24; 25), where each word k is represented as some n -dimensional vector k 2 R n .
2.2 Self-supervised learning frameworks: modeling sequences of brain activity akin tosequences of text
We ﬁrst establish several commonalities among the frameworks (see Fig. 1), before describing themin more detail.
2.2.1 Commonalities among frameworks
First, each framework takes as input a parcelated BOLD sequence X 2 R t ⇥ n (with t time pointsfor n networks) as well as the corresponding sampling frequency (i.e., repetition time or TR) of theunderlying fMRI sequence (typically in the range of 0 .
7 s to 4 s).
Second, all frameworks linearly project the parcelated BOLD sequence X into an embeddingrepresentation E X 2 R t ⇥ e : E Xt,e = b e +
P n w n,e X t,n , where w 2 R n ⇥ e and b 2 R e indicateweights and biases, thereby reducing the dimensionality of the input: R t ⇥ n !
R t ⇥ e (if e < n ).
Third, to account for any differences in the sampling frequency of the fMRI input sequences, theframeworks add one of r learnable TR embeddings E TR 2 R r ⇥ e to each time point t in the BOLDembedding E X .
Each TR embedding thus corresponds to one possible time point k of the input.
We set r = 1500 with a time increment of 0 .
2 s between any subsequent TR embeddings, such thatk 2 0 , 0 .
2 , ..., 300 s.
For example, for a sequence of 21 s with a TR of 0 .
7 s, we ﬁrst round eachtime point t 2 0 , 0 .
7 , ..., 21 to the closest k and then add the corresponding E TRk to E Xk .
We alwayslabel time points relative to the beginning of an input sequence, such that time point 0 is alwaysassigned to the ﬁrst sequence element.
Importantly, TR embeddings carry different informationthan the position embeddings used in transformer models (28), as the same position in an input cancorrespond to different time points, depending on the fMRI’s sampling frequency.
Note that we usethe same learnable TR embedding for any other embeddings that are inserted into an input sequenceduring training (see sections 2.2.3 and 2.2.4 and Fig. 1).
Fourth, all frameworks utilize some standard DL model f ( · ) that takes E in as input and predicts acorresponding output sequence f
( E in ) 2 R t ⇥ e .
Fifth, during upstream learning, the output sequence f ( E in ) is linearly projected back to the dimensionality of the parcelated BOLD sequence X , such that ˆ X t,n = b n +
P e w e,n f ( E in ) t,e , where wand b again indicate weights and biases.
This step is crucial, as all proposed learning frameworks involve reconstructing X (or parts of X ) after some of its elements have been masked or it hasbeen encoded in a lower-dimensional representation.
To measure reconstruction performance, weutilize the mean absolute difference between the parcelated BOLD sequence X and the reconstructed
BOLD sequence ˆ X : L rec = 1 n ( J )
P j 2 J | X j  ˆ X j | , where J indicates the set of elements that is tobe reconstructed and n ( J ) the number of elements in J .
Sixth, for downstream adaptation to a new mental state decoding task, the frameworks utilize asimple decoding head p ( · ) , composed of a dense hidden layer with e model units (one for eachembedding dimension, with tanh activation) as well as a softmax output layer with one model uniti for each considered mental state in the data.
p ( x ) i therefore indicates the probability that input xbelongs to mental state i .
Consequently, all frameworks treat the decoding task as a multinomialclassiﬁcation problem and use a standard cross entropy loss objective: L cls =
P i
y
i log p ( x ) i ,where y i indicates a binary variable that equals 1 if i is the correct mental state and 0 otherwise.
2.2.2 Autoencoding
The autoencoding framework is inspired by recurrent sequence-to-sequence neural machine translation models (e.g., 26).
These models take as input some text sequence, encode this sequence in alower-dimensional representation, and predict an output sequence from the lower-dimensional encoding (e.g., a translation of text to another language).
Accordingly, the autoencoding framework uses astandard recurrent autoencoder architecture f a ( · ) , with an encoder and decoder part, each comprisedof a stack of long short-term memory (LSTM; 29) units (see Fig. 1).
The encoder e ( · ) takes E inas input and encodes it in a lower-dimensional representation h 2 R e .
The decoder d ( · ) then takesh as input and predicts a corresponding output sequence d ( h ) 2 R t ⇥ e , such that f a ( E in ) =
d ( h ) .
Our encoder is composed of a stack of bidirectional LSTM units, whereas the decoder is buildfrom a stack of unidirectional LSTM units.
Note that we apply residual connections between the LSTM units in a stack: Let LSTM i and LSTM ( i +1) be the i -th and ( i + 1) -th
LSTM layer ina stack (with weights w i and w i +1 ).
At the t -th step of an input sequence, the input and outputsof layers i and ( i + 1) are deﬁned as: h it , c it
= LSTM i ( h it  1 , c it  1 , x i  1 t  1 ; w i ) ; x it
= h it + x i  1 t  1 ;h i +1 t
+1 ,
c i +1 t
+1 =
LSTM i +
( h i +1 t , c i +1 t , x it ; w i +1 ) , where x it 2 R e represents the input to LSTM i atstep t , while h
it 2 R e and c it 2 R e represent the LSTM ’s hidden and cell states respectively.
For theencoder, we average the hidden states h z 2 i t and h
y 2 i t of the two LSTM units z and y contained inany bidirectional LSTM layer i at each time point t : h it = 0 .
5( h z 2 i t + h
y 2 i t ) .
The encoded sequenceembedding h that is forwarded to the decoder is then deﬁned as the mean of the ﬁnal hidden states ofthe two LSTM units contained in the encoder’s ﬁnal layer.
Upstream learning.
During pre-training, we jointly train the encoder and decoder to reconstructthe parcelated BOLD sequence X by minimizing its mean absolute difference to the reconstructedsequence ˆ X (see section 2.2.1): min 1 t
P ti =1 1 e
P ej =1 | ˆ X i,j  X i,j | .
Importantly, we apply teacherforcing (30) during upstream learning by using E in in 50% of the cases as each next input for thedecoder, instead of using the decoder’s own preceding output.
Adaptation.
To adapt the autoencoding framework to a new decoding task, we forward h as inputto a corresponding decoding head p ( · ) (see section 2.2.1), thereby omitting the decoder d ( · ) .
2.2.3 Causal sequence modeling (CSM)
The causal sequence modeling (CSM) framework is inspired by recent advances in causal languagemodeling (20), where models are trained to predict the next word in a sequence of text.
Speciﬁcally,these models receive some text sequence as input, e.g., represented as a matrix X 2 R n ⇥ e with nwords each encoded by an e -dimensional vector, where the last sequence element (representing thelast word) has been replaced by some some learned mask embedding E msk 2 R e
(e.g., "When itrains, it [msk]").
The task of a model is then to predict the word that was replaced by E msk .
Incausal language modeling , models are therefore solely concerned with the part of the input sequencethat precedes the masked word.
To adapt causal language modeling to neuroimaging data, wemask the last time point T of the parcelated BOLD sequence X by replacing it with a learnablemask embedding E msk .
As for the autoencoding framework, X is then transformed to E in bymeans of linear projection and the addition of TR embeddings E TR .
In line with recent advances in causal language modeling, we utilize a standard transformer decoder model f d ( · ) (based on the GPTarchitecture; 20) in this learning framework.
As transformer models are not aware of the ordering oftheir input, we further add learnable position embeddings E pos 2 R p ⇥ e to E in , each representingone of p possible positions in the input sequence (we set p = 512 for all transformer models).
Given E in , the transformer decoder model predicts a corresponding output sequence f d ( E in ) 2 R t ⇥ e inthe form of the hidden states at the output of its last layer.
Upstream learning.
In line with causal language modeling, the upstream objective of the CSMframework is to reconstruct the masked last time point T of the parcelated BOLD sequence X byminimizing the mean absolute difference to its reconstruction ˆ X T : min 1 e
P ei =1 | ˆ X T,i  X T,i | .
Adaptation.
During downstream adaptation, we attach a learnable classiﬁcation embedding E cls
R e to the end of BOLD embedding E X and forward the resulting prediction f
( E in ) T +1 to acorresponding decoding head p ( · ) (see section 2.2.1).
2.2.4 Sequence-BERT
The Sequence-BERT framework is based on recent advances in bidirectional masked languagemodeling, particularly BERT (21).
In NLP, BERT is trained to jointly solve a masked-languagemodeling and next-sentence-prediction task.
To this end, BERT receives two masked sentences S a(e.g., "The sky is [msk].") and
S b (e.g., "The [msk] is shining.") as input, in which some fraction ofwords have been randomly masked.
BERT is then asked to I) predict the masked words (maskedlanguage-modeling) and II) determine whether S b follows S a (next-sentence-prediction).
In contrastto causal language modeling, masked language modeling is considered as a bidirectional learningtask, as the model is not just concerned with the parts of the input that precede a masked wordbut also the parts that follow it.
To adapt BERT to neuroimaging data, we ﬁrst sample two inputsequences X a [-
-]2 R t a ⇥ n and X b 2 R t b ⇥ n from the data (with lengths t a and t b ).
In 50% of the cases,the two sequences represent sequential parts of the same underlying fMRI sequence, while in theother 50% of cases, the two sequences are randomly sampled from two distinct fMRI sequences.
Due to the strong temporal autocorrelation of BOLD data, we randomly leave a gap of 1 - 5 TRs(generally corresponding to a gap of 1 to 20 s) between the two sequences when sampling them fromthe same underlying fMRI sequence, to encourage the models to consider the entire input sequencesand not just their connecting time points.
As for BERT, we next randomly mask time points of eachsampled sequence (with a probability of 20% ) by replacing them with a learnable mask embedding E msk .
The masked versions of X a and X b are then linearly projected to obtain respective BOLDembeddings E X a and E X b to which we then add TR and position embeddings E TR and E pos aswell as a sequence ID embedding E id 2 R 2 ⇥ e (to indicate each of the two input sequences).
In linewith BERT, we lastly also attach a separation embedding E sep 2 R e to the end of each sequence(to indicate their endpoints) and a classiﬁcation embedding E cls to the beginning of S a (for thenext-sentence-prediction task; see upstream learning below).
The resulting embedding representation
E in = [ E cls , { E X a t
[Equation]
+ [Equation] , E sep ] isthen forwarded to a standard transformer encoder model f e ( · ) (based on the BERT architecture; 21),which outputs a corresponding output sequence f e ( E in ) 2 R t ⇥ e in the form of the hidden states atthe output of its ﬁnal layer.
Upstream learning.
Similar to BERT, we deﬁne the upstream objective of the Sequence-BERTframework as the sum of two learning objectives:
In line with masked-language-modeling, theﬁrst objective is deﬁned as the reconstruction loss L rec for all masked time points J : L rec =1 n ( J )
P j 2 J 1 e
P ei =1 | X j,i  ˆ X j,i | .
In line with next-sentence-prediction, the other objective is deﬁnedas the binary cross entropy loss for a decoding head that receives f ( E in ) 1 (corresponding to thetransformer output for classiﬁcation embedding E cls ) as input and predicts probability p ( f ( E in ) 1 )that X b follows X a : L cls =
y log( p ( f ( E in ) 1 ))
(1  y ) log(1  p ( f ( E in ) 1 )) , where y indicatesa binary variable that equals 1 if the two sequences come from the same fMRI run and 0 otherwise.
Adaptation.
During downstream adaptation, we forward the transformer’s output for classiﬁcationembedding f e ( E in ) 1 as input to a corresponding decoding head p ( · ) (see section 2.2.1).
2.2.5 Network-BERT
All so far presented frameworks model their input on the level of the individual time points in asequence, by trying to reconstruct the parcelated whole-brain BOLD signal for speciﬁc time points.
Yet, the dynamics of brain activity data can also be viewed differently, by focusing on the interaction ofnetwork activities over time instead of focusing on the distribution of network activities at individualtime points.
We therefore created a variant of the Sequence-BERT framework that randomly (at arate of 10%) masks the activity of individual networks n in the parcelated BOLD sequence X byreplacing them with a learnable network mask embedding E msk 2 R t , comprising one value foreach possible time point in the input (see Fig. 1).
With the exception of the masking procedure, the
Network-BERT and Sequence-BERT frameworks are identical.
Upstream learning.
During upstream learning, Network-BERT uses the same cross-entropy loss
L cls as Sequence-BERT and adds it to a variant of the reconstruction loss that measures reconstructionperformance for the masked network time courses
J : L rec =
[Equation]
Adaptation.
During downstream adaptation, Network-BERT forwards the transformer’s output forthe classiﬁcation embedding f e ( E in ) 1 to a respective decoding head p ( · ) (see section 2.2.1).
2.3 Training details
If not reported otherwise, we train all models with stochastic gradient descent and the ADAMoptimizer (with  1 = 0 .
9 ,  2 = 0 . 999 , and ✏ = 1 e  8 ; (31)).
We further use a linear learning ratedecay schedule (with a warmup phase of 1% of the total number of training steps), gradient normclipping at 1 . 0 , and L 2 -regularisation (weighted by 0 . 1 ).
During upstream learning, we set themaximum learning rates to 2 e  4 , 5 e  4 , 1 e  4 , and 1 e  4 for the autoencoding, CSM, Sequence-BERT,and Network-BERT frameworks respectively (based on the learning rates used in corresponding NLP studies; 20; 21; 26) and randomly sample input sequences of 10 to 55 TRs from the fMRI ﬁlescontained in our upstream dataset.
For the two BERT frameworks, each sequence S i is split intotwo sequences S ai and S bi of equal length (with a randomly sampled gap of 1 to 5 TRs betweenthe two sequences), while S bi is then randomly exchanged (with 50% probability) with another S bj(where i 6 = j ).
We randomly divide the upstream data into distinct training and validation datasetsby randomly designating 5% of the fMRI run ﬁles of each upstream dataset as validation data (at aminimum of 2 ﬁles per dataset) and using the rest of the ﬁles for training.
At each evaluation step,we randomly sample 640 , 000 samples from the validation dataset to evaluate model performances.
During downstream adaptation, we begin training with the respective pre-trained model parametersand then allow all parameters to be freely changed during the adaptation.
Other optimization andarchitecture hyper-parameters are evaluated and reported in the experiments described below.
Compute resources.
Upstream learning was performed on Google Compute Engine n1-highmem64 nodes with four Nvidia Tesla P100 GPUs, 64 CPU threads, and 416 GB RAM memory, whiledownstream adaptation was performed on compute nodes of the Texas Advanced Computing Centerwith one Nvidia 1080-TI GPU, 32 CPU threads, and 128 GB RAM memory.
Training times varieddepending on model size but were generally in the range of 1 to 3 days for upstream learning and 0 .
5to 3 hours for downstream adaptations.
Experiments
3.1 Datasets
All unprocessed fMRI data used in this study are publicly available through OpenNeuro.org (3) andthe Human Connectome Project (HCP; 32).
For an overview of our preprocessing, see Appendix A.3.3.1.1 Upstream: 11 , 980 fMRI runs from 1726 individuals across 34 datasets
Our upstream dataset comprises 11 , 980 fMRI run ﬁles from 1726 individuals across 34 datasets(for an overview, see Fig.
2 and Appendix Table A1).
To our knowledge, this represents one of thebroadest fMRI datasets used for pre-training in neuroimaging to date, spanning many acquisition sites and a diverse set of experimental conditions and domains.
A few prominent examples of includeddatasets are The Midnight Scan Club (33), Individual Brain Charting (34), Amsterdam Open MRIcollection (35), BOLD5000 (36), and the Narratives collection (37).
3.1.2 Downstream: 2 benchmark datasets with 20 and 26 mental states
To evaluate downstream adaptation performance, we use two benchmark mental state decoding datasets, with a wide range of mentalstates, namely, task-fMRI data from the HCP (32) and the MultiDomain Task Battery (MDTB; 38).
Speciﬁcally, we include datafrom 100 individuals of the HCP, who each performed two fMRIruns for each of seven tasks, with a total of 19 distinct mental statesacross the tasks (see Appendix A.2).
We also include two restingstate fMRI runs per individual, in which individuals simply rest inthe fMRI without any particular task, thereby increasing the totalnumber of mental states to 20 for the HCP data.
The MDTB datainclude 24 individuals performing 26 tasks in two scanning sessionswith eight fMRI runs per session.
Due to the strong similarity ofexperimental conditions within individual tasks of the MDTB, wedeﬁned each of its tasks as one target mental state (see Appendix A.2).
During downstream adaptation, we sample sequences fromboth datasets according to their respective experiment event ﬁles,which deﬁne an on- and offset for each trial in an fMRI run (with durations generally ranging between 2 and 30 s, at a TR of 0 .
72 (HCP)and 1 s (MDTB)).
For the HCP’s resting state data, we randomlysampled sequences between 10 and 30 s.
3.2 Hyper-parameter evaluation: larger embeddings but not deeper models
In our ﬁrst experiment, we compared the performance of different hyper-parameter settings for theproposed frameworks.
Speciﬁcally, we evaluated three different embedding dimensions e ( 192 , 384 ,and 768 ; see section 2.2.1) and three different numbers of hidden layers for the used DL models ( 2 , 4 ,and 6 hidden layers for the encoder and decoder parts of the autoencoding framework and 4 , 8 , and12 hidden layers for the transformer models).
In line with other work (39), we scaled the numberof attention heads n ↵ per layer of the transformer models with the embedding dimension, such thatn ↵ = e 64 .
We then trained one model for each point of the resulting 9 -point grid of each frameworkfor 200 , 000 training steps with a mini-batch size of 256 random samples per step.
Overall, modelperformances improved (in terms of validation loss) with larger embedding dimensions but not withthe number of hidden layers (see Fig. 3).
We therefore decided to continue all further analyses withmodels comprising 768 embedding dimensions and 2 hidden layers for the encoder and decoder partsof the autoencoding framework and 4 hidden layers for the transformer models.
3.3 Upstream performance: models learn well in all frameworks
In our second experiment, we fully pre-trained one model in each framework according to ourpreviously selected set of hyper-parameters (see section 3.2) by training models for 300 , 000 trainingsteps at a mini-batch size of 768 .
All models learned well over the course of their training, withgradually decreasing training and validation losses (see Fig. 4).
Note that the transformer modeltrained in CSM slightly overﬁtted the upstream data, whereas the gap in training and validationperformance of the autoencoding framework results from our application of teacher-forcing duringtraining (see section 2.2.2).
After training, we also evaluated the reconstruction performance of theﬁnal models for different parts of the brain (see Appendix A.6) and found that the models exhibitedsimilar overall distributions of reconstruction error, with relatively higher errors in the posteriorparietal, occipital, and cingulate cortices as well parts of the limbic system.
3.4 Adapting pre-trained language models does not yield upstream performance gains
Given that the formatting of our input embedding representation E X (see section 2.2.1) is highlysimilar to the embedding representations of sentences in language modeling, where each word of a sentence is represented by a vector (e.g., 40; 21), we next tested whether adapting pre-trainedlanguage models to our upstream data would yield meaningful performance gains, when comparedto training models from scratch.
To this end, we adapted pre-trained variants of GPT-2 (40) and BERT (21) (with 12 hidden layers and an embedding dimension of 768 ; as provided by Hugging
Face’s Transformer library (41)) to our upstream dataset (see Appendix A.7), but found that theupstream validation performance of these two language models did not meaningfully improve uponthe performance of our models trained from scratch (see section 3.3), at ﬁnal upstream validationlosses of 0 .
73 (CSM with GPT-2) and 0 .
80 (Sequence-BERT with BERT) (Appendix Fig. A3).
3.5 Downstream adaptation: CSM outperforms other frameworks
Lastly, in our fourth experiment, we evaluated the downstream adaptation performance of our pretrained models in two benchmark mental state decoding datasets (see section 3.1.2).
To gauge theeffectiveness of pre-training, we adapted the pre-trained models to varying sizes of the downstreamdatasets, comprising data from 1 to 48 (HCP) and 11 (MDTB) individuals.
Importantly, for eachdataset size, we randomly split the data into distinct training, validation, and test datasets by randomlyassigning individuals to one of these three datasets.
Speciﬁcally, we ﬁrst randomly selected 10(HCP) and 3 (MDTB) individuals whose data we use for validation and 20 (HCP) and 9 (MDTB)other individuals whose data we use for testing, before randomly sampling the given number ofindividuals whose data we use for training from the remaining pool of individuals.
We then adaptedthe pre-trained models to the resulting training datasets at a mini-batch size of 256 with varyinglearning rates and numbers of training steps (see Appendix A.9).
Here, we only report ﬁnal testdecoding accuracies for those models that achieved the highest ﬁnal validation decoding accuracy(see Appendix Fig.
A6 and A7 for all validation accuracies).
To evaluate the decoding performanceof the pre-trained models, we also trained a linear baseline model (inline with the state-of-the-art inmental state decoding; 42) and a variant of our decoding head p ( · ) (see section 2.2.1) that we applieddirectly to the parcellated BOLD sequences X (see Appendix A.8).
The pre-trained models clearly outperformed the linear baseline model across all training dataset sizes,while the models’ performances also scaled well with the number of individuals in the training dataset(see Tables 1 and 2; at chance-levels of 5 . 0% (HCP) and 9% (MDTB)). Overall, the transformer
[Table] [Table]
decoder model trained with the CSM framework clearly outperformed the other models by achievingthe highest test decoding accuracies throughout all evaluated sizes of the training dataset.
In smallertraining dataset sizes (  6 (HCP) and  3 (MDTB) individuals), the other models performed onpar with the decoding head p ( · ) , while clearly outperforming it in larger training datasets. For the HCP data, for which other reported decoding performances exist, the pre-trained models perform aswell as other models trained on much larger fractions of the data (often comprising the data of manyhundred individuals; 10; 7; 6; 43), which generally report test decoding accuracies between 80% and93% . An ablation analysis of the models’ mental state decoding decisions in both datasets furtherrevealed that the these are strongly dependent on the activity of the occipital and inferior temporalcortex as well as parts of the pre- and postcentral gyrus (Appendix A.10).
We also tested whether the reported model performances are stable over the non-deterministic aspectsof their training (44) by replicating our adaptation analysis with different random seeds and splits ofthe data (see Appendix A.11). This replication conﬁrmed our results, yielding ﬁnal test decodingaccuracies that were not meaningfully different from our initial analysis (see Appendix Tables A3, A4, A5, A6).
Conclusion
In this work, we propose a set of novel self-supervised learning frameworks for neuroimaging databased on prominent counterparts in NLP.
By the use of these frameworks, we are able to pre-train DLmodels across broad, public neuroimaging data, comprising many individuals, experimental domains,and acquisition sites.
We show that the pre-trained models generalize well to two benchmark mentalstate decoding datasets and generally outperform baseline models.
To enable others to build on thiswork, we make our training data and pre-trained models publicly available (see Appendix A.4).
Limitations.
Currently, this work neglects a key aspect of mental state decoding, namely, the abilityto draw inferences about the association between the decoded mental states and the underlying brainactivity from the trained models.
First empirical evidence indicates that attribution methods fromexplainable artiﬁcial intelligence (XAI; 45) research are well-suited to provide insights in the mentalstate decoding decisions of DL models (19).
Similarly, this work currently does not provide anyinsights into how the proposed self-supervised learning frameworks compare to contrastive learningtechniques, which are often used in computer vision (46) or medical imaging (47).
Potential negative social impact. We are currently not aware of any direct negative social impactsthat could follow from pre-training DL models on public neuroimaging data, which are de-identiﬁedand collected and shared with human consent in a way approved by institutional review boards.