Functional Annotation of Human Cognitive States using Graph Convolution Networks
Abstract
A key goal in neuroscience is to understand brain mechanisms of cognitive functions.
An emerging approach is to study “brain states” dynamics using functional magnetic resonance imaging (fMRI).
So far in the literature, brain states have typically been studied using 30 seconds of fMRI data or more, and it is unclear to which extent brain states can be reliably identiﬁed from very short time series.
In this project, we applied graph convolutional networks (GCN) to decode brain activity over short time windows in a task fMRI dataset, i.e. associate a given window of fMRI time series with the task used.
Starting with a populational brain graph with nodes deﬁned by a parcellation of cerebral cortex and the adjacent matrix extracted from functional connectome, GCN takes a short series of fMRI volumes as input, generates high-level domain-speciﬁc graph representations, and then predicts the corresponding cognitive state.
We investigated the performance of this GCN "cognitive state annotation" in the Human Connectome Project (HCP) database, which features 21 different experimental conditions spanning seven major cognitive domains, and high temporal resolution task fMRI data.
Using a 10-second window, the 21 cognitive states were identiﬁed with an excellent average test accuracy of 89% (chance level 4.8%).
As the HCP task battery was designed to selectively activate a wide range of specialized functional networks, we anticipate the GCN annotation to be applicable as a base model for other transfer learning applications, for instance, adapting to new task domains.
Introduction
Identifying brain networks involved in human cognition has been one of the main goals of neuroscience research.
Modern imaging techniques, such as functional magnetic resonance imaging (fMRI), provide an opportunity to accurately map the neural substrates of human cognition.
An emerging topic in the literature is the identiﬁcation of “brain states”, characterized by a canonical spatial pattern of functional activity, which were found to associate with speciﬁc cognitive states.
A popular approach to identify these brain states, called multi-voxel pattern analysis (MVPA), uses machine learning tools to decode which task a subject performed based on recordings of brain activity in task fMRI
But the algorithm is usually limited to speciﬁc cognitive domains and relies on long acquisition of brain activity with repeated blocks to accurately decode a brain state.
In this project, we proposed a GCN architecture for annotating human brain activity on a cognitive battery of 21 task states.
Instead of using the averaging BOLD signals or statistical constrast maps, GCN takes a short series of fMRI volumes as input, generates task-speciﬁc graph representations, and then predicts the corresponding cognitive labels.
Comparing to the multi-class support vector machines classiﬁcation, GCN achieved much higher performance in identifying a variety of cognitive states.
Datasets and Preprocessing
In this project, we are using block-design task fMRI data from the Human Connectome Project (HCP)
S1200 release.
The minimal preprocessed fMRI data of the CIFTI format were used, which maps individual fMRI time series onto the standard surface template with 32k vertices per hemisphere.
Further details on fMRI data acquisition, task design and preprocessing can be found in (2) and (5).
The task fMRI data includes seven cognitive tasks, which are emotion, gambling, language, motor, relational, social, and working memory.
In total, there are 23 different cognitive states.
Considering the short event design nature of the gambling trials (1.5s for button press, 1s for feedback and 1s for
ITI), we evaluated the decoding models with and without the two gambling conditions and found a much lower precision and recall scores for gambling task (average f1-score = 61%) than other cognitive domains (average f1-score > 91%).
In the following experiments, we excluded the two gambling conditions and only reported results on the remaining 21 cognitive states.
The detailed description of the tasks can be found in (2).
A summary table is also shown in Tab. 1.
[Table]
Graph Signal Processing and Graph Convolution Network
Starting with a brain signal x , Graph signal processing (GSP) ﬁrst maps the signal onto a weighted graph G =
( V , E , W ) that deﬁnes a network structure among a set of brain regions.
The set V is a parcellation of cerebral cortex into N regions, and E is a set of connections between each pair of brain regions, with its weights deﬁned as W i,j .
Here we used the multimodal cortical parcellation of the cerebral cortex (4), which delineates 180 functional areas per hemispheree bounded by sharp changes in cortical architecture, function, connectivity, and topography.
The connections between brain areas were estimated by calculating the group averaged resting-state functional connectivity (RSFC) based on 1080 minimal prepossessed resting-state fMRI data (5).
The spectral analysis of the graph signal relies on the graph Laplacian, deﬁned as:
[Equation]
where D is a diagonal matrix of node degrees and I is the identity matrix.
As we assume the weights to be undirected and symmetric, the matrix L can be factored as U ∆ U T , where U =
( u 0 , . . .
, u N − 1 ) is the matrix of Laplacian eigenvectors, also called graph Fourier modes, and
∆ = diag ( λ 0 , . . .
, λ N − 1 ) is a diagonal matrix with the corresponding eigenvalues, specifying the frequency of the modes.
This eigendecomposition can be interpreted as a generalization of the standard Fourier basis onto a non-Euclidean domain. Thus, the graph Fourier transform is deﬁned as ˆ x = L{ x } = U T x and its inverse as x
= U ˆ x , where x is the graph signal. Merging the spectral graph theory with deep learning techniques, (3) ﬁrst proposed a new architecture of graph convolutional neural networks (GCN), which generates a linear combination of the graph modes across the full spectrum of graph Laplacian.
Kipf and colleagues (7) introduced a simpliﬁed version of GCN:
[Equation]
where θ is a single parameter to be learned.
Thus, the output of a graph convolution layer is deﬁned as:
[Equation]
where X l and F out outcome ﬁlters.
These parameters are shared among all nodes on layer l .
σ ( . ) denotes an activation function, such as the ReLU ( x ) = max (0 , x ) .
It’s worth noted that this type of graph convolution only takes into account the direct neighbours in the graph.
By stacking multiple GCN layers, we could propagate information of brain activity among k th -order neighbourhood, with k being the number of GCN layers.
Speciﬁcally, in the ﬁrst GCN layer, we treat the short series of fMRI volumes as multiple channels, with X 1 ∈ R N × F being a 2D matrix of N brain regions and F time windows.
During model training, the ﬁrst GCN layer learns various versions of the spatio-temporal convolution kernels (spatially only consider the direct neighbours, and temporally use the same size as the time window), as a replacement of the canonical hemodynamic response function (HRF).
The derived activation maps are then inputted to the following convolutional layers to generate high-level graph representations.
A brain state annotation model was proposed, consisting of 6 GCN layers with 32 graph ﬁlters at each layer, followed by a global average pooling layer and 2 fully connected layers.
The model takes short series of fMRI data as input, propagates information among inter-connected brain regions and networks, generates a high order graph-level representations and ﬁnally predicts the corresponding cognitive labels as a multi-class classiﬁcation problem.
The overview of the fMRI decoding model was illustrated in Figure 1.
The implementation of the GCN model was based on Tensorﬂow 1.12.0.
The network was trained for 100 epochs with the batch size set to 128.
We used Adam as the optimizer with the initial learning rate as 0.001.
Additional l2 regularization of 0.0005 on weights was used to control model overﬁtting and the noise effect of fMRI signals.
The entire dataset was split into training (70%), validation (10%), test (20%) sets using a subject-speciﬁc split scheme, which ensures that all fMRI data from the same subject was assigned to one set.
The best model with the highest prediction score on the validation set was saved and then evaluated separately on the test set.
Results 4.1 Cognitive states can be decoded with high accuracy from 10s of fMRI data
The GCN state annotation model was evaluated using the cognitive battery of HCP task-fMRI dataset.
Here we mainly focused on the 21 task conditions spanning over 6 cognitive domains, namely: emotion, language, motor, relational, social, and working memory.
Using a 10 second window of fMRI time series, the 21 conditions can be identiﬁed with an average test accuracy of 89.83%, signiﬁcantly different from the chance level of 4.8%.
After summarizing the confusion matrix according to the 6 cognitive domains, each cognitive domain could be identiﬁed with a recall accuracy > 91%.
Among the cognitive domains, the language tasks (story vs math) and motor tasks (left/right hand, left/right foot and tongue) were the most recognizable conditions, which showed the highest precision and recall scores (average f1-score = 95% and 94% respectively for language and motor conditions).
Meanwhile, the relational processing and working memory task conditions were identiﬁed with the lowest performance (average f1-score = 81.6% and 84.2% respectively), with some misclassiﬁcations between emotion/relational processing and working memory tasks.
4.2 Classiﬁcation errors due to high similarity in task stimuli
Misclassiﬁcations of cognitive states were found both between and within cognitive domains.
For instance, the emotion and relational processing conditions were misidentiﬁed as working memory task, probably due to the high similarity in task stimuli.
Speciﬁcally, the emotion task involves the matching of faces, overlapping with face encoding and retrieval in working memory tasks.
The relational processing requires matching of shapes and textures of drawn objects, somewhat similar to encoding and retrieval of bodies and tools in working memory task.
Also, some trials were misclassiﬁed within the same cognitive domain.
For instance, most misclassiﬁcations within working memory task were found between 0-back and 2-back conditions, which were still observed even when the decoding model was trained for the single cognitive domain.
By contrast, for face and place working memory stimuli, brain decoding reached high accuracy, regardless of using a multi-domain or single-domain classiﬁer (misclassiﬁcation less than 0.2%).
4.3 Performance of GCN annotation was associated with in-scanner behaviors
We found a strong association between prediction accuracy of GCN annotation and the median reaction time within scanner (Figure 2).
For instance, during relational processing task which consists of two conditions, i.e. relational processing and feature matching, participants reacted faster to the matching condition than relational processing.
Similarly, GCN also achieved higher prediction for matching (f1=0.96) than relational processing (f1=0.91).
Moreover, within each task condition, GCN achieved higher accuracy on trials when participants were more engaged and responded faster.
The analysis was performed on 200 subjects from the test set.
4.4 GCN achieved much better performance than SVC
We also wanted to establish if the performance of deep GCN represented a substantial improvement over more traditional machine learning tools.
We thus also evaluated the same brain decoding tasks using a multi-class support vector machines classiﬁcation (SVC) with a linear kernel, as our baseline model.
The results showed that, using 10s of fMRI data, SVC-linear achieved much lower prediction accuracy in classifying the 21 states (89% vs 63% respectively for GCN and SVC-linear) and took a longer time for training (560s vs 9518s).
Even when only focusing on a single cognitive domain,
SVC-linear still showed much lower performance (96% vs 87% for motor task; 86% vs 70% for working memory conditions).
4.5 Saliency map demonstrate biologically meaningful features learned by GCN
We investigated whether a set of biologically meaningful features were learned by the GCN.
For this purpose, we generated saliency maps on the model trained on the corresponding cognitive domain, by propagating the non-negative gradients backwards till the input layer (10).
An input feature is salient or important only if its little variation causes big changes in the decoding output.
Thus, high values in the saliency map indicate large contributions during prediction of cognitive states.
The two language conditions, story and mathematics, shared a number of salient features (Figure
A), likely related to shared cognitive processes.
Both conditions involved the processing of auditory statements, which may explain high salience in the primary auditory cortex and perisylvian languagerelated brain regions, consisting of inferior frontal gyrus (IFG), supramarginal gyrus/angular gyrus, and superior temporal gyrus (STG).
There were also some salient features found only for either mathematics or story.
For instance, the story condition involved salient features in more anterior part of left IFG, including pars triangularis and orbitalis.
By contrast, mathematical statements involved more posterior part, including pars opercularis of IFG and precentral sulcus.
Additional inferior temporal regions were salient for mathematics only, which have been shown to be more involved in mathematical than non-mathematical judgment tasks (1).
As expected, no salient features in the perisylvian language-related brain regions were found for the motor task (Figure 3 B).
Different types of movements were associated with high salience in the primary motor and somatosensory cortex, which has been shown as the main territories engaged during movements of the human body (9).
No clear somatotopic organization among different types of movements were identiﬁed here, which was somewhat expected because the primary motor and somatosensory cortex were parcellated into single strips in the Glasser’s atlas (4).
Some categoryspeciﬁc salient features were still identiﬁed, for instance in medial primary motor cortex for foot movement and in lateral orbitofrontal cortex for tongue movement.
Moreover, salient features in the ventral visual stream were identiﬁed for image recognition in working memory task (Figure 3 C).
Speciﬁcally, the place stimuli activated more medial areas in the temporal cortex including parahippocampal gyrus; while the face stimuli activated more lateral temporal regions including fusiform gyrus.
This observation is consistent with the known, strong spatial segregation of the neural representation for face vs place image, in fusiform face area (FFA) and parahippocampal place area (PPA) respectively (6).
In summary, the regions highlighted by the saliency maps are consistent with prior knowledge from the literature, and suggest that the GCN model has learned biologically meaningful features, rather than confounding effects for example motion artifacts.
Conclusion
We propose a new graph convolution based model to annotate human cognitive states using a short series of fMRI signals.
This model annotates human brain activity with ﬁne temporal resolution and ﬁne cognitive granularity.
Using a 10s window of fMRI signals, our model identiﬁed 21 different task conditions with a test accuracy of 89%.
Besides, the GCN performance was associated with participants’ in-scanner behaviors.
The saliency maps of GCN also demonstrated that biologically meaningful and domain-speciﬁc features have been learned during model training.
Together, our project provides an automated tool to annotate the dynamics of human cognition in real time