B RAIN N ETWORK T RANSFORMER
Anonymous Author(s)
AffiliationAddress email
Abstract
Human brains are commonly modeled as networks of Regions of Interest (ROIs)and their connections for the understanding of brain functions and mental disorders.
Recently, Transformer-based models have been studied over differenttypes of data, including graphs, shown to bring performance gains widely.
In thiswork, we study Transformer-based models for brain network analysis.
Drivenby the unique properties of data, we model brain networks as graphs with nodesof fixed size and order, which allows us to (1) use connection profiles as nodefeatures to provide natural and low-cost positional information and (2) learn pairwise connection strengths among ROIs with efficient attention weights acrossindividuals that are predictive towards downstream analysis tasks.
Moreover, wepropose an O RTHONORMAL C LUSTERING R EADOUT operation based on selfsupervised soft clustering and orthonormal projection.
This design accounts forthe underlying functional modules that determine similar behaviors among groupsof ROIs, leading to distinguishable cluster-aware node embeddings and informative graph embeddings.
Finally, we re-standardize the evaluation pipeline on theonly one publicly available large-scale brain network dataset of ABIDE, to enable meaningful comparison of different models.
Experiment results show clearimprovements of our proposed B RAIN N ETWORK T RANSFORMER on both thepublic ABIDE and our restricted ABCD datasets.
The implementation is availableat https://anonymous.4open.science/r/BrainTransformer .
Introduction
Brain network analysis has been an intriguing pursuit for neuroscientists to understand humanbrain organizations and predict clinical outcomes [36, 43, 42, 4, 14].
Among various neuroimagingmodalities, functional Magnetic Resonance Imaging (fMRI) is one of the most commonly used forbrain network construction, where the nodes are defined as Regions of Interest (ROIs) given an atlas,and the edges are calculated as pairwise correlations between the blood-oxygen-level-dependent(BOLD) signal series extracted from each region [39, 38, 43].
Researchers observe that some regionscan co-activate or co-deactivate simultaneously when performing cognitive-related tasks such asaction, language, and vision.
Based on this pattern, brain regions can be classified into diversefunctional modules to analyze diseases towards their diagnosis, progress understanding and treatment.
Nowadays Transformer-based models have led a tremendous success in various downstream tasksacross fields including natural language processing [40, 13] and computer vision [16, 9].
Recentefforts have also emerged to apply Transformer-based designs to graph representation learning.
GAT[41] firstly adapts the attention mechanism to graph neural networks (GNNs) but only considers thelocal structures of neighboring nodes.
Graph Transformer [17] injects edge information into theattention mechanism and leverages the eigenvectors of each node as positional embeddings.
SAN[29] further enhances the positional embeddings by considering both eigenvalues and eigenvectorsand improves the attention mechanism by extending the attention from local to global structures.
Graphomer [47], which achieves the first place on the quantum prediction track of OGB Large-Scale
Challenge [21], designs unique mechanisms for molecule graphs such as centrality encoding toenhance node features and spatial/edge encoding to adapt attention scores.
However, brain networks have several unique traits that make directly applying existing graph
Transformer models impractical.
First, a brain network is a correlation matrix defined on a completegraph.
This impedes the designs like centrality, spatial, and edge encoding because each node inthe brain network has the same degree and connects to every other node by a single hop.
Second, inprevious graph transformer models, eigenvalues and eigenvectors are commonly used as positionalembeddings because they can provide identity and positional information for each node.
Nevertheless,in brain networks, the connection profile, which is defined as each node’s corresponding row in thebrain network adjacency matrix, is recognized as the most effective node feature [12].
This nodefeature naturally encodes both structural and positional information, making the aforementionedpositional embedding design based on eigenvalues and eigenvectors redundant.
The third challenge isscalability.
Typically, the numbers of nodes and edges in molecule graphs are less than 50 and 2500,respectively.
However, for brain networks, the node number is generally around 100 to 400, while theedge number can be up to 160,000.
Therefore, operations like the generation of all edge features inexisting graph transformer models can be time-consuming, if not infeasible.
[Figure]
In this work, we propose to develop B RAIN N ETWORK T RANSFORMER (B RAIN N ET TF), whichleverages the unique properties of brain network data to fully unleash the power of Transformer-basedmodels for brain network analysis.
Specifically, motivated by previous findings on effective GNNdesigns for brain networks [12], we propose to use the effective initial node features of connectionprofiles.
Empirical analysis shows that connection profiles naturally provide positional featuresfor Transformer-based models and avoid the costly computations of eigenvalues or eigenvectors.
Moreover, recent work demonstrates that GNNs trained on learnable graph structures can achievesuperior effectiveness and interpretability [25].
Inspired by this insight, we propose to learn fullypairwise attention weights with Transformer-based models, which resembles the process of learningpredictive brain network structures towards downstream tasks.
One step further, when GNNs are used for brain network analysis, a graph-level embedding needsto be generated through a readout function based on the learned node embeddings [27, 31, 12].
Asis shown in Figure 1(a), a property of brain networks is that brain regions (nodes) belonging to thesame functional modules often share similar behaviors regarding activations and deactivations inresponse to various stimulations [6].
Unfortunately, the current labeling of functional modules israther empirical and far from accurate.
For example, [2] provides more than 100 different functionalmodule organizations based on hierarchical clustering.
In order to leverage the natural functions ofbrain regions without the limitation of inaccurate functional module labels, we design a new globalpooling operator, O RTHONORMAL C LUSTERING R EADOUT , where the graph-level embeddingsare pooled from clusters of functionally similar nodes through soft clustering with orthonormalprojection.
Specifically, we first devise a self-supervised mechanism based on [44] to jointly assignsoft clusters to brain regions while learning their individual embeddings.
To further facilitate thelearning of clusters and embeddings, we design an orthonormal projection and theoretically proveits effectiveness in distinguishing embeddings across clusters, thus obtaining expressive graph-levelembeddings after the global pooling, as illustrated in Figure 1(b).
Finally, the lack of open-access datasets has been a non-negligible challenge for brain networkanalysis.
The strict access restrictions and complicated extraction/preprocessing from fMRI data limit the development of machine learning models for brain network analysis.
Specifically, among all the large-scale publicly available fMRI datasets in literature, ABIDE [5] is theonly one provided with extracted brain networks fully accessible without permission requirements.
However, ABIDE is aggregated from 17 international sites with different scanners and acquisitionparameters.
This inter-site variability may conceal inter-group differences that are really meaningful,which is reflected in the unstable training performance and the significant gap between validation andtesting performance in practice.
To address these limitations, we propose to apply a stratified samplingmethod in the dataset splitting process and standardize a fair evaluation pipeline for meaningfulmodel comparison on the ABIDE dataset.
Our extensive experiments on this public ABIDE datasetand a restricted ABCD dataset [7] show significant improvements brought by our proposed B RAIN
N ETWORK T RANSFORMER .
Background and Related Work
2.1 GNNs for Brain Network Analysis
Recently, emerging attention has been devoted to the generalization of GNN-based models tofMRI-based brain network analysis [30].
GroupINN [46] utilizes a grouping-based layer to provideinterpretability and reduce the model size.
BrainGNN [31] designs the ROI-aware GNNs to leveragethe functional information in brain networks and uses a special pooling operator to select thesecrucial nodes.
In addition, FBNetGen [25] considers the learnable generation of brain networks andexplores the interpretability of the generated brain networks towards downstream tasks.
Anotherbenchmark paper [12] systematically studies the effectiveness of various GNN designs over brainnetwork data.
Different from other work focusing on static brain networks, STAGIN [28] utilizes GNNs with spatio-temporal attention to model dynamic brain networks extracted from fMRI data.
2.2 Graph Transformer
Graph Transformer raises many researchers’ interest currently due to its outstanding performancein graph representation learning.
Graph Transformer [17] firstly injects edge information into theattention mechanism and leverages the eigenvectors as positional embeddings.
SAN [29] enhancesthe positional embeddings and improves the attention mechanism by emphasizing neighbor nodeswhile incorporating the global information.
Graphomer [47] designs unique mechanisms for moleculegraphs and achieves the STOA performance.
Besides, a fine-grained attention mechanism is developedfor node classification [51].
Also, the Transformer is extended to larger-scale heterogeneous graphswith a particular sampling algorithm in HGT [22].
EGT [23] further employs edge augmentation toassist global self-attention.
In addition, LSPE [18] leverages the learnable structural and positionalencoding to improve GNNs’ representation power, and GRPE [35] enhances the design of encodingnode relative position information in Transformer.
B RAIN N ETWORK T RANSFORMER
[Figure]
3.1 Problem Definition
In brain network analysis, given a brain network X ∈ R V × V , where V is the number of nodes (ROIs),the model aims to make a prediction indicating gender, presence of a disease or other properties of thebrain subject.
The overall framework of our proposed B RAIN N ETWORK T RANSFORMER is shownin Figure 2, which is mainly composed of two components, an L -layer attention module MHSAand a graph pooling operator OCR EAD .
Specifically, in the first component of MHSA , the modellearns attention-enhanced node features Z L through a non-linear mapping X → Z L ∈ R V × V .
Thenthe second component of OCR EAD compresses the enhanced node embeddings Z L to graph-levelembeddings Z G ∈ R K × V , where K is a hyperparameter representing the number of clusters.
Z G isthen flattened and passed to a multi-layer perceptron for graph-level predictions.
The whole trainingprocess is supervised with the cross-entropy loss.
3.2 Multi-Head Self-Attention Module (MHSA)
To develop a powerful Transformer-based model suitable for brain networks, two fundamentaldesigns, the positional embedding and attention mechanism, need to be reconsidered to fit the naturalproperties of brain network data.
In existing graph transformer models, the positional informationis usually encoded via eigendecomposition, while the attention mechanism often combines nodepositions with existing edges to calculate the attention scores.
However, for the dense (often fullyconnected) graphs of brain networks, eigendecomposition is rather costly, and the existence of edgesis hardly informative.
ROI node features on brain networks naturally contain sufficient positional information, making thepositional embeddings based on eigendecomposition redundant.
Previous work on brain networkanalysis has shown that the connection profile
X i · for node i , defined as the corresponding row foreach node in the edge weight matrix X , always achieves superior performance over others suchas node identities, degrees or eigenvector-based embeddings [31, 25, 12].
With this node featureinitialization, the self-connection weight x ii on the diagonal is always equal to one, which encodessufficient information to determine the position of each node in a fully connected graph based onthe given brain atlas.
To verify this insight, we also empirically compare the performance of theoriginal Connection Profile with two variants concatenated with additional positional information,i.e. , Connection Profile w/ Identity Feature and Connection Profile w/ Eigen Feature.
The resultsindeed show no benefit brought by the additional computations (c.f. Appendix B).
As for the attentionmechanism, previous work [12] has empirically demonstrated that integrating edge weights into theattention score calculation can significantly degrade the effectiveness of attention on complete graphs,while the generation of edge-wise embedding can be unaffordable given a large number of edges inbrain networks.
On the other hand, the existence of edges provides no useful information for thecomputation of attention scores as well because all edges simply exist in complete graphs.
Based on the observations above, we design the basic B RAIN N ETWORK T RANSFORMER by(1) adopting the Connection Profile as initial node features and eliminating any extra positionalembeddings and (2) adopting the vanilla pair-wise attention mechanism without using edge weightsor relative position information to learn a singular attention score for each edge in the complete graph.
Formally, we leverage a L -layer non-linear mapping module, namely Multi-Head Self-Attention( MHSA ), to generate more expressive node features Z L = MHSA( X ) ∈ R V × V .
For each layer l ,the output Z l is obtained by
[Equation]
where Z 0 = X , ∥ is the concatenation operator, M is the number of heads, l is the layer index,
W l O , W l,m Q , W l,m K , W l,m V are learnable model parameters, and d l,m K is the first dimension of W l,m K .
O RTHONORMAL C LUSTERING R EADOUT (OCR EAD )
The readout function is an essential component to learn the graph-level representations for brainnetwork analysis ( e.g. , classification), which maps a set of learned node-level embeddings to a graphlevel embedding.
Mean( · ) , Sum( · ) and Max( · ) are the most commonly used readout functions for
GNNs.
Xu et al.
[45] show that GNNs equipped with Sum( · ) readout have the same discriminativepower as the Weisfeiler-Lehman Test.
Zhang et al.
[49] propose a sort pooling to generate thegraph-level representation by sorting the final node representations.
Ju et al.
[24] present a layer-wisereadout by extending the node information aggregated from the last layer of GNNs to all layers.
However, none of the existing readout functions leverages the properties of brain networks that nodesin the same functional modules tend to have similar behaviors and clustered representations, as shownin Figure 1(a).
To address this deficiency, we design a novel readout function to take advantage ofthe modular-level similarities between ROIs in brain networks, where nodes are assigned softly towell-chosen clusters with an unsupervised process.
Formally, given K cluster centers, each center has V dimensions, E ∈ R K × V , a Softmax projectionoperator is used as the function to calculate the probability P ik of assigning node i to cluster k ,
[Equation]
where ⟨· , ·⟩ denotes the inner product and Z L is the learned set of node embeddings from the lastlayer of MHSA module.
With this computed soft assignment P ∈ R V
×
K , the original learned noderepresentation Z L can be aggregated under the guidance of the soft cluster information, where thegraph-level embedding Z G is obtained by Z G
= P ⊤ Z L .
However, jointly learning node embeddings and clusters without ground-truth cluster labels is difficult.
To obtain representative soft assignment P , the initialization of K cluster centers E is critical andshould be designed delicately.
To this end, we leverage the observation illustrated in Figure 1(b),where orthonormal embeddings can improve the clustering of nodes in brain networks w.r.t.
thefunctional modules underlying brain regions.
Orthonormal Initialization .
To initialize a group of orthonormal bases as cluster centers, we firstadopt the Xavier uniform [20] to initialize K random centers and each center contains V dimensions
C ∈ R K × V .
Then, we apply the Gram-Schmidt process to obtain the orthonormal bases E , where
[Equation]
In the next section, we theoretically prove the advantage of this orthonormal initialization.
3.3.1 Theoretical Justifications
In OCR EAD , proper cluster centers can generate higher-quality soft assignments and enlarge thedifference between P from different classes.
[37, 32] showed the advantages of orthogonal initialization in DNN model parameters.
However, none of them proves whether it is an ideal strategy toobtain the cluster centers.
We propose two methods from the perspective of statistics as follows.
Firstly, to discern features of different nodes, we would expect a larger discrepancy among theirsimilarity probabilities indicated from the readout.
One way to measure the discrepancy is using thevariance of P for each feature.
Let ¯ P ≡ 1 /K denote the mean of any discrete probabilities with Kvalues.
Variance of P measures the difference between P and ¯ P .
We average over the feature vectorspace: if the result is small, then there is a large tendency that different P approaches ¯ P and hencecannot be discerned easily.
Specifically, the following theorem holds for our function Eq.
Theorem 3.1.
For arbitrary r > 0 , let B r = {Z ∈ R V ; ∥Z∥ ≤ r } denote the round ball centeredat origin of radius r with Z being fracture vectors.
Let V r be the volume of B r .
The variance of
Softmax projection averaged over B r
[Equation]
attains maximum when E is orthonormal.
Despite the concise form, it is unclear whether the above integral has an elementary antiderivative.
Even though, we can circumvent this problem and a rigorous proof is given in Appendix C.
The second method shows that for general readout functions without a known analytical form,initializing with orthonormal cluster centers has a larger probability of gaining better performance.
Toset up the proper statistical scenario, we assume that the unknown readout is obtained by a regressionof some samples ( ˆ Z ( s ) , ˆ E ( t ) , ˆ P ( st ) ) .
This formally converts the exact functional relationship between Z i · , E k · and P ik to a statistical relationship :
[Equation]
i · [Equation] [Equation] with P T being the probability truly reflecting similarities between nodes and clusters and ϵ i denotingthe stochastic error.
It is almost impossible to find P T , but by computing the so-called variationinflation factor [33], we show that regression in orthonormal case has a higher accuracy than that innon-orthonormal case.
Combining with a hypothesis testing, we obtain the following
Theorem 3.2.
The significance level α E k · which reveals the probability of rejecting a well-estimatedpooling is lower when sampling from orthonormal centers than that from non-orthonormal centers.
More details can be seen in Appendix C.
3.3.2 Generalizing OCR EAD to Other Graph Tasks and Domains
In this work, we tested the proposed OCR EAD on functional connectivity (FC) based brain networks.
Other popular modalities of brain networks include structural connectivities (SC), which describethe anatomical organization of the brain by measuring the fiber tracts between brain regions [3].
In
SC-based brain networks, ROIs that are positionally close to each other on the structural connectivitynetworks tend to share similar connection profiles.
This means the idea of OCR EAD is also naturallyapplicable to SC networks, where the orthonormal clustering is based on the physical distancesinstead of the functional modules on FC.
At a higher level, the idea of our proposed OCR EAD is not confined to graph-level prediction tasks onbrain networks but can also be generalized to other graph learning tasks and domains.
Precisely, thereis a growing tendency in node/edge level prediction tasks to enhance the node/edge representationlearning by utilizing the subgraph embeddings around each target node/edge [50, 48].
In this process,substructure learning needs to be performed on the subgraphs, where our proposed OCR EAD can beadapted for compressing a set of node embeddings to subgraph embeddings.
Besides, OCR EAD isalso potentially useful for other types of graphs in the biomedical domains.
For example, for proteinprotein interaction networks, proteins can be implicitly grouped by families that share commonevolutionary origins [34], whereas for gene expression networks, genes can be grouped based onthe latent pathway information [26].
Both of them are potential directions for the future applicationof OCR EAD , among many others driven by biological or other types of prior knowledge regardingunderlying node/edge groups.
Experiments
This section evaluates the effectiveness of our proposed B RAIN N ETWORK T RANSFORMER(B RAIN N ET TF) with extensive experiments.
We aim to address the following research questions:
RQ1.
How does B RAIN N ET TF perform compared with state-of-the-art models of various types?RQ2.
How does our proposed OCR EAD module perform with different model choices?
RQ3.
Does the learned model of B RAIN N ET TF exhibit consistency with existing neuroscienceknowledge and suggest reasonable interpretability?
4.1 Experimental Settings
Datasets.
We conduct experiments on two real-world fMRI datasets.
(a) Autism Brain Imaging
Data Exchange (ABIDE) :
This dataset collects resting-state functional magnetic resonance imaging(Rs-fMRI) data from 17 international sites, and all data are anonymous [5].
The used datasetcontains brain networks from 1009 subjects, with 516 (51.14%) being Autism spectrum disorder(ASD) patients (positives).
The region definition is based on Craddock 200 atlas [11].
As themost convenient open-source large-scale dataset, it provides generated brain networks and can bedownloaded directly without permission request.
Despite the ease of acquisition, the heterogeneity of the data collection process hinders its use.
Since multi-site data are collected from differentscanners with different acquisition parameters, non-neural inter-site variability may mask inter-groupdifferences.
In practice, we find the training unstable, and there is a significant gap between validationand testing performances.
However, we discover that most models can achieve a stable performanceif we follow an appropriate stratified sampling strategy by considering collection sites during thetraining-validation-testing splitting process for ABIDE.
Training curves in Appendix A also showhow different models achieve a stabler performance on our designed new splitting settings than therandom splitting.
Therefore, we use ABIDE as one of the benchmark datasets in this work, andwe share our re-standardized data splitting to provide a fair evaluation pipeline for various futuremethods.
(b) Adolescent Brain Cognitive Development Study (ABCD) :
This is one of the largestpublicly available fMRI datasets with restricted access (a strict data requesting process needs to befollowed to obtain the data) [7].
The data we use in the experiments are fully anonymized brainnetworks with only gender labels.
After the quality control process, 7901 subjects are included in theanalysis, with 3961 (50.1%) among them being female.
The region definition is based on the HCP360 ROI atlas [19].
Metrics.
The diagnosis of ASD is the prediction target on ABIDE, while gender prediction is usedas the evaluation task for ABCD.
Both prediction tasks are binary classification problems, and bothdatasets are balanced between classes.
Hence, AUROC is the most proper performance metricadopted for a fair comparison, and accuracy is applied to reflect the prediction performance.
All thereported performances are the average of 5 random runs on the test set with the standard deviation.
Implementation details.
For experiments, we use a two-layer Multi-Head Self-Attention Moduleand set the number of heads M to 4 for each layer.
We randomly split 70% of the datasets fortraining, 10% for validation, and the remaining are utilized as the test set.
In the training process of
B RAIN N ET TF, we use an Adam optimizer with an initial learning rate of 10 − 4 and a weight decayof 10 − 4 .
The batch size is set as 64.
All models are trained for 200 epochs, and the epoch with thehighest AUROC performance on the validation set is used for performance comparison on the test set.
The model is trained on an NVIDIA Quadro RTX 8000.
Please refer to the supplementary materialfor the full implementation of B RAIN N ET TF.
Computation complexity.
In B RAIN N ET TF, the computation complexity of Multi-Head SelfAttention Module and OCR EAD are O ( LMV 2 ) and O ( KV ) respectively, where L is the layernumber of Multi-Head Self-Attention Module, V is the number of nodes, M is the number of heads,and K is the number of clusters in OCR EAD .
The overall computation complexity of B RAIN N ET TFis thus O ( V 2 ) , which is on the same scale as common GNNs on brain networks such as BrainGNN[31] and BrainGB [12].
4.2 Performance Analysis (RQ1)
(a) B RAIN N ET TF vs. other graph transformers.
We compare B RAIN N ET TF with two populargraph Transformers, SAN [29] and Graphormer [47].
In addition, we also include a basic versionof B RAIN N ET TF without OCR EAD , composed of a Transformer with a 2-layer Multi-Head SelfAttention and a CONCAT-based readout named VanillaTF.
Our B RAIN N ET TF outperforms SANand Graphormer by significant margins, with up to 6% absolute improvements on both datasets.
VanillaTF also performs better than SAN and Graphormer.
We believe this downgraded performanceof existing graph transformers results from their design flaws facing the natures of brain networks.
Specifically, both the preprocessing and the training stages of the Graphormer model accepts onlydiscrete, categorical data.
A bin operator has to be applied on the adjacency matrix, coarseningthe node feature from connection profiles and dramatically hurting the performance.
Furthermore,since brain networks are complete graphs, key designs like centrality encoding and spatial encodingof Graphormer cannot be appropriately applied.
Similarly, for SAN, experiments in Appendix Bshow that adding eigen node features to connection profiles cannot improve the model’s performance.
Besides, the benchmark paper [12] reveals that injecting edge weights into the attention mechanismcan significantly reduce the prediction power.
(b) B RAIN N ET TF vs. neural network models onfixed brain networks.
We further introduce another two neural network baselines on fixed brainnetworks.
BrainGB [12] is a systematic study of how to design effective GNNs for brain networkanalysis.
We adopt their best design as the BrainGB baseline.
BrainnetCNN [27] represents state-ofthe-art of specialized GNNs for brain network analysis, which models the adjacency matrix of a brainnetwork similarly as a 2D image.
As is shown in Table 1, B RAIN N ET TF consistently outperforms both BrainGB and BrainnetCNN.
(c) B RAIN N ET TF vs. neural network models on learnablebrain networks.
Unlike classical GNNs, FBNETGEN [25] is a GNN that aggerates node features ongraphs with learnable weights based on learnable projections of the original BOLD signals, whichachieves SOTA performance on the ABCD dataset for gender prediction.
The learnable graphs canbe seen as a type of attention score.
Experiment results show that our proposed B RAIN N ET TF beats FBNETGEN on both datasets.
[Table]
4.3 Ablation Studies
on the OCR EAD Module (RQ2)
OCR EAD with varying
We vary the readout function for various Transformer architectures, including SAN, Graphormer and
VanillaTF, to observe the performance of each ablated model variant.
The results shown in Table 2demonstrate that our OCR EAD is the most effective readout function for brain networks and improvesthe prediction power across various Transformer architectures.
[Table]
OCR EAD with varying cluster initializations
To further demonstrate how the design of OCR EAD influences the performance of B RAIN N ET TF [Table] we investigate two key model selections, the initialization method for cluster centers and the clusternumber K .
For the initialization, three different kinds of initialization procedures are compared,
namely (a) Random : the Xavier uniform [20] is leveraged to randomly generate a group of centers,which are then normalized into unit vectors; (b) Learnable : the same initial process as Random, butthe generated centers are further updated with gradient descent; (c) Orthonormal : our proposedprocess as described in Eq.
Specifically, we test each initialization method with the cluster number K equals to 2, 3, 4, 5, 10, 50,100.
The results of adjusting these two hyper-parameters on ABIDE and ABCD datasets are shownin Figure 3(a).
We observe that: (1) When cluster centers are orthonormal, the model’s performanceincreases with the number of clusters ranging from 2 to 10, and then drops with the cluster numberrising from 10 to 100, suggesting the optimal cluster number to be relatively small, which leads toless computation and is consistent with the fact that the typical number of functional modules aresmaller than 25; (2) With a sufficiently large cluster number, all three initialization methods, Random,
Learnable and Orthonormal, tend to reach similar performance, but orthonormal performs stably better when the number of clusters is smaller; (3) It is also notable that our OCR EAD consistentlyachieves the best performance over other initialization methods regarding smaller standard deviations.
[Figure]
4.4 In-depth Analysis of Attention Scores and Cluster Assignments (RQ4)
Figure 3(b) displays the self-attention score from the first layer of Multi-Head Self-Attention.
Theattention scores are the average across all subjects in the ABCD test set.
This figure shows that thelearned attention scores well match the divisions of functional modules based on available labels,demonstrating the effectiveness and interpretability of our Transformer model.
Note that since thereexists no available functional module labels for the atlas of the ABIDE dataset, we cannot visualizethe correlations between attention scores and functional modules.
Figure 4 shows the cluster soft assignment results P on nodes in OCR EAD with two initializationmethods.
The cluster number K is set to 4.
The visualized numerical values are the average P of allsubjects in each dataset’s test set.
From the visualization, we observe that (a) Orthonormal initialization produces more discriminative P between classes than random initialization; (b) Within eachclass, orthonormal initialization encourages the nodes to form groups.
These observations demonstrate that our OCR EAD with orthonormal initialization can leverage potential clusters underlyingnode embeddings, thus automatically grouping brain regions into potential functional modules.
[Figure]
Conclusion
This paper presents B RAIN N ETWORK T RANSFORMER , a specialized graph Transformer model with
O RTHONORMAL C LUSTERING R EADOUT for brain network analysis.
Extensive experiments on twolarge-scale brain network datasets demonstrate that our B RAIN N ET TF achieves superior performanceover SOTA baselines of various types.
Specifically, to model the potential node feature similarity inbrain networks, we design OCR EAD and prove its effectiveness both theoretically and empirically.
Lastly, the re-standardized dataset split for ABIDE can provide a fair evaluation for new methods inthe community.
For future work, B RAIN N ET TF can be improved with explicit explanation modulesand used as the backbone for further brain network analysis, such as digging essential neural circuitsfor mental disorders and understanding cognitive development in adolescents.
Checklist
For all authors...
[Table] (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s [Table] contributions and scope?
[Yes]
[Table]
(b) Did you describe the limitations of your work?
[Yes] See Section 5.
[Table]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
[Table] (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
[Table]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See AppendixC.(b)
Did you include complete proofs of all theoretical results?
[Yes] See Appendix C.3.
If you ran experiments...
[Table] (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes]
[Table]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they [Table] were chosen)?
[Yes] See Implementation details in Section 4.1.
[Table] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Metrics in Section 4.1.
[Table] (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Implementation details inSection 4.1.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Datasets in
Section 4.1.
[Table]
(b) Did you mention the license of the assets?
[N/A] ABIDE and ABCD do not provide any license. [Table]
(c) Did you include any new assets either in the supplemental material or as a URL
Yes]
We provide
our implementation and share our repository with MIT license.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] See Datasets in Section 4.1.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes] See Datasets in Section 4.1.
If you
used crowdsourcing or conducted research with human subjects
(a) Did you include the full text of instructions given to participants and screenshots , if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]