MAtt: A Manifold Attention Network for EEG Decoding
Abstract
Recognition of electroencephalographic (EEG) signals highly affect the efficiency of non-invasive brain-computer interfaces (BCIs).
While recent advances of deeplearning (DL)-based EEG decoders offer improved performances, the development of geometric learning (GL) has attracted much attention for offering exceptional robustness in decoding noisy EEG data.
However, there is a lack of studies on the merged use of deep neural networks (DNNs) and geometric learning for EEG decoding.
We herein propose a manifold attention network (MAtt), a novel geometric deep learning (GDL)-based model, featuring a manifold attention mechanism that characterizes spatiotemporal representations of EEG data fully on a Riemannian symmetric positive definite (SPD) manifold.
The evaluation of the proposed MAtt on both time-synchronous and -asyncronous EEG datasets suggests its superiority over other leading DL methods for general EEG decoding.
Furthermore, analysis of model interpretation reveals the capability of MAtt in capturing informative
EEG features and handling the non-stationarity of brain dynamics.
Source codes are available at https://github.com/CECNL/MAtt .
Introduction and related works
A brain-computer interface (BCI) is a type of human-machine interaction that bridges a pathway from brain to external devices.
Electroencephalogram (EEG), a non-invasive neuromonitoring modality with high portability and affordability, has been widely used to explore practical applications of BCI in the real world [1, 2, 3].
For instance, disabled users can type messages through an EEG-based BCI that recognizes the steady-state visual evoked potential (SSVEP) induced by flickering visual targets presented on a screen [4, 5, 6].
Stroke patients who need restoration of motor function undergo motor-imagery (MI) BCI-controlled rehabilitation as an active training [7, 8].
Most EEG-based BCI systems are designed to detect/recognize reproducible time-asynchronous or time-synchronous EEG patterns of interest, depending on the schemes of BCI [9].
For example, the MI EEG pattern is an endogenous oscillatory perturbation sourced from the motor cortex without an explicit onset time [10].
On the other hand, a time-synchronous EEG pattern is time-locked to a specific event.
For example, the pattern of SSVEP is synchronized to the change of brightness on a flickering visual target.
The efficiency of BCI systems largely relies on the accuracy and robustness of the EEG decoder.
However, due to the low signal-to-noise ratio (SNR) [11] and non-stationarity [12] of EEG, translating perplexing EEG signals into meaningful information has been a grand challenge in the field.
Recent advances in deep learning (DL) have contributed to the rapid development of DL-based EEG decoding techniques [13].
DL models are capable of extracting features automatically according to given training data.
Convolutional neural network (CNN) is one type of the most common DL models and has achieved remarkable performance in tasks such as image recognition and object detection [14, 15, 16].
CNN models newly designed for EEG decoding use convolutional kernels that analogously function as conventional spatial and temporal filters but with extra flexibility to optimize the transformation of EEG data automatically through model training [17, 18, 19].
In addition to the fast growth of DL-based EEG decoders, geometric learning (GL) approaches, mostly based on
Riemannian geometry (RG), have been adopted in the field of BCI [20].
RG is a type of non-Euclidean geometry that has a different interpretation of Euclid’s fifth postulate (i.e. parallel postulate) [21].
In GL, geodesic between points on the manifold is a critical feature for classification tasks in BCI.
The power and spatial distribution of a segment of multi-channel EEG signals can be coded into a covariance matrix that is symmetric positive definite (SPD) in general.
The use of Riemannian geometry allows mapping of EEG data directly onto a Riemannian manifold where Riemannian metrics are insensitive to outliers and noise [22, 20].
RG can also avoid swelling effect [23], which is a common issue when employing Euclidean metric.
Furthermore, metrics on Riemannian manifold have several types of invariance properties [24, 22], which make the model have higher generalization capability to complex EEG signals.
In 2010, Barachant et al.
[25] proposed Minimum Distance to Mean (MDM) that maps target EEG data onto the SPD manifold to find the nearest class center.
Later on, they developed TSLDA [26] that projects data from the manifold to a specific tangent space where Euclidean classifiers are applicable.
RG-based classification for EEG decoding has shown extra robustness as the relationship between data samples can be stably preserved, leading to success in recent data competitions in the BCI field such as ’DecMEG2014’ 1 and the ’BCI challenge’ 2 .
The nascent field of geometric deep learning (GDL) [27] has expanded by emerging techniques to generalize the use of deep neural networks to non-Euclidean structures, such as graphs and manifolds.
Efforts have been made to transitioning useful operations from Euclidean to Riemannian spaces, including convolution [28, 29, 30], activation function [28, 29], batch normalization [31, 32], that facilitate the ongoing development of GDL tools.
SPDNet [28] is a Riemannian network for non-linear SPD-based learning on Riemannian manifolds using bi-linear mapping that mimics
Euclidean convolution for visual classification tasks.
ManifoldNet [29] offers high performance in medical image classification with manifold autoencoder.
[33] characterizes 3D movement via the manifold polar coordinate with a geodesic CNN.
[27] performs convolution on the manifold as a generalization of local graph or manifold pseudo-coordinate for vertex classification on graph and shape correspondence task.
In contrast of the vast develop of GDL in many other scientific fields, only few studies focus on decoding EEG data with a merge use of GL and DL.
[34] proposed a network architecture that integrates fusion of Euclidean-based module and manifold-based module with multiple LSTM and attention structures to extract spatiotemporal information of EEG.
[35] proposes a Riemannian-embedding-banks method that separates the entire embeddings into multiple sub-problems for learning spatial patterns of MI EEG signals based on the features extracted from the SPDNet.
[36] combines federated learning and transfer learning on Riemannian manifold using the spatial information of EEG.
[37] proposes deep optimal transport on the manifold to minimize the cost of domain adaptation from the source domain to the target domain.
[38] extracts multi-view representations of EEG.
These studies have established cornerstones toward the field of future GDL for EEG decoding, but the increment of performance is yet marginal.
Most of the above-mentioned techniques can not map the temporal information of EEG onto the manifold, or still rely on Euclidean tools to handle EEG features.
We herein propose a manifold attention network, a novel GDL framework, which maps EEG features on a Riemannian SPD manifold where the spatiotemporal
EEG patterns are fully characterized.
The main contributions of the present study are the following:
• a manifold attention network proposed for decoding general types of EEG data.
• a lightweight, interpretable, and efficient GDL framework that is capable of capturing spatiotemporal EEG features across Euclidean and Riemannian spaces.
• an empirical validation of our proposed model demonstrating its generalizable superiority over leading DL approaches in EEG decoding.
• neuroscientific insights interpreted by the model that not only echo prior knowledge but also offer a new look into the dynamical brain.
This article is organized as follows: we first brief the essential background of RG and manifold attention mechanism; next, we leverage the proposed MAtt architecture with details of model design; we then validate our proposed model experimentally; lastly, we interpret our proposed model with neuroscientific insights.
Our source code is released at https://github.com/CECNL/MAtt .
Preliminary
A manifold is considered as an expansion of curve and surface in Euclidean space.
It is a topological space that can locally regarded as an open set in Hilbert space.
Suppose a manifold is endowed with a differential structure (i.e. a collection of charts satisfying transition mapping, which is defined on the overlap of charts), it is then the so-called differential manifold [39].
Riemannian geometry is a differential manifold equipped with Riemannian metric.
We consider the symmetric positive definite (SPD) manifold, which allows us to manipulate manifold-valued data on the manifold directly.
The spatial information of EEG signal can be represented as a specific covariance matrix, which records the relationship between channels, and is a critical representation for us to understand EEG signals.
However, the solution of the Riemannian mean doesn’t have a close form once the manifold equipped with affine invariant metric (AIM), thus we need to calculate the approximate mean in an iteration manner [25, 29] until convergence conditions are satisfied.
However, Riemannian mean may cause a heavy computational load in deep learning because of its high complexity.
Therefore, we seek an approximation based on Log-Euclidean metric [24] as described below.
2.1 Notations
GL ( n, R ) :
= { A ∈ R n × n | determinant ( A ) ̸ = 0 } is a general linear group, which is the set of all real non-singular sqaure matrices.
( M , g ) denotes connected Riemannian manifold.
Sym ( n ) :=
{ S ∈ M n × n ( R ) | S T = S } is the space of all n × n real symmetric matrices, where M n × n ( R ) specifies the space of all real square matrices, ( . )
T is the transpose operator, and Sym + ( n ) :
= { P ∈ M n × n ( R )
| P = P T , v T Pv > 0 , ∀ v ∈ R n − { 0 }} is the set of all n × n symmetric positive definite(SPD) matrices.
<
A, B > F means the Frobenius inner product, defined as Tr ( A T B ) , where
Tr ( . ) is the trace operator.
Log ( . ) and Exp ( . ) are the principle logrithm operator for SPD matrix [40] and exponential operator for symmetric matrix respectively.
Both of them can be computed using the orthogonal diagonalization .
Exp :
Sym ( n ) (cid:55)→
Sym + ( n ) , an operator maps a symmetric matrix S ∈
Sym ( n ) to Sym + ( n ) by:
[Equation]
where V is the matrix of eigenvectors of S .
The inverse projection of Exp operation is Log operator: Log :
Sym + ( n ) (cid:55)→
Sym ( n ) is an operator that maps a SPD matrix P ∈
Sym + ( n ) to Sym ( n ) by:
[Equation]
where U is the matrix of eigenvectors of P , since P ∈
Sym + ( n ) , σ i > 0 , i = 1 , ..., n
2.2 Log-Euclidean metric
Log-Euclidean metric (LEM) offers an elegant, analogous, and efficient generalization to calculate the center on the SPD manifold than the affine-invariant metric (AIM) [24, 41].
LEM is a bi-invariant metric on the Lie group on the SPD manifold [24].
The geodesic distance from P 1 to P 2 on the
Sym + ( n ) is also given by [24]:
[Equation]
Furthermore, we can also define the Log-Euclidean mean( G ) via the Log-Euclidean distance:
[Equation]
where P 1 , ..., P k ∈
Sym + ( n ) .
Fortunately, the solution to the formula above has a closed form to follow, given by [42]:
[Equation]
We utilizes the weighted Log-Euclidean mean that is endowed with different weights in different
P l in our work.
We denote the weight of each P l as w l , where ∀ l ∈ { 1 , 2 , ..., k } .
Here, { w l } kl =
[Figure]
satisfies the convexity constraint definition (i.e. k (cid:88) l = w l = 1 , and w l > 0 ).
The definition and the corresponding weighted Log-Euclidean mean can be defined and derived as:
[Equation]
and
[Equation]
respectively.
Methodology
As shown in Figure 1(a), the architecture of MAtt includes components of the feature extraction (FE), the manifold attention module, transitioning from Euclidean to Riemannian space (E2R), and transitioning from Riemannian to Euclidean space (R2E).
3.1 Feature extraction of EEG signals
We adopt two convolutional layers to extract information of raw EEG signals, where the first convolutional layer performs spatial filtering to the multi-channel EEG signals and the second convolutional layer extracts spatiotemporal features.
Our parameter setting follows [19].
3.2 From Euclidean space to SPD manifold (E2R operation)
[Figure]
As illustrated in Figure1(b), we convert the embeddings from the feature extraction stage to the SPD data and map the feature embeddings from Euclidean space to the SPD manifold.
Suppose ˜ f denotes the embeddings after the feature extraction stage, we divide the whole embeddings into several epochs ˜ f 1 , ˜ f 2 , ..., ˜ f m , and calculate the sample covariance matrix(SCM) of each ˜ f
i , ∀ i ∈ { 1 , 2 , ..., m } .
By doing so, we get a sequence of covariance matrices that present the temporal information of the embeddings ˜ f
in the form of SPD data, called SCM ˜ f 1 , SCM ˜ f 2 , ..., SCM ˜ f m .
After we get some datapoints, we do trace-normalization and add a small number ϵ on each main diagonal element for each SCM ˜ f
i (i.e.
SCM ˜ fi tr (cid:16) SCM ˜ fi (cid:17) + ϵI ) where i ∈ { 1 , 2 , ..., m } , I is the identity matrix, and we set ϵ as 1e-5 in our source code.
The resulting SPD sequence is denoted as ˜ X =
[ ˜ x 1 , ˜ x 2 , ..., ˜ x m ] .
We add a small identity matrix on them to promise ˜ x i to be a well-defined SPD matrix.
3.3 Manifold attention module
Forward procedure: The input of this layer is a sequence of SPD data.
The overview of the manifold attention module is illustrated in Figure 2(a).
Motivated by [28] and [43], we capture the spatiotemporal information on the manifold.
Suppose the module takes a sequence of SPD matrices [ ˜ x 1 , ˜ x 2 , ..., ˜ x m ] , denoted as ˜ X .
Herein we have the query, key, and value in the form of SPD matrices on the manifold [43].
We convert the ˜ x
i to the q i , k i , and v i via bilinear mapping [28] and exploit non-linear and valid features from each segment.
Suppose the shape of ˜ x
i is d c × d c , and h q , h k , and h v is the mapping from ˜ x
i to q i , k i , and v i respectively.
We have:
[Equation]
where ˜ x i ∈
Sym + ( d c ) , W q , W k , and W v ∈ R d u × d c ( d u < d c ) denotes transformation matrices.
Moreover, to make sure the output q i , k i , and v i are also SPD matrices, transition matrices W q , W k , and W v are constrained as row-full rank matrices.
After we got q i , k i , and v i by bilinear mapping, we define the similarity for measuring the q i and k j
SPD matrices.
In Euclidean space, there are several ways to define the similarity.
A most common way is to use dot-product [43] to measure the similarity of query and key.
However, our query, key, and value are SPD matrices instead of vectors as regular attention.
We define the similarity based on the Log-Euclidean distance (Eq. 2) between query and key.
Suppose we have q i and k j , for some i, j ∈ { 1 , ..., m } .
The similarity sim ( . ) is a strictly decreasing function of distance [0 , ∞ )
(cid:55)→ [0 , 1] and is defined as: sim ( q i , k j ) =
[Equation]
i ,
[Equation]
: = α
[Equation]
Then, the attention matrix is:
[Equation]
We then use Softmax function to shrink the range along the row direction, making values in row have convexity constraint property.
The final attention probability matrix A ′ is:
[Equation]
where α ′ ij =
[Equation]
, m .
Finally, we combine the attention probability matrix and v 1 , v 2 , ..., v m to get the final output v ′ 1 , v ′ 2 , ..., v ′ m and define the output v ′
i ( ∀ i = 1 , 2 , ..., m ) via
Log-Euclidean mean as:
[Equation]
The forward procedure of proposed manifold attention module is illustrated in Algorithm 1.
Backward procedure: In order to perform gradient descent parameter updating on the Riemannian manifold, we employed the Riemannian gradient descent method to update the parameters.
The trainable parameters in this module are W q , W k , and W v .
We require the weight updated on Stiefel manifold [44, 28], denoted as St ( p, n ) =
{ X ∈ R n × p | X T X
= I p } .
Since our manifold attention module has a different mathematical architecture to those in Euclidean space, we herein extend
Euclidean gradients onto a Riemannian space.
To be precise, we expect our gradients on the Stiefel manifold to generate valid orthogonal weights.
The Euclidean gradients of the W q , W k , and W v within the attention module can be derived by the chain rule.
Suppose the L is the loss, the query, key, and value generated in the manifold attention module are q i , k i , and v i ∀ i = 1 · · · m respectively,
Require:
A sequence of SPD data { ˜ x i } mi =1 , transformation matrices W q , W k , W v 1: for i = 1: m do 2: q i =
W q ˜ x i W
Tq ; k i =
W k ˜ x
i W Tk ; v i
=
W v ˜ x i W Tv 3: end for 4: ∀ i, j ∈ { 1 , · · · , m } , A :=
[ α ij ] m × m = 1 1+ log (1+ δ L ( q i ,k j )) 5: A ′ =
Softmax ( A ) 6: for i = 1: m do 7: v ′ i =
Exp (cid:32) m (cid:88) l = α ′ il Log ( v l ) (cid:33) 8: end for 9: return a sequence of SPD data { v ′ i } mi = and the output of the attention module is v ′
i ∀ i = 1 · · · m (All notations here are the same as in the
Algorithm 1): ∇ W q L = m (cid:88) i =
∂ L ∂v ′
i ∂v ′
i ∂W
q = m (cid:88) i =
∂ L ∂v ′
i ∂v ′
i
∂q
i
∂q
i ∂W
q = m (cid:88) i =
∂ L ∂v ′
i   m (cid:88)
j = ∂v ′ i ∂α
ij ∂α ′
ij ∂q i 

∂q
i ∂W q ∇ W k L = m (cid:88) i =
∂ L ∂v ′
i ∂v ′
i
∂W k = m (cid:88) i =
∂ L ∂v ′
i   m (cid:88)
j = ∂v ′
i
∂k j ∂k j ∂W k 
 = m (cid:88) i =
∂ L ∂v ′
i   m (cid:88)
j = ∂v ′ i ∂α
[Equation]
ij ∂α
[Equation]
∂k
[Equation]

[Equation]
i
[Equation]

[Equation]
j
[Equation]
∂v
[Equation]

The way of the attention module to impose the Euclidean gradient to the Stiefel gradient on the manifold [44] is: Suppose ∇ W v L is a d u × d c matrix and W v is on the manifold.
Then the projection of ∇ W v L onto the normal space, which is the orthogonal complement of the tangent space based on the tangent point W v is:
[Equation]
where the D sym is defined as D + D T.
The tangent component of the St ( d c , d u ) can be defined as the subtraction between the Euclidean gradient and normal component π N ( ∇ W v L ) from Eq.
[Equation]
[Equation]
Now we have the Stiefel gradient ˜ ∇ W v L , which is the tangential direction that we update the parameters.
We use the retraction operation, which is a smooth mapping to map the final weight back to the St ( d c , d u ) .
Finally, We have the updated weight:
[Equation]
where λ is the learning rate, Γ is the retraction operation defined in the QR decomposition [45].
The backward procedure for W q and W k are the same.
For more concepts about the optimization in
Riemannian geometry, we recommend readers refer to [44, 45].
As shown in Figure 2 (b), the output v ′ 1 of our attention module can be comprehended as a projection that translates the weighted sum, on the tangent space, of three different matrices encoded by three different epochs and corresponding attention scores α ′ 11 , α ′ 12 , α ′ 13 (or weights) to a specific representative matrix v ′ 1 on SPD manifold.
Herein the weights for the weighted sum on the tangent space is assigned by its query matrix q 1 and corresponding keys k 1 , k 2 , k 3 to generate the relevance score between q 1 and k 1 , k 2 , k 3 [43, 46].
3.4 From Riemannian manifold to Euclidean space (R2E) and loss layers
After passing through attention module, ReEig layer is used to imitate the ReLU function.
But it is different from the ReLU function that sets the threshold to the value of input, ReEig sets a threshold to the eigenvalue of the input, can be defined in [28].
R2E operation aims to map the SPD data back to the Euclidean space for executing the final classification, which is composed of a Log layer and regular flatten layer in [28] sequentially.
Log layer is the most common skill in geometric deep learning to project the SPD data to the Euclidean space.
By doing so, we reduce the manifold to a flat space by Log ( . ) operation.
We take Log ( . ) operation on the output from the attention module layer v ′ 1 , v ′ 2 , ..., v ′ m ∈
Sym + ( d u ) .
Denote the whole R2E operation as h L :
Sym + ( d u ) (cid:55)→ R d u × ( d u +1) / 2 : h L ( v ′ i )
= flatten ( Log ( v ′ i ))
= flatten (cid:0) S ( diag ( log ( σ 1 ) , · · · , log ( σ d u )))
S T (cid:1) where S is the eigenvector matrix of v ′ i , and σ 1 , · · · , σ d u are the eigenvalues of v
i .
The Log ( . ) operation is the same as Eq. 1, and the flatten ( A ) operation flatten the upper triangle of the arbitrary symmetric matrix A .
Finally, we set a fully connected layer and regular softmax operation on embeddings after R2E operation.
Suppose the output from the whole model stream is ˆ y, the groundtruth is y, we define the loss L as the cross-entropy loss of ˆ y and y .
Experiments
Here we evaluate the proposed MAtt using both time-asynchronous and time-synchronous EEG data to give empirical evidence of the advantages.
The performance in a general use for EEG decoding is compared against leading DL-based techniques.
We incorporate the BCI Competition
IV 2a Dataset
(BCIC-IV-2a) [47] to assess the performance on time-asynchronous motor-imagery (MI) EEG decoding , the MAMEM EEG SSVEP Dataset II (MAMEM-SSVEP-II) [48] and the
BCI challenge error-related negativity (ERN) dataset (BCI-ERN) [49] to assess the performance on time-synchronous SSVEP and ERN EEG decoding.
Previous and current state-of-the-art DL-based models listed for comparison with MAtt include MBEEGSE [50], TCNet-Fusion [51], EEG-TCNet [52], FBCNet [53], SCCNet[19], EEGNet [17], and ShallowConvNet [18].
4.1 Datasets and preprocessing
Data set I (MI) :
We employ the BCIC-IV-2a as a representative dataset providing time-asynchronous
EEG data.
The BCIC-IV-2a dataset is one of the most commonly used public EEG dataset released for the BCI Competition IV in 2008 [47].
It contains EEG recordings in a four-class motor-imagery task from nine subjects with two repeated session each on different days.
During the task, the subjects were instructed to perform imagination of one of the four types of movements (right hand, left hand, feet, and tongue) for four seconds after an instructional cue.
Each session consists of a total of 288 trials with 72 trials for each type of the motor imagery.
The EEG signals were recorded by 22 Ag/AgCl EEG electrodes at the central and surrounding regions in a sampling rate of
Hz.
We performed standard preprocessing procedures for the 22-channel EEG signals, including 1)
Down-sampling from 256 Hz to 128 Hz, 2) Band-pass filtering at 4-38 Hz, and 3) Segmenting EEG signals at 0.5-4s (438 timepoints) of the onset of cue for each trial.
Data set II (SSVEP) :
The MAMEM-SSVEP-II dataset was incoporated as the representative dataset of time-synchronous EEG data.
It consists of EEG recordings from 11 subjects performing a SSVEPbased task where the subjects were instructed to gaze at one of the five visual stimuli flickering at different frequencies (6.66, 7.50, 8.57, 10.00, and 12.00 Hz) for five seconds.
Each subject executed five runs of five cue-based trials for each of the five stimulation frequencies.
The EEG signals were acquired using the EGI 300 Geodesic EEG System (GES 300) equipped with 256 channels in a sampling rate of 250 Hz.
There are 5 sessions in this dataset.
We assigned the early blocks (session 1, 2, and 3) as the training set, session 4 as validation set and the remaining (session 5) as the test set to maximize the efficacy of the cross-session validation.
The preprocessing procedures for this dataset were 1) Band-pass filtering at 1-50 Hz, 2) Selecting eight channels (PO7, PO3, PO, PO4, PO8, O1,
Oz, and O2) in the occipital area, the location of visual cortex, and 3) Segmenting each trial into four 1-second segments at 1s-5s of the onset of cue, yielding a total of 500 trials of 1-second 8-channel
SSVEP signals for each subject, thus the time length of input EEG data is 125.
[Table]
Data set III (ERN) :
Another time-synchronous BCI Challenge ERN dataset (BCI-ERN) is adopted to validate the capacity of presented MAtt.
The BCI-ERN dataset [49] was employed in the BCI
Challenge on Kaggle 3 .
Twenty-six subjects participated in a P300-based BCI spelling task and measured the ERN, a type of event-related potential (ERP), elicited from the feedback of erroneous input displayed by the BCI speller.
The aim of the experiment was to detect and determine the type of signal perturbation evoked by the erroneous feedback from a BCI speller to improve and assess the robustness.
Therefore this task is a binary-and-imbalanced decoding task because the BCI speller detects more amount of correct inputs than the amount of erroneous inputs.
EEG recordings were recorded by 56 Ag/AgCl EEG electrodes (VSM-CTF compatible system) whose region followed the extended 10-20 system in sampling rate of 600 Hz.
There are five sessions (60 trials for the first four sessions and 100 trials for the last session) per subject, and the duration of a single EEG trial is 1.25 seconds.
We used the 16 subjects released in the early stage of the Kaggle competition.
The preprocessing steps include 1) Downsampling from 600 Hz to 128 Hz and 2) Band-pass filtering at 1-40 Hz.
Each trial has a size of 56 channels by 160 timepoints after the preprocessing step.
Model training and validation: A series of experiments were conducted to evaluate the performances of the MAtt against other EEG decoders with the context of real-world BCI usage taken into account.
In the real-world usage of BCI, a user usually needs to go through a training session for collecting a sufficient amount of individual EEG data for training the decoding model before executing the BCI system.
To stick with the practical scenario, we performed an individual training scheme where a chunk of trials within a subject are assigned to the training set and the left-over trials within the same subject are used for testing [19, 53].
For the BCIC-IV-2a dataset, we used the first session of a subject to the training set where one out of eight was used for validation for
MAtt with m = 3 .
The model with the lowest validation loss within 350 iterations was used for testing on the second session of the same subject.
For the MAMEM-SSVEP-II/BCI-ERN dataset, we assigned the first four sessions of a subject to the training set where one out of four was used for validation for MAtt with m = 7 / m = 3 .
The model with the lowest validation loss within 180/ iterations was used for testing on the fifth session of the same subject.
The classification performances for BCIC-IV-2a and MAMEM-SSVEP-II datasets were estimated by the mean accuracy across ten repeats for each subject.
On the other hand, we use the same criterion as [17], herein the AUC score [54] is adopted to estimate the performance of BCI-ERN dataset due to the imbalanced issue.
4.2 Performance comparison
We validate the performance of MAtt against other leading methods.
The criteria of selecting the baseline model collection here is based on: 1) code availability and completeness and 2) solid evaluation (e.g. cross-session) without additional auxiliary procedures (e.g. manual feature extraction, data augmentation, pre-trained model, etc).
As shown in Table 1, MAtt outperforms all other leading
DL methods on both time-synchronous (SSVEP) and -asynchronous (MI) EEG decoding.
However, for the ERN dataset, MAtt seconds to the EEG-TCNet with 1% deficit but is at least 7% higher than EEG-TCNet on other two kinds of EEG decoding tasks.
As most DL-based EEG decoders are
[Table]
designed for a single type of EEG decoding task due to high variability between different types of
EEG data [53, 52, 19], the experimental results suggest that MAtt provides high-performancee EEG decoding across types of EEG data.
Our study attests the robustness of proposed MAtt, which has strong generalization capacity to adapt general types of EEG data compared to other leading DL models.
4.3 Ablation study
We assess the significance of each of the major component in MAtt via a series of ablation analysis.
As shown in Table 2, the accuracy reduces if we only use one component in our MAtt to do the classification.
However, the combination of our proposed manifold attention module and feature extractor (FE+MA) achieves the best accuracy on all datasets.
This implies that there is no redundant component in our proposed MAtt, and each part is needed for its non-negligible functions.
The feature extractor aims to denoise and preprocess the EEG signals, and the attention module focus on integrating the preprocessed EEG signals and capturing underlying dynamics in the latent features.
We can further compare the result between FE+SA and FE+MA to check the necessity of MA: the performance of FE+MA significantly outperform the regular self-attention based FE+SA on all validation EEG datasets.
4.4 Model interpretation
gradient response (computed as in [55, 56]) across time from the visualization of the model S3 in the
BCIC-IV-2a dataset for the four motor-imagery classes (left hand, right hand, feet, and tongue).
Dark red marks the brains region presenting strong gradient activation at C4 (over right motor cortex) for the left hand, C3 (over left motor cortex) for the right hand, CPz (over motor cortex) for the feet and the tongue motor imagery.
Through analysis for the interpretation of the proposed model, MAtt, we are able to uncover the underlying characteristics learnt from the data.
Figure 3 illustrates the gradient response for MI EEG decoding across channel and across time.
We can see left/right hand MI responses are strong at C4 and C3 corresponding to right/left motor cortices that control the lateral motor functions of the contralateral side of the body [57].
Both feet and tongue MI, that are not lateral movements, presents strong responses at
CPz above the midline of motor cortex.
The spatial distribution clearly exhibit asymmetric pattern for left/right hand MI and symmetric pattern for feet/tongue MI.
The temporal information is available in Figure 4, where all four types of MI induce strong response at mu band around 10 Hz.
This result is in line with the well-known association between motor function and mu rhythm in EEG recordings [57].
The visualization of our model for SSVEP decoding is exhibit in the appendix.
Figure 5 depicts the distribution of attention scores across epochs between the (a) MI, (b) SSVEP, and (c) ERN
EEG signals.
Here, the attention score refers to the average of the relevance score of an attention network, as described in [43] and has been applied to interpreting an attentionbased EEG decoder [46].
For MI EEG signals, attention score is the highest at the first epoch and decreases in the following epochs, which implies that the beginning of the motor imagery may contribute a higher importance determined by the manifold attention module.
The profile of attention score for SSVEP EEG signals presents a similar traits that the earlier epochs relate to higher importance.
For ERN EEG signals, attention score is the highest on the first two epochs.
As we observe a consistency cross EEG datasets that higher attention scores lie in earlier epochs, this may infers that the attention module relies largely on the similarity to the early stage of a trial, which is analogous to baseline correction, a major common procedure in conventional EEG signal processing [58].
This analysis reveals the capability of MAtt in handling the non-stationarity of the dynamical brain activity.
Conclusion
We propose a manifold attention network as a novel GDL framework for both time-synchronous and -asynchronous EEG decoding.
Using back propagation based on the Stiefel manifold, the proposed
MAtt is capable of mapping EEG features onto a Riemannian manifold, where spatiotemporal EEG patterns are captured and characterized, within a lightweight architecture.
The experimental results suggest the superiority of MAtt over current leading DL methods for both time-synchronous and -asynchronous EEG decoding.
Spatial and temporal EEG patterns interpreted from the model are in line with prior neuroscientific knowledge and shed light on potential possibility of tracking the brain dynamics.
In sum, our privileged method, MAtt, improves the SOTA performance of EEG decoding, and is expected to impact on GDL-based EEG processing with generalizable efficiency and robustness for future development of various BCI systems.
Acknowledgement
This work was supported in part by the National Science and Technology Council (109-2222-E009-006-MY3, 110-2221-E-A49-130-MY2, and 110-2314-B-037-061) and in part by the Higher
Education Sprout Project of the National Yang Ming Chiao Tung University and Ministry of Education of Taiwan.