A C AUSAL V IEWPOINT ON M OTOR -I MAGERY B RAINWAVE D ECODING
A BSTRACT
In this work, we employ causal reasoning to breakdown and analyze important challenges of the decoding of Motor-Imagery (MI) electroencephalography (EEG) signals.
Furthermore, we present a framework consisting of dynamic convolutions, that address one of the issues that arises through this causal investigation, namely the subject distribution shift (or inter-subject variability).
Using a publicly available MI dataset, we demonstrate increased cross-subject performance in two different MI tasks for four well-established deep architectures.
I NTRODUCTION
Brain-Computer Interface (BCI) technology primarily aspires to provide a direct neural communication and control between individuals (subjects) and computers.
This is feasible by analyzing brainwaves captured by EEG signal recordings using signal processing and Machine Learning (ML) techniques.
One of the first and most popular BCI paradigms is Motor-imagery (MI).
MI-BCIs are based on a cognitive process, by which a subject mentally simulates a motor action, for example the movement of a hand or foot, without actually executing it (Decety & Ingvar (1990)).
Developing
MI-BCI systems mainly relies on robust decoding of a subject’s motor intentions from the recorded
EEG signals, under the prior assumption that these signals encode that relevant information, and are mainly used for movement rehabilitation purposes (e.g. Mane et al.
(2020), Robinson et al.
, (Sebasti ´ an-Romagosa et al.
In recent years, Deep Learning (DL) techniques - and most specifically Convolutional Neural Networks (CNNs) - have largely alleviated the need for manual feature extraction, achieving state-ofthe-art performance in various areas, most notably Computer Vision (Chai et al.
Due to their massive progress, CNN-based feature extractors have been introduced in various paradigms in the field of BCIs (e.g. Antoniades et al.
(2016), Rezaeitabar & Halici (2017), L¨angkvist et al.
(2012), Wulsin et al.
, in an effort to become generic EEG signal processing tools.
One of the core challenges that a BCI - or more generally the decoding of EEG signals - faces is to cope with changes in data distributions across different subjects.
Each individual has a unique brain anatomy and functionality that makes the discovery and exploitation of shared invariant features extremely difficult.
Therefore, modern DL-based BCIs tend to fail to generalize well in unseen subjects due to this type of data distribution shift.
Causal reasoning provides tools to breakdown and analyze important aspects of a BCI task, identify and possibly resolve some of these challenges by employing appropriate ML strategies.
The methodical breakdown of a BCI task and the identification of the causal relationships between the various variables of interest take into account the expert’s knowledge of the involved biological and neurophysiological processes and can be of vital importance when designing and building ML-based models in the field of BCI.
In this work, we focus mainly on MI-BCI systems, and inspired by the work of (Sch¨olkopf et al.
, we analyze the task of MI EEG signal classification through the lens of causal reasoning.
Motivated by this causal analysis, we introduce a framework based on dynamic convolutions that tackles the identified problem of subject distribution shift.
C AUSALITY IN M OTOR - IMAGERY B RAINWAVE D ECODING 2. T ASK D EFINITION
In a Motor-Imagery (MI) classification problem, we want to accurately predict the mentally performed task from a recorded EEG signal.
Mathematically, given an input EEG signal X , we train a statistical model to predict the correct MI task Y , which can be the imagery movement of a hand or foot.
In essence, this statistical model tries to estimate the conditional probability P ( Y | X ) using an appropriate objective function.
Causal reasoning is the analysis of a task / problem in terms of cause-effect relationships between the different variables of interest: if a variable A is a direct cause of variable B, we express it as A → B (A causes B or B is the effect of A).
In ML-based tasks, given the input X and the prediction target Y , we can establish that the task to estimate P ( Y | X ) can be either (Castro et al.
• Causal: when X → Y , prediction effect from cause • Anti-causal: when Y → X , prediction cause from effect
Using the above basis, we can define an MI EEG classification task as an anti-causal problem, since the true MI intention (observed with the MI label Y ) can be considered the cause of the recorded
EEG signal X .
Additionally, inspired by (Castro et al.
, we can consider X as a sequence of imperfect observed measurements (in sensor-space) of the true unobserved brain activations Z within, mainly, the cortical areas responsible for the sensorimotor rhythms, i.e. Z → X .
Therefore, using a causal diagram, an MI EEG classification task can be described as:
[Equation]
As a consequence of the above anti-causal definition and causal diagram, we can explore the problem of MI EEG classification through the following causal factorization:
[Equation]
2. T ASK C HALLENGES
Through this causal breakdown, we can categorize the major challenges associated with MotorImagery (MI) EEG classification tasks into three main categories.
Challenges related with the:
Training EEG signals.
One of the renowned challenges in motor-imagery classification problem - as in any medical-related machine learning problem - is the scarcity of labelled data due to the lengthy acquisition process.
Subjects are required to spend hours in a laboratory facility performing subsequent motor-imagery tasks.
This process has been reported to cause fatigue and discomfort, even when devices with dry electrodes are utilized.
To make things worse, due to the wide variety of available EEG recorders in the market, the data acquisition can be undertaken with various devices which have completely different specifications (e
g. number of electrodes, sampling frequency to name just a few), making the combination of publically available EEG datasets extremely difficult.
Anatomical differences of subjects.
Each subject has a unique brain anatomy and functionality that results in polymorphous neural activity patterns when appeared in the surface observed EEG signal.
When designing a generic ML-based MI-BCI, researchers need to take this inter-subject variability (subject distribution shift) into account.
Class Imbalance.
Class imbalances can arise between the training and the deployment set of a MI-BCI.
It is necessary for the training set to be as closely balanced to the deployment set as possible when training machine learning models.
P ROPOSED F RAMEWORK
In this work, we mainly focus on the challenge of subject distribution shift (or inter-subject variability).
As described in the next section, we will use a publicly available MI dataset - which contains a large number of different subjects, is class balanced, has relatively enough trials per subject and all trials come from a single EEG recorder, essentially solving all the above identified challenges but the subject distribution shift.
In terms of the causal factorization 2, the problem of inter-subject variability can be seen as a distribution shift S where:
[Equation]
Here, we propose a framework that tackles the problem of subject distribution shift (or inter-subject variability) and can be applied to any established CNN-based MI-BCI architecture, resulting in a statistically significant performance increase.
Our framework is based on the concept of dynamic convolutions as originally introduced in (Chen et al.
In their work, they present dynamic convolution layer as an effective way to increase model’s complexity without increasing the network’s depth and / or width, since each convolutional layer is computed by dynamically mixing multiple parallel learnt convolutional kernels based on an input-dependent attention vector.
Inspired by this concept, we utilize dynamic convolutions in the domain of MI brainwave decoding.
Instead of having a BCI architecture that tries to discover a common latent space for all k subjects in the training set, we use k parallel trainable convolutional kernels (corresponding to the k available training subjects) for each convolutional block of a CNN-based BCI network.
Using a subject attention network that learns to distinguish between the available individuals, we decouple the subjects and essentially train simultaneously k parallel personalized models of the same BCI architecture, as illustrated in Figure 2.
Our subject attention network consists of the first order wavelet scalogram of the input EEG signal
X followed by a global average pooling across time and frequency.
Mathematically, let x ( t ) ∈ R T denote a one-dimensional input EEG signal, where T is the number of initial EEG time points, and ψ λ ( t ) be a wavelet.
The 1st order scalogram is defined as X ( λ, t ) =
| x ( t ) ∗ ψ λ ( t ) | .
To perform this operation, the raw input signal from each EEG channel is convolved with a wavelet kernel with size (1 , W ) =
(1 , F s 2 ) where F s is the sampling frequency.
This wavelet kernel follows the real Gabor wavelet format:
[Equation]
(4) with t ∈ [ − W 2 , ..., W 2 ] and 1 σ denotes the bandwidth and λ the normalized frequency of the Gabor wavelet and these two properties are the only trainable parameters of this layer.
During training, λ is restricted ( λ ∈ [0 , 12 ] ) to satisfy the Nyquist theorem.
The proposed framework takes the EEG signal X as input and tries to learn both the correct MI task Y (estimate the conditional probability P ( Y | X ) ) as well as the correct subject id π (estimate the conditional probability P ( π | X ) ).
The subject attention network and the k parallel convolutional kernels are trained simultaneously using the following loss function:
[Equation]
where acc is the training accuracy of the subject attention network and ℓ denotes the cross-entropy function ( ℓ Attention for the subject attention network and ℓ MI for the MI classification task).
This loss function effectively enforces first the training of the subject attention network and, as the attention’s accuracy increases, it switches its focus to train the parallel convolutional kernels for the different MI tasks.
As also suggested in (Chen et al.
since softmax does not work well due to its near one-hot output, we use a large temperature of 30 in the softmax of the attention network during training in order to flatten the framework’s attention, allow a broader gradient backpropagation and effectively assist in the attention network’s training in the early epochs.
On the other hand, during evaluation we use the hard softmax (temperature of 1).
During inference, when an input EEG signal from a new unseen subject S x is processed, it passes firstly through the attention network and the subject attention vector π is computed.
Empirically, it was shown that this vector is quite sparse, and if it was used during inference, only a handful of parallel convolutional kernels would be utilized during the kernel mixing.
Instead, we would ideally like to use knowledge from all k individuals and ”shift” the attention more to the most relevant subjects.
To accomplish that, we compute what we call the ”uniformly attended vector” A* .
If there was no attention network, the k parallel convolutional kernel would be mixed with a uniform factor
A i = 1 k .
To compute the ”uniformly attended vector”, the uniform attention vector A is combined with the subject attention vector π and the result is passed through a softmax activation to flatten the attention across all subjects - while maintaining the focus on the most relevant ones.
Mathematically, this operation can be described as:
[Equation]
where σ denotes the softmax operation, A the uniform attention vector with A i = 1 k , π the subject attention vector with (cid:80)
i π i
= 1 and A* the ”uniformly attended” vector where (cid:80) i
A ∗ i = 1 .
In other words, using the causal factorization 3, our proposed framework tries to to estimate the
P S x ( Z | Y ) of a new unseeen subject S x as the linear combination of k learned conditional probabilities.
More specifically:
[Equation]
E XPERIMENTS
We tested our proposed framework in four well-established BCI architectures, namely DeepConvNet (Schirrmeister et al.
(2017)), ShallowConvNet (Schirrmeister et al.
, EEGNet (Lawhern et al.
and EEG-Inception (Santamar´ıa-V´azquez et al.
in two MI tasks formed based on the publically available MI dataset Physionet (Goldberger et al.
The one is a binary classification task (MI Left vs Right Hand) and the other is a 3-class classification problem (MI Left Hand / Right Hand / Feet).
The original Physionet dataset includes brain recordings from 109 healthy participants, registered via 64 EEG sensors with a sampling frequency of 160 Hz, while performing a series of pseudo-randomized cue-triggered MI tasks.
In our experiments, we first excluded data from 6 participants (subjects 88, 89, 92, 100, 104 and 106) due to differences in either the sampling frequency or duration of the performed tasks.
We evaluated the performance of the normal networks and their equivalent dynamic networks in a leave-one-subject-out fashion.
We trained the normal networks for 50 epochs with learning rate of 0.001 while their dynamic versions for 30 epochs (to avoid overfitting) - in the first 20 epochs with learning rate of 0.01, to assist the attention’s Gabor filters to quickly adapt to the data
[Table]
and 10 epochs with learning rate of 0.001 and frozen attention, to fine-tune to the MI task.
In both cases [Table] we use an Adam optimizer.
Finally, we performed a paired t-test between the the normal and dynamic results
[Table]
C ONCLUSION
In this work [Table] we analyze the task of MI EEG classification through the lens of causal reasoning.
To the best of our knowledge, this is the first work that brings machine learning in conjunction with causal reasoning to the domain of EEG.
Through this analysis, we identify and analyze some of the major challenges and we introduce a framework based on dynamic convolutions that tackles the problem of subject distribution shift (inter-subject variability).
Our proposed framework demonstrates increased performance when applied to different BCI architectures.
Although this constitutes early results in this research direction, we strongly believe that the proposed framework can have further benefits in MI-BCIs and, in future work, we plan to use it to tackle more, if not all, challenges detailedly described in this causal analysis.