Spatio-temporal analysis of feedback-related brain activity in brain-computer interfaces
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Electroencephalography-based Brain-computer interfaces are systems that inferbrain signals recorded using electroencephalography (EEG) to provide means ofcommunication for patients suffering from the locked-in syndrome where normalneuromuscular pathways are not available.
One challenge in EEG-based BCIs isthe non-stationarity of the EEG signal.
A major contributor to this is the feedbackrelated brain activity.
Since EEG is a time series recorded at multiple sites on thescalp, one can estimate covariance matrices for both time and space.
On the otherhand, covariance matrices are symmetric positive deﬁnite matrices and lie on a
Riemannian manifold.
In this work, we investigate spatio-temporal aspects of thefeedback-related brain activity by considering both space and time covariancesin Euclidean space and on the Riemannian manifold.
We propose two novelmethods to incorporate both spatial and temporal features and show improvedresults compared to existing methods.
Introduction
Brain-computer interface (BCI) systems provide a means of communication for patients sufferingfrom the locked-in syndrome through bypassing the normal neuromuscular pathways [8, 18].
Thesesystems record signals directly from the brain using methods such as electroencephalography (EEG)and infer the intention of the user from them.
Motor imagery (MI) BCI are common BCI paradigmsin which the user imagines moving a part of her/his body.
Movement imagination results in anevent-related desynchronization (ERD) (decrease in power) in various frequency bands [16, 13].
Motor imagery of different body parts leads to different spatial desynchronization and this informationis used by the BCI to distinguish among the imagined movement classes.
In practice, for example, auser imagining moving her/his right or left hand can be mapped to a switch to turn on/off the light orto control the movement of a wheelchair, etc.
Non-stationarity in EEG signal is a barrier for real-world application of the EEG-based BCI systems.
Feedback-related brain activity is one contributor to this non-stationarity [6, 17, 14].
In this work,we investigate the spatio-temporal aspects of the feedback-related brain activity in a motor imagery
BCI paradigm.
Covariance matrices play an important role to extract features for classiﬁcation in
MI-BCI [5, 10].
By investigating both space and time covariances in the Euclidean space and on the
Riemannian manifold, our goal is to distinguish whether the BCI has made an error perceived by theuser in a motor imagery BCI paradigm.
Methods
Data were recorded from 10 participants participating in a motor imagery task to control a cursor ona monitor in front of them.
Data were recorded using a 64-channel BrainAmp system (Brain Products
GmbH).
Each trial began with the cursor at the center of the screen and the target at either left orright side.
The distance from the center to each side was ﬁxed to three cursor jumps.
The cursormoved at the speed of one step per second.
The trial ended when the cursor hit the target or the otherend of the screen.
Participants were lead to believe they were in control of the cursor, but in reality,the cursor moved based on a pre-determined sequence that was the same across participants.
Thereason was to have enough steps towards or away from the target irrespective of the participant’smotor imagery performance.
Data were downsampled to 500 Hz and sections that were contaminatedwith excessive noise were removed.
Next, independent component analysis (ICA)
[11] was appliedto remove muscle and eye artifacts.
Pre-processing was done in MATLAB [12] and EEGLAB [7].
Feature extraction and classiﬁcation were implemented in Python.
We used the pyRiemann toolbox[2] to calculate Riemannian distances and means.
For detailed information on the pre-processing,please refer to [citation removed for anonimity].
We looked at different frequency bands, namely 1-3, 2-5, 4-7, 6-10, 7-12, 10-15, 12-19, 18-25,19-30, 25-35, 30-40 Hz to cover the low and high theta, mu, and beta frequency bands and to coverfor potential individual differences [15].
Data were epoched at 100 to 1000 ms after each cursormovement.
This is called a “step".
Classiﬁcation results are reported on a single movement, i.e.for every step.
We train a classiﬁer to distinguish whether the user is satisﬁed with the last cursormovement or not, i.e., if the cursor had moved towards (good) or away (bad) from the target.
We callthis a good/bad classiﬁer.
The classiﬁcation methods we propose and the ones we compare them toare based on covariance of EEG epochs.
We estimated the time and space covariance of each EEGepoch as described next.
2.1 Space and time covariances
Let X i ∈ R c
× t be an EEG frame where c is the number of channels, t represents the number oftime samples and i ∈ { 1 , ..., N } where N is the number of EEG frames available.
The sample spacecovariance for each EEG epoch is deﬁned as follows:
[Equation]
Similarly, one can deﬁne a sample time covariance for each EEG epoch as follows:
[Equation]
Since the number of channels in our case c = 64 is smaller than the number of time samples,covariance matrices are not full rank and thus not positive deﬁnite.
This is speciﬁcally required for the
Riemannian methods.
On the other hand, since the highest cut-off frequency among the consideredfrequency bands is 40 Hz, we downsampled each EEG frame along the time dimension by 5 andregularized both space and time covariances as follows:
[Equation]
where I is the identity matrix with the same size as C , α is the regularization parameter and trace ( C )is the sum of values on the diagonal of matrix C .
In what follows, α = 0 .
It is a part of our futurework to use data-driven methods such as Ledoit-Wolf [9] to estimate the regularization parameter.
We compare 6 methods for this task that differ in how they estimate the covariance from EEG data(either time or space) and how they extract corresponding features and infer the class label from thisinformation.
These methods are explained next.
2.2 Filterbank
Common
Spatial Patterns (
FB-CSP
Proposed by Ang et al.
[1], this method ﬁrst ﬁnds the average of the sample covariances for each of thegood and bad classes in the Euclidean space as Σ g = 1 N (cid:80) i C g S
i and Σ b = 1 N (cid:80) i C b S i respectivelywhere N is the number of steps in the (balanced in number) good and bad classes.
C gS i and C bS irepresent the space covariance of the i th good and bad steps respectively.
CSP ﬁlters are estimatedfor each frequency band by simultaneous diagonalization of the two covariance matrices:
[Equation]
where Λ g and Λ b are diagonal matrices and W is selected such that Λ g + Λ b
= I [5].
This can beachieved by solving the generalized eigenvalue problem: Σ g w = λ Σ b w .
CSP ﬁlters w j , j = 1 , ..., Care the columns of W .
We considered the top 3 ﬁlters for each of the good and bad classes in eachfrequency band, i.e. the eigenvectors associated to the largest and smallest eigenvalues λ .
Features areselected as the log of the variance of the EEG signal passed through these ﬁlters.
A linear discriminantanalysis (LDA) classiﬁer was trained on the selected features.
2.3 Filterbank-Common Temporal Patterns (FB-CTP)
Common temporal patterns (CTP) were proposed by Yu et al.
CTPs are found by solving theequations that were described earlier for CSP, except the sample mean of the good and bad timecovariances are considered instead of space covariances.
In their work, Yu et al.
combined the featuresfrom common spatial and temporal ﬁlters.
However, here we only considered the common temporalﬁlters as a stand-alone method to compare with FB-CSP and another time covariance methods on the
Riemannian manifold that will be described next.
We proposed a ﬁlter-bank version of CTP method similar to FB-CSP.
We considered the top 3 ﬁltersfor each of the good and bad classes in each frequency band.
Features are selected as the log ofthe variance of the EEG signal passed through these ﬁlters.
A linear discriminant analysis (LDA)classiﬁer was trained on the selected features.
2.4 FB-CSP-CTP
We propose this method to combine spatial and temporal features from CSP and CTP methods.
First
CSP ﬁlters were trained and the top 3 ﬁlters for good and bad classes were selected.
Then the EEGframes were passed through the 6 selected CSP ﬁlters.
Next, CTP ﬁlters were trained and againthe top 3 ﬁlters for each class were selected.
The CSP ﬁltered EEG frames passed through theselected CTP ﬁlters were used as features ( 6 × 6 ).
This procedure was applied to each frequency bandseparately to have 11 × 6 × 6 features and a logistic regression was trained on them for classiﬁcation.
2.5 Filterbank-distance to spatial Riemannian means (FB-DRM-S)
Since covariance matrices are symmetric positive deﬁnite(SPD) matrices, they lie on a Riemannianmanifold [3].
Let P ( n ) =
{ P ∈ S ( n ) , P > 0 } be the set of all n × n SPD matrices.
The Riemanniandistance between two SPD matrices, P 1 and P 2 is deﬁned as follows:
[Equation]
where λ i are the eigenvalues of ( P − 1 1 P 2 ) which are real positive (non-zero) values as P 1 and P 2 areboth SPD.
Also, || .
|| F is the Frobenius norm of a matrix and Log ( P ) is the matrix logarithm of P .
Since SPD matrices are diagonalizable, let P = WDW − 1 .
Then Log ( P ) =
Wlog ( D ) W − 1 wherelog ( D ) i = log ( D i ) and log ( D ) i and D
i are the i th element on the diagonal of matrices log ( D ) and
D respectively.
Mean of the SPD matrices P 1 , P 2 , ..., P m on the Riemannian manifold is deﬁned as follows [3]:
[Equation]
Based on these mathematical tools on the Riemannian manifold, we propose a ﬁlter bank generalization of the minimum distance to Riemannian mean (MDM) classiﬁer proposed in [4].
First, we foundthe Riemannian mean of the good and bad space covariances on the training set.
Then, the featureswere chosen as the distances to the Riemannian means of good and bad classes in each frequencyband (a total of 22 features: 11 frequency bands × 2 good and bad classes) and logistic regressionwas trained on the selected features.
2.6 Filterbank-distance to temporal Riemannian means (FB-DRM-T)
Here, we ﬁrst found the Riemannian mean of the good and bad time covariances on the training set.
Then, the features were chosen as the distances to the Riemannian means of good and bad classes
[Table]
in each frequency band (a total of 22 features: 11 frequency bands × 2 good and bad classes) andlogistic regression was trained on the selected features. 2.7 Filterbank-distance to spatial and temporal Riemannian means (FB DRM -TS)
This method combines the two previous ones: we ﬁrst found the Riemannian mean of the good andbad time and space covariances on the training set.
Then, the features were chosen as the distances tothe Riemannian means of good and bad time and space covariances in each frequency band (a totalof 44 features: 11 frequency bands × 2 good and bad classes × 2 time and space covariances).
Alogistic regression was trained on the selected features.
Results
Since good and bad classes are not balanced, we sub-sampled the larger class to have balanced classes.
We randomly sub-sampled the larger class and made 5 instances of train-validation and test setseach having the same number of good and bad steps.
Results are presented in table 1.
The ﬁrst column represents the participant ID (P1 to P10).
In therest of the columns, the ﬁrst row speciﬁes the classiﬁcation method.
Each entry shows the averageclassiﬁcation accuracy on the test set across instances (ﬁrst number) and standard error of the mean(second number).
The last row shows the average performance of each method across participants.
Discussion
In this work, we investigated spatio-temporal aspects of the feedback-related brain activity.
Since
EEG is a time series recorded at multiple sites (electrodes on the scalp), we proposed methods tocombine aspects of space and time covariances.
We showed that time covariance can provide usefulfeatures for better classiﬁcation of feedback-related brain activity.
We proposed a ﬁlter-bank versionof the common temporal patterns (CTP) method.
Moreover, we proposed to combine spatial andtemporal features by applying CTP after CSP for improved classiﬁcation rates.
We also proposed a ﬁlter bank generalization of the MDM method proposed in [4], i.e. our FB-DRMS and another one to incorporate Riemannian distance to time covariances of the good and bad classesas extra features for the classiﬁer (FB-DRM-TS).
Our results show that FB-CSP-CTP performs best for majority of the participants.
On the otherhand, FB-DRM-T on average performs better than FB-CTP even though FB-DRM-TS does not beat
FB-CSP-CTP in performance.
This is an interesting observation and needs further investigation asit seems Riemannian geometry provides somewhat better time covariance features to classify thefeedback-related brain activity in our paradigm.
Future work includes investigating more sophisticatedclassiﬁers as introduced in [4], to replace the logistic regression on distance to the mean of each classin FB-DRM-TS to further improve our results.