Spatio-temporal analysis of feedback-related brain activity in brain-computer interfaces
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Electroencephalography-based Brain-computer interfaces are systems that inferbrain signals recorded using electroencephalography (EEG) to provide a means ofcommunication for patients suffering from locked-in syndrome where commonneuromuscular pathways are not available.
One challenge in EEG-based BCIs isnon-stationarity of the EEG signal.
A major contributor to this is feedback-relatedbrain activity.
Since EEG consists of time series data recorded at multiple siteson the scalp, one can estimate covariance matrices for both time and space whichlie on the Riemannian manifold of symmetric positive deﬁnite matrices.
In thiswork, we investigate spatio-temporal aspects of the feedback-related brain activityby considering both space and time covariances in Euclidean space and on the
Riemannian manifold.
We propose two novel methods to incorporate both spatialand temporal features and show improved results compared to existing methods.
Introduction
Brain-computer interface (BCI) systems provide a means of communication for patients sufferingfrom locked-in syndrome through bypassing the normal neuromuscular pathways [8, 18].
Thesesystems record signals directly from the brain using methods such as electroencephalography (EEG)and infer the intention of the user from them.
Motor imagery (MI) BCIs are common BCI paradigmsin which the user imagines moving a part of her/his body.
Movement imagination results in anevent-related desynchronization (ERD) (decrease in power) in various frequency bands [16, 13].
Motor imagery of different body parts leads to different spatial desynchronization and this informationis used by the BCI to distinguish among the imagined movement classes.
In practice, for example, auser imagining moving her/his right or left hand can be mapped to a switch to turn on/off the light orto control the movement of a wheelchair, etc.
Non-stationarity of EEG signals is a barrier for real-world application of EEG-based BCI systems.
Feedback-related brain activity is one contributor to this non-stationarity [6, 17, 14].
In this work,we investigate the spatio-temporal aspects of the feedback-related brain activity in a motor imagery
BCI paradigm.
Covariance matrices play an important role in extracting features for classiﬁcation in
MI-BCI [5, 10].
By investigating both space and time covariances in the Euclidean space and on the
Riemannian manifold, our goal is to distinguish whether the BCI has made an error perceived by theuser in a motor imagery BCI paradigm.
Methods
Data were recorded from 10 participants participating in a motor imagery task to control a cursor ona monitor in front of them.
Data were recorded using a 64-channel BrainAmp system (Brain Products
GmbH).
Each trial began with the cursor at the center of the screen and the target at either left or right side.
The distance from the center to each side was ﬁxed to three cursor jumps.
The cursormoved at the speed of one step per second.
The trial ended when the cursor hit the target or the otherend of the screen.
Participants were lead to believe they were in control of the cursor, but in reality,the cursor moved based on a pre-determined sequence that was the same for all participants.
Thereason was to have enough steps towards or away from the target irrespective of the participant’smotor imagery performance.
Data were downsampled to 500 Hz and sections that were contaminatedwith excessive noise were removed.
Next, independent component analysis (ICA)
[11] was appliedto remove muscle and eye artifacts.
Pre-processing was done in MATLAB [12] and EEGLAB [7].
Feature extraction and classiﬁcation were implemented in Python.
We used the pyRiemann toolbox[2] to calculate Riemannian distances and means.
For detailed information on the pre-processing,please refer to [citation removed for anonymity].
We looked at different frequency bands, namely 1-3, 2-5, 4-7, 6-10, 7-12, 10-15, 12-19, 18-25,19-30, 25-35, 30-40 Hz to cover the low and high theta, mu, and beta frequency bands and to coverfor potential individual differences [15].
Data were epoched at 100 to 1000 ms after each cursormovement which we call a “step".
We trained a classiﬁer to distinguish whether the user is satisﬁedwith the last cursor movement or not, i.e., if the cursor had moved towards (good) or away (bad) fromthe target on that step.
This is called a good/bad classiﬁer.
The methods that are proposed in thiswork have space and time covariances of these steps at their core.
2.1 Space and time covariances
Let X i ∈ R c
× t be an EEG epoch (step) where c is the number of channels, t represents the numberof time samples and i ∈ { 1 , ..., N } where N is the number of steps available.
The sample space andtime covariances for each step are deﬁned as follows:
[Equation]
Since the number of channels in our case c = 64 is smaller than the number of time samples (inour case 450 time samples at a rate of 500 Hz), covariance matrices are not full rank and thus notpositive deﬁnite.
This is speciﬁcally required for the Riemannian methods.
On the other hand, sincethe highest cut-off frequency among the considered frequency bands is 40 Hz, we downsampled eachstep along the time dimension by 5 and regularized both space and time covariances as follows:
[Equation]
where I is the identity matrix with the same size as C , α is the regularization parameter and trace ( C )is the sum of values on the diagonal of matrix C .
In what follows, the regularization parameter waschosen arbitrarily but small, i.e., α = 0 . We didn’t test for other values.
It is a part of our futurework to use data-driven methods such as Ledoit-Wolf [9] to estimate the regularization parameter.
In this work, six methods are compared that differ in how they estimate the covariance from EEGdata (either time or space) and how they extract corresponding features and infer the class label fromthis information.
These methods are explained next.
2.2 Filterbank-Common Spatial Patterns (FB-CSP)
Filterbank common spatial patterns (FB-CSP) was proposed by by Ang et al.
[1] and serves as ourbenchmark. In this method, ﬁrst the average of the sample covariances for each of the good and badclasses in the Euclidean space were estimated as Σ g = 1 N (cid:80) i C
gS i and Σ b = 1 N (cid:80) i C bS i respectivelywhere N is the number of steps in the (balanced in number) good and bad classes.
C gS i and C bS irepresent the space covariance of the i th good and bad steps respectively.
CSP ﬁlters were estimatedfor each frequency band by simultaneous diagonalization of the two covariance matrices:
[Equation]
where Λ g and Λ b are diagonal matrices and W is selected such that Λ g + Λ b
= I [5].
This can beachieved by solving the generalized eigenvalue problem: Σ g w = λ Σ b w .
CSP ﬁlters w j , j = 1 , ..., Care the columns of W .
We considered the top 3 ﬁlters for each of the good and bad classes in eachfrequency band, i.e. the eigenvectors associated with the largest and smallest eigenvalues λ .
Featureswere selected as the log of the variance of the EEG signal passed through these ﬁlters.
A lineardiscriminant analysis (LDA) classiﬁer was trained on the selected features [10].
2.3 Filterbank-Common Temporal Patterns (FB-CTP)
Common temporal patterns (CTP) were proposed by Yu et al.
CTPs were found by solvingthe equations that were described earlier for CSP, except the sample mean of the good and bad timecovariances were considered instead of space covariances.
In their work, Yu et al.
combined thefeatures from common spatial and temporal ﬁlters.
However, here we only considered the commontemporal ﬁlters as a stand-alone method to compare with FB-CSP and another time covariancemethod on the Riemannian manifold that will be described next.
We consider a ﬁlter-bank version of CTP method similar to FB-CSP.
The top 3 ﬁlters for each of thegood and bad classes in each frequency band were selected.
Features were the log of the varianceof the EEG signal passed through these ﬁlters.
A linear discriminant analysis (LDA) classiﬁer wastrained on the selected features [10].
2.4 FB-CSP-CTP
This method combines the spatial and temporal features from CSP and CTP methods.
First CSPﬁlters were trained and the top 3 ﬁlters for good and bad classes were selected.
Then the EEG frameswere passed through the 6 selected CSP ﬁlters.
Next, CTP ﬁlters were trained and again the top 3ﬁlters for each class were selected.
The CSP ﬁltered EEG frames passed through the selected CTPﬁlters were used as features ( 6 × 6 ).
This procedure was applied to each frequency band separately tohave 11 × 6 × 6 features and a logistic regression was trained on them for classiﬁcation.
2.5 Filterbank-distance to spatial Riemannian means (FB-DRM-S)
Since covariance matrices are symmetric positive deﬁnite(SPD) matrices, they lie on a Riemannianmanifold [3].
Let P ( n ) =
{ P ∈ S ( n ) , P > 0 } be the set of all n × n SPD matrices.
The Riemanniandistance between two SPD matrices, P 1 and P 2 is deﬁned as follows:
[Equation]
where λ i are the eigenvalues of ( P − 1 1 P 2 ) which are real positive (non-zero) values as P 1 and P 2 areboth SPD.
Also, || .
|| F is the Frobenius norm of a matrix and Log ( P ) is the matrix logarithm of P .
Since SPD matrices are diagonalizable, let P = WDW − 1 .
Then Log ( P ) =
Wlog ( D ) W − 1 wherelog ( D ) i = log ( D i ) and log ( D ) i and D
i are the i th element on the diagonal of matrices log ( D ) and
D respectively.
The mean of the SPD matrices P 1 , P 2 , ..., P m on the Riemannian manifold is deﬁned as follows [3]:
[Equation]
Based on these mathematical tools on the Riemannian manifold, a ﬁlter bank generalization of theminimum distance to Riemannian mean (MDM) classiﬁer [4] is proposed.
First, the Riemannianmean of the good and bad space covariances on the training set were estimated.
Then, the featureswere selected as the distances to the Riemannian means of good and bad classes in each frequencyband (a total of 22 features: 11 frequency bands × 2 good and bad classes) and logistic regressionwas trained on the selected features.
2.6 Filterbank-distance to temporal Riemannian means (FB-DRM-T)
Here, we ﬁrst estimated the Riemannian mean of the good and bad time covariances on the trainingset.
Then, the features were selected as the distances to the Riemannian means of good and badclasses in each frequency band (a total of 22 features
11 frequency bands 2 good and bad classes)and logistic regression was trained on the selected features.
2.7 Filterbank-distance to spatial and temporal Riemannian means (FB-DRM-TS)
This method combines the two previous ones: we ﬁrst estimated the Riemannian mean of the goodand bad time and space covariances on the training set.
Then, the features were selected as the [Table] distances to the Riemannian means of good and bad time and space covariances in each frequencyband (a total of 44 features: 11 frequency bands × 2 good and bad classes × 2 time and spacecovariances).
A logistic regression classiﬁer was trained on the selected features.
Results and Discussion
Since good and bad classes were not balanced, we sub-sampled the larger class to have balancedclasses.
We randomly sub-sampled the larger class and made 5 instances of train-validation andtest sets - each having the same number of good and bad steps.
Results are presented in table 1.
The ﬁrst column represents the participant ID (P1 to P10).
In the rest of the columns, the ﬁrstrow speciﬁes the classiﬁcation method.
Each entry shows the average classiﬁcation accuracy onthe test set across instances (ﬁrst number) and standard error of the mean (second number).
Thelast row shows the average performance of each method across participants.
Our results show that
FB-CSP-CTP performs best for majority of the participants. CSP is a very effective well developedway to remove the high amount of spatial (between electrodes) correlation in EEG data and it appearsto be a good pre-processing step for future temporal analysis. On the other hand, FB-DRM-T onaverage performs better than FB-CTP even though FB-DRM-TS does not beat FB-CSP-CTP inperformance. This is an interesting observation and implies that further investigation may be fruitfulto best determine how to integrate Riemannian methods in an end-to-end classiﬁer.
Importantly inboth the Riemannian methods and the CSP/CTP methods combining spatial and temporal informationimproves performance.
Conclusion
In this work, we investigated spatio-temporal aspects of the feedback-related brain activity.
Since
EEG is a time series recorded at multiple sites (electrodes on the scalp), we proposed methods tocombine aspects of space and time covariances.
We showed that time covariance can provide usefulfeatures for better classiﬁcation of feedback-related brain activity.
We proposed a ﬁlter-bank versionof the common temporal patterns (CTP) method.
Moreover, we proposed to combine spatial andtemporal features by applying CTP after CSP for improved classiﬁcation rates.
We also proposed aﬁlter bank generalization of the MDM method that was originally proposed in [4], i.e. our FB-DRM-Sand another one to incorporate Riemannian distance to time covariances of the good and bad classes asextra features for the classiﬁer (FB-DRM-TS).
Future work includes investigating more sophisticatedclassiﬁers as introduced in [4], to replace the logistic regression on distance to the mean of each classin FB-DRM-TS to further improve our results.