Spatio-temporal analysis of feedback-related brain activity in brain-computer interfaces
Abstract
Electroencephalography-based brain-computer interfaces are systems that infer brain signals recorded using electroencephalography (EEG) to provide a means of communication for patients suffering from locked-in syndrome where common neuromuscular pathways are not available.
One challenge in EEG-based BCIs is non-stationarity of the EEG signal.
A major contributor to this is feedback-related brain activity.
Since EEG consists of time series data recorded at multiple sites on the scalp, one can estimate covariance matrices for both time and space which lie on the Riemannian manifold of symmetric positive deﬁnite matrices.
In this work, we investigate spatio-temporal aspects of the feedback-related brain activity by considering both space and time covariances in Euclidean space and on the
Riemannian manifold.
We propose two novel methods to incorporate both spatial and temporal features and show improved results compared to existing methods.
Introduction
Brain-computer interface (BCI) systems provide a means of communication for patients suffering from locked-in syndrome through bypassing the normal neuromuscular pathways [8, 18].
These systems record signals directly from the brain using methods such as electroencephalography (EEG) and infer the intention of the user from them.
Motor imagery (MI) BCIs are common BCI paradigms in which the user imagines moving a part of her/his body.
Movement imagination results in an event-related desynchronization (ERD) (decrease in power) in various frequency bands [16, 13].
Motor imagery of different body parts leads to different spatial desynchronization and this information is used by the BCI to distinguish among the imagined movement classes.
In practice, for example, a user imagining moving her/his right or left hand can be mapped to a switch to turn on/off the light or to control the movement of a wheelchair, etc.
Non-stationarity of EEG signals is a barrier for real-world application of EEG-based BCI systems.
Feedback-related brain activity is one contributor to this non-stationarity [6, 17, 14].
In this work, we investigate the spatio-temporal aspects of the feedback-related brain activity in a motor imagery
BCI paradigm.
Covariance matrices play an important role in extracting features for classiﬁcation in MI-BCI [5, 10].
By investigating both space and time covariances in the Euclidean space and on the Riemannian manifold, our goal is to distinguish whether the BCI has made an error perceived by the user in a motor imagery BCI paradigm.
This information can be used to improve the BCI performance (e.g. [14]).
Methods
Data were recorded from 10 participants participating in a motor imagery task to control a cursor on a monitor in front of them.
Data were recorded using a 64-channel BrainAmp system (Brain Products
GmbH).
Each trial began with the cursor at the center of the screen and the target at either left or right side.
The distance from the center to each side was ﬁxed to three cursor jumps.
The cursor moved at one step per second.
The trial ended when the cursor hit the target or the other end of the screen.
Participants were lead to believe they were in control of the cursor, but in reality, the cursor moved based on a pre-determined sequence that was the same for all participants.
This was to have enough steps towards or away from the target irrespective of the participant’s motor imagery performance.
Data were downsampled and sections that were contaminated with excessive noise were removed.
Next, independent component analysis (ICA)
[11] was applied to remove muscle and eye artifacts.
Pre-processing was done in MATLAB [12] and EEGLAB [7].
Feature extraction and classiﬁcation were implemented in Python.
We used the pyRiemann toolbox [2] to calculate Riemannian distances and means.
For details on pre-processing, please refer to [14].
We looked at different frequency bands, namely 1-3, 2-5, 4-7, 6-10, 7-12, 10-15, 12-19, 18-25, 19-30, 25-35, 30-40 Hz to cover the low and high theta, mu, and beta frequency bands and to cover for potential individual differences [15].
Data were downsampled to 100 Hz and epoched at 100 toms after each cursor movement which we call a “step".
We trained a classiﬁer to distinguish whether the user is satisﬁed with the last cursor movement or not, i.e., if the cursor had moved towards (good) or away (bad) from the target on that step.
This is called a good/bad classiﬁer.
The methods that are proposed in this work have space and time covariances of these steps at their core.
2.1 Space and time covariances
Let X i ∈ R c
× t be an EEG epoch (step) where c is the number of channels, t represents the number of time samples and i ∈ { 1 , ..., N } where N is the number of steps available.
The sample space and time covariances for each step are deﬁned as follows:
[Equation]
Since the number of channels in our case c = 64 is smaller than the number of time samples (i.e.,time samples at a rate of 100 Hz), covariance matrices are not full rank and thus not positive deﬁnite.
This is speciﬁcally required for the Riemannian methods.
Therefore, we regularized both space and time covariances as follows:
[Equation]
where I is the identity matrix with the same size as C , α is the regularization parameter and trace ( C ) is the sum of values on the diagonal of matrix C .
In what follows, the regularization parameter was chosen arbitrarily but small, i.e., α = 0 .
We did not test for other values.
It is a part of our future work to use data-driven methods such as Ledoit-Wolf [9] to estimate the regularization parameter.
In this work, six methods are compared that differ in how they estimate the covariance from EEG data (either time or space) and how they extract corresponding features and infer the class label from this information.
These methods are explained next.
2.2 Filterbank-Common Spatial Patterns (FB-CSP)
Filterbank common spatial patterns (FB-CSP) was proposed by by Ang et al.
[1] and serves as our benchmark.
In this method, ﬁrst the average of the sample covariances for each of the good and bad classes in the Euclidean space were estimated as Σ g = 1 N (cid:80) i C
gs i and Σ b = 1 N (cid:80) i C bs i respectively where N is the number of steps in the (balanced in number) good and bad classes.
C gs i and C bs i represent the space covariance of the i th good and bad steps respectively.
CSP ﬁlters were estimated for each frequency band by simultaneous diagonalization of the two covariance matrices:
[Equation]
where Λ g and Λ b are diagonal matrices and W is selected such that Λ g + Λ b
= I [5].
This can be achieved by solving the generalized eigenvalue problem: Σ g w = λ Σ b w .
CSP ﬁlters w j , j = 1 , ..., C are the columns of W .
We considered the top 3 ﬁlters for each of the good and bad classes in each frequency band, i.e. the eigenvectors associated with the largest and smallest eigenvalues λ .
Features were selected as the log of the variance of the EEG signal passed through these ﬁlters.
A linear discriminant analysis (LDA) classiﬁer was trained on the selected features [10].
2.3 Filterbank-Common Temporal Patterns (FB-CTP)
Common temporal patterns (CTP) were proposed by Yu et al.
CTPs were found by solving the equations that were described earlier for CSP, except the sample mean of the good and bad time covariances were considered instead of space covariances.
In their work, Yu et al.
combined the features from common spatial and temporal ﬁlters.
However, here we only considered the common temporal ﬁlters as a stand-alone method to compare with FB-CSP and another time covariance method on the Riemannian manifold that will be described next.
We consider a ﬁlter-bank version of CTP method similar to FB-CSP.
The top 3 ﬁlters for each of the good and bad classes in each frequency band were selected.
Features were the log of the variance of the EEG signal passed through these ﬁlters.
A linear discriminant analysis (LDA) classiﬁer was trained on the selected features [10].
2.4 FB-CSP-CTP
This method combines the spatial and temporal features from CSP and CTP methods.
First CSP ﬁlters were trained and the top 3 ﬁlters for good and bad classes were selected.
Then the EEG frames were passed through the 6 selected CSP ﬁlters.
Next, CTP ﬁlters were trained and again the topﬁlters for each class were selected.
The CSP ﬁltered EEG frames passed through the selected CTP ﬁlters were used as features ( 6 × 6 ).
This procedure was applied to each frequency band separately to have 11 × 6 × 6 features and a logistic regression was trained on them for classiﬁcation.
2.5 Filterbank-distance to spatial Riemannian means (
FB- DRM S)
Since covariance matrices are symmetric positive deﬁnite (SPD) matrices, they lie on a Riemannian manifold [3].
Let P ( n ) =
{ P ∈ S ( n ) , P > 0 } be the set of all n × n SPD matrices.
The Riemannian distance between two SPD matrices, P 1 and P 2 is deﬁned as follows:
[Equation]
where λ i are the eigenvalues of ( P − 1 1 P 2 ) which are real positive (non-zero) values as P 1 and P 2 are both SPD.
Also, || .
|| F is the Frobenius norm of a matrix and Log ( P ) is the matrix logarithm of P .
Since SPD matrices are diagonalizable, let P = WDW − 1 .
Then Log ( P ) =
Wlog ( D ) W − 1 where log ( D ) i = log ( D i ) and log ( D ) i and D
i are the i th element on the diagonal of matrices log ( D ) and
D respectively.
The mean of the SPD matrices P 1 , P 2 , ..., P m on the Riemannian manifold is deﬁned as follows [3]:
[Equation]
Based on these mathematical tools on the Riemannian manifold, a ﬁlter bank generalization of the minimum distance to Riemannian mean (MDM) classiﬁer [4] is proposed.
First, the Riemannian mean of the good and bad space covariances on the training set were estimated.
Then, the features were selected as the distances to the Riemannian means of good and bad classes in each frequency band (a total of 22 features: 11 frequency bands × 2 good and bad classes) and logistic regression was trained on the selected features.
2.6 Filterbank-distance to temporal Riemannian means (FB-DRM -T)
Here, we ﬁrst estimated the Riemannian mean of the good and bad time covariances on the training set.
Then, the features were selected as the distances to the Riemannian means of good and bad classes in each frequency band (a total of 22 features: 11 frequency bands × 2 good and bad classes) and logistic regression was trained on the selected features.
[Table] 2.7 Filterbank-distance
This method combines the two previous ones: we ﬁrst estimated the Riemannian mean of the good and bad time and space covariances on the training set.
Then, the features were selected as the distances to the Riemannian means of good and bad time and space covariances in each frequency band (a total of 44 features: 11 frequency bands × 2 good and bad classes × 2 time and space covariances).
A logistic regression classiﬁer was trained on the selected features.
Results and Discussion
Since good and bad classes were not balanced, we sub-sampled the larger class to have balanced classes.
We randomly sub-sampled the larger class and made 10 instances of train-validation and test sets - each having the same number of good and bad steps.
Results are presented in table 1.
The ﬁrst column represents the participant ID (P1 to P10).
In the rest of the columns, the ﬁrst row speciﬁes the classiﬁcation method.
Each entry shows the average classiﬁcation accuracy on the test set across instances (ﬁrst number) and standard error of the mean (second number).
The last row shows the average performance of each method across participants.
Our results show that
FB-CSP-CTP performs best for majority of the participants.
CSP is a very effective well developed way to remove the high amount of spatial (between electrodes) correlation in EEG data and it appears to be a good pre-processing step for future temporal analysis.
On the other hand, FB-DRM-T on average performs better than FB-CTP.
This is an interesting observation and implies that further investigation may be fruitful to best determine how to integrate Riemannian methods in an end-to-end classiﬁer.
Importantly in both the Riemannian methods and the CSP/CTP methods combining spatial and temporal information improves performance.
Conclusion
In this work, we investigated spatio-temporal aspects of the feedback-related brain activity.
Since
EEG is a time series recorded at multiple sites (electrodes on the scalp), we proposed methods to combine aspects of space and time covariances.
We showed that time covariance can provide useful features for better classiﬁcation of feedback-related brain activity.
We proposed a ﬁlter-bank version of the common temporal patterns (CTP) method.
Moreover, we proposed to combine spatial and temporal features by applying CTP after CSP for improved classiﬁcation rates.
We also proposed a ﬁlter bank generalization of the MDM method that was originally proposed in [4], i.e. our FB-DRM-S and another one to incorporate Riemannian distance to time covariances of the good and bad classes as extra features for the classiﬁer (FB-DRM-TS).
Future work includes investigating more sophisticated classiﬁers as introduced in [4], to replace the logistic regression on distance to the mean of each class in FB-DRM-TS to further improve our results. Acknowledgments
This work was supported by NSF IIS 1219200, IIS 1817226, SMA 1041755, and IIS 1528214, FISP
G2171, G3155, NIH 5T32MH020002-18 and Mary Anne Fox dissertation year fellowship.