  Convolutional neural networks (CNNs) have become a powerful technique to
decode EEG and have become the benchmark for motor imagery EEG
Brain-Computer-Interface (BCI) decoding. However, it is still challenging to
train CNNs on multiple subjects' EEG without decreasing individual performance.
This is known as the negative transfer problem, i.e. learning from dissimilar
distributions causes CNNs to misrepresent each of them instead of learning a
richer representation. As a result, CNNs cannot directly use multiple subjects'
EEG to enhance model performance directly. To address this problem, we extend
deep transfer learning techniques to the EEG multi-subject training case. We
propose a multi-branch deep transfer network, the Separate-Common-Separate
Network (SCSN) based on splitting the network's feature extractors for
individual subjects. We also explore the possibility of applying Maximum-mean
discrepancy (MMD) to the SCSN (SCSN-MMD) to better align distributions of
features from individual feature extractors. The proposed network is evaluated
on the BCI Competition IV 2a dataset (BCICIV2a dataset) and our online recorded
dataset. Results show that the proposed SCSN (81.8%, 53.2%) and SCSN-MMD
(81.8%, 54.8%) outperformed the benchmark CNN (73.4%, 48.8%) on both datasets
using multiple subjects. Our proposed networks show the potential to utilise
larger multi-subject datasets to train an EEG decoder without being influenced
by negative transfer.
