Paper: Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCI (ICLR-2024)

Reviewer comment (Experiments):
While the authors acknowledge the challenge of varying configurations in EEG data collection, particularly concerning electrode variations across different datasets, the manuscript does not provide a comprehensive solution to this challenge. Specifically, the Temporal & Spatial embedding section offers an ambiguous explanation regarding spatial embedding (SE). The method appears to encode channels based merely on their sequential order, which may not accurately represent the channel's functional significance or location in the brain. Given the heterogeneity in electrode configurations and positions across various datasets, it's imperative for the authors to elucidate how their approach effectively manages this inconsistency, and use qualitative results to demonstrate the model has learned different electrode configurations.

Author response:
* Thanks for your valuable suggestion. First of all, our spatial embeddings are initialized according to the international 10-20 system (https://en.wikipedia.org/wiki/10%E2%80%9320_system_(EEG)) which was commonly followed by most existing EEG datasets. Though different datasets may have different configurations, they all utilize a subset of electrodes of the international 10-20 system, providing the possibility to solve this problem. Specifically, our solution is adding the corresponding spatial embedding according to the international 10-20 system. For example, both dataset A and dataset B have the electrode FP1, so the FP1 electrodes in the two datasets are added by the same spatial embedding. By this means, the spatial embeddings incorporate the spatial information into the model training, which is unremovable during pre-training. If we try to discard the spatial embeddings, the pre-training loss cannot converge because it does not have the ability to identify which masked electrodes to reconstruct. To demonstrate the effectiveness of the spatial embeddings, we provide the ablation study where the spatial embeddings are directly discarded during fine-tuning on downstream datasets as follows:

| TUAB | Balanced Accuracy | AUC-PR | AUROC |
| -------------------- | :---------------------: | :--------------------: | :------------------: |
| LaBraM | 0.8140 $\pm$ 0.0019 | 0.8965 $\pm$ 0.0016 | 0.9022 $\pm$ 0.0009 |
| w/o spatial embedding | 0.8004 $\pm$ 0.0037 | 0.8922 $\pm$ 0.0023 | 0.8888 $\pm$ 0.0018 |

|TUEV|Balanced Accuracy|Cohen's Kappa|Weighted F1|
|---|---|---|---|
|LaBraM|0.6409 $\pm$ 0.0065|0.6637 $\pm$ 0.0093|0.8312 $\pm$ 0.0052|
|w/o spatial embedding|0.5949 $\pm$ 0.0423|0.6069 $\pm$ 0.0248|0.8040 $\pm$ 0.0111|

The results clearly demonstrate the importance of spatial embeddings in capturing spatial information. We have updated this experiment to the revised paper in Appendix L.

Reviewer comment (Novelty):
The paper draws parallels to another LEM, BIOT, developed by Yang et al., and even follows some experimental settings from the same. Given the apparent similarities, it remains unclear as to what drives the performance improvements of the proposed model — is it solely attributed to the increased model size, enhanced training data volume, or specific architectural designs? A more in-depth comparative discussion and analysis between the two models would be beneficial to ascertain the genuine contributions and innovations of the current work.

Author response:
* Thanks for your valuable feedback. Our conclusion to your concern is that the well-designed large-scale self-supervised pre-training contributes most to our superior performance compared to BIOT. In Appendix J, we have conducted ablation to see whether the pre-training really works. It can be seen that without pre-training, the performance decreases steeply on the two datasets especially on TUEV.

Reviewer comment (Reproducibility):
Will the pre-trained checkpoints be released for open-source development as well?

Author response:
* Yes, we will release the pre-trained models as well as the codes.

Reviewer comment (Evaluation):
I don’t think the results are clear enough to deduce what is claimed in the analysis of Figure 5, i.e. that the performance saturates for the base model at 500 h and for the large model at 2000 h. For instance, the Large models seem to continue learning over 2000 h on TUEV. Similarly, the Base model might not be saturated yet; the performance curve seems pretty noisy. Maybe repeating this analysis with a log-scale would be clearer (1h, 10h, 100h, 1000h, 2500h).

Author response:
* Thanks for your valuable suggestion. We have updated Figure 5 with additional experiments on 1h, 10h, and 100h. Please refer to Figure 5 in the revised paper. As can be seen from the trend in Figure 5, the performance of the Base model consistently increases with the amount of data at 500 hours or less. Therefore, 500 hours of EEG data is enough for pre-training the Base model. For the Large model, the performance tends to saturation and achieve its optimum as the data size reaches 2,000 hours.

Reviewer comment (Writing):
The use of the term “BCI” (e.g. in the title) is confusing as this typically refers to a subset of the tasks/datasets considered in this work. For instance, the term BCI is usually used to describe cases where there is an interface between brain activity and a computer that bypasses normal communication pathways. Under this definition, tasks such as pathology detection (TUAB) or event detection (TUEV) are not BCI tasks. I would recommend adapting the language of the manuscript to make this clearer.

Author response:
* Thanks for your valuable opinion.  A Brain-Computer Interface (BCI) can be defined as "a system that measures central nervous system (CNS) activity and converts it into artificial output that replaces, restores, enhances, supplements, or improves natural CNS output, thereby altering the ongoing interactions between the CNS and its external or internal environment" [1]. Based on this definition, we propose that tasks like TUAB and TUEV contribute to the first part of the BCI system's measurement process. This involves measuring neural activity and utilizing it as input for subsequent processes, including control, intervention therapy, and neural circuit remodeling. Therefore, we contend that tasks such as TUAB and TUEV fall within the scope of BCI tasks.

Reviewer comment (Experiments):
Some of the hyperparameter choices for the windowing and tokenization steps are not clear to me. First, what were the window strides ($s$ in first paragraph of Section 2.1) used for the different datasets? Tables 3 and 4 report a “Data stride” value but I’m not sure whether that’s the same thing. Second, what is the impact of the selected patch size and patch stride on performance, i.e. are these choices important? Related to the point about how symmetric masking might be providing regularization that is useful for larger models, would a smaller window and/or stride help create more pretraining examples?

Author response:
* We apologize for not clarifying some hyperparameters. For the window stride, it is exactly the same thing as the data stride in Table 3 and Table 4. As for the setting of patch size and patch stride, we set them to 1 second because 1 second is the smallest unit for many EEG tasks. The common time length for most EEG tasks is between 1 second and 10 seconds. For example, the time length of a sample for datasets we use in this paper is 10 seconds for TUAB, 5 seconds for TUEV, 1 second for SEED-V, and 2 seconds for MoBI. On the other hand, this setting is consistent with some baselines such as BIOT where the authors conduct experiments with different settings of patch size and overlapping length, and the results show that 1 second is a relatively good choice for EEG segments. For the setting of data stride, we set it to 4 seconds for pre-training based on the following considerations: (1) covering all training data. As the maximum sequence length is 256, the sample for 64-channel EEG signals is 4 seconds. All data can be visited if we set the data stride to 4 seconds (no overlapping). (2) boosting the training speed (reducing the number of samples). If we set the data stride to 1 second or less, the number of samples could be so large that it will take several times as long to complete the training.

Reviewer comment (Reproducibility):
It is not clear to me whether the weights of the temporal encoder from the tokenization step are reused in the pretraining step, or if only the architecture is the same. From the dimensions of the large and huge models I assume the weights could not be reused as the sizes are not the same.

Author response:
* The weights of the temporal encoder from the tokenization step are not reused in the pretraining step. It is just as what you say that only the architecture is the same.

Reviewer comment (Experiments):
My understanding is that new spatial embeddings must be learned from each montage that is seen (i.e. the $i$ in Equation 2). How many different spatial embeddings were learned during pretraining? Also, what spatial embeddings were used in the different ablations of Table 10 if the model didn’t have a chance to learn a spatial embedding for TUAB and TUEV (or were examples from this EEG montage already seen in the pretraining data)?

Author response:
* First of all, our spatial embeddings are initialized according to the international 10-20 system (https://en.wikipedia.org/wiki/10%E2%80%9320_system_(EEG)) which was commonly followed by most existing EEG datasets. Though different datasets may have different configurations, they all utilize a subset of electrodes of the international 10-20 system. Another reason why we collect a variety of datasets with different configurations is that we hope LaBraM can see as many different electrodes as possible in the pre-training process. Consequently, examples from the EEG montage of TUAB and TUEV are already seen in the pretraining data. However, if there are unseen electrodes in the downstream dataset such as MoBI, we just randomly initialize new spatial embeddings for the unseen electrodes. The new spatial embeddings are trained along with the existing spatial embeddings throughout the fine-tuning process.

Reviewer comment (Theory):
Section 2.3: What is the self-distillation loss term? It doesn’t seem to be in the final pretraining objective of Equation 12.

Author response:
* We apologize for this mistake. This is a clerical error and we have removed it in the revised paper.

Reviewer comment (Experiments):
Is there an overlap between TUAB/TUEV and the different training sets taken from the Temple University EEG Corpus (TUAR, TUEP, etc.)? If so, could this explain why including TUAB and TUEV in the pretraining set didn’t change the results much (Section 3.5)? I believe this shouldn’t impact the comparison with BIOT as the reported results from the BIOT paper appear to be from the model also pretrained on TUAB and TUEV.

Author response:
* There is no overlap between TUAB/TUEV and the different training sets taken from the Temple University EEG Corpus (https://isip.piconepress.com/projects/tuh_eeg/html/downloads.shtml) and they are disjoint from TUAR, TUEP, TUSZ, and TUSL. Since we pre-train LaBraM with tremendous EEG data, the minor differences when including TUAB/TUEV demonstrate that our model has the capability to learn universal EEG representations.

Reviewer comment (Evaluation):
Related to the previous question, looking at Figure 4 it looks like adding TUEV to the pretraining dataset actually negatively impacts downstream performance. Is this effect significant and if so, what could be driving this decrease in performance?

Author response:
* The large-scale pre-training has brought about the strong capability for learning generic representations, which might be saturated even if we further utilize TUEV. Thus, the slight decrease on TUEV might also be due to the randomness of the pre-training process. Considering ablation on both TUAB and TUEV, we observe no obvious differences between pre-training with and without downstream tasks.

Reviewer comment (Experiments):
How are the pretraining examples sampled? Since this is not described in the manuscript I would assume sequences were sampled uniformly across the entire training corpus, however I was wondering if the authors have considered more balanced sampling schemes, e.g. taking datasets, recordings and/or experimental paradigm-related information into account when sampling.

Author response:
* For simplicity, the pre-training samples are uniformly sampled within a set of datasets that are of the same EEG montage. The order of the set of datasets is also random. This might have the potential to improve if we can design a more advanced sampling strategy and is indeed a promising direction for future work.

Reviewer comment (Reproducibility):
Where did the baseline results for Table 6 come from? They don’t seem to be in the BIOT paper.

Author response:
* These results are replicated by ourselves using the codes provided by the authors of BIOT.

Reviewer comment (Writing):
A few typos:  
- Figure 2: “Fuorier spectrum”  
- Appendix J: “PRE-TRAINGING”  
- Equation 14: Missing closing parenthesis

Author response:
* Thanks a lot for your careful check and we have corrected the typos in the revised paper.

Reviewer comment (Novelty):
Some papers have not been mentioned that work on heterogeneous datasets:  
* [Learning Topology-Agnostic EEG Representations with Geometry-Aware Modeling](https://openreview.net/forum?id=hiOUySN0ub)  
* [EEG Decoding for Datasets with Heterogenous Electrode Configurations using Transfer Learning Graph Neural Networks](https://arxiv.org/abs/2306.13109v1)  
* [Generalizable Movement Intention Recognition with Multiple Heterogeneous EEG Datasets](https://ieeexplore.ieee.org/document/10160462)

Author response:
* Thank you for your insightful suggestions. We appreciate your encouragement to broaden our study's scope. In response, we have integrated these recommendations into our revised paper, specifically in the related work section.

Reviewer comment (Evaluation):
The choice to use mean squared error on phase values is strange to me. Due to their cyclical nature, very nearby phases, e.g., -pi+eps, pi-eps would get a large squared error that would also depend on whether one uses phases from 0 to 2pi or -pi to pi etc. It would make more sense to either always only use the minimum distance (e.g., putting the phases on a unit circle and using the minimum distance on the circle), or maybe even regress Fourier coefficients instead of amplitude/phase. I even wonder what would happen if one just does not put any loss on the phase prediction at all, only on the amplitudes — that would be interesting to check as well.

Author response:
* Thank you for your insightful comments on the selection of mean squared error for phase values. We acknowledge your concerns about the cyclical nature of phases and the potential biases associated with different ranges. We are currently making efforts to conduct additional experiments based on this suggestion. However, due to limited computational resources and time constraints, we may not be able to complete these experiments in time. Rerunning the neural tokenizer training and LaBraM pre-training (especially with a total of 2,500 hours of data) can take several weeks. If the experiments are completed, we will update the results as promptly as possible. We believe that our comprehensive analysis will help researchers better understand the merits of each method and their applicability in different scenarios. We sincerely appreciate your constructive feedback, which has led to a more in-depth exploration of our research. Your insights have enriched our paper and improved its overall quality.

Reviewer comment (Presentation):
Why bold the lowest standard deviation in Table 1 and 2? Better to remove that, as it is rather confusing.

Author response:
* Thank you for your suggestion, and we have made the necessary changes accordingly.

Reviewer comment (Presentation):
Font could be a bit bigger in Figure 1 and also parts of Figure 2 (e.g., channel names on bottom). The amplitude-phase plot on the top right of Figure 2 is confusing to me. The presentation of "symmetric" in Figure 2 is not symmetric.

Author response:
* Thank you for your valuable feedback. We have taken your suggestions into account and made the necessary adjustments to improve the clarity and readability of the figures.

Reviewer comment (Experiments):
I assume Table 1/2 only compares to other self-supervised models? That should be written a bit more explicitly, otherwise there are other papers worth citing:  
https://www.springerprofessional.de/en/chrononet-a-deep-recurrent-neural-network-for-abnormal-eeg-ident/16824220   
https://www.sciencedirect.com/science/article/pii/S1053811920305073   
It might, in any case, be good to also show a purely supervised baseline.

Author response:
* Thanks for your valuable suggestion. We appreciate your attention to the details of the baselines presented in Tables 1 and 2. To provide a clearer understanding of these baselines, we present the following detailed descriptions: 

1) SPaRCNet: This is a 1D-CNN-based model equipped with dense residual connections. 

2) ContraWR: It converts biosignals into multi-channel spectrograms and then employs a 2D-CNN-based ResNet framework accompanied by contrastive learning.   

3) CNN-Transformer: This hybrid model combines the strengths of CNNs and Transformers for enhanced performance.   

4) FFCL: The Fine-tuning Framework for Classification (FFCL) integrates embeddings from both CNN and LSTM encoders for effective feature fusion.   

5) ST-Transformer: This multi-level EEG transformer concurrently learns spatial and temporal features for improved performance.  

6) BIOT: A framework based on the Transformer architecture, it takes spectral energy vectors extracted by FFT on the original signals as inputs. It can be pre-trained using either supervised or self-supervised learning techniques.

It is worth noting that these baselines represent state-of-the-art supervised (SPaRCNet, CNN-Transformer, FFCL, and ST-Transformer) and self-supervised (ContraWR and BIOT) methods in the recent literature. We have sincerely cited the mentioned paper in our revised manuscript, specifically in the introduction section. We hope this provides clarification and addresses your concerns.

Reviewer comment (Experiments):
Are TUAB and TUEV disjoint recording-wise from TUAR, TUEP, TUSZ, and TUSL?

Author response:
* Yes, the TUAB and TUEV are indeed disjoint from TUAR, TUEP, TUSZ, and TUSL. Detailed descriptions of these datasets can be found in the provided URL link (https://isip.piconepress.com/projects/tuh_eeg/html/downloads.shtml).