Paper: Decoding Natural Images from EEG for Object Recognition (ICLR-2024)

Reviewer comment (Evaluation):
There is no analysis of why NICE achieves better results than BraVL.

Author response:
A2. In our view, BraVL [R14] has good feature extraction but relatively basic Enc-Dec (MLP) and classifier (SVM). Besides, we think it’s a fMRI-oriented method due to more evaluation of two fMRI datasets. Thus, the performance on EEG is lower.

Reviewer comment (Experiments):
In the Encoder comparison experiment, the baseline compared with TSConv did not include methods used for EEG feature extraction in the past five years.

Author response:
A3. We used several most recognized CNN-based EEG baseline methods for comparison [R15] [R16], though not the latest but still very effective. We also compared with the new method based on CNN+Transformer [R17].

Reviewer comment (Evaluation):
The proposed architecture for EEG feature extraction, TSConv (described in Table 1 of the manuscript), is highly similar to that of the ShallowNet (as described in (Schirrmeister et al., 2017)) architecture, which the authors include as a baseline method. Specifically, the ShallowNet architecture originally consists of a temporal and a spatial convolutional layer, followed by average pooling and a fully-connected classification layer. Leaving aside the differences on the employed activation functions between TSConv and ShallowNet, one could say that the difference lies only on the order of placement for the average pooling layer. Given that the results reported in Table 3 are quite different for these two architectures (i.e. ShallowNet is the worst performing architecture and TSConv is the best performing architecture), what could be the interpretation?

Author response:
We have to clarify here that the architecture of the EEG encoder was totally not the main contribution of this work, as mentioned at the end of Sec. 1. Our contributions lie in proving the feasibility of EEG-based object recognition, due to EEG provides low-cost, portable attributes with high time resolution, compared to fMRI and MEG. For this reason, we propose the self-supervised framework with nice classification performance and give evidence from different aspects, including detailly resolving temporal, spatial, spectral, and semantic aspects. 

Weakness

W1. The architecture of EEG encoder.

A1. We mentioned that we adapted the EEG encoder we used as the widely use way in EEG analysis in the third paragraph of Sec. 3.1 and named it the basic temporal-spatial convolution (TSConv). TSConv was simply adapted based on the data. The comparison in Table 3 showed the effectiveness of the framework with different EEG feature extractors.

The encoder we used and the original ShallowNet encompassed several differences, mainly including the kernel size, activation function, and conv-pooling order. We compared these factors based on the final TSConv in the below table (**Appendix A.6**, Table 9). The three cases caused a significant effect on the top-1 accuracy separately compared to TSConv (p < 0.01, Bonferroni correction). We don’t want to prove which is better, but believe that any other well-designed EEG feature extractors can help us improve the performance in this framework.

| Condition | top-1 acc (std) | top-5 acc (std) |
| --- | --- | --- |
| original ShallowNet | 7.9 (4.3) | 24.2 (9.3) |
| kernel size as ShallowNet | 9.9 (2.4) | 33.5 (5.4) |
| pooling order | 10.9 (3.0) | 34.3 (7.2) |
| ELU->ReLU | 11.0 (2.5) | 33.3 (6.6) |
| TSConv | 13.8 (3.3) | 39.5 (6.5) |

Reviewer comment (Evaluation):
Regarding the two plug-and-play spatial modules, the authors state that they are "instrumental in preserving the spatial characteristics of EEG channels" (page 3, Section 3.1). Considering the self-attention (SA) module and looking at Equation (1), one can notice the spatial filtering nature of the proposed self-attention. However, the authors do not provide any insight on the spatial characteristics of the signals obtained through Equation (1), i.e. are the signals more or less correlated after this operation? The authors also mention that, following this transformation of the SA module, "Residual connection is employed by integrating the input and output of the SA module", but it is not clear how much this integrating procedure affects the spatial correlations between the EEG channels.

Author response:
W2. The spatial features captured by spatial modules, self-attention (SA), and graph attention (GA).

A2. We applied the two spatial modules for two reasons. (1) enhancing the performance by maintaining spatial features. (2) using the trained model to give implied evidence for the brain processing. In Table 2, we have proved that SA and GA improved the top-1 by 0.9% (p < 0.05) and 1.8% (p < 0.01), separately. Besides, we used grad-CAM in Fig. 4(C) and Appendix A.3 Fig. 5(A) to show evidence that SA and GA help the model tend to temporal and occipital regions, which are related to visual object recognition processing. We also supply the attention weight visualization here to show a similar tendency in **Appendix A.3** Fig. 5(B).

The residual connection helps to stabilize the training procedure habitually, especially for GA. We are sorry for neglecting to mention the use of residual connection for GA. In the table below (**Appendix A.6**, Table 8), we compare the effect of the residual connection. We can see that it significantly helps GA in top-1 accuracy (p<0.01) but has no significance for SA (p > 0.05). 

| Module | top-1 acc (std) | top-5 acc (std) |
| --- | --- | --- |
| SA w/ | res 14.7 (2.3) | 41.7 (4.6) |
| GA w/ | res 15.6 (2.9) | 42.8 (5.5) |
| SA w/o | res 13.8 (2.8) | 39.6 (5.7) |
| GA w/o | res 10.1 (3.2) | 29.9 (5.3) |

Reviewer comment (Novelty):
The graph attention mechanism proposed by the authors (page 5, Section 3.3.2) is similar to that of (Li et al., 2023). It would be interesting if the authors could explain the differences.

Author response:
W3. The design and use of GA.

A3. The STGATE is an interesting method that also uses graph attention networks for emotion recognition. Actually, both GA [R1] and SA [R8] in our application are very popular and usual ways to measure channel correlation, which introduces nothing new at the algorithm level. We would like to use these reliable ways to interpret the framework for learning plausible features.

Reviewer comment (Presentation):
In Figure 2, sub-figure (D) is not sufficiently explained, i.e. the exact method that was used for obtaining the time-frequency map is not mentioned, the quantity that is depicted is not stated and it is unclear to what does the range [-1, 0] correspond.

Author response:
Q3. Details of the Fig. 2(D).

A3. In Fig. 2(D), we average the channels belonging to different brain regions and use a short-time Fourier transform with Scipy to plot the time-frequency map. The x-axis denotes 1000 ms after the stimuli onset, the y-axis denotes the frequency, and the color bar unit is dB calculated by log. We would like to give a glance at the frequency distribution of the EEG signals, and then we perform quantitative analysis in Fig. 2(E).

Reviewer comment (Evaluation):
Some of the secondary claims do not appear to be adequately supported by the presented results. For instance, the claims on gamma-based decoding (see Q3 below) and on the use of occipital information in the graph attention layer (Q4).

Author response:
A3. It could be seen in Fig. 2(D) that the high-frequency responses in the temporal area were slightly more obvious than in the occipital and parietal areas. But the performance indeed achieved a chance level, where gamma was known for being related to object recognition. Two possible reasons were attached at the end of Sec. 4.5. (1) There was a longstanding controversy about whether scalp EEG can obtain high-frequency response due to artifacts like EMG [R4] [R5]. (2) Gamma had a small amplitude and easily to be modulated by other cognitive processing. Besides, the current dataset uses rapid sequences for stimuli, which makes the responses interfered by the pre- and post-stimulus. We plan to evaluate longer stimulus duration in future work to explore gamma responses. We have also mentioned the limitation in Sec. 5.

A4. We applied two popular ways to design “spatial filters” in the EEG encoder for two reasons. (1) enhancing the performance by maintaining spatial features. (2) using the trained models to give implied evidence for the brain processing. As in Fig. 4(C) and the supplements, we could see that the spatial modules tended to the occipital area (GA) and parts of the temporal area (SA).

We have visualized the attention weights of SA and GA with all training data of each subject, as shown in **Appendix A.3**, Fig. 5. Interestingly, more clear trends in occipital and temporal areas can be observed. However, we have to admit that it’s an intuitive way to use trained models to reflect the brain signal response. We still need other rigorous evaluations. 

Actually, both SA and GA used a residual connection, which was a mistake in the draft. We have made a comparison of the residual connection in the Table shown below (**Appendix A.6**, Table 8). We can see that residual connect significantly helps GA in top-1 accuracy (p<0.01) but not for SA (p > 0.05). The top-1 accuracy of GA is not significantly better than SA when both of them use the residual connection (p > 0.05). We will provide the source code later for reproduction.

| Module | top-1 acc (std) | top-5 acc (std) |
| --- | --- | --- |
| SA w/ res | 14.7 (2.3) | 41.7 (4.6) |
| GA w/ res | 15.6 (2.9) | 42.8 (5.5) |
| SA w/o res | 13.8 (2.8) | 39.6 (5.7) |
| GA w/o res | 10.1 (3.2) | 29.9 (5.3) |

There are other differences between these two methods, such as the way for attention calculation, the activation function, etc. We will attempt to go deeper into designing attention or graphs to help us perceive the signal dynamic in future works.

Reviewer comment (Novelty):
Based on previous image decoding work in fMRI and EEG, a logical continuation of the proposed contrastive learning pipeline is to then pass predicted latents to a generative image model (see Q5). I believe the presented image decoding performance results (along with the analyses of Sections 4.5-4.8) are interesting in themselves, however as a result the scope of the paper is narrower than existing studies on the topic. I believe the authors should discuss this in their manuscript.

Author response:
A5. We know that there are some works performing image reconstruction, even caption with brain signals, especially fMRI [R5] [R6]. Brain features are extracted and fed into well-designed generators like diffusion models. 

Although exquisite images are decoded, we are not easy to figure out what we really obtained from brain signals (low-level or high-level information?), even though some works separate structural and semantic features [R7] and prove them with different metrics. 

On the other hand, we think one of the most important gains is the concept/category-level semantic discrimination when visual decoding goes from low-level stimuli (contrast, color, orientation, etc.) to high-level stimuli (natural images). Therefore, we would like to begin with the recognition task to prove the feasibility of EEG signals. We have also mentioned it at the beginning of Sec. 5.

Reviewer comment (Reproducibility):
In Section 3.1, the authors write “These images are processed and averaged to obtain one template for each concept.” It is not clear to me how these averaged images are used in the experiments - was this in the semantic similarity analysis? Moreover, were the images averaged in pixel space or in latent space?

Author response:
Q1. The use of templates with averaged images.

A1. In the test phase, the incoming EEG trial was processed by the EEG encoder and then compared the cosine similarity with image templates. Therefore, we used several images for each image concept to construct the templates by averaging the image features at the latent level. 

It’s worth noting that we preferred to define it as a classification task, because we didn’t use stimulus images for the templates but other images from the corresponding concepts.

Reviewer comment (Presentation):
In Figure 2A: What is plotted on the topomaps? I assume it is the average EEG amplitude of each channel across the 100 ms window. If so, how was the data normalized (since the colorbar is from -1 to 1)? Also, in Figure 2B: What do the shaded areas represent? Is it variability across subjects?

Author response:
Q2. Explain the topomaps in Fig. 2(A) and shade in Fig. 2(B).

A2. Fig. 2(A) plotted EEG amplitude with a 100 ms window. We performed z-score standardization on the raw data and then normalized to [-1, 1]. In Fig. 2(B), the shade denoted the standard deviation of different subjects.

Reviewer comment (Evaluation):
In Section 4.5: “High-frequency responses could be observed from electrodes on the temporal cortex.” How is that measured? I find it hard to conclude from visual analysis of Figure 2D. Along with the chance-level performance of the gamma band (Figure 2E) I don’t see compelling evidence for the following claim: “These results align with established principles, coarsely indicating that the bottom-up feedforward is carried by the synchronization of theta and gamma bands.”

Author response:
A3. It could be seen in Fig. 2(D) that the high-frequency responses in the temporal area were slightly more obvious than in the occipital and parietal areas. But the performance indeed achieved a chance level, where gamma was known for being related to object recognition. Two possible reasons were attached at the end of Sec. 4.5. (1) There was a longstanding controversy about whether scalp EEG can obtain high-frequency response due to artifacts like EMG [R4] [R5]. (2) Gamma had a small amplitude and easily to be modulated by other cognitive processing. Besides, the current dataset uses rapid sequences for stimuli, which makes the responses interfered by the pre- and post-stimulus. We plan to evaluate longer stimulus duration in future work to explore gamma responses. We have also mentioned the limitation in Sec. 5.

Reviewer comment (Experiments):
Existing image decoding studies, e.g. in fMRI (Ozcelik & VanRullen, 2023) but also in EEG (Palazzo et al., 2020), have extended a similar approach up to image generation by feeding predicted image latents to a generative image model. Have the authors attempted this step?

Author response:
A5. We know that there are some works performing image reconstruction, even caption with brain signals, especially fMRI [R5] [R6]. Brain features are extracted and fed into well-designed generators like diffusion models. 

Although exquisite images are decoded, we are not easy to figure out what we really obtained from brain signals (low-level or high-level information?), even though some works separate structural and semantic features [R7] and prove them with different metrics. 

On the other hand, we think one of the most important gains is the concept/category-level semantic discrimination when visual decoding goes from low-level stimuli (contrast, color, orientation, etc.) to high-level stimuli (natural images). Therefore, we would like to begin with the recognition task to prove the feasibility of EEG signals. We have also mentioned it at the beginning of Sec. 5.

Reviewer comment (Writing):
The description of the analysis of Section A2 is a bit confusing. The comparison appears to be made between (1) using pretrained frozen image encoders and (2) using randomly initialized image encoders which are trained along the EEG encoders. The description doesn’t make it clear that the difference in computational resources between both approaches comes from the frozen/unfrozen dimension, not from the pretrained/non-pretrained dimension.

Author response:
A6. The increment of the computational cost was from training image encoders. Using pre-trained image encoders, we can process the images and obtain features before training the framework. Thus, the computational cost and training time of using pre-trained image encoders would be much lower than randomly initialed ones, which is meaningful for real-world use. We have updated the description for clarity.

R3: [Q6] You are right. The decrease in computational cost comes from the freezing of the image encoder. We have updated the content and footnote of Table 5 and the description to clarify that.
> In A.2: We should mention that the decrease in computational cost actually comes from the freezing of the pre-trained image encoder and getting image features beforehand.

> Table 5 footnote: pre-trained ResNet-50 and ViT-B/16. The image features were obtained before training.

Reviewer comment (Writing):
Algorithm 1 describes the CLIP loss (Radford et al., 2021). I think this should be introduced and cited as such, rather than through the lens of InfoNCE.

Author response:
A7. Clear introduction of contrastive loss.

Q7. We have updated the drafts for clear reference.

We enjoy these pertinent suggestions and look forward to further discussion!