Paper: Cognition-Supervised Learning: Contrasting EEG Signals and Visual Stimuli For Saliency Detection (ICLR-2024)

Reviewer comment (Novelty):
There has been prior works on image reconstruction from EEG, e.g. "NeuroGAN: image reconstruction from EEG signals via an attention-based GAN", "EEG2IMAGE: Image Reconstruction from EEG Brain Signals", "Photorealistic Reconstruction of Visual Texture From EEG Signals", "Visual Saliency and Image Reconstruction from EEG Signals via an Effective Geometric Deep Network-Based Generative Adversarial Network", and others. Given that the proposed paper seems to perform the opposite direction of these works, they could be discussed. Moreover, simple reverse versions of these can be developed to use as baselines.

Author response:
Regarding prior works on EEG-based image reconstruction, our introduction section discusses a series of earlier studies suffering from confounded EEG data due to specific experimental block designs. This includes the EEG-GAN approach [1], [2], Thoughtviz [3], Brain2image [4], EEG-ChannelNet[5], and subsequent research such as EEG2IMAGE [6], DM-RE2I [7], NeuroGAN [8], and others [9]. Many of these studies use the same datasets; for instance, NeuroGAN uses the same dataset from Thoughtviz, and [9] employs the EEG-ImageNet dataset from [5].

However, subsequent analyses [10], [11], [12] have identified a critical flaw in these approaches: the block design in data collection introduces temporal correlations between the stimulus class and the experiment's duration due to the same presentation order of stimulus class. Replication attempts have suggested that models were learning to condition on this temporal correlation rather than stimulus-related brain activity.

On the other hand, there are studies that utilize EEG signals from multiple trials to overcome the low signal-to-noise ratio inherent in EEG, like [13], which employs 24 repetitions of each stimulus and averages the visual evoked potentials for texture reconstruction. While these approaches have shown promising results, single-trial EEG-based high-resolution image reconstruction remains largely unexplored, to the best of our knowledge. 

We have appended this discussion to the related work section of our revised manuscript. 

[1] Palazzo, Simone, et al. "Generative adversarial networks conditioned by brain signals." Proceedings of the IEEE international conference on computer vision. 2017.

[2] Spampinato, Concetto, et al. "Deep learning human mind for automated visual classification." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.

[3] Tirupattur, Praveen, et al. "Thoughtviz: Visualizing human thoughts using generative adversarial network." Proceedings of the 26th ACM international conference on Multimedia. 2018.

[4] Kavasidis, Isaak, et al. "Brain2image: Converting brain signals into images." Proceedings of the 25th ACM international conference on Multimedia. 2017.

[5] Palazzo, Simone, et al. "Decoding brain representations by multimodal learning of neural activity and visual features." IEEE Transactions on Pattern Analysis and Machine Intelligence 43.11 (2020): 3833-3849.

[6] Singh, Prajwal, et al. "EEG2IMAGE: Image reconstruction from EEG brain signals." ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023.

[7] Zeng, Hong, et al. "DM-RE2I: A framework based on diffusion model for the reconstruction from EEG to image." Biomedical Signal Processing and Control 86 (2023): 105125.

[8] Mishra, Rahul, et al. "NeuroGAN: image reconstruction from EEG signals via an attention-based GAN." Neural Computing and Applications 35.12 (2023): 9181-9192.

[9] Khaleghi, Nastaran, et al. "Visual saliency and image reconstruction from EEG signals via an effective geometric deep network-based generative adversarial network." Electronics 11.21 (2022): 3637.

[10] Li, Ren, et al. "The perils and pitfalls of block design for EEG classification experiments." IEEE Transactions on Pattern Analysis and Machine Intelligence 43.1 (2020): 316-333.

[11] Ahmed, Hamad, et al. "Object classification from randomized EEG trials." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

[12] Ahmed, Hamad, et al. "Confounds in the data—Comments on “Decoding brain representations by multimodal learning of neural activity and visual features”." IEEE transactions on pattern analysis and machine intelligence 44.12 (2021): 9217-9220.

[13] Wakita, Suguru, Taiki Orima, and Isamu Motoyoshi. "Photorealistic reconstruction of visual texture from EEG signals." Frontiers in Computational Neuroscience 15 (2021): 754587.

Reviewer comment (Writing):
There's a small error in Fig 1, "Find-tuning" --> Fine-tuning.

Author response:
* Thank you for highlighting the typo in Figure 1. We have now corrected it in the revised manuscript.

Reviewer comment (Experiments):
While the RQs are very interesting, I wonder whether the outcome is completely expected. Given that we can map EEG to certain classes/actions/objects (e.g., affect classes, objects, directions, etc.), doesn't the outcome become expected? In other words, the method is mapping two high-dimensional data points onto a common lower dimensional feature space, where the locality of the image embeddings is used as the supervisory signal. Can the authors please (a) provide a discussion on this, (b) show the common embedding space (using tSNE, UMAP, etc) to extract some more insights about the reason behind why the method works. In general, I found the "analysis" part of the paper a bit weak. Further experiments to analyze the method, its components, and embedding space, are recommended.

Author response:
We think that our results are not entirely expected. Indeed, our embedding model maps high-dimensional EEG data to a lower-dimensional embedding, preserving relevant structures with the aid of paired stimuli data. However, this mapping is not trivial for two key reasons.

Firstly, the inherent noise in single-trial EEG signals makes even supervised binary classification a challenging task, often resulting in accuracies around 0.7. For instance, accuracies is 0.78 in [1] for classifying human faces against objects, around 0.7 in [2] for within-subject and around 0.4 for cross-subject,  0.708 in [3] and below 0.6 for text stimuli in [4]. This variability underscores that mapping EEG signals to a common feature space is not straightforward.

Secondly, while we use the locality of image embeddings to reduce the dimensions of EEG signals, these image embeddings alone are insufficient to discern semantic saliency. Section 4.1 and Table 1 demonstrate that clusters based on image embeddings alone do not align with the semantic saliency of our tasks. However, when clustering the EEG embeddings in the common feature space, there is a notable alignment with the target semantic feature. Furthermore, a PCA on these learned embeddings reveals that the most significant component correlates with the task's targeted semantic feature. We acknowledge the value of a deeper exploration into the learned EEG embeddings and their properties beyond mere correlation with task semantics. To this end, we have added new tSNE plots of both raw EEG signals and the learned embeddings in Appendix A.4 of our revised manuscript. 

[1] Bagchi, Subhranil, and Deepti R. Bathula. "EEG-ConvTransformer for single-trial EEG-based visual stimulus classification." Pattern Recognition 129 (2022): 108757.

[2] Lawhern, Vernon J., et al. "EEGNet: a compact convolutional neural network for EEG-based brain–computer interfaces." Journal of neural engineering 15.5 (2018): 056013.

[3] Ahmed, Hamad, et al. "Object classification from randomized EEG trials." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

[4] Eugster, Manuel JA, et al. "Predicting term-relevance from brain signals." Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval. 2014.