Paper: MindGPT: Interpreting What You See with Non-invasive Brain Recordings (ICLR-2024)

Reviewer comment (Experiments):
The main weakness to me is the lack of comparison to existing work on the topic of fMRI-to-caption decoding. I am aware of at least two papers proposing a similar fMRI-to-pretrained latent alignment + generative text modeling approach which also report qualitative and quantitative results on an fMRI-to-caption task (Mai & Zhang, 2023; Ferrante et al., 2023). However, the presented results are not compared to any previous baseline. Given the similarity of the approaches, a clear comparison must be made to establish whether the proposed approach provides an improvement over existing approaches.

Author response:
>1) **About the comparison.** Many thanks for pointing out the potential competitors for comparison. We did not compare with the approaches because they were not officially published nor provided source-code, at the time we submitted. Following your suggestion, we have added **comparative experiments (as well as a discussion) with UniBrain** [1], a recent preprint on arXiv, which demonstrated the strength and advantage of our approach (please refer to the Related Work and Appendix). We only included one subject currently due to time limitation and the experiment with all the subjects is running and will be added to the results. Many thanks.

Reviewer comment (Experiments):
I believe a useful result to include in the analysis of Table 1 would be the performance of the pipeline if GPT-2 receives the ground-truth CLIP latents, instead of the fMRI-based predictions. This would provide an upper-bound for the proposed approach and help situate the reported results.

Author response:
>9) **About the analysis of Section 4.4.** We appreciate your valuable and constructive advice. By utilizing the actual CLIP [CLS] token instead of the decoder’s prediction for computing similarity scores, we noted a significant discrepancy from the predictions made by the MindGPT. These findings have been integrated into the revised version.

Reviewer comment (Experiments):
The results in this paper are from DIR dataset (1200 training and 50 test images) which is smaller compared to NSD dataset (10k images). Another advantage of NSD is that images are from MS-COCO dataset which contain more semantically complex scenes as compared to imagenet images in DIR which contain a single object centered. So, I would like to know author’s reasoning on why NSD dataset was not considered for this paper.

Author response:
> 1) **Regarding the NSD dataset.** The reason we did not use the NSD was that the MS-COCO dataset contains detailed annotations for images, but it does not provide specific classification labels like ImageNet. In other words, semantically complex scenes make it difficult for us to establish proper correspondences between virtual neural signals and candidate images. On the contrary, the images of ImageNet (typically including a single object) allow us to exploit virtual fMRI-image correspondences under the same class semantics.  We do agree that NSD dataset has advantages over DIR dataset in terms of the scale (10k images) and semantics richness. Therefore, we have added **an extra experiment with NSD dataset in comparison of a preprint work [3] (also used NSD)**. We only included one subject currently (please see Appendix) due to time limitation and the experiment with all the subjects is running and will be added to the results. Many thanks.

Reviewer comment (Novelty):
There are a few relevant references missing (Matsuo et al., Sakata et al.) which generate captions from brain activity. I am not sure if the code of these papers are available that’s why I am not asking to compare the results but I believe they should be at least discussed for readers to understand how this paper is different from previous fMRI→caption generation methods.

Author response:
>4) **The discussion about relevant references.** Previous works [1-4] mapped fMRI recordings to the embeddings of pre-trained neural networks like VGGNet by relatively simple linear regression models, and then feeds the predicted result into language models to generate word sequences. Unlike existing decoding paradigms, our MindGPT is designed to explore self-supervised neural semantics learners by using cross-attention mechanisms and data augmentation techniques. To the best of our knowledge, generating linguistic semantic information directly from a single brain image in an end-to-end fashion has not been adequately explored. The above discussion has been incorporated into the revised version. Many thanks.

Reviewer comment (Evaluation):
Self-attention maps of MindGPT encoder can be used to inform which brain region was more relevant in generating captions. This analysis could have provided more insight into captions of which category of images are generated by which brain regions.

Author response:
>5) **About the visual cues.** The visual cues demonstrated that the model autonomously acquired information about the salient regions within an image, a process devoid of any explicit labels or supervision. By utilizing the actual CLIP [CLS] token for computing similarity scores, we noted a significant discrepancy from the predictions made by the MindGPT. These findings have been integrated into the revised version. Our preliminary hypothesis is that the visual cues are related to the subjects' eye movements when viewing the images. Comparing them (i.e., attention maps) to eye movement data may provide a more convincing answer. We think it would be a valuable research topic in our future studies.

Reviewer comment (Evaluation):
It is not clear what groundtruth is used to compute language similarity metrics. If it is compared to Image captions generated by SMALLCAP then it is a major limitation of this approach. Accuracy of image generated captions will be upper bound by SMALLCAP. Collecting human captions for at least the test set and comparing both SMALLCAP and MindGPT would have been more informative.

Author response:
>3) **About the quantitative results.** The results in the paper are obtained via calculating language similarity metrics between the pseudo-caption (provided by SMALLCAP) and the generated text of MindGPT. We agree that this is a potential limitation of the approach. During the training phase, we used 200.7k candidate images for augmentation, making it challenging to collect human captions, but we are working in that direction.

Reviewer comment (Evaluation):
The paper has circumvented the main goal of visual reconstruction, we are completely relying on the generative model to build the images. In other words, images generated this way won't be closer than a certain amount despite having enormous details.

Author response:
>1)  **About visual reconstruction.** Our goal is to interpret the scene we see rather than reconstruct the image. The underlying hypothesis is that our brain is not a camera, and we can’t actually store all the pixels we see. Instead, we process visual information (at a semantic level) in our brain. Thus, we think that decoding semantic information and expressing it linguistically could be a more efficient way to understand the brain.

Reviewer comment (Experiments):
Comparison with other methods in the field has been mentioned scarcely after the introduction part, e.g., in the experiments it's done only among the different versions of their own algorithm.

Author response:
>2) **The discussion and comparison with other competitors.** Decoding semantic sentences instead of reconstructing the image is a novel perspective and to best of our knowledge, there hasn’t been any similar open-source study officially published at the time we submitted. To validate the effectiveness of our approach, we have added **comparative experiments (as well as a discussion)** with UniBrain [1], a recent preprint on arXiv, which demonstrated the strength and advantage of our approach (please refer to the Related Work and Appendix).