Paper: CLIP-MUSED: CLIP-Guided Multi-Subject Visual Neural Information Semantic Decoding (ICLR-2024)

Reviewer comment (Writing):
The use of letters in the article needs to be consistent. For example, the representation of X.

Author response:
The expressions $\mathcal{X}^{(n)}$ and $\mathbf{X}^{(n)}$ actually represent two different meanings. $\mathcal{X}^{(n)}$ represents the neural response space of the n-th subject, while $\mathbf{X}^{(n)}$  represents the samples in the dataset, which is a subset of the former. We also checked the other symbols.

Reviewer comment (Experiments):
The high-level feature in RSMs uses the average of the image and text features from the last layer of CLIP. What are the advantages of using the CLIP image encoder or text encoder alone compared to directly using it alone? Can you provide corresponding results for an explanation?

Author response:
Although CLIP maximizes the similarity between image and text modal embeddings through contrastive learning, there still exist differences in the last-layer embeddings of images and text. By considering the embeddings from both modalities and averaging them, the model can obtain more information.

We also conduct related experiments. As you can see, Table E6 shows the results only using the last-layer features extracted by CLIP's image encoder to guide the learning of higher-level representations, which is not better than CLIP-MUSED guided by the multi-modal features.
The experimental results using only the last-layer features from CLIP's text encoder for guidance have also been added to Table E6 (See CLIP-Text). These results further highlight the advantages of using the RSM of the multimodal features as the guidance.

Reviewer comment (Writing):
In Section 2.1 "X^{(n)}" is recommended to be consistent with the previous notation. The rest of the article should also be consistent.

Author response:
Please see our response to W1.

Reviewer comment (Writing):
The meaning of B in "B visual stimuli" in Section 2.2 should be explained.

Author response:
The symbol B in "B visual stimuli" represents the batch size in the mini-batch training paradigm. We have provided the corresponding details in Section 2 as indicated.

Reviewer comment (Presentation):
Is the underline in the AUC results of MS-SMODEL-ViT in Table 1 redundant? It should remain in the same format as other tables.

Author response:
We clarify that the underlines in the data of Table 1 and Table E6 are not redundant. As indicated in the captions of these two tables, the underlines signify that the improvement of our method compared to the reference methods is not significant in the respective metric.

Reviewer comment (Experiments):
The relatively weak decoding the authors perform, which is implied to be just category decoding. It is very strange to me that they use so much compute relative to traditional methods, and end up just doing category decoding. I believe the results would be strengthened by performing visual decoding in the form of full image reconstruction, at least for NSD (single image passive viewing task). If this is not possible, I think an alternative would be perform image retrieval (show top-5) on a test set, where the test set contains images not used for the training of any subject. This would likely require training the decoder on a single subject's stimuli only, then testing on other subjects.

Author response:
Thank you for your valuable suggestions. We are supplementing with retrieval experiments. The experimental results will be promptly released once the experiments are completed. Although reconstruction tasks also have significant research value, reconstruction involves deep generative models such as diffusion models, whereas our method is a discriminative model. Research on algorithms for reconstruction tasks is beyond the scope of this paper. We will consider conducting further research in the future.

Reviewer comment (Evaluation):
The fact that the subject-wise token is not computed via amortized inference (via an encoder conditioned on a few BOLD responses). But instead, they are fixed per subject. In my view this weakens the approach somewhat as you cannot take this approach and apply it to a new subject in a few-shot fashion, after the paper talks about how their method does not require subjects to view the same stimulus.

Author response:
The amortized inference you mentioned is indeed a good idea, but our method also has advantages compared to the amortized inference. 
1. Amortized inference typically relies on assumptions about the prior distribution. If the prior assumptions are inappropriate or inaccurate, it may affect the effectiveness of inference and the quality of results. Our method does not rely on prior distribution assumptions.
2. Our method eliminates the need for an encoding network, reducing the number of parameters that need to be learned and preventing overfitting. 

Applying to new subjects in a few-shot fashion still is a common challenge faced in the multi-subject neural decoding field. We have minimized the parameters that new subjects need to learn as much as possible.

Reviewer comment (Reproducibility):
In both the paper eq 10, the authors discuss using the F-norm (frobenius norm), but in practice they are using the squared frobenius norm in the code (line 42 of `losses.py`). This is fine, as the squared F-norm is everywhere differentiable, but I ask the authors clarify/fix this.

Author response:
In the latest manuscript PDF, we have revised the original statement to "the squared F-norm of the difference matrix normalized by the matrix size," and made the corresponding modification to Eq 10. Based on your suggestion, we have also modified the description of the orthogonal constraint, "To encourage the low-level and high-level token representations for each stimuli to represent as different features as possible, the proposed method applies an orthogonal constraint by ...".

Reviewer comment (Writing):
In the paper eq 15, the authors apply an orthogonal constraint such that matrix $z_\text{llv}$ and matrix $z_\text{hlv}$ are orthogonal to each other via the regularization loss. Same issue listed above applies. You should probably also clarify here that you want the low/high level representations for each stimuli to be orthogonal, otherwise it is not super clear.

Author response:
In the latest manuscript PDF, we have revised the original statement to "the squared F-norm of the difference matrix normalized by the matrix size," and made the corresponding modification to Eq 10. Based on your suggestion, we have also modified the description of the orthogonal constraint, "To encourage the low-level and high-level token representations for each stimuli to represent as different features as possible, the proposed method applies an orthogonal constraint by ...".

Reviewer comment (Writing):
Page 3. It seems that $X^{(n)} \in \mathbb{R}^{n_i \times d} $ should be $\mathcal{X}^{(n)} \in \mathbb{R}^{n_i \times d} $.

Author response:
The expressions $\mathcal{X}^{(n)}$ and $\mathbf{X}^{(n)}$ actually represent two different meanings. $\mathcal{X}^{(n)}$ represents the neural response space of the n-th subject, while $\mathbf{X}^{(n)}$  represents the samples in the dataset, which is a subset of the former.

Reviewer comment (Presentation):
For Figure 1 in the paragraph below you mention `two bird images t1 and t2` as well as `t1 and t2 are birds, while t3 (duck) and t4 (building blocks) are not`. Ducks should be birds, so I encourage the authors to clarify this, or use other examples in the figure and paper.

Author response:
**response**: Thank you for your suggestion. We have replaced the duck with cat.

Reviewer comment (Theory):
In the paper, it is discussed that `Linear transformation and MLP are unsuitable for high-dimensional voxel responses`. However no justification on the MLP aspect is given, and the citation (Yousefnezhad & Zhang) explicitly uses a kernelized MLP approach. Could you clarify?

Author response:
**response**: First, we discussed that linear transformation is unsuitable for high-dimensional voxel responses because Yousefnezhad et al. pointed out in their paper (Yousefnezhad & Zhang) that "original hyperalignment does not work in a very high dimensional space". Second, since MLP is composed of stacked linear layers and activation functions, the multi-layer kernel function in (Yousefnezhad & Zhang) will have a large number of parameters. Based on this, the MLP-based method is not a parameter-efficient method and is not suitable for high-dimensional data. We have provided these explanations at the corresponding locations in the updated manuscript PDF.

Reviewer comment (Writing):
In section 3.2, you never define what a backbone network is. Does it just take as input the fMRI BOLD responses and output a category? In that case, is the ViT a 3D model?

Author response:
**response**: The inputs for different backbones are preprocessed BOLD responses, and the outputs are class labels. The structure of the backbone network varies, and this variation also determines the differences in the format of the input data. In SS-MLP, the patchified responses of multiple ROIs are concatenated together, while in SS-ViT, the patchified responses are treated as different tokens. For HCP dataset, the model utilizes a 3D-CNN to convert 3D volumes to 2D patches, followed by a 2D ViT, as depicted in Figure 3(a)(b). The NSD dataset, on the other hand, directly employs 2D ViT, as shown in Figure 3(b). We have added these clarifications to Section 3.2 of the updated manuscript PDF and the section "Preprocessing before input" of Appendix A of the supplementary materials.

Reviewer comment (Experiments):
It is not clear what exactly you are decoding. You mention that HCP has a wordnet label, and NSD has the 80 categories (I assume the binary 0/1 mask on if an object from a category is present in the image).

Author response:
**response**: In the HCP dataset, we utilize WordNet labels as semantic labels for the stimulus images. Each word in the WordNet is treated as a separate category or class. In NSD, the stimulus image labels are 80 categories. In both HCP and NSD, "1" indicates the presence of a certain category, and "0" indicates its absence. A single stimulus image may have multiple labels associated with it.

Reviewer comment (Reproducibility):
Can you clarify if you use ROI based patching for NSD, and the CNN based patching approach for HCP? Is the CNN jointly trained with the transformer?

Author response:
**response**: We use ROI based patching for NSD, and the CNN based patching approach for HCP. Due to space limitation, we provide additional descriptions on how we process the NSD and HCP data to adapt the input form of Transformer in the section "Preprocessing before input" of Appendix A. In the HCP dataset, the CNN and transformer layers are trained jointly. We provide the clarification in Section 3.3 of the updated manuscript PDF.

Reviewer comment (Writing):
I found that the abstract is not as clearly written as is the introduction. From the abstract, it is unclear what the paper sets out to achieve nor how this is done. The introduction does a much better job at this. Perhaps the authors could try to formulate the problem addressed and the contributions more clearly (I appreciate that this is more difficult to do in the space available).

Author response:
**W1: Abstract modification**: Thank you to the reviewer for the modification suggestions on the abstract. We have made revisions to the abstract in the updated manuscript PDF to better highlight the problem addressed and our contributions.

Reviewer comment (Experiments):
The evaluation is rather limited, especially since the size of the datasets is quite small.

Author response:
**W2: The evaluation is rather limited, especially since the size of the datasets is quite small**:
As mentioned in the Introduction, fMRI data collection is costly, resulting in relatively small sizes for publicly available datasets. Among the available visual fMRI datasets, the number of stimuli presented to each subject typically ranges from several hundred to a few thousand and the number of subjects always is no more than five. For example, Vim-1 [1] has 1750 stimuli and two subjects; the dataset in [2] has 6000 stimuli but only three subjects. Compared to other available datasets, NSD and HCP datasets are the largest datasets because they include much more stimuli and subjects. Based on them, we trained and evaluated our method and the comparative methods in the paper. Notably, our evaluation is not limited to classification metrics alone; we also demonstrate the interpretability of the algorithm through visualizing attention maps (See Fig. 4 and Fig. F8). In conclusion, our evaluation is not limited but rather comprehensive and extensive.

Reviewer comment (Experiments):
Why do you extract low-level and high-level feature RSM separately?

Author response:
**Q1. Why do extract low-level and high-level feature RSM seperately?**

**Response**: 

The task of object classification requires encoding both low-level and high-level features of stimuli, and there are differences in how subjects encode these two types of features. Therefore, we introduce low-level and high-level tokens to account for these differences. To encourage these tokens to encode low-level and high-level features, we utilize the guidance of low-level and high-level features from CLIP, necessitating two feature RSMs. Furthermore, extracting low-level and high-level feature RSM seperately has two advantages:

1. The topological structures of stimuli in the low-level feature (e.g., shape, color) space and high-level feature (e.g., semantics) space exhibit differences. By encoding different topological structures in the two RSMs and incorporating the orthogonal constraint of tokens, it facilitates the tokens to learn better representations.
2. By visualizing attention maps, we can observe the correspondence between primary brain regions and low-level tokens, as well as between high-level brain regions and high-level tokens, thereby ensuring the interpretability of the algorithm.

Reviewer comment (Experiments):
The HCP dataset contains more than 158 subjects. Why did you not use all subjects? How were the 158 subjects selected from the HCP? Randomly? Why is this further reduced to 9? What is the bottleneck here?

Author response:
**Q2. The HCP dataset contains more than 158 subjects. Why did you not use all subjects? How were the 158 subjects selected from the HCP? Randomly? Why is this further reduced to 9? What is the bottleneck here?**

**Response**:
    
Firstly, we would like to clarify that our proposed algorithm itself is not a bottleneck. Our method can be easily scaled to more subjects because when training on one more subject, the model only needs to learn two additional N-dimensional vectors, where N equals 512. We selected the same ten subjects as the literature [3] that conducts experiments with randomly-selected ten subjects. However, the data format for one of the subjects has an issue. As a result, we proceeded with the experiment using the remaining nine subjects. 

While the algorithm itself is not a bottleneck, increasing the number of subjects and training data, on the other hand, may extend the training time and require additional resources. 
Given that the stimuli presented to different subjects in the HCP dataset are the same, the data diversity is more limited compared to that of NSD. When the number of subjects reaches a certain threshold, it is akin to training the model with more iterations, as the model repeatedly processes similar responses from different subjects under the same stimuli. Therefore, training with all 158 subjects simultaneously may not significantly improve the decoding performance for individual subjects but will demand more training resources.
As a result, we have currently conducted experiments on nine subjects. 

In order to illustrate the broader applicability of our method to a larger number of subjects, we are presently conducting multi-subject decoding experiments with a cohort of 30 subjects. We will expediently provide updates on the experimental outcomes.