Paper: Aligning brain functions boosts the decoding of videos in novel subjects (ICLR-2024)

Reviewer comment (Experiments):
The loss function has three parameters. How to choose the parmeters is difficult.

Author response:
Reviewer jWwB is right that the chosen alignment method (FUGW) has 3 hyper-parameters, namely alpha, rho and epsilon. We use FUGW because, unlike alternatives, it comes with a simple python API comparable to that of sklearn and thus facilitates reproducibility. Previous work has already explored the impact of these three hyper-parameters, and showed that alignment is robust to changing values for alpha and rho, although epsilon needed to be carefully chosen. Our goal is a proof-of-existence, i.e. that it is possible to yield significant decoding gains with a standard functional alignment preprocessing.

Reviewer comment (Experiments):
The proposed method should be compared with the state-of-the-art methods.

Author response:
Since we did not introduce this alignment method, we did not feel compelled to compare it with a lot of other methods. However, note that our baseline (i.e. fsaverage) is the most commonly used alignment method. Moreover, we agree with you that our work could constitute an interesting downstream task, which a future benchmark could use to compare between functional alignment methods.

Reviewer comment (Experiments):
The proposed method don't compared with the state-of-the-art methods.

Author response:
Since we did not introduce this alignment method, we did not feel compelled to compare it with a lot of other methods. However, note that our baseline (i.e. fsaverage) is the most commonly used alignment method. Moreover, we agree with you that our work could constitute an interesting downstream task, which a future benchmark could use to compare between functional alignment methods.

Reviewer comment (Novelty):
The novelty needs to be further elaborated.

Author response:
We agree with the reviewer that we should clarify the novel contributions of our work. We propose to amend the introduction  with the following paragraph:  
“Overall, decoding and functional alignment studies have focused on largely disjoint objectives: decoding semantics is typically done at the individual level (e.g. Tang et al. 2023, Scotti et al. 2023, Ozcelik et al. 2023), whereas functional alignment across subjects is evaluated on simple features, such as retinotopy (Feilong et al. 2022), object categorization (Bazeille et al. 2021), and binary contrasts (Thual et al. 2022). With the rise of naturalistic datasets (NSD, Wen2017, Lebel2023, Things), it is pressing to know the conditions necessary for future studies to directly benefit from the high-level representations identified in a small set of high-quality recordings. Here we systematically investigate the data regime in which a set of standard semantic decoders would benefit from functional alignment.”

Reviewer comment (Experiments):
The proposed method in this paper employs one reference subject. How to deal with multiple reference subjects in practice?

Author response:
We agree with tcU3 that the present method is limited to a single reference subject. To mitigate this issue, we propose to generalise our result section, by reproducing the analysis with each possible reference subject. We kindly invite you to read the description of two additional experiments in the response we make to reviewers uqYK and HiZ5. Overall, however, building a multi-subject reference is still an open research question. Thual et al. 2022 introduces a method to build functional barycenters of individuals, which we did not try here because we feel that this way of computing reference subjects is fairly new and would require extensive testing of its own.  

Let us further expand on our previous answer by providing results to the two additional experiments described in our previous message, which we think bring insight on the robustness of our finding. We kindly invite you to read them in the second comment we have addressed to reviewer uqYK.

These two experiments bring new information regarding your third question. In short, the first experiment shows that our results hold regardless of which individual is used as the reference subject. Moreover, the second experiment shows that training on multiple aligned subjects can help boost results of our out-of-subject setup even further. It also shows that any subject can be used as reference to train the decoder and alignment.

We hope that these new elements will strengthen our results and would happily provide additional information.

In order to further elaborate on the point raised by your third question, we replicated our main experiment on the Natural Scenes Dataset. This replication shows results similar to that exposed in our paper, i.e. that functional alignment dramatically boosts the performance of semantic decoders when they are used on left-out subjects. They also show that more data will be needed to figure out a principled way to choose or build the reference subject. We kindly invite you to read them in the second comment we have addressed to reviewer HiZ5.

We hope that these new elements strengthen our current results. We will integrate them in our manuscript.

Reviewer comment (Experiments):
The major weakness of this paper is the limited number of subjects that are available for testing. Given that only three different subjects are used, it is unclear whether the findings of the paper and insights will generalize for a larger population.

Author response:
We agree that it is unclear whether our multi-subject alignment will scale to larger numbers of individuals.
Our decision to focus on a dataset with few individuals with a lot of natural stimuli stems from a simple question: under what data regime can functional alignment be beneficial to semantic decoding. 
In this regard, our contribution is a proof-of-concept: if decoders trained on a single participant can generalize to other unseen individuals for whom little data has been acquired, then it constitutes (1) a strong incentive to acquire very high amounts of data in a limited number of participants and (2) an additional legitimation of recent initiatives (e.g. Natural Scene Dataset, Kay, Naselaris et al. 2023; deep-fMRI-dataset, Lebel, Huth et al. 2023). 

To further address this issue, we propose to add two more experiments:
- Experiment 1: evaluate decoding gains on each possible reference subject of the Wen2017 dataset. This will allow us to triple the number of experiments, hopefully strengthening our original conclusions.
- Experiment 2: evaluate decoding gains for out-of-subject setups in which the decoder has been trained on multiple aligned subjects.

We will test all possible combinations to measure the effect of multi-subject training on this cohort.
We will post results for both these experiments as soon as we get them.

Reviewer comment (Evaluation):
How was the choice of retrieval metric made? Is there a reason standard metrics such as mean average precision or NDCG are not appropriate for evaluating this task?

Author response:
2. We chose to report retrieval metrics to be comparable with other recent papers (e.g. Scotti et al. 2023) and because image reconstruction metrics (like used in Ozcelik et al. 2023) depend on the generative models used, but are also difficult to assess. Also, top-k accuracy is used in similar work (Scotti et al. 2023; Defossez et al. 2023; Chen et al 2022). Finally, we think that relative median ranks are easier to understand intuitively and to compare between settings.

Reviewer comment (Writing):
It would be nice to provide more context to explaining the design and modeling strategy in Eq. (1). The way it is currently presented requires the reader to flip back and forth between this manuscript and Thual et al. 2022 to understand the methodology properly.

Author response:
3. We agree that this part of the paper is confusing. We propose to rearrange the method sections to avoid this back-and-forth.

Reviewer comment (Presentation):
It would be nice if a higher resolution image for Figure 2 could be made available to understand the false retrieval cases. Additionally, it would be nice to provide more description in the appendix to help the reader interpret the extended experimental observations.

Author response:
We agree and will make a higher-resolution version of Figure 2 in the appendix and add more comprehensive captions and paragraphs to introduce supplementary results.