From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: The aim of this paper is to use sparse information from literature to solve the seemingly ill-posed problem of mapping from peak coordinates to fmri brain activity images. Prior work in deep learning in recent years has shown that going for such tasks can lead to highly non-trivial models. 

The reviewers although finding the paper well-written and clear are not entirely convinced about the merits of the paper. This indicates that although the paper is tackling an important and in this AC's opinion well-posed problem, there is still some work to do before it is ready for publication.

It is therefore recommended that the authors improve the paper further and submit to the next relevant conference. 
justification_for_why_not_higher_score: This is a potentially important and novel contribution that requires to be done completely right to get the impact it deserves.
justification_for_why_not_lower_score: N/A

From: ICLR.cc/2023/Conference/Paper3126/Authors
title: We would gladly discuss evolutions
comment: Dear reviewers,

We appreciate again your commitment in this review and the constructive feedback you sent. This allowed us to improve our manuscript along the following points:

- better reconstruction performance by leveraging a standardization step in the peak preprocessing
- clearer method figure
- more precise and flowing writing of the manuscript

We would gladly discuss those evolutions with you.

 

Best regards,

The authors

From: ICLR.cc/2023/Conference/Paper3126/Authors
title: Answer to Reviewer eWu3
comment: We thank the reviewer for their thorough lecture of our paper.

In the following, we will address the reviewer’s questions and comments:

**Brain image vs DiFuMo representation** - There is indeed a potential confusion between the two across the paper. We added some clarification early in the paper to clearly indicate that we use the DiFuMo 1024-dimensional representation of the image instead of the voxels.

**Clarity of Figure 1** - We agree with the reviewer that the figure was confusing in multiple aspects, from the representations of the data to the different learning procedures. We made a major revision to the figure, hopefully fixing those points. In details, we:

- switched to a more detailled view of each of the 3 steps of Peaks2Image (learning to reconstruct statistical images from peaks, predicting dense spatial representations for neuroscientific studies, learning to decode real-world fMRI images from reconstructed images of studies). We believe that it better separates tasks that are learning procedures from evaluations.
- specify the input data to each step while stating the dependencies between them in term of models and data
- gave each module the same design (e.g models and operators vs representations of elements)

**Reconstruction performance** - We improved the peak extraction preprocessing pipeline, leading to much better results both in terms of sparse Gaussian correlations (from 0.27 to 0.42) and the reconstructions (from 0.28 to 0.57), see Fig. 2.

We identified two shortcomings in our initial peak extraction pipeline:

- isolated voxels with high statistical values, leading to a peak extracted in an inactive region
- activated regions that present multiple local maxima, leading to multiple peaks extracted within in restricted area.

This led us to update our preprocessing to:

- an initial embedding of the image in DiFuMo space, which regularizes regions with high activity
- a standardization in the voxel space to prevent extraction from false positive voxels

With those adjustments, we managed to greatly improve the correlation of the sparse Gaussian representations. We consider the sparse Gaussian correlation gain a good metric to show our peak extraction pipeline progress. 

We noticed that having better initial representations facilitated the learning of the reconstruction process, leading to:

- a better overall correlation of the reconstructions (from 0.28 to 0.57)
- the elimination of anti-correlated reconstructed that we could observe initially


**Typo in page 5** - We fixed the typo. Thank you for pointing it out.

From: ICLR.cc/2023/Conference/Paper3126/Authors
title: Answer to Reviewer GJ9C
comment: We thank the reviewer for their thorough lecture of our paper.

In the following, we will address the reviewer’s questions and comments:

**Relevance of extracted peaks?** - With respect to the literature, we choose to represent the statistical brain images by peaks that have a significant values (standardized statistic > 3, corresponding approximately to  p<.001). We believe that the correlation between the sparse Gaussian representations we build from those peaks with the original images (Fig 2) is a good indicator of the relevance of the extracted peaks. The levels of correlations observed indicate that a large part of the information is retrieved in those sparse representations.


**Why reducing the images to individual peaks?** - This peak extraction task has to be understood in the meta-analysis context. In most brain imaging studies, reporting peaks location is the way to describe and share significant statistical effects in group-level studies. Most neuroscientific studies only report peak activation tables without a link to whole brain data, which prevents running meta-analysis at the image level. Peaks2Image aims at reducing this gap, allowing meta-analysis on whole-brain data instead of discrete locations.

We believe this traduces a loss of the narrative in our paper. We have reformulated it to better keep track of the use of the reconstructions to perform meta-analysis on full brain images (instead of peaks).

**Ill-defined variables** - We have modified the paper to better define $f$ and $D$. We added the sparsity and non-negativity constraints under which the DiFuMo dictionary $D$ was learnt.

**Comparison to SOTA** - We carefully looked at the references from the reviewers.

* Manning 2018 performs reconstruction of the temporal fMRI signal using a hierarchical topographic factor analysis. Reconstructions are evaluated through a decoding task where they identify specific moments of an audio story participants are listening to.
* Wu 2019 enforces region sparsity on regression problems and experiments on different binary classification tasks.
* CAI 2016 recovers the covariance structure from voxel activity, which is used to estimate structure similarity between 6 different animals.

We added appropriate references in the related work section but did not perform comparative baselines, as these papers are obviously pursuing different goals.

**Multi-site multi-subject data**  Please note that
- NeuroVault is an open-database, freely contributed by researchers in a relatively unconstrained manner, hence all relevant information (individual versus group data, subject or population characteristics) is not necessarily available
- We are working in a meta-analysis mindset, meaning that we consider a "third-level" type of analysis, on top of individual and group-level analyses. We do not expect to meaningfully leverage individual information at this level of analysis, and are clearly focussed on idetifying cognitive effects that generalize across studies, hence inherently with limited spaial accuracy.

From: ICLR.cc/2023/Conference/Paper3126/Authors
title: Answer to Reviewer LK9E
comment: We thank the reviewer for their thorough lecture of our paper.

In the following, we will address the reviewer’s questions and comments:

**Is brain reconstruction a reasonable task?** - We believe that the reconstruction problem is feasible as initiated in [1]. Even though the peaks might seem like a small amount of information compared to the thousands of voxels, brain images live in a small manifold within the large voxel space. This makes small pieces of information such as peaks relevant to disclose broader pattern of the image. In essence, recovering an image from its peaks is a compressed sensing problem.

Unsatisfying reconstruction performance has also been evoked by reviewer eWu3. Experimentally, we have improved the peak extraction during the discussion period, adding a standardization of the image. Indeed, we observe that scaling differences on the training set was having a deleterious impact on reconstruction. This leads to (1) a better sparse gaussian representation of the image and (2) to much better reconstruction performance, eliminating reconstructions that were anti-correlated. This confirms our intuition that one can successfully reconstruct relevant brain maps from the given sets of peaks.

Please see Fig.2 with the new results.

[1] K. J. Gorgolewski, T. Yarkoni, and R. A. Poldrack. peaks2maps: reconstructing unthresholded statistical maps from peak coordinates using deep neural networks. F1000Research, 8, 2019

**Lack of an ablation study** - In the experiments, we trained the decoding model on both (1) the sparse Gaussian representations and (2) the reconstruction from our model. (1) corresponds to a baseline analysis on the raw peaks. We show that reconstructions yield higher accuracy than the sparse Gaussian representations in Fig 4. We agree with the reviewer that we did not compare to sets (Deep Sets, Set Transformers) methods, as we could not add it reasonably during the discussion period. We anticipate that they would not be more succesful in average than the sparse Gaussian representations.


**Results in table format** - We added results in a table format to complete Figure 4.

From: ICLR.cc/2023/Conference/Paper3126/Authors
title: Revision Summary 
comment: We thank all the reviewers for their reviews and their constructive comments and questions. We have tried our best to answer and take into account those comments during the discussion period, and it led to what we consider significant improvements to the paper.

For clarity, we list all the changes that have been made during the period below.

**Reconstruction performance** - We improved our peak extraction preprocessing pipeline without modifying Peaks2Image architecture. This leads to better sparse Gaussian representations (overall correlation improves from 0.27 to 0.42), and a better learnt reconstruction process (from 0.28 to 0.57 correlation). This has been done by adding a normalization step before performing the peak extraction.

**Figure clarity** - We reworked Fig 1. In particular, we:

- reviewed the overall design to better illustrate the 3 steps of Peaks2Image (learning to reconstruct statistical images from peaks, predicting dense spatial representations for neuroscientific studies, learning to decode real-world fMRI images from reconstructed images of studies)
- gave each module the same design (e.g models and operators vs representations of elements)


**Paper clarity** - we improved sections related to remarks or questions from the reviewers. In particular, we:
  * modified the abstract to better highlight the purpose of combining brain map images and neuroscientific studies.
  * precised the definition of *f* and *D* along with the constraints on *D*
  * clarified that we use DiFuMo representations for the learning process, and not the image voxels
  * added logistic regression decoding baseline
  * added decoding performance in table format

From: ICLR.cc/2023/Conference/Paper3126/Reviewer_LK9E
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This work proposes a neural-network approach to reconstruct continuous spatial representations of brain activity from peak activation tables. Using reconstructions of studies published in the neuroscientific literature, the authors train a decoder leading to a much broader set of decoded terms than current image-based studies.
strength_and_weaknesses: Strength:
1.This paper is well-written and easy to read. The definition of tasks and problems are very clear.
2.For a decoding task, the idea that fully utilize the rich descriptions of the studied cognitive tasks in scientific report is novel.
Weaknesses
1.My major concern is the necessity of image reconstruction from peaks. As illustrated in the paper, the authors train a reconstruction network to convert the peak coordinates into images. In my opinion, this ill-posed problem is too hard to be reasonable. Reconstructing the fMRI from only a few peak coordinates (less than 100 according to the paper) seems impossible to recover the details of images. Thus, the reconstruction is unnecessary. If the peak coordinates contain enough information for the following decoding tasks, reconstruction would introduce inevitable noise. If the peak coordinates are not enough, reconstruction still can’t help the decoding since extra information comes from the ‘guessing’ of the trained network. Can the authors give more explanation about this?
2.Also, the experiments do not verify that image reconstruction can help in improving the decoding performance. The ablation study should compare with model using peak coordinates as input. So, I am wondering about the performance of model that uses peak coordinates as input.
3.Experimental results are all shown in figures, which is good for analysis. However, giving some extra tables with exact values can be better.
clarity,_quality,_novelty_and_reproducibility: This paper is well-written and easy to read. 
The definition of tasks and problems are very clear. 
The idea that fully utilize the rich descriptions of the studied cognitive tasks in scientific report is novel, but reconstructing image from peaks is not quite reasonable. 
Reproducibility is good since details are all given in the experimental setup.
summary_of_the_review: This paper contains novel ideas such as utilizing the rich descriptions of the studied cognitive tasks in scientific reports to assist in brain decoding task. However, I have several concerns about the image reconstruction from peaks and experiments. For more details please see the ‘Strength And Weaknesses’.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3126/Reviewer_GJ9C
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper proposes Peaks2Image as a neural-network approach to reconstruct continuous spatial representations of brain activity from peak activation tables. This study aims to learn a decoding model by using TF-IDF features as labels so that the number of decoded terms will be much broader than in current image-based studies. Based on 43,000 NeuroVault images, they successfully decoded 58 out of 81 concepts. In summary, this paper is interesting in terms of neuroscience, but it is unclear to what extent the proposed method is novel in terms of machine learning.
strength_and_weaknesses: Please refer to the review section.

clarity,_quality,_novelty_and_reproducibility: Please refer to the review section.

summary_of_the_review: This paper contains several controversial claims. 

The paper uses Nilearn to estimate peaks and then uses these peaks to learn a model from data. However, there is no evaluation of the error associated with the selection of these peaks. Further, what evidence shows that you are more likely to see neural responses in these peaks than examining the whole brain data?

Another concern is the proposed formulation. As an example, what is the definition of the function f() in Equation 1? To learn f*, what constraints are considered? The same question can be asked for D in Equation 2. Due to the fact that D without any constraint may increase the chance of overfitting. 

Empirical studies are another concern. The proposed method only compares with some classic machine learning approaches, such as Kernel Ridge or Space Gaussian. There are several groups of state-of-the-art techniques that aim at similar types of analysis and should be compared with the proposed approach, e.g., HTFA (Manning 2018), BRSA (Cai 2016), and DRD (Wu 2019). 

In addition to the aforementioned concerns, the proposed approach should clarify some related concepts for multi-subject and multi-resource analysis. It seems that this study uses multi-subject data. However, I could not find anywhere that discusses how the author(s) functionally align brain responses from different subjects. Alternatively, Neuro Vault is a multi-resource repository where data are provided from different machines and environments. I also did not find any information on how the proposed method deals with bach effects to learn a model from multi-site data. You may provide additional information in a preprocessing/preparation section.

correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3126/Reviewer_eWu3
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: Cognitive neuroscience aims to reveal the underlying brain regions corresponding to different cognitive processes, and such studies are usually performed by comparing and contrasting the fMRI scans of subjects performing designed cognitive tasks. A major drawback in this field is the high cost of data acquisition, which limits the sample size and hence undermines the statistical power. Meta-analysis over many individual studies, though may partially overcome the sample size issue, usually runs into data unavailability issues where only the locations (in form of standardized coordinates) of peaks of activations are reported in literature while the raw image data are not publicly shared.

On the other hand of the story are the recent efforts (e.g., NeuroVault, OpenNeuro) of aggregating and curating data from individual studies yielding large-scale public image repositories. In these repositories, however, the annotations and metadata are usually scarce and inconsistent.

The authors claimed that the rich text data from neuroscientific literature and the large-scale public image data are complementary. Leveraging the advantages from both sides, they proposed a novel meta-analysis method that can decode fMRI statistical maps corresponding to concepts in neuroscientific literature.
strength_and_weaknesses: Strengths
1. The authors proposed an interesting workflow. If I understand it correctly, the authors are using the following priors: (1) fMRI brain maps is tightly correlated to the underlying cognitive states, and (2) distribution of peak locations in fMRI brain maps is highly correlated to the overall global pattern in fMRI brain maps. With these priors, the authors proposed to train a neural network model to predict the global fMRI pattern from sparse locations of peaks, and with that trained model they synthesized a paired dataset of (cognitive descriptions, global fMRI patterns) using the (cognitive descriptions, sparse locations of peaks) from the neuroscientific literatures. Finally, they trained a separate decoder with the (cognitive descriptions, global fMRI patterns) pairs to decode the former from the latter. The decoder is the ultimate piece of mastery that can describe the underlying cognitive states of an arbitrary fMRI image.

Weaknesses
1. In my humble opinion, the abstract may need to be better articulated. I was very disoriented and confused by the abstract until I finished reading the introduction section. Specifically, I was confused by (1) what the two complementary sources of data are, (2) why they are complementary, and (3) what is the proposed method trying to accomplish. For (1) and (2), even though the two sources of data are both explicitly mentioned, I did not have enough context to parse their relationship. In contrary, the introduction section has a very natural flow and is painless to follow.
2. It is a bit misleading that when the authors mentioned the word “image” they did not necessarily mean “image”, but rather the “1x1024 vector representation in the DeFuMo space of the image”. I believe it might be helpful to provide a clarifying statement somewhere early in the paper to avoid potential misunderstanding.
3. Figure 1 can be further polished. (1) Figure 1A did not clearly show what data format is provided and what are derived. It might be better if that information is illustrated more explicitly. (2) A lot of things are represented as “blue thin rectangle”, yet some of them are modules/operations while others are entities. It may be better to replace some of them. For example, “Reconstruction” and “decoder” can be replaced by sketches of the neural networks, and “Labels” can be replaced by some clipart object, etc. (3) The caption mentioned that in Figure 1B, the brain map is reconstructed using the trained model, so it would be better to show the sketch of model in an appropriate place and indicate that its weights are frozen.
4. The reconstruction performance (Section 5.1, Figure 2) is quite poor. a correlation of 0.3 does not seem too promising.

Minor Things
1. Is there some typo in the following sentence? Page 5, 3rd paragraph. “When the percentile corresponds to a To avoid selecting studies with a null tf-idf value, we replace the percentile threshold with a strictly positive tf-idf value criteria.”
clarity,_quality,_novelty_and_reproducibility: Clarity and quality of the paper is mediocre. Novelty is good, but not remarkable.
summary_of_the_review: I like the idea behind the paper, but the current presentation is selling it short, significantly. I do not believe the paper in its current form meets the publication standard at ICLR.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold