Paper: Brain2Music: Reconstructing Music from Human Brain Activity (ICLR-2024)

Reviewer comment (Experiments):
The main drawback of this work is that the conclusion drawn is not persuasive by adopting powerful pre-trained models and the limited data size of the dataset utilized in this work. Mulan is a music text embedding model that's pre-trained on a large amount of music/text data. Similarly, MusicLM is a powerful conditional music generation model pre-trained on large amounts of data. Say we randomly sample embeddings from the hidden state space of Mulan and use them as a condition to guide MusicLM; we get a piece of 'meaningful' music. By meaningful, I mean it sounds like music, not noise. Now, linearly mapping fMRI to the embedding space of Mulan to generate music does not necessarily show the correspondence between fMRI and music but the powerful representation ability of Mulan and the generation ability of MusicLM. I also carefully listened to the demos given. This confirmed my thought that the generated or reconstructed music sounds like real music but is not very similar to the stimulus music. This is also verified by the low correlation results demonstrated in Table 1. It would also be interesting to see the correlation result of only using the mean vector or random vectors to reconstruct the music.

Author response:
a) While we acknowledge in Section 5 (Limitations) that our system's representation power is limited by MuLan, we emphasize that our quantitative and qualitative results demonstrate that our system's performance significantly outperforms chance (randomly sampling from the MuLan embedding space). While the use of MusicLM contributes to the generation of pleasant-sounding music, the results presented in Table 1 and Figure 1 would not surpass chance levels without the extraction of meaningful information from the input signal (fMRI recordings).

Reviewer comment (Evaluation):
No visualization results of the embeddings from fMRI to Mulan are given. Ideally, the embeddings $\hat{T}$ on the test set should show a clustering effect.

Author response:
b) Thanks for this suggestion. We ran a new analysis in which we take all MuLan embeddings predicted on the validation data and reduce their dimensionality from 128 down to 2 using a PCA. The resulting plot (with colors indicating genres) can now be found in [Figure A.6](https://github.com/f2mu/f2mu.github.io/blob/main/rebuttal-figures/fig-a6.png?raw=true) in the updated paper. We observe that the predictions form noisy clusters, most prominently for the genres _Classical_, _Hiphop_, and _Jazz_. To get a better feel for what to expect, we also PCA-dimensionality-reduced the ground truth MuLan embeddings (computed directly from the GTZAN eval data) and plotted them in juxtaposition.

Reviewer comment (Experiments):
Given the small size of the dataset, a K-fold evaluation should be adopted rather than a fixed test set with only 60 data points.

Author response:
c) During the fMRI recording, our test-set is repeated several times to enhance SNR. This is a common approach in fMRI encoding/decoding analyses. It’s not suitable for a k-fold CV, because the test examples are available with lower SNR. Apart from that, they are recorded in a separate run (i.e., temporally separated) from the train examples to be as certain as possible that there is no leakage (details are in Nakai et al. (2022)).

Reviewer comment (Experiments):
No comparisons are conducted with respect to other works. Especially seq2seq approaches, meaning directly predicting music waveform using fMRI. Thus, it is hard to evaluate the solidness of this work.

Author response:
d) In terms of comparison with other models, in our response to Reviewer QVSZ, we report the new results of constructing encoding models using different models. A detailed comparison of the prediction performance between different models and different brain regions yielded interesting insights. Particularly in terms of prediction performance, MuLan was found to be the most effective in predicting the auditory cortex during listening music.

Furthermore, to the best of our knowledge, there are no published works that have performed music reconstruction using seq2seq in fMRI. Therefore, we believe our study is the first to establish a correspondence between fMRI brain activity for music and DNN features of music.

Reviewer comment (Evaluation):
The analyses are fairly preliminary and there are currently no clear neuroscience insights.

Author response:
Our study is the first fMRI study to explore the relationship between the brain activity during music listening and DNNs for music generation. Given that music is very complex, the significance of finding models that accurately describe brain activity during musical engagement is substantial. Therefore, we believe that the findings of our study are of great importance.

However, the reviewer's comments have been extremely beneficial in making our research more intriguing. In response to the comments, we have conducted several additional analyses as follows.

Reviewer comment (Experiments):
There is no comparison against other DNN audio models such as wav2vec2.0 or HuBERT which have shown promising prediction accuracy in auditory cortex. The relation between w2v-BERT-avg and these prior models is unclear.

Author response:
Following the comments, we constructed encoding models using non-DNN features (Mel-spectrogram), which are thought to capture lower-level music features, and different DNN features (Hubert and wav2vec2.0). Additionally, to address subsequent questions, we compared the prediction performance of each layer of the auditory cortex. Our findings are summarized as follows (please see the newly added [Figure 3](https://raw.githubusercontent.com/f2mu/f2mu.github.io/main/rebuttal-figures/fig-3.png) in the revised manuscript):

1. MuLan demonstrated superior overall performance compared to the other models. Since it is an integral component of MusicLM, this may indicate the effectiveness of music generation models in exploring brain activity related to music. 
2. As anticipated, the prediction accuracy of the mel spectrogram decreased with the increasing hierarchy of the auditory cortex. 
3. However, strikingly, DNN features more accurately predicted the Lbelt/Pbelt regions (also known as A2/A3; early layers within the auditory hierarchy) than A1 (the earliest, primary auditory cortex), yet demonstrated lower performance in A4/A5 (association, higher auditory cortex). This trend has not been observed in other auditory neuroscience studies that focus on general sounds or speech, suggesting that it might be a phenomenon unique to music motivating further exploration.
4. The observed decline in the prediction performance of mel spectrogram with ascending hierarchy suggests that the results for DNN features might reflect inherent differences in the features themselves, rather than mere differences in signal-to-noise ratio (SNR) across different ROIs.

In summary, thanks to the reviewer, we have gained new and interesting insights into the neural representation of music in neuroscience and machine learning models.

Reviewer comment (Evaluation):
There are no perceptual experiments done to evaluate the quality of the reconstructions.

Author response:
During the short rebuttal period it is not feasible for us to undertake the requested rater study. We recognize this as a valuable addition for future work.

Reviewer comment (Experiments):
There is no serious investigation of how encoding and decoding results might vary across the auditory hierarchy.

Author response:
This is also an important point. As mentioned above, we now compared the prediction performance of each layer of the auditory cortex and found the results as described.

Reviewer comment (Experiments):
There is no investigation of how performance might vary across different layers of the network in the case of the encoding models.

Author response:
Indeed, we believe this is interesting to explore in addition, but due to time constraints during the rebuttal period, we would like to address it as a future task.

Reviewer comment (Experiments):
For encoding models there are no attempts to estimate the unique contribution of different models by comparing the performance of individual models against combined models. This is important as the features from different models are highly correlated. Thus a text-based model might predict auditory responses due to correlated features rather than a genuine response to text. As a consequence, the scatter plots showing correlated predictions is not surprising or particularly informative.

Author response:
Thank you very much for your feedback. We acknowledge that our initial analysis already demonstrates that both MuLanText and MuLanMusic effectively predict similar voxels, with MuLanMusic showing superior performance. However, we agree with your suggestion that a further analysis could elucidate how these two representations differ in their predictive capabilities. Accordingly, we conducted an investigation using banded ridge regression (la Tour et al., 2022) to assess the unique variance of these features. Our findings are summarized below (please see new supplementary [Figure A.3](https://github.com/f2mu/f2mu.github.io/blob/main/rebuttal-figures/fig-a3.png?raw=true) in the revised manuscript):

1. MuLanText lacks unique explained variance when compared to MuLanMusic.
2. This observation is notably different from trends in visual neuroscience (e.g. Takagi and Nishimoto, CVPR 2023), where text embeddings have distinct prediction performance. This might suggest a phenomenon unique to musical processing.
3. Our additional analysis is complementary to the initial analysis: although MuLanText predicts many areas within the auditory cortex, most of the information contributing to this prediction is contained in MuLanMusic.

In conclusion, the insightful suggestions from the reviewer have enabled us to uncover new perspectives in neuroscience research.

Reviewer comment (Evaluation):
The statistical approach used to compute p-values does not seem appropriate, since it assumes the samples are independent and Gaussian distributed. They could use a permutation test across stimuli as an alternative.

Author response:
Thank you for the important point. We changed to a more conservative p-value using a permutation test for the encoding model and all figures have been replaced. We confirmed that the auditory cortex was still central to the predictions, and there were no changes to any of the interpretations.

Reviewer comment (Evaluation):
For Figure 1A, it is unclear whether the identification accuracy is based on the latent embeddings or the reconstructions. The reviewer requests clarification on exactly what was done to compute this figure.

Author response:
Figure 1a: The identification accuracy is obtained by computing the embedding (MuLan or w2v-BERT-avg) of the reconstructed music.

The idea is to evaluate end to end (in the chain: music stimulus → fMRI scans → MusicLM→ music reconstruction) by comparing the similarity of music stimulus and music reconstruction in two embedding spaces. For that we compute the embeddings of the music stimulus and those of the reconstruction and determine the identification accuracy based on them. We've updated the respective caption in the paper to make this clearer.

Reviewer comment (Experiments):
According to your [demos](https://f2mu.github.io), the presented examples are nearly music clips with a strong rhythm, which may be easy to reconstruct from fMRI. Could you give some instances where the music clips come from a symphony (with a weak rhythm)?

Author response:
Reconstruction of music without precise rhythm, e.g., a symphony: Out of the 10 genres that GTZAN contains, one is classical. A non-cherry-picked example from it is in the supplementary material in sections _Comparison of Retrieval and Generation_ and _Comparison Across Subjects_ in the second row. Generally speaking, information about the exact timing of beats is very unlikely to be contained in fMRI scans due to their low temporal resolution. We do see, however, that a non-existent beat in the stimulus also leads to no beat being in the reconstruction.

Reviewer comment (Experiments):
As the authors mentioned in section 5, the relatively high TR of fMRI is a limitation. Have you explored the retrieval/reconstruction performance of music clips with different frequencies?

Author response:
fMRI sampling rate: We have not explored the reconstruction of music based on fMRI scans obtained at a higher sampling rate. Exploring this is interesting future work, perhaps alongside the use of EEG, for example. We thank the reviewer for this suggestion.

Reviewer comment (Experiments):
Are the fMRI recordings able to encode some complex music without a precise rhythm, like a symphony?

Author response:
Reconstruction of music without precise rhythm, e.g., a symphony: Out of the 10 genres that GTZAN contains, one is classical. A non-cherry-picked example from it is in the supplementary material in sections _Comparison of Retrieval and Generation_ and _Comparison Across Subjects_ in the second row. Generally speaking, information about the exact timing of beats is very unlikely to be contained in fMRI scans due to their low temporal resolution. We do see, however, that a non-existent beat in the stimulus also leads to no beat being in the reconstruction.