Paper: Revealing Vision-Language Integration in the Brain with Multimodal Networks (ICLR-2024)

Reviewer comment (Evaluation):
Model performance seems to be low. Figure 2a suggests that the Pearson correlation between model predictions and the true neuron activity is about 0.1. This means the model explains about 1% of the variance of the data (linear model r2). This is pretty low. Please explain why we should care about a model with limited prediction power.

Author response:
Note that Figure 2a averages the Pearson correlation over many models, neurons (electrodes) and time bins so this is not entirely representative of the meaningful range of possible Pearson r values. Recall that our window size extends from -2s to 2s and predicting activity in a time bin close to the tail end of this window size gets very low Pearson correlation. We are including these in the average in order to have a fuller representation in the plot. As stated in Section 4, we can obtain max Pearson correlation values of ~0.52 for particular time bins and neurons.

Reviewer comment (Evaluation):
The performance difference between brain regions, or between models, is low compared to the error bar per condition (figure 2a, and in multimodality tests). It is not clear what the error bar stands for. Is the difference between models or between electrodes sufficient for discrimination?

Author response:
Apologies for the confusion. We should have a better caption for Figure 2, which now has been moved to Figure 7 in the appendix. Once again, the error bar in Figure 2a is generated from averaging the raw Pearson correlation scores from running a single base regression. All scores in section 4c refer to the intervals that we obtain from bootstrapping analyses so this range will be different. Based on the differences we report in this section, we believe that the difference between multimodal and unimodal models is significant. Multimodal models outperform unimodal models by 0.07 points on the language-aligned dataset and 0.09 on the vision-aligned dataset.

Reviewer comment (Evaluation):
Whether the model selection is consistent with the known biological function of the brain areas.

Author response:
For selecting unimodal models, we selected models with general architectures that were used in prior work extensively such as vision transformers (SLIP-SimCLR, BEIT), CNNs (ConvNeXt), or language transformers (SBERT, SimCSE). These models have been shown to fit the brain consistently. For our multimodal selection, we stayed with the general motif of transformer architectures which generally have seen success at modeling activity in the brain. We selected the best-performing SOTA DNN models of vision-language processing at the time. Connecting DNN computations and circuitry to neural computations and circuitry is still an open question but newer work has shown connections between the transformer architecture and the brain.

Reviewer comment (Evaluation):
Do all multimodal tests identify the same set of multimodal selection electrodes? Quantitative results should be provided. A standard procedure should be followed to identify brain areas as potential vision-language integration regions.

Author response:
Our dataset-specific tests of multimodality (test 1 and 3) do identify potentially different electrodes which is why we take an intersection between electrodes that pass test 1 for the language-aligned dataset and vision-aligned dataset for test 2 (and do the same between test 3 and test 4). We are a bit unclear on how this isn’t a standard procedure or what quantitative results the reviewer wants to see. We believe we are using a standard procedure (bootstrapping model performance difference) as a way of comparing models over the significant time bins -- this is a general procedure.

Reviewer comment (Reproducibility):
How the regression model is trained? What is the input and what is the output? For each model with a different feature size, how does the parameter space for the ridge regression model differ between models? Whether the model performance was affected by the number of parameters?

Author response:
Sorry for the confusion! We do discuss these details in Appendix C -- due to space this discussion can’t find entirely in the main paper. To summarize, the regression model is trained as follows. We have a set of image-text pairs, $(i, t)$. We feed the pair (or one input in the tuple) to our DNN model, $M$, to get $r = M(i, t)$, where r is the activations of the DNN in response to $(i, t)$. We also extract a 4s window of sEEG activity from one electrode e and split this into 161 time bins of averaged activity, $e_t$. The regression is trained to take $r$ as input and predict $e_t$ for each time bin across all electrodes. So, the number of parameters in the regression is dependent on the dimensionality of $r$ which is affected by the dimensionality reduction procedure. We found that the number of parameters in the regression did not affect the regression performance across models.

Reviewer comment (Writing):
It would be helpful to include a better explanation of what the "event structures" are exactly, possibly with a picture.

Author response:
Apologies, we realize the concept is a bit confusing. We include a figure in the appendix showing event structures selection in Appendix A.2.

Reviewer comment (Evaluation):
Page 6, Section 4, second paragraph: This section is overloaded with numbers, and while careful readers may try to interpret them, most will not. Additional explanations should be included to guide the reader on what to think about the meaning and significance of the numbers presented.

Author response:
Thank you! We have updated the results section to better address the numbers here. Our goal was to present the results without overselling/overclaiming our findings but we realize that our current writing may come off as dry or needlessly confusing to a reader.

Reviewer comment (Presentation):
Figure 2: The caption does not explain the figure well. Panels (a) and (b) are not mentioned, “mid left” and “bottom right” labels are confusing, model names should be directly on figure plots, and the dots on the freesurfer brain surfaces are unexplained. Additionally, the red dot is hard to distinguish for color-blind readers and should be changed to blue.

Author response:
Thank you! We have decided to focus on some new results and have moved Figure 2 to the appendix as Figure 7. In addition, we have updated the caption and figure.

Reviewer comment (Evaluation):
Clarification is needed on whether the singular electrodes shown in Figure 8 as “multimodal across vision and language” provide sufficient evidence for multimodal processing in those areas. The yellow dots appear scattered, and it is unclear whether the absence in visual areas reflects a true lack or lack of electrode implantation.

Author response:
The improvement in Pearson’s r that we achieve is significant. We introduce many controls on the dataset, architecture, and statistics. We still achieve significant gains from multimodal models, sometimes close to r=0.1. We agree with the reviewer that our current draft may leave the reader feeling confused as to whether our results are trustworthy so we include some new text focusing on this.
In general, we approach these results with caution due to the fact that little is known about multimodal integration, especially with vision and language. We are excited to see our results match prior work. We are also excited to see that our results are not explained by better task performance. We believe that this is a strong start to deeper exploration into these areas and what in our networks drives our prediction. That being said, our motivation for showing numbers was not to deter readers or reduce confidence but instead to be factual. We believe these studies are invaluable given the lack of understanding around multimodal integration.

Reviewer comment (Experiments):
The implantation sites of sEEG electrodes in your dataset were clinically motivated. To what extent did they cover the areas you were interested in? This includes whether all areas of interest were covered, and if so, whether the coverage was sufficient for your analysis.

Author response:
Great question, and this likely connects with the weakness you were bringing up. There were some areas that were not covered in the implantation such as early visual cortex. These are areas where we know significant visual processing occurs but these could not be included in our analyses. We also had fewer electrodes in the occipitotemporal cortex or the superior frontal cortex than we would have liked, making the percentage of multimodal electrodes small in these regions as shown in Figure 4. 

Figure 5 shows that we have broad coverage across the brain but a majority of the electrodes naturally fell into the superior temporal and middle temporal cortex -- one subject had all of their electrodes in this location. We had coverage across all desirable locations of the brain other than EVC and having more coverage in OTC or the superior frontal cortex would have given more evidence for these regions to be multimodal.

Reviewer comment (Experiments):
Are there multimodal models combining vision, text, and audio? Given that your data contains all three modalities, what was/is the main obstacle to identifying tri-modal predictive brain regions? Is it due to the lack of appropriate DNNs or another factor?

Author response:
There are [1]! The main obstacle for this paper is processing time -- none of our methods prevent us from extending to one more modality. Obviously, we would need to increase the number of baselines to have a meaningful comparison which would be difficult in the time we have for this rebuttal. We would like to have this as future work where we compare 3 modalities vs 2 modalities vs 1 modality with these models. We would also like to include video models as a comparison as well.

Reviewer comment (Experiments):
In your experiments, were the subjects able to hear the audio track of the movie?

Author response:
Yes, subjects were able to hear the audio track. The movie presentation was unaltered from a normal movie viewing experience.

Reviewer comment (Novelty):
For your literature review, here is another work comparing vision to DNN specifically on sEEG data from 100+ subjects: https://www.nature.com/articles/s42003-018-0110-y

Author response:
Thank you! We will add this to our related work section.

Reviewer comment (Experiments):
Did the authors perform SLIP-CLIP vs SLIP-SimCLR vs SLIP-Combo comparison? Because SLIP-CLIP is also multimodal I am curious what was the motivation for only showing SLIP-CLIP vs Combo results in Figure 4.

Author response:
This is a good point! We could have included SLIP-CLIP as well but chose SLIP-Combo and SLIP-SimCLR as our main comparison. The choice between SLIP-CLIP and SLIP-Combo as our multimodal choice was arbitrary. We show the results between SLIP-CLIP and SLIP-SimCLR for comparison in Appendix H.

Reviewer comment (Evaluation):
It is not clear to me why language alignment and vision alignment event structures leads to difference in results. I would like to read author’s explanation on why results depend on how event structures and if this is a limitation of this approach.

Author response:
This is an important point and we add a few sentences to Appendix A.2. Event structures are a first-order approximation for choosing image-text pairs without imposing a strong hypothesis on the structure of real vision-language integration. Vision-aligned data selects scene cuts and the closest sentence and this may be modeling longer-term integration of vision and language than the language-aligned dataset which uses context segments and frames. This is because the vision-aligned data is less aligned than the language-aligned data since a frame occurs with every word but a sentence can occur much later (sometimes, 2 seconds) after a scene cut. 
We believe the event structure selection has strengths and limitations. Our event structures are naturalistic and intuitive, prioritizing one modality and sampling the other modality. They do not rely on model architecture or make assumptions about vision-language integration. They also cover many forms of possible integration by strictly aligning (language-aligned) and loosely aligning (vision-aligned) the modalities. However, we recognize that in an ideal world, we would have defined vision-language events similar to words or objects that are easily extractable from our movie dataset where we have reasonable evidence that vision-language integration is occurring. In the end though, we argue that the event structure selection makes for a more difficult benchmark by greatly expanding the potential forms of vision-language integration that multimodal architectures need to consider.

Reviewer comment (Novelty):
Is the dataset also part of this paper or is it from an already published paper? If yes then this is an additional contribution of this paper and should be emphasized.

Author response:
The neural recordings in this dataset are unpublished. We plan to release the dataset of 7 neural recordings from the 7 subjects when making this work public. However, for the purposes of this paper, we don’t plan to emphasize the dataset since this will be under review in a different submission as a dataset paper.

Reviewer comment (Reproducibility):
Will this dataset be publicly released? (if not released already)

Author response:
The neural recordings in this dataset are unpublished. We plan to release the dataset of 7 neural recordings from the 7 subjects when making this work public. However, for the purposes of this paper, we don’t plan to emphasize the dataset since this will be under review in a different submission as a dataset paper.

Reviewer comment (Evaluation):
The biggest weakness is the central claim that the proposed analyses of multi-modal models can localize vision-language **integration** in the brain. Can the authors please define what they mean by vision-language integration? Even on the model side, it is an open question to what degree multi-modal models actually integrate information from multiple modalities as opposed to increasing the alignment between individual modalities (Liang et al. 2022 https://arxiv.org/pdf/2209.03430.pdf). It is not clear whether the results that are observed are due to vision-language integration or whether they are due to improved representations of the language-only or vision-only modality. For example, in Fig 4, all regions that are identified as multimodal (e.g. marked with a green star), are canonical language regions. How can the authors disentangle the effect of integration of modalities vs the effect of improving language-only information in the model? For instance, let's do a thought experiment and apply the authors' methods to study different layers of a language-only encoder: I predict that these results will be similar to what has been shown before which is that some layers in the language model predict exactly the same regions they call multimodal substantially better than other layers (see Jain and Huth, 2018 NeurIPS; Toneva and Wehbe, 2019 NeurIPS for some of the earlier work showing this). That clearly is not due to multimodality though because the input is only language.

Author response:
**Weakness 1**: We address this point-by-point below.

>The biggest weakness is the central claim that the proposed analyses of multi-modal models can localize vision-language integration in the brain. Can the authors please define what they mean by vision-language integration?

This is a good point. We define vision-language integration to be any non-linear combination of vision-language features as carried out by the candidate neural networks. Our candidate DNN models like ALBEF, Flava, BLIP, SLIP-Combo etc. represent a hypothetical integration of vision-language features. Effectively, we are testing whether these DNNs model activity in the brain better than language or vision models. 

Using neural networks in this way is desirable. We are not attaching ourselves to a specific hypothesis of vision-language integration but instead allowing us to have a larger search space. 


>Even on the model side, it is an open question to what degree multi-modal models actually integrate information from multiple modalities as opposed to increasing the alignment between individual modalities (Liang et al. 2022 https://arxiv.org/pdf/2209.03430.pdf). It is not clear whether the results that are observed are due to vision-language integration or whether they are due to improved representations of the language-only or vision-only modality. 

Could the reviewer explain what they mean by “increasing the alignment between individual modalities”? How is this distinct from integration? The citation the reviewer provides also does not seem to provide any evidence for their claim as we understand it -- Section 4 gives evidence that models perform some alignment between modalities that allows for improvement on many tasks that require some form of multimodal reasoning such as visual question answering, etc. The paper doesn’t show that better unimodal representations are necessarily what leads to improved performance on tasks VQA -- if this was true, architectures like LXMERT or VisualBERT would still be SOTA as long as we introduce better visual or language representations which is not the insights that led to models like BLIP, ALBEF or Flava. Furthermore, we are unsure how our results don’t dispel this doubt. We show that multimodal models significantly out-perform both vision and language models at predicting vision- and language-aligned data. We show that these models do not have strictly better language or visual reasoning, even on our dataset in Table 5 and Table 6. We include a combined version of Table 5 and 6 which presents average language and vision task performance and a new subsection discussing this in Section 4.3. We also include Tables 5 and 6 below. 

| Model                       | Test Set Perplexity (One Movie) |
|-----------------------------|---------------------------------|
| BLIP                        | 168.7                          |
| ALBEF                       | 223.8                        |
| Flava                       | 202.2                           |
| SLIP-Combo Language Encoder | 197.6      |
| SLIP-CLIP Language Encoder  | 259.3  |
| SBERT                       | 121.3                        |
| SimCSE                      | 145.4                       |


| Model                      | Scene Classification Accuracy (Places365) |
|----------------------------|-------------------------------------------|
| SLIP-Combo Vision Encoder  | 73.9                                      |
| SLIP-CLIP Vision Encoder   | 71.2                                      |
| SLIP-SimCLR Vision Encoder | 71.0                                      |
| BEIT                       | 75.4                                      |
| ConvNeXt                   | 76.2                                      |
| ALBEF                      | 60.1                                      |
| Flava                      | 58.4                                      |
| BLIP                       | 62.4                                      |

Reviewer comment (Presentation):
The presentation can be much improved: there is very little discussion of what the different models that are used are and how they are trained. This is key to understand the contributions of this work. I suggest the authors include a table of all models and model variations used with clearly marked information about what modality was used to train the model and what modality is used to evaluate the model (e.g. even if the authors are using only a vision encoder at inference time, if the vision encoder was jointly trained with a language encoder, this should be noted as this may make a difference in the representations).

Author response:
**Weakness 2**:

>The presentation can be much improved: there is very little discussion of what the different models that are used are and how they are trained. This is key to understand the contributions of this work. I suggest the authors include a table of all models and model variations used with clearly marked information about what modality was used to train the model and what modality is used to evaluate the model (e.g. even if the authors are using only a vision encoder at inference time, if the vision encoder was jointly trained with a language encoder, this should be noted as this may make a difference in the representations)

We do include a longer discussion in appendix B of the paper. We can make this into a table as well which we include in Appendix B. If the reviewer is still unsatisfied with the discussion, specific points on what remains confusing would be much appreciated.

Reviewer comment (Novelty):
The work is still not positioned well in the current literature on multi-modal modeling of brain recordings, and it’s not clear what the novelty here is for people who are unfamiliar with this area. The authors should discuss the work of Oota et al. 2022 COLING and Wang et al. 2022 bioRxiv https://www.biorxiv.org/content/10.1101/2022.09.27.508760v1.

Author response:
**Weakness 3**:

>The work is still not positioned well in the current literature on multi-modal modeling of brain recordings, and it’s not clear what the novelty here is for people who are unfamiliar with this area. The authors should discuss the work of Oota et al. 2022 COLING and Wang et al. 2022 bioRxiv https://www.biorxiv.org/content/10.1101/2022.09.27.508760v1.

Both papers are discussed in our current related work section but we expand the discussion as part of a re-worked related work section in the paper to focus more on the multimodal processing and less on fitting models to the brain.

Reviewer comment (Novelty):
The contribution to an ML audience is not very clear. I believe this work will be better suited to be evaluated by neuroscientists, since the claimed contributions are on the neuroscience side, and will also be more appreciated at a neuroscience venue.

Author response:
**Weakness 4**:

>The contribution to an ML audience is not very clear. I believe this work will be better suited to be evaluated by neuroscientists, since the claimed contributions are on the neuroscience side, and will also be more appreciated at a neuroscience venue.

We believe that there is a core contribution to the ML audience for comparing neural networks with the brain. We design a standard pipeline with statistical comparisons that lend credibility to a conclusion that one model fits the brain better than another. 
We highlight our ML contributions in an updated subsection 4.4. In Figure 4 we show the best multimodal model across our multimodal electrodes that we find. We believe this investigation can lead to natural follow-up work with our data and pipeline such that in future years, new papers with new multimodal transformers can be released and these architectures are hopefully more similar to the brain. We also want to point out that prior papers have been accepted to ICLR despite not having considerable ML contributions such as [1] or [2]. We believe our work is in-line with what is presented at ICLR. 

[1] Wang et. al. BrainBERT: Self-supervised representation learning for intracranial recordings. ICLR 2023.

[2] Loong Aw and Toneva. Training language models to summarize narratives improves brain alignment. ICLR 2023.