Paper: System Identification of Neural Systems: Going Beyond Images to Modelling Dynamics (ICLR-2024)

Reviewer comment (Experiments):
In the simulated environment, I3D ResNet-50, ViT-B, and MViT-B can be used for purposes (1) and (2), but not for (3). I would suggest including more target models for the purpose (3).

Author response:
We leave it for future work to expand on target models due to the limited scope of the rebuttal period.

Reviewer comment (Experiments):
Figure 4. if I understand correctly, OmniMAE-B pretrained is indeed self-supervised. But OmniMAE-B finetuned should be self-supervised + supervised finetuning. Is this comparison fair to show the differences between fully-supervised vs. self-supversied??

Author response:
For fair comparison, we added TimeSformer to Fig. 3b (which is trained fully-supervised on SSV2 relying on ViT base without self supervised pre-training). It still confirms that fully supervised models surpass self supervised ones. Below is a table showing the regression scores of OmniMAE Pretrained compared to TimeSformer. 

|     | OmniMAE Pretrained | TimeSformer |
|-----|--------------------|-------------|
| V1  | 0.23               | 0.28        |
| V2  | 0.21               | 0.27        |
| V3  | 0.19               | 0.26        |
| V4  | 0.18               | 0.26        |
| LOC | 0.20               | 0.28        |
| EBA | 0.23               | 0.33        |
| FFA | 0.21               | 0.27        |
| STS | 0.14               | 0.20        |
| PPA | 0.15               | 0.21        |

Reviewer comment (Theory):
The DNN models are not intended as models of biological vision and do not contain any interpretable dynamics that would enable conclusions about theories. Also, they usually run at so slow timescales (typically the frame rate of the video), that they could never have the temporal dynamics of biological networks in the first place. Thus, it is not surprising that the conclusions are not particularly clean.

Author response:
Several studies (Lahner et al., 2023; Zhou et al., 2022) have used DNNs models compared to brain computation, and it is well established to use fMRI in these studies, we are not the first to. Despite the slower timescales, our results demonstrate the superiority of video understanding over image ones in neural encoding.

Reviewer comment (Evaluation):
fMRI is not able to resolve dynamics of visual processes. Thus, it cannot provide evidence about these dynamics.

Author response:
Several studies (Lahner et al., 2023; Zhou et al., 2022) have used DNNs models compared to brain computation, and it is well established to use fMRI in these studies, we are not the first to. Despite the slower timescales, our results demonstrate the superiority of video understanding over image ones in neural encoding.

Reviewer comment (Novelty):
The relationship to the Algonauts challenge remains unclear. The challenge website is open for post-challenge submissions, so the authors could have submitted their models to the competition to get scores for the official test set. If that was not desirable for some reason, the results for the top entries of this challenge would be useful to get an idea of how close to the state of the art the models from this paper perform. There does not appear to be a substantial step of this manuscript beyond the Algonauts papers.

Author response:
Challenge website prevents new submissions on the test set. Although our focus is on comparing existing SOA video models in neural encoding, we still compare to the challenge winner on the validation set, Fig. S4. It shows comparable results, without employing an ensemble of models, different modalities or fine-tuning the backbone as the winning entry.

Reviewer comment (Novelty):
The manuscript does not provide the promised insights into dynamics and instead becomes an incremental step repeating things that have been done for image neural networks with video networks without providing substantial new insights.

Author response:
We thank the reviewer for the timely feedback. We want to re-iterate that previous literature would not be able to give meaningful insights comparing video understanding models to biological neural systems without a large-scale and comprehensive study. As such it is not trivial to provide a study of such breadth and depth for other researchers to benefit from.

We want to re-iterate on the differences to prior research:
1- The first system identification study to differentiate image vs. video understanding, this will pave the way for other researchers studying this question by establishing the feasibility of it through simulated experiments.

2- We are the first to conduct such large-scale (more than 30 models) and comprehensive study on video understanding models including various aspects (convolutional/transformer, self/fully supervised, two-stream/single stream and image/video understanding). 

3- Having a deeper understanding of which video understanding models are better (e.g., MViT and SlowFast), and why (as the first to establish connections to why ViTs are worse on the early-mid regions when they rely on the patchifying stem and why MViT could be overcoming such concern).

We believe our research is paving the road for other researchers to have a better understanding of the problem. If the reviewer can clarify in how these contributions is trivial we can address the concern and we are thankful for the timely response.

Reviewer comment (Experiments):
The main question is why this dataset was chosen and why the study was conducted without the official evaluations. To improve the perception of this work, a substantial insight into how we might capture the dynamics of human visual perception better would be necessary.

Author response:
We provide a large-scale study of deep video understanding from a neuroscience perspective. Thus, the dataset is driven by the characteristics of short video stimuli, which is in-line with the typical datasets used for training deep video understanding models for a better comparison.

Reviewer comment (Novelty):
The authors claim “previous work did not consider the time aspect and how video and dynamics (e.g., motion) modelling in deep networks compare to these biological neural systems”. This is incorrect. Several previous works [i-iv] have investigated modeling temporal aspects of videos and comparing it to brain responses. These works have been completely overlooked and not cited. Further seminal works on encoding and neural system identification from Jack Gallant’s group and Marcel Van Gerven’s group are not cited.

Author response:
**1.** We incorporate the requested works in the revised submission Sec. 2 (in red). Note that the last two works mentioned are not published and are concurrent. These works do not provide a large-scale study of different deep video understanding models. We study around 30 video understanding models and include new and comprehensive aspects in our study missing from previous literature: e.g., single/two-stream, fully/self supervised and transformer/convolutional. We conduct a system identification of image vs. video understanding models through simulated experiments which is missing from the literature.

Reviewer comment (Experiments):
When comparing convolutional vs. transformer or self-supervised in Figure 2b,c; did you consider both video and image models?

Author response:
2. Kindly check the supplementary (Appendix A.2) for the comparison you requested. The statistical significance test when the group (models family) has only two members can become less representative of the family. That is why we opted to use more models. However, since we already provide the results requested on the sub-family it should be sufficient as a response to this point.

Reviewer comment (Experiments):
If both the video and image models are considered for comparison, what was the reasoning? If video models better predict brain activity, doesn’t it make sense to restrict only to video models? If both the video and image models are considered, do you see the same pattern for video and image family of models?

Author response:
**2.1.1.** For Fig.2 b,c, yes we considered all models, to incorporate more models in the statistical significance test for a stable result. Nonetheless, we add Appendix A.2 results showing the consistency of our conclusions in video understanding only excluding image models in Fig. S1.

Reviewer comment (Evaluation):
In Figure 3, it is not clear whether the results are statistically significant or not.

Author response:
**3.** Figure 3(a, b, c) statistical significant results are added and re-iterated in Appendix A.3 Tables S5-7 confirming in most regions there is statistically significant comparing single and two-stream models (Fig. 3a & Tab. S5), comparing self and fully supervised ones (Fig. 3b & Tab. S6), and comparing pairs of video understanding models  (Fig. 3c & Tab. S7).

Reviewer comment (Evaluation):
In Figure 1a (MViT-B) and 1c (I3D R-50), it can be clearly seen that variance in regression score by video models is quite high compared to image models, suggesting some models are better predictors and some are worse. Which ones are worse/best predictors and why? This answer is important to understand how temporal information in video should be modelled.

Author response:
**4.1.** Tab. S1-3 were included for Fig. 1a-c, highlighting the best and worst source models. The alignment between the source and target model architecture and supervision signal is evident, except for the MViT target model, clarified in Appendix A.2 showing consistency to MViT performance in real experiments.

**The best source models for I3D target model are shown below:**

|    | Max (Video)   | Max (Image) |
|----|---------------|-------------|
| B1 | SlowFast-R101 | ResNet-18   |
| B2 | C2D           | ResNet-50   |
| B3 | C2D           | ResNet-50   |
| B4 | C2D           | ResNet-18   |
| B5 | R(2+1)D       | ResNet-50   |
| B6 | C2D           | ResNet-18   |
| B7 | SlowFast-R101 | ResNet-50   |

**The best source models for MViT-B-16x4 target model are shown below:**

|    | Max (Video) | Max (Image) |
|----|-------------|-------------|
| B1 | MViT-B-32x3 | ResNet-50   |
| B2 | MViT-B-32x3 | ResNet-18   |
| B3 | MViT-B-32x3 | ResNet-50   |
| B4 | MViT-B-32x3 | ResNet-50   |
| B5 | MViT-B-32x3 | ResNet-50   |

Reviewer comment (Presentation):
Similar variance can be observed in Figure 2a-c as well, raising the question why these models are grouped into one family. A simple classification such as transformer vs conv or self-supervised vs supervised is not helpful when there is so much variance within a family of models.

Author response:
**4.2.** The use of “models families” is established in previous literature, Han et. al. (2023), where they studied convolutional vs. transformer models. Computing statistical significance among populations of models is standard practice, regardless of the degree of variance within the population.

Reviewer comment (Presentation):
From Figure 2a: three video models predict brain responses similar to or worse than image models, while others predict better. Which are similar to image models and which are better is not answered.

Author response:
**4.2.1.** The 3 video models performing worse and comparable to image models are the self supervised ones (stMAE and omniMAE variants). The best video models are the SlowFast variants (two-stream). Both results align with findings in Fig. 3.

Reviewer comment (Evaluation):
From Figure 2b: some transformer models predict as well as conv models.

Author response:
**4.2.2.**  Fig. S2 compares slow (conv) and timesformer (transformer) models, trained using two different datasets. Evidently, within the early-mid regions of the brain, the convolutional model consistently yields higher scores. This reaffirms our finding that convolutional models excel in capturing orientation and frequencies due to their early local connections.

Reviewer comment (Evaluation):
The above conclusions are quite weak and less helpful and informative for readers without a deeper analysis.

Author response:
**4.2.3.** Strong findings are shown in Sec.4.5 and Appendix A.1,2 and 4 provide deeper analysis showing strong alignment to the results in the main submission.

Reviewer comment (Evaluation):
Overall, the paper contains multiple results with unclear findings.

Author response:
**5.** Findings summary and emphasis added to the Abstract and Sec. 4.5 and the main response for all reviewers.