Paper: Deep Models modelled after human brain boost performance in action classification (ICLR-2024)

Reviewer comment (Experiments):
The bottom line is that highly uncontrolled and bad datasets lead to spurious and uninterpretable results. This is the main challenge throughout.

Author response:
Thank you for your review. Naturalistic datasets such as the HAA500 used in this manuscript are uncontrolled, however this does not invalidate our conclusions. The use of naturalistic datasets is very common in the field (see for example the Kinetics dataset). The use of such datasets makes it possible to train artificial neural networks to generalize across sources of variation that are encountered in the wild. Importantly, we used the same dataset to train both Baseline and DomainNet models. Therefore, differences between the two models cannot be due to differences in the datasets.The only difference between the training schemes was the architecture of the networks. As such, our results can only be explained by the difference between the architectures, with the two-stream architecture that separates body and background being advantageous over the single-stream architecture for the task of action classification.

Reviewer comment (Evaluation):
The fact that humans can identify actions purely from the background frames with over 0.7 accuracy shows that background is a major confounding factor.

Author response:
2) That is correct, and was the motivation for the paper. This problem is not unique to HAA500, actions tend to be associated with particular backgrounds in naturalistic experience (e.g., “playing basketball” will most often happen in a basketball court). Therefore, it is important to build models that are robust to this. Results demonstrate that DomainNet was more robust to the confounding effect of background compared to the Baseline model.

Reviewer comment (Experiments):
The fact that humans can identify actions purely from the background frames with over 0.7 accuracy shows that time is not needed in such an easy task.

Author response:
3) “Time is not needed in such an easy task”. In contrast with this statement, our results show that including dynamic, time-dependent information improved the accuracy on our task compared to using only static information (see Table 1). Specifically, when processing the original videos (ORIG), DomainNet(frames+flows) – that uses both static frames and optic flow (which are time-dependent) – obtained an accuracy of 75%, while DomainNet(frames) – that does not use time-dependent information – only obtained an accuracy of 66.25%. This was also the case for processing videos showing only the body: DomainNet(frames+flows) obtained an accuracy of 73.75%, while DomainNet(frames) obtained an accuracy of only 62.5%. These results show that time-dependent information (optic flow) was indeed helpful to perform the task, and that it led to improvements in action recognition accuracy.

Reviewer comment (Experiments):
It would be great to present actual results on how well Yolo v8 separates body and background.

Author response:
“It would be great to present actual results on how well Yolo v8 separates body and background.”  

These are available in figure S1.

Reviewer comment (Experiments):
A key aspect of action recognition is likely to be dynamics and time, which is not studied here.

Author response:
Dynamics and time do indeed contribute to action information and we did study the effects of both static and static plus dynamic information using optic flow. Those results are reported in Table 1 and are discussed in the Results section. Even when dynamic information is available, the model additionally benefits from separating background and body information (52.50% accuracy vs 75% accuracy), showing that dynamics alone does not account for our findings.

Reviewer comment (Experiments):
Within the study of action recognition from frames, it would be useful to use rigorous datasets.

Author response:
Thank you for your review. Naturalistic datasets such as the HAA500 used in this manuscript are uncontrolled, however this does not invalidate our conclusions. The use of naturalistic datasets is very common in the field (see for example the Kinetics dataset). The use of such datasets makes it possible to train artificial neural networks to generalize across sources of variation that are encountered in the wild. Importantly, we used the same dataset to train both Baseline and DomainNet models. Therefore, differences between the two models cannot be due to differences in the datasets.The only difference between the training schemes was the architecture of the networks. As such, our results can only be explained by the difference between the architectures, with the two-stream architecture that separates body and background being advantageous over the single-stream architecture for the task of action classification.