The complex multi-stage architecture of cortical visual pathways provides the neural basis for efficient visual object recognition in humans. However, the stage-wise computations therein remain poorly understood. Here, we compared temporal (magnetoencephalography) and spatial (functional MRI) visual brain representations with representations in an artificial deep neural network (DNN) tuned to the statistics of real-world visual recognition. We showed that the DNN captured the stages of human visual processing in both time and space from early visual areas towards the dorsal and ventral streams. Further investigation of crucial DNN parameters revealed that while model architecture was important, training on real-world categorization was necessary to enforce spatio-temporal hierarchical relationships with the brain. Together our results provide an algorithmically informed view on the spatio-temporal dynamics of visual object recognition in the human visual brain. Visual object recognition in humans is mediated by complex multi-stage processing of visual information emerging rapidly in a distributed network of cortical regions [1] [2] [3] [4] [5] [6] [7] . Understanding visual object recognition in cortex thus requires a quantitative model that captures the complexity of the underlying spatio-temporal dynamics [8] [9] [10] . A major impediment in creating such a model is the highly nonlinear and sparse nature of neural tuning properties in mid-and high-level visual areas [11] [12] [13] that is difficult to capture experimentally, and thus unknown. Previous approaches to modeling object recognition in cortex relied on extrapolation of principles from well understood lower visual areas such as V1 8,9 and strong manual intervention, achieving only modest task performance compared to humans. Here we take an alternative route, constructing and comparing against brain signals a visual computational model based on deep neural networks (DNNs) [14] [15] [16] , i.e. computer vision models in which model neuron tuning properties are set by supervised learning without manual intervention 14, 17 . DNNs are the best performing models on computer vision object recognition benchmarks and yield human performance levels on object categorization 18, 19 . We used a tripartite strategy to reveal the spatio-temporal processing cascade underlying human visual object recognition by DNN model comparisons. First, as object recognition is a process rapidly unfolding over time 3, [20] [21] [22] , we compared DNN visual representations to millisecond resolved magnetoencephalography (MEG) brain data. Our results delineate an ordered relationship between the stages of processing in a DNN and the time course with which object representations emerge in the human brain 23 . Second, as object recognition recruits a multitude of distributed brain regions, a full account of object recognition needs to go beyond the analysis of a few pre-defined brain regions [24] [25] [26] [27] [28] , determining the relationship between DNNs and the whole brain. Using a spatially unbiased approach, we revealed a hierarchical relationship between DNNs and the processing cascade of both the ventral and dorsal visual pathway.
In REF , the authors are interested in stage-wise computation.
Comparison of deep neural networks to spatio-temporal cortical dynamics of human visual object recognition reveals hierarchical correspondence
Scientific Reports
Scientific Reports
Computer Science Medicine