Signiﬁcance of feedforward architectural differences between the ventral visual stream and DenseNet
Abstract
There are many differences between convolutional networks and the ventral visualstreams of primates.
For example, standard convolutional networks lack recurrentand lateral connections, cell dynamics, etc.
However, their feedforward architectures are somewhat similar to the ventral stream, and warrant a more detailedcomparison.
A recent study found that the feedforward architecture of the visualcortex could be closely approximated as a convolutional network, but the resulting architecture differed from widely used deep networks in several ways.
Thesame study also found, somewhat surprisingly, that training the ventral streamof this network for object recognition resulted in poor performance.
This paperexamines the performance of this network in more detail.
In particular, I made anumber of changes to the ventral-stream-based architecture, to make it more likea DenseNet, and tested performance at each step.
I chose DenseNet because ithas a high BrainScore, and because it has some cortex-like architectural featuressuch as large in-degrees and long skip connections.
Most of the changes (whichmade the cortex-like network more like DenseNet) improved performance.
Furtherwork is needed to better understand these results.
One possibility is that detailsof the ventral-stream architecture may be ill-suited to feedforward computation,simple processing units, and/or backpropagation, which could suggest differencesbetween the way high-performance deep networks and the brain approach coreobject recognition.
Introduction
Deep convolutional networks trained for object recognition have a number of things in common withthe primate ventral stream, including architectural similarities (e g. spatially localized connections),comparable performance, and closely related representations [1].
There are also many differences,notably related to cell and network dynamics and development of representations over time.
However, feedforward convolutional networks appear to be promising abstract models of core objectrecognition.
Can convolutional networks be used to model the signals and transformations of coreobject recognition in detail?
A recent study developed a data-driven convolutional network that wasconsistent with many architectural details of primate visual cortex [2].
However, that study reportedperformance far below state of the art when the ventral stream was trained on CIFAR-10.
Here Iconﬁrm this result, and investigate whether there is a speciﬁc feature of the cortex-like architecturethat limits performance. There is not.
Rather, multiple architectural details are responsible for theperformance gap.
This may suggest that the feedforward architecture of the ventral stream is notparticularly well suited to core object recognition in the idealized context of standard deep-networktraining.
[Figure]
V
V V2pale
[Figure]
Ventral-Stream Network
I experimented with a simpliﬁed sub-network of the macaque single-hemisphere visual cortexarchitecture of [2].
This sub-network spans much of the ventral stream (Figure 1, including V1,
V2 (thin and pale stripes), V4, VOT, and PITv.
The retrograde-tracer measure, fraction labelledneurons extrinsic to the injection site (FLNe), was used to identify and omit the weakest inter-areaconnections (FLNe<0.02).
To make training tractable, the number of neurons in each populationwas reduced by a factor of ten.
The full visual resolution was taken to be 500x500 pixels, andsmaller images from ImageNet and CIFAR-10 were taken to activate the central part of the visualﬁeld.
L2/3 of PITv was connected to a ﬁnal convolutional layer with 64 channels, and then eithertwo 512-unit fully-connected layers (for CIFAR-10) or one 2048-unit fully-connected layer (for
ImageNet), followed by a softmax classiﬁer layer.
Each linear layer in the model was followed bybatch norm and ReLU layers.
The code for this model can be found at github.com/bptripp/calc.
Results
[2] reported validation accuracy of 79% on CIFAR-10, for a similar ventral-stream sub-network thatomitted connections with FLNe<0.15, and was trained for 50 epochs with the Adam update algorithm.
In the present study, networks were trained for 300 epochs, using SGD with momentum 0.9, startingwith learning rate 0.1 and reducing it by 10x every 100 epochs.
This resulted in validation accuracyof 84.59%.
A standard DenseNet (github.com/kuangliu/pytorch-cifar) was trained using the sameprocedure, resulting in validation accuracy of 95.36%.
To understand the basis of this performance gap, I created hybrid networks, with features of both theventral-stream network (VSN) and DenseNet.
The VSN has a wide range of kernel sizes, optimizedto ﬁll realistic receptive ﬁeld sizes.
In the ﬁrst hybrid (H1), all kernel sizes of the VSN were setto 3x3.
The VSN also has a wide range of sparsity, with some connections consisting mostly ofzeros.
In the second hybrid network (H2), in addition to using 3x3 kernels, I eliminated pixel-wisesparsity, and limited channel-wise sparsity so that at least half of the input channels were used ineach connection.
Thirdly (H3), I replaced each layer with a two-layer bottleneck module, speciﬁcallya 1x1-kernel layer followed by 3x3 layer with four times fewer channels.
The number of channelsin the second layer was 0.3x the original number of channels.
Fourthly (H4), I replaced the LGNlayers with a DenseNet-like 24-channel input stage, eliminated remaining sparsity, and organized theremaining layers into three blocks with DenseNet-like transition layers between them.
Each blockgrouped layers with the same spatial resolution.
Connections that spanned multiple blocks werereplaced by connections with the transition layers.
Fifthly (H5), I set the numbers of channels ineach module within a given block equal to the block-wise average, emulating the block-wise constant“growth rate” parameter of DenseNet.
Finally (H6), I added any missing connections within eachblock, introducing many connections that do not exist in the brain. These changes are summarized in
Table 1. [Table] [Table] Most of these modiﬁcations improved performance, when applied cumulatively to make the ventralstream increasingly similar to a DenseNet (Table 2, left result column).
Making the growth rateuniform resulted in a slight performance drop.
Replacing simple layers with bottleneck modulessharply impaired performance, including on training data (83% correct), although performancerecovered in the next variation with transition layers.
The bottleneck experiment was repeated withslight variations (i.e. a different training schedule, growth rate 0.2x rather than 0.3x the originalnumber of channels, and a DenseNet-like input stage), and similar performance impairment wasfound.
With these exceptions, hybrid networks performed better the more they had in common with DenseNet.
Similar modiﬁcations were also applied one at a time (Table 2, right result column).
Improvementdue to dense kernels alone was less pronounced than in conjunction with 3x3 kernels.
To approximate
DenseNet blocks (H4), most of the connections between layers with different resolutions wereeliminated. However, some layers only had incoming connections from higher-resolution layers. For these layers, the single inbound connection with the largest number of parameters was retained. This resulted in an approximately block-wise structure, without introducing connections that do notexist in the brain.
This change strongly impaired performance.
To approximate DenseNet’s uniformgrowth rate within each block (H5), the number of channels in each layer was changed to the averagenumber of channels among layers with the same resolution.
This improved performance, but theredistribution of channels also caused a large increase in the number of parameters in the network.
Repeating this experiment while controlling for the number of parameters reduced validation accuracyslightly below baseline (83.71%).
Overall, each of these changes alone had little beneﬁt.
As discussed in [2], connection sparsity in the VSN is qualitatively distinct from various forms ofsparsity in deep learning.
I further explored the effect of sparsity on performance.
Each connection inthe VSN has two sparsity parameters, σ (pixel-wise probability of non-zero weight), and c (channelwise probability of non-zero weights); the probability that a weight is non-zero is σc .
I chose non-zero [Figure] channels at random, and non-zero weights using SNIP [3].
The sparsity parameters vary over severalorders of magnitude in different connections.
I trained variations of the VSN in which I scaledthe logarithms of these parameters (e.g. σ = .
0001 with a scale factor of . 5 would become .
Unsurprisingly, reductions in sparsity tended to improve performance (Figure 2).
So, physiologicallyrealistic connection sparsity seems to be counter-productive in these deep-network models of coreobject recognition.
A network with full sparsity, but with numbers of channels increased so that thetotal number of parameters matched that of a network with scale factors of 0.6, performed nearly aswell as the network with scale factors of 0.6 (squares in Figure 2.
This suggests that sparsity affectsperformance largely via the number of network parameters.
Representations in DenseNet are closely related to those in primate ventral stream [1].
However, I wondered whether, despite poorer performance of the VSN, it might have even more realisticrepresentations due to its architecture.
To explore this issue, I performed approximations of severalexperiments from the primate literature, with both a DenseNet-161 model pretrained on ImageNet(from torchvision.models), and a VSN trained on ImageNet, with c-scale= σ -scale=0.5.
The VSNwas trained with SGD for 90 epochs, with learning rate 0.1, decreasing by 10x every 30 epochs;best validation accuracy 63.14%(top-1)/84.97%(top-5).
I recorded from the second-last ReLU layerthat followed a convolutional layer in each network, and compared with published recordings frommacaque inferotemporal cortex.
The two networks had somewhat different representations, butoverall these experiments did not reveal either to be much more realistic (Figure 3).
The representation results were sensitive to implementation details such as whether stimulus imageswere normalized or cropped (center 224x224 pixels taken from 256x256 images). It is not clearwhether the stimulus images’ standard deviations should be normalized (as is usually done withnatural images). This is because they consist, as in the primate experiments, of small content on alarger neutral background. In the size tuning experiments, normalizing each image would give thelarger stimuli systematically lower contrast, while in the other experiments, normalizing would givethin objects (e.g. a hammer) higher local contrast.
The results in Figure 3 are without normalization orcropping, but other choices produced somewhat different results.
Normalization tended to narrow sizetuning bandwidth in both VSN and DenseNet.
The RDMs from normalized stimuli had correlationsof r = 0 . 62 (VSN) and r = .
68 (DenseNet) with those in the ﬁgure.
The RDMs from normalizedand cropped stimuli had correlations of r = 0 .
34 (both VSN and DenseNet) with those in the ﬁgure.
Sensitivity to such factors complicates comparison with macaque data.  Discussion Architectural details of the ventral-stream model tended to impair core object recognition in ourdeep-learning setting.
This may indicate inaccuracies in the architecture of [2].
Alternatively, it maysuggest that the feedforward ventral-stream architecture is not well suited for such idealized settings,or even that it is not optimized speciﬁcally for core object recognition. The ventral stream architecturemight be better suited for more diverse or naturalistic tasks, or more biological mechanisms. Forexample, biological schemes such as predictive coding may use sparse feedforward connections more
[Figure]
efﬁciently, or the connection pattern of the ventral stream may be better suited to extracting certainfeature combinations in unsupervised learning than to communicating gradients through many layers.