A UNIFIED THEORY OF EARLY VISUAL REPRESENTATIONS FROM RETINA TO CORTEX THROUGH ANATOMICALLY CONSTRAINED DEEP CNN S
A BSTRACT
The vertebrate visual system is hierarchically organized to process visual information in successive stages.
Neural representations vary drastically across the ﬁrst stages of visual processing: at the output of the retina, ganglion cell receptive ﬁelds (RFs) exhibit a clear antagonistic center-surround structure, whereas in the primary visual cortex (V1), typical RFs are sharply tuned to a precise orientation.
There is currently no uniﬁed theory explaining these differences in representations across layers.
Here, using a deep convolutional neural network trained on image recognition as a model of the visual system, we show that such differences in representation can emerge as a direct consequence of different neural resource constraints on the retinal and cortical networks, and for the ﬁrst time we ﬁnd a single model from which both geometries spontaneously emerge at the appropriate stages of visual processing.
The key constraint is a reduced number of neurons at the retinal output, consistent with the anatomy of the optic nerve as a stringent bottleneck.
Second, we ﬁnd that, for simple downstream cortical networks, visual representations at the retinal output emerge as nonlinear and lossy feature detectors, whereas they emerge as linear and faithful encoders of the visual scene for more complex cortical networks.
This result predicts that the retinas of small vertebrates (e.g. salamander, frog) should perform sophisticated nonlinear computations, extracting features directly relevant to behavior, whereas retinas of large animals such as primates should mostly encode the visual scene linearly and respond to a much broader range of stimuli.
These predictions could reconcile the two seemingly incompatible views of the retina as either performing feature extraction or efﬁcient coding of natural scenes, by suggesting that all vertebrates lie on a spectrum between these two objectives, depending on the degree of neural resources allocated to their visual system.
I NTRODUCTION
Why did natural selection shape our visual representations to be the way they are?
Traditionally, the properties of the early visual system have been explained with theories of efﬁcient coding , which are based on the premise that the neural representations are optimal at preserving information about the visual scene, under a set of metabolic constraints such as total ﬁring rate or total number of synapses.
These theories can successfully account for the antagonistic center-surround structure of receptive ﬁelds (RFs) found in the retina (Atick & Redlich, 1990; 1992; Vincent & Baddeley, 2003;
Karklin & Simoncelli, 2011; Doi et al., 2012), as well as for the oriented structure of RFs found in the primary visual cortex V1 (Olshausen & Field, 1996; 1997; Bell & Sejnowski, 1997).
However, a number of properties of the early visual system remain unexplained.
First, it is unclear why RF geometries would be so different in the retina and V1.
A study (Vincent et al., 2005) has proposed that both representations are optimal at preserving visual information under different metabolic constraints: a constraint on total number of synapses for the retina, and one on total ﬁring rate in V1.
However, it is unclear why the two systems would be optimized for these two different objectives.
Second, there is a great diversity of ganglion cell types at the output the retina (Gollisch & Meister, 2010), with each cell type tiling the entire visual ﬁeld and performing a speciﬁc computation.
Interestingly, some of these types perform a highly nonlinear computation, extracting speciﬁc, behaviorally-relevant cues from the visual scene (e.g. direction-selective cells, objectmotion-selective cells), whereas other types are better approximated by a quasi-linear model, and respond to a broad range of stimuli (e.g. midget cells in the primate (Roska & Meister, 2014) and quasi-linear pixel-encoders in the mouse (Johnson et al., 2018)).
Intriguingly, although quasi-linear and more nonlinear types exist in species of all sizes (e.g. primate parasol cells are nonlinear (Crook et al., 2008)), the proportion of cells performing a rather linear encoding versus a nonlinear feature detection seems to vary across species.
For example, the most common ganglion cell type in the primate retina is fairly well approximated by a quasi-linear pixel-encoder (midget cells, 50% of all cells and > 95% in the central retina (Roska & Meister, 2014; Dacey, 2004)), whereas the most common cell type in mouse acts as a speciﬁc feature detector, thought to serve as an alarm system for overhead predators (W3 cells, 13% of all ganglion cells (Zhang et al., 2012)).
Again, theories of efﬁcient coding have not been able to account for this diversity of computations found across cell types and across species.
The limitations of current efﬁcient coding theories might reside in the simplistic assumption that the objective is to simply relay indiscriminately all visual information to the next stages of processing.
Indeed, the ultimate goal of the visual system is to extract meaningful features from the visual scene in order to produce an adequate behavioral response, not necessarily to faithfully encode it.
A recent line of work has proposed using the information bottleneck framework as a way to move beyond the simplistic objective of information preservation towards more realistic objectives (Chalk et al., 2016; 2018).
Another study has shown that by changing the objective from efﬁciently encoding the present to efﬁciently encoding the future (predictive coding), one could better account for the spatio-temporal RFs of V1 cells (Singer et al., 2018).
Although promising, these approaches were limited to the study of a single layer of neurons, and they did not answer the aforementioned questions about cross-layer or cross-species differences.
On the other hand, deep convolutional networks have proven to be accurate models of the visual system, whether they are trained directly on reproducing neural activity (McIntosh et al., 2016;
Cadena et al., 2017), or on a behaviorally relevant task (Yamins et al., 2014; Eberhardt et al., 2016; Cadena et al., 2017), but they have not yet been used to study the visual system through the lens of efﬁcient coding theories.
In this study, we trained deep convolutional neural networks on image recognition (CIFAR-10,
Krizhevsky (2009)) and varied their architectures to explore the sets of constraints that could have shaped vertebrates’ early visual representations through natural selection.
We modeled the visual system with a series of two convolutional networks, one corresponding to the retina and one downstream network corresponding to the ventral visual system in the brain.
By varying the architecture of these networks, we ﬁrst found that a reduction in the number of neurons at the retinal output – corresponding to a realistic physical constraint on the number of ﬁbers in the optic nerve – accounted simultaneously for the emergence of center-surround RFs in our model of the retina, and for the emergence of oriented receptive ﬁelds in the primary visual relay of the brain.
Second, we found that the degree of neural resources allocated to visual cortices in our model drastically reshaped retinal representations.
Given a deep visual cortex, the retinal processing emerged as quasi-linear and retained substantial information about the visual scene.
In contrast, for a shallow cortex, the retinal processing emerged as nonlinear and more information-lossy, but was better at extracting features relevant to the object classiﬁcation task.
These observations make testable predictions on the qualitative differences that should be found in retinal representations across species, and could reconcile the seemingly incompatible theories of retinal processing as either performing efﬁcient encoding or feature detection.
F RAMEWORK : A DEEP C ONVOLUTIONAL N EURAL NETWORK MODEL OF THE VISUAL SYSTEM
The retinal architecture is strongly conserved across species (Masland, 2001), and consists of three layers of feed-forward convolutional neurons (photoreceptors, bipolar cells, ganglion cells) and two layers of inhibitory interneurons (horizontal, amacrine cells).
However, we chose to model the retina as a convolutional neural network (LeCun et al., 2015) with only two layers (ﬁg. 1A).
Indeed the retinal response of many species to complex stimuli has been modeled successfully with only one or two-layer models (Deny et al., 2017; Maheswaranathan et al., 2018; Gollisch & Meister, 2010), with some rare exceptions of models requiring more layers (McIntosh et al., 2016).
We refer to this network as the retina-net.
In our simulations, we varied the number of neurons in the second layer of the retina-net, which is the output of the retina, corresponding to the physical bottleneck of the optic nerve conveying all the visual information to the brain (ﬁg. 1B).
We modeled the ventral visual system – the system associated with object recognition in the brain (Hubel, 1995) – as a convolutional neural network taking its inputs from the retina-net (ﬁg. 1A).
We varied the neural resources allocated to the ventral visual system network (VVS-net) by changing the number of layers it is composed of (ﬁg. 1B).
We trained the neural network composed of the retina-net and VVS-net end-to-end on an object classiﬁcation task (CIFAR-10, ﬁg.
1A-B-C).
Even though the visual system does much more than just classify objects in natural images, this objective is already much more complex and biologically realistic than the one used in previous studies of efﬁcient coding, namely preserving all information about the visual scene.
Moreover, we are encouraged by the fact that previous studies using this objective have found a good agreement between neural activity in artiﬁcial and biological visual networks (Yamins et al., 2014; Cadena et al., 2017).
More speciﬁcally, we trained a convolutional neural network on a grayscale version of the standard
CIFAR-10 dataset for image classiﬁcation.
The retina-net consisted of two convolutional layers with 32 channels and N BN channels respectively, and with ReLU nonlinearities at each layer.
The VVSnet consisted of a varying number D V V S of convolutional layers with 32 channels followed by two fully connected layers (the ﬁrst one with 1024 neurons and the second one with 10 neurons mapping to the 10 object categories), with ReLU nonlinearities at each layer and a softmax nonlinearity at the last layer.
The full system encompassing the retina-net and VVS-net thus had 32 !
N BN !
channels respectively, where we varied the retinal bottleneck width, N BN , as well as the number D V V S of convolutional brain layers (not counting the fully connected layers).
In each convolutional layer, we used 9x9 convolutional ﬁlters with a stride of 1 at each step.
The large ﬁlter size was chosen to give the network ﬂexibility in determining the optimal ﬁlter arrangement.
We trained our network with the RMSProp optimizer for 20 epochs on the training set with batches of size 32.
All optimizations were performed using Keras and TensorFlow.
For all results presented, we
A “ H o r s e ”
V
Fully Connected
VVS - ne t
R e t i na l
B o tt l ene ck
S i z e
B
C “ H o r s e ” “ H o r s e ” “ H o r s e ” “ H o r s e ” . . .
D    R e t i na - ne t
A
[Figure]
[Figure]
[Figure]
[Figure]
VVS-net depth
[Figure]
R e t i na l
B o tt l ene ck
S i z e
Figure1Final tested statistical signiﬁcance by training 10 identical networks with different random initializations of weights and biases taken from a Glorot-uniform distribution (Glorot & Bengio, 2010). After training, we determined the linear approximation of RFs of each convolutional channel of the network in each layer. This was achieved by computing the gradient of the activation of that channel with respect to a blank image. This gradient map gives a ﬁrst-order approximation of the image pattern that maximally activates the cells in the channel of interest.
In the limit of small noise variance, this computation is mathematically equivalent to measuring the cell’s spike-triggered average in response to a perturbative white-noise stimulus (Koelling & Nykamp, 2008; Schwartz et al., 2006), a commonly used method for determining receptive ﬁelds in experimental biology (Chichilnisky, 2001).
This equivalence allowed us to compare directly the geometries of RFs experimentally measured in biological networks with the ones found in our models.
The test accuracy of our neural network model of the visual system at the recognition task increased both with the number of channels in the retinal bottleneck, and with the number of layers in the
VVS-net (ﬁg. 1D), conﬁrming that we were in a regime where the restrictions on neural resources in the VVS-net and at the output of the retina were critical to the ability of the network to perform the task.
A UNIFIED MODEL FOR CENTER - SURROUND RF S IN THE RETINA AND ORIENTED RF S IN V
Here we investigate the effects of a dimensionality bottleneck at the retinal output on early visual representations in our model of the visual system
. 3. A DIMENSIONALITY BOTTLENECK AT THE RETINAL OUTPUT YIELDS THE EXPECTED REPRESENTATIONS IN RETINA AND V
When reducing the number of neurons at the output of the retina we found that RFs with antagonistic center and surround emerged.
For N BN = 32 , our control setting with no bottleneck at the retinal output, we observed mostly oriented receptive ﬁelds in the second layer of the network (ﬁg. 2A).
For N BN = 4 , 2 , and 1 , we observed center-surround receptive ﬁelds in the second layer of the network and mostly oriented receptive ﬁelds in the third layer, which is the ﬁrst layer of the ventral visual system in our model (ﬁg. 2B).
We quantiﬁed these results in App.
A.
The RF geometries did not depend qualitatively on the VVS-net depth D V V S (results shown for D V V S = 2 ), except for the shallowest VVS-net tested ( D V V S = 0 , no convolutional layer, and thus no dimensionality expansion), for which the shape of emergent retinal RFs were variable across trials and difﬁcult to interpret.
These results are in good agreement with the organization of the biological visual system, where retinal RFs are center-surround and most downstream RFs in primary visual cortex (V1) are sharply oriented (Hubel, 1995), suggesting that the dimensionality bottleneck at the output of the retina is sufﬁcient to explain these differences in representations.
It is worth noting that for both conditions (bottleneck and no bottleneck), the RFs of downstream layers in the VVS-net after the ﬁrst layer exhibited complex shapes that were neither clearly oriented, nor circular, and the RFs in the ﬁrst layer of the retina did not appear to have any well-deﬁned structure (data not shown).
We then tested in our model the hypothesis of Hubel and Wiesel concerning how center-surround cells are pooled to give rise to oriented RFs in V1 (Hubel, 1995).
We found that orientation-selective neurons in the VVS-net typically draw primarily from center-surround neurons in the retina-net that are aligned with the direction of the edge, with positive or negative weights corresponding to whether the polarity (light-selective / dark-selective) of the two neurons are consistent or inconsistent (ﬁg.
2C, and App.
A for a quantiﬁcation).
These qualitative results are in good agreement with Hubel and Wiesel’s hypothesis.
Of course, this hypothesis remains to be tested in the real brain, since there is no evidence that the micro-circuitry of the brain matches that of our simulation.
In the visual system of mammals, the main relay of visual information taking its input from the retina is the LGN (thalamus), which has center-surround RFs and a similar total number of neurons as the retinal output (Hubel, 1995).
We created a network reﬂecting this architecture by having two lowdimensionality layers in a row instead of just one (ﬁg. 2C).
After training, we found center-surround
RFs in the two layers with a bottleneck (retinal output and LGN), and oriented RFs in the next layer, corresponding to the primary visual cortex (V1).
These results suggest that center-surround representations remain advantageous as long as the dimensionality of the representation remains low, and hence dimensionality expansion seems to be the crucial factor explaining the qualitative change of RFs found between LGN and V1.
It is an interesting question to ask whether neurons in our model of the VVS are more similar to simple or complex cells (Hubel, 1995).
To test this, we performed a one-step gradient ascent on the neural activity of VVS neurons with respect to the image, starting from several random initial images (App. B).
If the neurons were acting as simple cells (i.e. are approximately linear in the stimulus), we would expect all optimized stimuli to converge to the same preferred stimulus.
On the other hand, if the cells were complex (i.e. OR function between several preferred stimuli), we would expect the emergent preferred stimuli to depend on the exact initialization.
Interestingly, we found that most neurons in the ﬁrst layer of the VVS-net behaved as simple cells, whereas most neurons in the second layer of the VVS-net behaved as complex cells.
Note that in biology, both simple and complex cells are found in V1.
These results expose the fact that anatomical regions of visual cortex involve multiple nonlinearities and hence may map onto more than one layer of our simple model.
Indeed, V1 itself is a multilayered cortical column, with LGN inputs coming in to layer 4, and layer 4 projecting to layers 2 and 3 (Hubel, 1995).
Simple cells are predominantly found in layer 4 and complex cells are predominantly found in layers 2 and 3.
These observations bolster the interpretation that biological V1 may correspond to multiple layers in our model.
Local divisive normalization (i.e. local gain control) is an ubiquitous source of nonlinearity in the visual system (Geisler & Albrecht, 1992; Heeger, 1992; Deny et al., 2017).
We thus tested the robustness of our main result to a more realistic model of the visual system with local normalization, by adding it at every layer of the network (App. C).
We found that receptive ﬁelds still emerged as center-surround in the retina-net, and as oriented in our model of V1.
We note that the local normalization slightly degraded the performance of the network on the task for all parameter settings we tried. Figure2Final [Figure] t
[Figure]
R e t i na - ne t [Figure] B E
F “Horse” “Horse”
D ... t-SNE
R e t i na - ne t “ L G N ”
VVS - ne t V
V
V       C 3. E MERGENCE OF ON AND OFF POPULATIONS OF CENTER - SURROUND CELLS IN THE RETINA We then veriﬁed that the emergence of center-surround RFs in the retina-net is a consequence of reducing the number of neurons at the retinal output, not of reducing the number of channels, our model’s equivalent of biological retinal cell types.
In the retina, there exist 20-30 types of ganglion cells (Roska & Meister, 2014), each with a different and stereotyped receptive ﬁeld, density, polarity (i.e. ON or OFF), and nonlinearities.
Cells of each type tile the entire visual ﬁeld like a convolutional channel in our model, so there is a direct analogy between channels in our model and ganglion cell types in the retina.
In order to test whether the emergence of center-surround RFs depends on the number of types that we allow, or just on the number of neurons that we allow at the output of the retina (i.e. dimensionality bottleneck), we employed locally connected layers – equivalent to convolutional layers, but without parameter-tying between artiﬁcial neurons within a channel at different spatial locations.
In this manner, we can limit the number of neurons at the retinal output without imposing a constraint on the number of cell types.
Such a network contains too many parameters to be trained from scratch by gradient descent; to work around this, we trained the model stage-wise by ﬁrst training our convolutional control network ( N BN = 32 with parameter tying) and then substituting the untied retina-net in place of the ﬁrst two layers of the control model for the rest of training.
Even in the untied retina-net, in which each neuron is effectively its own channel, we found that center-surround RFs emerged (ﬁg. 2E), indicating that center-surround RFs are the network’s preferred strategy for passing information through a dimensionality bottleneck even when no constraint on the number of cell types is imposed.
We then found that the cells cluster in two distinct populations.
To demonstrate this, we measured their activations in response tonatural images, computed the ﬁrst 20 principal components of this 10000-dimensional space, and ran t-SNE to visualize the clustering of neuron types.
We found that two distinct clusters emerged, that corresponded visually to ON and OFF center-surround RFs (ﬁg. 2F).
We thus observe in our model the emergence of one of the most prominent axes of dichotomy of biological ganglion cell types, namely the classiﬁcation of cells in ON and OFF populations with RFs of opposite polarity.
R ETINAL REPRESENTATIONS ARE A FUNCTION OF THE NEURAL RESOURCES ALLOCATED TO THE VENTRAL VISUAL STREAM
To what extent are retinal representations in our model shaped by the degree of neural resources allocated to downstream processing?
To investigate this question, we studied the effects of varying the degree of neural resources in the VVS-net, on emergent visual representations in the retina-net.
4. T HE RETINA BECOMES MORE LINEAR AS BRAIN COMPLEXITY INCREASES
As we increased the number of layers in the VVS-net, the retinal computation became more linear (ﬁg. 3A), as measured by the ability of the raw image to linearly map onto the neural representation at the retinal output (see methods, and App.
F for a visualization of retinal representation as
VVS-net depth increases).
This observation is consistent with the current state of knowledge of the differences found in retinal representations across vertebrate species with different brain sizes.
The linearization of the retinal response with increased brain complexity was true for different values of bottleneck N BN .
However, when we did not use any bottleneck ( N BN = 32 ), the trend became non-monotonic, with a peak in linearity of the response when the VVS-net had 1 conv layer (data not shown).
Another interesting phenomenon to note is that linearity of the retinal response decreased as we increased the number of channels in the bottleneck, at any ﬁxed brain depth (ﬁg. 3A).
The two main sources of nonlinearity in the retina are thought to be the inner retinal rectiﬁcations (bipolar and amacrine cells, corresponding to the ﬁrst rectiﬁed layer in our model) and the ganglion cell rectiﬁcation (corresponding to the second rectiﬁed layer in our model).
As we decreased VVSnet depth, we observed that the retinal response became more nonlinear.
Is this increase in response nonlinearity due to the ﬁrst or second stage of nonlinearity in our retina-net?
To test this, we plotted the real response against the response predicted by a purely linear model for the most shallow and for the deepest VVS-nets tested (ﬁg. 3B).
If the linear prediction were inaccurate because of the ﬁrst stage of nonlinear processing in the retina-net, we would expect the points on the scatter plot to be scattered around the unit line.
If the prediction error were due to the second-stage of nonlinearity, we would expect the linear approximation to make incorrect negative predictions for inactive neurons.
In practice, we found that the prediction error of the linear model was partly explained by both stages of nonlinearity in the retina-net model, predicting that both inner retinal nonlinear processing and ganglion cell rectiﬁcations should be more pronounced in animals with fewer neural resources in their visual cortices.
4. T HE RETINAL REPRESENTATION IS THE RESULT OF A TRADE - OFF BETWEEN INFORMATION TRANSMISSION AND FEATURE EXTRACTION
Why would retinal representations be more linear when the subsequent ventral visual stream has more resources?
One hypothesis is that with a restricted number of neurons, the retina must trade-off between the two incentives of (1) compressing visual information in order to transmit it to downstream layers and (2) extracting nonlinear features from the scene to start disentangling the manifolds corresponding to different classes of objects (Chung et al., 2018a;b).
According to this hypothesis, when the VVS is shallow, the priority of the retina should be to work toward extracting relevant features.
When the VVS is deep, the priority of the retina should be to transmit as much visual information as possible for downstream processing.
We validated this hypothesis in two ways in our model.
First we showed that the retinal representation retained more information about the image as VVSnet complexity increased (ﬁg. 3C).
To estimate information retention, we trained a linear decoder
A
B D
E
[Table]
[Figure]
[Table]
[Figure]
Network Layer
R e t i na l R e s pon s e
L i nea r
i t y
I m age
C l a ss
S epa r ab illi t y
I m age
C l a ss
S epa r ab ili t y
R e c on s t r u c t i on
P e
r
f
o r m an c e
Predicted Activations V V S - d e p t h = V V S - d e p t h = C
R ea l A c t i v a t i on s N e
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
t i na l e s pon s e
L i nea r
i t y
I
[Table]
y I m age
C l a ss
S epa r ab illi t y
[Table]
A
[Table]
D
[Table]
[Figure] [Table]
t i na l R pon
[Table]
i nea r i t y
[Table]
I m age
C l a s s
S epa r ab illi t y
[Figure]
[-( C i f a r-) Retina Representation RawPixels 1 Channel2 Channels ChannelsRaw Pixels 1 Channel2 Channels
-][Table]
A
B
D
E
[Figure]
R e t i na l
R e s pon s e
L i nea r
i t y
I m age
C l a s s
S epa r ab illi t y
I m age
C l a s s
S epa r ab illi t y
R e c on s t r u c t i on
P e
r
f
o r m an c e
Predicted Actvations
V V S - d e p t h = V V S - d e p t h =
C
R ea l A c t i v a t
i on s
N e t w o r k
P e
r
f
o r m an c e
( C i f a r-
[Figure]
[-ChannelsRaw Pixels 1 Channel2 Channels
 ChannelsRaw Pixels 1 Channel32 Channels ***-][Figure] [Figure] (see methods) from the output of the retina to reconstruct the image and we measured the reconstruction error.
The reconstruction error provided a lower bound on the information that the retina retained about the stimulus (note that more information for reconstruction might be accessible by a nonlinear decoder). This result corroborated our hypothesis that, as the VVS-net becomes more complex, the retinal representation gets better at retaining visual information for further processing by the VVS-net.
Second, we found that different classes of objects of CIFAR-10 (e.g. trucks, frogs) were more linearly separable from the retina-net representation when the VVS-net was shallow than when it was deep (ﬁg. 3D).
To measure linear separability of manifolds, we trained a linear SVM decoder to separate all pairs of classes and evaluated the performance of the SVM classiﬁer on held-out images (see methods).
Moreover, we showed that a VVS-net consisting of two fully connected layers only (no convolutional layers) equipped and trained end-to-end with a retina with a tight bottleneck N BN = 1 (dimensionality of retinal output matches dimensionality of the input image) performed better at image recognition than the same VVS-net trained without a retina-net, taking raw images as input (ﬁg. 3E).
Both these results corroborate our hypothesis that retinas followed by a simple cortex performs meaningful feature extraction, whereas retinas followed by more complex visual cortices prioritize non-lossy encoding, postponing feature extraction to downstream layers that are better equipped to do it.
Next, we show that within a single network, each retinal channel is trading-off between (1) linearly transmitting visual information to the brain, and (2) extracting relevant features for the object classiﬁcation task.
For 10 instantiations of a network with a retinal bottleneck containing 4 channels, we represented the linearity of each of these 4 channels against the linear separability of object categories obtained from each of these representations.
We found, across all networks, a systematic negative correlation between linearity and linear separability across all 4 channels (App. D).
Again, this result strongly suggests that extracting features and transmitting visual information are indeed two competing goals shaping representations in our model of the retina.
In the case of the deepest VVS-nets tested, the retinal processing was quasi-linear for the tightest bottleneck (var.expl.
= 0.9,
N BN
= 1 , ﬁg. 3A).
One might take this result to suggest that the retinanet in such models does little more than copy image information.
However the very ﬁrst layer of the VVS-net after the retina disentangled classes (as measured by linear separability) almost as well as the second layer of a VVS-net without a retina (ﬁg. 3F), suggesting that the retinal representation, while only moderately linearly separable itself, is especially transformable into a representation with a high linear separability.
This result suggests that even when the retina-net is quasi-linear, it can still participate in extracting relevant features for downstream processing by the brain.
The increased separability allowed by the retinal pre-processing for this deep VVS-net could be due to (1) the linear processing or (2) the slightly nonlinear part of the retinal processing (3) a combination of both linear and nonlinear processing.
To distinguish between these hypotheses, we replaced the true retinal processing by its best linear approximation, retrained the VVS-net on the output of this linearized retina, and tested whether separability was as high as with the true retinal processing (App. E).
We found that the ﬁrst layer trained on the output of the linearized retinal representation was indeed much more separable than the ﬁrst layer of the control network (trained directly on natural images) at separating classes of objects, suggesting that the linear operation done by the retina does indeed play a crucial role in making the representation especially separable for subsequent layers.
M ETHODS
To estimate the linearity of the response of retinal neurons, we ﬁt a linear model to predict the neural response from the image on 8,000 images.
In order to prevent overﬁtting, we regularized the linear weights with an L2 penalty and optimized the weights using ridge regression.
The value of the penalty term was chosen by 10-fold cross-validation on the training set.
We then measured the
Pearson correlation between the linearized responses and original model responses on a testing set of 2,000 images.
To estimate the information about the input image retained by the retinal output representation, we ﬁt a linear model to reconstruct the image from the (ﬁxed) outputs of the trained retina-net of interest.
All numerical ﬁgures given are variance-explained results on the held-out test set.
To estimate the linear separability of classes of objects from the neural representation, we trained an SVM classiﬁer between all pairs of classes on half of the testing set of CIFAR-10 (1,000 images that were not used to train the network), and we tested the performance of the SVM classiﬁer on 1,000 held-out images from the testing set, as measured by the percentage of images classiﬁed correctly.
We then averaged the performance of the SVM across all pairs of classes to obtain the linear separability score.
D ISCUSSION
A uniﬁed theoretical account for the structural differences between the receptive ﬁeld shapes of retinal neurons and V1 neurons has until now been beyond the reach of efﬁcient coding theories.
Karklin & Simoncelli (2011) found that efﬁcient encoding of images with added noise and a cost on ﬁring rate produce center-surround RFs, whereas the same task without noise produces edge detectors.
However, this observation (as they note) does not explain the discrepancy between retinal and cortical representations.
Vincent et al.
(2005) propose a different set of constraints for the retina and V1, in which the retina optimizes for a metabolic constraint on total number of synapses, whereas V1 optimizes for a constraint on total ﬁring rate.
It is not clear why each of these constraints would predominate in each respective system.
Here we show that these two representations can emerge from the requirement to perform a biologically relevant task (extracting object identity from an image) with a bottleneck constraint on the dimensionality of the retinal output.
Interestingly, this constraint differs from the ones used previously to account for center-surround RFs (number of synapses or total ﬁring rate).
It is worth noting that we unsuccessfully tried to reproduce the result of Karklin & Simoncelli (2011) in our network, by adding noise to the image and applying an L regularization to the retina-net activations.
In our framework (different than the one of Karklin & Simoncelli (2011) in many ways), the receptive ﬁelds of the retina-net without bottleneck remained oriented across the full range of orders of magnitude of noise and L1 regularization that permitted successful task performance.
There is a long-standing debate on whether the role of the retina is to extract relevant features from the environment (Lettvin et al., 1959; Gollisch & Meister, 2010; Roska & Meister, 2014), or to efﬁciently encode visual information (Barlow, 1961; Atick & Redlich, 1990; 1992).
In this work, we show that our model of the visual system, trained on the same task and with the same input statistics, can exhibit different retinal representations depending on the degree of neural resources allocated to downstream processing by the ventral visual stream.
These results suggest the hypothesis that, despite its conserved structure across evolution, the retina could prioritize different computations in different species.
In species with fewer brain resources devoted to visual processing, the retina should nonlinearly extract relevant features from the environment for object recognition, and in species with a more complex ventral visual stream, the retina should prioritize a linear and efﬁcient transmission of visual information for further processing by the brain.
Although all species contain a mix of quasi-linear and nonlinear cell types, the proportion of quasi-linear cells seems to vary across species.
In the mouse, the most numerous cell type is a two-stage nonlinear feature detector, thought to detect overhead predators (Zhang et al., 2012). In contrast, the most common ganglion cell type in the primate retina is fairly well approximated by a linear ﬁlter (midget cells, 50% of all cells and > 95% in the central retina (Roska & Meister, 2014; Dacey, 2004)).
Note however that two-stage nonlinear models are also present in larger species, such as cat Y-type cells and primate parasol cells (Crook et al., 2008), making it difﬁcult to make deﬁnitive statements about inter-species differences in retinal coding.
To gain a better understanding of these differences, it would be useful to collect a dataset consisting of recordings of complete populations of ganglion cells of different species in response to a common bank of natural scenes.
A related question is the role of the parcellation of visual information in many ganglion cell types at the retinal output.
A recent theory of efﬁcient coding has shown that properties of midget and parasol cells in the primate retina can emerge from the objective of faithfully encoding natural movies with a cost on the total ﬁring rate traversing the optic nerve (Ocko et al., 2018).
On the other hand, many cell types seem exquisitely sensitive to behaviorally relevant features, such as potential prey or predators (Gollisch & Meister, 2010).
For example, some cell types in the frog are tuned to detect moving ﬂies or looming predators (Lettvin et al., 1959).
It is an intriguing possibility that different cell types could subserve different functions within a single species, namely efﬁcient coding of natural scenes for some types and extraction of behaviorally-relevant features for others.
In this study we allowed only a limited number of cell types (i.e. convolutional channels) at the retinal output (1 to 4), in order to have a dimensionality expansion between the retinal representation and the representation in the ventral visual stream (32 channels), an important condition to see the retinal center-surround representation emerge.
By using larger networks with more channels in the retina-net and the VVS-net, we could study the emergence of a greater diversity of neuron types in our retina-net and compare their properties to real retinal cell types.
It would also be interesting to extend our model to natural movies.
Indeed, most feature detectors identiﬁed to date seem to process some form of image motion: wide-ﬁeld, local or differential (Roska & Meister, 2014).
Adding a temporal dimension to the model would be necessary to study their emergence.
In conclusion, by studying emergent representations learned by a deep network trained on a biologically relevant task, we found that striking differences in retinal and cortical representations of visual information could be a consequence of the anatomical constraint of transmitting visual information through a low-dimensional communication channel, the optic nerve. Moreover, our computational explorations suggest that the rich diversity of retinal representations found across species could have adaptively co-evolved with the varying sophistication of subsequent processing performed by the ventral visual stream.
These insights illustrate how deep neural networks, whose creation was once inspired by the visual system, can now be used to shed light on the constraints and objectives that have driven the evolution of our visual system. A CKNOWLEDGMENTS
We would like to thank Lane McIntosh, Niru Maheswaranathan, Aran Nayebi, SueYeon Chung,
Vardan Papyan, Nora Brackbill, E.J. Chichilnisky for useful discussions and Stephen Baccus for his comments that greatly improved the manuscript.
S.G. thanks the Burroughs-Wellcome, McKnight,
James S. McDonnell and Simons foundations for support.
A PPENDIX A Q UANTIFICATION OF RECEPTIVE FIELD ISOTROPY IN RETINA AND V
A
B
VVS - ne t
R e t i na - ne t “Horse”
V
Edgyness Quantification
Figure (filters and HW)
[Figure]
The following analysis corroborates our qualitative observation that a dimensionality bottleneck in the retina-net yields center-surround retinal receptive ﬁelds and oriented, edge-detecting receptive ﬁelds in the ﬁrst layer of the VVS-net (V1).
For a given receptive ﬁeld, we quantiﬁed its orientedness as follows: we displayed rectangular bar stimuli of all possible combinations of width, orientations and spatial translations that ﬁt in the input image window.
Among all these combinations, we selected the bar stimulus width, orientation, and translation that yielded the strongest response from the
RF.
Bars with the same width as the best stimuli were presented at all orientations and translations, and for each orientation, we select the strongest response it produced (across all translations).
In this manner we obtained a measure of the strength of a receptive ﬁeld’s preference for all orientations.
We measured the strength of each RF preference (maximum strength of response) for its preferred orientation and for the orthogonal orientation, and computed the ratio of these strengths.
Completely isotropic ﬁlters would be expected to give a ratio of 1, while oriented ﬁlters should give higher ratios.
Note however that some deviation from 1 may indicate noise in the ﬁlter rather than true orientedness.
For each network layer, we averaged this ratio across ﬁlters (for convolutional layers with multiple layers) and trials (re-training of the same neural network architecture with different random initializations).
We found that the average ratios were 1 . 56( ± 0 . 22) for the retinal output, 3 .
( ± 0 . 30) for the ﬁrst VVS-net layer, and 2 . 57( ± 0 . 27) for the second VVS-net layer, where error margins given are 95% conﬁdence intervals.
To help assess whether retinal RFs were more isotropic than expected by chance, we compared them to receptive ﬁelds composed of random Gaussian noise as a baseline.
These give an average ratio (as computed above) of 1 . 97( ± 0 . 08) , signiﬁcantly higher than that for retinal RFs.
Furthermore, the standard deviation of RF preference across orientations was signiﬁcantly lower for the retinal RFs (0 .
118 ± 0 . 036) than for random RFs (0 .
177 ± 0 .
007) , also indicating that retinal RFs were more isotropic than expected by chance.
We also plot the average RF preference for different orientations at each layer to more comprehensively assess the isotropy of RFs at each network layer.
To aggregate results across multiple trials and ﬁlters, we rotated the coordinates of each receptive ﬁeld such that its preferred orientation was vertical, and averaged our results across ﬁlters and trials.
(See Figure 4).
The results conﬁrm our qualitative observations that (1) RFs in the second layer of a vanilla network ( N BN = 32 ) are highly oriented (Figure 4A) (2) RFs in the second layer (retina output) of a bottleneck network ( N BN = 1 ) are much more isotropic, consistent with center-surround RFs (Figure 4B top), and (3) RFs in the layer immediately following the retina-net in the bottleneck network are oriented (Figure 4B bottom).
We also quantitatively corroborate our observation that oriented receptive ﬁelds in the V1 layer pool input from oriented arrays of center-surround ﬁlters in the retina-net output layer.
We apply our method of isotropy quantiﬁcation described above to the weight matrix for each input-output ﬁlter combination in the V1 convolutional layer.
We ﬁnd that this weight matrix itself exhibits orientedness across ﬁlters and trials, conﬁrming our observation (Figure 4C).
B S IMPLE AND COMPLEX CELLS
[Figure]
VVS
[Figure]
net Layer
VVS-net Layer
[Figure]
D i ff e r en t R ando m I n i t i a li z a t
i on s f o r L
i nea r
R F A pp r o x .
[Figure]
Same as B, for the second layer of the VVS-net.
In contrast to the ﬁrst layer, the emergent preferred stimuli are always different across different initializations, indicative of a complex-cell like behavior.
To investigate whether neurons in our model’s early layers more closely resembled simple or complex cells, we performed the following analysis.
As before, we obtained local linear approximations of receptive ﬁelds by computing the gradient in input space with respect to the response of a given neuron.
Rather than beginning with a blank input, we ran multiple trials with different randomly initialized inputs.
A purely linear cell would give the same result no matter the initialization; a somewhat nonlinear but still “simple” cell is expected to give similar results across initializations.
A “complex” cell is expected to give different RF visualizations for different random inputs, reﬂecting multiple peaks in its response as a function of input.
In Figure 5 we show examples of receptive ﬁelds at different layers of our retina-net + VVS-net model (with N BN
= 1 , D V V S = 2 ) for different random intializations of the image (uniform random in [0, 1]).
The retina-net output and ﬁrst VVS-net layer exhibit “simple” behavior, but the second VVS-net layer exhibits observably “complex” behavior.
To quantify this effect, we measure the average (across ﬁlters within each layer and re-trainings of the same network architecture) standard deviation of computed RFs (normalized to the range [0, 1]) for each network layer.
We found that the average standard deviations were 7 . 9( ± 1 . 1) ⇥ 10  3 , 15 . 4( ± 0 .
8) ⇥ 10  3 , and 35 .
9( ± 0 .
8) ⇥ 10  3 for the retina-net output, ﬁrst
VVS-net layer, and second VVS-net layer, respectively, where the margins of error given are 95% conﬁdence intervals.
These results corroborate the observation of signiﬁcantly more complex behavior in the second VVS-net layer, mirroring the biological phenomenon in which complex cells pool from simple cells in V1.
C E FFECTS OF LOCAL RESPONSE NORMALIZATION ON EARLY VISUAL REPRESENTATIONS Normalization
Figure
[Figure]
We tested the robustness of our ﬁrst main ﬁnding – that a bottlenecked retina-net + VVS-net model yields center-surround receptive ﬁelds in the retina and oriented receptive felds in V1 – to the use of biologically realistic local response normalization at every layer of the network.
In particular, we normalized the output x of each channel (row r , column c ) of each layer as follows (during training and testing):
[Equation]
where the subscripts of x indicate the spatial location (row/column), and k , ↵ , ad  are constants.
We used k = 2 ,  = 0 .
5 and  = 0 .
75 , and ↵ = 5 ⇥ 10  4 and ↵ = 5 . 0 .
All parameter settings tested yielded RFs with the same qualitative properties as in the model without normalization.
Figureshows example RFs from the no-normalzation model next to example RFs from the normalization model with k = 2 ,  = 0 .
5 , ↵ = 5 . 0 .
D R ETINAL CELL TYPES TRADE OFF BETWEEN LINEAR INFORMATION TRANSMISSION AND NONLINEAR FEATURE EXTRACTION
[Figure]
[Figure]
E L INEARIZED RETINA ALSO INCREASES SEPARABILITY IN SUBSEQUENT LAYERS
[Figure]
[Figure]
r ab ili t y , the additionally the
) as a of the jump in between layers survives suggests that the main effect of retinal processing in this network is whitening (see Fig. 9) rather than nonlinear
In the case of the deepest VVS-nets tested, the retinal processing was quasi-linear for the tightest bottleneck (var.expl.
= 0.9,
N BN
= 1 , ﬁg. 3A).
However the very ﬁrst layer of the VVS-net after the retina disentangled classes (as measured by linear separability) almost as well as the second layer of a VVS-net without retina (ﬁg. 3F), suggesting that the retinal representation, while only moderately linearly separable itself, is especially transformable into a representation with a high linear separability.
To determine to what degree this increased separability was due to (1) the linear processing or (2) the slightly nonlinear part of the retinal processing, we performed an ablation experiment to eliminate factor (2).
We ﬁrst replaced the true retinal processing by its best approximation by a onelayer linear convolution (of sufﬁcient ﬁlter width to correspond to two convolutional layers withby 9 ﬁlters).
After this linearization process, we retrained the VVS-net using the linearized retinal representation as input, keeping the linearized retina weights frozen.
We found that the ﬁrst layer trained on the output of the linearized retinal representation was indeed much better than the ﬁrst layer of the control network (trained directly on natural images) at separating classes of objects (Fig.
8), suggesting that the linear operation done by the retina does indeed play a crucial role in making the representation especially separable for subsequent layers.
Visualization of retinal processing in
App.
F suggest that whitening is an important part of this linear processing.
F R ETINAL REPRESENTATION VISUALIZATION AS A FUNCTION OF VVS- NET DEPTH FOR BOTTLENECK N BN =
[Figure]