Learning low-dimensional generalizable natural features from retina using a U-net
Anonymous Author(s)
AffiliationAddress email
Abstract
Much of sensory neuroscience focuses on presenting stimuli that are chosen by theexperimenter because they are parametric and easy to sample and are thought tobe behaviorally relevant to the organism.
However, it is not generally known whatthese relevant features are in complex, natural scenes.
This work focuses on usingthe retinal encoding of natural movies to determine the presumably behaviorallyrelevant features that the brain represents.
It is prohibitive to parameterize a naturalmovie and its respective retinal encoding fully.
We use time within a natural movieas a proxy for the whole suite of features evolving across the scene.
We thenuse a task-agnostic deep architecture, an encoder-decoder, to model the retinalencoding process and characterize its representation of “time in the natural scene”in a compressed latent space.
In our end-to-end training, an encoder learns acompressed latent representation from a large population of salamander retinalganglion cells responding to natural movies, while a decoder samples from thiscompressed latent space to generate the appropriate future movie frame.
Bycomparing latent representations of retinal activity from three movies, we find thatthe retina has a generalizable encoding for time in the natural scene: the precise,low-dimensional representation of time learned from one movie can be used torepresent time in a different movie, with up to 17 ms resolution.
We then showthat static textures and velocity features of a natural movie are synergistic.
Theretina simultaneously encodes both to establishes a generalizable, low-dimensionalrepresentation of time in the natural scene.
Introduction
The flexibility and computational power of convolutional neural networks (CNNs) has helped sensoryneuroscience model the neural code for natural stimuli with rich feature repertoires.
It has beenshown that CNNs carryout encoding computations similar to those observed in the retina [1, 2].
A
CNN also makes the inverse problem of decoding complex stimuli from the retinal response [3] moretractable.
Understanding decoding has its own significant merit because neural systems of the retina can only ‘see’ the world through the retinal code.
For any visual information to be used to guide behavior, it must first be decoded from retinal responses
Historically the decoding problem for natural stimuli has been challenging because both the retinal response and the stimuli are high-dimensional
Although deep neural networks can capture the high dimensionality of neural inputs and responses they so by projecting the neural code into another high the neural decoding this work propose a novel artificial neural architecture that can decode complex natural scenes from retinalresponses with high fidelity while providing a low-dimensional latent space that is interpretable.
Using this architecture, we obtain unique insights into what features are important for reading outthe retina’s population code, and why encoding these features might enable an animal to navigate acomplex, dynamic natural environment.
We anchor our results on a unique dataset from the salamander retina.
Nearly one hundred outputganglion cells were recorded simultaneously while several different natural movies were projectedonto the photoreceptor layer.
The relatively long (20 s) movie clips were played many times, in apseudo-random order.
During the lifespan of a salamander, it goes through a transition from beingaquatic to terrestrial.
The sampled movies attempt to span these different motion environments.
Amovie of small fish in an aquarium with live plants was set up to match what a larval tiger salamandermight see underwater while it hunts for food.
A movie of leaves blowing in the wind resembles thescene a salamander may live in after it undergoes metamorphosis.
Does a salamander retina re-usehow it encodes features during the aquatic larval phase to represent features in a terrestrial scene?
This motivates us to investigate whether the encoding of natural features from one particular movie isgeneralizable to a novel movie.
Natural movies contain complex spatio-temporal features on multiple scales.
This makes enumeratingall possible stimulus states in natural movies intractable.
It is more feasible to investigate how theretina encodes time in the natural scene, as has been done in other studies [5].
The salamander retinaelicits precisely timed spikes [6].
The idea that the retina may encode how features change overtime to discriminate between frames has been explored before [7].
It has an intricate connection tostimulus-dependent representational drift in sensory systems [8].
Previous work [5] reported thata low dimensional compressed representation of activity from a mouse V1 population can be usedto discriminate frames that are 1s apart.
Furthermore, the authors showed that if such an encodingof time in the natural scenes exists, it is likely to be low-dimensional [5].
We train our deep neuralnetwork (an encoder-decoder in machine learning parlance) (Fig 1 and Section 2) for decoding.
Itreconstructs movie frames from retinal responses.
This is different from previous works [1, 2] thatuse CNNs as "in-silico retinas" to understand how specific stimuli drive the retina.
During training,the encoder part of the deep neural network learns a continuous, compressed latent representationof the retinal responses from which the decoder part samples to reconstruct target movie frames.
We enforce structural constraints [9, 10, 11, 12, 13] to obtain meaningful, continuous latent space.
We find that using this compressed representation, we can decode time in the natural scene up tosingle-frame resolution (17ms) (Fig 1).
This learned, compressed representation allows for precisedecoding in a novel set of natural scenes.
We show that the retina is responding to spatio-temporalfeatures that change over time, rather than having a clock.
We divided these features into static(texture) and dynamic (optic flow–a vector field describing the motion between subsequent frames[14]) motifs.
This allows us to construct two distributions which reflect how static features anddynamic features cluster to discriminate between frames.
We find that these features are synergisticwith respect to the encoding.
By simultaneously encoding static and dynamic features, the retinaestablishes a generalizable, low-dimensional representation of time in natural scenes.
Data and encoder-decoder architecture
Data : Our dataset contains retinal recordings of 93 cells responding to repeated, 20 second presentations of three natural movies at 60 frames per second.
There are 85-90 presentations of eachmovie interleaved in pseudo-random order.
Spikes are binned at 17ms, to align with the movieframe duration.
We compute each neuron’s firing rate as a function of time within each movie, (theperi-stimulus time histogram, or PSTH) by averaging spikes across trials in these bins.
Encoder-decoder architecture
: We use a U-net [15] as the backbone architecture.
The U-netsupplements an encoder of contracting layers with a nearly symmetric decoder of expansive layers(hence the U-shape, see Supplementary Information) [16, 17].
It is successful in domain-conversionproblems, i.e., text-to-speech [18].
We modify its skip connections to supply the decoder noisy features from the encoder.
Because the intermediate features from the feedforward encoder havedifferent resolutions at their convolutional layers, these skip connections enable the decoder to forma multi-scale and multi-level feature representation of the input.
Our particular network has anencoder with the same architecture as ResNet18.
We also initialize it with the ResNet18 [19] weightspre-trained by ImageNet [20].
Our decoder mirrors the feedforward architecture of the encoder (see Supplementary Information).
In addition, we turn these skip connections into variational samplinglayers and use them as compressed representations of the input retinal activity.
The set of activationvalues in this latent space in response to the neural activity (input) from a given stimulus movie isreferred to as Z movie , where ‘movie’ is either ‘fish’, ‘water’, or ‘leaf’).
We constrain all variationalsampling layers to have the same dimensionality, to simplify the training.
Knowing that the decoder reconstructs the movie frame from features across multiple spatial andtemporal scales, we use the perceptual loss [21] as the objective function for the encoder-decoder. This loss function compares a reconstructed frame with its target frame with respect to learnedfeatures from a pretrained VGG, as opposed to their raw pixels. Throughout the paper, we usefeatures from the pretrained VGG19 to represent movie frames.
Results
3.1 Retinal activity has a low intrinsic dimension
Fig 1A shows how we train an encoder-decoder for a specific movie (in this example the fishmovie).
The input is the retinal PSTHs from the 45 most reliably spiking cells in the population(see Supplementary Information).
We take trial-averaged firing rates, and ignore ‘noise’ correlationsbetween cells, to make an initial pass at this high-dimensional problem.
Considering that many ofthe observed retinal computations happen on a timescale below 400ms [23], we restrict ourselvesto 500 ms long snippets of the PSTHs.
To encourage the model to learn temporal structure within the movie, we ask the decoder to reconstruct a movie frame 100ms in the future, after the endof the 500ms snippet of neural response.
We choose this particular ∆ t based on the timescale ofpredictive information in retinal populations [24].
We train one predictive encoder-decoder for aspecific movie using 40,000 training samples (see Supplementary Information for details) with a90%/10% training/validation split.
The reconstruction is from an additional held-out test set of 10,000samples (100 frames, 100 PSTH patterns in each frame, see Supplementary Information).
We alsotrain a static encoder-decoder that learns to reconstruct a frame centered within the 500ms windowof neural activity (using 250ms before and after the target frame as the input, similar to [3]).
Thepredictive encoder-decoder achieves a reconstruction performance similar to the static one.
We focuson the predictive encoder-decoder in our subsequent analysis because it may capture both static andtemporal structures of a natural movie by design.
The latent space is a compressed representation, Z , of the mean firing rate patterns from the retinalpopulation.
We estimate the intrinsic dimension [25, 26] of the retinal activity and use it to guidethe selection of the dimensionality of the latent space.
This intrinsic dimension is the number ofvariables needed to describe a data distribution [27].
It is also a complexity measure of data becauseit determines the number of samples needed to characterize a data manifold [28, 29].
We find thatour estimate of the intrinsic dimensionality for all retinal activity in response to the three moviesyields the same result as the estimate of the intrinsic dimensionality from the activity for each movieseparately, i.e., including retinal activity from a different movie does not add complexity to the latentspace representation of the retinal response.
This suggests that features in natural movies may beencoded by the retina in a generalizable way across movies.
We use the intrinsic dimension of theretinal activity as a lower bound to determine latent dimension, i.e., dim Z in the encoder-decoder.
We then add additional latent dimensions to help encode factors that may result from combinations ofdifferent intrinsic dimensions.
We observe that the increase in the estimated intrinsic dimension ofthe latent space decreases after dim Z > 4 .
Meanwhile, we empirically observe good reconstructionperformance (the pixel MSE is about 0.02 averaged over 100 test frames of size 64X64 with pixelintensity ∈ [0 , 255] ) when we use latent dimension equal to 10 to reconstruct the held-out moviesegments (see Supplementary Information).
Thus, for further analyses, we use dim Z = 10 unlessotherwise stated.
To address questions about whether features learned from retinal activity respondingto one movie are generalizable to another novel movie, we generate "mismatched" Z ’s. These"mismatched" Z ’s are compressed representations of the retinal responses to Z water and Z leafmovies by passing those inputs through the encoder trained for the Z fish movie (Fig 1B).
The resultspresented below are similar regardless of which movie (water, leaf, fish) we train on and which othertwo movies are used to generate mismatched Z ’s.
3.2 The retina encodes complex, but interpretable spatio-temporal features in natural movies
The goal in this section is to qualitatively assess latent space stimulus features to show that theencoder-decoder contains features that are plausibly encoded by the retina.
We visualize highlyactivated features from decoding layers of multiple spatial scales (threshold by top 1% activationintensity, see Supplementary Information for more visualization).
We find both features that resemblebackground motion and features that resemble object motion (Fig 2) in a specific decoding layer.
In particular, the object motion feature closely traces the movement of the fish in the target moviesegment.
The average activation of this feature along the x-axis shows that this feature may beresponsible to decode both position and velocity of the fish movement in the target movie frames(top row).
This observation agrees with previous experiments that the retina can encode features thatevolve across time and space [30].
This motivates us to investigate whether we can decode the "timein the natural movie", i.e., discriminate between different frames from this specific decoding layer.
3.3 Low dimensional, generalizable representation of time in multiple movies
In this section, we investigate whether we can discriminate frames in natural movies and thus decodetime in natural scenes.
We can also ask whether the feature space of the retinal population used forone movie can be used for the other two natural movies in our three-movie dataset.
If this is the case, it would suggest that there is a general representation of spatio-temporal features in retinal activitythat supports this decoding of time.
Inspired by previous work that the visual cortex may contain a low-dimensional representationof complex stimulus features evolving over time [5], we decode time in the natural scene withthe particular latent space that corresponds to the decoding layer shown in Fig 2.
We obtain thecompressed representation of retinal activity on the held-out test frames for the fish movie, i.e., Z fish ,as well as test frames for two other novel movies (e.g., leaf and water).
This use of an encoderdecoder to generate a compressed representation of a data distribution has been investigated in detailin representation learning [9].
For each Z fish,leaf,water separately, we linearly decode the 1D framelabel of held-out test frames from the corresponding Z .
Fig 3A shows all decoding performance as afunction of the number of dimensions allowed in Z (we trained a series of encoder-decoders withdifferent dim Z ).
Because all encoder-decoder models are trained with the fish movie, the decodingfor the fish movie outperforms the other two.
However, we observe that dim Z = 5 is sufficient forthe encoder-decoder to build a latent space Z which can decode >80% frames in all three movies.
This is also the dimensionality needed for decoding in the natural scene reported in mouse visualcortex [5].
Since these frames are 17ms apart, this decoding performance shows that salamanderretina establishes a low dimensional representation of "time in the natural scene" with fine temporalresolution.
The high decoding performance from the mismatched Z ’s suggests that retinal activity may establisha general encoding for "time in the natural scene".
How is this possible?
The retina is presumablyencoding general space-time features that are predictive of the future space-time features in naturalscenes.
These could be complex, but they seem to generalize across movies.
We use informationtheory to directly evaluate the information that all three Z s contain that is relevant for decoding time.
We first calculate the mutual information between latent representations of retinal activity respondingto different movies (here, Z fish is the learned compressed retinal responses to the fish movie, so weshow the mutual information of fish vs. leaf
I ( Z fish ; Z leaf ) and fish vs. water, I ( Z fish ; Z water ) ,respectively).
We also observed that I ( Z fish ; Z leaf , time ) =
I ( Z fish ; Z leaf ) , i.e., including timedoes not add additional mutual information.
This tells us the latent representation obtained from retinalactivity for one movie encodes the generalizable "time in natural scenes" for a different movie, up to
[Figure]
[Figure]
the full entropy of time itself.
To show this, we use the chain rule of mutual information and subtractmutual information that is independent of time, i e., I ( Z fish ; Z leaf | time )
=
I ( Z fish ; Z leaf , time ) −
I ( Z leaf ; time ) =
I ( Z fish ; Z leaf ) − I ( Z leaf ; time ) , in Fig 4A.
The cyan bar shows the MI between
Z fish and mismatched Z that is about time, I ( Z fish ; Z leaf ) − I ( Z fish ; Z leaf | time ) .
It nearlysaturates H ( time ) .
We also observe that the mutual information between different movies is mostlyabout time.
The difference between the pink and cyan bar is very small.
This estimate confirms thatthe retina has a generalizable, precise representation of time in natural scenes that can discriminateconsecutive frames that are only 17ms apart.
3.4 Synergistic features for encoding "time in the natural scene"
Fig 3 shows that the retina has a low-dimensional, generalizable representation for time in thenatural scene.
We next ask what features the retina uses for this low-dimensional, generalizablerepresentation.
Although there are retinal circuits that encode object motion, most decoding work only uses naturalimages as their input stimuli [31].
Our stimulus set allows us to explore how natural texture andmotion might interact.
In Fig 4, we show two example decoding errors (Fig 4B), one where the predicted and true frame have similar static features and one other where the predicted and true frameswith similar optic flow (dynamic features).
This indicates that our observed latent representationof "time in the natural scene" can be confused when either the static or dynamic structure betweenframes is similar.
These examples are not sufficient to exclude the possibility that static texturesmay also be used to discriminate between different dynamic frames.
To investigate this, we performtwo parallel hierarchical clusterings: one on the frames themselves (static) and one on the optic flowframes (dynamic) of the test movie segment.
If textures govern the discrimination between framesin both dynamic and static settings, then these clusterings should produce similar clusters.
Beforeclustering, we convert all static frames and their corresponding optic flow frames to features that arethe activation from the last ReLu layer in a pre-traind VGG19 network [32].
These activations arebelieved to mimic features used in human perception of generic natural stimuli [21].
We observe thathierarchical clusterings on dynamic versus static motifs of all three movies yield different results (see
Supplementary Information for details).
The difference between clusters of frames based on static versus dynamic features enables us toconstruct three distributions of "time in the natural scene" (Fig 5A).
One uses the clustering based onthe static features Y static , another using the dynamic features Y dynamic , and a third combining bothsets of features Y joint .
By construction, we would like the joint distribution Y joint to have an entropyas close as possible to the full entropy of time H ( time ) = log 2(100) =6.6 bits.
This constructionnarrows down our search of coding schemes to how the retina combines dynamic and static structureswithin natural movies to encode the "time in the natural scene".
Previous work showed that the retina performs efficient coding [33, 34].
Efficient coding predicts thatredundancy should be minimized among different features of interest.
Therefore, while we would likethe joint distribution H ( Y joint ) to contain most of the entropy of time, we also want to minimize themutual information between the two components Y static and Y dynamic (minimize their redundancy).
We varied the threshold on the clustering hierarchies to coarse grain the distributions Y static and
Y dynamic , and also to create the joint distribution Y joint .
Because we do not know a priori theirrelative contributions, we coarse-grain both clusterings with thresholds such that the coarse grainedentropy of the two components are comparable, i.e., E ( Y static )
∼ E ( Y dynamic ) .
In Fig 4A, we showthe coarse-grained joint distribution we use for subsequent analysis.
It contains a small amount ofredundancy, i.e., I ( Y dynamic , Y static )
= 0 .
8 bits, while Y joint includes most of the entropy for time( H ( Y joint )
= 5 .
05 bits =
76% of H ( time ) = 6 . 6 bits).
These two components are dominated byeither static or dynamic features, respectively.
In Supplementary Information, we discuss in detailhow we find these distributions.
A different threshold may either introduce a significantly higherredundancy or sacrifice too much information from H ( time ) .
Using these distributions, now we can ask how the neural population encodes time through encodingthe both the static and dynamic features of the natural scene.
The latent activation of retinal inputsare compressed representations, so we frame this encoding problem using the information bottleneckmethod [35].
The information bottleneck method identifies whether a compressed representation
T retains as much information as possible about the relevant variable Y while compressing awayirrelevant components of the input X .
In this context, the information bottleneck shows how muchinformation a compressed representation needs, in order to encode a specific amount of informationabout the features of interest, Y .
The information bottleneck method minimizes the following objective function:
[Equation]
To make a meaningful measurement of I ( X ; T ) and I ( Y ; T ) , we first ensure that we have a meaningful latent representation Z .
The challenge is to prevent the so-called “posterior collapse” [11].
This isa phenomenon previously reported in encoder-decoders with highly expressive architectures (like the
ResNet18 network that we use here) [22].
These expressive architectures are capable of decodingcomplex features, e.g., our movie frames, without using Z .
This results in the latent code Z onlycontaining noise, i.e. I ( X ; Z ) = 0 .
Here, we use a simple heuristic to circumvent this scenario.
Asdiscussed in [36, 12, 37], we can obtain a meaningful Z by making the posterior to have a small but nonzero noise.
To be specific, we have σ 2 > 0 for p ( X | Z ) .
This is also dubbed a ’committed rate’for the encoder.
In our case, we choose log σ 2 = − 1 . 0 to further ensure numerical stability of themutual information estimator that we use [38].
The mutual information estimator also requires the prior of p ( Z ) to have a factorized marginal (eachmarginal is an independent Gaussian), N (0 , I ) (they are independent Gaussians).
This is a typicalconstraint introduced in the original variational autoencoder [13].
Combining this constraint on p ( Z )and the above constraint on p ( X | Z ) , we can approximate I ( X ; Z ) with the following estimator,
[Equation]
P is the number of test samples ( P = 10000 in our case) and z i , z
j are the latent activations for thei th or j th sample.
Because P ( Y ) has a uniform distribution (100 retinal inputs per frame and there are 100 held-outframes), we can use the following estimator for I ( Y ; Z ) (we also validated this estimation with awidely used non-parametric estimator [39], the difference is less than 0.1 in all our calculations):
[Equation]
Note that p l is the number of test samples for the l th frame.
p l = 100 in all of the test distributions.
Using p ( Z ) with a factorized Gaussian marginal brings an additional benefit: it is the sufficientand necessary condition for the latent space to exhibit orthogonal symmetry [40, 41, 42].
This
Darmois-Skitovitch characterization was first introduced to identify unique factors for independentcomponent analysis [43, 44].
Recent work also showed that this factorized marginal Z is necessaryto capture “ground truth factors of variation” [10] in the latent representation Z [45, 12].
In Fig 5B, we show the information plots for time, dynamic features, static features and the joint( { Y dynamic , Y static } ) features.
X is the retinal activity.
Y are features of interest, which are time,
Y dynamic , Y static and Y joint , shown in different colors.
Z are the latent activations of a series ofencoder-decoders with different dim Z .
Because the encoding of time reaches its full entropy of
H ( time ) (specifically, we observe this saturation at dim Z > 5 ), this shows that the compressedrepresentations Z from the encoder-decoder are near-optimal.
In the information bottleneck technique,the best any representation can do is to encode the full entropy of the relevant variable Y , i.e., here
H ( Y ) =
H ( time ) .
This is similar to the near-optimality shown in previous work in the variationalinformation bottleneck [46, 38].
In addition, we use many Z ’s spanning a range of dimensionsto construct these information curves.
We observe that all of these compressed representationsencode comparable amounts of information about both dynamic and static features.
This suggeststhat the dynamic features are as prominent as the static features in the retinal population’s internalrepresentation of complex natural scenes.
We highlight the benefit of simultaneous encoding of dynamic and static features in Fig 5C.
Bycomparing the information about Y joint = { Y static , Y dynamic } and the sum of the information aboutthese two components, we observe that there is synergy =
I ( Y joint ; Z ) − ( I ( Y static ; Z )
+ I ( Y dynamic ); Z ) .
By construction, the presence of synergistic information corresponds to latent representations withlower dimensions.
This suggests why the retina can compress its encoding of both static and dynamicfeatures.
When it encodes both features simultaneously, the synergy between these features helps theretina to represent the full entropy of time itself in fewer latent dimensions.
[Figure]
Discussion
This work uses a U-net-based deep learning architecture to reverse engineer a retinal encoding processfor complex natural movies.
Using the PSTHs of a large salamander retinal population, we identifystereotypical features that are generalizable across multiple natural movies.
We find that the retinauses a transferable, low dimensional representation to encode a rich set of natural space-time features.
The encoding obtained from one movie can be used to decode "time in the natural scene" for adifferent movie, despite differences in their particular spatio-temporal structures.
We also discoverthat the retina encodes time through synergistic coding of both dynamic and static features.
Here, we only observed synergy within the feature space (using mean firing rates of retinal activity,we assume all cells are independent).
Here, we decode time in its simplest form by asking how wellwe discriminate between different frames.
In future work, we would like to extend our analysis totemporal structure with proper predictive constraints, i.e., predicting a future at a longer ∆ t shouldbe more challenging than predicting a smaller ∆ t [47, 24].
We are also aware that the synergy here isdifferent from what can be observed between cells in the neural data.
The synergy in the neural codemay combine synergy in the feature space with synergy in the population code, itself [48, 49].
Compared to other method that learns a latent representation between neural activity and externalstimuli, our work is the most similar to [50, 51]. They also used a highly expressive, feedforwardencoder (a multilayer perceptron (MLP)).
MLP is fully-connected, so that its learned latent representation corresponds to a single global scale.
Unlike the MLP, our U-net architecture uses a ResNetas the encoder.
The ResNet encoder attains the same performance as the MLP, but by cascading Resblocks from coarse-to-fine scales.
This enables the U-net architecture to learn compressed latentrepresentation at multiple scales simultaneously.
We did not explore this feature explicitly, but it maybe interesting for future work in understanding neural dynamics in complex natural environments.
For example, there is a hierarchy of timescales both in natural scenes and output behaviors, fromhundreds of milliseconds to minutes (whisking to walking to making action plans [52, 53]).
Withadditional constraints [54], these variational saampling layers may learn hierarchically interpretablelatent representations for each timescale, individually, and understand how they can be combinedto drive complex behavioral outputs. Outside of neuroscience, This U-net is compatible to learnlatent representations between other temporal sequences (e.g., text) and complex spatio-temporalsignals (speech or video). Two potential applications are text-to-speech and video summarization. Combining latent representation at multiple scales may also reveal semantic relationships betweencomplex features in general object recognition, e.g, how does a model combine local features (nose,eye) with global shape (e.g., body size) to discriminate between cats and dogs.