Natural image synthesis for the retina with variational information bottleneck representation
Anonymous Author(s)
AffiliationAddress email
Abstract
In the early visual system, high dimensional natural stimuli are encoded into thetrains of neuronal spikes that transmit the information to the brain to produceperception.
But, is all the visual scene information required to explain the neuronalresponses?
In this work, we search for answers to this question by developing ajoint model of the natural visual input and neuronal responses using the Information
Bottleneck (IB) framework that is able to represent features of the input data into afew latent variables that play a role in the prediction of the outputs.
The correlationsbetween data samples, acquired from published experiments on ex-vivo retinas, isaccounted for in the model by a Gaussian Process (GP) prior.
The proposed IBGP model performs competitive to the state-of-the-art feedforward convolutionalnetworks in prediction of spike responses to natural stimuli.
Finally, the IB-GPmodel is used in a closed loop iterative process to obtain reduced-complexityinputs that elicit responses as those elicited by the original stimuli.
We foundthree properties of the retina’s IB-GP model.
First, the reconstructed stimuli fromthe latent variables show robustness in spike prediction across models.
Second,surprisingly the dynamics of the high-dimensional stimuli and RGCs’ responses arevery well represented in the embeddings of the IB-GP model. Third, the minimumstimuli consist of different patterns: Gabor-type locally high-frequency filters, onand off-center Gaussians, or a mixture of both. Overall, the IB-GP model not onlyprovides a principled approach for joint learning of the stimuli and retina codes,which could help understand the computation of the early visual system, but couldalso be potentially used in the closed loop with the visual prostheses to increasetheir efficiency.
Introduction
A fundamental challenge in neuroprosthetics is finding the proper input to a sensory or motor systemthat yields a desired functional output.
This is achieved naturally by the underlying physiologicalcircuit in an unimpaired system.
For example, in the vision system, when an image is formed ontothe photoreceptors, the electrical neural activity is processed by several layers of neurons withinthe retina, and a train of electrical neural spikes is sent to the visual cortex via the optic nerveproducing a perception of the visual scene.
But one can ask: is it possible to reduce the complexityof the input stimuli and still obtain the same functional behavior?
In other words, what are thereduced spatiotemporal stimuli that elicit the same response as that of the higher-dimensional originalinput?
These are important questions both for understanding the neural computations of the Retinaand designing the visual prosthesis [35] [18] that brings vision to the blind.
Nirenberg et al.
[22]showed that a good encoder of the retinal responses dramatically increases the prosthesis performance.
Therefore, finding the minimal information stimuli that optimally derive neurons could be helpfulfor the hardware design as well as the stimulation protocols of the prostheses.
For example, by structuring the energy of the electrical stimuli based on the optimal stimuli, it might be possible toavoid wasting resources on stimulating neurons (only based on the complex structure of the visualstimuli) that may not contribute to the perception.
In this work, we focus on the natural visionsystem and search for answers to these questions.
Our system consists of the spatiotemporal stimuliimpinging on the photoreceptors in the retina up to the spiking responses emitted by the Retinal Ganglion Cells (RGCs).
Linear-nonlinear (LN) [11] and generalized linear models (GLMs) [24] [25]were proposed for encoding of the retinal neural codes.
Although successful for artificial stimulisuch as white noise, they could not faithfully describe RGCs responses to natural stimuli [15].
Theconvolutional neural network (CNN) model of McIntosh et al.
[21] was the first model capable ofsuch computation.
This was followed by other efforts such as employing recurrent neural networks [5]or variants of CNNs [20].
Although high-dimensional in nature, neural activity can be well explainedby low-dimensional representations [12] [14] [27].
This opens up the possibility of using models thatlearn a latent representation of data, such as VAEs [16] and ICAs [19].
Accordingly, latent dynamicalmethods that inspect sources of low-dimensional structure in the neural response have recently gainedgrowing interest [17] [34].
Inspired by the Deep Variational Information Bottleneck (DVIB) [2][9], we introduce the Information Bottleneck Gaussian Process (IB-GP), a latent space model thatcan extract the principal features of the joint stimuli-responses distribution that are sufficient forpredicting the retina responses.
The schematic of the IB-GP is depicted in Fig.
This model is avariational approximation to the original Information Bottleneck framework of Tishby et al.
The objective of IB is to obtain a reduced representation of the input source that preserves maximuminformation about the output response.
DVIB assumes the IB’s latent codes are i.i.d.
This is a poorassumption for the retina dataset.
The reason for this is two-fold.
First, while observing a scene,images captured by the retina are naturally very similar.
Second, during fixation, the eyes undergodynamic movements that shift the gaze’s center.
Therefore, incorporating the temporal correlationsamong data samples should provide a better model.
Accordingly, IB-GP uses a Gaussian Processprior to modeling the retina dataset.
Our contributions in this work are:
• A probabilistic model : We propose IB-GP, a new model for the RGCs spike train in response to
the complex natural visual stimuli using the IB framework.
The model integrates a GP prior intothe IB latent code that can learn the temporal dynamics of the data in a lower dimension space.
This model shows favorable performance compared to the state-of-the-art feedforward CNN models ofthe retina tested on real-world experimental data.
• Latent dynamics analysis : We analyze the latent space of the proposed model and extract low- dimensional dynamics consistent with the high-dimensional data dynamics.
We show that theextracted dynamics are more predictive of the neural responses compared to dynamics extracted by a model with no temporal prior constraint.• Closed loop image synthesis : Utilizing the IB-GP model, which only allows principal features ofthe input stimuli to pass the bottleneck, the original input stimuli are pruned to contain the minimuminformation required for producing the RGC responses.
The resulting complexity-reduced stimulicould be used for the next round of measurements in an iterative process.
Related work
In the RGC modeling literature, our work is the closest to the work of McIntosh et al.
[21] in which a CNN is used to model the responses of RGCs to a set of input stimuli.
Despite the simplicity of thisapproach, the feedforward CNNs for image classification or regression cannot be directly utilized toextract the principal features of the stimuli contributing to the neural responses.
On the other hand,neural latent models such as [17] [23] [28] are primarily applied to cortical recordings to learn sourcesof neural variability in the motor cortex data.
Unlike our model, which directly relates the stimuli tothe neural responses, the stimuli in those lines of work are utilized as a guide star to disentangle thelatent factors.
Therefore, these methods are not directly applicable to modeling the RGC responses toinput stimuli.
In the closed-loop physical system input optimization literature, our work is relatedto [26][29][33][4].
In particular, for neuroscience applications, authors in [4] control the activityof the individual neuronal sites in V4 by optimizing individual input stimulation, whereas in ourwork, thanks to the learned latent code of the entire population of the targeted neurons, all stimuliare optimized together.
In the work of Shah et al.
[29], the RGCs responses to electrical stimulationwere optimized by first developing a model for the electrical stimuli and the spiking probabilities andthen using the model for adaptive stimulation.
The model is obtained by parameterizing the spikeamplitude and electrical stimulation threshold by a few parameters and then maximizing an evidentlower bound on the spiking probabilities.
Our work does not assume any relevant relation betweenthe input stimuli and the spiking probabilities.
Instead, the relevant parts of the input for predictingthe output are discovered automatically.
Methods3.1 IB-GP model
Notation and problem formulation The input stimuli dataset X ∈ R M × T consists of T consecutive spatial inputs of size M , i.e. X =
[ x M, 1 , x M, 2 , · · · , x M,T ] , that are projected in the retina at aconstant rate to elicit count responses of the form Y ∈ N N × T , where Y =
[ y N, 1 , y N, 2 , · · · , y N,T ]and N denotes the number of RGCs.
With the above assumptions, we formulate the problem asfinding the latent variables Z ∈ R L ×T that have maximal mutual information
I ( Z , Y ) with targets
Y .
We note that Z is comprised of T consecutive data points, i.e. Z =
[ z L, 1 , z L, 2 , · · · , z L, T ] where
T ≤ T .
Z is simultaneously constrained to have minimal mutual information with X .
Therefore, theconstraint optimization problem can be written as
[Equation]
where
[Equation]
ζ designates the parameters of the model (notation henceforth dropped for brevity) and β is a variableto adjust the amount of reduced and preserved information in Z .
To obtain these latent variables, weapproximate their posterior distribution with a parametric stochastic encoder p ϕ
( Z | X ) .
Due to thespatio-temporal nature of the data, assumption of data independence along the time dimension is notvalid.
The same is true in the latent space.
To account for this fact, we assume a posterior of the form:
[Equation]
where z l denotes the l -th latent vector of size T .
In Eq. 2, µ lϕ ∈ R T and Σ lϕ ∈ R T ×T are the meanand covariance matrix of a multi-variate normal distribution that are functions of the inputs.
Wechoose the structure of the covariance of the multivariate normal distribution in Eq.
2 to reflect thetime correlations in the data.
Hence, similar to previous work [3] [7] [13], we construct the Σ lϕ in ourmodel by the product of bidiagonal matrices:[ Σ l ϕ ] − 1 = V T l V l + I where [ V l ] ττ ′
[Equation]
We note that the simple GP kernel assumption makes the computation required for drawing samplesfrom
p ϕ linear in time [3].
Expanding the IB objective in Eq.
1 (details in the Appendix 1), we have:
[Equation]
Although all terms in the RHS of Eq. 4 are fully defined, computing marginal distributions p ( Z ) andp ( Y | Z ) may be intractable.
We use a variational approximation for p ( Y | Z ) that is parameterized withθ .
On the other hand, the prior on the latent variables, i.e. p ( Z ) , is modeled using GPs.
In particular,we assume a GP prior on Z defined as a multivariate normal:
[Equation]
where ρ is the variational approximation for the prior and K is the covariance function that modelsthe temporal correlations in the latent space.
In more details, the covariance between the τ -th and theτ ′ -th
samples is computed as K ττ ′ =
K ( τ, τ ′ ) where K is the kernel function.
We used the Cauchykernel:
[Equation]
where σ and l are the magnitude and temporal scales of the kernel function, respectively.
Cauchykernels have proved successful in modeling of the time-series systems with multi-scale dynamics asshown before [13].
Substituting the variational approximations of the intractable marginals in Eq.
4and using the fact that the Kullback Leibler (KL) divergence is always positive, we obtain a lowerbound for the IB objective:
[Equation]
where τ ( t ) is to represent time dependence up to time t .
See Appendix 1 for more details.
3.2 Other models considered
Maximum likelihood feedforward CNN is used as a comparison baseline.
We use the same CNNnetwork that was shown in the previous work to obtain state-of-the-art results on a larger version ofthe Natural dataset [21].
The predicted outputs of the network are the averaged maximum likelihoodestimation of the retina responses given the input stimuli.
Artificial noise is injected into the modelduring optimization to account for the variability in retinal spiking.
IB-Disjoint optimizes the objective from Eq.
7 using the same training procedure as that of theoriginal method.
The difference lies in that the IB-Disjoint assumes latent samples are temporallyindependent.
Accordingly, an isometric Gaussian distribution is employed as the prior in Eq.
Thismodel is akin to the standard Variational Auto-encoder [16], or vanilla DVIB [2] in which samples inthe data and latent space are assumed to be independent (no latent GP).
Closed loop stimulation
In this section, we take advantage of the model developed in the previous sections to devise analgorithm that uses the prior recorded data to optimize the stimulation by iteration in subsequentmeasurements in a closed loop.
Specifically, we assume the latent variables of the learned model,henceforth referred to as the forward model , have captured the principal rules governing the underlyingbiological system.
We intend to find a transformed version of the original input stimuli that yields aset of latent variables that produce the most correlated responses with the original target responsesin the subsequent round of measurements.
Of course, the original stimuli themselves are one set ofsolutions.
However, this is not a useful transformation.
Instead, we are interested in obtaining the besttransformation subject to a constraint on their complexity.
Hence, we define a parametric functionf ξ : R M × T → R M × T that maps the complex original stimuli X onto the transformed stimuli X ∗ .
Passing X ∗ to the forward model, the parameter of the mapping functions, i.e. ξ , are optimized sothat the forward model’s output responses Y ∗ are the most correlated with the original responses
Y .
Denoting the original stimuli and responses as X orig := X and Y orig :
=
Y , the objective functionreads as:
[Equation]
, Y ∗ )
+ α (cid:2)
D 2 ( X orig , X ∗ )
+ D 3 ( X ∗ ) (cid:3)
where:
[Equation]
Therein, p ϕ and q θ are the encoder and decoder of the forward model trained on the prior data.
D 1is a measure to ensure the closeness of the original target outputs and the responses of the forwardmodel to X ∗ .
D 2 is a similarity measure between the original and synthesized stimuli X orig and X ∗which encourages the image synthesizer to focus on finding the essential features in the stimuli ratherthan synthesizing entirely new solutions.
Finally, D 3 is used to constrain the synthesized stimuli
X ∗to be smooth.
Contributions of D 2 and D 3 to the total loss in Eq.
8 are tuned by hyperparameter α .
Two major mechanisms help to prevent f ξ from being simply an identity mapping.
One, during thetraining of the f ξ , the encoder’s latent variables Z ∗ are masked by zeroing out all the latent variablesexcept for the most informative one.
This way, X orig is no longer the optimal solution that minimizesthe first term in the objective function.
Two, f ξ is implemented using an Autoencoder architecturewith a limited latent capacity.
Finally, the optimized stimuli for subsequent rounds of measurementscan be obtained by the following procedure: (1) map the original stimuli to the optimized stimuliusing mapping f ξ , (2) encode the new input X ∗ to the posterior Z ∗ , (3) decode the latent variables,(4) construct the metric D and backpropagate the error to fit f ξ , (5) once converged, use the mappingfunction to obtain the optimized stimuli X ∗ (6) send new inputs X ∗ to the true biological system (orits model) and observe the new true system outputs, (7) to repeat the procedure, use the true system’snew inputs and outputs to fine-tune the forward model and (8) repeat.
Experiments5.1 Dataset
We focus on applying our model to the time-series dataset containing the 2D train of input stimuli(images) entering the vision circuit of an example subject and the corresponding elicited countresponses of a group of neurons.
Specifically, we use two real-world experimental datasets.
The firstone, Natural dataset, contains natural scene images as input stimuli and spike trains from nine RGCsin Salamander [21] 1 .
The second dataset, Brownian-movement , is a public dataset which containsimages with disc-like shapes having Brownian movements and the spike trains elicited in 91 Rat’s
RGCs [8].
5.2 Forward modeling
Training specifications We used the models outlined in section 3 to fit the neural activities andinput stimuli.
The Natural dataset is from the Natural images from the birthplace of the human [31].
This dataset consists of Natural images in the wild that do not belong to any particular category.
Thetrain set contains 3950 unique images (each unique image is jittered 100 times) of which 20% wasrandomly chosen for validation.
The test set contains 50 averaged repeated trials of novel stimuliwhere each unique image is jittered 100 times.
One batch of the test data consists of 10 uniqueimages.
We note that the output responses were first binned at 10 ms and then smoothed using a
Gaussian filter with a standard deviation equal to the bin size.
The Brownian-movement datasetcontains 129600 pairs of input images and output spikes binned at 12.5 ms in the training set, ofwhich 20% was randomly chosen for validation.
The test set contains 1800 novel stimuli.
We variedthe number of neurons in the Brownian dataset for which we tested our model.
We report the resultsfor nine neurons in the main paper and 15, 21, 27, and 36 neurons in Appendix 6.
Models and evaluation metrics To account for the discrete nature of the output responses (counttime series), we employ a Poisson regression model for both IB-GP and IB-Disjoint decoder aswell as the feedforward CNN loss term, i.e. q θ ( y t | Z ) =
Po ( y t | f θ ( Z )) where f θ is the parametricnetwork.
We sweep the number of latent factors and the prior pressure constraint β for both IB-GPand IB-Disjoint.
For the Natural and Brownian datasets, the number of latents is swept between 1to 15 and 1 to 50 variables, respectively.
Except for the latent space of the IB-GP, which requiresmore variables to model its covariance matrix, IB-GP and IB-Disjoint are implemented using the [Figure] but for the Brownian dataset. same architectures.
All hyperparameters in the training of the two models are fixed to be the same.
The architectures of the networks and the training setup are explained in more detail in Appendix 3.
We perform a three-fold fitting of the models for the spike prediction results.
We report the models’
Poisson negative log-likelihoods on the ground truth data.
Moreover, we use the Pearson Correlation[6] between the ground truth data and the networks’ prediction as another metric for evaluating theperformance of the models.
Traversal analysis We perform traversal analysis on the latent space of IB-GP and IB-Disjointmodels to study the amount of information picked by each latent variable.
To this end, the encoderp ϕ ( Z | X ) is fed with a batch of input stimuli to initialize the latents.
In the IB-GP model, the l -thlatent is sampled from N ( 0 , m K ) where m is a multiplier that is traversed in the range [ − 3 , 3] whilekeeping all the other latents to their inferred values.
The latent variables in the IB-Disjoint model aresampled similarly but from the normal univariate distribution N ( 0 , m I ) .
The latents are then passedto the decoder that reconstructs the output responses.
The output responses of the traversed model arethen compared against the non-traversed responses.
5.3 Closed-loop image synthesis
We use both the IB-GP and IB-Disjoint with a latent size of 4.
The prior constraint parameter β is setto 0.05.
These hyperparameters are chosen as a trade-off between the latent space compression and
RGCs neural activity fitting accuracy.
The image synthesizer f ξ is implemented using an Autoencoderarchitecture [32] with a latent size of 15.
The rationale for this choice of architecture is explained in
Appendix 3.
To allow for multi-phase iterative optimization, we use a neural network as the proxyfor the true biological system trained on the Natural dataset.
This model is trained once and keptfixed.
We use the feedforward CNN network of McIntosh et al.
[21] as the true model in order torule out the possibility of obtaining correlated responses due to the similarity of the IB-GP modeland the true model.
We optimize the stimuli in a three-phase closed-loop experiment.
We used the
Poisson regression loss as the D 1 measure in the objective function 8.
The similarity measure D 2between the synthesized and original stimuli is computed using a Mean Squared Error (MSE).
Total
Variation (TV) metric [10] is used as the D 3 measure.
We analyze the contributions of all measuresto the quality of the synthesized stimuli by sweeping the hyperparameter α .
Results
6.1 Forward modeling
RGC neural activity fitting Performances of the IB-GP, IB-Disjoint, and Feedforward CNN infittings to neural activities versus the number of latents for both Natural and Brownian datasets areplotted in Figure 2.
Figure 2a-d shows that IB-GP outperforms IB-Disjoint almost in all cases.
For afixed β value, the performance of the IB-Disjoint gets closer to that of the IB-GP with the increase inthe number of latents.
This indicates that IB-GP learns neural activities with a fewer number of latents.
Similar is the case with β value decreasing to the extent where IB-Disjoint’s performance reachesthat of the IB-GP.
For less constrained models, IB-Disjoint surpasses IB-GP (refer to Appendix 5).
Interestingly, IB-GP achieves good neural prediction performance for the Natural dataset even withtwo latents.
For the Brownian dataset, IB-GP keeps its performance gap with IB-Disjoint even for alarge number of latents.
This is due to the strong correlation between the Brownian stimuli dataset(refer to Appendix 4 to observe some examples of stimuli).
Both IB-GP and IB-Disjoint outperformthe Feedforward CNN when a sufficient number of latents are used (4 latents and β < 0 .
for
Natural and (27 latents and β < 0 .
01 ) for the Brownian dataset.
Comparison of the IB-GP and IB-Disjoint learned dynamics In Figure 3, we analyze the learneddynamics of the IB-GP and IB-Disjoint trained under the same conditions on the Natural dataset(results for the Brownian dataset are in Appendix 7).
For β = 0 .
and 15 latents, performancesof the IB-GP and IB-Disjoint in prediction of RGCs responses are similar (correlation ∼ 0 .
45 vs.∼ 0 .
43 , respectively).
Nevertheless, the amount of information picked by latents in each model variessignificantly.
To show this, we perform a traversal analysis of the latents and report the variability inthe output responses for each traversed latent in Figure 3(a,d).
The KL divergence for each latentis also plotted.
As observed in the figure, IB-GP lumps most of the information required to explainthe neural responses in one factor.
In comparison, this information is spread over five factors in
IB-Disjoint.
Analyzing the dynamics of the latents in each model can provide further insight.
Figure3(b,e) visualizes the inferred dynamics of the most informative latent in each model after reducingthe dimension to 2 with T-SNE.
As observed in the figure, the dynamics of the most informativelatent in the IB-GP have 50 clusters.
Strikingly, these 50 clusters correspond to the 50 unique stimuliin the dataset.
The test set consists of 5 batches in which each batch has 10 unique images (eachjittered 100 times).
Each batch is colored differently in the figure.
Auto-Correlation (AC) plots of thelatents corroborate these findings.
As evident from Figure 3(c,f), AC of the most informative latent inthe IB-GP closely follows the statistics of the true and predicted responses of the model (averagedover all neurons).
While the AC of the IB-Disjoint’s predicted responses is similar to that of the trueresponses, the AC of its latents is very different.
To demonstrate the efficacy of the IB-GP model infitting the RGCs responses, neural activity predictions of both models are shown in Figure 3 (g,h).
Refer to Appendix 8 for a more detailed interpretation of IB-GP’s latent dynamics.
6.2 Closed loop image synthesis
Figure 4 reports the training results of the closed-loop experiment after the first round of measurements.
Both the IB-GP and IB-Disjoint are used as the forward model in the training of the image synthesizer.
We consider two scenarios when tuning the hyperparameters in the objective function of the imagesynthesizer.
Scenario one: the image synthesizer is trained only to reduce the complexity of theoriginal stimuli independently from the RGCs responses (dropping the D 1 term in the objectivefunction of the image synthesizer).
In this scenario, we also consider the case where the TVsmoothness constraint is dropped.
These are important cases because they answer this valid question:can a simple dimensionality-reduction technique be used for synthesizing images that elicit responsescorrelated with the original ones?
Scenario Two: the image synthesizer is trained in conjunction with
RGCs responses ( α ̸ = 0 ).
Figure 4(a, b) plot the Pearson correlation and negative log-likelihood ofthe RGCs responses predicted by the true model in response to the stimuli obtained by the imagesynthesizer network for the two scenarios.
The TV loss of the obtained synthesized stimuli in each caseis plotted in Figure 4c.
As can be observed in the Figure 4d, dimensionality reduction of the stimuliwithout neural information results in non-smoothed images (column 2) or smoothed images (column3) but poor RGC response accuracy (rightmost results in Fig.
4a-c).
On the contrary, incorporating
RGC neural information obtains synthesized stimuli with excellent smoothness and significant RGCresponse accuracy (Fig.
4a-c and columns 5,6 in F4d).
These results demonstrate the robustness of [Figure] the proposed algorithm in finding synthesized images that provide accurate responses.
Strikingly,when IB-GP is used as the forward model in the closed-loop experiment, complex curvature shapesappear in the synthesized images.
Expectedly, these shapes appear where the neurons’ receptivefield is located.
The receptive fields (RFs) of the neurons, obtained by Spike-triggered Average(STA) analysis [1], are also shown in the figure.
Finally, we continue the optimization of the imagesynthesizer for the IB-GP model with hyperparameter α = 0 .
1 for another two rounds.
The Pearsoncorrelation of the synthesized stimuli in all three rounds reads as: 0.353, 0.366, and 0.361.
Detailsof the iterative optimization and evolution of the synthesized images in each phase are explained in
Appendix 9.
The last row in the Figure 4d shows a failure mode of the image synthesizer algorithm.
We tested the image synthesizer (trained on the Natural dataset) on the white noise data.
As seenfrom the figure, IB-Disjoint provides a superior synthesized image than other methods.
This is in linewith our observations that the IB-GP model fails to fit neural data elicited by the white noise.
Moreresults are available in Appendix 10.
This work presents the IB-GP model, a latent space variational model that learns a low-dimensionalrepresentation of the Natural stimuli and RGCs responses. We observed that the IB formulation ofthe neural activity fitting allows learning of a low-dimensional representation of the stimuli and RGC
[Figure] [Figure] [Figure]
[Figure]
8 ∞ 9 . 5 · 10 −0 .0 .0 .0 .α
N e g a ti v e L og L i k e li hood a 0.0 0.1 0.2 0.
0.8 ∞ 0 .0 .0 .0 .α
P ea r s on C o rr e l a ti on b
[Figure]
T V L o ss ]
[Figure]
Disjoint train No TV 0.
[Figure] [Figure]
Fields
Disjoint train
[Figure]
d
Original stimuli
Joint train IB-Disjoint a = 0.
a = 0.
Joint train IB-GP
The RFs of the neurons are depicted at scale.
responses which is more predictive than the feedforward models.
We demonstrated that incorporatinga temporal prior on the latent space of the IB model factorizes the sources of neural variability intofewer latents.
The Latent analysis of the model revealed the superior performance of the IB-GP to IBDisjoint with no temporal prior on the latents.
We observed that the dynamics of the high-dimensionalstimuli and RGCs’ responses were very well represented in the embeddings of the IB-GP model.
Themodel was then used in a closed-loop experiment to synthesize stimuli that elicit neuronal responsesas those elicited by the original complex stimuli.
We observed various types of optimal stimuli forthe Natural dataset such as Gabor-like filters, on- and off- center Gaussian, or a mixture of both.
Given the promising results demonstrating the usefulness of the model, the IB-GP model has somelimitations too.
First, it is unclear what type of temporal prior is sufficient for the best factorizationof latents.
Although the Cauchy kernel has proved efficient in modeling multi-scale dynamics [13],it might be that other temporal kernels, a combination of several kernels, or even spatio -temporalkernels [9] could improve disentangling of the latents.
Second, we observed the failure mode of the IB-GP model in the fitting of the white noise stimuli due to the lack of temporal correlation.
Third,computing the covariances of the IB-GP prior is computationally intensive, especially for the longtemporal datasets.
This might also be a possible negative societal impact of our work, resulting in alarge carbon footprint.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes] See section 6 1 and
(b) Did you describe the limitations of your work?
[Yes] See section
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See section7(d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See section 3and Appendix
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See the footnotein section1 and Appendix 11(b)
Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes] See section5 and Appendix
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes] See section
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See the footnote in section1 and Appendix
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators
[Yes] The dataset asset are
cited in section
(b) Did you mention the license of the assets?
[Yes One dataset is public and the other isproprietary
See Dataset subsection in
section 5(c) Did you include any new assets either in the supplemental material or as a URL?
[No]a sample of data will be added in the supplementary material for verification purposes.(d)
Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[Yes] One dataset is public.
The other was provided by the authors ofthe cited paper.
See Dataset subsection in section 5(e)
Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
[N/A]5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A](b)
Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A](c)
Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]