Original: Maximum a posteriori natural scene reconstruction from retinal ganglion cells with deep denoiser priors
Revised: Maximum a posteriori natural scene reconstruction from retinal ganglion cells with deep denoiser priors

Original: Abstract
Revised: Abstract

Original: Visual information arriving at the retina is transmitted to the brain by signals in the optic nerve, and the brain must rely solely on these signals to make inferences about the visual world.
Revised: Visual information arriving at the retina is transmitted to the brain by signals in the optic nerve, and the brain must rely solely on these signals to make inferences about the visual world.

Original: Previous work has probed the content of these signals by directly reconstructing images from retinal activity using linear regression or nonlinear regression with neural networks.
Revised: Previous work has probed the content of these signals by directly reconstructing images from retinal activity using linear regression or nonlinear regression with neural networks.

Original: Maximum a posteriori (MAP) reconstruction using retinal encoding models and separately-trained natural image priors offers a more general and principled approach.
Revised: Maximum a posteriori (MAP) reconstruction using retinal encoding models and separately-trained natural image priors offers a more general and principled approach.

Original: We develop a novel method for approximate MAP reconstruction that combines a generalized linear model for retinal responses to light, including their dependence on spike history and spikes of neighboring cells, with the image prior implicitly embedded in a deep convolutional neural network trained for image denoising.
Revised: We develop a novel method for approximate MAP reconstruction that combines a generalized linear model for retinal responses to light, including their dependence on spike history and spikes of neighboring cells, with the image prior implicitly embedded in a deep convolutional neural network trained for image denoising.

Original: We use this method to reconstruct natural images from ex vivo simultaneously-recorded spikes of hundreds of retinal ganglion cells uniformly sampling a region of the retina.
Revised: We use this method to reconstruct natural images from ex vivo simultaneously-recorded spikes of hundreds of retinal ganglion cells uniformly sampling a region of the retina.

Original: The method produces reconstructions that match or exceed the state-of-the-art in perceptual similarity and exhibit additional fine detail, while using substantially fewer model parameters than previous approaches.
Revised: The method produces reconstructions that match or exceed the state-of-the-art in perceptual similarity and exhibit additional ﬁne detail, while using substantially fewer model parameters than previous approaches.

Original: The use of more rudimentary encoding models (a linear-nonlinear-Poisson cascade) or image priors (a 1 /f spectral model) significantly reduces reconstruction performance, indicating the essential role of both components in achieving high-quality reconstructed images from the retinal signal.
Revised: The use of more rudimentary encoding models (a linear-nonlinear-Poisson cascade) or image priors (a 1 /f spectral model) signiﬁcantly reduces reconstruction performance, indicating the essential role of both components in achieving high-quality reconstructed images from the retinal signal.

Original:  Introduction
Revised:  Introduction

Original: A torrent of visual information arrives at each of our eyes where it is transduced by a large population of photoreceptors and transformed by retinal circuitry.
Revised: A torrent of visual information arrives at each of our eyes where it is transduced by a large population of photoreceptors and transformed by retinal circuitry.

Original: The resulting signals are transmitted to the brain in the spikes of a relatively small number of retinal ganglion cells (RGCs), suggesting that a large fraction of this visual information is lost.
Revised: The resulting signals are transmitted to the brain in the spikes of a relatively small number of retinal ganglion cells (RGCs), suggesting that a large fraction of this visual information is lost.

Original: Elucidating the nature of these encoded signals, and the inference processes the brain could use to interpret them, is fundamental to understanding biological vision.
Revised: Elucidating the nature of these encoded signals, and the inference processes the brain could use to interpret them, is fundamental to understanding biological vision.

Original: Image reconstruction provides a method of visualizing the content encoded by
Revised: Image reconstruction provides a method of visualizing the content encoded by

Original: RGC signals, evaluating it using standard image quality metrics, and reasoning about how the brain might use it [1, 2, 3, 4].
Revised: RGC signals, evaluating it using standard image quality metrics, and reasoning about how the brain might use it [1, 2, 3, 4].

Original: The fidelity and quality of reconstructed images also provides a useful objective function for optimizing the design of electrical stimulation patterns delivered by devices implanted to restore vision to people blinded by retinal degeneration [5, 6].
Revised: The ﬁdelity and quality of reconstructed images also provides a useful objective function for optimizing the design of electrical stimulation patterns delivered by devices implanted to restore vision to people blinded by retinal degeneration [5, 6].

Original: The simplest and most well-studied image reconstruction method is linear regression [1, 3, 7].
Revised: The simplest and most well-studied image reconstruction method is linear regression [1, 3, 7].

Original: Optimal reconstruction kernels are learned for each RGC using least-squares regression of recorded responses to many visual images, and the reconstruction of a new incident image is computed with the sum of the filters weighted by the measured response of each cell.
Revised: Optimal reconstruction kernels are learned for each RGC using least-squares regression of recorded responses to many visual images, and the reconstruction of a new incident image is computed with the sum of the ﬁlters weighted by the measured response of each cell.

Original: The quality of linearly reconstructed images can be enhanced by applying an autoencoder neural network to impose natural image priors [8], or by using deep neural networks to non-linearly recover additional high spatial frequency image components [4].
Revised: The quality of linearly reconstructed images can be enhanced by applying an autoencoder neural network to impose natural image priors [8], or by using deep neural networks to non-linearly recover additional high spatial frequency image components [4].

Original: Neural networks can also be directly trained (supervised) for reconstruction from raw data, but this is data-intensive, and has limited their use to simulated data, or low-dimensional stimuli and small numbers of cells [9].
Revised: Neural networks can also be directly trained (supervised) for reconstruction from raw data, but this is data-intensive, and has limited their use to simulated data, or low-dimensional stimuli and small numbers of cells [9].

Original: These direct approaches leave substantial room for improvement and interpretation.
Revised: These direct approaches leave substantial room for improvement and interpretation.

Original: A Bayesian formulation, in which encoding model and image priors are made explicit and are separately designed and optimized, offers a more flexible and interpretable solution.
Revised: A Bayesian formulation, in which encoding model and image priors are made explicit and are separately designed and optimized, offers a more ﬂexible and interpretable solution.

Original: Here, we present a method for approximate maximum a posteriori (MAP) image reconstruction from recorded RGC spikes, combining a retinal encoding model that accurately captures retinal responses [10] with state-of-the-art image priors that are implicitly embedded in deep denoising networks [11, 12, 13, 14, 15, 16].
Revised: Here, we present a method for approximate maximum a posteriori (MAP) image reconstruction from recorded RGC spikes, combining a retinal encoding model that accurately captures retinal responses [10] with state-of-the-art image priors that are implicitly embedded in deep denoising networks [11, 12, 13, 14, 15, 16].

Original: Separating the contributions of image prior and RGC spiking response likelihood allows independent training of both elements, and offers flexible exploration and interpretation.
Revised: Separating the contributions of image prior and RGC spiking response likelihood allows independent training of both elements, and offers ﬂexible exploration and interpretation.

Original: Specifically, (1) any pre-trained or closed-form natural image prior can be used, and the effects of different priors can be compared; and (2) any model of RGC encoding that provides an explicit likelihood can be used, allowing examination of the relative importance of different model components in representing the visual signal, such as spike train history, cell-to-cell correlations and output nonlinearities.
Revised: Speciﬁcally, (1) any pre-trained or closed-form natural image prior can be used, and the effects of different priors can be compared; and (2) any model of RGC encoding that provides an explicit likelihood can be used, allowing examination of the relative importance of different model components in representing the visual signal, such as spike train history, cell-to-cell correlations and output nonlinearities.

Original: We apply our method to reconstruct flashed natural images from responses of several hundred macaque RGCs of identified types recorded with large-scale electrode arrays.
Revised: We apply our method to reconstruct ﬂashed natural images from responses of several hundred macaque RGCs of identiﬁed types recorded with large-scale electrode arrays.

Original: We compare these results to published state-of-the-art linear and neural network regression methods.
Revised: We compare these results to published state-of-the-art linear and neural network regression methods.

Original: The new method matches or outperforms previous methods, producing sharper, more naturalistic reconstructions, and similar or greater perceptual similarity to ground truth.
Revised: The new method matches or outperforms previous methods, producing sharper, more naturalistic reconstructions, and similar or greater perceptual similarity to ground truth.

Original: However, our method also produces some reconstructions with distinctive hallucinated image structure, as would be expected when RGC signals are noisy and image priors dominate the reconstruction process.
Revised: However, our method also produces some reconstructions with distinctive hallucinated image structure, as would be expected when RGC signals are noisy and image priors dominate the reconstruction process.

Original: Finally, comparisons to reconstructions using more conventional encoding models and image priors reveal that both aspects of the approach are important for the most accurate reconstructions.
Revised: Finally, comparisons to reconstructions using more conventional encoding models and image priors reveal that both aspects of the approach are important for the most accurate reconstructions.

Original: Source code, fitted models, and a subset of the test dataset used to produce example reconstructions are available at https://github.com/wueric/RGC_MAP_reconstruction .
Revised: Source code, ﬁtted models, and a subset of the test dataset used to produce example reconstructions are available at https://github.com/wueric/RGC_MAP_reconstruction .

Original:  Retinal data and stimuli
Revised:  Retinal data and stimuli

Original: Extracellular recordings from RGCs in the peripheral macaque retina were performed ex vivo using a 512-electrode system [17] (see [3] for details).
Revised: Extracellular recordings from RGCs in the peripheral macaque retina were performed ex vivo using a 512-electrode system [17] (see [3] for details).

Original: Retinas were obtained from terminally anesthetized macaque monkeys used by other laboratories, in accordance with Institutional Animal Care and
Revised: Retinas were obtained from terminally anesthetized macaque monkeys used by other laboratories, in accordance with Institutional Animal Care and

Original: Use Committee requirements.
Revised: Use Committee requirements.

Original: Spikes from individual RGCs were identified with the YASS [18] spike-sorter.
Revised: Spikes from individual RGCs were identiﬁed with the YASS [18] spike-sorter.

Original: A 30-minute spatio-temporal white noise stimulus [19] was used to compute spatiotemporal receptive fields, and to identify cells of distinct types.
Revised: A 30-minute spatio-temporal white noise stimulus [19] was used to compute spatiotemporal receptive ﬁelds, and to identify cells of distinct types.

Original: Analysis focused on the four major
Revised: Analysis focused on the four major

Original: RGC types of the primate retina (ON parasol, OFF parasol, ON midget, and OFF midget) [20, 21], totaling roughly 700 cells per recording.
Revised: RGC types of the primate retina (ON parasol, OFF parasol, ON midget, and OFF midget) [20, 21], totaling roughly 700 cells per recording.

Original: The receptive fields [22] of all four cell types formed regular mosaics, with each providing nearly-complete coverage of a region of visual space (Figure 1B).
Revised: The receptive ﬁelds [22] of all four cell types formed regular mosaics, with each providing nearly-complete coverage of a region of visual space (Figure 1B).

Original: Natural images were presented to the retina as described previously [3].
Revised: Natural images were presented to the retina as described previously [3].

Original: Images from the ImageNet database [23] were converted to grayscale and cropped to 256x160.
Revised: Images from the ImageNet database [23] were converted to grayscale and cropped to 256x160.

Original: Each pixel measured approximately 11 × 11 µ m at the retina.
Revised: Each pixel measured approximately 11 ⇥ 11 µ m at the retina.

Original: Each stimulus image was displayed statically for 100 ms, followed by a 400 ms uniform gray display, allowing each image presentation to be treated as an independent trial (Figure 1A).
Revised: Each stimulus image was displayed statically for 100 ms, followed by a 400 ms uniform gray display, allowing each image presentation to be treated as an independent trial (Figure 1A).

Original: Recordings were gathered from retinas of two animals, in response to 19,000 and 10,000 images, respectively.
Revised: Recordings were gathered from retinas of two animals, in response to 19,000 and 10,000 images, respectively.

Original: Details are summarized in Tables 4 and 5 in the Appendix.
Revised: Details are summarized in Tables 4 and 5 in the Appendix.

Original:  MAP image reconstruction from RGC spikes
Revised:  MAP image reconstruction from RGC spikes

Original: The MAP estimate of the stimulus image x (a vector of pixel luminances) from observed RGC spike trains s corresponds to the minimum of the negative log of the posterior distribution, − log p ( x | s ) , which can be expressed using Bayes’ Rule as:
Revised: The MAP estimate of the stimulus image x (a vector of pixel luminances) from observed RGC spike trains s corresponds to the minimum of the negative log of the posterior distribution,  log p ( x | s ) , which can be expressed using Bayes’ Rule as:

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: The first term is the negative log likelihood of the stimulus, as obtained from an encoding model describing the probabilistic spiking of RGCs given a stimulus.
Revised: The ﬁrst term is the negative log likelihood of the stimulus, as obtained from an encoding model describing the probabilistic spiking of RGCs given a stimulus.

Original: In principle, any model can be used, and its parameters learned from experimentally measured RGC responses.
Revised: In principle, any model can be used, and its parameters learned from experimentally measured RGC responses.

Original: The second term is the negative log prior of the stimulus which can be learned from natural images, independent of retinal responses.
Revised: The second term is the negative log prior of the stimulus which can be learned from natural images, independent of retinal responses.

Original: Because encoding models with varying levels of fidelity and detail can be mixed with priors of varying sophistication, the MAP approach allows us to probe the distinct roles of these two components in image reconstruction [ 24 , 25 ] .
Revised: Because encoding models with varying levels of ﬁdelity and detail can be mixed with priors of varying sophistication, the MAP approach allows us to probe the distinct roles of these two components in image reconstruction [ 24 , 25 ] .

Original: 3.1 RGC encoding models
Revised: 3.1 RGC encoding models

Original: Encoding models for each RGC must be fitted to the experimental data before performing MAP reconstruction.
Revised: Encoding models for each RGC must be ﬁtted to the experimental data before performing MAP reconstruction.

Original: Two types of encoding models were used: (1) a linear-nonlinear-Poisson (LNP) cascade model with an exponential nonlinearity, the most commonly used model of RGC responses to visual stimuli [7]; and (2) a generalized linear model (GLM) that augments the LNP model with a feedback loop and cross-connections between neighboring cells, and which can accurately capture fine spike timing structure and cell-to-cell correlations [10].
Revised: Two types of encoding models were used: (1) a linear-nonlinear-Poisson (LNP) cascade model with an exponential nonlinearity, the most commonly used model of RGC responses to visual stimuli [7]; and (2) a generalized linear model (GLM) that augments the LNP model with a feedback loop and cross-connections between neighboring cells, and which can accurately capture ﬁne spike timing structure and cell-to-cell correlations [10].

Original: Encoding models were fitted to a training partition of the data by minimizing the negative log-likelihood, and regularization hyperparameters were tuned by evaluating the negative log-likelihood on a test partition for a small subset of RGCs of each type.
Revised: Encoding models were ﬁtted to a training partition of the data by minimizing the negative log-likelihood, and regularization hyperparameters were tuned by evaluating the negative log-likelihood on a test partition for a small subset of RGCs of each type.

Original: Linear-nonlinear-Poisson (LNP) encoding model
Revised: Linear-nonlinear-Poisson (LNP) encoding model

Original: The LNP model is the de facto standard model for describing the probabilistic spiking of RGCs in response to visual stimuli [7].
Revised: The LNP model is the de facto standard model for describing the probabilistic spiking of RGCs in response to visual stimuli [7].

Original: The model parameters for each RGC consist of a linear spatial filter m , and a scalar bias b .
Revised: The model parameters for each RGC consist of a linear spatial ﬁlter m , and a scalar bias b .

Original: In the model, a scalar generator signal m T x + b is passed through an exponential nonlinearity to yield a spike rate λ ( x ) .
Revised: In the model, a scalar generator signal m T x + b is passed through an exponential nonlinearity to yield a spike rate  ( x ) .

Original: Responses, s , are defined as the RGC spike count in a 150 ms interval, which are assumed to be
Revised: Responses, s , are deﬁned as the RGC spike count in a 150 ms interval, which are assumed to be

Original: Poisson-distributed with rate λ ( x ) .
Revised: Poisson-distributed with rate  ( x ) .

Original: The resulting negative log-likelihood for each RGC is
Revised: The resulting negative log-likelihood for each RGC is

Original: [Equation]
Revised: [Equation]

Original: which is convex in m and b .
Revised: which is convex in m and b .

Original: In practice, to ensure that the spatial filters were spatially compact and corresponded approximately with receptive fields obtained using reverse correlation with white noise, the negative log-likelihood objective was augmented with two regularization terms: an L1 penalty to induce sparsity in the filter weights, and an L2 penalty to minimize the difference between the filter and the receptive field.
Revised: In practice, to ensure that the spatial ﬁlters were spatially compact and corresponded approximately with receptive ﬁelds obtained using reverse correlation with white noise, the negative log-likelihood objective was augmented with two regularization terms: an L1 penalty to induce sparsity in the ﬁlter weights, and an L2 penalty to minimize the difference between the ﬁlter and the receptive ﬁeld.

Original: The complete LNP-fitting objective function with both regularization terms is described in A.4.
Revised: The complete LNP-ﬁtting objective function with both regularization terms is described in A.4.

Original: The parameters of each cell were optimized separately using FISTA [26].
Revised: The parameters of each cell were optimized separately using FISTA [26].

Original: MAP reconstruction requires the joint negative log-likelihood of the image based on the observed spikes from every RGC.
Revised: MAP reconstruction requires the joint negative log-likelihood of the image based on the observed spikes from every RGC.

Original: Since the LNP model assumes that the responses for each cell are statistically independent, this is simply the sum of the single-cell negative log-likelihoods, which is convex in the image x :
Revised: Since the LNP model assumes that the responses for each cell are statistically independent, this is simply the sum of the single-cell negative log-likelihoods, which is convex in the image x :

Original: [Equation]
Revised: [Equation]

Original: Generalized linear encoding models The generalized linear model (GLM) is an augmentation of the LNP model that incorporates the effects of spiking history and cell-to-cell correlations on neural response [10].
Revised: Generalized linear encoding models The generalized linear model (GLM) is an augmentation of the LNP model that incorporates the effects of spiking history and cell-to-cell correlations on neural response [10].

Original: In the GLM, the i th RGC is parameterized by a spatio-temporal stimulus filter k i [ x, y, t ] , a feedback (spike history) filter f i [ t ] , a set of neighboring cell coupling filters c ( j ) i [ t ] , and a bias b i .
Revised: In the GLM, the i th RGC is parameterized by a spatio-temporal stimulus ﬁlter k i [ x, y, t ] , a feedback (spike history) ﬁlter f i [ t ] , a set of neighboring cell coupling ﬁlters c ( j ) i [ t ] , and a bias b i .

Original: The GLM was fitted to spike counts measured within 1 ms time bins, a duration approximately matched to the spike refractory period [27, 28], yielding binary spike counts.
Revised: The GLM was ﬁtted to spike counts measured within 1 ms time bins, a duration approximately matched to the spike refractory period [27, 28], yielding binary spike counts.

Original: To limit the number of parameters and improve computational efficiency, the stimulus filter was assumed to be space-time separable k i [ x, y, t ]
Revised: To limit the number of parameters and improve computational efﬁciency, the stimulus ﬁlter was assumed to be space-time separable k i [ x, y, t ]

Original: = m i [ x, y ] h i [ t ] .
Revised: = m i [ x, y ] h i [ t ] .

Original: Letting ∗ denote time-domain convolution, w [ t ] denote the time course of the stimulus (boxcar), and { i } C denote the set of coupled neighboring cells, the generator signal g [ t ] is
Revised: Letting ⇤ denote time-domain convolution, w [ t ] denote the time course of the stimulus (boxcar), and { i } C denote the set of coupled neighboring cells, the generator signal g [ t ] is

Original: [Equation]
Revised: [Equation]

Original: Assuming a sigmoidal nonlinearity σ ( x ) =
Revised: Assuming a sigmoidal nonlinearity  ( x ) =

Original: e x / (1 + e x ) and a Bernoulli spiking distribution, the negative log-likelihood used to fit a single cell (see A.5.2 for complete derivation) is − logp
Revised: e x / (1 + e x ) and a Bernoulli spiking distribution, the negative log-likelihood used to ﬁt a single cell (see A.5.2 for complete derivation) is  log

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation][Equation]
Revised: [Equation] [Equation]

Original: To simplify the GLM, the filters h i , f i , and c ( j ) i were parameterized as weighted sums over a set of cosine-shaped “bump” functions [10] (discussed further in A.5.4).
Revised: To simplify the GLM, the ﬁlters h i , f i , and c ( j ) i were parameterized as weighted sums over a set of cosine-shaped “bump” functions [10] (discussed further in A.5.4).

Original: As with the LNP model, L and L2 regularization terms were added to constrain the spatial filters, and an additional L 1 , 2 group sparsity penalty was imposed on the coupling filters to eliminate spurious cell-to-cell correlations.
Revised: As with the LNP model, L and L2 regularization terms were added to constrain the spatial ﬁlters, and an additional L 1 , 2 group sparsity penalty was imposed on the coupling ﬁlters to eliminate spurious cell-to-cell correlations.

Original: The complete objective function is described in detail in A.5.3.
Revised: The complete objective function is described in detail in A.5.3.

Original: Model parameters were found by alternating between convex minimization steps for the spatial and temporal filters of each RGC, using
Revised: Model parameters were found by alternating between convex minimization steps for the spatial and temporal ﬁlters of each RGC, using

Original: FISTA [26, 29] for each step.
Revised: FISTA [26, 29] for each step.

Original: For MAP image reconstruction, we compute the joint negative log-likelihood of the image based on responses of all cells over time bins N, N + 1 , . . .
Revised: For MAP image reconstruction, we compute the joint negative log-likelihood of the image based on responses of all cells over time bins N, N + 1 , . . .

Original: , T .
Revised: , T .

Original: This can be constructed from the single cell negative log-likelihoods using the chain rule (see A.5.5 for derivation), and is again convex in the image x : [Equation] [Table]
Revised: This can be constructed from the single cell negative log-likelihoods using the chain rule (see A.5.5 for derivation), and is again convex in the image x :  log p ( s [ N, ..., T ] | x , s [0 , ..., N  1]) =

Original: [Equation]
Revised: [Equation]

Original: 3.2 MAP with Gaussian 1 /f priors
Revised: 3.2 MAP with Gaussian 1 /f priors

Original: A stationary Gaussian prior with 1 /f spectral energy is the simplest and most commonly used image prior [30], and is the basis for most classical image processing algorithms.The 1 /f prior [Figure] C
Revised: A stationary Gaussian prior with 1 /f spectral energy is the simplest and most commonly used image prior [30], and is the basis for most classical image processing algorithms.

Original: D

Original: E

Original: F

Original: G

Original: H

Original: I

Original: J

Original: K L [Figure] assumes that pixels of the image are drawn from a stationary jointly Gaussian distribution (and thus that the spatial covariance matrix is diagonalized by the 2D Fourier basis) and that spectral power (variance of each spatial frequency component) falls off in inverse proportion to the square of the frequency f 2 .
Revised: The 1 /f prior assumes that pixels of the image are drawn from a stationary jointly Gaussian distribution (and thus that the spatial covariance matrix is diagonalized by the 2D Fourier basis) and that spectral power (variance of each spatial frequency component) falls off in inverse proportion to the square of the frequency f 2 .

Original: Discarding terms that do not depend on the image x , the negative log prior can be written as − log p ( x ) =
Revised: Discarding terms that do not depend on the image x , the negative log prior can be written as  log p ( x ) =

Original: (cid:80) k | a k ( x ) | 2 /f 2 k where a k ( x ) is the amplitude of the k th Fourier coefficient at frequency f k .
Revised: P k | a k ( x ) | 2 /f 2 k where a k ( x ) is the amplitude of the k th Fourier coefﬁcient at frequency f k .

Original: MAP image reconstruction using this prior can be performed using standard unconstrained convex minimization methods because both the negative log prior and the
Revised: MAP image reconstruction using this prior can be performed using standard unconstrained convex minimization methods because both the negative log prior and the

Original: RGC encoding negative log-likelihoods described in (3) and (6) are convex.
Revised: RGC encoding negative log-likelihoods described in (3) and (6) are convex.

Original: 3.3 Approximate MAP with denoising convolutional neural network (dCNN
Revised: 3.3 Approximate MAP with denoising convolutional neural network (dCNN) priors

Original: priors

Original: Modern denoising convolutional neural networks can represent powerful image priors, but these priors are implicit [12, 31]: they are not expressed in closed form, and their values cannot be computed directly, making exact MAP inference difficult.
Revised: Modern denoising convolutional neural networks can represent powerful image priors, but these priors are implicit [12, 31]: they are not expressed in closed form, and their values cannot be computed directly, making exact MAP inference difﬁcult.

Original: The “plug-and-play” methodology provides an approximate iterative procedure for using such denoisers in MAP estimation problems [11], by incorporating them into variable-splitting optimization methods such as half-quadratic splitting (HQS)
Revised: The “plug-and-play” methodology provides an approximate iterative procedure for using such denoisers in MAP estimation problems [11], by incorporating them into variable-splitting optimization methods such as half-quadratic splitting (HQS)

Original: [32] or alternating direction method of multipliers (ADMM)
Revised: [32] or alternating direction method of multipliers (ADMM)

Original: Here, we use a method based on the HQS method presented in [15] to perform MAP reconstruction from RGC spikes.
Revised: Here, we use a method based on the HQS method presented in [15] to perform MAP reconstruction from RGC spikes.

Original: Following [15], we introduce an auxiliary variable z , split the original problem in equation (1) into two complementary sub-problems, incorporate a regularization parameter λ p to control the strength of the prior term, and iteratively alternate between solving the two sub-problems:
Revised: Following [15], we introduce an auxiliary variable z , split the original problem in equation (1) into two complementary sub-problems, incorporate a regularization parameter  p to control the strength of the prior term, and iteratively alternate between solving the two sub-problems:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Since the sub-problem in equation (8) has the same objective function as MAP Gaussian denoising with known noise variance λ p /ρ ( k ) , we compute its solution using a single forward pass of a pretrained DRUNet denoiser network [15].
Revised: Since the sub-problem in equation (8) has the same objective function as MAP Gaussian denoising with known noise variance  p / ⇢ ( k ) , we compute its solution using a single forward pass of a pretrained DRUNet denoiser network [15].

Original: This is an approximation, because the denoiser was trained with an MSE objective and hence targets the mean of the posterior rather than its maximum.
Revised: This is an approximation, because the denoiser was trained with an MSE objective and hence targets the mean of the posterior rather than its maximum.

Original: The approximation is correct in the limit of small noise, and thus can be used within an iterative procedure that gradually decreases the objective function [11].
Revised: The approximation is correct in the limit of small noise, and thus can be used within an iterative procedure that gradually decreases the objective function [11].

Original: The overall iterative MAP reconstruction procedure is given in Algorithm 1.
Revised: The overall iterative MAP reconstruction procedure is given in Algorithm 1.

Original: Unlike most applications of HQS, the encoding term in equation (7) is non-quadratic in x and hence (7) was solved iteratively (gradient descent with momentum and backtracking line search) rather than in closed form.
Revised: Unlike most applications of HQS, the encoding term in equation (7) is non-quadratic in x and hence (7) was solved iteratively (gradient descent with momentum and backtracking line search) rather than in closed form.

Original: z (1) was initialized as the linear solution; using random Gaussian initialization does not significantly affect the results (see Appendix A.6).
Revised: z (1) was initialized as the linear solution; using random Gaussian initialization does not signiﬁcantly affect the results (see Appendix A.6).

Original: Because convergence in the mathematical sense is not necessary for most imaging applications [36], K = 25 iterations were used in Algorithm 1.
Revised: Because convergence in the mathematical sense is not necessary for most imaging applications [36], K = 25 iterations were used in Algorithm 1.

Original: As in [15], ρ ( k ) was increased per-iteration on a log-spaced schedule.
Revised: As in [15], ⇢ ( k ) was increased per-iteration on a log-spaced schedule.

Original: The hyperparameters λ p and [ ρ (1) , ρ (25) ] were determined by performing a grid search and evaluating reconstruction quality on an 80-image subset of the test partition.
Revised: The hyperparameters  p and [ ⇢ (1) , ⇢ (25) ] were determined by performing a grid search and evaluating reconstruction quality on an 80-image subset of the test partition.

Original: Variations in hyperparameters over a reasonable range ( ρ (1) ∈ [10 − 2 , 10 − 1 ] , ρ (25) ∈ [30 , 500] , λ p ≈ 0 . 1 ) produced similar reconstruction quality, and optimal hyperparameters were similar across the two retinas and across LNP and GLM encoding models.
Revised: Variations in hyperparameters over a reasonable range ( ⇢ (1) 2 [10  2 , 10  1 ] , ⇢ (25) 2 [30 , 500] ,  p ⇡ 0 . 1 ) produced similar reconstruction quality, and optimal hyperparameters were similar across the two retinas and across LNP and GLM encoding models.

Original: Approximate MAP reconstructions using this algorithm are termed MAP-GLM-dCNN and
Revised: Approximate MAP reconstructions using this algorithm are termed MAP-GLM-dCNN and

Original: MAP-LNP-dCNN for the GLM and LNP encoding models, respectively.
Revised: MAP-LNP-dCNN for the GLM and LNP encoding models, respectively.

Original: A
Revised: A

Original: C
Revised: C

Original: B
Revised: B

Original: D
Revised: D

Original: 3.4 Benchmark: nonlinear regression with artificial neural networks
Revised: 3.4 Benchmark: nonlinear regression with artiﬁcial neural networks

Original: Current state-of-the-art methods for reconstruction of natural images from RGC spikes rely on an initial linear reconstruction step [1, 3], followed by application of nonlinear neural networks to further improve the image.
Revised: Current state-of-the-art methods for reconstruction of natural images from RGC spikes rely on an initial linear reconstruction step [1, 3], followed by application of nonlinear neural networks to further improve the image.

Original: In one approach, Parthasarathy et al.
Revised: In one approach, Parthasarathy et al.

Original: [8] use a deep convolutional autoencoder (L-CAE) trained with MSE loss to incorporate image priors in the reconstruction.
Revised: [8] use a deep convolutional autoencoder (L-CAE) trained with MSE loss to incorporate image priors in the reconstruction.

Original: The authors trained and tested the model on simulated RGC spikes.
Revised: The authors trained and tested the model on simulated RGC spikes.

Original: For our comparisons, we used the published architecture and hyperparameters but trained the model using backpropagation on experimentally measured retinal spike counts.
Revised: For our comparisons, we used the published architecture and hyperparameters but trained the model using backpropagation on experimentally measured retinal spike counts.

Original: Further details and characterization of the L-CAE on experimental data are provided in Appendix A.7.
Revised: Further details and characterization of the L-CAE on experimental data are provided in Appendix A.7.

Original: Another method was developed by Kim et al.
Revised: Another method was developed by Kim et al.

Original: [4] who partition the target image into high and low spatial frequency components.
Revised: [4] who partition the target image into high and low spatial frequency components.

Original: Linear regression from RGC spike counts is used to recover the low-frequency component and a fully-connected neural network is used to non-linearly reconstruct the high-frequency component.
Revised: Linear regression from RGC spike counts is used to recover the low-frequency component, and a fully-connected neural network is used to non-linearly reconstruct the high-frequency component.

Original: The two components are summed and then passed through a final deblurring CNN.
Revised: The two components are summed and then passed through a ﬁnal deblurring CNN.

Original: Both the high-frequency network and the deblurring CNN are trained with backpropagation on RGC responses to natural images.
Revised: Both the high-frequency network and the deblurring CNN are trained with backpropagation on RGC responses to natural images.

Original: This method has achieved the most accurate reconstructions in the literature to date.
Revised: This method has achieved the most accurate reconstructions in the literature to date.

Original: Details of our implementation of the Kim et al.
Revised: Details of our implementation of the Kim et al.

Original: method are provided in Appendix A.8.
Revised: method are provided in Appendix A.8.

Original:  Results 4.1 Approximate MAP with GLM/dCNN matches or exceeds state-of-the-art results
Revised:  Results 4.1 Approximate MAP with GLM/dCNN matches or exceeds state-of-the-art results

Original: To compare our MAP-GLM-dCNN method to current state-of-the-art approaches, image reconstructions were generated from the test partitions of the datasets.
Revised: To compare our MAP-GLM-dCNN method to current state-of-the-art approaches, image reconstructions were generated from the test partitions of the datasets.

Original: Example reconstructions are shown in columns 3-5 of Figure 2 for the L-CAE [8], for Kim et al.
Revised: Example reconstructions are shown in columns 3-5 of Figure 2 for the L-CAE [8], for Kim et al.

Original: [4], and for our method, respectively.
Revised: [4], and for our method, respectively.

Original: Qualitatively, the MAP-GLM-dCNN reconstructions are seen to be sharper than those of L-CAE, and contain additional image details (especially extended contours, as in rows C, E, G, H, I, L).
Revised: Qualitatively, the MAP-GLM-dCNN reconstructions are seen to be sharper than those of L-CAE, and contain additional image details (especially extended contours, as in rows C, E, G, H, I, L).

Original: When
Revised: When

Original: [Table] [Table]
Revised: [Table] [Table]

Original: compared to Kim et al.
Revised: compared to Kim et al.

Original: [Table]
Revised: [Table]

Original: MAP-GLM-dCNN tended to recover more content, particularly straight edges (rows E, G, H, I, L)
Revised: MAP-GLM-dCNN tended to recover more content, particularly straight edges (rows E, G, H, I, L)

Original: [Table]
Revised: [Table]

Original: but sometimes exaggerated contrast (rows H, I, N).
Revised: but sometimes exaggerated contrast (rows H, I, N).

Original: MAP-GLM-dCNN produced qualitatively different artifacts than the other methods.
Revised: MAP-GLM-dCNN produced qualitatively different artifacts than the other methods.

Original: In particular, it sometimes hallucinated naturalistic structure not present in the stimulus images (rows J, K, N), including striking irregularities in contours (rows D, K, L, M)
Revised: In particular, it sometimes hallucinated naturalistic structure not present in the stimulus images (rows J, K, N), including striking irregularities in contours (rows D, K, L, M)

Original: [Table]
Revised: [Table]

Original: Reconstruction quality for each method was quantified over the test and heldout partitions using
Revised: Reconstruction quality for each method was quantiﬁed over the test and heldout partitions using

Original: MS-SSIM and PSNR.
Revised: MS-SSIM and PSNR.

Original: Scatter plots comparing MS-SSIM and PSNR on the test partition of one retina are shown in Figure 3A and 3B for Kim et al.
Revised: Scatter plots comparing MS-SSIM and PSNR on the test partition of one retina are shown in Figure 3A and 3B for Kim et al.

Original: and the L-CAE, respectively, and summary statistics over the test and held out partitions for both retinas are presented in Tables 1 and 2.
Revised: and the L-CAE, respectively, and summary statistics over the test and held out partitions for both retinas are presented in Tables 1 and 2.

Original: On a per-image basis, the perceptual similarity (measured with MS-SSIM) of MAP-GLM-dCNN reconstructions to ground truth is higher than L-CAE (3B).
Revised: On a per-image basis, the perceptual similarity (measured with MS-SSIM) of MAP-GLM-dCNN reconstructions to ground truth is higher than L-CAE (3B).

Original: The perceptual similarity of MAP-GLM-dCNN results is comparable to that of the much more complicated Kim et al.
Revised: The perceptual similarity of MAP-GLM-dCNN results is comparable to that of the much more complicated Kim et al.

Original: method (3A).
Revised: method (3A).

Original: But note that the PSNR of MAP-GLM-dCNN reconstructions was systematically lower than either benchmark.
Revised: But note that the PSNR of MAP-GLM-dCNN reconstructions was systematically lower than either benchmark.

Original: This is not surprising, as the MAP optimization procedure does not necessarily minimize MSE.
Revised: This is not surprising, as the MAP optimization procedure does not necessarily minimize MSE.

Original: These results held for both retinas (Tables 1 and 2).
Revised: These results held for both retinas (Tables 1 and 2).

Original: 4.2 Deep denoiser prior substantially improves image quality over 1 /f prior
Revised: 4.2 Deep denoiser prior substantially improves image quality over 1 /f prior

Original: To test the importance of the image prior, MAP-GLM-dCNN results were compared against reconstruction using the GLM encoding model with the classical 1 /f Gaussian prior (MAP-GLM-1F).
Revised: To test the importance of the image prior, MAP-GLM-dCNN results were compared against reconstruction using the GLM encoding model with the classical 1 /f Gaussian prior (MAP-GLM-1F).

Original: Example reconstructed images using the denoiser prior and 1 /f prior are shown in columns 5 and 7, respectively, of Figure 2.
Revised: Example reconstructed images using the denoiser prior and 1 /f prior are shown in columns 5 and 7, respectively, of Figure 2.

Original: Images reconstructed with the denoiser prior are less “grainy”, and tend to have both crisper edges and smoother surfaces.
Revised: Images reconstructed with the denoiser prior are less “grainy”, and tend to have both crisper edges and smoother surfaces.

Original: The artifacts seen in the 1 /f examples are expected, since this simple prior does not constrain phase, whose alignment is essential for generating sharp spatially-localized features.
Revised: The artifacts seen in the 1 /f examples are expected, since this simple prior does not constrain phase, whose alignment is essential for generating sharp spatially-localized features.

Original: Scatter plots of image quality on the test partition using MS-SSIM and
Revised: Scatter plots of image quality on the test partition using MS-SSIM and

Original: PSNR are shown for one retina in Figure 3C, and mean values for both retinas are summarized in
Revised: PSNR are shown for one retina in Figure 3C, and mean values for both retinas are summarized in

Original: Tables 1 and 2.
Revised: Tables 1 and 2.

Original: Consistent with the visual appearance, PSNR and MS-SSIM were systematically higher when using the denoiser prior, in both retinas.
Revised: Consistent with the visual appearance, PSNR and MS-SSIM were systematically higher when using the denoiser prior, in both retinas.

Original: Thus, using the more sophisticated denoiser image prior substantially increased the perceptual similarity of the reconstructions to ground truth.
Revised: Thus, using the more sophisticated denoiser image prior substantially increased the perceptual similarity of the reconstructions to ground truth.

Original: 4.3 GLM encoding model recovers additional image structure over LNP encoding model
Revised: 4.3 GLM encoding model recovers additional image structure over LNP encoding model

Original: To test the importance of the encoding model, we compared images reconstructed using the GLM and
Revised: To test the importance of the encoding model, we compared images reconstructed using the GLM and

Original: LNP encoding models, both in combination with the the denoiser prior.
Revised: LNP encoding models, both in combination with the the denoiser prior.

Original: Example images are shown in columns 5 and 6 of Figure 2.
Revised: Example images are shown in columns 5 and 6 of Figure 2.

Original: Images reconstructed using both models exhibit natural image structure, including smooth surfaces and well-defined edges, but the GLM-reconstructed images tended to have more realistic-looking textures, whereas the LNP-reconstructed images tended to be overly simplified.
Revised: Images reconstructed using both models exhibit natural image structure, including smooth surfaces and well-deﬁned edges, but the GLM-reconstructed images tended to have more realistic-looking textures, whereas the LNP-reconstructed images tended to be overly simpliﬁed.

Original: Moreover, the GLM method recovered more fine details (e.g., the legs of the insect in row C, the horizontal stripes on the tape cassette in row D, and the details on the hammock in row E, and the structure on the file cabinets in row G).
Revised: Moreover, the GLM method recovered more ﬁne details (e.g., the legs of the insect in row C, the horizontal stripes on the tape cassette in row D, and the details on the hammock in row E, and the structure on the ﬁle cabinets in row G).

Original: The quality of image reconstructions for each image-response pair in the test partition for one retina were compared using MS-SSIM and PSNR in Figure 3D, and their mean values over the test and held out partitions for both retinas are summarized in Tables 1 and 2.
Revised: The quality of image reconstructions for each image-response pair in the test partition for one retina were compared using MS-SSIM and PSNR in Figure 3D, and their mean values over the test and held out partitions for both retinas are summarized in Tables 1 and 2.

Original: In both retinas, images reconstructed using the GLM encoding model had systematically higher
Revised: In both retinas, images reconstructed using the GLM encoding model had systematically higher

Original: MS-SSIM scores, indicating greater perceptual similarity to ground truth, than those reconstructed using the LNP encoding model.
Revised: MS-SSIM scores, indicating greater perceptual similarity to ground truth, than those reconstructed using the LNP encoding model.

Original: This demonstrates that the choice of encoding model significantly affects reconstruction quality, and that the inclusion of temporal spike dependencies and cell-to-cell correlations in the more sophisticated GLM encoding model provides important constraints on the visual signal encoded by the RGC spikes.
Revised: This demonstrates that the choice of encoding model signiﬁcantly affects reconstruction quality, and that the inclusion of temporal spike dependencies and cell-to-cell correlations in the more sophisticated GLM encoding model provides important constraints on the visual signal encoded by the RGC spikes.

Original: This finding is consistent with previous work showing that decoding using the GLM (without priors) can access more information than simplified models lacking the cell-to-cell correlations or spiking history [10].
Revised: This ﬁnding is consistent with previous work showing that decoding using the GLM (without priors) can access more information than simpliﬁed models lacking the cell-to-cell correlations or spiking history [10].

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  Discussion
Revised:  Discussion

Original: We have presented a novel approximate MAP method for reconstructing natural images from the simultaneously recorded spikes of several hundred retinal ganglion cells, using an accurate probabilistic model of retinal encoding and a natural image prior implicit in a pre-trained denoising neural network.
Revised: We have presented a novel approximate MAP method for reconstructing natural images from the simultaneously recorded spikes of several hundred retinal ganglion cells, using an accurate probabilistic model of retinal encoding and a natural image prior implicit in a pre-trained denoising neural network.

Original: The method matches or outperforms the current state-of-the-art in terms of recovering naturalistic image structure and/or the perceptual similarity of reconstructions to ground truth, while also being more principled and interpretable due to the explicit Bayesian separation of the encoding model and prior.
Revised: The method matches or outperforms the current state-of-the-art in terms of recovering naturalistic image structure and/or the perceptual similarity of reconstructions to ground truth, while also being more principled and interpretable due to the explicit Bayesian separation of the encoding model and prior.

Original: The new approach uses substantially fewer parameters than previous state-of-the-art methods based on neural networks, and does not require training neural networks on retinal data (the prior is obtained from a network trained exclusively on image denoising).
Revised: The new approach uses substantially fewer parameters than previous state-of-the-art methods based on neural networks, and does not require training neural networks on retinal data (the prior is obtained from a network trained exclusively on image denoising).

Original: We showed that both encoding model and image prior contributed to the high-quality image reconstructions: simplification of either substantially degraded performance.
Revised: We showed that both encoding model and image prior contributed to the high-quality image reconstructions: simpliﬁcation of either substantially degraded performance.

Original: Thus, we expect that cell-to-cell correlations and temporal structure of spike trains, as well as sophisticated image priors, will prove important in understanding how the retinal signal is used by the brain.
Revised: Thus, we expect that cell-to-cell correlations and temporal structure of spike trains, as well as sophisticated image priors, will prove important in understanding how the retinal signal is used by the brain.

Original: Several previous studies have used GLM encoding models for stimulus reconstruction from experimentally-recorded retinal signals, revealing the significance of cell-to-cell correlations for decoding temporal structure in white noise stimuli [10, 38], and the significance of the temporal structure of spike trains in tracking moving features [2].
Revised: Several previous studies have used GLM encoding models for stimulus reconstruction from experimentally-recorded retinal signals, revealing the signiﬁcance of cell-to-cell correlations for decoding temporal structure in white noise stimuli [10, 38], and the signiﬁcance of the temporal structure of spike trains in tracking moving features [2].

Original: By including a complex natural image prior into a Bayesian reconstruction method, the present work more efficiently leverages both the GLM model and the experimental data to produce state-of-the-art natural image reconstructions.
Revised: By including a complex natural image prior into a Bayesian reconstruction method, the present work more efﬁciently leverages both the GLM model and the experimental data to produce state-of-the-art natural image reconstructions.

Original: The enhanced reconstructions and interpretability obtained with our method could lead to improved function of retinal implants for restoring vision.
Revised: The enhanced reconstructions and interpretability obtained with our method could lead to improved function of retinal implants for restoring vision.

Original: Previous work [5] has suggested that electrical stimulation with a retinal implant can be guided by minimizing the expected MSE of linearly reconstructed images.
Revised: Previous work [5] has suggested that electrical stimulation with a retinal implant can be guided by minimizing the expected MSE of linearly reconstructed images.

Original: This method ignored fine temporal structure in RGC spike trains, and assumed that image priors captured by linear regression are sufficient for high performance.
Revised: This method ignored ﬁne temporal structure in RGC spike trains, and assumed that image priors captured by linear regression are sufﬁcient for high performance.

Original: The method presented here offers an alternative approach to choosing simulation patterns to produce higherfidelity artificial vision, while potentially being more robust than neural network methods.
Revised: The method presented here offers an alternative approach to choosing simulation patterns to produce higherﬁdelity artiﬁcial vision, while potentially being more robust than neural network methods.

Original: However, achieving this in real time with minimal latency presents a substantial technical challenge.
Revised: However, achieving this in real time with minimal latency presents a substantial technical challenge.

Original: Our Bayesian reconstruction methodology relied on the convexity of the GLM encoding model, which allows for robust convergence.
Revised: Our Bayesian reconstruction methodology relied on the convexity of the GLM encoding model, which allows for robust convergence.

Original: It is worthwhile to consider generalizations that might be used to evaluate more complex encoding models that are non-convex in the stimulus, such as subunit models [39, 40, 41] or artificial neural network retinal models[42, 43].
Revised: It is worthwhile to consider generalizations that might be used to evaluate more complex encoding models that are non-convex in the stimulus, such as subunit models [39, 40, 41] or artiﬁcial neural network retinal models[42, 43].

Original: We are currently working on methods based on score-based sampling methods [44, 14] to answer this question.
Revised: We are currently working on methods based on score-based sampling methods [44, 14] to answer this question.

Original: Though the present work is limited to reconstruction of flashed static natural images from RGC spikes, extensions of the method could be used to probe how neurons encode visual information under more natural conditions.
Revised: Though the present work is limited to reconstruction of ﬂashed static natural images from RGC spikes, extensions of the method could be used to probe how neurons encode visual information under more natural conditions.

Original: For example, a central problem is understanding how the visual system achieves high-acuity perception in the presence of “jitter” in eye position, even during fixation [45].
Revised: For example, a central problem is understanding how the visual system achieves high-acuity perception in the presence of “jitter” in eye position, even during ﬁxation [45].

Original: Previous computational efforts have probed this question, but have been largely limited to simulated data with simple encoding models and stimuli [46, 47, 48].
Revised: Previous computational efforts have probed this question, but have been largely limited to simulated data with simple encoding models and stimuli [46, 47, 48].

Original: Combining the methods put forth here with modern algorithms for image deblurring and motion-correction [49, 15] could yield more powerful methods to decode images from jittered retinal inputs.
Revised: Combining the methods put forth here with modern algorithms for image deblurring and motion-correction [49, 15] could yield more powerful methods to decode images from jittered retinal inputs.

Original: A related problem is understanding how the retina encodes the information contained in complex naturalistic movies [2], including movement of objects within a scene and other non-rigid transformations over time.
Revised: A related problem is understanding how the retina encodes the information contained in complex naturalistic movies [2], including movement of objects within a scene and other non-rigid transformations over time.

Original: The dimensionality of such stimuli and the consequent data requirements are high, so the use of machine learning methods to capture spatio-temporal priors [50, 51] separately from the retinal data will be important for understanding reconstruction in these contexts.
Revised: The dimensionality of such stimuli and the consequent data requirements are high, so the use of machine learning methods to capture spatio-temporal priors [50, 51] separately from the retinal data will be important for understanding reconstruction in these contexts.

Original:  Funding
Revised:  Funding

Original: National Defense Science and Engineering Graduate (NDSEG) Fellowship (EGW), NSF IGERT 0801700 (NB), Wu Tsai Neurosciences Institute Big Ideas award, NSF CRCNS grant(EJC/EPS), and NIH grants EY017992 and EY029247 (EJC).A Appendix A.1 Potential negative impacts A potential negative impact of any animal research is use of the animals.
Revised: National Defense Science and Engineering Graduate (NDSEG) Fellowship (EGW), NSF IGERT 0801700 (NB), Wu Tsai Neurosciences Institute Big Ideas award, NSF CRCNS grant(EJC/EPS), and NIH grants EY017992 and EY029247 (EJC).

Original: In the present work, eyes were obtained from animals that were euthanized in the course of experiments already being performed by other laboratories.

Original: If the eyes had not been used for these recordings, they would have been discarded.

Original: We are unaware of any other significant potential negative impacts.

Original: A.2 Summary of experimental datasets

Original: [Table]

Original: [Table]

Original: A.3 Computational resources

Original: All computations were performed using Pytorch [52] on a single NVIDIA V100 GPU with 32 GB of

Original: VRAM.

Original: Fitting encoding models for all cells in a single retina required approximately 24 hours of computational time.

Original: A.4 Linear-nonlinear-Poisson (LNP) model and fitting

Original: The LNP response model for a single RGC is

Original: [Equation]

Original: [Equation]

Original: To fit this model to spike count data, we optimize the parameters m (the spatial stimulus filter) and b (the scalar bias) according to:

Original: [Equation]

Original: The first two terms correspond to the negative log-likelihood of the model of equation (9).

Original: The third term is an L1 sparsity-inducing penalty on the spatial filter m .

Original: The fourth term is an L2 penalty that induces similarity to m ′ , which is an initial fit of the spatial filter obtained using reverse correlation with white noise.

Original: Together, the two regularization terms help to ensure that the spatial filters are spatially compact, contiguous, and resemble the white noise receptive fields.

Original: The hyperparameters for this objective function were optimized by performing a grid search and evaluating the log-likelihood of the responses of a small number of RGCs of each type in test partition.

Original: Optimization problems were solved separately for each RGC using FISTA [26].

Original: A.5 Generalized linear model (GLM) and fitting

Original: The GLM model of RGCs augments the LNP model to include feedback and coupling filters to account for refractoriness, bursting, and cell-to-cell correlations [10].

Original: We make the following definitions and assumptions:

Original: • The spikes for each RGC are binned into T + 1 total time bins indexed as t ∈ [0 , 1 , . . .

Original: T − 1 , T ]

Original: The stimulus onset (transition from gray screen to presented image) occurs at time t = N .

Original: • The time bin durations are sufficiently brief (1 ms) that at most one spike per RGC can occur in any bin.

Original: • The spike train for the i th single RGC is denoted s i .

Original: The spike trains for a set of RGCs A is denoted s A , and the spike trains for the set of all RGCs except cell i is referred to as s { i } C • The visual stimulus within each trial is space-time separable, because each trial consists of a single statically-flashed image.

Original: In particular, the visual stimulus v [ x, y, t ] = w [ t ] x where x is the stimulus image, and w [ t ] is a boxcar corresponding to the time component of the stimulus.

Original: A.5.1 GLM with Bernoulli spiking

Original: We assume that the spatio-temporal stimulus filter for the RGC is space-time separable.

Original: The GLM for the i th cell thus the following parameters:

Original: • The spatial component of the stimulus filter

Original: m i • The temporal component of the stimulus

Original: filter h

Original: i [ t ] , assumed causal •

Original: The additive (bias) constant b i • A feedback (spike history) filter f i [ t ] , assumed causal • Coupling filters for the neighboring RGCs, c ( j ) i

Original: [ t ] , assumed causal.

Original: The set of cells coupled to cell i is denoted { i } C

Original: Approximate nearest-neighbor distances for each of the cell types (ON parasol, OFF parasol, ON midget, OFF midget) were computed based on the reverse-correlation receptive field centers.

Original: Coupled neighbors consisted of cells within a multiple of the median nearest neighbor distances for each respective cell type (2 times the median nearest neighbor distance for parasols, 2.5 times the median nearest neighbor distance for midgets).

Original: The generator signal of the i th RGC, which determines its instantaneous spiking probability, is defined as

Original: [Equation]

Original: Since filters h

Original: i , f i , and c ( j ) i are causal, g i [ t ] depends only on the stimulus and spikes that occur before time t .

Original: The generator signal is passed through a sigmoidal nonlinearity p [ t ] =

Original: exp { g i [ t ] } / (1 + exp { g i [ t ] } ) , and a spike is generated with this probability.

Original: [-The log-likelihood of observing s 
-]
Revised: [-The log-likelihood of observing s 
-]

Original: [ t ] spikes from cell i at time t given the stimulus and previously observed spikes is

Original: [Equation]

Original: A.5.2 Derivation of GLM log-likelihood objective function (equation (5))

Original: When fitting optimal GLM parameters for the i th RGC, the spikes from the coupled RGCs s { i } C [0 , . .

Original: , T ] are observed and known, and the spikes for RGC

Original: i that occur before time N s i [0 , . .

Original: , N − 1] are also known.

Original: The optimal GLM parameters for the

Original: i th RGC m i , h i , f i , c ( j ) i , and b i can be found by minimizing the negative log-likelihood (negative of equation (12)) over the parameters.

Original: The negative log-likelihood can be simplified using the chain rule:

Original: − log p ( s i [ N, . . .

Original: , T ] | x , s i [0 , . . .

Original: N − 1] , s { i } C [0 , . . .

Original: , T ])

Original: = −

Original: T (cid:88) t =

Original: N log p ( s i [ t ] | x , s i [0 , . .

Original: , t − 1] , s { i } C [0 , . . .

Original: , t − 1]) =

Original: T (cid:88) t = N (cid:8) log(1 + exp { g i [ t ] } ) − s i [ t ] g i [ t ] (cid:9) , which results in the expression presented in (5).

Original: A.5.3 Regularized GLM objective function

Original: To reduce the number of GLM model parameters, the stimulus time filter h i [ t ] , feedback (spike history) filter f i [ t ] , and the neighboring cell coupling filters c ( j ) i [ t ] were represented in terms of a small number of cosine bump basis vectors [10] (basis functions discussed in more detail in A.5.4).

Original: Letting B h , B f , and B c represent the basis matrices for stimulus time, feedback, and coupling filters, respectively, the full time-domain filters can be recovered from the low dimensional representations

Original: [Equation]

Original: Substituting these low-dimensional temporal filters into the encoding negative log-likelihood from equation (5) yields an expression for that is jointly convex in { m i ,

Original: ˜f i , ˜c ( j ) i , b i } and in { ˜h i , ˜f i ,

Original: ˜c ( j ) i , b i } but not with the stimulus spatial filter m and stimulus temporal filter ˜h i together because of the space-time stimulus filter separability assumption.

Original: As with the LNP models, L1 sparsity and L2 prior regularization terms were added.

Original: In addition, to eliminate spurious cell-cell correlations, an L 2 ,group-sparsity term was added to constrain the neighboring cell coupling filters ˜c ( j ) i , b i .

Original: Because of the joint convexity issue described above, we fitted GLMs by alternating between solving two convex minimization problems:

Original: [Equation]

Original: and arg

Original: [Equation]

Original: Each of the problems were solved using FISTA [26] with the proximal operator for the L 2 , 1 penalty from [29].

Original: A.5.4 GLM time basis functions

Original: Proper selection of temporal basis functions for the stimulus time filter, spiking feedback filter, and coupling filters is necessary for well-fitted GLM models.

Original: As in [10], basis functions corresponding to the first lobe of

Original: [Equation]

Original: (13) were used.

Original: The quantities, a , c , and ϕ l , as well as the number of basis functions used are hyperparameters.

Original: Because the GLM was fit with a 1 ms bin width and the filters were 250 samples long, t takes values 0 , 1 , ..., 249 .

Original: The paper used the following basis hyperparameters:

Original: • For the stimulus time filter, a = 5 . 5 , c = 1 . 0 , and 10 total basis functions were used, corresponding to ϕ l ∈ { 8 π 2 , 9 π 2 , ..., 16 π 2 , 17 π 2 } .

Original: • For the feedback filter, a = 5 . 5 , c = 1 . 0 , and 18 total basis functions were used, corresponding to ϕ l ∈ { 0 , π 2 , ..., 16 π 2 , 17 π 2 } .

Original: • For the coupling filter, a = 3 . 2 , c = 1 . 0 , and 10 total basis functions were used, corresponding to ϕ l ∈ { 0 , π 2 , ..., 9 π 2 } .

Original: The basis set for the stimulus time filter was uniformly shifted backward in time to approximately match each retina’s response latency to the stimulus.

Original: A.5.5 Derivation of the total encoding log-likelihood term (equation (6))

Original: The log-likelihood of the spike trains of all of the RGCs given the stimulus image and the spikes occurring prior to presentation of the stimulus can be computed using the chain rule.

Original: In particular, the total log-likelihood over all of the cells corresponding to time bin t

Original: [Equation]

Original: can be computed simply by summing over the cells, since the spiking responses in time bin t for each cell are independent of the other cells given the image and the spike histories for every cell up until time t − 1 .

Original: The overall log-likelihood corresponding to time bins N, N + 1 , ..., T can then be computed using the chain rule

Original: [Equation]

Original: i ∈ cells log p ( s i [ t ] | x , s i [0 , ..., t − 1] , s { i } C [0 , ..., t − 1])

Original: A.6 Initialization of HQS for MAP-GLM-dCNN method

Original: We tested different methods of initializing z (1) in Algorithm 1 for the MAP-GLM-dCNN.

Original: We compared initialization with the linear solution (as used for all results shown in the main text), with

Original: Gaussian noise initialization with standard deviations σ ∈ { 10 − 4 , 10 − 3 , 10 − 2 , 10 − 1 , 1 . 0 } , where images are defined on the interval [ − 1 , 1] .

Original: Example test reconstructions for each initialization are shown in Figure 4, and mean test MS-SSIM and PSNR values for each initialization method are summarized in Tables 6 and 7.

Original: Qualitatively, the content and structure of the reconstructions did not show significant dependence on the initialization (with exception of the horizontal antenna of the insect in row C).

Original: However, the mean luminance of the image reconstructions did depend on the initialization method.

Original: Specifically, initialization with the linear solution tended to better recover the mean luminance (rows C, J, N) in cases where the stimulus image was particularly bright or particularly dark.

Original: Quantitatively, mean test and heldout MS-SSIM was largely independent of the initialization method (Table 6), while PSNR was worse for the randomly-initialized reconstructions (Table 7).

Original: A

Original: Ground truth

Original: Linear init =

Original: 1e- =

Original: 1e- =

Original: 1e- =

Original: 1e- =1e+

Original: B

Original: C

Original: D

Original: E

Original: F

Original: G

Original: H

Original: I

Original: J

Original: K

Original: L

Original: M

Original: N

Original: [Table]

Original: [Table]

Original: A.7 Implementation details for L-CAE and performance on experimental data

Original: We used the published L-CAE model architecture [8].

Original: This consisted of a linear regression decoder with parameters fitted by solving the normal equations, followed by a 4-layer convolutional encoder and 4-layer convolutional decoder to improve the linear reconstructions.

Original: The convolutional encoder and decoder were trained with backpropagation using a masked MSE loss (including only the regions of the image covered by recorded RGCs), the Adam optimizer [53] with learning rate 4 · 10 − 3 , and batch size 32.

Original: The number of training epochs was determined by evaluating the masked MSE loss on the test partition of the dataset.

Original: The results in [8] were based on simulated RGC responses, and to our knowledge there are no published examples applying the L-CAE technique to experimentally recorded data.

Original: To verify that the L-CAE performs well with experimental data, we compared the performance of L-CAE and linear regression on the test partition (Figure 5).

Original: As expected, the L-CAE systematically produces improved PSNR, relative to linear regression.

Original: It also produces more perceptually accurate image reconstructions.

Original: A.8 Implementation details for Kim et al. linear/nonlinear regression benchmark

Original: The Kim et al. paper [4] reconstructs images from RGC spikes by breaking up the problem into three sequential ad hoc steps.

Original: The target image is decomposed into low spatial frequency and high spatial frequency components.

Original: The low-frequency component is reconstructed using learned reconstruction filters, while the high-frequency component is reconstructed using a trained fully-connected neural network mapping the responses of the most relevant RGCs to the desired high-frequency pixel values.

Original: Finally, the low-frequency and high-frequency components are summed together and passed through a deblurring network with an architecture based on that of DeblurGANv2 [54].

Original: The linear reconstruction filter was fitted by solving an L1-regularized linear-least-squares problem aiming to reconstruct the low-frequency image component, where the L1 hyperparameter was chosen by performing a grid search and evaluating MSE on the test partition.

Original: Cell/variable selection for the pixel-wise high-frequency component neural network was done by performing a separate

Original: L1-regularized linear-least-squares problem aiming to reconstruct the full image, with the L1 hyperparameter again chosen using grid search on the test partition.

Original: As in the published paper, the top 25 RGCs for each pixel were selected as inputs to the pixel-wise high-pass neural networks by taking the cells with the largest absolute reconstruction filter value for each respective pixel.

Original: The high-frequency neural networks were trained with using backpropagation with an MSE loss with respect to the high-frequency component of the target image.

Original: The DeblurGANv2 final deblur network was trained using backpropagation with an L1 loss with respect to the target image as well as an L perceptual similarity loss computed using the VGG [55] network.

Original: To verify our implementation of the Kim et al.

Original: method, we compared reconstruction quality for images produced using this method against linear regression (Figure 6).

Original: As expected, Kim et al.

Original: systematically produces better MS-SSIM and PSNR than linear regression.

Original: [Table]

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] We have made comparisons to the appropriate benchmarks, and have described how the particular image reconstruction task differs from true natural vision.

Original: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] Potential positive impacts of the work regarding retinal prostheses for treating blindness are discussed in the third paragraph of the Discussion section 5.

Original: Because the work is specific to the retina, the negative impacts are limited and discussed in the Appendix A. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] Code and fitted models are provided in the supplemental material, and a small subset of the experimental dataset used to generate the example figures is also provided.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] Descriptions of hyperparameters and how they were selected are discussed in the methods.

Original: The train/test/heldout data partitions and descriptions of the datasets are provided in Tables 4 and 5, respectively.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] Mentioned in the Appendix 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?

Original: [No] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]