Original: A NEW PHOTORECEPTOR - INSPIRED CNN LAYER ENABLES DEEP LEARNING MODELS OF RETINA TO GENERALIZE ACROSS LIGHTING CONDITIONS
Revised: A NEW PHOTORECEPTOR - INSPIRED CNN LAYER ENABLES DEEP LEARNING MODELS OF RETINA TO GENERALIZE ACROSS LIGHTING CONDITIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: As we move our eyes, and as lighting changes in our environment, the light intensity reaching our retinas changes dramatically and on multiple timescales.
Revised: As we move our eyes, and as lighting changes in our environment, the light intensity reaching our retinas changes dramatically and on multiple timescales.

Original: Despite these changing conditions, our retinas effortlessly extract visual information that allows downstream brain areas to make sense of the visual world.
Revised: Despite these changing conditions, our retinas effortlessly extract visual information that allows downstream brain areas to make sense of the visual world.

Original: Such processing capabilities are desirable in many settings, including computer vision systems that operate in dynamic lighting environments like in self-driving cars, and in algorithms that translate visual inputs into neural signals for use in vision-restoring prosthetics.
Revised: Such processing capabilities are desirable in many settings, including computer vision systems that operate in dynamic lighting environments like in self-driving cars, and in algorithms that translate visual inputs into neural signals for use in vision-restoring prosthetics.

Original: To mimic retinal processing, we first require models that can predict retinal ganglion cell (RGC) responses reliably.
Revised: To mimic retinal processing, we first require models that can predict retinal ganglion cell (RGC) responses reliably.

Original: While existing state-of-the-art deep learning models can accurately predict RGC responses to visual scenes under steady-state lighting conditions, these models fail under dynamic lighting conditions.
Revised: While existing state-of-the-art deep learning models can accurately predict RGC responses to visual scenes under steady-state lighting conditions, these models fail under dynamic lighting conditions.

Original: This is because changes in lighting markedly alter RGC responses: adaptation mechanisms dynamically tune RGC receptive fields on multiple timescales.
Revised: This is because changes in lighting markedly alter RGC responses: adaptation mechanisms dynamically tune RGC receptive fields on multiple timescales.

Original: Because current deep learning models of the retina have no in-built notion of light level or these adaptive mechanisms, they are unable to accurately predict RGC responses under lighting conditions that they were not trained on.
Revised: Because current deep learning models of the retina have no in-built notion of light level or these adaptive mechanisms, they are unable to accurately predict RGC responses under lighting conditions that they were not trained on.

Original: We present here a new deep learning model of the retina that can predict RGC responses to visual scenes at different light levels without requiring training at each light level.
Revised: We present here a new deep learning model of the retina that can predict RGC responses to visual scenes at different light levels without requiring training at each light level.

Original: Our model combines a fully trainable biophysical front end capturing the fast and slow adaptation mechanisms in the photoreceptors with convolutional neural networks (CNNs) capturing downstream retinal processing.
Revised: Our model combines a fully trainable biophysical front end capturing the fast and slow adaptation mechanisms in the photoreceptors with convolutional neural networks (CNNs) capturing downstream retinal processing.

Original: We tested our model’s generalization performance across light levels using monkey and rat retinal data.
Revised: We tested our model’s generalization performance across light levels using monkey and rat retinal data.

Original: Whereas conventional CNN models without the photoreceptor layer failed to predict RGC responses when the lighting conditions changed, our model with the photoreceptor layer as a front end fared much better in this challenge.
Revised: Whereas conventional CNN models without the photoreceptor layer failed to predict RGC responses when the lighting conditions changed, our model with the photoreceptor layer as a front end fared much better in this challenge.

Original: Overall, our work demonstrates a new hybrid approach that equips deep learning models with biological vision mechanisms enabling them to adapt to dynamic environments.
Revised: Overall, our work demonstrates a new hybrid approach that equips deep learning models with biological vision mechanisms enabling them to adapt to dynamic environments.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: A key problem in visual neuroscience is to generate models that can accurately predict how neurons will respond to visual stimuli.
Revised: A key problem in visual neuroscience is to generate models that can accurately predict how neurons will respond to visual stimuli.

Original: Along with their role in basic neuroscience, these models have applications in prosthetic devices and can form the basis for bio-inspired computer vision systems that aim to mimic the impressively robust functions of the human visual system.
Revised: Along with their role in basic neuroscience, these models have applications in prosthetic devices and can form the basis for bio-inspired computer vision systems that aim to mimic the impressively robust functions of the human visual system.

Original: Given their strong performance in other visual tasks like object recognition (Chollet, 2017; Simonyan & Zisserman, 2015; 
Revised: Use of machine learning models for such applications in neuroscience has become increasingly ubiquitous given their strong performance in computer vision applications like object recognition (Chollet, 2017; Simonyan & Zisserman, 2015; Krizhevsky et al., 2017).

Original: Krizhevsky et al., 2017), many recent studies trained convolutional neural networks (CNNs) to predict responses of neurons in visual cortex (Kindel et al., 2019; Cadena et al., 2017) and retina (McIntosh et al., 2016; Tanaka et al., 2019; Yan et al., 2022; Goldin et al., 2022) to visual stimuli.
Revised: For example using convolutional neural networks (CNNs) to predict responses of neurons in visual cortex (Kindel et al., 2019; Cadena et al., 2017) and retina (McIntosh et al., 2016; Tanaka et al., 2019; Yan et al., 2022; Goldin et al., 2022) to visual stimuli.

Original: We focus here on  retina.
Revised: We focus here on the retina.

Original: Under carefully controlled experimental conditions,  with constant lighting conditions, CNN models can predict responses of retinal ganglion cells (the “output” cells of the retina, whose axons form the optic nerve) to visual stimuli with high accuracy (McIntosh et al., 2016).
Revised: Under carefully controlled experimental conditions, and with constant lighting conditions, CNN models can predict responses of retinal ganglion cells (RGCs, the “output” cells of the retina, whose axons form the optic nerve) to visual stimuli with high accuracy (McIntosh et al., 2016).

Original: In natural vision conditions, however, lighting conditions are highly dynamic: the amount of light falling on the retina can change by several orders of magnitude at multiple timescales.
Revised: In natural vision, however, lighting conditions are highly dynamic: the amount of light falling on the retina can change by several orders of magnitude at mul- tiple timescales.

Original: For example, light input can change locally on a region of the retina  in less than a second following rapid eye movements such as saccades; global light levels may fluctuate by an order of magnitude every few seconds on a cloudy day, as clouds pass between the observer and the sun.
Revised: For example, light input can change locally on a region of the retina by an order of magnitude in less than a second following rapid eye movements such as saccades; global light levels may fluctuate by an order of magnitude every few seconds on a cloudy day, as clouds pass between the observer and the sun.

Original: These changes in light level substantially alter RGC responses to visual stimuli (Tikidji-Hamburyan et al., 2015; Ruda et al., 2020; Idrees et al., 2020; 2022;
Revised: These changes in light level substantially alter RGC responses to visual stimuli (Tikidji-Hamburyan et al., 2015; Ruda et al., 2020; Idrees et al., 2020; 2022;

Original: Farrow et al., 2013), and pose the as-of-yet-unanswered question of whether CNNs can accurately predict RGC responses under these more challenging conditions.
Revised: Farrow et al., 2013), and pose the as-of-yet-unanswered question of whether CNNs can accurately predict RGC responses under these more challenging conditions.

Original: To answer this question, we first tested the ability of conventional CNN models (i.e., the Deep
Revised: To answer this question, we first tested the ability of conventional CNN models (i.e., the Deep

Original: Retina model of (McIntosh et al., 2016)) to predict RGC responses in the presence of changing light levels.
Revised: Retina model of (McIntosh et al., 2016)) to predict RGC responses in the presence of changing light levels.

Original: These CNNs could accurately predict RGC responses when tested under the same lighting conditions under which they were trained. However, when tested under different lighting conditions, they were unable to accurately predict RGC responses.
Revised: These CNNs could accurately predict RGC responses when tested under the same lighting conditions under which they were trained but were unable to accurately predict responses under different lighting conditions.

Original: To address this shortcoming and create new machine learning (ML) models that can predict RGC responses under varied lighting conditions, we created a new type of convolutional neural network layer as a front-end for CNNs.
Revised: To improve the performance of machine learning (ML) models in predicting RGC responses under varied lighting conditions, we created a new type of convolutional neural network layer as a frontend for CNNs.

Original: This (fully trainable) input layer mimics the transformation of light into electrical signals at the retina’s photoreceptors, including the adaptation mechanisms that modulate photoreceptor response sensitivity and kinetics that enable the retina to operate under a wide and dynamic input range.
Revised: This fully trainable input layer mimics the transformation of light into electrical signals at the retina’s photoreceptors, including the adaptation mechanisms that modulate photoreceptor response sensitivity and kinetics and by doing so enable the retina to operate under a wide and dynamic input range.

Original: This in-built adaptation allows the model to adapt to changing light levels locally, at each pixel location in the input, the same way our retinas do.
Revised: This in-built adaptation allows the model to adapt to changing light levels locally, at each pixel location in the input, the same way our retinas do.

Original: With this new input layer, we found that our CNNs could generalize between light levels in ways that conventional CNNs (without our photoreceptor-inspired input layer) cannot: our photoreceptor-inspired CNN models can accurately predict RGC responses under test lighting conditions very different from the lighting conditions under which they were trained.
Revised: CNNs with this new input layer could generalize to test lighting conditions very different from the lighting conditions under which they were trained.

Original: We anticipate that our new photoreceptor-inspired CNN layer could have several significant impacts on visual neuroscience and computer vision.
Revised: We anticipate that our new photoreceptor-inspired CNN layer could have several significant impacts in various fields of vision science.

Original: First, our model can be used directlyto translate visual inputs into neural responses that can then be stimulated inthe eyes of blind individuals to create vision-restoring prosthetics. By enabling the translator algorithms to function under diverse lighting conditions, our work could dramatically improve the operating range of these prosthetics. 
Revised: First, our model can be used directly in vision-restoring prosthetics by enabling the translator algorithms to function under diverse lighting conditions, and thereby improve the operating range of these prosthetics. Second, our model can be used by visual neuroscientists to investigate dynamic retinal computations.

Original: Second, our models can be used as a front-end for algorithms that predict the responses of neurons in visual cortex, and thus enable those cortical models to operate under more naturalistic and more varied lighting conditions.
Revised: Using the model as a front-end for algorithms that predict the responses of neurons in visual cortex, will enable those cortical models to operate under more naturalistic and more varied lighting conditions.

Original: Finally, because their local (pixel-by-pixel) adaptation to changing light levels can filter out image changes due to changing lighting and shadow, our photoreceptor-inspired CNNs could have substantial application for computer vision systems operating in outdoor environments.
Revised: Finally, because their local (pixel-bypixel) adaptation to changing light levels can filter out image changes due to changing lighting and shadow, our photoreceptor-inspired CNNs could have substantial application for computer vision systems operating in outdoor environments.

Original: There, lighting changes and shadows are known to confuse existing object detection and recognition algorithms (Yadron & Tynan, 2016; Levin, 2018;Janai et al., 2020; Gomez-Ojeda et al., 2015;
Revised: There, lighting changes and shadows are known to confuse existing object detection and recognition algorithms (Yadron & Tynan, 2016; Levin, 2018;

Original:  Kolaman et al., 2019) because those algorithms do not naturally separate image changes due tostrong local lighting variations from image changes due to changing object content.
Revised: Janai et al., 2020; Gomez-Ojeda et al., 2015; Kolaman et al., 2019) as they do not naturally separate image changes due to local lighting variations from image changes due to changing object content.

Original: To reinforce this last point, we include a proof-of-concept demonstration that photoreceptor-inspired CNN models can discount disruptive events in a video stream, such as sudden large changes in the amount of light reflected off a tracked object due to shadows or changes in intensity of the light source.
Revised: Here, photoreceptor-inspired CNN models can discount disruptive events in a video stream, such as sudden large changes in the amount of light reflected off a tracked object due to shadows or changes in intensity of the light source.

Original:  R ELATED WORK 2. D EEP LEARNING MODELS OF THE VISUAL SYSTEM
Revised:  R ELATED WORK 2. D EEP LEARNING MODELS OF THE VISUAL SYSTEM

Original: Deep learning approaches have become increasingly popular in modeling the visual system as they can describe neural responses to a visual scene markedly more accurately than linear-nonlinear (LN) models.
Revised: Deep learning approaches have become increasingly popular in modeling the visual system as they can describe neural responses to a visual scene markedly more accurately than linear-nonlinear (LN) models.

Original: The current state-of-the-art retina predictor, called Deep Retina (McIntosh et al., 2016), is based on a 2-layer CNN.
Revised: The current state-of-the-art retina predictor, called Deep Retina (McIntosh et al., 2016), is based on a 2-layer CNN.

Original: This model takes as input a movie from which spatio-temporal features are extracted by the CNN layers, and outputs the spike rate of retinal ganglion cells.
Revised: This model takes as input a movie from which spatio-temporal features are extracted by the CNN layers, and outputs the spike rate of retinal ganglion cells.

Original: This model, when trained to predict tiger salamander retinal ganglion cell responses to spatiotemporal white noise stimuli, outperformed other models such as linear-nonlinear (LN) and the generalized linear models (GLMs).
Revised: This model, when trained to predict tiger salamander retinal ganglion cell responses to spatiotemporal white noise stimuli, outperformed other models such as LN and the generalized linear models (GLMs).

Original: Such CNN based models of the retina have been used for describing responses to natural stimuli (McIntosh et al., 2016; Tanaka et al., 2019) and for explaining the underlying neural computations that lead to ganglion cell responses (Tanaka et al., 2019; Yan et al., 2022;
Revised: Such CNN based models of the retina have been used for describing responses to natural stimuli (McIntosh et al., 2016; Tanaka et al., 2019) and for explaining the underlying neural computations that lead to RGC responses (Tanaka et al., 2019; Yan et al., 2022;

Original: Goldin et al., 2022).
Revised: Goldin et al., 2022).

Original: CNN based models can also better capture the activity of other visual areas such as the primary visual cortex (V1), than standard LN models (Kindel et al., 2019; Cadena et al., 2017).
Revised: CNN based models can also better capture the activity of other visual areas such as the primary visual cortex (V1), than standard LN models (Kindel et al., 2019; Cadena et al., 2017).

Original: While these models have been shown to reliably predict responses to visual stimuli having similar image statistics as the training set, it is unclear whether they will generalize to stimuli with different image statistics, such as different ambient luminance levels.
Revised: While these models have been shown to reliably predict responses to visual stimuli having similar image statistics as the training set, it is unclear whether they will generalize to stimuli with different image statistics, such as different ambient luminance levels.

Original: 2. H OW THE VISUAL SYSTEM FUNCTIONS IN DYNAMIC AMBIENT LIGHT CONDITIONS
Revised: 2. H OW THE VISUAL SYSTEM FUNCTIONS IN DYNAMIC AMBIENT LIGHT CONDITIONS

Original: The light falling on our retina spans over twelve orders of magnitude and can change at different timescales: local changes on retina can occur within a few hundred milliseconds as we make eye movements such as saccades or by shadows cast on a tracked object whereas slower global changes can occur on a cloudy day.
Revised: The light falling on our retina spans over twelve orders of magnitude and can change at different timescales: local changes on retina can occur within a few hundred milliseconds as we make eye movements such as saccades or by shadows cast on a tracked object whereas slower global changes can occur on a cloudy day.

Original: While extracting features under such dynamic conditions is still a challenge for artificial vision systems(Janai et al., 2020; Gomez-Ojeda et al., 2015; Kolaman et al., 2019), our retina effortlessly extracts reliable information for the downstream areas to make sense of our visual world (Tikidji-Hamburyan et al., 2015; Idrees et al., 2020; 2022; Goldin et al., 2022; Schreyer & Gollisch, 2021; Hosoya et al., 2005; Kastner & Baccus, 2011).
Revised: While extracting features under such dynamic conditions is still a challenge for artificial vision systems (Janai et al., 2020; Gomez-Ojeda et al., 2015; Kolaman et al., 2019), our retina effortlessly extracts reliable information for the downstream areas to make sense of our visual world (Tikidji-Hamburyan et al., 2015; Idrees et al., 2020; 2022; Goldin et al., 2022; Schreyer & Gollisch, 2021; Hosoya et al., 2005; Kastner & Baccus, 2011).

Original: This is achieved in part by signal processing done by photoreceptors (Angueyra et al., 2021; Yu et al., 2022; Clark et al., 2013), the very first cells in the visual processing hierarchy.
Revised: This is achieved in part by signal processing done by photoreceptors (Angueyra et al., 2021; Yu et al., 2022; Clark et al., 2013), the very first cells in the visual processing hierarchy.

Original: They adapt strongly to match their neural sensitivity to the ambient light conditions.
Revised: They adapt strongly to match their neural sensitivity to the ambient light conditions.

Original: This adaptation itself is highly dynamic, and occurs at multiple timescales.
Revised: This adaptation itself is highly dynamic, and occurs at multiple timescales.

Original: As a result, the photoreceptor dynamics affects both the gain and the kinetics with which light inputs are converted into electrical signals in the retina.
Revised: As a result, the photoreceptor dynamics affects both the gain and the kinetics with which light inputs are converted into electrical signals in the retina.

Original: This adaptation is also highly local: each photoreceptor can adapt independently.
Revised: This adaptation is also highly local: each photoreceptor can adapt independently.

Revised: Biophysical models of phototransduction in photoreceptors that translate light input into electrical signals are well established (Lamb & Jr, 1992; Rieke & Baylor, 1998; Invergo et al., 2014), and faithfully capture a range of responses.

Original: A newly-developed biophysical model (Angueyra et al., 2021) of cone photoreceptors, describing the phototransduction signalling cascade, faithfully captures these adaptive mechanisms.
Revised: A more recent model by Angueyra et al. (2021) incorporates adaptive changes in the photoreceptor output in conditions where light falling on the retina changes by large magnitudes rapidly.

Original: We implemented this signalling cascade as a fully-trainable neural network layer, and used it as the input to deep CNNs that could predict RGC responses to visual stimuli.
Revised: We implemented this signalling cascade as a fully-trainable neural network layer, and used it as the first layer to deep CNNs to predict RGC responses to visual stimuli.

Original:  M ETHODS 3. P HOTORECEPTOR LAYER
Revised:  M ETHODS 3. P HOTORECEPTOR LAYER

Original: Our new photoreceptor CNN layer builds upon a biophysical model of the phototransduction cascade  Angueyra et al.
Revised: Our new photoreceptor CNN layer builds upon a biophysical model of the phototransduction cascade by Angueyra et al.

Original: (2021) that incorporates the various feedforward and feedback molecular processes that convert photons into electrical signals, and therefore faithfully captures the photoreceptors’ adaptation mechanisms.
Revised: (2021) that incorporates the various feedforward and feedback molecular processes that convert photons into electrical signals, and therefore faithfully captures the photoreceptors’ adaptation mechanisms (Fig. 1).

Original: For example, the model captures the dynamic gain adjustment that matches the sensitivity of the photoreceptors to the prevailing light level (Fig. 1).

Original: We implemented this biophysical model as a fully-trainable neural network layer using Keras (Chollet et al., 2015) in Python.
Revised: We implemented this biophysical model as a fully-trainable neural network layer in Keras (Chollet et al., 2015) in Python.

Original: This layer converts time-varying light intensity signal in units of rhodopsin isomerizations per rod per second (R*rod − 1 s − 1 ) into photocurrents (pA) by solving the differential equations that underlie the phototransduction cascade (biophysical model from Angueyra et al.
Revised: This layer converts time-varying light intensity signal in units of receptor activations per photoreceptor per second (R*receptor − 1 s − 1 ) into photocurrents (pA) by solving the differential equations that underlie the phototransduction cascade (biophysical model from Angueyra et al.

Original: (2021) reproduced in Appendix A).
Revised: (2021) reproduced in Appendix A).

Original: The photoreceptor layer has twelve parameters corresponding to the twelve interaction parameters of the biophysical model, all of which can (if needed) be trained through backpropagation using the  TensorFlow package in Python.
Revised: The photoreceptor layer has twelve parameters corresponding to the twelve interaction parameters of the biophysical model, all of which can (if needed) be trained through backpropagation using the Keras and TensorFlow package in Python.

Original: We fixed 8 parameters to their values from photoreceptor data fits (Angueyra et al., 2021).
Revised: We fixed 8 parameters to their values from photoreceptor data fits.

Revised: These parameters relate to molecular processes such as the concentration of cyclic guanosine monophosphate (cGMP) in darkness and other factors governing cGMP conversion into current, and are similar for both rod and cone photoreceptors.

Original: We set the remaining 4 parameters, governing the scaling factors andtime constants for the various molecular processes in the model, to be trainable.
Revised: The remaining 4 parameters, set to be trainable, include the rate and the gain factors of the molecular processes such as the opsin activation, the phosphodiesterase activation and decay, and the rate of calcium removal.

Original:  In the current version of our model, these parameters are shared by all the input pixels: I.e.,a photoreceptor model with thesame parameter values is used at each pixel location. The conversion of each pixel into photocurrents only depends on that pixel’s previous values and not on the values of the other pixels.
Revised: These trainable parameters differ across photoreceptor types. In the current version of our model, the photoreceptor parameters are shared by all the input pixels: I.e., the conversion of each pixel into photocurrents only depends on that pixel’s previous values and not on the values of the other pixels.

Original: 3. M ODEL ARCHITECTURE
Revised: 3. M ODEL ARCHITECTURE

Original: We used our new photoreceptor layer at the input stage of a deep CNN whose architecture is shown in Fig.
Revised: We used our new photoreceptor layer at the input stage of a deep CNN (Fig. 2).

Original: The general architecture of our deep CNN was similar to Deep Retina (McIntosh et al., 2016), which we describe briefly here.
Revised: The general architecture of our deep CNN was similar to Deep Retina (McIntosh et al., 2016), which we describe briefly here.

Original: In Deep Retina, the model input was a movie which passed through a convolutional layer operating in both space and time, followed by another convolutional layer that only operated in the spatial dimension.
Revised: In Deep Retina, the model input was a movie which passed through a convolutional layer operating in both space and time, followed by another convolutional layer that only operated in the spatial dimension.

Original: The output of the final dense layerof Deep Retina was the predicted spike rate of RGCs.
Revised: The output of the final dense layer was the predicted spike rate of RGCs.

Original: Our model differs from Deep Retina in two key ways: 1) the addition of the photoreceptor layer at the input stage; and 2) the use ofnormalization layers (layer norm and batch norm), which were not included in Deep Retina.
Revised: Our model differs from Deep Retina in two key ways: 1) the addition of the photoreceptor layer at the input stage; and 2) the use of batch normalization layers.

Original: We performed some experiments on models in which the photoreceptor layer was removed, and  in which the normalization layers were removed.
Revised: We performed some experiments on models in which the photoreceptor layer was removed, and some in which the normalization layers were removed.

Original: These latter experiments provide the closest comparisons to Deep Retina.
Revised: These latter experiments provide the closest comparisons to Deep Retina.

Original: Similar to Deep Retina, our model takes as input a movie (180 frames per training example where each frame corresponds to 8 ms) and outputs an instantaneous spike rate for each RGC at the end of that movie segment.
Revised: Similar to Deep Retina, our model takes as input a movie (180 frames per training example where each frame corresponds to 8 ms) and outputs an instantaneous spike rate for each RGC at the end of that movie segment.

Original: To obtain the time series of RGC responses to longer movie stimuli, we feed into the model many 180-frame video samples taken from that longer movie, that correspond to 1-frame shifts.
Revised: To obtain the time series of RGC responses to longer movie stimuli, we feed into the model many 180-frame video samples taken from that longer movie, that correspond to 1-frame shifts.

Original: I.e., the model receives as inputs frames 1–180, 2–181, 3–182, etc., and outputs
Revised: I.e., the model receives as inputs frames 1–180, 2–181, 3–182, etc., and outputs

Original: RGC responses at the times of movie frames 180, 181, 182, etc.
Revised: RGC responses at the times of movie frames 180, 181, 182, etc.

Original: The photoreceptor layer of our model,which is the first layer, converts each pixel of the input movie (where each pixel is in R*rod − 1 s − 1 ) into photocurrents (pA), based on the history of the pixel’s value.
Revised: The photoreceptor layer of our model, converts each pixel of the input movie (where each pixel is in R*receptor − 1 s − 1 ) into photocurrents (pA), based on the history of the pixel’s value.

Original: The first 60 time points are then discarded to account for edge effects.
Revised: The first 60 time points are then discarded to account for edge effects.

Original: The output of this layer is a movie that is 120 frames long, and the same spatial dimensions as the input visual stimuli.
Revised: The output of this layer is a movie that is 120 frames long, and the same spatial dimensions as the input visual stimuli.

Original: This output gets normalized  using a LayerNorm layer and is then passed through a conventional 3-layer CNN, where the first layer is a 3D convolutional layer operating in both the spatial and temporal dimensions.All the temporal information from the movie is extracted by this layer as the temporal dimension of the convolutional filter is the same as the temporal dimension of the movie (120 in this case).
Revised: This photocurrents movie is then normalized by its mean and variance using a Layer Normalization layer (LayerNorm). The resulting movie is then passed through a conventional 3-layer CNN, where the first layer is a 3D convolutional layer operating in both the spatial and temporal dimensions.

Original: The output of the 3D convolutional layer isthus a 2D image which is normalized using Batch Normalization  and then passed through a rectifying (ReLU) nonlinearity.
Revised: The output of the 3D convolutional layer is a 2D image which is normalized using Batch Normalization (BatchNorm) and then passed through a rectifying (ReLU) nonlinearity.

Original: To reduce the spatial dimensions, we applied a 2D max pool operation  of size 2 after this layer.
Revised: To down sample the spatial dimensions, we applied a 2D max pool operation that took the maximum value over 2x2 patches of the previous layer’s output.

Original: The subsequent 2D CNN layers are followed by a final, fully connected layer that outputs the predicted spike rate for each RGC in the dataset.
Revised: The subsequent 2D CNN layers are followed by a final, fully connected layer that outputs the predicted spike rate for each RGC in the dataset.

Original: We refer to this model as the photoreceptor-CNN model, or PR-CNN model for short.
Revised: We refer to this model as the photoreceptor-CNN model, or PR-CNN model for short.

Original: Model parameters were optimized using Adam (Kingma & Ba, 2015), where the loss function was given by the negative log-likelihood under Poisson spike generation.
Revised: Model parameters were optimized using Adam (Kingma & Ba, 2015), where the loss function was given by the negative log-likelihood under Poisson spike generation.

Original:  The network layers were regularized with an L 2 weight penalty at each layer, including the photoreceptor layer,  and an L 1activity penalty at the final layer.
Revised: Photoreceptor parameters were initialized to their known values from fits to experimental data. This initialization helped stabilize the joint training of the photoreceptor model with the downstream layers. The network layers were regularized with a L 2 weight penalty at each layer, including the photoreceptor layer, to prevent loss of information and be more robust to outliers. In addition, a L 1 penalty was applied to the output of the dense layer because the neural activity itself is relatively sparse and L 1 penalties are known to induce sparsity.

Original: The number of channels in each CNN layer and the filter sizes were optimized by a grid search for each model type.
Revised: The number of channels in each CNN layer and the filter sizes were optimized by a grid search for each model type.

Original: These details therefore differed for the models with and without the photoreceptor layer or normalization layers.
Revised: These details therefore differed for the models with and without the photoreceptor layer or normalization layers.

Original: Learning rates were set to 0.001.
Revised: Learning rates were set to 0.001.

Original: 3. R ETINA ELECTROPHYSIOLOGY
Revised: 3. R ETINA ELECTROPHYSIOLOGY

Original: To develop and test our models of the retina, we used data from ex vivo macaque monkey and rat retina electrophysiology experiments.
Revised: To develop and test our models of the retina, we used data from ex vivo macaque monkey and rat retina electrophysiology experiments.

Original: The rat data was previously published by Ruda et al.
Revised: The rat data was previously published by Ruda et al.

Original: (2020), whereas the monkey data were collected by us as part of this study.
Revised: (2020), whereas the monkey data were collected by us as part of this study.

Original: The basic experimental approach was the same in both sets of experiments: the retina of an euthanized animal was extracted and placed with the RGC-side down on a multi-electrode array (MEA).
Revised: The basic experimental approach was the same in both sets of experiments: the retina of an euthanized animal was extracted and placed with the RGC-side down on a multi-electrode array (MEA).

Original: We recorded monkey RGC activity to the same stimuli at three different  light levels, each differing by 1 log unit: 0.3 R*rod − 1 s − 1 , 3 R*rod − 1 s − 1 and 30 R*rod − 1 s − 1 .
Revised: We recorded monkey RGC activity to the same stimuli at three different mean light levels, each differing by 1 log unit: 0.3 R*receptor − 1 s − 1 , 3 R*receptor − 1 s − 1 and 30 R*receptor − 1 s − 1 .

Original: The retina was first allowed to adapt at each light level for 5 minutes before we started the recordings.

Original: These light levels fall under the scotopic regime, where mostly rod photoreceptors contribute to vision.
Revised: These light levels fall under the scotopic regime, where mostly rod photoreceptors contribute to vision.

Original: The stimuli was a binary white noise checkerboard movie with 39 pixels x 30 pixels, where each pixel edge corresponded to approximately 140 µ m on the retina surface.
Revised: The stimulus was a binary white noise checkerboard movie with 39 pixels x 30 pixels, where each pixel edge corresponded to approximately 140 µ m on the retina surface.

Revised: The movies across the three light levels only differed in their mean pixel values which were 0.3 R*receptor − 1 s − 1 , 3 R*receptor − 1 s − 1 and 30 R*receptor − 1 s − 1 .

Original: In this work, we used a subset of 37 recorded RGCs that were classified as high quality units after spike sorting.
Revised: In this work, we used a subset of 37 recorded RGCs that were classified as high quality units after spike sorting.

Original: This subset contained ON parasol, ON midget, OFF parasol  and OFF midget RGCs.
Revised: This subset contained both the midget RGCs (that process fine details given their small receptive fields) and parasol RGCs (sensitive to motion given their large receptive fields) of ON and OFF types.

Original: The rat experiments of Ruda et al.
Revised: The rat experiments of Ruda et al.

Original: (2020) were performed at two light levels differing by 4 log units: 1 R*rod − 1 s − 1 (scotopic light level where mostly rod photoreceptors contribute to vision) and 10,000 R*rod − 1 s − 1 (photopic light level where cone photoreceptors predominantly contribute to vision).
Revised: (2020) were performed at two light levels differing by 4 log units: 1 R*receptor − 1 s − 1 (scotopic light level where mostly rod photoreceptors contribute to vision) and 10,000 R*receptor − 1 s − 1 (photopic light level where cone photoreceptors predominantly contribute to vision).

Original: The white noise checkerboard movie in these experiments had 10 pixels x 11 pixels, with each pixel edge corresponding to approximately 252 µ m on the retina.
Revised: The white noise checkerboard movie in these experiments had 10 pixels x 11 pixels, with each pixel edge corresponding to approximately 252 µ m on the retina.

Original: In this work, we used data from two rat experiments: a subset of 61 RGCs from Retina A and a subset of 58 RGCs fromRetina
Revised: In this work, we used data from two rat experiments: a subset of 61 RGCs from Retina A and a subset of 58 RGCs from

Original:  B that were classified as high quality units after spike sorting.
Revised: Retina B that were classified as high quality units after spike sorting.

Original: This subset contained OFF brisk sustained and OFF brisk transient RGC subtypes.
Revised: This subset contained OFF brisk sustained and OFF brisk transient RGC subtypes.

Original: 3. M ODEL TRAINING AND EVALUATION
Revised: 3. M ODEL TRAINING AND EVALUATION

Original: Models were trained using 40 minutes of neural recordings and associated stimulus movies for at least 50 epochs. Weights were saved for the epoch with highest performance on the validation data.
Revised: Models were trained using 40 minutes of neural recordings and associated stimulus movies for at least 50 epochs with early stopping based on a validation dataset separate from the test dataset used for measuring the model’s performance.

Original: Trained models were then evaluated using the held out validation dataset that contained multiple trials of 5-10 s of a binary white noise checkerboard movie not seen during the training, along with the associated neural recordings (also not used during training).
Revised: Trained models were then evaluated using the held out test dataset that contained multiple repeats of 5-10 s of a binary white noise checkerboard movie not seen during the training, along with the associated neural recordings.

Original: We quantified the model performance with the fraction of explainable variance in each RGC’s response that was explained by the model (FEV).
Revised: We quantified the model performance with the fraction of explainable variance in each RGC’s response that was explained by the model (FEV).

Original: This quantity (Eq. 1) was calculated as the ratio between the variance accounted for by the model and the explainable variance (denominator in Eq. 1), and has been used in previous studies like Cadena et al. (2017) to quantify how well a model predicts neural data.
Revised: This quantity (Eq. 1) was calculated as the ratio between the variance accounted for by the model and the explainable variance (denominator in Eq. 1). Such metrics to quantify how well a model predicts neural data have been used in previous studies like Cadena et al. We calculate FEV as

Original: [Equation]
Revised: [Equation]

Original: where ˆ y t represents the predicted spike rate by the model at time bin t , and y  the observed spike rate averaged across all trials.
Revised: where, [Equation] y A and y B are the observed spike rate of an RGC calculated as an average across set of repeats A and set of repeats B respectively.

Original: The  explainable variance  is the variance of each RGC attributable to the stimulus, computed bytaking the variance in the actual RGC’s responses at each time point and subtracting an estimate of the observed noise fromit. Observed noise, σ 2 noise was estimated by splitting the number of trials into two equal sets A and B and averaging across time their squared difference, given as
Revised: The sets A and B were obtained by randomly splitting the total number of repeats into two. ˆ y t represents the predicted spike rate by the model at time bin t . The explainable variance (denominator in Eq. 1) is the variance of each RGC attributable to the stimulus, computed by subtracting an estimate of the observed noise from the variance across time (Eq. 3) in the actual RGC’s responses, calculated as

Original: [Equation]
Revised: [Equation]

Original: where ¯ y A and ¯ y B are the average responses of an RGC across set of trials A and set of trials B respectively.
Revised: where ¯ y A is the the observed spike rate y A averaged across time.

Original: (Note that, in both neural data sets we considered, the number of trials was large and hence the estimated noise variance was quite low.
Revised: In all neural data sets we considered, the number of trials was large and hence the estimated noise variance was quite low.

Original: As a result, our FEV values are quite similar to what is obtained using the usual fraction explained variance calculation, which does not correct for unexplainable noise.)
Revised: As a result, our FEV values are quite similar to what is obtained using the usual fraction explained variance calculation, which does not correct for unexplainable noise.

Original: By definition, FEV can be negative if the prediction error is larger than the variance in the actual responses.
Revised: By definition, FEV can be negative if the prediction error is larger than the variance in the actual responses.

Original: We report each model’s performance across all RGCs as the median FEV across the set of RGCs.
Revised: We report each model’s performance across all RGCs as the median FEV across the set of RGCs.

Original: For ease in interpretation, we present FEV as a percentage throughout our results.
Revised: For ease in interpretation, we present FEV as a percentage throughout our results.

Original:  R ESULTS 4. E XISTING DEEP LEARNING MODELS OF THE RETINA DO NOT GENERALIZE WELL ACROSS LIGHT LEVELS
Revised:  R ESULTS 4. E XISTING DEEP LEARNING MODELS OF THE RETINA DO NOT GENERALIZE WELL ACROSS LIGHT LEVELS

Original: We first tested whether existing deep learning models of the retina could generalize across light levels.
Revised: We first tested whether existing deep learning models of the retina could generalize across light levels.

Original: To do this, we used the Deep Retina architecture (McIntosh et al., 2016)
Revised: To do this, we used the Deep Retina architecture (McIntosh et al., 2016)

Original: (i.e., the architecture in Fig.
Revised: (i.e., the architecture in Fig.

Original: 2 with the photoreceptor layer and normalization layers removed), and re-optimized the numbers of convolutional layers, filters in each layer, and the filter sizes with a grid search.
Revised: 2 with the photoreceptor layer and normalization layers removed), and re-optimized the numbers of convolutional layers, filters in each layer, and the filter sizes with a grid search.

Original: This model had three CNN layers followed by a dense layer with 8, 16 and 18 channels in each of the
Revised: This model had three CNN layers followed by a dense layer with 8, 16 and 18 channels in each of the

Original: CNN layers, respectively, and filter sizes of 9x9, 7x7 and 5x5 respectively.
Revised: CNN layers, respectively, and filter sizes of 9x9, 7x7 and 5x5 respectively.

Original: We trained this model to predict monkey RGC responses using the training dataset that contained binary white noise movie at 30 R*rod − 1 s − 1 and 3 R*rod − 1 s − 1 , the two higher light levels in the monkey data set.
Revised: We trained this model to predict monkey RGC responses using the training dataset that contained binary white noise movies at 30 R*receptor − 1 s − 1 and 3 R*receptor − 1 s − 1 , the two higher light levels in the monkey data set.

Original: We then used the validation data set to evaluate its performance at all light levels including the testing light level (0.3 R*rod − 1 s − 1 ) not used during the training (Fig. 3a). 
Revised: We then used the test data set to evaluate its performance at all light levels including the testing light level (0.3 R*receptor − 1 s − 1 ) not used during the training (Fig. 3a). A

Revised: Layer Normalization layer at the input normalized each pixel in input movie sample by the mean and variance across all pixels in the movie.

Original: The model could reliably predict responses to stimuli at the two training light levels, 30 R*rod − 1 s − 1 and 3 R*rod − 1 s − 1 (Fig.
Revised: The model could reliably predict responses to stimuli at the two training light levels, 30 R*receptor − 1 s − 1 and 3 R*receptor − 1 s − 1 (Fig.

Original: 3b, columns 1-2), withmedian FEV  of 81% and 72% respectively (Fig. 4a).
Revised: 3b, columns 1-2), with FEV ( median ± 95% c.i. ) of 84% ± 11% and 78% ± 3% respectively (Fig. 4a).

Original: However, this model performed poorly at the testing light level (0.3 R*rod − 1 s − 1 ) (Fig.
Revised: However, this model performed poorly at the testing light level (0.3 R*receptor − 1 s − 1 ) (Fig.

Original: 3b, column 3) explaining only 4% of the variance in the population data (Fig. 4a).
Revised: 3b, column 3) explaining only 16% ± 15% of the variance in the population data (Fig. 4a).

Original: To test whether this lack of generalization to a different light level was simply because of differences in the scales of the input and  the subsequent layers we included  normalization layersfollowing the input and prior to each activation function in the model.
Revised: To overcome internal covariate shift and stabilize the network, we included batch normalization layers prior to each activation function in the model.

Original: This marginally increased the overall performance of the model at each light level (Fig. 4b), however the generalization performance at the test light level was  still quite low witha median FEV of 19% (Fig. 4b).
Revised: This marginally increased the overall performance of the model at each light level (Fig. 4b). The generalization performance at the test light level was however still quite low with FEV of 24% ± 15% (Fig. 4b).

Original: 4. P HOTORECEPTOR FRONT END ENABLES DEEP LEARNING MODELS OF RETINA TO GENERALIZE ACROSS LIGHT LEVELS
Revised: 4. P HOTORECEPTOR FRONT END ENABLES DEEP LEARNING MODELS OF RETINA TO GENERALIZE ACROSS LIGHT LEVELS

Original: To test whether our new PR-CNN model was better at generalizing across lighting conditions, we repeated the same task as above, i.e., trained the model end-to-end including the photoreceptor layer to predict monkey RGC responses to the binary white noise movie at the two higher light levels, 30 R*rod − 1 s − 1 and 3 R*rod − 1 s − 1 .
Revised: To test whether our new PR-CNN model was better at generalizing across lighting conditions, we repeated the same task as above, i.e., trained the model end-to-end including the photoreceptor layer to predict monkey RGC responses to the binary white noise movie at the two higher light levels, 30 R*receptor − 1 s − 1 and 3 R*receptor − 1 s − 1 .

Original: Similar to the model without the photoreceptor layer, this model could also reliably predict responses to test stimuli at the two training light levels (Fig.
Revised: Similar to the model without the photoreceptor layer, this model could also reliably predict responses to test stimuli at the two training light levels (Fig.

Original: 4c columns 1-2 and Fig. 4c).
Revised: 4c columns 1-2 and Fig. 4c).

Original: Importantly, this model was able to explain 54%  of the variance in responses at a light level lower than those at which it was trained (Fig.
Revised: Importantly, this model was able to explain 54% ± 11% of the variance in responses at a light level lower than those at which it was trained (Fig.

Original: 4c column 3 and Fig. 4c).
Revised: 4c column 3 and Fig. 4c).

Original: This is a notable improvement over the model without the photoreceptor layer (but with the normalization layers included), which only explained 19% of the variance (Fig. 4b).
Revised: This is a notable improvement over the model without the photoreceptor layer (but with the normalization layers included), which only explained 24% ± 15% of the variance (Fig. 4b).

Original: Next, we repeated this analysis with all different combinations of training and testing light levels: each time, we trained the model on data from two light levels, and then evaluated the model on test data from a different light level (Fig. 4d).
Revised: Next, we repeated this analysis with all different combinations of training and testing light levels: each time, we trained the model on data from two light levels, and then evaluated the model on test data from a different light level (Fig. 4d).

Original: For all combinations of training and testing light levels, the PR-CNN model performed better at generalizing to new light levels than the conventional CNN model without the photoreceptor layer (but with normalization layers included).
Revised: For all combinations of training and testing light levels, the PR-CNN model performed better at generalizing to new light levels than the conventional CNN model without the photoreceptor layer (but with normalization layers included).

Original: The difference was smallest for the case where the testing light level (3 R*rod − 1 s − 1 ) was intermediate between the training light levels (30 R*rod − 1 s − 1 and 0.3 R*rod − 1 s − 1 ) (Fig.
Revised: The difference was smallest for the case where the testing light level (3 R*receptor − 1 s − 1 ) was intermediate between the training light levels (30 R*receptor − 1 s − 1 and 0.3 R*receptor − 1 s − 1 ) (Fig.

Original: 4d, column 3).
Revised: 4d, column 3).

Original: This suggests that while standard CNN based models can generalize to conditions that can be interpolated from training conditions, they are not as well equipped to generalize to conditions that have to be extrapolated.
Revised: This suggests that while standard CNN based models can generalize to conditions that can be interpolated from training conditions, they are not as well equipped to generalize to conditions that have to be extrapolated.

Original: Our new PR-CNN model leverages the biophysics of photoreceptor adaptation to do much better at this challenging extrapolation task.
Revised: Our new PR-CNN model leverages the biophysics of photoreceptor adaptation to do much better at this challenging extrapolation task.

Original: 4. G ENERALIZATION ACROSS EXTREMELY DIFFERENT LIGHT LEVELS
Revised: 4. G ENERALIZATION ACROSS EXTREMELY DIFFERENT LIGHT LEVELS

Original: Having observed that our PR-CNN model can generalize well between light levels that differ by 1-2 orders of magnitude, we next wondered whether it could also generalize well across more extreme variations in lighting.
Revised: Having observed that our PR-CNN model can generalize well between light levels that differ by 1-2 orders of magnitude, we next wondered whether it could also generalize well across more extreme variations in lighting.

Original: To address this question, we trained our model to predict responses at a relatively bright (photopic) light level (10,000 R*rod − 1 s − 1 ) where cone photoreceptors predominantly contribute to vision and evaluated the model’s generalization performance at a much dimmer (scotopic) light level (1 R*rod − 1 s − 1 ) where only rod photoreceptors are active.
Revised: To answer this question, we trained our model to predict responses at a relatively bright (photopic) light level (10,000 R*receptor − 1 s − 1 ) where cone photoreceptors predominantly contribute to vision and evaluated the model’s generalization performance at a much dimmer (scotopic) light level (1 R*receptor − 1 s − 1 ) where only rod photoreceptors are active.

Original: For comparison, we first performed this experiment with the conventional CNN model including normalization layers (as in Fig. 4b).
Revised: For comparison, we first performed this experiment with the conventional CNN model including normalization layers (as in Fig. 4b).

Original: That model failed  to predict responses at the 1 R*rod − 1 s − 1 light level after having been trained at the 10,000 R*rod − 1 s − 1 level (Fig.
Revised: That model failed (FEV of − 52% ± 9% ) to predict responses at the 1 R*receptor − 1 s − 1 light level after having been trained at the 10,000 R*receptor − 1 s − 1 level (Fig.

Original: 5a,c).
Revised: 5a,c).

Original: Our proposed PR-CNN model, however, did surprisingly well (Fig. 5b), achievinga median FEV of 54%  on this task (Fig. 5).
Revised: Our proposed PR-CNN model, however, did notably well (Fig. 5b), achieving FEV of 54% ± 8% on this task (Fig. 5).

Original: For this experiment, we trained a PR-CNN model at 10,000 R*rod − 1 s − 1 and then replaced that model’s photoreceptor parameters (which correspond to cone cells at this light level) with those corresponding to rod cells, as explained below and in Fig.
Revised: For this experiment, we trained a PR-CNN model at 10,000 R*receptor − 1 s − 1 and then replaced that model’s photoreceptor parameters (which correspond to cone cells at this light level) with those corresponding to rod cells, as explained below and in Fig.

Original: B.1 in Appendix B.
Revised: B.1 in Appendix B.

Original: First, we trained our model end-to-end to predict rat RGC responses (from Retina A) to stimuli at the bright (photopic) light level (10,000 R*rod − 1 s − 1 ).
Revised: First, we trained our model end-to-end to predict rat RGC responses (from Retina A) to stimuli at the bright (photopic) light level (10,000 R*receptor − 1 s − 1 ).

Original: This led to an estimate of photoreceptor and
Revised: This led to an estimate of photoreceptor and

Original: CNN parameters for the cone pathway.
Revised: CNN parameters for the cone pathway.

Original: We then estimated parameters reflecting rod photoreceptors by re-training the model at the dim (scotopic) light level (1 R*rod − 1 s − 1 ), while keeping the
Revised: We then estimated parameters reflecting rod photoreceptors by re-training the model at the dim (scotopic) light level (1 R*receptor − 1 s − 1 ), while keeping the

Original: CNN weights and biases fixed to their already-learnt values.
Revised: CNN weights and biases fixed to their already-learnt values.

Original: Thus, the retraining altered only the photoreceptor parameters based on the data from Retina A.
Revised: Thus, the retraining altered only the photoreceptor parameters based on the data from Retina A.

Original: We then trained the PR-CNN model to predict RGC responses of another rat’s retina (Retina B) to stimuli at the bright light level.
Revised: We then trained the PR-CNN model to predict RGC responses of another rat’s retina (Retina B) to stimuli at the bright light level.

Original: Here, we fixed the photoreceptor parameters to the values learned from Retina A at the bright light level.
Revised: Here, we fixed the photoreceptor parameters to the values learned from Retina A at the bright light level.

Original: We then exchanged those photoreceptor parameters with the ones learned from Retina A at the low light level.
Revised: We then exchanged those photoreceptor parameters with the ones learned from Retina A at the low light level.

Original: In this manner, we took the model where the
Revised: In this manner, we took the model where the

Original: CNN weights were learned on model of Retina B at the bright light level and simply exchanged the cone photoreceptor parameters for rod parameters.
Revised: CNN weights were learned on model of Retina B at the bright light level and simply exchanged the cone photoreceptor parameters for rod parameters.

Original: Notably, in this procedure, none of the data from
Revised: Notably, in this procedure, none of the data from

Original: Retina B at the dim light level was used in training the model.
Revised: Retina B at the dim light level was used in training the model.

Original: When tested on Retina B at the dim light level, the resulting model was able to explain 54% of the variance in Retina B RGC responses (Fig. 5c).

Original: This generalization is surprisingly good given that this model was never trained on scotopic data from Retina B: the CNNs were trained on Retina B at the photopic light level, and the photoreceptor model was fit to scotopic data from Retina A.

Original: In retrospect, models without inherent knowledge of light levels, like the conventional CNN, are likely to fail at this generalization task asbecause response properties of RGCs are substantially different at bright (photopic) and dim (scotopic) light levels.
Revised: In retrospect, models without inherent knowledge of light levels, like the conventional CNN, are likely to fail at this generalization task as response properties of RGCs are substantially different at bright (photopic) and dim (scotopic) light levels.

Original: For example, rod mediated RGC responses at dim conditions are slower than cone mediated responses at brighter light levels (Ruda et al., 2020; Baylor & Fettiplace, 1977).
Revised: For example, rod mediated RGC responses at dim conditions are slower than cone mediated responses at brighter light levels (Ruda et al., 2020; Baylor & Fettiplace, 1977).

Original: This could explain the temporal lag between the predicted and actual response at dim light level (Fig.
Revised: This could explain the temporal lag between the predicted and actual response at dim light level (Fig.

Original: 5a): the model trained at bright light level (10,000 R*rod − 1 s − 1 ) learnt filters representing the cone pathway and therefore predicted responses with the cone pathway kinetics even at dim conditions (1 R*rod − 1 s − 1 ) (blue line in Fig. 5a). Whereas the actual responses at dim light levels were in fact slower (gray line in Fig. 5a) because they were mediated by the rod pathway.
Revised: 5a): the model trained at bright light levels (10,000 R*receptor − 1 s − 1 ) learnt faster kinetics of the cone pathway.

Original: Other differences in response properties across rod and cone pathways might explain the additional peaks in the predicted response.

Original: While some of these differences is RGC responses may arise due to faster response kinetics of the cones themselves (Cao et al., 2007; Baylor & Hodgkin, 1973; Ingram et al., 2016; Schneeweis & Schnapf, 1995), the relative contribution of photoreceptors and downstream retinal circuits are not well understood.
Revised: While some of the differences in RGC responses may arise due to faster response kinetics of the cones themselves (Cao et al., 2007; Baylor & Hodgkin, 1973; Ingram et al., 2016; Schneeweis & Schnapf, 1995), the relative contribution of photoreceptors and downstream retinal circuits are not well understood.

Original: Models of retina that can leverage the power of deep learning to modelmultiple ganglion cells simultaneously, together with biologically tractable components, could be used to dissect the relative contributions by different components. At least in our photoreceptor-CNN model, switching to rod photoreceptor substan- tially reduces (Fig. 5b) the lag and the response frequency that was observed in the conventional CNN model predictions at dim light level of 1 R*rod − 1 s − 1 (Fig. 5a).
Revised: Our PR-CNN model could be one way to dissect such underlying contributions.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: Here, we introduced a  novel CNN architecture that uses a biophysical photoreceptor modelat its front end to leverage photoreceptor adaptation mechanisms and enable generalization across light levels.
Revised: Here, we introduced a new class of machine learning models for neuroscience applications. The input layer to the model is a novel convolutional layer that incorporates a biophysical photoreceptor model to leverage photoreceptor adaptation mechanisms and enable generalization across light levels.

Original: We found that this model strongly outperforms other CNN architectures,including ones that make heavy use of normalization layers, in generalizing across light levels.
Revised: We found that this model strongly outperforms other CNN architectures, in generalizing across light levels, making it a potentially desirable tool for many applications, including visual prosthetics.

Original: Our photoreceptor-CNN retina model is able to function well at light levels very different from those at which it was trained, making it a potentially desirable tool for many applications, including visual prosthetics. Those could benefit substantially from algorithms that can translate visual stimuli into patterns of nerve impulses (to be stimulated by the device in order to restore sight), and that work well under varied lighting conditions. Moreover, the photoreceptor-CNN model can provide the scientific community with an opportunity to investigate visual processing of dynamic stimuli.
Revised: The photoreceptor model while complex, has parameters that map directly onto the biology, presenting the scientific community with an opportunity to investigate visual processing of dynamic stimuli.

Original: For example, how much of the adaptation in the retina can be accounted for by photoreceptors alone?
Revised: For example, how much of the adaptation in the retina can be accounted for by photoreceptors alone?

Original: and how much does the sensitivity of each RGC to different stimulus features vary as the lighting conditions change?
Revised: And how much does the sensitivity of RGCs to different stimulus features vary with lighting conditions?

Revised: Moreover, models of retina that can leverage the power of deep learning to model multiple ganglion cells simultaneously, together with biologically interpretable components, could be used to dissect the relative contributions by different components.

Original: From a wider neuroscience perspective, our approach demonstrates the power of embedding neural dynamics in deep learning models of the brain:our new biophysical model layer can match its sensitivity to prevailing input conditions that change dynamically whereas the downstream layers of the model  progressively extract relevant features from those dynamically adapting input stages.
Revised: From a wider neuroscience perspective, our approach demonstrates the power of embedding neural dynamics in deep learning models of the brain: biophysical layers can match the sensitivity to prevailing input conditions that change dynamically whereas the downstream layers of the model can progressively extract relevant features from those dynamically adapting input stages.

Revised: The wider scientific community might not be interested in biological investigations, but could still benefit from the photoreceptor’s light adaptation mechanisms to overcome challenges in computer vision.

Revised: To this end we have developed an adaptive convolutional layer based upon a simplified phenomenological photoreceptor model (Appendix C.1).

Original: Our model also potentially enables better computer vision systems for applications where lighting is highly dynamic, such as the object detection and recognition systems on autonomous vehicles.
Revised: This model could potentially enable better computer vision systems for applications where lighting is highly dynamic, such as the object detection and recognition systems on autonomous vehicles.

Original: Here, pixel-wise local luminance adaptation (from using our photoreceptor model as an input layer) could help filter out disruptive lighting changes induced by changing shadows or clouds, that would otherwise cause errors in object recognition and/or detection (Janai et al., 2020; Gomez-Ojeda et al., 2015; Kolaman et al., 2019).
Revised: Here, pixel-wise local luminance adaptation (from using our adaptive-convolutional layer as an input layer) could help filter out disruptive lighting changes induced by changing shadows, that would otherwise cause errors in object recognition and/or detection (Janai et al., 2020; Gomez-Ojeda et al., 2015; Kolaman et al., 2019).

Original: A proof-of-conceptdemonstration that a photoreceptor-inspired convolution layer, Adaptive-Conv, may be better than conventional CNN layers at filtering out lighting fluctuations from a visual scene, is presented in Appendix C.1. 
Revised: A proof-of-concept that a photoreceptor-inspired convolution layer, Adaptive-Conv, may be better than conventional CNNs at this task, is presented in Appendix C.1. Future work includes overcoming some of the limitations of this work. For example, the manual switch from cone to rod photoreceptors that is required for a model trained at cone light levels to generalize to rod light levels (Fig. 5). In addition, the photoreceptor parameters learnt may also reflect some of the adaptive mechanisms in downstream retinal circuits.

Revised: While this is not a concern when developing models that perform better at different light levels, it may limit some biological inferences.

Revised: One way to overcome this limitation would be by adding layers downstream of the photoreceptor layer that incorporate other known biological sources of adaptation in the retina.

Original: C ODE AVAILABILITY
Revised: R EPRODUCIBILITY STATEMENT

Original: The photoreceptor-CNN model and Adaptive-Conv model can be downloaded from [Redacted for
Revised: The photoreceptor-CNN model and Adaptive-Conv model can be downloaded from [Redacted for

Original: Double-Blind Review].
Revised: Double-Blind Review].

Revised: All retina electrophysiology data used in this study can be made available upon reasonable request.

Original: E THICS STATEMENT
Revised: E THICS STATEMENT

Original: All electrophysiology experiments were performed in accordance with the guidelines of Institutional
Revised: All electrophysiology experiments were performed in accordance with the guidelines of Institutional

Original: Animal Care and Use Committee at [Redacted for Double-Blind Review]
Revised: Animal Care and Use Committee at [Redacted for Double-Blind Review]

Original: A A PPENDIX : P HOTORECEPTOR BIOPHYSICAL MODEL
Revised: A A PPENDIX : P HOTORECEPTOR BIOPHYSICAL MODEL

Original: The biophysical model of the photoreceptor that we used is described in Angueyra et al.
Revised: The biophysical model of the photoreceptor that we used is described in Angueyra et al.

Original: Below we reproduce the model to facilitate the readers.
Revised: Below we reproduce the model to facilitate the readers.

Original: Input to the model is the stimulus intensity as a function of time (in units of R*rod − 1 s − 1 ) given by Stim ( t ) in Eq.
Revised: Input to the model is the stimulus intensity as a function of time (in units of R*receptor − 1 s − 1 ) given by Stim ( t ) in Eq.

Original: The output of the model is the photoreceptor outer segment current as a function of time, I ( t ) given in Eq.
Revised: The output of the model is the photoreceptor outer segment current as a function of time, I ( t ) given in Eq.

Original: [Equation]
Revised: [Equation]

Original: In our Keras photoreceptor layer, the parameters σ , γ , ϕ , η , q , β , C dark , K GC , B slow , m , k Ca , and h can all be set to trainable.
Revised: In our Keras photoreceptor layer, the parameters σ , γ , ϕ , η , q , β , C dark , K GC , B slow , m , k Ca , and h can all be set to trainable.

Original: In this study however, we only allowed σ , β , ϕ and η to be trained.
Revised: In this study however, we only allowed σ , β , ϕ and η to be trained.

Original: Remaining parameters were fixed to their values (Angueyra et al., 2021) from photoreceptor data fits.
Revised: Remaining parameters were fixed to their values (Angueyra et al., 2021) from photoreceptor data fits.

Original: B A PPENDIX : T RAINING PROCEDURE FOR GENERALIZING ACROSS EXTREME LIGHT LEVELS
Revised: B A PPENDIX : T RAINING PROCEDURE FOR GENERALIZING ACROSS EXTREME LIGHT LEVELS

Revised: Step 1: PR-CNN model was trained end-to-end to predict Rat Retina ’A’ RGC responses at photopic light level (10,000 R*receptor − 1 s − 1 ).

Original: C A PPENDIX : A DAPTIVE -C ONV C. P ROOF - OF - CONCEPT DEMONSTRATION OF APPLICATION IN COMPUTER VISION
Revised: This led to an estimate for cone photoreceptor parameters and a model for the inner retina circuit (the CNN layers). Step 2: The model was re-trained at scotopic light level (1 R*receptor − 1 s − 1 ) but the CNN layers were set to non-trainable and fixed to weights learnt in Step 1. In this case, the photoreceptor model learnt parameters reflecting rods. Step 3: The model was trained to predict Rat Retina ’B’ responses at photopic light level.

Revised: Photoreceptor layer was set to non-trainable with parameters fixed to cone parameters learnt in Step 1 .

Revised: Step 4 (testing step):

Revised: Model was tested to predict Rat Retina ’B’ responses at scotopic light level.

Revised: Here, we used the rod photoreceptor parameters learnt in Step 2 and rest of the model representing inner retina pathways of Retina B learnt in Step 3 .

Revised: C A PPENDIX : A DAPTIVE -C ONV C. P ROOF - OF - CONCEPT DEMONSTRATION OF APPLICATION IN COMPUTER VISION

Original: Given that the photoreceptor adaptive mechanisms enable the retina (and the PR-CNN model discussed in the main paper) to work under a wide range of steady-state and dynamic light intensities we hypothesized that those same adaptive mechanisms could also be useful for computer vision tasks in dynamic lighting conditions.
Revised: Given that the photoreceptor adaptive mechanisms enable the retina (and the PR-CNN model discussed in the main paper) to work under a wide range of steady-state and dynamic light intensities, we hypothesized that those same adaptive mechanisms could also be useful for computer vision tasks in dynamic lighting conditions.

Original: Specifically, we consider conditions with potential disruptive events such as shifting cloud cover or shadows due to moving tree branches, that can rapidly and dramatically change the lighting in part or all of a visual scene.
Revised: Specifically, we consider conditions with potential disruptive events such as shifting cloud cover or shadows due to moving tree branches, that can rapidly and dramatically change the lighting in part or all of a visual scene.

Original: These events can shift the scale of the pixel values in an image, either on some pixels or on all of them, depending on the spatial scale of the event.
Revised: These events can shift the scale of the pixel values in an image, either on some pixels or on all of them, depending on the spatial scale of the event.

Original: If the spatial scale is large, and the whole image exhibits the change in lighting level, simple normalization (i.e., layer norm) could remove the change.
Revised: If the spatial scale is large, and the whole image exhibits the change in lighting level, simple normalization (i.e., layer norm) could remove the change.

Original: For localized changes, however, there is no obvious strategy through which normalization can remove the lighting change.
Revised: For localized changes, however, there is no obvious strategy through which normalization can remove the lighting change.

Original: Moreover, these local changes in pixel intensities can be easily confused for changes in the edges in the scene, hence confounding object detection and identification (Gomez-Ojeda et al., 2015; Kolaman et al., 2019;
Revised: Moreover, these local changes in pixel intensities can be easily confused for changes in the edges in the scene, hence confounding object detection and identification (Gomez-Ojeda et al., 2015; Kolaman et al., 2019;

Original: Janai et al., 2020).
Revised: Janai et al., 2020).

Original: To address this challenge, we have developed a novel convolution layer, the Adaptive-Conv, which contains a simplified functional model of the photoreceptors (see Appendix.
Revised: To address this challenge, we have developed a novel convolution layer, the Adaptive-Conv, which contains a simplified functional model of the photoreceptors (see Appendix.

Original: C.2 for technical details).
Revised: C.2 for technical details).

Original: As a proof-of-concept that such photoreceptor-inspired models could be better at solving computer vision tasks, we considered a toy example where the input to the model is an object’s intensity represented by a single pixel in space.
Revised: As a proof-of-concept that such photoreceptor-inspired models could be better at solving computer vision tasks, we considered a toy example where the input to the model is an object’s intensity represented by a single pixel in space.

Original: The pixel value (intensity) varies in time around a fixed mean value (object intensity in Fig.
Revised: The pixel value (intensity) varies in time around a fixed mean value (object intensity in Fig.

Original: C.1a, column 1).
Revised: C.1a, column 1).

Original: At different time points, brief disruptive events of different durations and intensities (spanning 1 to 5 log units) (source intensity in Fig.
Revised: At different time points, brief disruptive events of different durations and intensities (spanning 1 to 5 log units) (source intensity in Fig.

Original: C.1a, column 2), change the amount of light reflected off the object( Fig.
Revised: C.1a, column 2), change the amount of light reflected off the object( Fig.

Original: C.1a, column 3).
Revised: C.1a, column 3).

Original: We attempted to train neural networks with, or without, our new Adaptive-Conv layer, to extract the original intensity of the object (red line in Fig.
Revised: We attempted to train neural networks with, or without, our new Adaptive-Conv layer, to extract the original intensity of the object (red line in Fig.

Original: C.1a, column 3), thereby removing the effects of the disruptive light change (blue line in Fig.
Revised: C.1a, column 3), thereby removing the effects of the disruptive light change (blue line in Fig.

Original: C.1a, column 3)
Revised: C.1a, column 3)

Original: We trained a model with the Adaptive-Conv layer to perform this task (Fig.
Revised: We trained a model with the Adaptive-Conv layer to perform this task (Fig.

Original: C.1b, left panel).
Revised: C.1b, left panel).

Original: The model takes as input a movie with 200 frames.
Revised: The model takes as input a movie with 200 frames.

Original: The first layer is an Adaptive-Conv layer with 40 channels followed by 7 dense layers each containing 80 units.
Revised: The first layer is an Adaptive-Conv layer with 40 channels followed by 7 dense layers each containing 80 units.

Original: These hyperparameters were optimized using a grid search.
Revised: These hyperparameters were optimized using a grid search.

Original: The width of the temporal kernel in the Adaptive-Conv layer was set to 200.
Revised: The width of the temporal kernel in the Adaptive-Conv layer was set to 200.

Original: This model could reliably extract the original intensity of the object, discounting the disruptive events (Fig.
Revised: This model could reliably extract the original intensity of the object, discounting the disruptive events (Fig.

Original: C.1b, right panel).
Revised: C.1b, right panel).

Original: In contrast, a conventional 1D CNN model where the standard convolution layer was used (Fig.
Revised: In contrast, a conventional 1D CNN model where the standard convolution layer was used (Fig.

Original: C.1c, left panel), failed to extract the original intensity following the disruptive events (Fig.
Revised: C.1c, left panel), failed to extract the original intensity following the disruptive events (Fig.

Original: C.1c, right panel), despite having 4 times as many parameters as the model with the Adaptive-Conv.
Revised: C.1c, right panel), despite having 4 times as many parameters as the model with the Adaptive-Conv.

Original: (For this model as well we optimized the hyperparameters using a grid search which resulted in 100 channels for the first convolution layer, followed bydense layers each with 200 units.)
Revised: (For this model as well we optimized the hyperparameters using a grid search which resulted in 100 channels for the first convolution layer, followed bydense layers each with 200 units.)

Original: Our ongoing work will test this Adaptive-Conv approach on more complex vision tasks and see if the same observation holds, namely that the photoreceptor model discounts the sudden large changes in intensity, thereby removing the impacts of those intensity changes on object detection and/or categorization.
Revised: Our ongoing work will test this Adaptive-Conv approach on more complex vision tasks and see if the same observation holds, namely that the photoreceptor model discounts the sudden large changes in intensity, thereby removing the impacts of those intensity changes on object detection and/or categorization.

Original: C. A DAPTIVE -C ONV LAYER
Revised: C. A DAPTIVE -C ONV LAYER

Original: The Adaptive-Conv layer is a trainable layer that we constructed using Keras (Chollet et al., 2015) package in Python and builds upon a modified version of a phenomenological photoreceptor model described in (Clark et al., 2013).
Revised: The Adaptive-Conv layer is a trainable layer that we constructed using Keras (Chollet et al., 2015) package in Python and builds upon a modified version of a phenomenological photoreceptor model described in (Clark et al., 2013).

Original: In this layer, the input signal at each pixel, x ( t ) is convolved in time, t , with two temporal kernels, as in 
Revised: In this layer, the input signal at each pixel, x ( t ) is convolved in time, t , with two temporal kernels, as in [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: where x ( t ) is the pixel intensity at time t , K y and K z are the two temporal, of the form
Revised: where x ( t ) is the pixel intensity at time t , K y and K z are the two temporal, of the form

Original: [Equation]
Revised: [Equation]

Original: Here, τ y , τ c , τ z , specifies the time scale and n y , n c , n z specifies the rise behaviour of the kernels.
Revised: Here, τ y , τ c , τ z , specifies the time scale and n y , n c , n z specifies the rise behaviour of the kernels.

Original: Parameter γ (constrained at 0 ≤ γ ≤ 1 ) weights the relative importance of the two kernels with different temporal profiles making up K z , and θ is the Heavyside function.
Revised: Parameter γ (constrained at 0 ≤ γ ≤ 1 ) weights the relative importance of the two kernels with different temporal profiles making up K z , and θ is the Heavyside function.

Original: All these parameters are trainable within the Adaptive-Conv layer.
Revised: All these parameters are trainable within the Adaptive-Conv layer.

Original: The two convolved input signals, y ( t ) and z ( t ) are then weighted and combined divisively to form the Adaptive-Conv layer’s output, through the following operation
Revised: The two convolved input signals, y ( t ) and z ( t ) are then weighted and combined divisively to form the Adaptive-Conv layer’s output, through the following operation

Original: [Equation]
Revised: [Equation]

Original: where α , κ and γ are trainable weights.
Revised: where α , κ and γ are trainable weights.

Original: In total, the Adaptive-Conv layer has 10 parameters, all of which are fully trainable through backpropagation.
Revised: In total, the Adaptive-Conv layer has 10 parameters, all of which are fully trainable through backpropagation.

Original: Similar to the biophysical model used in the main paper, a key feature of this phenomenological model is that the gain of the input is dynamically controlled by α κ + βz ( t ) based on prevailing inputs over the time scale of K z .
Revised: Similar to the biophysical model used in the main paper, a key feature of this phenomenological model is that the gain of the input is dynamically controlled by α κ + βz ( t ) based on prevailing inputs over the time scale of K z .