Paper: Exploring Effective Stimulus Encoding via Vision System Modeling for Visual Prostheses (ICLR-2024)

Reviewer comment (Reproducibility):
What is the adjusted ReLU function for surrogate gradient?

Author response:
**A1:** The adjusted ReLU function is defined as:

$$f(x) = ln(1+e^{(x-\eta)}),$$ where $\eta$ is the threshold to control the intensity in EQ. (4) of the original manuscript.

Reviewer comment (Writing):
What are the differences between LIF and MP_LIF?

Author response:
**A2:** The output of the LIF neuron is a spike train, while the output of the MP_LIF neuron is the numerical values of membrane potentials. Naively speaking, the LIF neurons output 'where the spikes are' and  MP_LIF neurons output 'how large is the membrane potentials'. The predicted membrane potentials will be set to zero after a spike is fired.

Reviewer comment (Reproducibility):
How are the parameters of the phosphene model selected? It says "fitting the experimental data recorded from patients with retinal prostheses". I didn't see any fitting procedure or description of the data.

Author response:
**A3:** The hyperparameters of the phosphene model utilized in this work are based on various experimental and modeling results from different components of corresponding research:

In Eq. 2, we use $a = 0.75$, $b = 120$, $k = 1.73$, based on a fitting method of [1] on data of a human experiment from [2].

In Eq. 3, Eq. 4, $Pw = 3.3ms$, $f = 30Hz$, $threshold = 0.3 mW/mm^{2}$ are obtained from the setting and human experimental results with retinal prosthesis [3]. We use $I_{leak} = 23.9 \mu A$ based on a data fitting from [4].

In Eq. 5, Eq. 6, $a_{1}$, $a_{2}$, $b_{1}$, and $b_{2}$, $\lambda$ is adopted from a retinal phosphene model [5].

In Eq. 7, $d_{s}$, $d_{e}$, $\rho$, and $\gamma$ were obtained from a human retina model [6].

Reviewer comment (Evaluation):
Table 3 shows the energy cost of SRNN and CNN with a similar architecture. The advantage of SRNN is not surprising given the intrinsically low energy cost of SRNN. However, the energy cost of SRNN should be compared to that of a CNN with similar performance (such as Cadena, 2009) rather than similar architecture.

Author response:
**A5:** In the proposed StimuSEE, we predict stimulation patterns using **spike trains** recorded from a dynamic vision sensor as input. The frameworks designed by Cadena et al. [7] and Yan et al. [8] take images as input and output responses of V1 cells and Ganglion cells, respectively. Therefore, directly comparing power consumption under the same performance is intractable given different inputs and outputs. In Tab. 3, we demonstrate that our designed StimuSEE is energy efficient, leading to a more practical visual prosthesis.

Reviewer comment (Novelty):
It's unclear to me what is the novelty of the proposed phosphene model compared to Granley et al. (2022).

Author response:
**A1:** Compared to the phosphene model in Granley et al. [1], the phosphene model of StimuSEE utilizes coordinate system transition for a more accurate phosphene location calculation. Also, the leakage $I_{leak}$ during the electrical stimulation process is also taken into account in the calculation of stimulation intensity. Furthermore, our proposed phosphene model incorporates a threshold function to determine the generation of phosphene.

Reviewer comment (Reproducibility):
How is PVS-net pre-trained? Was the same dataset utilized both for the PVS-net pretraining and the end-to-end training?

Author response:
**A2:** We utilized CIFAR-10 dataset to pretrain the PVS-net with a classification task, and adopted the DVS-V1 dataset in the end-to-end training process.

Reviewer comment (Experiments):
What is the temporal resolution of the label (neural firing rate)? How many time points per image?

Author response:
**A3:** The visual stimulus is presented to the subject to capture neural responses within a 60 ms time window. The spike firing rate is assessed by counting neuron firings within a fixed recording window of 60 ms. For one image, as the recording covers 166 neurons, the label consists of 166 numbers, each representing the corresponding neuron's firing count.

Reviewer comment (Experiments):
In Fig. 5a, the authors only showcased 20 neurons. It would be informative if the authors could provide quantitative results on the model's fit across all 166 neurons. Is Fig. 5b showing the results from a single neuron? If so, is it the best-performing one?

Author response:
**A4:** We first present the image-level Pearson Correlation Coefficient (PCC) distribution for all 166 neurons in Table 1. Specifically, for each image, we compute the PCC of the firing rate across all 166 neurons with respect to the ground truth, providing statistical information for all testing images. Additionally, in Table 2, we offer the neuron-level PCC distribution for all 166 neurons. This involves calculating the PCC of the firing rate for each individual neuron in relation to the ground truth across all testing images, and presenting statistics for all 166 neurons. These experimental results show that our proposed StimuSEE fits the neural response of V1 cells well. Fig. 5b shows one of the best-performing ones.

Table 1: The distribution of (PCC) of 166 neurons to each image  
| PCC | >0.85 | 0.75-0.85 | 0.65-0.75 | 0.55-0.65 | <0.55 |  
| :----: | :----: | :----: | :----: | :----: | :----: |   
| Percentage | 0.32| 0.50 | 0.08 | 0.07 |  0.03 |  

Table 2:  The distribution of PCC of each neuron against all test images  
| PCC | >0.75 | 0.65-0.75 | 0.55-0.65 | 0.45-0.55 | <0.45 |  
| :----: | :----: | :----: | :----: | :----: | :----: |  
| Percentage | 0.22| 0.47 | 0.16 | 0.09 |  0.06 |

Reviewer comment (Evaluation):
What would be the chance-level performance for both Fig. 5a and Fig. 5b?

Author response:
**A5:** We provide statistics Tab. 3 and 4 as in **A4** based on our defined chance-level predictor.

Table 3: The distribution of PCC of 166 neurons to each image  
| PCC | >0.15 | 0.05 - 0.15 | -0.05 - 0.05 | -0.15 - -0.05 | <-0.15 |  
| :----: | :----: | :----: | :----: | :----: | :----: |  
| Percentage | 0.04| 0.25 | 0.48 | 0.21 |  0.02 |  

Table 4:  The distribution of PCC of each neuron against all test images  
| PCC | >0.15 | 0.05 - 0.15 | -0.05 - 0.05 | -0.15 - -0.05 | <-0.15 |  
| :----: | :----: | :----: | :----: | :----: | :----: |  
| Percentage | 0.07| 0.24 | 0.42 | 0.22 |  0.05 |

Reviewer comment (Evaluation):
Is Fig. 5a showing spike counts from 20 neurons for a single time point or single image? If yes, why not show the overall statistics for all images in the test dataset?

Author response:
**A6:** Please refer to **A4** for the overall statistics.

Reviewer comment (Evaluation):
Fig. 5b shows the neural spike train over time. What is the temporal resolution here? Also, why compare the spike train randomly generated 10 times from the firing rate, instead of directly comparing the time series of the predicted and groundtruth firing rate?

Author response:
**A7:**  We apologize for any confusion caused by the x-axis of Figure 5b. When referring to 'times,' we meant images. Figure 5b illustrates the firing pattern of a single neuron across the first 20 test images (x-axis). The temporal resolution of both the labeled and predicted neuron responses is modeled using a Poisson distribution. The fitting process is iterated ten times to mitigate the impact of randomness. We will revise Fig. 5b in the camera-ready version for better presentation.

Reviewer comment (Evaluation):
How did you get the PCC in Table 2? Is it the correlation between the predicted and groundtruth firing rate along time, then averaged across all 166 neurons?

Author response:
**A8:** Since the spike firing rate is assessed by counting neuron firings within a fixed recording window of 60 ms. The label consists of 166 numbers, each representing the corresponding neuron's firing count. Thus, we can directly calculate the PCC of all 166 neurons.

Reviewer comment (Presentation):
Some fonts in equations (8) and (9) are not standard. Typically, functions or operators such as min, max, and ReLU are not italicized.

Author response:
**A9:** The font errors are fixed in the camera-ready version.

Reviewer comment (Presentation):
It will be more reader-friendly to explain some abbreviations (e.g., DVS, LIF) in the caption of Fig. 2.

Author response:
**A10:** We have included the full forms of these abbreviations in the caption of Fig. 2.

Reviewer comment (Evaluation):
Though the framework demonstrate reliable results in predicting V1 neurons, because of the different performance metric (neuronal responses vs. reconstruction error), it is hard to directly compare this method against other existing methods. One potential solutions is adding a decoder layer on top of the trained StimuSEE framework and see whether it indeed also facilitate reconstruction as well.

Author response:
**A1:** We could potentially replace StimuSEE's PVS-net with a decoder for image reconstruction. However, unlike other existing frameworks (hybrid neural autoencoders) that use the original image as input for reconstruction, our proposed StimuSEE utilizes spike trains recorded from a dynamic vision sensor as input. From an information perspective, reconstructing the original image with spike train input is a more challenging task compared to using the original image as input.

Reviewer comment (Presentation):
Since the StimuSEE frameworks consists of multiple models, the paper perhaps should make it more clear which of the models are trained with the error signals from predicting V1 responses. For example, in Figure 2, it could be helpful to use a bounding box with a different color or the SRNN model or an arrow signifying the direction of training feedback.

Author response:
**A2:** The parameters of the retina-net and PVS-net are updated based on the error signals during model training, and the parameters of the phosphene model are kept fixed. As suggested, we will revise Fig. 2 to better indicate the gradient flow.

Reviewer comment (Experiments):
The paper only briefly mentions that V1 neuron response in blind animals are comparable to those in sighted animals, it is unclear how this approach directly transfers to blind animals. Does this framework end up with different learned model based on coverage of V1 neurons in recording? Could you use two sighted animal to estimate how transferrable the model is from one animal to the other? Also since the frameworks requires many pre-determined parameters for other non-trainable models, could the authors provide some insight on how sensitive is the SRNN network training with respect to different parameter setting among the other models?

Author response:
**A3:**  The concept of using visual cortical prostheses for vision restoration has long been proposed. Beauchamp et al. [1] demonstrated that dynamic stimulation could facilitate the recognition of letter shapes in both sighted and blind individuals. Building upon this insight, we introduce StimuSEE, a system that leverages the V1 neuron response in sighted animals as a reference to create stimulation for the blind. In parallel, researchers [2,3] have explored the correlation between stimulation and the responses of vision neurons.  

**A4:** We appreciate Reviewer Fc3Q for posing these intriguing open questions. Given that the dataset employed in this study is obtained from a single subject with a fixed electrode placement corresponding to one coverage pattern of V1 neurons, direct answers to these questions are not feasible. Considering that StimuSEE generates stimulation patterns based on V1 neuron firing, we posit that transitioning to a different V1 coverage pattern would result in a different model. The verification of transferability among animals will be addressed in our future research endeavors.  

**A5:**  The hyperparameters of the phosphene model utilized in this work are based on various experimental and modeling results from different components of corresponding research:

In Eq. 2, we use $a = 0.75$, $b = 120$, $k = 1.73$, based on a fitting method of [4] on data of a human experiment from [5].

In Eq. 3, Eq. 4, $Pw = 3.3ms$, $f = 30Hz$, $threshold = 0.3 mW/mm^{2}$ are obtained from the setting and human experimental results with retinal prosthesis [6]. We use $I_{leak} = 23.9 \mu A$ based on a data fitting from [7].

In Eq. 5, Eq. 6, $a_{1}$, $a_{2}$, $b_{1}$, and $b_{2}$, $\lambda$ is adopted from a retinal phosphene model [8].

In Eq. 7, $d_{s}$, $d_{e}$, $\rho$, and $\gamma$ were obtained from a human retina model [9].

The phosphene model demonstrates robustness across various hyperparameter combinations, and the framework consistently produces similar prediction performance [10,11].

Reviewer comment (Writing):
Could you elaborate on the purpose of the parameter index in equations (8)?

Author response:
**A6:** Eq. 8 and Eq. 9 reduce the correlation among different neurons when predicting spike firing rates, simultaneously acting as a form of regularization.

Reviewer comment (Reproducibility):
The method and experimental procedure is still a bit vague to me, i.e. I’m not sure how data and gradient flow through this system and how this system trains. Correct me if I’m wrong, during training you use the DVS → spiking RNN → stimulation model → phospene model → PVS-net V1 model and then fit all parameters with the primate V1 data? Some parameters of the phosphene model are constrained by human patient data, is that also transferred to fit primate V1 data?

Author response:
**A1:** The StimuSEE is composed of a retinal network, a phosphene model, and a primary visual system network (PVS-net). The retinal network utilizes dynamic vision sensor and spiking RNN to perceive and process the scene signals. Then, a phosphene model is designed to simulate phosphene in the retina. Finally, PVS-net is adopted to predict the firing rate of V1 neurons with the simulated phosphene. The phosphene model is fixed during training. The parameter of retinal network and PVS-net are trained to fit the neural responses of V1 cells.

Reviewer comment (Writing):
Eq. 8 and Eq. 9, the notations are somewhat confusing, are capital `Index` and `index` the same thing? What does $\cdot$ and $[]$ mean in the Eq. 9? I’m also not sure about the dimension of any of the variables… are they scalar or vector integer or what? Please clarify these in the paragraph following it.

Author response:
**A2:** $x \in R^{1 \times 166}$ is the extracted feature of the PVS-net. $params \in R^{166 \times D}$ refers to a learnable matrix, where $D$ is set to 50 in this work. The selected parameters are further dot producted with the input value $x$ to yield the final prediction results. $\cdot$ denotes dot production and $[]$ refers to indexing operation. Eq. 8 and Eq. 9 reduce the correlation among different neurons when predicting spike firing rates, simultaneously acting as a form of regularization.

Reviewer comment (Evaluation):
The biological data validation figure seems a bit weak. Usually you’d like to plot the scatter plot between predicted spikes and actual spikes for all images and summarize the correlation for each cell. Figure 5 a) looks nice but lacks the population summary.

Author response:
**A3:** We first present the image-level Pearson Correlation Coefficient (PCC) distribution for all 166 neurons in Table 1. Specifically, for each image, we compute the PCC of the firing rate across all 166 neurons with respect to the ground truth, providing statistical information for all testing images. Additionally, in Table 2, we offer the neuron-level PCC distribution for all 166 neurons. This involves calculating the PCC of the firing rate for each individual neuron in relation to the ground truth across all testing images, and presenting statistics for all 166 neurons. These experimental results show that our proposed StimuSEE fits the neural response of V1 cells well. Fig. 5b shows one of the best-performing ones.  

Table 1: The distribution of PCC of 166 neurons to each image  
| PCC | >0.85 | 0.75-0.85 | 0.65-0.75 | 0.55-0.65 | <0.55 |  
| :----: | :----: | :----: | :----: | :----: | :----: |   
| Percentage | 0.32| 0.50 | 0.08 | 0.07 |  0.03 |  

Table 2: The distribution of PCC of each neuron against all test images  
| PCC | >0.75 | 0.65-0.75 | 0.55-0.65 | 0.45-0.55 | <0.45 |  
| :----: | :----: | :----: | :----: | :----: | :----: |   
| Percentage | 0.22| 0.47 | 0.16 | 0.09 |  0.06 |

Reviewer comment (Experiments):
If the data presented in Figure 5 b is real biological spike data from times, it seems overly sparse. Can you clarify the image stimulation onset and offset time schedule of it? —— if the no spike period is just gray screen stimulation, then the temporal accuracy is just defined by the stimuli time course, which doesn’t mean much.

Author response:
**A4:** The visual stimulus is presented to the subject to capture neural responses within a 60 ms time window. The spike firing rate is assessed by counting neuron firings within a fixed recording window of 60 ms. For one image, as the recording covers 166 neurons, the label consists of 166 numbers, each representing the corresponding neuron's firing count. The offset time is 940 ms.  
Figure 5b illustrates the firing pattern of a single neuron across the first 20 test images (x-axis). The temporal spike train of both the labeled and predicted neuron responses is modeled using a Poisson distribution. The fitting process is iterated ten times to mitigate the impact of randomness.

Reviewer comment (Experiments):
Further the temporal precision between the data and the prediction in Figure 5b seems **overly high.** With 10 repetitions, i.e. it can predict all the cases where spikes are missing? Maybe I misunderstood what data means in this case, is it the firing rate data for 166 V1 neurons, with some Poisson spike generator attached? How could the model know the differences between these different runs?

Author response:
**A5:** The true label (data in Figure 5b) of spike firing rate is given by neuron firing counts during recorded time during vision stimulus. No recording is done during offset (no vision stimulus). The temporal resolution is fitted using a Poisson distribution, which is repeated ten times. Since the model is deterministic. The Poisson fitting is applied to the output of the PVS-net.

Reviewer comment (Presentation):
Is Figure 5 training set or test set?

Author response:
**A6:** The displayed results in Figure 5 are test set results.

Reviewer comment (Presentation):
Also not sure about the bottom panel in Figure 5 b), the x axes `Times` means different images?

Author response:
**A7:** In Figure 5b, the x axes can be times or images. In the neural response recording experiment, the duration of recording is 60 ms. The offset time between two stimuli is 940 ms. The whole period of one image is 1000 ms. In Figure 5b, we demonstrate the firing pattern of a single neuron across 20 seconds, which also means 20 test images. We will revise Fig. 5b in the camera-ready version for better presentation.

Reviewer comment (Novelty):
The work is technically solid, but the contribution and impact seem limited to the neuroengineering domain, not sure the technique will fit or benefit the NeurIPS community.

Author response:
**A8:** This study introduced an end-to-end processing framework for retinal prostheses, aligning with the domain of **applications to neuroscience & cognitive science** within ICLR. Comparable researches [1,2,3,4] to StimuSEE, have been acknowledged by NIPS and ICLR in recent years.

Reviewer comment (Presentation):
Not quite clear to me what “*the front 20 times prediction of*” mean in Figure 5 (b) caption. What are the x-axes “Times” in this plot? Is it seconds?

Author response:
**A9:** Please refer to **A7**. We will revise Fig. 5b in the camera-ready version for better presentation.

Reviewer comment (Evaluation):
Given the four repeats in your V1 data, could you calculate the noise ceiling / self-consistency of V1 response itself? Then you can kind of compare your obtained Pearson correlation to that “ceiling”.

Author response:
**A10:** Visual neurons exhibit varying responses even to the same stimulus. In this study, we utilized the average value of these neural responses across four trials as the ground truth label. From this perspective, the average inherently provides a degree of self-consistency.

Reviewer comment (Evaluation):
How do you compute the energy efficiency of Spiking RNN? Also by counting the multiplication and addition in the simulated neuron firing process? —— if that’s the case won’t that depend on how you discretize time?

Author response:
**A11:** The energy efficiency is calculated based on the number of multiplication and addition operations. In this study, a 33 ms output from a DVS sensor with a 1 $\mu s$ time resolution serves as input to the Spiking RNN.

Reviewer comment (Experiments):
For more data to train the model, [1] seems to be quite relevant.  

[1] Chen, X., Wang, F., Fernandez, E., & Roelfsema, P. R. (2020). Shape perception via a high-channel-count neuroprosthesis in monkey visual cortex. *Science*, *370*(6521), 1191-1196.

Author response:
**A12:** Thank you for sharing this reference. We will examine it closely and explore ways to enhance our future work by incorporating insights from [5].