Paper: A Light-robust Reconstruction Method for Spike Camera (ICLR-2024)

Reviewer comment (Experiments):
How does the delta T, which sets the length of temporal window, affect the overall performance?

Author response:
**Q1:  How does the delta T affect the overall performance?**

Our recurrent-based reconstruction method demonstrates flexibility in efficiently covering and extracting temporal information from spike streams. The number of spike streams, combined with delta T, determines the method's coverage of temporal information from spike streams. When employing the same number of spike streams, a too-small delta T results in insufficient coverage of the temporal information. This hinders our method of predicting and reconstructing scenes effectively.  The increasing delta T is beneficial to our method of reconstructing scenes until stable. Generally, delta T is 41  (Zhao et al. ,2021); chen et al. ,2022); Zhang et al. ,2023). We have also opted for this value.

Reviewer comment (Experiments):
Does the number of forward path of LR-Rep reduce if we increase delta T (i.e., the length of $S_t$ increases)?

Author response:
**Q2:  Does the number of forward path of LR-Rep reduces if we increase delta T?**

The number of spike streams and delta T are independent.  We can choose the different number of spike streams for inference as shown in Fig.11.

Reviewer comment (Reproducibility):
Current pipeline seems to occupy large memory. How much does the training take in terms of memory and time?

Author response:
**Q3: The current pipeline seems to occupy a large memory. How much the training takes for memory and time?**


The time and memory required for training our method are, 18 hours and 34.4 GB which can be satisfied with 1 NVIDIA A100 GPU. This is attributed to the fact that recurrent-based networks typically consume more time and memory during training due to Backpropagation Through Time (BPTT). During testing, our method demonstrates high efficiency, i.e., our inference time is 818 ms while the inference time of WGSE is 1344 ms as shown in Table 7.

Reviewer comment (Presentation):
Figure 1 Middle(b) is quite confusing. Does the temporal features for both blue and purple are same? If it is different, it should be illustrated differently. Also, the color difference between blue and purple is too small, and also it does not look like purple. Please make it more distinctive. Also, blue and purple arrows seem not like arrows, but just lines, so please improve it for better clarity. Same for Fig. 3.

Author response:
**Q4: Related to display**

Thanks for your careful reading. We have made relevant modifications based on your suggestions:


1. Figure 1 \& Figure 3: 
We selected a color pair (green and red) with high contrast to represent forward and backward data streams. We have added a description of that temporary features for both forward and back are different.

2. Explanation of LISI and GISI transform: We divided the original paragraph into two blocks to describe proposed method more clearly.

3. Figure 4 \& Figure 3: 
We have selected a more intuitive result (GISI$_t$) for display in **Fig.4**. The reconstruction results in **Fig.3** are also updated accordingly.


4. Figure 5: We have made standardized adjustments to the formula (Not limited to **Fig.5**).

5. Figure 9: Thanks for your suggestion. The official source code of STP has been used for testing which is comparison a fair comparison. Therefore, we still hope to keep the initial results in the main paper. Besides, we have also provided the adjusted results of STP in **Fig.20**.




6. typo "camrea": We have updated the relevant typo.

Reviewer comment (Presentation):
Explanation of LISI transform is in the caption, not in the section. In section 4.3, GISI and LISI suddenly appear, which make readers a bit confused. If these are also one of the new blocks proposed, they should be described well.

Author response:
**Q4: Related to display**

Thanks for your careful reading. We have made relevant modifications based on your suggestions:


1. Figure 1 \& Figure 3: 
We selected a color pair (green and red) with high contrast to represent forward and backward data streams. We have added a description of that temporary features for both forward and back are different.

2. Explanation of LISI and GISI transform: We divided the original paragraph into two blocks to describe proposed method more clearly.

3. Figure 4 \& Figure 3: 
We have selected a more intuitive result (GISI$_t$) for display in **Fig.4**. The reconstruction results in **Fig.3** are also updated accordingly.


4. Figure 5: We have made standardized adjustments to the formula (Not limited to **Fig.5**).

5. Figure 9: Thanks for your suggestion. The official source code of STP has been used for testing which is comparison a fair comparison. Therefore, we still hope to keep the initial results in the main paper. Besides, we have also provided the adjusted results of STP in **Fig.20**.




6. typo "camrea": We have updated the relevant typo.

Reviewer comment (Presentation):
In Figure 4, the example image of GISI does not show anything. While the figure can be illustrative, it is suggested that authors use a better image that can give a clue what the GISI output looks like.

Author response:
**Q4: Related to display**

Thanks for your careful reading. We have made relevant modifications based on your suggestions:


1. Figure 1 \& Figure 3: 
We selected a color pair (green and red) with high contrast to represent forward and backward data streams. We have added a description of that temporary features for both forward and back are different.

2. Explanation of LISI and GISI transform: We divided the original paragraph into two blocks to describe proposed method more clearly.

3. Figure 4 \& Figure 3: 
We have selected a more intuitive result (GISI$_t$) for display in **Fig.4**. The reconstruction results in **Fig.3** are also updated accordingly.


4. Figure 5: We have made standardized adjustments to the formula (Not limited to **Fig.5**).

5. Figure 9: Thanks for your suggestion. The official source code of STP has been used for testing which is comparison a fair comparison. Therefore, we still hope to keep the initial results in the main paper. Besides, we have also provided the adjusted results of STP in **Fig.20**.




6. typo "camrea": We have updated the relevant typo.

Reviewer comment (Presentation):
In Figure 5, notation is confusing. For example, if the (a) LISI ti=21 means the LISI output at ti equals 21, "=21" needs to be subscript along with ti. Currently it is $LISI_{ti} = 21$. Also suggest to use LaTeX outputs in the figures for math expressions.

Author response:
**Q4: Related to display**

Thanks for your careful reading. We have made relevant modifications based on your suggestions:


1. Figure 1 \& Figure 3: 
We selected a color pair (green and red) with high contrast to represent forward and backward data streams. We have added a description of that temporary features for both forward and back are different.

2. Explanation of LISI and GISI transform: We divided the original paragraph into two blocks to describe proposed method more clearly.

3. Figure 4 \& Figure 3: 
We have selected a more intuitive result (GISI$_t$) for display in **Fig.4**. The reconstruction results in **Fig.3** are also updated accordingly.


4. Figure 5: We have made standardized adjustments to the formula (Not limited to **Fig.5**).

5. Figure 9: Thanks for your suggestion. The official source code of STP has been used for testing which is comparison a fair comparison. Therefore, we still hope to keep the initial results in the main paper. Besides, we have also provided the adjusted results of STP in **Fig.20**.




6. typo "camrea": We have updated the relevant typo.

Reviewer comment (Presentation):
In Figure 9, maybe the dynamic range is uniform for all reconstructions. The STP shows an extremely different range, which makes other reconstruction methods totally not visible. (I expect STP images are totally saturated if we use dynamic range of other images.) How about using two different dynamic ranges, one only for STP and the other for other images? It would be okay with descriptions and specifying the min-max values. Anyway, currently the contrast is too low for reconstructed images.

Author response:
**Q4: Related to display**

Thanks for your careful reading. We have made relevant modifications based on your suggestions:


1. Figure 1 \& Figure 3: 
We selected a color pair (green and red) with high contrast to represent forward and backward data streams. We have added a description of that temporary features for both forward and back are different.

2. Explanation of LISI and GISI transform: We divided the original paragraph into two blocks to describe proposed method more clearly.

3. Figure 4 \& Figure 3: 
We have selected a more intuitive result (GISI$_t$) for display in **Fig.4**. The reconstruction results in **Fig.3** are also updated accordingly.


4. Figure 5: We have made standardized adjustments to the formula (Not limited to **Fig.5**).

5. Figure 9: Thanks for your suggestion. The official source code of STP has been used for testing which is comparison a fair comparison. Therefore, we still hope to keep the initial results in the main paper. Besides, we have also provided the adjusted results of STP in **Fig.20**.




6. typo "camrea": We have updated the relevant typo.

Reviewer comment (Writing):
In the last line of page 4, typo "camrea".

Author response:
**Q4: Related to display**

Thanks for your careful reading. We have made relevant modifications based on your suggestions:


1. Figure 1 \& Figure 3: 
We selected a color pair (green and red) with high contrast to represent forward and backward data streams. We have added a description of that temporary features for both forward and back are different.

2. Explanation of LISI and GISI transform: We divided the original paragraph into two blocks to describe proposed method more clearly.

3. Figure 4 \& Figure 3: 
We have selected a more intuitive result (GISI$_t$) for display in **Fig.4**. The reconstruction results in **Fig.3** are also updated accordingly.


4. Figure 5: We have made standardized adjustments to the formula (Not limited to **Fig.5**).

5. Figure 9: Thanks for your suggestion. The official source code of STP has been used for testing which is comparison a fair comparison. Therefore, we still hope to keep the initial results in the main paper. Besides, we have also provided the adjusted results of STP in **Fig.20**.




6. typo "camrea": We have updated the relevant typo.

Reviewer comment (Reproducibility):
In Fig. 2, the GT scenes seem to be RGB. How are they converted to gray scale?

Author response:
**Q1:  In Fig. 2, the GT scenes seem to be RGB. How are they converted to grayscale?**

We employ the grayscale conversion function provided by OpenCV, i.e., cv2.cvtColor(img, cv2.COLOR\_BGR2GRAY).

Reviewer comment (Evaluation):
Why is the PSNR value so high? For eg. the S2I image looks so noisy compared to GT in Fig.1, but according to table 1, the PSNR is > 40dB, which does not make sense.

Author response:
**Q2:  Why is the PSNR value so high?  The S2I image looks so noisy compared to GT in Fig.8, but according to table 1, the PSNR is higher than 40dB.**

When the scene is so dark, both the Ground Truth (GT) and the reconstructed images can be at a lower grayscale level. This results in a significantly lower MSE compared to normal-light scenes. Consequently, their PSNR is high.


The Peak Signal-to-Noise Ratio (PSNR) can be expressed as $PSNR = 10 \cdot \log_{10}\left(\frac{{\text{255}^2}}{{\text{MSE}}}\right),$
where MSE represents the Mean Squared Error. It can be further defined as,
$MSE = \frac{1}{mn} \sum_{i=0}^{m-1} \sum_{j=0}^{n-1} \left( I(i, j) - K(i, j) \right)^2,$where m (n) is the height (width) of the image, respectively and $I(i, j)$($K(i, j)$) represents the pixel value at (x, y) in ground truth (the reconstructed image). Hence, when both the Ground Truth (GT) and the reconstructed images are at a lower grayscale level, MSE is lower (PSNR is higher).  In Fig.8, the scene is exceptionally dark, and we applied the same gamma correction to the reconstructed result for better visibility to readers.

Reviewer comment (Experiments):
The forward model presented in the main paper doesn't model noise (except quantization). It's unclear how the proposed method would perform in a photon starved regime where a significant amount of Poisson noise would be present.

Author response:
**Q2: How the proposed method would perform where a significant amount of Poisson noise would be present?**

Our method can deal with Poisson noise better than the state-of-the-art methods. First, the mean of the error from Poisson noise in the temporal dimension is 0. Our bidirectional recurrent-based reconstruction method can effectively aggregate temporal information. Its structure is suitable for learning to filter Poisson noise. Besides, our training set is generated from the spike camera simulator  (Zhao et al.
2022a) and it considers Poisson noise and fixed-pattern noise in the spike camera.

Reviewer comment (Evaluation):
The paper doesn't link to any reconstructed videos. I can't evaluate if the proposed method introduced significant flickering artifacts.

Author response:
**Q1: Reconstructed videos**

We add reconstructed videos to **the supplementary material** where the state-of-the-art method WGSE is compared. We can find that the results of WGSE are very unstable and fluctuate greatly in the time domain, while our method can effectively handle temporal noise.

Reviewer comment (Presentation):
Figure 5 isn't particularly informative.

Author response:
**Q3: Figure 5 \& typos**

Thanks for your suggestions. We aim to enhance the reader's understanding of GISI through Fig.5. Following your advice, we have included links to the algorithm. Additionally, we have addressed typos in our paper.

Reviewer comment (Presentation):
Rather than stating, "The algorithm is in appendix", please state where in the appendix the algorithm can be found.

Author response:
**Q3: Figure 5 \& typos**

Thanks for your suggestions. We aim to enhance the reader's understanding of GISI through Fig.5. Following your advice, we have included links to the algorithm. Additionally, we have addressed typos in our paper.

Reviewer comment (Writing):
There are a number of typos (e.g., extra capitalizations) that a spell-checker should be able to catch and a few incomplete sentences (e.g., "A fusion module.").

Author response:
**Q3: Figure 5 \& typos**

Thanks for your suggestions. We aim to enhance the reader's understanding of GISI through Fig.5. Following your advice, we have included links to the algorithm. Additionally, we have addressed typos in our paper.

Reviewer comment (Experiments):
How would the algorithm behave in the presence of significant Poisson noise?

Author response:
**Q2: How the proposed method would perform where a significant amount of Poisson noise would be present?**

Our method can deal with Poisson noise better than the state-of-the-art methods. First, the mean of the error from Poisson noise in the temporal dimension is 0. Our bidirectional recurrent-based reconstruction method can effectively aggregate temporal information. Its structure is suitable for learning to filter Poisson noise. Besides, our training set is generated from the spike camera simulator  (Zhao et al.
2022a) and it considers Poisson noise and fixed-pattern noise in the spike camera.

Reviewer comment (Novelty):
What were the intuitions behind selecting the chosen architecture? Why not leverage any of the wavelet structure from WGSE?

Author response:
**Q4: What were the intuitions behind selecting the chosen architecture?**

This is an excellent question. We initially observed a pronounced performance degradation of reconstruction methods in low-light conditions. Subsequently, we noticed a significantly sparser distribution of spikes in low-light spike streams compared to those in normal-lighting conditions. The recurrent-based structure proves effective in aggregating information from different time. This forms the basis of our work. Finally, we design a tailored bidirectional recurrent reconstruction framework specifically for spike cameras. 

**Q6:  Why not leverage any of the wavelet structure from WGSE?**

The wavelet structure in WGSE (Zhang et al., 2023). exists solely within its representation. We replaced our representation with that of WGSE and retrained it. As shown in Table.3, the performance of our method (PSNR: 45.075 and SSIM: 0.9868) is better than that (PSNR: 42.302 and SSIM: 0.9744).

Reviewer comment (Evaluation):
The paper has not established benchmarks for the light robustness. It is very unclear how low is the "low light" used in this paper. It is also not addressed how robust the algorithm functions when comparing normal and low light conditions. A better approach would be to provide a quantification of performance versus light intensity.

Author response:
**Q1: Setting of light source power and How to achieve consistency with the real world?**

We set the lighting parameters in the advanced 3D graphics software, blender, to make the lighting conditions as consistent as possible with the real world. The following are the configuration details in Blender.

In Blender, various types of lighting simulation functions, including sunlight, point lights, and area lights, have been integrated into the graphical interface. We can adjust lighting parameters to control brightness and darkness. 

1. For sunlight in Blender, the watts per square meter can be modified. Typically, 100 watts per square meter corresponds to a cloudy lighting environment. For the 'Car$_L$' scene in Fig.13, we have set sunlight to 10 watts per square meter, which is deemed sufficiently low. 

2. For point lights and area lights, Blender allows modification of radiant Power, measured in watts.  This is not the electrical power of consumer light bulbs. A light tube with the electrical power of 30W approximately corresponds to a radiant Power of 1W. In the 'Cook$_L$' scene  in Fig.13, we have set an area light with the radiant Power to 1W (the electrical power of 30W) . It already represents a very dim indoor light source.

**Q2:How dark is our dataset?**

The low light scene in our dataset cover various dark range. The brightness is not only determined by the light source, but also by factors such as camera distance, object occlusion, and so on. These factors are ultimately reflected in the grayscale of the rendered image. Therefore, we calculate the grayscale histograms of images in low light scenes. As shown in **Fig.14** (each bar represents 5 grayscale levels). We can see that the grayscale is diverse and in a lower range.

**Q3: A quantification for performance vs light intensity.**

We generated spike streams under different lighting conditions by modifying the light source parameters. Due to the large time-consuming of once rendering for each type of lighting, we only use one scene to demonstrate the performance of our method. Related results are updated in **Table.4**. Our method demonstrates excellent performance.


Specific modification details: We set sunlight (see Q1) to 30, 50, 70, 90, 110 and 130 watts per square meter to render images respectively.

Reviewer comment (Experiments):
It appears that the LLR dataset has only two lighting conditions, i.e., normal and low. There is a question of whether this is enough for benchmarking. Additionally, there is a question about whether 5 motions are enough. The paper mentions that "... the power of light source is consistent with the real world", but it is unclear how this consistency is achieved. The dataset section lacks technical details and justification.

Author response:
**Q1: Setting of light source power and How to achieve consistency with the real world?**

We set the lighting parameters in the advanced 3D graphics software, blender, to make the lighting conditions as consistent as possible with the real world. The following are the configuration details in Blender.

In Blender, various types of lighting simulation functions, including sunlight, point lights, and area lights, have been integrated into the graphical interface. We can adjust lighting parameters to control brightness and darkness. 

1. For sunlight in Blender, the watts per square meter can be modified. Typically, 100 watts per square meter corresponds to a cloudy lighting environment. For the 'Car$_L$' scene in Fig.13, we have set sunlight to 10 watts per square meter, which is deemed sufficiently low. 

2. For point lights and area lights, Blender allows modification of radiant Power, measured in watts.  This is not the electrical power of consumer light bulbs. A light tube with the electrical power of 30W approximately corresponds to a radiant Power of 1W. In the 'Cook$_L$' scene  in Fig.13, we have set an area light with the radiant Power to 1W (the electrical power of 30W) . It already represents a very dim indoor light source.

**Q2:How dark is our dataset?**

The low light scene in our dataset cover various dark range. The brightness is not only determined by the light source, but also by factors such as camera distance, object occlusion, and so on. These factors are ultimately reflected in the grayscale of the rendered image. Therefore, we calculate the grayscale histograms of images in low light scenes. As shown in **Fig.14** (each bar represents 5 grayscale levels). We can see that the grayscale is diverse and in a lower range.

**Q4: Diversity of motion**


The motion in LLR are diverse. Each designed scene contains 1000 frames of images and the motion at different times is different. We generate a optical flow every 40 frames for LLR. The degree distribution of the optical flow is statistically analyzed in **Fig.15**. We can find that the motion in LLR covers all kinds of directions.

Reviewer comment (Novelty):
The global inter-spike interval (GISI) is a small extension of previous LISI. It is also very confusing what LISI is referring to. The two references [Chen 2022] proposed TFI and [Zhao 2022b] proposed DSFT (differential of spike firing). It is unclear whether the authors are referring to TFI and DSFT as the same thing. According to Table 2, the significance of GISI is marginal and is hardly considered a contribution.

Author response:
**Q6: TFI \& DSFT**

Both TFI and DSFT first calculate the interval between adjacent spikes in the input spike stream, and they both employ a brute-force search approach. The difference lies in the fact that TFI takes the reciprocal of each interval. 

**Q7: LISI VS. GISI**


GISI (our final method) not only outperform LISI (baseline (E) in Table.2) in both PSNR and SSIM on synthetic datasets but also have better generalization  on real spike streams. 


We update reconstruction results of two methods in **Fig.17**. More importantly, the cost of using GISI instead of LISI is negligible (we only need to use two 400x250 matrices to store the time of the forward spike and the backward spike, respectively), which does not affect the parameter and efficiency of the network at all.

Reviewer comment (Presentation):
The figures are well-made but lack explanation of technical details. Figure 5 generated confusion about what "LISI", "update", and "maintain" mean. Mathematical formulation is needed. Figure 7 did not provide useful information and is redundant to present after Equations 5-9.

Author response:
**Q8: Fig.5 \& Fig.7**


Thanks for your suggestions. 

1. For **Fig.5**, we have included additional supplementary explanations for LISI in the appendix and highlighted them in the main paper. The "maintain" and "update" are unique operations in GISI. For better understanding, it is suitable to be written as pseudocode. The pseudocode has been written in the appendix. In the caption of Fig.5, we have updated the link for the specific location of the algorithm. 

2. For **Fig.7**, we have also adjusted its positioning to better illustrate the specific modules of the network, placing it before the formulas.

Reviewer comment (Writing):
The presentation requires improvement, as there are many grammar errors.

Author response:
**Q9: Grammar errors**

We have corrected the relevant Grammar errors.

Reviewer comment (Experiments):
From Figure 11, it seems that the method converges well at 5 frames and even has worse results when comparing 21 to 13 for PSNR. This may contradict the choices for frame numbers.

Author response:
**Q11: From Fig.11, it seems that the method converges well at 5 frames and even has worse results comparing 21 to 13 for PSNR. Is this contradicting the choices for frame numbers?**

Our method is robust to the setting of the number of spike streams within a certain range, i.e., when the number of spike streams exceeds a certain value, the method performance can converge (with certain fluctuations). Therefore, as long as the selected hyperparameter is within the appropriate range for test, it is reasonable.

Reviewer comment (Reproducibility):
It is unclear where exactly noise is being handled. The only indication seen is an I_{dark} on top of I as in current, but it is unclear if that is all.

Author response:
**Q12: Noise of spike cameras**

We use the spike camera simulator  (Zhao et al. 2022a) to generate data. It takes into account Poisson noise and fixed-pattern noise. By doing so, we believe our data is closer to that from the real scenes since noises are inevitable in the real world. In addition, training on data with noises can help the model avoid overfitting as shown in **Fig.16**. Besides, we never claim our method aims to deal with the noise problem as a contribution, this discussion is out of the scope of the main paper.