Paper: Seeing Video Through Optical Scattering Media using Spatio-Temporal Diffusion Models (ICLR-2024)

Reviewer comment (Novelty):
The key issue and the reason I chose a rating of 2 for the presentation is the lack of contextualization for this work. While the authors do a good job comparing to some baselines like a TV method and a 2D approach, the problem of seeing through scattering media has a long history, which this paper largely ignores. This problem arises in the context of de-hazing and underwater imaging (see e.g. Akkaynak et al and Berman et al below).

Author response:
> **W1**. The key issue and the reason I chose a rating of 2 for the presentation is the lack of contextualization for this work. While the authors do a good job comparing to some baselines like a TV method and a 2D approach the problem of seeing through scattering media has a long history, which this paper largely ignores. This problem arises in the **(1) context of de-hazing and underwater imaging**. 
I also point out that the inverse problem is very similar to the **(2) DiffuserCam**. **(3) Further works by Satat et al. Bar et al. Alterman et al. all looked into seeing through scattering and Bar et al. offer a simple model for speckle formation**. 
Lastly, the **(4) discussion on speckles seem redundant** as the paper reduces the model to a simple convolution with a gaussian kernel. This opens the discussion to a whole host of works done on blind and non-blind deconvolution.

Please see General Comment 2 & 3. 

DiffuserCam has not been mentioned in General Comment 2, so we will provide following response here. We started the derivation by substituting z=0 to derive a Gaussian blur kernel. We emphasize that the distance from the imaging plane from the output plane of scattering media can be non-invasively controlled (e.g. Imaging right in front of the skin). If this **distance becomes large**, it can be considered similar to the case of a DiffuserCam, leading to a speckle PSF rather than a Gaussian psf.

In DiffuserCam, they used this speckle PSF for compressed sensing encoding of 3D information, which has significance. However, due to the Memory effect, the forward model has a significant limitation with a hard limit on the field of view. Therefore, the speckle PSF is unknown without the speckle pattern calibration according to various distances(not applicable to an unseen diffuser), and the object must be spatially incoherent amplitude target. In this point of view, our forward model does not require any calibration to applicate to an unseen diffuser.

Reviewer comment (Novelty):
The discussion on speckles seems redundant as the paper reduces the model to a simple convolution with a Gaussian kernel. This opens the discussion to a whole host of works done on blind and non-blind deconvolution, which are not addressed.

Author response:
> **W3**. A minor point: the paper alternates between a differentiable model and a closed form model, which do not overlap. One can have a differential scattering-based model that is nonetheless a non-closed form model.

Thank you for your interesting idea. We will consider it for the future work.

Reviewer comment (Experiments):
All the results and experiments assume a simple convolution model to generate data and then show the recovery based on that model, meaning there is no model mismatch at all. The authors should expand on the implications of this assumption.

Author response:
> **W4**. Some relevant work that should be acknowledged and contextualized.

Please see the General comment 3. 

In the context outlined above, we undertook a thorough review of all the work in this field, categorizing it into the three specified regimes. Additionally, we engaged in a detailed discussion on the state-of-the-art advancements. Thank you for the suggestion which brings our paper could give the readers a more complete perspective on the field.

Reviewer comment (Writing):
The paper alternates between a differentiable model and a closed-form model, which do not overlap. One can have a differential scattering-based model (e.g. Nimier-David, Merlin, et al., "Mitsuba 2: A retargetable forward and inverse renderer.") that is nonetheless a non-closed form model, and this distinction needs clarification.

Author response:
> **W3**. A minor point: the paper alternates between a differentiable model and a closed form model, which do not overlap. One can have a differential scattering-based model that is nonetheless a non-closed form model.

Thank you for your interesting idea. We will consider it for the future work.

Reviewer comment (Experiments):
There is a lack of context and comparison with other state-of-the-art approaches that have shown good results in real-world hazy images, which weakens the paper’s positioning in the field.

Author response:
> **Q1**. lack of context and comparison with other state-of-the-art approaches that have shown good results in real-world hazy images.

Please see the General comment 2 & 3. 

Per your suggestion, we engaged in a detailed discussion on the state-of-the-art advancements including de-hazing and underwater imaging papers suggested by reviewer dFYB. On the other hand, due to the different perspective in the context, utilizing it as a comparative method proves challenging.

Reviewer comment (Theory):
The authors should elaborate on the lack of model mismatch when assuming a simple Gaussian kernel and then recovering under this assumption. It is unclear how to evaluate the figure in the appendix, and whether other, more physically realistic methods for rendering scattering effects might perform much better.

Author response:
> **Q2**. I would like the authors to elaborate on the lack of model mismatch by assuming a simple gaussian kernel and then recovering under this assumption.

Please see General comment 1.

Per your suggestion, we demonstrate a description of the model mismatch in Appendix A.5 of the revised manuscript, especially for noise and phase errors. Our real experiment results serve as a demonstration that our forward model can effectively overcome model mismatch in practical scenarios.

> **Q3**. I'm not sure how to evaluate the figure in the appendix.

We demonstrate how well our forward model models the imaging of an object located at a distance $d$ behind a dynamic scattering medium. 

First, according to our forward model, we propagate a phase object by 6/12mm using the angular spectrum method. We take the intensity and convolve it with the Gaussian PSF of the scattering medium, resulting in the image in the simulated column.
Then, we position the actual phase target at a distance of 6/12mm from the scattering medium and get the image in the real measurement.
The correlation between the two images is calculated using the following formula, with a maximum value of 1:

$r(A,B) = \frac{\sum\limits_{m}\sum\limits_{n}(A_{mn}-\bar{A})(B_{mn}-\bar{B})}{\sqrt{(\sum\limits_{m}\sum\limits_{n}(A_{mn}-\bar{A})^2)(\sum\limits_{m}\sum\limits_{n}(B_{mn}-\bar{B})^2)}}$
where, $\bar{X}$ is the mean value of image $X$.

Per your thorough feedback, we include the description in Appendix A.3 in the revised manuscript.

> **Q4**. I do not know if other, more physically realistic methods for rendering scattering effects might do much better.

In tasks such as de-hazing or underwater imaging which fall in to the regime 2 of General comment 2, which shares the similar context, the approach suggested by the reviewer may indeed be effective. However, applying the approach to the problem we are dealing with might be challenging.

Reviewer comment (Writing):
The paper is well introduced and clearly explained but the derivations in the appendix are quite hard to follow.

Author response:
Per your suggestion, we simplified the explanation on the forward model with technical reports to support the description. The key focus of the forward model lies on coherent propagation and the incoherent scattering kernel. We revised it to emphasize concentration on these aspects. The Gaussian approximation of the scattering kernel shape is a minor point, so we omitted the details.

Reviewer comment (Theory):
A reference to a technical report with the complete derivations would be very useful.

Author response:
Per your suggestion, we simplified the explanation on the forward model with technical reports to support the description. The key focus of the forward model lies on coherent propagation and the incoherent scattering kernel. We revised it to emphasize concentration on these aspects. The Gaussian approximation of the scattering kernel shape is a minor point, so we omitted the details.

Reviewer comment (Reproducibility):
I was not able to find the description about the learning step in the paper and would like more information about this important step.

Author response:
We have listed the training details in Appendix C for training our diffusion prior. It follows exactly same learning step with VDM which learns target(video data before scattering) distribution which means it does not require any information about physical forward model during learning(training) step. This is one the key advantages of our method that it can handle any variations in forward physical model, once training done.

Reviewer comment (Experiments):
This paper only contains restoration results from **simulated** scattering. No successful restoration on any real-world scattering were demonstrated.

Author response:
Please see the General comment 1. 

We agree the results on real scattering measurements are required. In the revised manuscript, we add experimental results in real-world optical setting. Our model still works while Poissonian shot noise and camera dark noise were added real scattering measurements.

Reviewer comment (Theory):
This paper only focuses on a naive special case of scattering, where the forward operator is trivially a Gaussian blur. Real-world scattering is much more complicated and requires the modeling of phase error caused by the scattering medium.

Author response:
Please see the General comment 2. 

Our model is the first to combine a wave propagation model and a convolution with a blur kernel in the context of seeing through dynamic scattering media. With these modifications, may deal with the problem of object positioned at arbitrarily distance from scattering medium and the medium’s thickness of far beyond the regime discussed in Regime 3 of general comment 2.

We emphasize that dealing with the problem of object positioned at arbitrarily distance from scattering medium and the medium’s thickness is sufficiently complicated problem representing real-world scattering.
We believe our model has a great significance in imaging through scatter problem and makes an advancement compared to the previous methods.

Reviewer comment (Novelty):
Ignoring the significance on the problem of optical scattering, the technical contribution on the algorithm side is very limited. The paper introduces minimal changes to existing approaches that apply the diffusion posterior sampling strategy on other inverse problem tasks.

Author response:
As reviewer AVri kindly summarize our work, our method shows similar extension allowing to extend Diffusion models as Video Diffusion Models (VDM). The main contribution on the algorithm side is that the exploiting the  spatio-temporal correlation can bring huge improvements not only for generative models but also for inverse problem solvers like DPS. To best of our knowledge, this work is first video reconstruction study using diffusion prior and optical forward model. It is simple but very effective method for real-world, because real-world signals mostly temporally coherent. We firmly believe that our approach will serve as a source of inspiration for both the computer vision and optics research fields.

Reviewer comment (Experiments):
What's stopping the proposed method from successfully working on real-world scattering? Can the proposed framework handle more sophisticated forward model?

Author response:
This method retains all the advantages of DPS. In other words, it can restore measurements even in cases where shot-noise and Gaussian noise are added to the measurement, also for the forward operator is nonlinear. 
The capacity to address shot noise is a pivotal factor, our method highly effective in precisely restoring real optical measurements. In this context, our paper makes a significant contribution by illustrating the feasibility of such a forward model design in practical applications. Consequently, we demonstrate state-of-the-art results within this framework.