Paper: LDINet: Latent Decomposition and Interpolation for Single Image FMO Deblatting (ICLR-2024)

Reviewer comment (Novelty):
The contributions are somehow incremental. The encoder-decoder framework and most of the training losses mentioned in this article have already been proposed in DeFMO.

Author response:
**Q4: The contributions are somehow incremental. The encoder-decoder framework and most of the training losses mentioned in this article have already been proposed in DeFMO.**

A4: Although our method is built with a similar encoder-decoder framework and loss functions, we want to emphasize that the novelty of our method mainly lies in the decomposition-interpolation paradigm, which allows naturally introducing the required time indexes via interpolation operation. And the well-designed AffNet helps perform feature alignment between different frames during the interpolation. Further, the splitting scheme of the scalar-like and gradient-like feature maps could further improve the alignment effect as the processing way of scalar and gradient fields under the affine coordinate transformations is different. This helps generate a more accurate latent frame at the target time index for decoding. These designs can well distinguish our method from DeFMO. We will add such explanations in the revision.

Reviewer comment (Experiments):
Although interpolating in hidden space seems to have some intuitive benefits, and the proposed network achieve empirically improvements, the authors do not seem to provide ablation study to verify the improvement of DIB module. The authors should conduct further experiments to explore the impact of the DIB module and add some understanding experiments if possible to illustrate how DIB "exploits the intrinsic structure of the latent space" and thus benefiting the deblatting.

Author response:
**Q5: More ablation studies and experiments for the DIB module.**

A5: We have provided an ablation study to compare the performance between settings of linear interpolation and affine interpolation in the DIB module. Please refer to the common response part.  
In addition, we also provide the visualization of their output images in the supplemental material for better understanding.

Reviewer comment (Theory):
I'm puzzled as to why the weights in the formula at the bottom of page 4 (which has no formula label) are set this way and why they serve the purpose of "fully leverage the information from both the two neighboring parts."

Author response:
**Q1: why the weights in the formula at the bottom of page 4 (which has no formula label) are set this way and why they serve the purpose of "fully leverage the information from both the two neighboring parts.**

A1: The weights in this formula are set in such way to guarantee that $Q_{\tau}$ is identity to $P_{\tau}$ when $\tau = t$ or $\tau=t’$. In the case that $\tau \in (t, t’)$, a larger weight is given to the transformed result from the closer latent part and a smaller one to the other transformed result. Since the target latent frame $Q_{\tau}$ is a combination of two transformed results from two neighboring latent parts, it would fully leverage the information from both sides.

Reviewer comment (Theory):
Could you explain the motivation for choosing the direction with a smaller relative error rate when determining the trajectory direction?

Author response:
**Q3: The motivation for choosing the direction with a smaller relative error rate when determining the trajectory direction.**

A3: For the direction selection, we actually follow DeFMO to leverage such a strategy for determining the appropriate trajectory direction. The difference is that our method only calculates the relative error rate on the masks. Since in the generated results, only the masked part of the appearance is what we need, it can be considered that the reconstruction of the mask needs to be focused on.

Reviewer comment (Theory):
The sum that denotes the number of pixels within the FMO blur in Equation (6) should rather be written as $\sum_D \sum_{\tau} (M_{\tau} > 0)$. Otherwise, I don't understand how it can normalize the losses. Moreover, why is this normalization different for the first (without $>0$) and the second term (with $>0$)?

Author response:
A6: Since $I$ denotes the blur image which is the average of a series of sharp images at the given time indexes within the exposure time, the formula reviewer provided could count the same pixel multiple times because of its appearance in masks of different time indexes. In our formula, we use a pixel-wise indicator function $[\cdot >0]$ to set each pixel that appears in any mask to 1 which avoids the multiple-counting.  

On the other hand, $>0$ is part of the pixel-wise indicator function $[\cdot >0]$. The first normalization term is without the function because the mask $M_\tau$ is a binary map whose value is either 0 or 1 for different pixels and thus it could be directly summed. While the second normalization needs the function to convert the summation over time indexes $\sum_{\tau} M_{\tau}$ to a binary map to further count the size of the blurred area.

Reviewer comment (Presentation):
Arch. (bi-branched) ablation in Table 2 is not clear. The paper says that it provides improvement by separating the estimation of the appearance and the mask. Does it mean that otherwise, they are not separated? What does it even mean? This is very confusing.

Author response:
A7: In the decoder, we separate the estimation of the appearance and the mask with two different branches as in Figure 1. In particular, this separation happens after the first two Resblocks of the decoder. Otherwise, they are estimated with one shared branch setup. We will add more explanations to the revision.  
In addition, for the dependence of pretraining, we provide the following ablation study to show its advantage:  
| Method |Falling TIoU| Falling PSNR|Falling SSIM| TbD-3D TIoU | TbD-3D PSNR | TbD-3D SSIM | TbD TIoU |TbD PSNR| TbD SSIM |  
| :-: | :-: | :-: | :-: | :-: |:-: |:-: |:-: |:-: |:-: |  
|DeFMO | 0.684| 26.83 |0.753 |0.879 |26.23| 0.699 | 0.550 | 25.57 | 0.602|  
|w/ pretrain| 0.686(0.007) | 28.087(0.006) | 0.771 (0.001) | 0.906(0.002) | 26.503(0.136) | 0.707(0.003) | 0.616(0.004) | 25.242(0.122) | 0.626(0.008) |  
| w/o pretrain| 0.697 |27.706 | 0.766 | 0.906 | 26.490 | 0.706 | 0.615 | 25.702 | 0.626 |  

Since time is limited, we only report the one-run results for w/o pretraining here and we will further complement the results.

Reviewer comment (Writing):
There are many typos in the paper:  
- Abstract: with backgrounds -> with background  
- Abstract: the feature maps of the long blur is -> are  
- Abstract: experiments are conducted and has shown -> have shown  
- Intro (p1): which is accordance -> is in accordance  
- Conclusion: feature maps is -> feature maps are

Author response:
A10: Thanks a lot, and we will correct the typos and revise our paper more carefully. For the description of "Conventional methods mostly recover a clear image at the median of the motion", such expression is only to illustrate the common problem setup in the existing literatures. We will revise accordingly to avoid confusion.

Reviewer comment (Theory):
"Conventional methods mostly recover a clear image at the median of the motion": this is not true, I'd even say they recover a clear image at an arbitrary location of the motion.

Author response:
A10: Thanks a lot, and we will correct the typos and revise our paper more carefully. For the description of "Conventional methods mostly recover a clear image at the median of the motion", such expression is only to illustrate the common problem setup in the existing literatures. We will revise accordingly to avoid confusion.

Reviewer comment (Experiments):
It would be helpful to add SfB results if sub-frame silhouettes are used from the proposed method as input.

Author response:
A2: We have provided the results for SfB with silhouettes generated by LDINet as follows:  
| Method |Falling TIoU| Falling PSNR|Falling SSIM| TbD-3D TIoU | TbD-3D PSNR | TbD-3D SSIM | TbD TIoU |TbD PSNR| TbD SSIM |  
| :-: | :-: | :-: | :-: | :-: |:-: |:-: |:-: |:-: |:-: |  
|  SfB (ours)  | 0.658(0.012) | 25.348(0.421) | 0.730(0.004) | 0.843(0.003) | 24.172(0.096) | 0.651(0.001) | 0.601(0.000) | 24.130(0.062) | 0.609(0.001) |  
|SfB (DeFMO)| 0.650(0.001) | 24.982(0.096) | 0.732(0.003) | 0.864(0.000) | 23.982(0.027) | 0.640(0.002) | 0.561(0.001) | 24.164(0.030) | 0.592(0.002) |  

Since time is limited, we only report the 8-step results with two prototypes and set the learning rate to 0.03 for stable optimization in SfB. And the std is provided in parentheses for 3 times experiments. We will add this comparison into the revision.

Reviewer comment (Theory):
Convolution is defined as a linear combination of the signals within a spatial window in an image. However, it is not clear to the reviewer why we need to further decompose the convolution as a linear combination of summation and series of directional derivatives. What is the benefit of this further decomposition? It seems that the Taylor expansion of the latent is applied at time t to approximate the latent at any time t? It was not clearly motivated in the paper.

Author response:
A1: The main reason is to help improve the alignment effect by the affine transformation. In particular, by decomposing the convolution as a linear combination of summation and series of directional derivatives, the resulting feature maps are correspondingly split into scalar-like and gradient-like feature maps. Further, the scalar-like and gradient-like feature maps have different processing ways under the affine coordinate transformation. Thus, different ways of transformation operations are enforced on these two kinds of feature maps to fully exploit their particular characteristics, and thus better alignment effect could be achieved, which helps improve the embedding quality for any time index and obtain better overall performance.

Reviewer comment (Theory):
Affine transformation. The paper mainly describes what has been implemented to approximate the affine transformation via prediction of two affine transformations (one is computed from I_t to I_t’ and the other one is computed from I_t’ to I_t). Would this be redundant as one transformation should be the inverse of the other?

Author response:
A2: We predict the two affine transformations mainly for the consideration of the numerical stability of the model. Empirically, using the inverse operation directly has the risk of numerical instability, especially in the first several epochs in the training. Such a design of modeling the transformation from both directions also provides some flexibility when either one happens to be not precise.

Reviewer comment (Theory):
Eq.10 is not correct. It should be the Frobenius Norm for computing the relative distance between two matrices not the standard MSE.

Author response:
A3: Thanks for pointing this out. We will correct it by using the Frobenius Norm for computing the relative distance between two matrices.

Reviewer comment (Theory):
There is no guarantee that the predicted affine transformation is invertible. While losses are introduced to guide the learning process, no hard constraint is enforced in the framework.

Author response:
A4: For the invertibility, based on the experimental statistics, the id loss in Equation 10 is less than 1e-4 during the training process, which empirically shows that the predicted affine transformation approximates to be invertible. We will add such analysis in the revision.

Reviewer comment (Theory):
All the equations should be written properly. D is referred to as the domain for the pixel coordinate. However, no index ever appears in the loss function for Eq. (5,6,7,8).

Author response:
A5: Thanks for pointing them out. We will revise the Equations 5, 6, 7, and 8 accordingly.

Reviewer comment (Evaluation):
No supplementary is provided in the submission. Could the authors show the recovery of the appearance of the FMO in a video as the proposed method can reconstruct the image at any time within the exposure time? The qualitative results in Figure 6 in the appendix cannot demonstrate the effectiveness of the proposed method. The masks recovered by the proposed method and DeFMO look the same.

Author response:
A6: In the supplemental material, we have provided the compared results for the recovery of the appearance of the FMO in the form of videos.

Reviewer comment (Presentation):
The authors fail to provide evidence of why the proposed method works. Also, the presentation of this paper is too poor to be fully understood.

Author response:
A2: Thanks for your suggestion. We would reorganize the paper and provide more required explanations for the important components of the proposed method as is pointed out. To further validate the effectiveness of our work, we would add more ablation/experimental studies. For example, (1) We would compare with the results by linear interpolation to show the benefits of the affine transformation in our interpolation method, (2) We would provide an ablation setting to show the importance of the pre-training, and (3) We would provide more qualitative results to show the benefits of the decomposition-interpolation paradigm.

Reviewer comment (Novelty):
More details should be added to discuss the difference between the proposed network and the prior work DeFMO and describe why the proposed method is better.

Author response:
(1)	The main difference between our method and DeFMO lies in the way of constructing latent embedding for different time indexes. DeFMO concatenates the target time index with shared latent feature maps as the corresponding embedding for the subsequent decoding. Instead, our method first decomposes the feature maps in latent space into several latent parts and then generates the required corresponding embedding for any time index via an interpolation method with affine transformation. This helps greatly improve the flexibility of generating diverse outputs for different time indexes. Further, in the decoder part, DeFMO generates the mask and appearance of the object with only one branch, while our method generates them with two separate branches as they have different value distributions. More details about the difference will be added in the revision.

Reviewer comment (Experiments):
What are scalar-like and gradient-like feature maps? How does the network get them? Why are they so important? It also lacks an ablation study to validate it.

Author response:
(2) For the scalar-like and gradient-like feature maps, such design mainly considers the characteristics of convolution operation under the affine coordinate transformations to improve the interpolation quality. In particular, on the one hand, the convolution operation could be seen as a linear combination of summation and series of directional derivatives, where the directional derivatives are linear projections of the gradient fields. On the other hand, the processing way of scalar and gradient fields under the affine coordinate transformations is different. Thus, it is very beneficial to represent the feature maps output by the convolution operation as a combination of scalar-like and gradient-like feature maps. Such a well-designed splitting scheme helps improve the alignment effect by the affine transformation.
In practice, our model learns the scalar-like and gradient-like feature maps with a pre-training stage. In the pre-training, we first obtain the latent parts for the original input image. Then we split the channels of latent parts into the scalar-like and gradient-like categories where each scalar-like feature map has a single channel and each gradient-like feature map has two channels as the two components of the gradient field, which are processed by an estimated affine transformation by AffNet. Further, the resulting feature maps are forced to be consistent with the corresponding ones for a distorted input version which is produced by applying a pre-set affine transformation to the FMO blur contained in the original input image. Meanwhile, in this process, the estimated affine transformation is also encouraged to be similar to the pre-set one. The related ablation study is provided in Table 4 to validate the importance of generating two kinds of feature maps instead of only one kind. More details will be added in the revision accordingly.

Reviewer comment (Theory):
The detailed process from eq. 3 to eq. 5 is not clear which makes it hard to understand. Also, how this process is presented in Fig. 2 is also not clear.

Author response:
(3) In particular, Equation 3 and Equation 4 formulate the supposed transformations for the scalar-like and gradient-like feature maps, respectively. In Equation at the bottom of page 4, a function $\Phi$ is first defined to split the input feature map into the scalar-like and gradient-like feature maps, which are then processed with the transformations in Equation 3 and 4 correspondingly. The two respective results are concatenated as the output. Therefore, the Equation 3 and 4 provide the important component for the operation of the Equation at the bottom of page 4. For Figure 2, since it is defined that the latent feature map $P_t$ is the concatenation of the scalar part $P_t’$ and the gradient part $P_t’’$, only the process of the Equation at the bottom of page 4 is explicitly presented in Figure 2(a), where the function $\Phi$ takes as input ($P_t$, $A_{\tau\rightarrow t}$) and ($P_{t’}$, $A_{\tau\rightarrow t’}$) for generating $Q_{\tau}$. More explanations will be added in the revision.