Paper: Image Inpainting via Iteratively Decoupled Probabilistic Modeling (ICLR-2024)

Reviewer comment (Writing):
In Eq. 1, does $\mathcal(N)$ mean the density function of the Gaussian distribution?

Author response:
Correct, $\mathcal N(\cdot, \cdot)$ is the density function of Gaussian.

Reviewer comment (Writing):
In Eq. 1, what's the relationship between the $y$ being integrated and the $y$ in the integral bounds? Are they the same or different?

Author response:
They are different. The $\boldsymbol y$ in the integral bounds refers to pixel values of the ground-truth unmasked image. Meanwhile, the $y$ being integrated follows a Gaussian distribution with the estimated mean and variance. We appreciate your highlighting this distinction. For improved clarity, we have represented the integrated variable as $z$ in the revised manuscript.

Reviewer comment (Theory):
In Eq. 1, where does $y_i$ come from during training? Is it the ground-truth unmasked image or the GAN-generated one? If it is the latter, it also depends on $\theta$?

Author response:
The $\boldsymbol y^i$ denotes the $i$-th pixel of the ground-truth unmasked image. We have explicitly clarified this in the revised manuscript.

Reviewer comment (Writing):
Above Eq. 9, there seems to be a typo: 'filed' -> 'field'

Author response:
Thanks for pointing this out. We have fixed this typo.

Reviewer comment (Evaluation):
There are some suspiciously accurate in-painting results in the appendix - particularly in Fig. M7. Can authors explain how the proposed model can be so accurate? I can understand being realistic, however, the third column shows that the generation is pretty much the same. Similar in-painting results are given in Figure M8.

Author response:
There is no problem with training and test data leakage. Following established methodologies like LaMa [a] and MAT[b], we utilize the official training (24183 images) and validation (2993 images) sets of CelebA-HQ. We have carefully checked that individuals present in the validation set do not overlap with the training data. 

    Our model performs well when the input image contains sufficient visible pixel information, enabling high-quality generation while maintaining coherence with the existing content. This success originates from our model's pixel spreading mechanism, which initiates from visible pixels and progressively diffuses valuable information throughout the image.  This approach is particularly effective for facial images characterized by strong inherent priors, such as symmetry and structural attributes. In the third example in Figure N.7, where the right half of the face is visible, our model reconstructs a comparatively realistic and consistent outcome using visible features like eyes and beard, albeit with potential discrepancies in details like ears and nose. However, when the entire face is masked, as in the first example in Figure N.7, our generated facial image notably diverges from ground truth. This phenomenon also appears in natural scenes; for instance, in the third example in Figure N.4, our method successfully restores the airplane structure due to the visibility of the frontal part, while other methods fail to inpaint such regions.

    We will release the training code and models. All results are reproducible.
    
    [a] Suvorov, Roman, et al. "Resolution-robust large mask inpainting with fourier convolutions." WACV. 2022.  
    [b] Li, Wenbo, et al. "Mat: Mask-aware transformer for large hole image inpainting." CVPR. 2022.

Reviewer comment (Experiments):
The in-painting results are suspiciously good in some cases. Can authors explain how their model can be so accurate - not just realistic - in M7 and M8? May there be a problem with training and test data leakage?

Author response:
There is no problem with training and test data leakage. Following established methodologies like LaMa [a] and MAT[b], we utilize the official training (24183 images) and validation (2993 images) sets of CelebA-HQ. We have carefully checked that individuals present in the validation set do not overlap with the training data. 

    Our model performs well when the input image contains sufficient visible pixel information, enabling high-quality generation while maintaining coherence with the existing content. This success originates from our model's pixel spreading mechanism, which initiates from visible pixels and progressively diffuses valuable information throughout the image.  This approach is particularly effective for facial images characterized by strong inherent priors, such as symmetry and structural attributes. In the third example in Figure N.7, where the right half of the face is visible, our model reconstructs a comparatively realistic and consistent outcome using visible features like eyes and beard, albeit with potential discrepancies in details like ears and nose. However, when the entire face is masked, as in the first example in Figure N.7, our generated facial image notably diverges from ground truth. This phenomenon also appears in natural scenes; for instance, in the third example in Figure N.4, our method successfully restores the airplane structure due to the visibility of the frontal part, while other methods fail to inpaint such regions.

    We will release the training code and models. All results are reproducible.
    
    [a] Suvorov, Roman, et al. "Resolution-robust large mask inpainting with fourier convolutions." WACV. 2022.  
    [b] Li, Wenbo, et al. "Mat: Mask-aware transformer for large hole image inpainting." CVPR. 2022.

Reviewer comment (Writing):
What's the meaning of $y^i$ in equation (1)? Is it the groundtruth?

Author response:
The $y^i$ represents the $i$-th pixel of the ground-truth image. We have revised the paper to improve the clarity of this illustration.

Reviewer comment (Theory):
More explanation is necessary for equation (5). What happens when $m_t=1$ and $m_0=0$ or vice versa?

Author response:
In the mask $\boldsymbol m$, we denote known regions with values of 1 and missing regions as 0. In Eq.5, we assign pixel values in the missing regions to 0 while preserving the original pixel values in the initially known area. Subsequently, we progressively fill in the missing regions using sampled results based on newly estimated mean and variance values. It's important to note that the previously inpainted content in the missing regions is updated at each iteration to maintain consistency with the newly inpainted pixels. 

In cases when $\boldsymbol m_0=0$, the initially visible pixels are absent. We consider this as a generation task. However, in our inpainting mask, such a phenomenon does not occur since we do not set $\boldsymbol m_0=0$. Furthermore, we only set $\boldsymbol m_t=1$ during the final iteration to ensure training convergence. When $\boldsymbol m_0=1$, all pixels are regarded as valid. So we will not update anymore.

Reviewer comment (Reproducibility):
Is the uncertainty map computed only in the mask region or the entire image?

Author response:
The uncertainty map is computed for the entire image. As discussed in the second step (i.e., 'Pick'), the uncertainty score of each initially known pixel is set to 0. Additionally, for the pixels that have been inpainted up to the current time step, their uncertainty scores  are computed by using the $\boldsymbol \sigma_t$ map. For the pixels located in regions that are still missing, their values are set as 1.

Reviewer comment (Experiments):
Comparison to Controlnet is necessary.

Author response:
Thank you for the constructive comments. In addition to ControlNet [a], we also include MI-GAN \[b\] (ICCV 2023) in our comparisons. Although ControlNet is not trained on the Places2 dataset, it undergoes training on a more extensive LAION dataset. Results from our proposed PSM show significant improvements across all metrics, highlighting the effectiveness of our method. We observe that ControlNet might produce undesired results due to its tendency to generate new objects that might not align harmoniously with the existing content. We have added the comparisons to Sec.I of the supplementary material due to the page limit.
    
#### 512x512 Places2 under the small mask ratio setting
| | | | | | |
|---|---|---|---|---|---|
| Method | FID$\downarrow$ | P-IDS$\uparrow$ | U-IDS$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ |
| **PSM (ours)** | **0.72** | **30.95** | **43.91** | **0.084** | **25.51** |
| MIGAN | 1.40 | 18.43 | 39.35 | 0.103 | 24.38 | 
| ControlNet | 1.86 | 12.63 | 35.71 | 0.117 | 24.68 |
| | | | | | |
    
#### 512x512 Places2 under the large mask ratio setting
| | | | | | |
|---|---|---|---|---|---|
| Method | FID$\downarrow$ | P-IDS$\uparrow$ | U-IDS$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ |    
|**PSM (ours)**| **1.68** | **25.33** | **39.30** | **0.161** | **20.89** |
|MIGAN| 3.81|13.50|32.42|0.195|20.00|
|ControlNet|5.55|6.60|25.65|0.219|19.73|
| | | | | | |

[a] Zhang, Lvmin, Anyi Rao, and Maneesh Agrawala. "Adding conditional control to text-to-image diffusion models." ICCV. 2023.  
[b] Sargsyan, Andranik, et al. "MI-GAN: A Simple Baseline for Image Inpainting on Mobile Devices." ICCV. 2023.

Reviewer comment (Evaluation):
It seems that sometimes the model is hard to grasp some structure of the image. For example, in the last row of Fig.4, the result of Model A cannot reconstruct the structure of stairs.

Author response:
We agree that our model sometimes hard to grasp some structures during generation. This is attributed to our pixel spread model, which functions by spreading informative pixels from initially visible regions across the entire image. In the last example depicted in Figure 4, despite some remaining stairs, the spreading process appears to be primarily influenced by the window representation due to its frequent appearance in the input. Consequently, our model effectively reconstructs the windows. However, even in the absence of stairs, our output remains reasonable.

Indeed, it is very challenging to consistently generate all subtle structures with high fidelity, as the content of the free-form masked regions is typically complex and ambiguous. This challenging factor also deteriorates diffusion models during generation although they have shown great generation abilities, as shown in Figure 6. We will concentrate on improving the generation quality in our future work.

Reviewer comment (Reproducibility):
Please explain in detail how the uncertainty is obtained in each iteration.

Author response:
1. **Please explain in detail how the uncertainty is obtained in each iteration.**

    We provide the details of obtaining the uncertainty in each iteration. Taking the $t$-th iteration as an example, we forward the masked image $\boldsymbol x_{t-1}$, binary mask $\boldsymbol m_{t-1}$ and uncertainty map $\boldsymbol u_{t-1}$ to the network to obtain the estimated mean $\boldsymbol \mu_t$ and variance $\boldsymbol \sigma_t^2$. To obtain a preliminary uncertainty map $\tilde {\boldsymbol u}_t$ in \[0,1\], we normalize the standard deviation map $\boldsymbol \sigma_t$ by subtracting its min value and dividing by absolute max-min value (the smaller $\boldsymbol \sigma_t$, the lower uncertainty). Note that the final sampling (Step 3 in Sec.3.1.2) is performed using the original $\boldsymbol \sigma_t$ without normalization.  
    
    We then sort the uncertainty scores only for unknown pixels based on $\boldsymbol m_{t-1}$. According to the pre-defined mask schedule, we figure out the pixels that will be newly added in this iteration. Based on the preliminary uncertainty map $\tilde {\boldsymbol u}_t$, by marking pixel locations that are still missing as 1 and the initially known pixel locations (referring to $\boldsymbol m_0$) as 0, while keeping the $\tilde {\boldsymbol u}_t$ values of inpainted pixels unchanged, we obtain the final uncertainty map $\boldsymbol u_t$.

Reviewer comment (Presentation):
The third paragraph in Section 3.1.1 is inconsistent with Figure 2. The input image x_t at time t should be x_{t-1} at time t-1, right?

Author response:
2. **The third paragraph in Section 3.1.1 is inconsistent with Figure 2. The input image $\boldsymbol x_t$ at time t should be $\boldsymbol  x_{t-1}$ at time $t-1$, right?**

    Thanks for pointing this out. We have fixed this typo. It should be $\boldsymbol x_{t-1}$.

Reviewer comment (Experiments):
Please discuss in detail the effect of the learnable function F in Eq. (6).

Author response:
3. **Please discuss in detail the effect of the learnable function $\mathcal F$ in Eq.6.**

    The learnable function $\mathcal F$ introduces the proposed uncertainty-guided attention mechanism. In traditional attention based solely on feature similarity, all pixels have an equal chance to exchange information. However, in image inpainting, where missing pixels are initialized with specified identical values, conventional attention mechanisms are not effective in conveying valuable information from visible regions to missing regions. This phenomenon often yields unsatisfied pixels, resulting in blurry content and undesirable artifacts.

    By incorporating the uncertainty map as an input, the function $\mathcal F$ discerns between valid and invalid pixels. This enables our network to prioritize attention on valid pixels when filling in the holes. Removal of this design (i.e., 'model F' in Table 1) results in a performance drop compared to our full 'model A'. Additionally, the structures are sharper and clearer in the inpainted results via our uncertainty-guided attention in Figure 4.

Reviewer comment (Experiments):
The authors state several times that the proposed method improve the efficiency. Please demonstrate in detail, e.g., the comparison of inference time.

Author response:
4. **The authors state several times that the proposed method improves efficiency. Please demonstrate in detail, e.g., the comparison of inference time.**

    We provide efficiency comparisons in Table F.5, where our model takes less time for inference. We also compare inference speed tested on the A100 GPU and inpainting quality in the table below where our model achieves favorable performance.

    | | | | | | |
    |---|:---:|:---:|:---:|:---:|:---:|
    |Method |**Ours**|Stable Diffusion|LDM|ControlNet|MAT|
    |FID$\downarrow$ | **1.68** | 2.11 | 2.76 | 5.55 | 2.90 |
    |Speed$\downarrow$ (512×512) |**0.25s**|3.6s|2.7s|3.1s|0.26s|
    | | | | | | |

Reviewer comment (Reproducibility):
What is the mask size in Table 4?

Author response:
5. **What is the mask size in Table 4?**

    The mask size is 512x512, consistent with the image sizes of Places2 and CelebA-HQ as indicated in Table 4. The small and large mask settings refer to different masking ratios. The mask statistics have been provided in Sec.B of Mat's [e] supplementary file. The small and large masks are established with an average masking ratio of approximately 20% and 50%, respectively.

Reviewer comment (Experiments):
In the experiments, please add the comparisons with more recent methods, e.g.,  
[a] Chu et al., Rethinking Fast Fourier Convolution in Image Inpainting, ICCV 2023.  
[b] Sargsyan et al., MI-GAN: A Simple Baseline for Image Inpainting on Mobile Devices, ICCV 2023.  
[c] Ko et al., Continuously Masked Transformer for Image Inpainting, ICCV 2023.

Author response:
6. **In the experiments, please add the comparisons with more recent methods, e.g., [a] Chu et al., Rethinking Fast Fourier Convolution in Image Inpainting, ICCV 2023. [b] Sargsyan et al., MI-GAN: A Simple Baseline for Image Inpainting on Mobile Devices, ICCV 2023. [c] Ko et al., Continuously Masked Transformer for Image Inpainting, ICCV 2023.**

    Thanks for the valuable comments. We have cited all the referred works. In practice, we find out that only MI-GAN [b] provides implementation code and models, so we compare our method to MI-GAN and ControlNet \[d\] (ICCV 2023 Best Paper Award) below. Note that ControlNet is trained on a more extensive LAION dataset rather than the Places2 dataset. The results obtained from our proposed PSM exhibit significant improvements across all metrics. We observe that ControlNet might produce undesired results due to its tendency to generate new objects that might not align harmoniously with the existing content. A detailed comparison is shown in Sec.I due to page limit.
    
    #### 512x512 Places2 under the small mask ratio setting
    | | | | | | |
    |---|---|---|---|---|---|
    | Method | FID$\downarrow$ | P-IDS$\uparrow$ | U-IDS$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ |
    | **PSM (ours)** | **0.72** | **30.95** | **43.91** | **0.084** | **25.51** |
    | MIGAN | 1.40 | 18.43 | 39.35 | 0.103 | 24.38 | 
    | ControlNet | 1.86 | 12.63 | 35.71 | 0.117 | 24.68 |
    | | | | | | |
    
    #### 512x512 Places2 under the large mask ratio setting
    | | | | | | |
    |---|---|---|---|---|---|
    |**PSM (ours)**| **1.68** | **25.33** | **39.30** | **0.161** | **20.89** |
    |MIGAN| 3.81|13.50|32.42|0.195|20.00|
    |ControlNet|5.55|6.60|25.65|0.219|19.73|
    | | | | | | |

    [a] Chu, Tianyi, et al. "Rethinking Fast Fourier Convolution in Image Inpainting." ICCV. 2023.     
    [b] Sargsyan, Andranik, et al. "MI-GAN: A Simple Baseline for Image Inpainting on Mobile Devices." ICCV. 2023.      
    [c] Ko, Keunsoo, and Chang-Su Kim. "Continuously Masked Transformer for Image Inpainting." ICCV. 2023.     
    [d] Zhang, Lvmin, Anyi Rao, and Maneesh Agrawala. "Adding conditional control to text-to-image diffusion models." ICCV. 2023.
    [e] Li, Wenbo, et al. "Mat: Mask-aware transformer for large hole image inpainting." CVPR. 2022.