Paper: Image Inpainting via Tractable Steering of Diffusion Models (ICLR-2024)

Reviewer comment (Reproducibility):
The readability of this paper is relatively low. We believe that the author should explain in the introduction and method sections whether the method proposed is training or non-training. If it is the former, the loss function and the parts of the network that need to be updated should be stated explicitly. If it is the latter, the pseudo-code of the algorithm should be given.

Author response:
Thank you for pointing out aspects of the paper that were not explained clearly in the main text. The PC used in the proposed inpainting algorithm needs to be first trained (unconditionally) on the datasets (e.g., CelebA-HQ, ImageNet, and LSUN-Bedroom) using the MLE objective: $\max \sum_{\mathbf{x} \in \mathcal{D}} \log p_{n} (\mathbf{x})$, where $p_{n}$ is the PC and $\mathcal{D}$ is the dataset.

Following prior art, a natural way to optimize PC parameters w.r.t. the MLE objective is by the Expectation-Maximization algorithm since PCs can be viewed as a latent variable model with a hierarchically nested latent space. The implementation of the EM algorithm is very similar to the backpropagation algorithm since the feedforward computation following Definition 1 can be treated as a computational graph. To update the parameters, we first run the standard backpropagation algorithm to compute the gradient of all parameters (w.r.t. the MLE objective, which is the output of the feedforward PC computation). Suppose the gradient of $\theta_{n,c}$ is $g_{n,c}$, for every sum node $n$, we update all its child parameters by 

$$\theta_{n,c} \leftarrow (1 - \alpha) \cdot \theta_{n,c} + \alpha \cdot \frac{g_{n,c} \cdot \theta_{n,c}}{\sum_{m \in \mathtt{in}(n)} g_{n,m} \cdot \theta_{n,m}},$$

where $\alpha$ is the step size. In addition to the EM update, we follow prior arts and use the LVD technique to initialize the parameters. The full details of the learning process of PCs are included in Appendix C of the revised paper. At the end of Section 4.1, we also added a paragraph to explain the PC learning pipeline and points to the Appendix for more details.

We are happy to make further clarifications if there are other unclear parts in the paper.

Reviewer comment (Reproducibility):
I cannot fully understand the details of the method proposed in this paper (e.g., how were the weights in the PC obtained). I would appreciate it if the authors could provide pseudocode to enhance the method's readability further.

Author response:
Thank you for pointing out aspects of the paper that were not explained clearly in the main text. The PC used in the proposed inpainting algorithm needs to be first trained (unconditionally) on the datasets (e.g., CelebA-HQ, ImageNet, and LSUN-Bedroom) using the MLE objective: $\max \sum_{\mathbf{x} \in \mathcal{D}} \log p_{n} (\mathbf{x})$, where $p_{n}$ is the PC and $\mathcal{D}$ is the dataset.

Following prior art, a natural way to optimize PC parameters w.r.t. the MLE objective is by the Expectation-Maximization algorithm since PCs can be viewed as a latent variable model with a hierarchically nested latent space. The implementation of the EM algorithm is very similar to the backpropagation algorithm since the feedforward computation following Definition 1 can be treated as a computational graph. To update the parameters, we first run the standard backpropagation algorithm to compute the gradient of all parameters (w.r.t. the MLE objective, which is the output of the feedforward PC computation). Suppose the gradient of $\theta_{n,c}$ is $g_{n,c}$, for every sum node $n$, we update all its child parameters by 

$$\theta_{n,c} \leftarrow (1 - \alpha) \cdot \theta_{n,c} + \alpha \cdot \frac{g_{n,c} \cdot \theta_{n,c}}{\sum_{m \in \mathtt{in}(n)} g_{n,m} \cdot \theta_{n,m}},$$

where $\alpha$ is the step size. In addition to the EM update, we follow prior arts and use the LVD technique to initialize the parameters. The full details of the learning process of PCs are included in Appendix C of the revised paper. At the end of Section 4.1, we also added a paragraph to explain the PC learning pipeline and points to the Appendix for more details.

We are happy to make further clarifications if there are other unclear parts in the paper.

Reviewer comment (Evaluation):
LPIPS alone may not be sufficient to assess the quality of generated images. We recommend that the authors report metrics such as FID, U-IDS, etc.

Author response:
We conduct a user study based on all three datasets and two masks (“Expand1” and the newly added “Wide” masks adopted from [1,2]). Detailed settings and results are added in the updated paper (Appendix E.1). We copy the result table from the paper:

| &nbsp;&nbsp;&nbsp;&nbsp; Dataset & Mask     &nbsp;&nbsp; &nbsp;  |  &nbsp;  Tiramisu &nbsp; | CoPaint | RePaint | DPS |

| CelebAHQ & Expand1  |  Reference    |   &nbsp;&nbsp;&nbsp;&nbsp;  22    &nbsp;&nbsp;&nbsp;&nbsp;  |    &nbsp;&nbsp;&nbsp;&nbsp; 34   &nbsp;&nbsp;&nbsp;&nbsp;  |  &nbsp;14 &nbsp; |

| CelebAHQ & Wide  &nbsp; &nbsp; &nbsp; |  Reference    |   &nbsp;&nbsp;&nbsp;&nbsp;  26    &nbsp;&nbsp;&nbsp;&nbsp;  |    &nbsp;&nbsp;&nbsp;&nbsp; 30   &nbsp;&nbsp;&nbsp;&nbsp;  |  &nbsp; 22 &nbsp; |

| ImageNet & Expand1  |  Reference    |   &nbsp;&nbsp;&nbsp;&nbsp;  32    &nbsp;&nbsp;&nbsp;&nbsp;  |    &nbsp;&nbsp;&nbsp;&nbsp; 20   &nbsp;&nbsp;&nbsp;&nbsp;  |  &nbsp; 24 &nbsp; |

| ImageNet & Wide &nbsp;&nbsp; &nbsp;&nbsp;    |  Reference    |   &nbsp;&nbsp;&nbsp;&nbsp;  14    &nbsp;&nbsp;&nbsp;&nbsp;  |    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 6   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;  |  &nbsp; 12 &nbsp; |

| LSUN & Expand1&nbsp; &nbsp;&nbsp; &nbsp; &nbsp;    |  Reference    |   &nbsp;&nbsp;&nbsp;&nbsp;  18    &nbsp;&nbsp;&nbsp;&nbsp;  |    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 2   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;  |  &nbsp; 8 &nbsp; |

| LSUN & Wide &nbsp;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;   |  Reference    |   &nbsp;&nbsp;&nbsp;&nbsp; &nbsp; 4    &nbsp;&nbsp;&nbsp;&nbsp; &nbsp; |    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 6   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;  |  &nbsp; -6 &nbsp; |

We use the vote difference (%) metric, which is the percentage of votes to Tiramisu subtracted by that of the baseline. A positive vote difference value means images generated by Tiramisu are preferred compared to the baselines, while a negative value suggests that the baseline is better than Tiramisu.

We compare against the three strongest baselines (CoPaint, RePaint, and DPS) as their average LPIPS scores are not significantly worse than Tiramisu. As shown in the table, in all but one task the vote difference score is positive (and often quite large), which indicates the superiority of Tiramisu compared to the baselines.

We also tried the metrics suggested by the reviewer. However, FID is not specific to image pairs and is less informative for image inpainting tasks; when using the official U-IDS metric, we found that in most cases we got a U-IDS close to 0 and the results are not very informative.

Reviewer comment (Evaluation):
The computing cost is related to the size of the hole as well as the network architecture. As authors have mentioned, it also related to resolution. So instead of claiming 10% additional computation overhead, a detailed analysis is more helpful.

Author response:
Thank you for the suggestion. In the updated paper (Section 6.2, the “computational efficiency” paragraph), we provide an ablation study over the % of denoising steps using PCs, and report the LPIPS score and runtime-per-image for each case. We adopt the “CelebA-HQ + the Expand1 mask” task. As shown in Figure 4 of the updated paper (for convenience, we copied the results in the figure to the table below) as we engage the PC in more denoising steps, the LPIPS score first decreases and then increases. Therefore, incorporating PCs in a moderate amount of steps gives the best performance (around 20% in this case). In terms of computational efficiency, the additional computation cost is around 10% when we engage the PC for the first 20% of the denoising steps and around 25% when the PC is used in the first 50% of the denoising steps.

One explanation for the diminishing performance gain when using PCs in too many denoising steps is that the later stages of denoising are mainly responsible for refining the details in the image. And the guidance of PCs could be less effective in this aspect. This could be the consequence of using latent space PC (latent space generated by VQ-GAN). In the future, we will explore possibilities to train good PCs directly in the pixel space. This could lead to better performance in terms of refining details in generated images.

| % denoising steps using PC  |   &nbsp;&nbsp;&nbsp;  5    &nbsp;&nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;   10    &nbsp;&nbsp;&nbsp;|&nbsp; &nbsp;&nbsp;   20   &nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;    30   &nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;   40    &nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;  50 &nbsp;&nbsp; |

| &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; LPIPS &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; |  0.470 | 0.459 | &nbsp; 0.454  | 0.456 | 0.464 &nbsp; | 0.476 |

| &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Runtime &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; |  &nbsp; 109 &nbsp;  | &nbsp; 111 &nbsp;  |  &nbsp; 115 &nbsp;  |  &nbsp; 118 &nbsp; |  &nbsp;121 &nbsp;  | &nbsp; 125 &nbsp; |

Reviewer comment (Evaluation):
Number-wise the improvement on LPIPS value compared to CoPaint, or even RePaint is minor. How about other metrics? Or user studies?

Author response:
We conduct a user study based on all three datasets and two masks (“Expand1” and the newly added “Wide” masks adopted from [1,2]). Detailed settings and results are added in the updated paper (Appendix E.1). We copy the result table from the paper:

| &nbsp;&nbsp;&nbsp;&nbsp; Dataset & Mask     &nbsp;&nbsp; &nbsp;  |  &nbsp;  Tiramisu &nbsp; | CoPaint | RePaint | DPS |

| CelebAHQ & Expand1  |  Reference    |   &nbsp;&nbsp;&nbsp;&nbsp;  22    &nbsp;&nbsp;&nbsp;&nbsp;  |    &nbsp;&nbsp;&nbsp;&nbsp; 34   &nbsp;&nbsp;&nbsp;&nbsp;  |  &nbsp;14 &nbsp; |

| CelebAHQ & Wide  &nbsp; &nbsp; &nbsp; |  Reference    |   &nbsp;&nbsp;&nbsp;&nbsp;  26    &nbsp;&nbsp;&nbsp;&nbsp;  |    &nbsp;&nbsp;&nbsp;&nbsp; 30   &nbsp;&nbsp;&nbsp;&nbsp;  |  &nbsp; 22 &nbsp; |

| ImageNet & Expand1  |  Reference    |   &nbsp;&nbsp;&nbsp;&nbsp;  32    &nbsp;&nbsp;&nbsp;&nbsp;  |    &nbsp;&nbsp;&nbsp;&nbsp; 20   &nbsp;&nbsp;&nbsp;&nbsp;  |  &nbsp; 24 &nbsp; |

| ImageNet & Wide &nbsp;&nbsp; &nbsp;&nbsp;    |  Reference    |   &nbsp;&nbsp;&nbsp;&nbsp;  14    &nbsp;&nbsp;&nbsp;&nbsp;  |    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 6   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;  |  &nbsp; 12 &nbsp; |

| LSUN & Expand1&nbsp; &nbsp;&nbsp; &nbsp; &nbsp;    |  Reference    |   &nbsp;&nbsp;&nbsp;&nbsp;  18    &nbsp;&nbsp;&nbsp;&nbsp;  |    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 2   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;  |  &nbsp; 8 &nbsp; |

| LSUN & Wide &nbsp;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;   |  Reference    |   &nbsp;&nbsp;&nbsp;&nbsp; &nbsp; 4    &nbsp;&nbsp;&nbsp;&nbsp; &nbsp; |    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 6   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;  |  &nbsp; -6 &nbsp; |

We use the vote difference (%) metric, which is the percentage of votes to Tiramisu subtracted by that of the baseline. A positive vote difference value means images generated by Tiramisu are preferred compared to the baselines, while a negative value suggests that the baseline is better than Tiramisu.

We compare against the three strongest baselines (CoPaint, RePaint, and DPS) as their average LPIPS scores are not significantly worse than Tiramisu. As shown in the table, in all but one task the vote difference score is positive (and often quite large), which indicates the superiority of Tiramisu compared to the baselines.

Reviewer comment (Presentation):
Section 4.2 reads slightly disconnected from Section 4.1, especially the introduction of the equation 7.

Author response:
Thank you for spotting this. Section 4.2 continues to introduce how to compute Equation (4) (from the TPM), which is a key step in the general algorithm described in Section 3. In the revised manuscript, we have added clarification sentences at the beginning of Section 4.2 to remind readers why we need to compute $p_{TPM} (\tilde{\mathbf{x}}_0 | \mathbf{x}_t, \mathbf{x}_0^k)$.

Reviewer comment (Reproducibility):
Will code be released upon publication of this work?

Author response:
Yes, we will release the training/inference code as well as the pretrained PCs if this work gets accepted.

Reviewer comment (Experiments):
The TPMs seem to be a general design, while this work constrain the application to image inpainting only, I am not sure about the intuition of this specific application. How about the potential of this method for general conditional generation?

Author response:
> How about the potential of this method for general conditional generation?

There are various conditional/constrained image generation tasks that could be done by (variants of) the proposed method. As a first attempt at using TPMs (PCs) for conditional image generation tasks, this paper uses image inpainting to demonstrate the effectiveness and potential of the proposed method. In the following, we list several classes of conditional generation tasks that could be accomplished by the proposed method or some (non-trivial) extension of our method.

- Other types of constraints belong to the independent soft evidence constraint family (formally defined in Section 4.2). For example, the constraint of image super-resolution and deblurring could be written in the form of such constraints and thus can be tackled by the proposed method.

- Other inverse problems that can be formulated into “simple” constraints could potentially be tackled by a generalized version of the proposed method thanks to the ability of PCs to efficiently *condition on certain logical constraints*. For example, in image coloring, the constraint would be that for every pixel, its R, G, B value have a fixed weighted sum (specified by the input gray-scale image).

- Controlled generation tasks without formally described conditions. For example, the semantic fusion task described in Section 6.3 aims to “fuse” the semantics of several reference images while maintaining the “naturalness” of the resultant image.

Reviewer comment (Experiments):
In section 6.2, it says "we only need to incorporate guidance from the TPM in the early denoising stages to control the global semantics of the image; fine-grained details can be later refined by the diffusion model. As a result, TPM is only required in the first ∼20% denoising steps". Here, more experimental analysis of the TPM steps are expected, including the effect on generation quality, semantic coherence, and computational efficiency.

Author response:
> Here, more experimental analysis of the TPM steps are expected, including the effect on generation quality, semantic coherence, and computational efficiency.

Thank you for the suggestion. In the updated paper (Section 6.2, the “computational efficiency” paragraph), we provide an ablation study over the % of denoising steps using PCs, and report the LPIPS score and runtime-per-image for each case. We adopt the “CelebA-HQ + the Expand1 mask” task. As shown in Figure 4 of the updated paper (for convenience, we copied the results in the figure to the table below) as we engage the PC in more denoising steps, the LPIPS score first decreases and then increases. Therefore, incorporating PCs in a moderate amount of steps gives the best performance (around 20% in this case). In terms of computational efficiency, the additional computation cost is around 10% when we engage the PC for the first 20% denoising steps and around 25% when the PC is used in the first 50% denoising steps.

One explanation for the diminishing performance gain when using PCs in too many denoising steps is that the later stages of denoising are mainly responsible for refining the details in the image. And the guidance of PCs could be less effective in this aspect. This could be the consequence of using latent space PC (latent space generated by VQ-GAN). In the future, we will explore possibilities to train good PCs directly in the pixel space. This could lead to better performance in terms of refining details in generated images.

| % denoising steps using PC  |   &nbsp;&nbsp;&nbsp;  5    &nbsp;&nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;   10    &nbsp;&nbsp;&nbsp;|&nbsp; &nbsp;&nbsp;   20   &nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;    30   &nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;   40    &nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;  50 &nbsp;&nbsp; |

| &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; LPIPS &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; |  0.470 | 0.459 | &nbsp; 0.454  | 0.456 | 0.464 &nbsp; | 0.476 |

| &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Runtime &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; |  &nbsp; 109 &nbsp;  | &nbsp; 111 &nbsp;  |  &nbsp; 115 &nbsp;  |  &nbsp; 118 &nbsp; |  &nbsp;121 &nbsp;  | &nbsp; 125 &nbsp; |

Reviewer comment (Evaluation):
Limitations and Failure Cases: A discussion of the method's limitations and failure cases are not addressed in this research, which would be beneficial for the application of this work.

Author response:
> Limitations and Failure Cases: A discussion of the method's limitations and failure cases are not addressed in this research, which would be beneficial for the application of this work.

As described in the response to the above question, engaging the guidance of PCs in too many denoising steps could lead to worse performance. This suggests that the current method may not be good at refining details. Therefore, Tiramisu could be less effective when dealing with small-hole inpainting.

Another potential weakness of the proposed method is the expressiveness of PCs. While we are able to train good PCs on datasets such as ImageNet 256*256, it is not sure whether this could be generalized to higher-dimensional images.

Reviewer comment (Experiments):
The comparison only contains six different masks. In real application, the cases where the images are masked by some texts or patterns are also very common. It would be ideal to see more comparisons of such masks in arbitrary shapes.

Author response:
Thank you for the suggestion. We additionally adopted a set of randomly generated “wide” masks following [1] and [2]. The masks are generated by uniformly sampling from polygonal chains dilated by a high random width and rectangles of arbitrary aspect ratios. We downloaded a set of 100 such masks provided by [1] and ran all algorithms with the same set of hyperparameters used in other tasks (no hyperparameter tuning specific to these masks is done). We have added the full results in the updated paper (Table 1) and below is a summary of the new results:

|    Dataset & Mask &nbsp;&nbsp;&nbsp;  | Tiramisu |  CoPaint  |  RePaint  |  DDNM  |  DDRM  |  &nbsp; DPS   &nbsp;  | Resampling |

| CelebAHQ & Wide  |  &nbsp; 0.069 &nbsp; &nbsp;  |   0.072   &nbsp;&nbsp; |   &nbsp; 0.075  &nbsp;  |  &nbsp; 0.112  &nbsp; |   0.132 &nbsp;  |  0.078    |   &nbsp;  0.128     &nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; |

| ImageNet & Wide  |  &nbsp;  0.125 &nbsp; &nbsp;  |    0.128  &nbsp;&nbsp; |   &nbsp;  0.127 &nbsp;  |  &nbsp;   0.198 &nbsp; |  0.197  &nbsp;  |  0.132 |   &nbsp;  0.196 &nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; |

| LSUN & Wide &nbsp;  &nbsp;  &nbsp;  &nbsp;    |  &nbsp;    0.116  &nbsp; &nbsp;  |     0.115 &nbsp;&nbsp; |   &nbsp;   0.124 &nbsp;  |  &nbsp;   0.135 &nbsp; |  0.204  &nbsp;  |  0.108  |   &nbsp;   0.202 &nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; |

Tiramisu achieves the best LPIPS score on CelebA-HQ and ImageNet, and is behind DPS in LSUN-Bedroom. Overall, Tiramisu achieves the best LPIPS score on 14 out of 21 tasks. Since the LPIPS scores alone may not fully justify the performance of Tiramisu, please also refer to the response to the next question (about user study).

Reviewer comment (Evaluation):
The table only contains LPIPS for quantitative measurement, however, as image inpainting is an ill-posed problem, a user study would be beneficial in this case as previous works such as [1][2] perform.

Author response:
We conduct a user study based on all three datasets and two masks (“Expand1” and the newly added “Wide” masks adopted from [1,2]). Detailed settings and results are added in the updated paper (Appendix E.1). We copy the result table from the paper:

| &nbsp;&nbsp;&nbsp;&nbsp; Dataset & Mask     &nbsp;&nbsp; &nbsp;  |  &nbsp;  Tiramisu &nbsp; | CoPaint | RePaint | DPS |

| CelebAHQ & Expand1  |  Reference    |   &nbsp;&nbsp;&nbsp;&nbsp;  22    &nbsp;&nbsp;&nbsp;&nbsp;  |    &nbsp;&nbsp;&nbsp;&nbsp; 34   &nbsp;&nbsp;&nbsp;&nbsp;  |  &nbsp;14 &nbsp; |

| CelebAHQ & Wide  &nbsp; &nbsp; &nbsp; |  Reference    |   &nbsp;&nbsp;&nbsp;&nbsp;  26    &nbsp;&nbsp;&nbsp;&nbsp;  |    &nbsp;&nbsp;&nbsp;&nbsp; 30   &nbsp;&nbsp;&nbsp;&nbsp;  |  &nbsp; 22 &nbsp; |

| ImageNet & Expand1  |  Reference    |   &nbsp;&nbsp;&nbsp;&nbsp;  32    &nbsp;&nbsp;&nbsp;&nbsp;  |    &nbsp;&nbsp;&nbsp;&nbsp; 20   &nbsp;&nbsp;&nbsp;&nbsp;  |  &nbsp; 24 &nbsp; |

| ImageNet & Wide &nbsp;&nbsp; &nbsp;&nbsp;    |  Reference    |   &nbsp;&nbsp;&nbsp;&nbsp;  14    &nbsp;&nbsp;&nbsp;&nbsp;  |    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 6   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;  |  &nbsp; 12 &nbsp; |

| LSUN & Expand1&nbsp; &nbsp;&nbsp; &nbsp; &nbsp;    |  Reference    |   &nbsp;&nbsp;&nbsp;&nbsp;  18    &nbsp;&nbsp;&nbsp;&nbsp;  |    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 2   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;  |  &nbsp; 8 &nbsp; |

| LSUN & Wide &nbsp;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;   |  Reference    |   &nbsp;&nbsp;&nbsp;&nbsp; &nbsp; 4    &nbsp;&nbsp;&nbsp;&nbsp; &nbsp; |    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 6   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;  |  &nbsp; -6 &nbsp; |


We use the vote difference (%) metric, which is the percentage of votes to Tiramisu subtracted by that of the baseline. A positive vote difference value means images generated by Tiramisu are preferred compared to the baselines, while a negative value suggests that the baseline is better than Tiramisu.

We compare against the three strongest baselines (CoPaint, RePaint, and DPS) as their average LPIPS scores are not significantly worse than Tiramisu. As shown in the table, in all but one task the vote difference score is positive (and often quite large), which indicates the superiority of Tiramisu compared to the baselines.

Reviewer comment (Experiments):
From Figure 1 and Table 3, there are some hyper-parameters specifically tuned for different datasets especially $t_{cut}$ which has also been applied in many previous works, could the authors provide an ablation study over the selection of the hyper-parameters?

Author response:
We provide an ablation study over the value of t_cut in the updated paper (Section 6.2, the “computational efficiency” paragraph). We adopt the “CelebA-HQ + the Expand1 mask” task. All other hyperparameters are kept the same and we change t_cut (we used “% denoising steps using PC”, which has a one-to-one correspondence with t_cut). As shown in Figure 4 of the updated paper (for convenience, we copied the results in the figure to the table below), as we engage the PC in more denoising steps, the LPIPS score first decreases and then increases. Therefore, incorporating PCs in a moderate amount of steps gives the best performance (around 20% in this case).

One explanation for the diminishing performance gain when using PCs in too many denoising steps is that the later stages of denoising are mainly responsible for refining the details in the image. And the guidance of PCs could be less effective in this aspect.

| % denoising steps using PC  |   &nbsp;&nbsp;&nbsp;  5    &nbsp;&nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;   10    &nbsp;&nbsp;&nbsp;|&nbsp; &nbsp;&nbsp;   20   &nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;    30   &nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;   40    &nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;  50 &nbsp;&nbsp; |

| &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; LPIPS &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; |  0.470 | 0.459 | &nbsp; 0.454  | 0.456 | 0.464 &nbsp; | 0.476 |

| &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Runtime &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; |  &nbsp; 109 &nbsp;  | &nbsp; 111 &nbsp;  |  &nbsp; 115 &nbsp;  |  &nbsp; 118 &nbsp; |  &nbsp;121 &nbsp;  | &nbsp; 125 &nbsp; |