Paper: MaGIC: Multi-modality Guided Image Completion (ICLR-2024)

Reviewer comment (Evaluation):
The fair comparison is not easy since most SOTA are not considering multiple resources in the same time. It'd be nice to share some insight into this, and share failure cases.

Author response:
We deeply appreciate your observation regarding the difficulty in constructing a fair comparison, given the absence of baselines with similar multi-conditioning features. In response, we have included an analysis of failure cases in Figure 8 (Appendix) in our updated version.

Reviewer comment (Novelty):
In the related work section (page 3 last paragraph), the authors give the impression that T2I-Adapter (and ControlNet) "fails to simultaneously use multi-modality as guidance". This is completely wrong. While T2I-Adapter does not explicitly train jointly for multiple modalities, it can combine multiple modalities (see section 4.3.2 in T2I-Adapter).

Author response:
**A1**: Thank you for your thorough review. We acknowledge that the T2I-Adapter represents an outstanding contribution to controllable image generation, exhibiting exponential performance in single-modality guidance tasks.
However, our claim that it "**fails** to simultaneously use multi-modality as guidance" remains valid. 
The T2I-Adapter, as outlined in its paper, implements multimodal guidance through a straightforward feature-level addition approach, while showcasing results with bimodal guidance. 
However, theoretically, this feature-level addition leads to a distribution mismatch with the pre-trained model, a phenomenon illustrated in Figure 7 of our updated paper. This implies that with an increased number of guiding modalities (more than two), such straightforward strategy can drastically perturb the distribution, resulting in distorted outputs. 
Empirically, our hypothesis is corroborated in Appendix Table 4 and Figure 9 of our updated version. In Table 4, our approach shows improvements of 230.30%, 71.40%, and 37.07% in image quality metrics P-IDS, U-IDS, and FID, respectively, compared to the multimodal guidance of T2I-Adapter. 
Qualitatively, Figure 9 demonstrates that while the T2I-Adapter aligns well with the shape and layout of multiple guiding modalities, it produces images lacking in realism. The loss of high-frequency details and the failure to restore content in unmasked regions are due to the simultaneous addition of features from four modalities during inference, which does not match the distribution of the T2I-Adapter's backbone U-Net.

Reviewer comment (Experiments):
The authors need to provide a solid justification for why a simple "feature-level addition" mentioned in Page-6 paragraph-2 is not good. T2I-Adapter (in broad terms) does exactly that. More detailed experiments and intuition comparing both T2I-Adapter and ControlNet for the multi-modality case are needed.

Author response:
**A2**: Thanks for your suggestion. As highlighted by Reviewer VxR9, the absence of multimodal guided image completion methods makes it challenging to conduct fair comparative experiments. ControlNet and T2I-adapter are specialist in various single modality image generations, and we implement a latent-level blending to enable them to perform image completion. This way ensures maximum fairness in comparing the best multimodal guided image completion methods currently available.
We have put the Table 4 (Appendix) for detailed quantitative experiments with T2I-Adapter and ControlNet in our new version. Additionally, in the Figure 9 (Appendix), we also provide qualitative comparison with T2I-Adapter under multi-modal conditioning. This supports our findings in Section 4.2, where T2I-adapter* and ControlNet* are observed to fall short in preserving spatial consistency.

Reviewer comment (Theory):
The argument given by the authors that "denoiser is trained solely on the distribution of $$\hat{F}_{c} = F_{enc} + F_{c}$$" makes even less sense when considering ControlNet with its zero-convolutions. In ControlNet, plausible generation is obtained from the starting iterations thanks to zero-convolutions, and with training, the conditioning branch becomes good. Hence, the distribution mismatch should not really be an issue to begin with.

Author response:
**A3**: Thanks for raising such an interesting question,  and we would like to provide a comprehensive answer addressing two crucial aspects. 

1. **The function of zero-convolution:** Our understanding of zero-convolution aligns with its purpose in ControlNet for stabilizing the initial training process. This technique avoids introducing "noise" to deep features generated by production-ready weights, as discussed in ControlNet. In both T2I-Adapter and ControlNet, the modality features are directly added to the features of the original Stable Diffusion, which is trained for conditional generation, and that is why we said the denoiser is trained on $\hat{F_c} = F_{enc} + F_c$. Consequently, we disagree with there exists any direct correlation between the zero-convolution and training distribution.
2. **The origin of distribution mismatch:** Our primary focus resolves around the process of combining separately trained condition modules (such as adapter in T2I-Adapter) to support multi-modality input in a training-free manner. Analogous to how modality features might act as noise during the initial training phase for a well-established Stable Diffusion, we posit that features produced by separately trained conditional modules could similarly be perceived as noise by each other. For example, suppose the trained networks supports segmentation ($\hat{F_{seg}} = F_{enc} + F_{seg}$) and depth $\hat{F_{dep}} = F_{enc} + F_{dep}$ modalities individually, the combined features $F_{seg}+F_{dep}$ could be noise inputs for both networks, leading to distribution mismatch. 
   We apologize for any confusion in our initial manuscript concerning this question and welcome further discussion to clarify these aspects.

Reviewer comment (Writing):
I think Eq. 4 should be $$ l (\hat{F}_{C}, F_{*}) = \frac{1}_{L} \sum_{l=0}^{L} \sum_{i=1}^{N} \delta_{c} || \hat{F}_{c}^{l} - F_{*}^{l} ||_{2}^{2} $$ (Note: subscript is C and not c?).

Author response:
**A4.1**: Thanks for your nice suggestion. We have revised the equation 4 as:

$\ell(\hat{F_\mathcal{C}}, F_*) = \frac{1}{L} \sum_{l=0}^L \sum_{c\in \mathcal{C}} \delta_{c} \| \hat{F_c^l} - F_{*}^l\|_2^2$

Reviewer comment (Writing):
There are typos, e.g., in the 2nd last paragraph just after Equation 2 ("Denoising").

Author response:
**A4.2**: Thanks for your detailed review. We have corrected the typo in our new version.

Reviewer comment (Novelty):
The MCU-Net is basically T2I-Adapter. There appears to be no reason to have a new name for it (which may only rebrand something and create more confusion). Given MCU-Net is the same as T2I-Adapter, the only merit of this paper is CMB.

Author response:
**A4.3**: Sorry for confusion. It's important to clarify that MCU-Net is distinct from the T2I-Adapter; we cannot simply use the T2I-Adapter as our backbone for single modality guided image completion. Our work involves gathering multi-modality datasets and training MCU-Net with masked images. This training enables our model to adeptly fill masked regions while ensuring harmony with the unmasked areas. This way significantly contributes to our method's superior performance over baseline models, as evidenced by our experimental results.

Reviewer comment (Experiments):
Since the only contribution of this paper is CMB, there should be a very detailed comparison with respect to T2I-Adapter, ControlNet, and more, particularly for multiple modalities.

Author response:
**A5**: Thanks. We have included both quantitative and qualitative comparisons with respect to T2I-adapter and ControlNet. Please refer to Table 4 and Figure 9 in our updated version.

Reviewer comment (Novelty):
Apart from just a few qualitative results and some incremental metrics improvement, the authors should explain why Converse Amplification (or simply a variant of classifier guidance) is a better approach than zero-convolutions with ControlNet.

Author response:
**A6**: Thank you. As **A3** clarified, zero-convolution differs significantly from Converse Amplification in both motivation and implementation. If there are any further questions or concerns, please feel free to discuss them.

Reviewer comment (Evaluation):
The paper should include some failure cases of CMB to give a better idea of where ControlNet lacks and where CMB lacks (while acknowledging that CMB can be coupled with ControlNet or T2I-Adapter).

Author response:
**A7**: Thank you for your thoughtful suggestion. We have included the failure cases in Figure 8 in our revised version.

Reviewer comment (Writing):
The paper is challenging to follow and contains multiple typos, which can impede understanding. Improved clarity in the presentation and thorough proofreading would enhance the paper's quality.

Author response:
**A1**: We are sorry for this and truly appreciate your feedback on improving our paper's clarity and addressing the typos. We have corrected the expression and typos in our new version.

Reviewer comment (Theory):
The update scheme presented in equation (5) appears inhomogeneous, as it involves gradient descent with respect to $z_t$ but updates $z'_{t-1}$. This choice could be a reasonable heuristic but is not discussed or justified, which leaves questions about its validity.

Author response:
**A2**: Our approach aligns with what classifier-free guidance (CFG) [1] does and it makes common sense in this field. The core idea is to manually design reasonable loss and impose its gradient on latent via backpropagation. This process allows for the adjustment of the latent space towards generating the desired output. Please refer to applications [2-4] of CFG if you need further clarification. If there are any further questions or confusion, we are open to discussing them.

`[1]` Classifier-Free Diffusion Guidance. Arxiv. 2022.

`[2]` Freedom: Training-free energy-guided conditional diffusion model. Arxiv. 2023.

`[3]` Training-free layout control with cross-attention guidance. Arxiv. 2023.

`[4]` Unite and Conquer: Plug & Play Multi-Modal Synthesis using Diffusion Models. CVPR. 2023.

Reviewer comment (Evaluation):
The paper only provides qualitative results without quantitative evaluation metrics when compared with recent baselines such as ControlNet and T2I-adapter. In particular, the performances with respect to ControlNet should be carefully assessed.

Author response:
**A3**: Thanks for your valuable suggestion. We have added the Table 4 (Appendix) for quantitative evaluation with ControlNet and T2I-Adapter in our updated manuscript. Additionally, in the Figure 9 (Appendix), we also provide qualitative comparison with T2I-Adapter under multi-modal conditioning.

Reviewer comment (Experiments):
A more in-depth exploration of the impact of the weights $\delta_c$ in equation (4) would offer valuable insights. The stability of these parameters is critical as their tuning could rapidly be cumbersome.

Author response:
**A5**: Thank you for your feedback. As suggested, we have conducted ablation study on $\delta_c$ and the results are shown in the table below:

| $\delta_c$ | FID$\downarrow$ | PickScore$\uparrow$/% |
| :----------- | :---------------- | :---------------------- |
| 1.0          | 37.65±0.22        | 49.57±0.17              |
| 0.1          | 37.75±0.25        | 50.98±0.37              |
| 0.5          | 37.27±0.26        | 50.96±0.15              |
| 2.0          | 37.93±0.18        | 49.18±0.16              |

In our initial manuscript, due to resource constraints and setting $\delta_c$ to 1 is trivial, we had not conducted this part of the experiment. However, the results suggest that adjusting the value within a practical range (0-1) yields improved results. This finding holds promise and offers valuable insights for future practical applications. Thanks again for the valuable suggestion.

Reviewer comment (Evaluation):
The results in Table 3.b appear to be inconsistent, with FID scores not following the expected pattern. This raises questions about the efficiency of the CMB method and its need for complex hyperparameter tuning. In particular, for a fixed P=30, FID(Q=1) > FID(Q=10) > FID(Q=5).

Author response:
**A6**: Thank you for your detailed review. When Q is sufficiently large, the latent will match the conditioning modality. However, in the case of multimodal conditioning, a large Q can lead to stiff competition among multiple modalities for the latent. This competition can naturally lead to worse performance, as seen in metrics like FID. To clarify, we've emphasized in the revision that Table (b) is evaluated under multimodal conditioning. 

And we acknowledge the reviewer's concern regarding hyperparamter tuning. But our work targets training-free scenarios with introducing merely two additional hyperparameter. It is not excessively burdensome compared to training a large, multimodal guided model.

Reviewer comment (Evaluation):
The authors did not provide CLIP score evaluation as well as reconstruction performances.

Author response:
**A7**: PickScore, a **CLIP-based** metric, aligns more closely with human judgments, which is why we chose it. FID, a **reconstruction** metric, together with U-IDS and P-IDS, evaluates image quality and aligns well with human perception, making it suitable for our purposes. PSNR and SSIM, being **pixel-wise** metrics, are not ideal for **editing** or large mask settings [1,2]. Nevertheless, based on your suggestion, we have included Table 5 in the Appendix for a quantitative evaluation with ControlNet and T2I-Adapter in terms of CLIP score, and additional reconstruction metrics PSNR, SSIM, and LPIPS.

`[1]` Large scale image completion via co-modulated generative adversarial networks. ICLR. 2021.

`[2]` Mat: Mask-aware transformer for large hole image inpainting. CVPR. 2022.

Please let us know if there are further questions. Thanks again!