Paper: Completing Visual Objects via Bridging Generation and Segmentation (ICLR-2024)

Reviewer comment (Theory):
It's not rigorous to name the progressive completion process as 'mask denoising'. The object's mask generated from SAM is taken as a condition to diffusion instead of the denoising uint as $x_t$.

Author response:
**2. It's not rigorous to name the progressive completion process as 'mask denoising'. The object's mask generated from SAM is taken as a condition to diffusion instead of the denoising unit as $x_t$.**

Thanks for the comments. We agree that the mask denoising is very different from the image denoising diffusion while we hope the word "mask denoising" can help the reader understand our iterative mask completion process. Given a partially visible object $I_p$ and its corresponding partial mask $M_p$, the conventional object completion task aims to find a generative model $\mathcal{G}$ such that $I_c\leftarrow\mathcal{G}(I_p)$, where $I_c$ is the complete object. Here, we additionally add the partial mask $M_p$ to the condition  $I_c\leftarrow\mathcal{G}(I_p, M_p)$, where $M_p$ can be assumed as an addition of the complete mask and a noise $M_p=M_c+\Delta$. By introducing a segmentation model $\mathcal{S}$, we can find a mask denoiser $\mathcal{S}\circ\mathcal{G}$ from the mask completion process: $M_c\leftarrow\mathcal{S}\circ\mathcal{G}(I_p, M_c+\Delta)$. If we consider the occlusion $\Delta$ as a noise, $\mathcal{S}\circ\mathcal{G}$ can be assumed as conducting mask denoising for each IMD step. We will further clarify the usage of "mask denoising" in the revision.

Reviewer comment (Reproducibility):
It is not clear that whether the other comparison method are retrained in the evaluation data. If yes, the training details and the incomplete mask's interaction in their networks should be claimed, otherwise the authors should explain how to obtain these completion results with the off-the-shelf generation models.

Author response:
**3. It is not clear whether the other comparison methods are retrained in the evaluation data. If yes, the training details and the incomplete mask's interaction in their networks should be claimed, otherwise the authors should explain how to obtain these completion results with the off-the-shelf generation models.**

We would like to clarify that the baseline methods are not retrained in the evaluation dataset. We consider the baselines can be directly utilized due to (1) the object completion can be treated as a subtask of the in/outpainting objectives that the baseline methods are trained on (2) the object categories in the evaluation set are common which should be within the large-scale training data of the baselines. In addition, to evaluate on a more general object setting, our model is also not trained on the evaluation set (DYCE) but on OpenImage (Line 231). In this way, we consider our comparison can fairly reflect the effectiveness of the proposed method.

Reviewer comment (Evaluation):
It would be better if the authors can give some analysis on the sampling speed and more qualitative results of the proposed method.

Author response:
**4. It would be better if the authors can give some analysis on the sampling speed and more qualitative results of the proposed method.**

|Reduce SD step|Generation stage|Segmentation Stage|Total|FID|
| :----:| :----: | :----: | :----: | :----: |
|| 14.3s | 1.2s |15.5s |16.9|
|$\checkmark$| 8.6s | 1.2s|9.8s| 17.4|

Table D: Inference time for each stage of IMD on a single V100 GPU. 

Thanks for your suggestion. We provide the sampling time of each component as shown in Table D. We notice that the most time-consuming part is the diffusion process. To improve the inference speed, we notice that decreasing the diffusion step number in the first several IMD steps will not severely degrade the performance. By incorporating this idea into MaskComp, the average running time was reduced to 2/3 original time. We also provide more qualitative results and results in a potential application (layered image generation) as shown in Figure A ([github.com/iclr23anonymous739/more_vis.pdf](https://github.com/iclr23anonymous739/rebuttal/blob/main/more_vis.pdf)) and Figure B ([github.com/iclr23anonymous739/application.pdf](https://github.com/iclr23anonymous739/rebuttal/blob/main/application.pdf)). We believe the idea of using segmentation to boost generation and the strong capability of MaskComp can be of interest to the community. We will incorporate the suggested results in the revision.

Reviewer comment (Theory):
The justification for the entire iterative procedure is lacking. While it is ideal to achieve improvements as shown in Figure 5, there is no guarantee that such improvement can be realized in a realistic setting. In particular, the mask-denoising controlnet is trained in a local manner, which may generate a worse image, and the segmentation stage is largely dependent on the segmentation model S, which may produce noisy segmentation output. Therefore, it is unclear whether this design would work in general cases.

Author response:
**1. The justification for the entire iterative procedure is lacking. It is possible to have noisy output (worse image/inaccurate mask) in both the generation and segmentation stages.**

We would like to provide an analysis of the noise-tolerant capability of the proposed MaskComp. We agree that worse images and inaccurate masks can appear in the generation and segmentation stages respectively. Since all the generation processes are conditioned on the original partial image (as shown in Figure 1), both worse images and inaccurate masks will just influence the conditioned mask for the next iteration. We have two mechanisms to mitigate the influence of the noisy conditioned masks: 

- *Error Suppression through Mask Voting*: We vote among multiple masks to form the input of the next iteration. In this way, if only a few masks are degraded, the impact of them can be recovered by the voting operation. 

- *Error Tolerance in IMD Iteration*: We train the mask-denoising ControlNet to handle a wide range of occluded masks. Consequently, if the conditioned mask undergoes minimal improvement or degradation due to the noises in a given iteration, it can still be improved in the subsequent iteration. While this may slightly extend the convergence time, it is not anticipated to have a significant impact on the ultimate image quality.

In this way, we consider the IMD process can work in a robust manner. We will show a quantitative analysis in the later response to question 7.

Reviewer comment (Theory):
The proposed method lacks mode diversity for object completion. The segmentation stage uses an averaging operator, which seems problematic since it would lead to mode average and is unable to capture potential different modes in object completion. This is important to generate different candidates for image editing since there are multiple possibilities for the occluded regions.

Author response:
**2. The proposed method lacks mode diversity for object completion due to the averaging operator in the IMD process.**

We agree that MaskComp lacks shape diversity due to the averaging operator. However, we consider that object completion tasks typically only prioritize the estimation closest to the ground truth object. For example, in the closely related amodal segmentation task, the evaluation is conducted by calculating the IoU between the predicted mask and the GT mask. We consider retaining only the most plausible estimation to be reasonable since the rough shape of objects can be deduced by the visible parts in most cases.

In addition, we consider the lack of shape diversity will not hinder the usage of MaskComp in practice. We discuss the potential application of MaskComp in Figure 10. Since the current image editing models struggle to modify partial images, MaskComp can serve as the first step to complete objects and then feed to an image editing model to further process the image. We show another cool application, layered image generation, of MaskComp as shown in Figure A ([github.com/iclr23anonymous739/application.pdf](https://github.com/iclr23anonymous739/rebuttal/blob/main/application.pdf)).

Reviewer comment (Reproducibility):
In the generation stage, how does the interpolation is implemented? How does the method generate the interpolated masks and its time embedding?

Author response:
**3. In the generation stage, how does the interpolation is implemented? How does the method generate the interpolated masks and its time embedding?**

We designed several occlusion strategies to create diverse occlusion (the details can be found in Line 417-424). During training, we conduct the random occlusion process twice for each complete mask $M_c$. The partial mask $M_p$ is achieved by considering the occluded areas in both of the occlusion processes. And the interpolated mask $M$ is generated by using one of the occlusions. The time embedding used for the gating operation is shared with the time embedding for encoding the diffusion step in the stable diffusion.

Reviewer comment (Writing):
What are the notations M, E in the paragraph of "Diffusion model"? The details of the adopted diffusion model are missing in the main text.

Author response:
**4. What are the notations M, E in the paragraph of "Diffusion model"? The details of the adopted diffusion model are missing in the main text.**

The $M$ denotes the interpolated mask (Line 146) with an occlusion rate between the complete mask $M_c$ and partial mask $M_p$ and $E$ denotes the text prompt of the object (please note that the text prompt is necessary to fine-tune stable diffusion while optional during inference). We leverage frozen stable diffusion (Line 242) as our underlining diffusion model. We further clarified the notations in the revised version.

Reviewer comment (Evaluation):
It is unclear how the baselines and this method are compared. It is worth noting that the proposed method only generates the foreground object while the other methods also produce the background. Are the background removed before comparison, for the FID computation and user study? It seems unfair if the background is treated differently.

Author response:
**5. It is unclear how the baselines and this method are compared. Are the background removed before comparison, for the FID computation and user study?**

Yes, we aim to ignore the background during all the evaluations. For the quantitative comparison, the computing of FID only considers the ground-truth object area to eliminate the influence of the generated background (Caption of Table 1). For the user study, we give detailed instructions to participants to focus only on the foreground object region and ignore the difference presented in the background (Line 457).

Reviewer comment (Experiments):
A reasonable baseline is to combine amodal segmentation with condition control image generation, which is missing in the comparison.

Author response:
**6. Comparison with a combination of amodal segmentation and condition image generation as a baseline.**

|Method|FID|
| ----| :----: | 
|AISFormer+ControlNet| 29.4 |
|MaskComp| 16.9 |

Table A: Performance comparison with amodal segmentation baseline.

Thanks for the suggestion. We conducted an additional experiment with the masks from the SOTA open-sourced amodal segmentation method ([A] AISFormer) and leveraged ControlNet to generate images based on the amodal masks. We report the performance as shown in Table A. We notice that our method shows obvious superior performance compared to the baseline setting.

[A] AISFormer: Amodal Instance Segmentation with Transformer

Reviewer comment (Experiments):
It is unclear how the method is sensitive to the segmentation quality. What if there are different degrees of segmentation error?

Author response:
**7. What if there are different degrees of segmentation error?**

|Noise degree|Iter 1|Iter 3|Iter 5|Iter 7|Iter 9|
| ----| :----: | :----: | :----: | :----: | :----: | 
|15\% area| 28.4 | 22.7| 18.9 | 17.2 |16.5|
|10\% area| 26.4 | 21.4 | 18.1 | 17.0| 16.4|
|5\% area| 24.9 | 19.6 | 17.0 | 16.2 | 16.0|
|No noise| 24.7 | 19.4 | 16.9 | 16.1 | 15.9|

Table B: Ablation with noisy conditioned mask. We leverage FID to evaluate the performance. The error is added based on the size of the foreground object.

We conducted an experiment to manually add random errors to the masks. As shown in Table B, we ablate on the iteration number and the degree of segmentation error. We notice that the segmentation error will just increase the converge iteration number while the final performance will not be severely influenced. In addition, since MaskComp predicts a black background, it is easy for segmentation models to segment the foreground objects. Therefore, a large error in segmentation is not expected.

Reviewer comment (Experiments):
It would be more convincing if the method is evaluated on the cases with diverse modes of object shape in the occluded regions.

Author response:
**8. It would be more convincing if the method is evaluated on the cases with diverse modes of object shape in the occluded regions.**

|Occ.|20\%|40\%|60\%|80\%|
| ----| :----: | :----: | :----: | :----: | 
|FID| 13.4 | 15.7 | 17.2 | 29.9 |

Table 2 (b): Performance with different occlusion rates.

|Occ.|Rectangle|Oval|Object|
| ----| :----: | :----: | :----: | 
|FID| 15.3 | 15.1| 16.9 |

Table C: Performance with different occlusion types. Object denotes the occlusion is created with a random object shape.

We reported the performance with different occlusion rates as shown in Table 2 (b). We additionally report the performance with different occlusion types as shown in Table C. Since the object occlusion has a more complex boundary, the results with rectangle and oval occlusions show a lower FID than that with more complex object occlusions.

Reviewer comment (Novelty):
The technical contributions of the proposed method could be further improved. For now, the proposed method is mainly a mask-guided stable diffusion model with an off-the-shelf segmentation model to produce the mask condition. Using SAM to generate masks is straightforward and the mask voting process gives no surprises.

Author response:
**1 & 2. Using SAM to generate masks is straightforward and the mask voting process gives no surprises. If generating a mask is the key to generating high-quality images, why not directly use an encoder-decoder model like U-Net or an SD to predict the target complete mask in one step?**

|Method|SD|UNet |IMD (Ours)|
| ----| :----: |:----: |:----: |
|IoU| 82.4 | 75.4 |88.5 | 

Table A: IoU of predicted masks on AHP dataset.

In this paper, we investigate the relation between object generation and segmentation. We found that even if the segmentation model is not trained, the segmentation can still benefit the generation. It is a good idea to conduct experiments to investigate other segmentation models.
We report the results of using a ControlNet (frozen SD) and UNet with the partial object as the condition to predict the complete mask in Table A. We notice that the complete mask predicted from the IMD process shows obviously higher IoU to the ground truth which indicates the IMD process is an effective mask prediction approach.

In addition, we would like to clarify that the mask prediction is not simply conducted by the SAM but by the entire IMD process with both generation and segmentation stages. We treat the segmentation as the shape property of the generated object. Therefore, with an object generation model, we can directly obtain the shape using SAM without training another mask-specific model.

We also want to further clarify the usage of the voting operation. Given a partial object $I_p$, the generation model samples from a distribution that contains both realistic and unrealistic images. As shown in Figure A ([github.com/iclr23anonymous739/dist.pdf](https://github.com/iclr23anonymous739/rebuttal/blob/main/dist.pdf)), since the image distribution is complex, the expectation $E[I_c]$ cannot represent a realistic image. However, when we consider the shape of the generated images, we are excited to find that the expectation $E[\mathcal{S}(I_c)]$ leads to a more realistic shape. We consider this observation interesting as, for most of the other generation tasks (non-conditioned), averaging the object shape will just yield an unrealistic random shape. This observation (Figure 4) serves as one of our core observations to build the IMD process. Here, SAM serves as the tool to extract the object shape and voting is a way to binarize the expectation of object shapes to a binary mask.

Reviewer comment (Experiments):
If generating a mask is the key to generating high-quality images, why not directly use an encoder-decoder model like U-Net or an SD to predict the target complete mask in one step? As generating the mask is relatively a simple task, I believe a U-Net might be enough to obtain an accurate object mask (this can be regarded as an outpainting task for binary images). It would be interesting to have conducted such an experiment.

Author response:
**1 & 2. Using SAM to generate masks is straightforward and the mask voting process gives no surprises. If generating a mask is the key to generating high-quality images, why not directly use an encoder-decoder model like U-Net or an SD to predict the target complete mask in one step?**

|Method|SD|UNet |IMD (Ours)|
| ----| :----: |:----: |:----: |
|IoU| 82.4 | 75.4 |88.5 | 

Table A: IoU of predicted masks on AHP dataset.

In this paper, we investigate the relation between object generation and segmentation. We found that even if the segmentation model is not trained, the segmentation can still benefit the generation. It is a good idea to conduct experiments to investigate other segmentation models.
We report the results of using a ControlNet (frozen SD) and UNet with the partial object as the condition to predict the complete mask in Table A. We notice that the complete mask predicted from the IMD process shows obviously higher IoU to the ground truth which indicates the IMD process is an effective mask prediction approach.

In addition, we would like to clarify that the mask prediction is not simply conducted by the SAM but by the entire IMD process with both generation and segmentation stages. We treat the segmentation as the shape property of the generated object. Therefore, with an object generation model, we can directly obtain the shape using SAM without training another mask-specific model.

We also want to further clarify the usage of the voting operation. Given a partial object $I_p$, the generation model samples from a distribution that contains both realistic and unrealistic images. As shown in Figure A ([github.com/iclr23anonymous739/dist.pdf](https://github.com/iclr23anonymous739/rebuttal/blob/main/dist.pdf)), since the image distribution is complex, the expectation $E[I_c]$ cannot represent a realistic image. However, when we consider the shape of the generated images, we are excited to find that the expectation $E[\mathcal{S}(I_c)]$ leads to a more realistic shape. We consider this observation interesting as, for most of the other generation tasks (non-conditioned), averaging the object shape will just yield an unrealistic random shape. This observation (Figure 4) serves as one of our core observations to build the IMD process. Here, SAM serves as the tool to extract the object shape and voting is a way to binarize the expectation of object shapes to a binary mask.

Reviewer comment (Experiments):
More ablation study of the proposed method should be given. E.g., different segmentation models, different mask voting strategies.

Author response:
**3. More ablation studies of the proposed method should be given. E.g., different segmentation models, and different mask voting strategies.**

|Method|Mask2Former|ClipSeg|SAM|
| :----| :----: | :----: | :----: |
|FID| 22.5| 19.9 |16.9|

Table B: Performance with different segmentation models.

|Method|Voting with logits|Mean with logits|Voting with mask| Mean with mask|
| :----| :----: | :----: | :----: | :----: |
|FID| 16.9| 17.2 |17.6| 17.0 |

Table C: Performance with different voting strategies. Logits: mask logits before binarzing. 

We conducted ablation experiments to determine the design choice in the segmentation stage. We report the ablation studies about segmentation models and voting strategies in Table B and Table C. The current design choice of using SAM and voting with logits is based on the ablation results.

Reviewer comment (Evaluation):
One of the main drawbacks of the proposed method is its slow inference speed. Each iteration in the generation process involves generating multiple image candidates and their segmentations (segment anything model is quite slow), let alone running for multiple iterations. The authors did not report the running time statistics of different methods and omitted this limitation in the limitations section. I encourage the authors to have more comparisons and discussions of the running time.

Author response:
**4. Comparisons and discussions of the running time.**

|Reduce SD step|Generation stage|Segmentation Stage|Total|FID|
| :----:| :----: | :----: | :----: | :----: |
|| 14.3s | 1.2s |15.5s |16.9|
|$\checkmark$| 8.6s | 1.2s|9.8s| 17.4|

Table D: Inference time for each stage of IMD on a single V100 GPU. 

Thank you for your suggestion. We demonstrate the running time of each component in IMD as shown in Table D. We acknowledge that the inference speed of MaskComp is slow. To improve the inference speed, we notice that decreasing the diffusion step number in the first several IMD steps will not severely degrade the performance. By incorporating this idea into MaskComp, the average running time was reduced to 2/3 original time. We will add the discussion to our limitation section.

Reviewer comment (Evaluation):
The FID metric shown in Table 1 is problematic. The FID only considers the ground-truth object area; however as shown in Figure 6, different methods generate different foreground areas that may not match the ground-truth object mask. Thus, this metric will yield a higher FID score if the generated object has a large discrepancy with the ground truth even if it is realistic.

Author response:
**5. The FID only considers the ground-truth object area. Thus, this metric will yield a higher FID score if the generated object has a large discrepancy with the ground truth even if it is realistic.**

|Method|ControlNet|Kandinsky 2.1|SD 1.5|SD 2.1|MaskComp (Ours)|
| ----| :----: | :----: | :----: | :----: | :----: |
|AHP|45.4|43.9|41.4|39.9|21.3|
|DYCE|49.4|47.7|43.4|41.1|25.4|

Table E: Performance comparison with FID scores calculated with SAM masks. 

Thanks for the comment. We report the FID scores evaluated by cropping the object with SAM in Table E. We notice that, for the FID scores calculated with both GT mask and SAM mask, our method outperforms baseline methods with an obvious marginal which indicates the effectiveness of the proposed method.

Reviewer comment (Experiments):
I cannot find the qualitative comparisons with ControlNet and Kandinsky.

Author response:
**6. Qualitative comparisons with ControlNet and Kandinsky.**

Thanks for pointing this out. We add additional comparisons with ControlNet and Kandinsky in Figure 6 ([github.com/iclr23anonymous739/comparison.pdf](https://github.com/iclr23anonymous739/rebuttal/blob/main/comparison.pdf)). We notice that MaskComp achieves superior performance compared to baseline methods.

Reviewer comment (Experiments):
I would like to see the results of using the ground-truth mask as input to see how different methods perform with such an oracle. This also tells the generation upper bound of mask-guided methods.

Author response:
**7. The results of using the ground-truth mask as input to see how different methods perform with such an oracle (This tells the generation upper bound of mask-guided methods).**

|Mask|Visible|Noisy|Complete|
| ----| :----: | :----: | :----: |
|FID| 16.9| 15.3 |12.7|

Table 2 (a): Performance with different conditioned masks. Occlusion rate: partial mask > noisy mask > complete mask.

We reported the results of using the ground-truth mask as the condition in Table 2 (a) and there is also a visual comparison among different conditioned masks in Figure 2. With the ground-truth mask, the generation model shows a promising performance to complete the objects. This also serves as the core motivation for our iterative mask denoising process (IMD) which starts from a partial mask to gradually generate a complete mask to boost the generation process.