Paper: CustomNet: Zero-shot Object Customization with Variable-Viewpoints in Text-to-Image Diffusion Models (ICLR-2024)

Reviewer comment (Experiments):
The proposed CustomNet focuses on the limitation of 256 × 256 resolution, which may affect the quality of generated images.

Author response:
## **W1. Limitation of 256$\times$256 Resolution.**



- **Limitation of Resolution**

    - The constraint of the resolution of $256\times256$ can be addressed in two ways. For the first one, we can use an improved Zero1-to-3, *e.g.*, The Zero1-to-3 ++ [8], which has extended the resolution to $512\times512$. The second one is to train our CustomNet on large-resolution data from the current zero-1-to-3 checkpoint. 

    
    -  We update more real-world object customization cases in the update paper, shown in **Figures 12, 13, 14**.

***

Reviewer comment (Experiments):
Non-rigid transformations and object style changes are not supported, limiting flexibility.

Author response:
## **W2. Non-rigid transformations and style changes.**

- It can be solved by training our model on a larger web-scale dataset with our dataset construction pipeline. The Non-rigid transformations and style changes can be controlled by text prompts, which has been proven in the SOTA Text-to-Image Diffusion model. Current Text-to-image diffusion models present promising results by training with large web-scale (text, image) pair datasets. With our dataset construction pipeline, we can convert the web-scale text-image pair datasets into (text, image, and novel-view image) pair datasets, then train our model to achieve non-rigid transformations and style changes.

***

Reviewer comment (Novelty):
How is CustomNet different from past methods for object customization? How does it achieve better identity preservation and varied output generation?

Author response:
## **Q1. Difference between the proposed CustomNet and existing methods. How does CustomNet achieve better performance?**


- **Difference from other works.**

    - We should kindly remind you that we novelty incorporate the 3D novel view synthesis capability into the task of object customization, obtaining more flexible customization results with various backgrounds. Note that previous object customization methods (optimization-based Dreambooth[4], Textual Inversion[5], and encoder-based Paint-by-example[2], Anydoor[3]) do not consider the 3D properties of the inserted object but learn a compressed token embedding to represent the object visual information. Therefore, they struggle to balance identity preservation and output diversity.  For example, Paint-by-example[2] loses identity, while Anydoor [3] produces copy-paste results, lacking diversity. other optimization-based methods Dreambooth[4], and Textual Inversion[5] suffer long test-time optimization times and are prone to over-fitting.

    - In 3D novel-view synthesis fields, following Zero-1-to-3, most works focus on how to improve it to generate better geometry and quality, e.g. Magic123 [6], One-2-3-45 [7]. They still have the limitation in Zero-1-to-3 that can not be applied in real-world customization applications directly. However, CustomNet provides such a possibility to bridge the 3D novel-view synthesis and object customization with flexible multiple controls. 

- **CustomNet achieves better identity preservation and varied output generation in the following three aspects**:

    - **Model architecture.**  Other methods mainly extract object identity visual information into a compressed embedding vector (whose dimension is normally 1 $\times$ 768), Which makes it hard to preserve the identities of different objects. CustomNet preserves objects' identity by contacting objects corresponding latent with the UNet input latent, and sending them into UNet together. the object latent is the output of the VAE encoder, which can reconstruct the object input image by the VAE decoder. Therefore, the object latent contains the most information as the total model (VAE + Unet) can do.

    - **Explicit control with 3D prior.** Other methods change the object generation by text control. The text prompt may be ambiguous in semantics, and the words that are not used be describe the object could also affect the object's identity. We design a separate viewpoints control branch that learns to warp the object into different view synthesis. 3D viewpoint warping is a rigid process that has less uncertainty than text description. Therefore, with our explicit control, we can preserve object identity better when generating various results.

    - **Dataset construction.**  Other methods mainly use the text-image pair dataset for customization, while we further generate a multi-view for the object image which provides more information for the model to learn the object visual concepts. 

***

Reviewer comment (Reproducibility):
Can you explain the detailed designs mentioned in the text that allow for location and background control in CustomNet?

Author response:
## **Q2. Location control and background control.**


- **Location control** is achieved by specifying a bounding box where the object is located in the result-generated image. Assume that we are going to generate a customized image that the object will locate at [x, y, w, h], then we just need to resize the reference object image into the size of [w, h] and place its left-top corner at the [x, y] coordinate of a background-free image (this image is the same size as the target image being denoised but without background). We have demonstrated it in detail in the main paper, please see Sec. 3.1 and model pipeline Fig. 2.

- **Background control**  is controlled by two branches: text generation and image composition. In text generation, the user can provide a text prompt that describes the background, and take it into text encoder input. In image composition, the user can provide a background image directly, and concat it with the UNet Input latent.

***

Reviewer comment (Experiments):
How does CustomNet handle real-world objects and complex backgrounds in its dataset construction pipeline, and what's the motivation?

Author response:
## **Q3. How to handle real-world images in dataset construction pipeline and motivation**

- Given a real-world image, we need to process it to obtain the text description and object novel-view image as input. 

    - For text prompt, we send the input image into the BLIP2 model to generate the corresponding text description of the image
    
    - For the novel-view images, we first use the BLIP-2 5T version to extract the foreground object with the instruction {"image": image, "prompt": "Question: What foreground objects are in the image? find them and separate them using commas. Answer:"}. Then we feed the object to SAM, SAM can receive text as input and output the segmentation mask of the corresponding object, and we can get the single object image by the segmentation mask. Finally, we use Zero-1-to-3 to generate novel-view images of the objects.



- The motivation of dataset construction is to facilitate fusing real-world 3D prior in CustomNet training for better object identity preservation and harmonious natural generation. Training our model only with synthetic datasets will generate 'floating' artifacts between the object and background. Training with real-world data, the artifacts are alleviated significantly.

***

Reviewer comment (Theory):
Can you explain zero-shot object customization without test-time optimization and its multi-aspect control?

Author response:
## **Q4.  Explanations of zero-shot object customization without test-time optimization and its multi-aspect control**

- **Zero-shot object customization** is a task or scenario that which a user provides a reference object image that may not be seen in model training to do generation while preserving their identities.

- **Test-time optimization** corresponds to the optimization-based methods used in customization. When users provide multiple images of the same object during inference, this method iterative optimizes model parameters to perform online training for these images for minutes to hours, and then the users can use the model to generate customized images. The phrase 'test-time' refers to the user inference process, and 'optimization' means the category of optimization method. CustomNet can directly perform customized generation when the user provides only a single image during inference. Therefore CustomNet can achieve generation **without test-time optimization**.

- **Multi-aspect control** means that our CustomNet can control customized object generation by viewpoints, locations, text prompts, or background images simultaneously while other methods mainly control it only by text prompts.


***

Reviewer comment (Novelty):
The novelty of the approach is somewhat limited, as it essentially plugs background generation/compositing into the Zero-1-2 architecture. Placing the object into a localized bounding box as a guide instead of keeping it centralized would be straightforward to do in Zero-1-2.

Author response:
## **W1. Novelty and object placement.**



### **About Novelty**
We would like to kindly remind you that CustomNet is designed for **the task of object customization** that can place the object into desired scenes harmoniously with the explicit viewpoint, location, and background controls. We are not a simple application of zero-1-to-3 in the object customization field, but we introduce it to image customization for the first time, by addressing the inherent limitations of zero-1-to-3. 
Compared to previous image customization methods, CustomNet is the *first to investigate the novel view synthesis in the customization field*. 
It can preserve the object's identity while controlling the viewpoint and increasing the output diversity.

- **Improvements of zero-1-to-3.** 

    - Our module designs, together with the training dataset construction, are important improvements for the final outcome of object customization. They cannot be viewed as a simple addition to zero-1-to-3. They are specialized designs for object customization. Without those designs for object customization in the zero-1-to-3 model, the synthesized image lacks the background, and the object is centrally localized, which is far from our goal of object customization. The comparison to the Zero-1-to-3 limitation in our updated Figure 10 shows that our designs are necessary and effective. 
    Besides, the concurrent paper TOSS [1] also mentions in their paper that the extra text input could improve the generation quality of novel views over zero-1-to-3.
    In our CustomNet, incorporating real-world data with text could also benefit the generation results, which is insightful to improve the novel-view synthesis model.  

        
    - Specifically, as described in the main paper, we demonstrate that Zero-1-to-3 has the following limitations when applied to object customization, and we make specific designs to address those limitations and adapt them to the task of object customization.
    
        - I. Zero-1-to-3 cannot be controlled by text for background. **v.s.** We design a dual-attention mechanism to simultaneously control viewpoint and text. 
        - II. Zero-1-to-3 cannot generate/composite backgrounds with a given background image. **v.s.** We designed a unified pipeline to combine it.
        - III. Zero-1-to-3 can only process images located in the image's center. **v.s.** We design the concatenation module to place the object in arbitrary locations flexibly. 
        - IV. Zero-1-to-3 usually generates unreal effects. **v.s.** We design a real dataset construction pipeline to extend its capability on real data. 

- **Difference from other works.**

    - We should kindly remind you that we novelty incorporate the 3D novel view synthesis capability into the task of object customization, obtaining more flexible customization results with various backgrounds. Note that previous object customization methods (optimization-based Dreambooth[4], Textual Inversion[5], and encoder-based Paint-by-example[2], Anydoor[3]) do not consider the 3D properties of the inserted object but learn a compressed token embedding to represent the object visual information. Therefore, they struggle to balance identity preservation and output diversity.  For example, Paint-by-example[2] loses identity, while Anydoor [3] produces copy-paste results, lacking diversity. other optimization-based methods Dreambooth[4], and Textual Inversion[5] suffer long test-time optimization times and are prone to over-fitting.

    - In 3D novel-view synthesis fields, following Zero-1-to-3, most works focus on how to improve it to generate better geometry and quality, e.g. Magic123 [6], One-2-3-45 [7]. They still have the limitation in Zero-1-to-3 that can not be applied in real-world customization applications directly. However, CustomNet provides such a possibility to bridge the 3D novel-view synthesis and object customization with flexible multiple controls. 


- **Object placement.** The original Zero-1-to-3 model struggles to synthesize novel views of an object at an arbitrary location, in their camera settings, they cannot control translation but only azimuth angle, polar angle, and size. Therefore, we introduce a localized bounding box as guidance. 
It is a simple but effective design for solving this problem. 
When applying the same location control strategy to zero-1-to-3 directly, we can see that significant artifacts would occur, while our CustomNet has reasonable results in the updated **Figure 10**.

Reviewer comment (Reproducibility):
An interesting component of the system is the synthetic dataset of objects composed onto backgrounds. Publishing that dataset for future research would add to the contribution.

Author response:
## **W2. Dataset construction pipeline.**

- Thank you for your advice. Our dataset can help CustomNet obtain more harmonious results since the real images serve as the model target. We will release the
dataset, together with the full pipeline for constructing the dataset to contribute to the development of future research.

Reviewer comment (Theory):
How come you need to specify both the camera translation and image location? Wouldn't it be enough to only encode camera rotation and the bounding box to place the object anywhere in the image?

Author response:
## **Q1. Camera translation and object location.**

- We should clarify that the definition and design of **RT** is consistent with the setting in Zero-1-to-3. More specifically, Zero-1-to-3 renders 3D objects into images with a predefined camera pose that always points at the center of the object. 
This means the viewpoint is actually controlled by polar angle $\theta$, azimuth angle $\phi$, and radius $r$ (distance away from the center), not the full rotation and translation matrices, resulting all objects to be central of the rendered image. Thus, the Zero-1-to-3 can only synthesize objects at the image center, lacking the capacity to place the object at the **arbitrary spatial location** in the image.

- Note that zero-1-to-3 cannot perform translation and can only generate objects localized centrally. We tackle this problem by specifying the bounding box at the desired location to the latent. This design is actually consistent with your comments.

Reviewer comment (Theory):
You mention that Zero-1-to-3 method cannot produce non-centered objects. But they allow users to specify full 3D camera rotation and translation, which is a component that this paper inherits. Why wouldn't translation enable users to place objects off-center?

Author response:
## **Q2. Translation in Zero-1-to-3.**

- Note that the zero-1-to-3 model cannot synthesize objects with arbitrary spatial translations, and can only control the size of the objects. Please refer to Q1 for the specified reasons. We address this limitation by placing objects in a specified location bounding box.

Reviewer comment (Presentation):
If I understood the section 3.2 correctly, to make Figure 2 more clear, you could show 2 branches as the input to the text encoder: "sitting on the beach" for the Generation branch and NULL for the Composition branch. Although I may be wrong, since the model could possibly take in both the background image and the textual description. What happens in that case in terms of the output?

Author response:
## **Q3. Generation and composition branches.**

- You are right. We provide two ways to control the background: textual description and given background image, referred to as generation and composition branches, respectively. 
For the generation branch, We added a new text branch to enable generating the background complying with the given text description with the Dual-Attention module. 

- As for the composition branch, when both given text and background image, the model generation will be dominated by the background images, as it provided more information than text when training.

Reviewer comment (Writing):
Phrases like "delicate designs" and "intricate designs" seem out of place in this paper. The network design, if that's what this refers to, seems reasonable and straightforward.

Author response:
## **Q4. Delicate or intricate.**

- We will modify our phrase more precisely. Our goal that customizing an object with identity preservation and diverse control is straightforward. However, to achieve this goal, we have several careful designs from dataset construction (both synthetic and real datasets) to model architecture (DualAttn for text and viewpoints, and location control in initial latent concat).

Reviewer comment (Writing):
You did not define d in equation 2.

Author response:
## **Q6. Definition of d.**

- Thank you for pointing out the mistakes. We have updated the definition on it in the **Sec. 3.2**.

Reviewer comment (Evaluation):
Paper should discuss some remaining artifacts in the limitation section, such as:  
  * Minor change of identity in Figure 1 Row 3 (look at the number 1 on the toy car)  
  * Bad aspect ratio of the dog in Figure 1 Row 4 (it's too squished)  
  * The white pixels around the character in Figure 4 Row 1 Last Column  
Where do these come from and how would one go about improving them?

Author response:
## **Q8. Artifacts in the limitation section.**

- First, this is a hard case in which other models usually fail to preserve their identity, see the updated **Figure 11**. Some minor changes would indeed occur when customizing objects with highly detailed textures. Compared to previous methods, our identity preservation is highly competitive. 
We will mention it in the limitation, and continuously improve the identity in future work.

- The white square in the background causes the artifacts. When testing another method, we place a white square to mask the original object.
When testing our CustomNet, we mistakenly keep the white square and the model interprets the white pixel as the background.
Our CustomNet will generate a good image without artifacts when provided with a pure background image. Please refer to the revised **Figure 4** for further clarification.

- The model allows for modifications to be made to the location bounding box and viewpoints as specified by the user. We have set an overhead view of the dog in the figure, resulting in it appearing squished.
Such an artifact can be addressed by modifying the location bounding box and viewpoints.

Reviewer comment (Novelty):
A significant concern with this paper is its limited novelty. It leans heavily on the "Zero-1-to-3" model as its foundation. While the addition of background inpainting offers an enhancement, the core mechanism—explicit 3D novel view synthesis—remains unchanged. Moreover, it inherits constraints from "Zero-1-to-3", particularly the resolution limitation of 256 × 256, which might not be practical for real-world scenarios.

Author response:
## **W1. Novelty and Resolution Constraint.** 


### **About Novelty**
We would like to kindly remind that CustomNet is designed for **the task of object customization** that can place the object into desired scenes harmoniously with the explicit viewpoint, location, and background controls. We are not a simple application of zero-1-to-3 in the object customization field, but we introduce it to image customization for the first time, by addressing the inherent limitations of zero-1-to-3. 
Compared to previous image customization methods, CustomNet is the *first to investigate the novel view synthesis in the customization field*. 
It can preserve the object's identity while controlling the viewpoint and increasing the output diversity.

- **Improvements of zero-1-to-3.** 

    - Our module designs, together with the training dataset construction, are important improvements for the final outcome of object customization. They cannot be viewed as a simple addition to zero-1-to-3. They are specialized designs for object customization. Without those designs for object customization in the zero-1-to-3 model, the synthesized image lacks the background, and the object is centrally localized, which is far from our goal of object customization. The comparison to the Zero-1-to-3 limitation in our updated Figure 10 shows that our designs are necessary and effective. 
    Besides, the concurrent paper TOSS [1] also mentions in their paper that the extra text input could improve the generation quality of novel views over zero-1-to-3.
    In our CustomNet, incorporating real-world data with text could also benefit the generation results, which is insightful to improve the novel-view synthesis model.  

        
    - Specifically, as described in the main paper, we demonstrate that Zero-1-to-3 has the following limitations when applied in object customization, and we make specific designs to address those limitations and adapt them to the task of object customization.
    
        - I. Zero-1-to-3 cannot be controlled by text for background. **v.s.** We design a dual-attention mechanism to simultaneously control viewpoint and text. 
        - II. Zero-1-to-3 cannot generate/composite backgrounds with a given background image. **v.s.** We designed a unified pipeline to combine it.
        - III. Zero-1-to-3 can only process images located in the image's center. **v.s.** We design the concatenation module to place the object in arbitrary locations flexibly. 
        - IV. Zero-1-to-3 usually generates unreal effects. **v.s.** We design a real dataset construction pipeline to extend its capability on real data. 

- **Difference from other works.**
    
    - We should kindly remind that we novelty incorporate the 3D novel view synthesis capability into the task of object customization, obtaining more flexible customization results with various backgrounds. Note that previous object customization methods (optimization-based Dreambooth[4], Textual Inversion[5], and encoder-based Paint-by-example[2], Anydoor[3]) do not consider the 3D properties of the inserted object but learn a compressed token embedding to represent the object visual information. Therefore, they struggle to balance identity preservation and output diversity.  For example, Paint-by-example[2] loses identity, while Anydoor [3] produces copy-paste results, lacking diversity. Other optimization-based methods Dreambooth[4], and Textual Inversion[5] suffer long test-time optimization times and are prone to over-fitting.

    - In 3D novel-view synthesis fields, following Zero-1-to-3, most works focus on how to improve it to generate better geometry and quality, e.g. Magic123 [6], One-2-3-45 [7]. They still have the limitation in Zero-1-to-3 that can not be applied in real-world customization applications directly. However, CustomNet provides such a possibility to bridge the 3D novel-view synthesis and object customization with flexible multiple controls. 

### **Limitation of Resolution**

- The constraint of the resolution of $256\times256$ can be addressed by in two ways. For the first one, we can use an improved Zero1-to-3, *e.g.*, The Zero1-to-3 ++ [8], which has extended the resolution to $512\times512$. The second one is to train our CustomNet on large-resolution data from the current zero-1-to-3 checkpoint. 

-  We update more real-world object customization cases in the update paper, shown in **Figure 12, 13, 14**.

Reviewer comment (Reproducibility):
The approach to decompose foreground objects using SAM and augmenting them using "Zero-1-to-3" is intriguing. However, there are lingering questions about the methodology. For instance, considering that SAM segments every element in a scene, how does the paper specifically determine the foreground object? Furthermore, if the selected foreground object falls outside the "Zero-1-to-3" training domain, is the efficacy of the method compromised?

Author response:
## **W2. Details of the Dataset construction pipeline.**

We provide more details about our dataset construction pipeline. 

- How to specifically determine the foreground object:  Given an image, We use BLIP2 (Github repo: https://github.com/salesforce/LAVIS/blob/main/examples/blip2_instructed_generation.ipynb) to extract the foreground object with the following instruction:
'''
{"image": image, "prompt": "Question: What foreground objects are in the image? find them and separate them using commas. Answer:"}
'''
Then we feed the queried object and its corresponding text to SAM, SAM can receive text as input and output the segmentation mask of the corresponding object.

- When the selected foreground object falls outside the Zero-1-to-3 training domain, we may get unsatisfying results. We have tried to alleviate this issue in our CustomNet
    1. We jointly train with both synthetic and real datasets, which will complement each other and narrow the gap between the synthetic and real datasets. 
    2. During training, though the input object image may be unsatisfying, we adopt the original natural image with the object as the target. It is helpful for model training as the target image provides information about how to place an object into a result image harmoniously and naturally, which helps in real-world scenarios. On the other hand, the synthetic dataset endows the model with the warping ability among multiple views. Combined with the two datasets, CustomNet demonstrates natural and high-fidelity customization. 
    3. In our CustomeNet, except for the image input, CustomNet also receives text as input. The text helps to improve the viewpoints control ability that Zero-1-to-3 is hard to manage, as shown in the updated **Figure 10**.

Reviewer comment (Experiments):
The paper mostly compares CustomNet with "Zero-1-to-3" and a few other models. A broader discussion with state-of-the-art models in the domain would have given a better quality assessment of CustomNet's performance. For example, [1] also employs stable diffusion for background inpainting while preserving the appearance of foreground objects through a dual-branch composition.

Author response:
## **W3. Broad Discussion with other methods.**

- To achieve customization, We can use a one-stage model like the optimization-based method Dreambooth[4] or Encoder-based Paint-by-example [2]. The optimization-based method suffers long test-time optimization times and is prone to over-fitting, while the encoder-based method may lose identity. We can also use two-stage methods that first get the foreground object image, then paste it to a new background directly or with diffusion inpainting. The OVTrack [9] uses stable diffusion background inpainting while preserving the appearance of foreground objects for a data hallucination strategy tailored to appearance modeling in multi-object tracking. Our Customnet can generate different views of the foreground object, and use text to generate background for it.

- We have conducted quantitative and qualitative comparisons to the state-of-the-art encoder-based and optimization-based image customization methods in Sec. 4 of the main paper. We update further discussion with the OVTrack[9] and other models, such as Blended latent diffusion[10], etc. in **the Discussion in Sec. 3 in the updated paper**.