Paper: Sin3DM: Learning a Diffusion Model from a Single 3D Textured Shape (ICLR-2024)

Reviewer comment (Writing):
The paper writing is sometimes overly convoluted and the same arguments are repeated too many times. In my opinion, this makes the paper at times a bit hard to read. (See the Questions section).

Author response:
__Q: Paper writing.__  
__A:__ Thanks for the suggestion. We have fixed the typos and mentioned less frequently the need for relightable assets in the revised paper. These changes are highlighted with color in the revised paper.

Reviewer comment (Evaluation):
While limitations are somewhat adequately discussed, it would have been interesting to see failure cases.

Author response:
__Q: Failure cases. What are the results of this method for shapes without repeated patterns?__  
__A:__ With a single input shape, our method learns its local patterns. Thus, for an input shape that has a highly semantic structure but no repeated patterns, e.g., a 3D character, the output may lose the semantic and thus looks unrealistic. In the revised paper, we show failure cases in Figure 16 of the appendix.

Reviewer comment (Novelty):
The related work section can be, in my opinion, improved. In particular, I believe that this paper should mention an important use-case for single-sample generative models, which is texture synthesis. This is even more important as this paper is working with Textured 3D shapes, and most of the examples shown contain repetitions (eg bricks or windows). I suggest including important work like "Non-stationary texture synthesis by adversarial expansion", Zho et al., Siggraph 2018; "Self-organising textures", Niklasson et al. 2021; and "SeamlessGAN: Self-Supervised Synthesis of Tileable Texture Maps", Rodriguez-Pardo et al. TVCJ 2022.

Author response:
__Q: Improve the related work section by discussing prior papers on texture synthesis.__  
__A:__ Thanks for the suggestion and we have improved the related work section accordingly in the revised paper.

Reviewer comment (Evaluation):
Some components of the model are detrimental to the diversity of the generated samples. This is a traditional trade-off in many generative models, where increased diversity can lead to less realistic outputs. However, I believe that this paper leans heavily towards realism, hindering diversity. I believe that some level of control on this tradeoff should be allowed to the user, and changing the receptive field of the model is, in my eyes, the only possible way to achieve this. This could be discussed in the limitations section or some future work could be hinted to address this.

Author response:
__Q: The trade-off between diversity and quality can only be controlled by changing the receptive field size. This could be discussed in the limitations section.__  
__A:__ We acknowledge this limitation and include a discussion on it in the revised paper.

Reviewer comment (Experiments):
What are the results of this method with unstructured shapes? Most of the results shown contain repeating patterns (like columns, windows or bricks). What would happen with a human 3D shape, for instance?

Author response:
__Q: Failure cases. What are the results of this method for shapes without repeated patterns?__  
__A:__ With a single input shape, our method learns its local patterns. Thus, for an input shape that has a highly semantic structure but no repeated patterns, e.g., a 3D character, the output may lose the semantic and thus looks unrealistic. In the revised paper, we show failure cases in Figure 16 of the appendix.

Reviewer comment (Experiments):
Can this method generate tileable texture shapes? For example, it would be valuable to generate a tileable version of the shape in Figure 6 (a).

Author response:
__Q: Can this method generate tileable texture shapes?__  
__A:__ No, we didn’t come up with a way to enforce the output to be tileable.

Reviewer comment (Writing):
Regarding the resolution of the method, it is mentioned that the maximum resolution across one dimension is 256 for the input shape, and that the receptive field is 128, while it is also mentioned that the receptive field is 40% of the input resolution. Could the authors clarify this?

Author response:
__Q: Clarify the receptive field size.__  
__A:__ Sorry for this confusion. The size of the receptive field is 54, which is roughly 40% of the triplane size 128.

Reviewer comment (Experiments):
Does this model allow for texture transfer or for structural analogies? See "Non-Stationary Texture Synthesis by Adversarial Expansion", "Drop the GAN" or "Neural Photometry-guided Visual Attribute Transfer" for examples of this. It would be very powerful to be able to condition the generation of one or more of the tri-planes with a structure from another image.

Author response:
__Q: Does this model allow for texture transfer or for structural analogies?__  
__A:__ No. The geometry and texture is coupled in our triplane latent space, which makes it difficult to do texture transfer or structural analogies. This is left for future work.

Reviewer comment (Evaluation):
Is there a middle ground between using and not using the $\epsilon$-prediction? I am concerned that this introduces a trade-off between diversity and quality that may be limiting the impact of the proposed method.

Author response:
__Q: Is there a middle ground between using and not using the epsilon-prediction?__  
__A:__ In the diffusion model literature, there is another alternative objective ‘v-precition’ which is formulated as a combination of $x_0$ and $\epsilon$, $v=\alpha_t \epsilon - \sigma_t x_0$. This might be a middle ground that gives better a trade-off.

Reviewer comment (Experiments):
Can the authors show more examples of results of this method? I think it would be valuable to see more results on controlled generation, retargeting and PBR generation.

Author response:
__Q: Need more examples on controlled generation and retargeting.__  
__A:__ In the revised paper, we show more examples of retargeting and controlled generation in Figure 15 of the appendix.

Reviewer comment (Writing):
I suggest the author mention the need for relightable assets less frequently. I think the introduction motivates the problem well, however, it is mentioned more times during the paper, hindering its readability.

Author response:
__Q: Paper writing.__  
__A:__ Thanks for the suggestion. We have fixed the typos and mentioned less frequently the need for relightable assets in the revised paper. These changes are highlighted with color in the revised paper.

Reviewer comment (Novelty):
After training, the model could only generate 3D contents with the same global structure. Since we already have the basic 3D mesh at the beginning, it seems that the application is a bit narrow and limited. Besides, achieving this goal needs to train two models, an auto-encoder and a diffusion model. The outcome does not seem to match the effort.

Author response:
__Q: The application seems a bit narrow and limited.__  
__A:__ We stress that synthesizing new samples that resemble the input example has numerous downstream applications in the content production industry, and research on this topic dates back to the 1990s in vision and graphics [1][2]. The 2D counterpart of our method is the texture synthesis, which has been studied for a couple of decades (see discussion of related works in our revised paper) and many products are built on this idea. The 3D asset synthesis is highly desired, but remains challenging, in the gaming industry, film making, advertisement, to name a few. It is these high demands that motivate our work and justify the effort.

[1] Efros, Alexei A., and Thomas K. Leung. "Texture synthesis by non-parametric sampling." ICCV, 1999.  
[2] Merrell, Paul. "Example-based model synthesis." I3D, 2007.

Reviewer comment (Novelty):
The novelty of this work lies at the usage of tri-plane as the input of diffusion model and the introduction of tri-plane convolution. Both aspects do not offer a solid contribution. Using tri-plane as a compact 3D representation has been explore by many prior works. The contributino tri-plane convolution also lacks complete analysis, such as discussion and comparison with similar modules that could fuse three axis information.

Author response:
__Q: Technical novelty. Discussion on tri-plane convolution.__  
__A:__ Though the network components (triplane autoencoder and latent diffusion model) are established techniques, our paper’s novelty lies in the adaptation for the single 3D shape generation task with limited receptive field and triplane-aware convolution to achieve experimentally verified strong results. 
In addition, we have an ablation study on tri-plane convolution to show its effectiveness (see Table 2 and Figure 12). If the reviewer has in mind a similar module that can fuse three axis information, we'd be happy to test it out.

Reviewer comment (Experiments):
What would happens if multiple objects instead of single object are encoded and fed into diffusion model in training? Would it not be able to generate realistic samples?

Author response:
__Q: What would happen if training the model on multiple objects?__  
__A:__ We tried this in our early experiments. With multiple objects, the model is in effect trained to learn the patch distribution over all the input objects. As a result, the generated sample contains different patches from multiple inputs. It often looks unnatural if the multiple input objects have different global structures. However, this problem is out of the scope of this paper.

Reviewer comment (Writing):
What does retargeting mean in this context? I could only get the meaning of outpaining and local editing.

Author response:
__Q: What does retargeting mean in this context?__  
__A:__ It means generating new shapes whose sizes and aspect ratios are different from the input example.

Reviewer comment (Experiments):
How do the input noise and target tri-plane feature match each other when training the diffusion model on only one input shape? One input shape could only correspond to one tr-plane feature maps, while different noise is sampled in each iteration.

Author response:
__Q: How do the input noise and target tri-plane feature match each other with only one input shape?__  
__A:__ It is correct that “one input shape could only correspond to one tr-plane feature map, while different noise is sampled in each iteration”. The training objective is essentially the typical diffusion model objective, with the expectation being solely over random noise (see Eqn. 6).

Reviewer comment (Writing):
It is mentioned several times that a unique benefit of the proposed method is that inputs and outputs meshes. This is not the case, as instead the 3D representation is triplanes encoding signed distance as well as texture color. While this can indeed be converted to a mesh using marching cubes, it's not fair to claim that the method outputs a mesh.

Author response:
__Q: The claim that the method outputs a mesh.__  
__A__: In the revised version, we clarify that the proposed method does not directly output a mesh, but rather outputs a triplane representation that can be converted to a high quality textured mesh.

Reviewer comment (Experiments):
While the authors cite and briefly discuss [Li et al. 2023], no qualitative or quantitative comparisons are provided, with the explanation that Li et al. do not produce parameterized meshes. While there are merits to the proposed representation, [Li et al. 2023] seems like the most natural point of comparison—many of the experiments are nearly identical to those in that work. For this reason, the reviewer cannot recommend acceptance as is.

Author response:
__Q: Comparison to [Li et al 2023].__  
__A:__ We add the quantitative and qualitative comparison to Sin3DGen [Li et al 2023] in the revised paper. As reported in Table 1, our method achieves better scores in terms of geometry quality, texture quality and diversity. As shown in Figure 14 of the appendix, Sin3DGen [Li et al 2023], which is based on Plenoxels radiance fields, often produces broken surfaces and distorted textures. In contrast, our method is able to generate 3D shapes with high quality geometry and fine texture details.

Reviewer comment (Writing):
Typo in Abstract: "example as long" should be "example has long".

Author response:
__Q: Typos.__  
A: Thanks for pointing these out. We have corrected them in the paper.

Reviewer comment (Writing):
Typo in Section 4: "The decoder first refine..." should be "The decoder first refines... gathers...".

Author response:
__Q: Typos.__  
A: Thanks for pointing these out. We have corrected them in the paper.