Paper: What Does Stable Diffusion Know about the 3D Scene? (ICLR-2024)

Reviewer comment (Experiments):
The paper does not compare the proposed method to other generative models, such as StyleGAN[1] and VQGAN[2], and other diffusion models such as DDPM[3] and GLIDE[4]. This would help to provide more insights on whether generative model generally gains good performance on 3D scene understanding or only diffusion models. This is a significant weakness, as it is difficult to assess the significance of the power of the diffusion model without knowing how it compares to other methods.

Author response:
W1 (Evaluation of other generative models): See Reviewer G3yb W3. As mentioned in the overall comments, our objective was to propose a protocol and use this to evaluate a set of models to compare to Stable Diffusion, but not to comprehensively evaluate all models. We have added an evaluation for a second generative model, VQGAN, and updated the results in our revision, showing that Stable Diffusion still has a superior performance (Table 4). DINO and CLIP models are compared because they are strong self-supervised features and are used extensively for scene understanding tasks.

Reviewer comment (Experiments):
Diffusion models have been used in different tasks for 3D data [5][6] and demonstrated promising results in different tasks. There are also other different important aspects of 3D scene understanding, eg. scene categories, attributes, and structures.[7] How does the author choose these 7 properties, are these 7 properties enough to “understand” the 3D scene, what problems this paper is trying to solve by experimenting on these properties, and the challenges that lie on them?

Author response:
W2 (Choice of 7 properties): The 7 properties we choose are important in terms of understanding the 3D physical scene. As mentioned in the discussion and future work section of the paper, we aim to provide a protocol; other properties can be studied in the same way as introduced in the paper. The discovery of the paper indicates that although Stable Diffusion has been trained for generative tasks, features extracted from the model can also be applied in a variety of downstream tasks including material segmentation, shadow association, support relation reasoning, 3D reconstruction, depth estimation, occlusion handling (separated parts association), which are the important challenges in terms of physical scene understanding. The paper provides the best time step and layer for different tasks of Stable Diffusion features.

Reviewer comment (Experiments):
This paper lacks of a more complete ablation study, which could provide more insight. For example, how does the number of time steps or layers affect the task? Is it sensitive to these parameters?

Author response:
W3 (Different time steps and layers): As mentioned in Section 4.1 of the paper, we have conducted a grid search in terms of different time steps and layers to find the best time step and layer for each task. We can observe that usually D3 and D2 perform best and the best time step is usually between 0-400 as later time steps might have added too much noise, as in Section 4.2 and Table 2.

Reviewer comment (Experiments):
The author's experiments are limited to feature quality evaluation, but there are no explicit visualization of the latent feature, and there is no example how such evaluation can benefit and help us design the next-generation T2I model, or Text-to-3D models by improving over the feature space.

Author response:
W2 (Visualisation of latent feature): We have updated the t-SNE visualization of the feature vectors in our revision (Section D in the supplementary), where we can observe that the better performance the SVM is getting, the better distribution these vectors are getting in the embedding space, i.e., the positive vectors and negative vectors are distributed apart from each other. In the conclusion of the paper, we conclude that current Stable Diffusion model could understand a variety of physical properties while less performant at the occlusion and material tasks, which indicates the future research could focus on improving these properties of Stable Diffusion by explicitly incorporating these tasks into training.  
Thanks so much for your reply. We are focusing on exploring whether the pre-trained generative Stable Diffusion model could understand the physical properties of the scene, rather than design a better generative model for image/3D generation. Through the thorough experiments, we demonstrate the generative diffusion model could outperform other large pre-trained models on various tasks such as scene geometry, material, shadow, support relation, occlusion and depth. This is significant for the follow-ups to explore the large generative models for understanding tasks in our community.  
Furthermore, we found that the features of Stable Diffusion model perform worse on occlusion and materials. We believe this is a reasonable analysis on why the existing Stable Diffusion model can not simultaneously generate many objects in one scene with complex 3D layout, such as occlusion. We think the future training of Stable Diffusion can explicitly incorporate these task into training, for example, with classifier-guidance of occlusion and material, so that the feature space could also be improved for occlusion and material (Figure 9 in the supplementary).

Reviewer comment (Experiments):
For the Question 6 design on occlusion, I feel it is not appropriate since there are two concepts: 1. belonging to same object; 2. whether there are occlusion. Before we evaluate occlusion, we need to first know how the classifier will work for parts belonging to same object but without occlusion.

Author response:
W3 (Design of occlusion question): We understand the reviewer’s point about asking the question first about whether two parts belong to the same object. We will clarify this in the discussion. It is partially covered already by the co-planar question. In the current question we are following the dataset and challenge proposed by [1], since we want to investigate the view-point dependence of occlusion here (rather than grouping `parts’ into an object, which is a segmentation problem). Segmentation has already been studied for Stable Diffusion.

Reviewer comment (Experiments):
These datasets with GT labels might serve as domain-specific experts, in order to fully capture the learnt image distribution of SD, how do we select these training/test images?

Author response:
Q1 (Selection of train/test images): As mentioned in Section A of the supplementary, the training/testing images are randomly sampled from the original datasets. We are testing on these datasets because they have ground truth annotations for different physical properties.

Reviewer comment (Evaluation):
The paper oversells the approach at the beginning of the paper. It would be better to have the title/abstract/introduction define the constraint of in-filling, as this changes the problem significantly. In general, we see that in-filling is a significantly easier task than generating from scratch, which is more the common use case for Stablediffusion. Therefore, it would be better to constrain the arguments of the paper to be specific to this, as the benchmark would not generalise to unconstrained image generation. (This is why reduced Soundness and Presentation scores as the paper is misleading).

Author response:
We suspect that the reviewer has misunderstood the paper as they stress inpainting as our goal. Our proposed protocol does not involve inpainting at all. We included one figure showing inpainting as a motivational example for the power of stable diffusion. We say in the text that this figure “is an indication that the diffusion network is 3D and physics aware”, and have now clarified in the figure caption of the revision that it is a motivational example. All the other figures do show the properties we evaluate on. However, given the reviewer’s apparent misunderstanding, it is difficult to respond to some of their questions.

Reviewer comment (Experiments):
The use of ScanNet is problematic for plane tests as the segmentation is quite incomplete. Therefore, in a large case the part you are infilling you can see the rest of the wall/floor/surface giving strong context clues. These tests provide limited insights.

Author response:
W2 (Use of ScanNet for plane test): As we are not studying the inpainting model, the masks in Figure 2 are only used to extract the Stable Diffusion feature of the specified region, and it is okay if the mask is smaller than the whole region (the mask can be any sub-region of the whole region).

Reviewer comment (Experiments):
It would have been good to see other models evaluated in the same setting, even older approaches like GANs or VQ-VAE or StableDiffusion derivatives to provide contextual information. While DINO and CLIP provide insights, they are focused on a different task and probably explain the worse performance. A single focus to the paper has a limited setting for arguing a benchmark.

Author response:
W3 (Evaluating other models): As mentioned in the overall comments, our objective was to propose a protocol and use this to evaluate a set of models to compare to Stable Diffusion, but not to comprehensively evaluate all models. We have added an evaluation for a second generative model, VQGAN, and updated the results in our revision, showing that Stable Diffusion still has a superior performance (Table 4). DINO and CLIP models are compared because they are strong self-supervised features and are used extensively for scene understanding tasks.

Reviewer comment (Experiments):
It is unclear why they only ablate across the SVM C parameter and not the Gamma as well. While in a large amount of cases, this has limited effect, it should be justified ideally empirically.

Author response:
W4 (Gamma in SVM): We have mentioned many times in the paper that we are using a linear SVM. This does not have a gamma parameter.

Reviewer comment (Experiments):
The authors argue a complete study in the beginning and then only evaluate DINO and Clip on Material and Support. The claim should either be consistent or the experiments completed.

Author response:
W5 (Grid search for other models): See Reviewer oLXV W3. We have now updated results to have a full grid search for all the foundation models we evaluate on (Table 3,4,5,6,7,8,9 in the revision). Stable Diffusion still outperforms all of them.

Reviewer comment (Evaluation):
Whether they tested human performance on the model? And how they came up with the groundtruth.

Author response:
Q2 (Ground truth for evaluation): On the Ground Truth (GT), as explained in the paper (Section 3.2) we are using standard image benchmark datasets where the GT for these properties is provided. As mentioned in Section B of our supplementary, “the reason why we do not study the generated images directly is there are no annotations about different properties for these synthetic images”. The quality of GT annotations for these real images datasets is of high quality, and they are all from well-established works.

Reviewer comment (Evaluation):
It is not clear to me that the proposed probing method is sufficient in proving that a model understands 3D relations. The biggest assumption made by the paper is that the result of the linear SVM is equal to the actual 3D relation in the image. This is simply not true. First, the linear SVM cannot get 100% on classifying the relations (this number is not mentioned anywhere in the paper). Second, even if the linear SVM gets 100% on the training dataset, it is not guaranteed to work on out of distribution images. Using the result of such a linear probe as a quantitative measure on how well the model understands 3D structure is very unreliable.

Author response:
W1 (SVM training and testing): The train/val sets and test set are disjoint, so the SVM is testing generalization. We agree that more datasets can always be added to test out of (dataset) generalization.

Reviewer comment (Experiments):
The paper seems to claim that stable diffusion can be used as a way to extract features from images and is better than alternatives like OpenCLIP and DINO. However, I’m not sure whether it is a good idea of adding 8 different noises to the image and average the features. It can be very slow since diffusion is a sequential process and multiple diffusion passes are needed, while OpenCLIP and DINO just need a single forward pass on the original image.

Author response:
W2 (8 different noises for Stable Diffusion): We follow DIFT[1] to add 8 different noises to get the more stable feature from Stable Diffusion. We set the batch size to be 8 so we only need a single forward pass.

Reviewer comment (Experiments):
The comparison between other foundation models and stable diffusion in Table 4 is not entirely fair, since stable diffusion uses grid search to find the best layer for the task, but other models use the last layer features. Although in Table 3 the authors mention that the difference between last layer and best layer is about 4% for two of the tasks. I don’t think it is enough to conclude that grid search is not needed for all tasks.

Author response:
W3 (Grid search for other models): We have now updated results to have a full grid search for all the foundation models we evaluate on (Table 3,4,5,6,7,8,9 in the revision). Stable Diffusion still outperforms all of them.

Reviewer comment (Experiments):
Please provide accuracy of the linear SVM on the train/val set for each question.

Author response:
Q1 (Linear SVM train/val accuracy): We will add this to the supplementary of the paper.

Reviewer comment (Experiments):
In the qualitative examples, the inpainting mask is very tight around the desired region (e.g. the shadow of an object). What if a larger region is masked (e.g. the entire ground)? Can stable diffusion still generate shadow in the correct location and direction?

Author response:
Q2 (Larger inpainting mask):  In the example of Figure 1, we cannot mask out the entire ground as we need the shadows of the other objects to indicate the direction of the shadow of the target object. In the revision, we have updated the shadow example of Figure 1 where a larger region is masked out for the inpainting task, but Stable Diffusion still manages to generate shadow in the correct direction.