Paper: Unsupervised Discovery of Object-Centric Neural Fields (ICLR-2024)

Reviewer comment (Experiments):
The main additional contribution of this paper should be disentangling the learning of object intrinsics and extrinsics compared with the previous method uORF. The performance comparisons in the experimental parts demonstrate the effectiveness of the whole pipeline. However, it is not very clear to me which parts contribute most to the performance gain. The object-centric sampling mentioned in Sec. 3.3 seems reasonable and should lead to better rendering performance. But I didn't see any relevant ablation study on this.

Author response:
**Q1. Ablation studies on encoder design, training pipeline, and object-centric sampling.**

**A1.** We provide additional ablation studies in Appendix E.1 following your suggestions. These studies justify the necessity of our three-stage training pipeline, showing that omitting synthetic dataset training stages significantly impairs the model's performance. Additionally, our findings illustrate that the integration of DINO features, while beneficial, requires the concurrent use of a U-Net route for better performance. Ablation studies on object-centric sampling also confirm its role in enhancing background reconstruction. We provide a snapshot of the results below, and the full details are available in Tables 4, and 5 and Figures 12, and 13. 

**Ablation Studies on Encoder Design**
| Method | ARI$\uparrow$ | FG-ARI$\uparrow$ | NV-ARI$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
| uORF | 0.649 | 0.108 | 0.587 | 24.37 | 0.688 | 0.251 |
| uORF-DINO | 0.688 | 0.677 | 0.656 | 25.33 | 0.739 | 0.240 |
| uOCF-DINO | 0.652 | 0.346 | 0.621 | 26.06 | 0.723 | 0.251 |
| uOCF | **0.802** | **0.785** | **0.747** | **28.96** | **0.803** | **0.121** |

**Ablation Studies on Training Pipeline**
| Method | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |
| :--- | :---: | :---: | :---: |
| uORF | 19.23 | 0.602 | 0.336 |
| Omit stage 1&2 | 19.67 | 0.565 | 0.568 |
| Omit stage 2 | 24.52 | 0.769 | 0.157 |
| Omit object-centric sampling | 27.89 | **0.843** | 0.083 |
| uOCF | **28.29** | 0.842 | **0.069** |

As for the most crucial component, we emphasize the integration of position disentanglement and a three-stage training pipeline. The former enables the model to learn object-centric NeRFs and object positions in 3D scenes from single images, while the latter enables generalization to complex real-world scenes. These two components are complementary and indispensable for our method's success.

Reviewer comment (Evaluation):
The authors did not provide a detailed analysis of the limitations or failure cases of the proposed approach, which could limit its applicability in certain scenarios.

Author response:
**Q2. Limitation analysis**

We have added discussions on our method's limitations in Appendix D. The current constraints include limited diversity in object appearance and background complexity in our datasets and challenges in reconstructing foreground objects with complex textures. These limitations, shared with other generalizable NeRF methods, are areas we aim to improve in future work.

Reviewer comment (Reproducibility):
The paper does not provide a clear explanation of the training process and hyperparameters used in the experiments, which could limit its reproducibility and further research.

Author response:
**Q3. Explanation of training configurations**

Details on datasets and training configurations are provided in Appendix C. Additionally, to facilitate reproducibility, we have dedicated a section in our paper and committed to releasing all codes, datasets, models, and detailed instructions upon paper acceptance.

Reviewer comment (Experiments):
How to decide the hyperparameter K in Eq.(1)? How to guarantee the redundant slots do not affect the learning of other slots?

Author response:
**Q4. Hyperparameter $K$ (number of objects)**

While the scenes in our datasets have $K=4$ objects, our method can also reconstruct and discover all objects to scenes with fewer objects. Further details and visual demonstrations are available in Appendix E.6 and Figure 19.

Reviewer comment (Experiments):
Did you test with transparent objects? Would it also work properly?

Author response:
**Q5. Results on transparent objects**

As for transparent objects, we have included new experiments in Appendix E.7, with visual results in Figure 20. Our model ignores transparency due to the absence of transparent objects in its training dataset. However, it still demonstrates reasonable object segmentation and reconstruction capabilities.

Reviewer comment (Evaluation):
Did you test the geometric quality of the learned object representations? Can your method recover accurate shape of the objects?

Author response:
**Q6. Geometric quality of learned object representations**

Recovering object geometric shapes from NeRFs is indeed a challenging problem. NeRF primarily represents objects using density values for rendering, which doesn't explicitly encode object geometry. This can lead to inaccuracies, especially in complex geometries or scenarios with limited view coverage. However, significant advancements have been made in this area. Recent research efforts have focused on enhancing NeRF's capability to reconstruct object geometry more accurately. These include improved sampling strategies [1], integration of geometric constraints [2], and hybrid approaches that blend traditional 3D representations with NeRF [3]. Integrating these advancements into our method is a promising direction for future work.

---

### References
[1] Li et al. "NerfAcc: Efficient Sampling Accelerates NeRFs". In ICCV, 2023.

[2] Rematas et al. "ShaRF: Shape-conditioned Radiance Fields from a Single View". In ICML, 2021.

[3] Yariv et al, "Volume Rendering of Neural Implicit Surfaces". In NeurIPS, 2021.

Reviewer comment (Experiments):
Which module do you think contributes most to the final performance improvement? Could you provide more detailed ablation study on this?

Author response:
**Q1. Ablation studies on encoder design, training pipeline, and object-centric sampling.**

**A1.** We provide additional ablation studies in Appendix E.1 following your suggestions. These studies justify the necessity of our three-stage training pipeline, showing that omitting synthetic dataset training stages significantly impairs the model's performance. Additionally, our findings illustrate that the integration of DINO features, while beneficial, requires the concurrent use of a U-Net route for better performance. Ablation studies on object-centric sampling also confirm its role in enhancing background reconstruction. We provide a snapshot of the results below, and the full details are available in Tables 4, and 5 and Figures 12, and 13. 

**Ablation Studies on Encoder Design**
| Method | ARI$\uparrow$ | FG-ARI$\uparrow$ | NV-ARI$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
| uORF | 0.649 | 0.108 | 0.587 | 24.37 | 0.688 | 0.251 |
| uORF-DINO | 0.688 | 0.677 | 0.656 | 25.33 | 0.739 | 0.240 |
| uOCF-DINO | 0.652 | 0.346 | 0.621 | 26.06 | 0.723 | 0.251 |
| uOCF | **0.802** | **0.785** | **0.747** | **28.96** | **0.803** | **0.121** |

**Ablation Studies on Training Pipeline**
| Method | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |
| :--- | :---: | :---: | :---: |
| uORF | 19.23 | 0.602 | 0.336 |
| Omit stage 1&2 | 19.67 | 0.565 | 0.568 |
| Omit stage 2 | 24.52 | 0.769 | 0.157 |
| Omit object-centric sampling | 27.89 | **0.843** | 0.083 |
| uOCF | **28.29** | 0.842 | **0.069** |

As for the most crucial component, we emphasize the integration of position disentanglement and a three-stage training pipeline. The former enables the model to learn object-centric NeRFs and object positions in 3D scenes from single images, while the latter enables generalization to complex real-world scenes. These two components are complementary and indispensable for our method's success.

Reviewer comment (Presentation):
In sec 3.1, the description of Encoder/Latent Extraction Module seems not matched with the provided Figure 2. It's unclear how the two sets of feature maps f_g and f_l are fed into the second module.

Author response:
Section 3.4 details our approach to integrating local and global features. We utilize $f_g$ as the "$\mathrm{inputs}$" for the latent extraction module and concatenate its outputs with an attention-weighted mean of $f_l$. This results in the final slot latent $ \{ \mathbf{z_i}\}_{i=0}^K$, where attention weights are also derived from the latent extraction module. This design efficiently balances global and local information, maximizing the utility of pre-trained models.

Reviewer comment (Theory):
In the Latent Extraction Module, for each object, its position p_i^{wd} in the world frame will be learned. In this case, what is the world coordinate? Is it predefined for the whole set of multi-view images of every single scene? In this case, given two different input images of the same scene, the same 3D object needs to learn exactly the same 3D location. In this way, the network actually needs to learn a separate coordinate system conditioning on every input image, which seems not that sensible if I understand correctly.

Author response:
Instead of learning the object position in the world coordinates $p_i^{wd}$, our approach first estimates an object's position on the image plane $p_i^{img}$ by the attention-weighted mean over a spatial grid, which is then converted to the world coordinates using the intersection of the ray with the ground plane. A bias term is added to $p_i^{img}$ to adjust for discrepancies in this conversion. Since the position estimation happens in image space instead of 3D space, our approach does not need to learn separate coordinate systems for each image.

Reviewer comment (Theory):
The paper models every object/background by a separate NeRF. How is such a strategy able to deal with the potential under/over segmentation of 3D objects? For example, what if two chairs are grouped into a single NeRF (the center of two chairs may be learned as the combined object center)? What if a single chair is learned as two objects at the very beginning? Why will the entire background structure (usually complex) must be grouped into a single NeRF? In practice, there are typically a variable number of objects in each 3D scene to discover. There could be objects that are visible in input image, but invisible in multi-view images, or otherwise.

Author response:
Under/over-segmentation is a common challenge in existing approaches, often resulting in binding multiple objects into a single foreground slot (missing objects) or all objects into the background slot (blurry object reconstruction), as visualized in Figures 5 and 7. The technical contribution of our work is exactly to help address this challenge. In particular, we introduce the generalizable object-centric prior learning pipeline. This pipeline begins with learning object priors from a synthetic dataset. The learned object priors might include generalizable visual clues such as physical coherence [4], which is critical to unsupervised segmentation [5]. The subsequent stage refines the model's ability to predict object positions and segregate them into individual slots. As shown in Figure 7, our method can effectively separate and reconstruct individual objects in real scenes, even when the object is largely occluded.

However, although our method can tackle object occlusion, it cannot handle invisible objects in the input image, as the model cannot discover them. Reversely, our model can still discover and reconstruct the object when the object is visible in the input image but invisible in the multi-view images. This property is naturally inherited from the volumetric rendering technique utilized in NeRF.

Reviewer comment (Writing):
The paper states that it incorporates two novel techniques: 1) object-centric prior learning and 2) object-centric sampling. However, in page 5, such two techniques are very briefly described by two small paragraphs. As to object-centric prior learning, what types of priors are planned to be learned? Why is it helpful for the subsequent object discovery? As to object-centric sampling, how does it work in detail and why is it important?

Author response:
We have discussed the effect of object-centric prior learning in previous responses. For qualitative results, ablation studies (Appendix E.1) show that omitting synthetic dataset training stages drastically reduces performance. Meanwhile, our object-centric technique enables $4\times$ samples within the same computational budget, significantly enhancing background reconstruction quality. Key results are summarized below, with full details in Tables 4, 5, and Figures 12, 13.

| Method | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |
| :--- | :---: | :---: | :---: |
| uORF | 19.23 | 0.602 | 0.336 |
| Omit stage 1&2 | 19.67 | 0.565 | 0.568 |
| Omit stage 2 | 24.52 | 0.769 | 0.157 |
| Omit object-centric sampling | 27.89 | **0.843** | 0.083 |
| uOCF | **28.29** | 0.842 | **0.069** |

Reviewer comment (Experiments):
For the collected three datasets, how many objects are in each image? From the samples in Figure 4, it seems every image has exactly four objects.

Author response:
While the scenes in our datasets have $K=4$ objects, our method can also reconstruct and discover all objects to scenes with fewer objects. Further details and visual demonstrations are available in Appendix E.5 and Figure 17.

Reviewer comment (Experiments):
As to the two real-world datasets Kitchen-easy/hard, the objects are rather simple in terms of both shape and appearance. It's suggested to evaluate on more complex real-world images such as ScanNet.

Author response:
Given that ScanNet does not offer textured meshes or calibrated images, we use the HM3D dataset [6], a dataset focused on building-scale indoor 3D reconstruction, for evaluating more complex real-world scenes. We conducted our evaluation using zero-shot inference, with detailed results presented in Appendix E.2 and illustrated in Figure 15. Our model, pre-trained on the Room-Texture dataset, demonstrates robustness in discovering and segmenting chair instances even in unseen complex scenes.

Reviewer comment (Evaluation):
For the evaluation metrics of object segmentation, only ARI-related scores are reported. It's suggested to report additional metrics such as AP scores as analyzed by the NeurIPS'22 paper in the community of unsupervised object segmentation.

Author response:
We have adopted the official AP computation code from [7] and included the AP evaluation metric in Appendix E.4, with detailed results in Table 7 (also attached below). Specifically, we consider two kinds of APs: Input view-AP for the input view, and Novel view-AP for novel views. Our method's high AP scores demonstrate its superior scene segmentation capabilities.

| Metric | uORF | QBO | COLF | uOCF |
| :--- | :---: | :---: | :---: | :---: |
| Input view-AP | 0.005 | 0.359 | 0.315 | **0.782** | 
| Novel view-AP | 0.001 | 0.195 | 0.015 | **0.770** |

Reviewer comment (Evaluation):
For the experimental results in Tables 1/2/3, why are the existing methods inferior in both segmentation and novel view synthesis? It's suggested to give more concrete discussions.

Author response:
A key strength of our work lies in its superior reconstruction quality. In contrast, uORF struggles to distinguish between background and foreground elements, particularly in complex datasets such as *Kitchen-Hard* and *Room-Texture*, resulting in low segmentation scores. This difficulty extends to uORF's background NeRF, leading to blurry reconstructions due to its inability to effectively focus on background elements.

Our approach builds upon uORF but introduces a critical enhancement: the disentanglement of object positions. This advancement not only enhances systematic generalization but also facilitates the application of techniques like object-centric prior learning and sampling. Specifically, object-centric prior learning allows better disentanglement of background, and object-centric sampling allows for $4\times$ samples with the same amount of computation. This results in a significant improvement in reconstruction quality for both background environment and foreground objects, setting our method apart from uORF's limitations.

Reviewer comment (Experiments):
There is a lack of ablation studies to analyze the effectiveness of the proposed components.

Author response:
Additional ablation studies (Appendix E.1) reveal that while substituting DINO for a shallow U-Net encoder improves performance, it does not inherently overcome its limitations. Specifically, Table 4 (also attached below) illustrates how uORF-DINO and uORF-DR (uORF with our dual-route encoder) improve upon the standard uORF, yet still fail to correctly disentangle foreground and background elements. Note that uORF-DR binds all foreground objects to the background, leading to an ARI score of zero. Besides, uOCF-DINO, which drops the U-Net route and utilizes the DINO encoder only, and uOCF-IM, which substitutes DINO's intermediate layer features for the shallow encoder in uOCF, both achieve inferior performance than uOCF. This reaffirms the superiority of our encoder design.

| Method | ARI$\uparrow$ | FG-ARI$\uparrow$ | NV-ARI$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
| uORF | 0.649 | 0.108 | 0.587 | 24.37 | 0.688 | 0.251 |
| uORF-DINO | 0.688 | 0.677 | 0.656 | 25.33 | 0.739 | 0.240 |
| uORF-DR | 0 | 0 | 0 | 25.38 | 0.698 | 0.322 |
| uOCF-DINO | 0.652 | 0.346 | 0.621 | 26.06 | 0.723 | 0.251 |
| uOCF-IM | **0.806** | 0.749 | **0.752** | 27.77 | 0.753 | 0.182 |
| uOCF | 0.802 | **0.785** | 0.747 | **28.96** | **0.803** | **0.121** |

Reviewer comment (Novelty):
In page 2, the claimed first contribution seems not very meaningful at all.

Author response:
Following your advice, we have revised the contributions for clarity and conciseness, updated the mathematical notations for positional encodings, and added discussions about the ONeRF paper.

Reviewer comment (Writing):
For equations 1/2/3/4/5, it's suggested to use mathematical symbols rather than English words. It's a bit hard to track the meanings.

Author response:
Following your advice, we have revised the contributions for clarity and conciseness, updated the mathematical notations for positional encodings, and added discussions about the ONeRF paper.

Reviewer comment (Novelty):
The work ONeRF should be discussed, as it also uses NeRF for unsupervised 3D object discovery.

Author response:
Following your advice, we have revised the contributions for clarity and conciseness, updated the mathematical notations for positional encodings, and added discussions about the ONeRF paper.

Reviewer comment (Experiments):
The major weakness of the paper is the lack of ablations performed. The paper mentions various design choices, however, there are no ablations in the main paper justifying any of them.

Author response:
We appreciate your suggestion regarding the inclusion of additional ablation studies. In Appendix E.1, we have expanded our analysis to emphasize the critical role of our three-stage training pipeline in achieving robust generalization to complex real-world scenes. The ablation results in Table 5 (also attached below) and Figure 13, demonstrate how omitting the initial synthetic dataset training stages significantly compromises the model's capability to distinguish foreground from background elements. These results confirm the indispensability of each stage in our pipeline for achieving the model's full potential.

| Method | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |
| :--- | :---: | :---: | :---: |
| uORF | 19.23 | 0.602 | 0.336 |
| Omit stage 1&2 | 19.67 | 0.565 | 0.568 |
| Omit stage 2 | 24.52 | 0.769 | 0.157 |
| Omit object-centric sampling | 27.89 | **0.843** | 0.083 |
| uOCF | **28.29** | 0.842 | **0.069** |

Reviewer comment (Experiments):
The paper mentions that 3 stage training helps in getting a better performance, however there is no ablation to justify that. What if you skip stage 2 and directly train on stage 3 after learning object priors using stage 1? How much will the performance degrade? How much does stage 2 training actually help here?

Author response:
We appreciate your suggestion regarding the inclusion of additional ablation studies. In Appendix E.1, we have expanded our analysis to emphasize the critical role of our three-stage training pipeline in achieving robust generalization to complex real-world scenes. The ablation results in Table 5 (also attached below) and Figure 13, demonstrate how omitting the initial synthetic dataset training stages significantly compromises the model's capability to distinguish foreground from background elements. These results confirm the indispensability of each stage in our pipeline for achieving the model's full potential.

| Method | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |
| :--- | :---: | :---: | :---: |
| uORF | 19.23 | 0.602 | 0.336 |
| Omit stage 1&2 | 19.67 | 0.565 | 0.568 |
| Omit stage 2 | 24.52 | 0.769 | 0.157 |
| Omit object-centric sampling | 27.89 | **0.843** | 0.083 |
| uOCF | **28.29** | 0.842 | **0.069** |

Reviewer comment (Experiments):
The authors mention learning a global representation using ViT and a local representation using shallow UNet, but there is no ablation justifying the use of two separate networks for these representations. Why not use some intermediate layer (or the spatial features) from ViT as local features? What is the intuition behind using a shallow UNet? Is it to prevent features from different objects entangling with each other? The authors should provide such ablations or intuitions in the main paper.

Author response:
Following your suggestions, we conducted ablation studies on the encoder design in Appendix E.1. The results in Table 4 and Figure 12 indicate that while integrating DINO features enhances performance, the parallel maintenance of a U-Net route is crucial for better performance. The comparative results on the Room-Texture dataset illustrate this point:

| Method | ARI$\uparrow$ | FG-ARI$\uparrow$ | NV-ARI$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
| uORF | 0.649 | 0.108 | 0.587 | 24.37 | 0.688 | 0.251 |
| uORF-DINO | 0.688 | 0.677 | 0.656 | 25.33 | 0.739 | 0.240 |
| uOCF-DINO | 0.652 | 0.346 | 0.621 | 26.06 | 0.723 | 0.251 |
| uOCF-IM | **0.806** | 0.749 | **0.752** | 27.77 | 0.753 | 0.182 |
| uOCF | 0.802 | **0.785** | 0.747 | **28.96** | **0.803** | **0.121** |

Specifically, while uORF-DINO improves over standard uORF by replacing the shallow U-Net encoder with DINO, it still fails to correctly disentangle foreground and background elements. Besides, uOCF-DINO, which drops the U-Net route and utilizes the DINO encoder only, and uOCF-IM, which substitutes DINO's intermediate layer features for the shallow encoder in uOCF, both achieve inferior performance than uOCF. This reaffirms the superiority of our encoder design.

Reviewer comment (Novelty):
This model is essentially an integration of prior research efforts, with key components such as slot-attention, DINO, and object-compositional NeRF having been individually introduced in previous works. Consequently, the training pipeline itself lacks a significant degree of novelty.

Author response:
**Q8. Novelty Concerns**

We argue that our core contribution is proposing object-centric modeling, which addresses a fundamental bottleneck in generalizing 3D object discovery to more complex real scenes than synthetic scenes. In particular, existing 3D object discovery methods represent objects in the viewer's coordinate frame, entangling camera extrinsics with object intrinsics. This entanglement prevents them from effectively learning object priors and generalizing to complex scenes. Our object-centric modeling allows object-centric prior learning instantiated by our 3-stage training pipeline, which shows promising results in generalization, even in a zero-shot setting.

Therefore, our innovation mainly lies in object-centric modeling which further enables prior learning and object-centric sampling.

Regarding the integration of DINO, our new ablation studies (Appendix E.1, Table 4) show that a mere substitution of DINO in existing frameworks cannot overcome their limitations. The three-stage training pipeline, particularly the initial stages on synthetic datasets, is essential for our model’s generalization, as evidenced by the substantial performance drop when omitted (Appendix E.1, Table 5). Moreover, the object prior obtained on synthetic datasets are agnostic to the real-world dataset's object categories, and thus can even be learned from simple CLEVR shapes. These results underscore the effectiveness of the proposed object-centric modeling.

Reviewer comment (Novelty):
I have major concerns regarding the utilization of a pre-trained DINO encoder. DINO is already proficient at extracting meaningful and somewhat object-aware features from images, leading to the possibility that the model merely aligns these features with latent variables. This raises questions about how much the proposed pipeline genuinely contributes to object discovery.

Author response:
**Q1. Utilization of DINO encoder**

We note that simply substituting DINO for the shallow U-Net encoder in existing models does not inherently overcome their limitations. To see this, we have conducted comprehensive ablation studies detailed in Appendix E.1. Specifically, Table 4 (also attached below) illustrates how uORF-DINO and uORF-DR (uORF with our dual-route encoder) improves upon the standard uORF, yet still fails to correctly disentangle foreground and background elements. Note that uORF-DR binds all foreground objects to the background, leading to an ARI score of zero. Besides, Figure 12 provides visual comparisons to demonstrate these distinctions in performance.

| Method | ARI$\uparrow$ | FG-ARI$\uparrow$ | NV-ARI$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
| uORF | 0.649 | 0.108 | 0.587 | 24.37 | 0.688 | 0.251 |
| uORF-DINO | 0.688 | 0.677 | 0.656 | 25.33 | 0.739 | 0.240 |
| uORF-DR | 0 | 0 | 0 | 25.38 | 0.698 | 0.322 |
| uOCF | **0.802** | **0.785** | **0.747** | **28.96** | **0.803** | **0.121** |

Reviewer comment (Experiments):
The "l_pos" loss, which calculates the positional disparity between object poses from different viewing directions, relies on the fact that, during the initial training stage, there is only one object per scene, eliminating object-matching challenges. However, strictly speaking, the use of a single-object dataset disqualifies the proposed pipeline as an unsupervised model while making the learning problem considerably less challenging. To apply this method to a general dataset, one must manually select scenes containing only one object, a process akin to labeling. In other words, the training pipeline appears tailored specifically for the proposed dataset. While the author presents object-centric prior learning as a contribution, it might be perceived more as a limitation.

Author response:
**Q2. Utilization of single-object datasets**

We clarify that in the first stage of training, our model is trained on a single-object synthetic dataset (which can be easily synthesized) to learn general object priors **agnostic to** the real-world dataset's object categories. Therefore, it is **not** akin to the manual process. For instance, we trained our model on synthetic chairs and successfully applied the learned priors to real-world kitchen datasets. We further validate this generalizability using a very simple CLEVR dataset (colored primitives like cubes and spheres) instead of the chair dataset for object prior learning. We added the results in Table 5 and Figure 12 in Appendix E.1. We also attach the table below. 

| Method | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |
| :--- | :---: | :---: | :---: |
| uOCF (adapt from CLEVR) | 27.32 | 0.833 | 0.092 |
| uOCF (adapt from chairs) | 28.29 | 0.842 | 0.069 |

Reviewer comment (Experiments):
The fixed number of objects in each scene restricts the method, while most object-centric learning frameworks merely set a maximum number of objects in the scenes.

Author response:
**Q5. Fixed number of objects in the scene**

While the scenes in our datasets have $K=4$ objects, our method can also reconstruct and discover all objects to scenes with fewer objects. Further details and visual demonstrations are available in Appendix E.6 and Figure 19.

---

Experiment configuration: We render a new synthetic dataset with each scene containing 2-4 chair instances. Real scenes still include exactly 4 objects. We test both $K=4$ and $K=5$. The results are shown in Tables 8, 9 (also attached below), and Figure 18. 

After using a training set consisting of a randomized number of objects, we observe that empty slots now appear (please see Figure 18), overcoming the over-segmentation problem. Besides, we note that our method does incorporate a specific design to prevent over-segmentation. Firstly, the slot attention mechanism enforces areas of similar features to bind to the same slot. Moreover, we add slot-specific positional encoding to the keys and values to let the slot latents emphasize local information, thus preventing parts of different instances from binding together, which is a common problem in previous methods (Figures 12, 23, also ObSuRF in Figure 4 in [Sajjadi et al.]).

[Sajjadi et.al.] M. Sajjadi et.al., *Object Scene Representation Transformer*. NeurIPS'22.

Qualitative Results on Room-Texture (15 epochs, 75000 iterations)
| Method | ARI$\uparrow$ | FG-ARI$\uparrow$ | NV-ARI$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
| K=4, n_obj=4 | 0.819 | 0.643 | 0.743 | 28.68 | 0.803 | 0.139 |
| K=4, n_obj$\in${2,3,4} | 0.828 | 0.743 | 0.756 | 30.11 | 0.831 | 0.112
| K=5, n_obj$\in${2,3,4} | 0.819 | 0.559 | 0.769 | 28.72 | 0.805 | 0.132 

Qualitative Results on Kitchen-Hard (150 epochs, 48600 iterations)
| Method | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |
| :--- | :---: | :---: | :---: |
| K=4, n_obj=4 | 27.36 | 0.820 | 0.140 |
| K=4, n_obj$\in${2,3,4}  | 27.10 | 0.825 | 0.131 |
| K=5, n_obj$\in${2,3,4}  | 28.26 | 0.837 | 0.120 |

As shown in these tables, uOCF performs reasonably well when trained on scenes with $\leq K$ objects, justifying that the outstanding performance of uOCF in real-world scenes is not because of the identical number of objects between stages 2 and stage 3. In other words, applying the object prior learned from synthetic scenes to real-world scenes neither requires two datasets to have the same number of objects nor $K$ to equal the number of objects in the scene. We can still assume a shared maximum number of objects of the two datasets, similar to previous unsupervised object discovery literature.

Reviewer comment (Writing):
The limitations of the work are not adequately discussed or disclosed.

Author response:
**Q4. Limitation analysis**

We have added discussions on our method's limitations in Appendix D. The current constraints include limited diversity in object appearance and background complexity in our datasets and challenges in reconstructing foreground objects with complex textures. These limitations, shared with other generalizable NeRF methods, are areas we aim to improve in future work.