Paper: OpenNeRF: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Views (ICLR-2024)

Reviewer comment (Novelty):
Many of the details of the paper were presented in previous works (e.g. per-pixel feature supervision was presented in NeRF-SOS fo DINO), and a big part of the novelty here is in the details of applying them to 3D segmentation.

Author response:
It is correct that details of this work have been addressed in previous research, for example [Kerr at al. 2023, Fan et al. 2022, Kobayashi et al. 2022, Zhi et al. 2021] etc. Given this, we appreciate your acknowledgment that the devil lies in the details, and that our novelty includes the combination of these distributed efforts into a single method for open-set 3D segmentation. On top of that, we are the first to directly lift pixel-level CLIP features to 3D within the scope of neural radiance fields. Further, as stressed earlier, a major contribution is how to leverage these CLIP features within open set 3D segmentation. To this end, we presented a novel confidence driven inference strategy and established a rigorous evaluation setup for the semantic segmentation task, demonstrating that our simpler yet more intuitive and faster setup leads to superior results (e.g., when compared to the more complex baseline of LeRF which requires multi-scale feature computation and additional DINO regularization terms).

Reviewer comment (Experiments):
While the Replica dataset may be challenging for long-tailed settings, it would be great to evaluate on larger-scaled datasets and outdoor benchmarks such as Matterport3D, ScanNet, and nuScenes.

Author response:
__Experiments on Larger-Scale Datasets__   
We provide additional results on the ARKitScenes [Baruch et al. NeurIPS’21] dataset in the attached supplementary material. ARKitScenes is a large-scale, real-world dataset providing posed RGB-D video sequences from an iPhone/iPad device along with 3D reconstructions. We focus on long-tail, in the wild objects that are small and typically hard to detect and segment. As explained in LERF, we found ScanNet to be challenging to optimize high-quality NeRFs due to limited views and poor poses. However, more recent datasets, such as ARKitScenes, do not exhibit these problems. In fact, as shown in Fig.7, the posed RGB-D recordings from an iPhone can directly be used to optimize our NeRF representation.

Reviewer comment (Reproducibility):
The uncertainty map in Figure 3 indicates that the door and cabinet are the most uncertain points, while the generated novel pose in Figure 2 seems to focus primarily on the table. Given high uncertainty points, how are the camera views sampled?

Author response:
__Confidence Estimation and Novel Camera View Selection__  
The reviewer comments on the uncertainty visualized in Fig.3 and the selected camera poses shown in Fig.2, and raises the question why most of the selected camera poses (yellow) are around the table in the center of the scene, instead of the door or cabinet which exhibit larger uncertainty. We agree with you that the illustrations is a bit misleading;
instead of the camera pose, we should visualize the camera frustum to show more clearly what is visible from the selected camera. Indeed, the current illustration might give the impression that the novel camera poses around the table are also looking at the table. However, this is not the case: what the cameras are looking at is (i.e., the camera target) is selected based on the estimated uncertainty. The actual camera position is placed at a distance from the target (to make sure the object and its context is visible) while making sure that the camera itself is not colliding with the scene geometry (see Sec.3.3 for details).

Reviewer comment (Evaluation):
Although there is a positive correlation between per-point uncertainty and per-point error, the heat maps appear to emphasize all areas other than the wall and floor. It is also intriguing that the peaks of uncertainties don't correspond to object boundaries, but rather the entire object. It would be better to demonstrate more samples to showcase these findings.

Author response:
__Object Boundaries vs. Entire Objects__  
The reviewer is curious why entire objects instead of object boundaries are emphasized in the uncertainty visualization (Fig.3). Uncertainty at object boundaries is an effect that is typically observed when visualizing the entropy of semantic-class probability distributions from a trained model. This means that the model is confident about the object label, but less sure about the object mask boundaries. In such a setup, the model acts as a regularization over the training data.

Instead, Fig.3 shows the variance over open-set features computed independently for each view. These features originate from OpenSeg, an approach that provides pixel-aligned CLIP features based on a segmentation of the input image. When computing the uncertainty, there is no regularization across views, such that these features can disagree significantly. This explains why entire objects are highlighted in the heatmap instead of only object boundaries.

Reviewer comment (Evaluation):
The performance drop when rendering novel views from random positions suggests that the approach is sensitive to the quality and meaningful context of the additional views.

Author response:
__Baseline Experiment__  
Finally, the reviewer mentions the performance drop in one of the baseline experiments, when rendering novel views from *random* positions (Sec.4.3, “Impact of Novel Views”). In the above review, this aspect is listed as a weakness of our method. However, only to clarify, this is not our method, but merely a baseline experiment. In fact, this analysis shows the benefit of our proposed uncertainty-guided camera pose selection over  randomly selecting poses. We will make this clearer in the paper.

Reviewer comment (Evaluation):
Have you considered the scalability and computational efficiency of OpenNeRF? How does it perform in terms of runtime and memory requirements compared to other methods?

Author response:
__Runtime and Memory__  
The reviewer is asking about computational requirements of our method compared to other models. We measure the runtime and memory requirements for our model and compare it to LERF. For a fair comparison, we report all measurements on the same hardware and settings (image resolution, feature backbone, etc.).
The table below shows the results:

| Method 	| Model Parameters	| Avg.Runtime per Frame  | mIoU |
| --- | --- | --- | --- |
| LERF 		| 234 million 		| 22.6			| 10.5 |
| OpenNeRF	|   84 million 		| 3.4 		 	| 20.4 |

We look at the number of model parameters and the average runtime to perform a query. Our model relies on approx. 3 times fewer parameters than LERF while being 6-7 times faster and performing 2 times better on segmentation. These significant differences are largely due to our less complex pipeline: our model does not require the additional head for DINO regularization, our CLIP branch is significantly smaller than LERFs which implements a multi-scale feature field. In terms of runtime, for each prompt, LERF needs to compute features at multiple scales to select the best one, and additionally performs negative-prompt  to suppress background noise. We also integrated negative prompts into our model, but did not find them to be helpful in terms of performance. We are grateful for your input and will add this table to the updated version of the paper.

Reviewer comment (Novelty):
Though the uncertainty measure is sound, I wonder if this improves LeRF and OpenScene's performance as well. A similar measure was introduced in Semantic-NeRF and the authors didn't highlight the difference between their formulation and Semantic NeRF's formulation.

Author response:
2) Second, beyond the actual NeRF model as scene representation, a key aspect of the work is the demonstrated use of rendered novel views, based on the proposed uncertainty estimation, as a promising approach to extract refined open-set features. In fact, the reviewer pointed out an additional interesting experiment, to analyze the performance of baselines models using the proposed novel-view rendering technique. We integrated it into the LERF pipeline as shown in Table 2 below. Interestingly, the relative performance boost due to the novel views appears even more significant for LERF.

| Method | LERF| + NovelViews | OpenNeRF | + NovelViews |
| --- | --- | --- | --- | --- |
| mIoU | 10.4 | 12.1 | 19.4| 20.4 |

_Table 2: Effect of adding uncertainty-based novel-views to LERF and OpenNeRF (metric is IoU)._

Since this mechanism depends on the novel-view rendering capabilities of NeRF, it can be easily integrated into NeRF-based methods such as LERF. However, it is a bit harder for OpenScene which does not support rendering of novel views. Nevertheless, one could imagine an extension of OpenScene where the provided textured meshes are rendered from novel camera poses based on our uncertainty measurement. This would then require the regeneration of the image-based training data as well as training/fine-tuning the 3D sparse convolutional Minkowski networks to obtain the improved 3D open-set features.

__Comparison with SemanticNERF’s Uncertainty Formulation__  
The reviewer asks for a comparison of our confidence estimate (Sec.3.3) and a related uncertainty formulation in SemanticNeRF. Thanks a lot for this question, this aspect is less highlighted in the paper but nevertheless important.  
In our work, we compute the _generalized variance_ over high-dimensional OpenSeg/CLIP features projected to a single 3D point. SemanticNeRF computes the _entropy_ of the per-semantic-class probability distribution for each 2D pixel.  Both metrics are used to quantify uncertainty, however what they measure is different:
- The entropy in SemanticNeRF measures the confidence of the semantic class prediction. Ideally, there would only be one peak for a single class in the predicted distribution. However, if the semantic prediction is uncertain, the probabilities will be spread out over multiple classes, resulting in larger entropy. SemanticNeRF uses this measure to analyze the effect of noisy and sparse supervision (Fig.5 and Fig.6 in SemanticNeRF).
- Since our OpenNeRF (and also LERF) operates in the open-vocabulary setting, we cannot assume to know all existing semantic classes in advance, i.e., we cannot compute the entropy over class-probabilities. However, we can measure the variance of the open-set features in the high-dimensional space. Ideally, for each point in 3D space the projected open-set features would be equal. The more they disagree, the more they are distributed around their mean. As a measure for uncertainty, we therefore compute the multi-dimensional variance, which is summarized in a single scalar by the generalized variance used in this work. We use this single scalar to select novel camera views.

Unlike SemanticNeRF which computes the entropy only for analyses, in our work, the generalized variance guides the novel-view selection mechanism used to improve the open-set features.

Reviewer comment (Novelty):
This looks like an incremental work that extends LeRF by using a different encoder backbone which is very straightforward to implement. Can the authors justify it with really good segmentation performance on long tail in-the-wild queries etc? I didn't see that comparison.

Author response:
__Additional “In-the-wild” Examples__  
The reviewer would like to see additional results from real-world scenes recorded in-the-wild. Unlike Replica,  real-world recordings typically do not have perfect viewpoint annotations which can make training of NeRFs harder. In the __attached supplementary we show additional “in-the-wild” real-world results__ on the ARKitScenes dataset which consists of iPhone RGB-D recordings with imperfect poses in natural environments. Specifically, we tried to pick “long-tail” objects, such as “door handle”, “remote control” etc. that are hard to localize and segment.

Reviewer comment (Experiments):
Not many in-the-wild examples could be seen in the paper. Replica dataset is easier since we have perfect viewpoint annotations. I wonder if the performance stays steady or breaks for more in the wild examples where there are imperfect viewpoint annotations.

Author response:
__Additional “In-the-wild” Examples__  
The reviewer would like to see additional results from real-world scenes recorded in-the-wild. Unlike Replica,  real-world recordings typically do not have perfect viewpoint annotations which can make training of NeRFs harder. In the __attached supplementary we show additional “in-the-wild” real-world results__ on the ARKitScenes dataset which consists of iPhone RGB-D recordings with imperfect poses in natural environments. Specifically, we tried to pick “long-tail” objects, such as “door handle”, “remote control” etc. that are hard to localize and segment.

Reviewer comment (Experiments):
Does uncertainty measure really help? What if the initial NeRFs are not that good? Features tend to break for those viewpoints/areas. Do the authors have improved results in those cases?

Author response:
__Dependence on NeRF quality__  
The reviewer wonders if our mechanism based on novel-views still works if the rendering quality of the NeRF deteriorates. As correctly pointed out, our method strongly depends on the quality of the rendered novel views. In case of low-quality novel-views, the extracted open-set features will naturally not be helpful. This is a clear limitation of our method that we will further discuss in the paper. A similar observation is made in LERF, which equally depends on high-quality views to train a NeRF.

Reviewer comment (Experiments):
I would have liked to see a thorough comparison with LeRF and what technical improvement the work brings (in addition to changing the pre-trained backbone), a comparison with Semantic-NeRF's uncertainty formulation, more in-the-wild examples, and examples where the quality of NeRF is not that good to see whether the semantic features break or whether uncertainty helps in those areas.

Author response:
__Comparison with LERF__  
The reviewer asked for a detailed comparison to the LERF baseline as shown in the paper. Below we highlight the most important aspects:

1) First, OpenNeRF proposes an overall *simpler* model (with fewer components that are easier to tune) which results in notably faster and improved scene understanding performance (see Table 1 below). In particular, the key differences in the model are:
- The usage of pixel-aligned open-set features (from OpenSeg) as opposed to per-image features (from CLIP). This results in better localized open-set features as evidenced by our improved segmentation masks. 
- A direct consequence is that we do not need complex multi-scale mechanisms for feature representation as in LERF. Therefore our CLIP head is significantly smaller, since it does not need to represent a multi-scale feature field. Importantly, during inference, LERF requires a rendering of each scale, which linearly slows down the querying process by the number of scales. This effect is further emphasized by the negative prompts proposed in LERF to suppress background noise, which did not help seem to bring improvements in our setup.
- Finally, our model does not depend on additional regularization terms; LERF requires an additional head for DINO regularization to keep the features consistent. In our setting, DINO regularization did not make a significant difference due to the pixel-aligned features.
Overall, our simplified design avoids the representation of multi-scale CLIP and DINO features, which results in an overall slimmer and faster model.  
| Method 	| Model Parameters	| Avg.Runtime per Frame  | mIoU |
| --- | --- | --- | --- |
| LERF 		| 234 million 		| 22.6			| 10.5 |
| OpenNeRF	|   84 million 		| 3.4 		 	| 20.4 |

_Table1: Our model relies on approx. 3 times fewer parameters than LERF while being 6-7 times faster and performing 2 times better on segmentation._

2) Second, beyond the actual NeRF model as scene representation, a key aspect of the work is the demonstrated use of rendered novel views, based on the proposed uncertainty estimation, as a promising approach to extract refined open-set features. In fact, the reviewer pointed out an additional interesting experiment, to analyze the performance of baselines models using the proposed novel-view rendering technique. We integrated it into the LERF pipeline as shown in Table 2 below. Interestingly, the relative performance boost due to the novel views appears even more significant for LERF.

| Method | LERF| + NovelViews | OpenNeRF | + NovelViews |
| --- | --- | --- | --- | --- |
| mIoU | 10.4 | 12.1 | 19.4| 20.4 |

_Table 2: Effect of adding uncertainty-based novel-views to LERF and OpenNeRF (metric is IoU)._

Since this mechanism depends on the novel-view rendering capabilities of NeRF, it can be easily integrated into NeRF-based methods such as LERF. However, it is a bit harder for OpenScene which does not support rendering of novel views. Nevertheless, one could imagine an extension of OpenScene where the provided textured meshes are rendered from novel camera poses based on our uncertainty measurement. This would then require the regeneration of the image-based training data as well as training/fine-tuning the 3D sparse convolutional Minkowski networks to obtain the improved 3D open-set features.

__Comparison with SemanticNERF’s Uncertainty Formulation__  
The reviewer asks for a comparison of our confidence estimate (Sec.3.3) and a related uncertainty formulation in SemanticNeRF. Thanks a lot for this question, this aspect is less highlighted in the paper but nevertheless important.  
In our work, we compute the _generalized variance_ over high-dimensional OpenSeg/CLIP features projected to a single 3D point. SemanticNeRF computes the _entropy_ of the per-semantic-class probability distribution for each 2D pixel.  Both metrics are used to quantify uncertainty, however what they measure is different:
- The entropy in SemanticNeRF measures the confidence of the semantic class prediction. Ideally, there would only be one peak for a single class in the predicted distribution. However, if the semantic prediction is uncertain, the probabilities will be spread out over multiple classes, resulting in larger entropy. SemanticNeRF uses this measure to analyze the effect of noisy and sparse supervision (Fig.5 and Fig.6 in SemanticNeRF).
- Since our OpenNeRF (and also LERF) operates in the open-vocabulary setting, we cannot assume to know all existing semantic classes in advance, i.e., we cannot compute the entropy over class-probabilities. However, we can measure the variance of the open-set features in the high-dimensional space. Ideally, for each point in 3D space the projected open-set features would be equal. The more they disagree, the more they are distributed around their mean. As a measure for uncertainty, we therefore compute the multi-dimensional variance, which is summarized in a single scalar by the generalized variance used in this work. We use this single scalar to select novel camera views.

Unlike SemanticNeRF which computes the entropy only for analyses, in our work, the generalized variance guides the novel-view selection mechanism used to improve the open-set features.

__Additional “In-the-wild” Examples__  
The reviewer would like to see additional results from real-world scenes recorded in-the-wild. Unlike Replica,  real-world recordings typically do not have perfect viewpoint annotations which can make training of NeRFs harder. In the __attached supplementary we show additional “in-the-wild” real-world results__ on the ARKitScenes dataset which consists of iPhone RGB-D recordings with imperfect poses in natural environments. Specifically, we tried to pick “long-tail” objects, such as “door handle”, “remote control” etc. that are hard to localize and segment.

__Dependence on NeRF quality__  
The reviewer wonders if our mechanism based on novel-views still works if the rendering quality of the NeRF deteriorates. As correctly pointed out, our method strongly depends on the quality of the rendered novel views. In case of low-quality novel-views, the extracted open-set features will naturally not be helpful. This is a clear limitation of our method that we will further discuss in the paper. A similar observation is made in LERF, which equally depends on high-quality views to train a NeRF.