Decomposing NeRF for Editing via Feature Field Distillation
Abstract
Emerging neural radiance fields (NeRF) are a promising scene representation for computer graphics, enabling high-quality 3D reconstruction and novel view synthesis from image observations.
However, editing a scene represented by a
NeRF is challenging, as the underlying connectionist representations such as MLPs or voxel grids are not object-centric or compositional.
In particular, it has been difficult to selectively edit specific regions or objects.
In this work, we tackle the problem of semantic scene decomposition of NeRFs to enable query-based local editing of the represented 3D scenes.
We propose to distill the knowledge of off-the-shelf, supervised and self-supervised 2D image feature extractors such as CLIP-LSeg or DINO into a 3D feature field optimized in parallel to the radiance field.
Given a user-specified query of various modalities such as text, an image patch, or a point-and-click selection, 3D feature fields semantically decompose 3D space without the need for re-training and enable us to semantically select and edit regions in the radiance field.
Our experiments validate that the distilled feature fields can transfer recent progress in 2D vision and language foundation models to 3D scene representations, enabling convincing 3D segmentation and selective editing of emerging neural graphics representations.
Introduction
Emerging neural implicit representations or neural fields have been shown to be a promising approach for representing a variety of signals [82, 53, 65, 106, 56].
In particular, they play an important role in 3D scene reconstruction and novel view synthesis from a limited number of context images.
Neural radiance fields (NeRF) [56] enabled the recovery of a continuous volume density and radiance field from a limited number of observations, producing high-quality images from arbitrary views via volume rendering with promising applications in computer graphics.
However, editing a scene reconstructed by NeRF is non-obvious because the scene is not object-centric and is implicitly encoded in the weights of a connectionist representation such as an MLP [56] or a voxelgrid [23].
Although we can transform the scene in input or output space or via optimization-based editing [37, 97], this does not enable selective object-centric or semantic , local edits, such as moving a single object.
Prior work has addressed this challenge via coordinate-level, semantic decompositions which allow to selectively move, deform, paint, or optimize parts of a NeRF, but relies on costly annotation of instance segmentations and training of instance-specific networks [104].
While this can be alleviated with pre-trained segmentation models [25, 41], such models require pre-defined closed label sets and domains (e.g., traffic scenes), limiting decomposition and editing.
Local editing of NeRFs ideally requires an efficient, open-set method for coordinate-level decomposition.
In this work, we present distilled feature fields (DFFs), a novel approach to query-based scene decomposition for local, interactive editing of NeRFs.
We focus on 3D neural feature fields, which map every 3D coordinate to a semantic feature descriptor of that coordinate.
Conditioned on a user query such as a text or image patch, this 3D feature field can compute a decomposition of a scene without re-training.
We train a scene-specific DFF via teacher-student distillation [34], using supervision from feature encoders pre-trained on the image domain.
Unlike the domain of 3D scenes, the image domain boasts massive high-quality datasets and abundant prior work on self-supervised and supervised training of effective feature extraction models.
Notably, recently proposed transformer-based models [96, 22] have demonstrated impressive capabilities across various vision- and text-based tasks (e.g., CLIP [69], LSeg [44], DINO [12]).
Such feature spaces capture the semantic properties of regions and make it possible to correspond and segment them well by text, image queries, or clustering.
We employ these models as teacher networks and distill them into 3D feature fields via volume rendering.
The trained feature field enables us to semantically select and edit specific regions in 3D NeRF scenes and render multi-view consistent images from the locally edited scenes.
In extensive experiments, we investigate the applications of neural feature fields with two different pre-trained teacher networks, (1) LSeg [44], a CLIP-inspired language-driven semantic segmentation network, and (2) DINO [12, 3], a self-supervised network aware of various object boundaries and correspondences.
LSeg and DINO features allow us to select 3D regions by a simple text query or an image patch, respectively.
We first quantitatively demonstrate that LSeg-based DFFs with label queries can have high 3D segmentation performance compared with an existing point-cloud based 3D segmentation baseline trained on ScanNet [20], a supervised point-cloud dataset.
We then demonstrate a variety of 3D appearance and geometry edits across real-world NeRF scenes with no annotations of segmentation; and show that we may edit regions with a single query of text, image, pixel, or cluster choice.
Related Work
Neural Implicit Representations.
Neural implicit representations or neural fields have recently advanced neural processing for 3D data and multi-view 2D images [82, 53, 65, 106, 56].
For a review of this emerging space we point the reader to the reports by Kato et al.
[39], Tewari et al.
[90], and Xie et al.
In particular, a neural radiance field (NeRF) can be fitted to a set of posed 2D images and maps a 3D point coordinate and a view direction to RGB color and density.
When observations are limited, NeRF often overfits and fails to synthesize novel views with correct geometry and appearance.
Pre-trained vision models have been used for regularizing NeRF via flows [62], multi-view consistency [35], perceptual loss [110], or depth estimation [100, 77].
Some pre-trained models operate not only in the visual world but also in other modalities such as language.
The recently proposed CLIP model [69] has demonstrated impressive performance in image-and-text alignment, with strong generalization to various textual and visual concepts.
Wang et al.
[97], Jain et al. [36], and Poole et al. [68] use CLIP or Imagen [79] to edit or generate a single-object NeRF with a text prompt query by optimizing the NeRF parameters to generate images matched with the text.
While such methods are promising, they do not enable accurate selective editing of specific scene regions.
For example, the prompt “yellow flowers” may affect unintended scene regions, such as the leaves of a plant.
Our proposed decomposition method leverages pre-trained foundation models to enable selective editing of real-world NeRF scenes.
Neural descriptor fields [80] use intermediate features that emerge in a 3D occupancy field network [53] for efficiently teaching robots object grasping.
Instead of a pre-trained object-centric 3D model, we use 2D vision models as teacher networks via distillation, exploiting recent progress in pre-trained foundation models [7].
Geometric Decomposition of Neural Scene Representations Kohli et al.
[40] and Zhi et al.
[112] show that neural implicit representations can be combined with the supervision of semantic labels.
Yang et al.
[104] demonstrate that given view-consistent ground-truth instance segmentation masks during training, NeRF can be trained to represent each object as different volumes, although such an annotation is expensive in practice. Concurrently, Benaim et al.
[6] also experiment with the different parametarization. Conditional [49, 37, 21, 63] and generative models [60, 61, 31] enable a degree of category-specific decomposition (e.g., human bodyparts) and editing on constrained domains with large datasets.
Regular structures such as voxelgrids or octrees [13, 48, 14, 88, 94, 89, 43, 81, 107, 59, 60] or unsupervised decomposition [73, 85, 109, 83] enable editability via manipulation of localized parameters.
However, the decomposition is limited due to the inflexibly structured boundaries or strong assumptions about scenes; self-supervised object-centric learning is a difficult task.
Other studies also explored reconstruction with more structured hybrid representations via pipelines specialized to a domain (e.g., traffic scene)
[64, 25, 41] or situation (e.g., each object data is independently accessible) [28, 27, 105].
Note that this line of work defines and constrains domains or the types of segmentation during or before training and thus limits the degrees of freedom for editable scenes and objects.
In contrast, our method can decompose scene-specific NeRFs into arbitrary semantic units via text and image queries, enabling versatile scene edits without retraining. A concurrent paper by Tschernezki et al.
[93] also explores the same training framework and, in particular, investigates how fused features are improved from 2D teacher networks.
It also complementarily shows the results with other teacher models (MoCo-v3 [17] and DeiT [91]), dimension reduction via PCA, and NeuralDiff [92]-based neural fields. Other concurrent studies explore decomposition through training scene-specific segmentation field [113] or 3DCNN [76] supervised by click or scribble annotations. Lastly, in a different but related task, video editing, Kasten et al. [38] use foreground-background decomposition and atlas representation for time-consistent, local editing; Loeschcke et al.
[51] and Bar-Tal et al.
[5] further use CLIP for editing.
Zero-shot Semantic Segmentation. Zero-shot semantic segmentation is a challenging task [24, 2, 10] where a model has to predict semantic labels of pixels in images without a-priori information of the categories.
A typical solution is to use vision-and-language cross-modal encoders.
They are trained to encode images (pixels) and text labels into the same semantic space and perform zero-shot prediction based on the similarity or alignments of the two inputs.
Recent development of image encoder architectures [96, 22, 71] and large-scale training [69, 12] have improved the ability and generalization of vision models, including zero-shot models [44, 52, 99, 103, 114, 72].
On the other hand, ongoing studies on zero-shot perception in 3D still suffer from the lack of effective, efficient, and high-resolution architectures and large-scale annotated datasets [54, 33, 29, 101, 78, 30].
Our method is a new approach to perform zero-shot semantic segmentation on scene-specific 3D fields by exploiting progress in the image domain without semantic 3D supervision.
We note that the goal of this paper is not to achieve state-of-the-art performance on 3D semantic segmentation tasks.
Instead, our goal is the decomposition of neural scene representations for editing, which requires smooth segmentation results on continuous 3D space rather than segmentation of discrete point clouds or voxelgrids.
Preliminaries 3.1 Neural Radiance Fields (NeRF)
NeRF [56] uses MLPs to output density σ and color c given a point coordinate x
= ( x, y, z ) in a 3D scene.
This simple scene representation can be rendered and optimized via volume rendering.
Given a pixel’s camera ray r ( t ) =
o + t d , depth t with bounds [ t near , t far ] , camera position o , and its view direction d , NeRF calculates the color of a ray using quadrature of K sampled points { x k } Kk =1 with depths { t k } Kk =1 as
[Equation]
where α ( x ) = 1 − exp( − x ) , and δ k = t k +1 − t k is the distance between adjacent point samples.
NeRFs are optimized solely on a dataset of images and their camera poses by minimizing a rerendering loss.
3.2 Pre-trained Models and Zero-shot Segmentation of Images
Most semantic segmentation models pre-define a closed set of labels, and cannot flexibly change the segmentation categories or boundaries without supervised training.
In contrast, zero-shot semantic segmentation predicts target regions given open-set queries.
Li et al.
[44] proposes LSeg, a model to
[Table]
example, we can use image or pixel 144 feature as a query f q in the same or similar formulation of Equation 3.
For example, DINO [Caron 145 et al., 2021], a self-supervised vision model, unsupervisedly solves video instance segmentation 146 (tracking) by calculating similarity among the features in adjacent frames.
Amir et al.
[2021] also 147 investigates co-segmentation and point correspondence by similarity and clustering of DINO features.
In our experiments, we use these two publicly available models, LSeg and DINO, for producing 149 image or text features of observed view images or users’ queries for 3D decomposition.
150 4 Neural Perceptual Fields 151 4.1 Learning Feature via Volume Rendering 152 ThebasicNeRFlearnsaﬁeldtocomputethedensityandtheview-dependentcolorofapoint,  ( x ) and 153 c ( x , d ) .
We can extend NeRF by additionally modeling quantities.
For example, SemanticNeRF
[Zhiet al., 2021a, Fu et al., 2022] adds a branch, whose output is a probability distribution of closed-set 155 semantic labels.
They train the branch by supervision from images with ground- semantic labels and use the model to produce more labeled images from novel views for augmenting the labeled 157 dataset.
We further extend such ideas and enable NeRF to perform 3D zero-shot segmentation using open-set 159 text labels or other feature queries.
Instead of a branch performing closed-set classiﬁcation, wepropose to add a feature branch, whose output is a feature vector.
This branch models a feature ﬁeld 161 describing some kind of characteristics of each spatial point.
We supervise the feature ﬁeld by apretrained pixel-level image encoder f img as a teacher network.
Speciﬁcally, given a 3D coordinate x , 163 the new NeRF outputs a feature vector f ( x ) in addition to density  ( x ) and color c ( x , d ) .
Volumerendering is also performed in the feature ﬁeld as follows
[Equation]
We can optimize f through SGD on minimizing the difference between rendered features
3.2 Pre-trained Models and Zero-shot Segmentation of Image 125 Zero-shot semantic segmentation is a task of predicting target regions after training without infor- 126 mation about what semantic concepts are targeted during test time.
A typical approach is training 127 encoders on large-scale datasets for obtaining generalizable features.
Recent development of self- 128 supervised learning improves the performance of such a feature-based approach.
129 Li et al.
[2022]’s LSeg employs an image feature encoder using dense prediction transformers [Ranftl 130 et al., 2021] and a text label feature encoder using CLIP [Radford et al., 2021], which are trained on 131 large-scale language-image contrastive learning.
Probability of a label l of a pixel r in an image I , 132 p ( l | I,r ) , are predicted by dot product of pixel-level image feature f img ( I,r ) and queried text label 133 feature f q ( l ) followed by softmax:p ( l | I,r ) =
exp( f img ( I,r ) f q ( l ) T ) P l 0 2 L exp( f img ( I,r ) f
q ( l 0 ) T ) , (3) where L is a set of possible labels.
We omit the temperature parameter ⌧ in softmax.
During 135 training, LSeg optimizes only the image encoder f img ( I,r ) through SGD on minimizing cross entropy 136 on supervised semantic segmentation image datasets.
The text encoder f q ( l ) is frozen from the 137 pre-trained CLIP checkpoint [Radford et al., 2021].
The pre-trained CLIP becomes a platform 138 and has been extended with further training another modules sharing the same latent space.
For 139 example, Reimers and Gurevych [2019, 2020] trains a multi-lingual (more than 50+ languages) text 140 encoder, which enables CLIP and CLIP-inspired variants to use non-English queries like Japanese 1 . 141 Becauseourworkusesthesamelatentspace, wecandecomposeNeRFwithsuchnon-Englishqueries 142 similarly.
143 Segmentation can be performed with other types of query.
For example, we can use image or pixel 144 feature as a query f q in the same or similar formulation of Equation 3.
For example, DINO [Caron 145 et al., 2021], a self-supervised vision model, unsupervisedly solves video instance segmentation tracking calculating
147 investigates co
In our experiments, we use these two publicly available models, and DINO, for producing 149 image or text observed view images or
3D
4 Neural Perceptual Fields4.1 Learning Feature via Volume Rendering
ThebasicNeRFlearnsaﬁeldtocomputethedensityandtheview-dependentcolorofapoint,  ( x ) and 153 c ( x , d ) .
We can extend NeRF by additionally modeling quantities.
For example, SemanticNeRF
[Zhiet al., 2021a, Fu et al., 2022] adds a branch, whose output is a probability distribution of closed-set 155 semantic labels.
They train the branch by supervision from images with ground-truth semantic labelsand use the model to produce more labeled images from novel views for augmenting the labeled 157 dataset.
We further extend such ideas and enable NeRF to perform 3D zero-shot segmentation using open-set 159 text labels or other feature queries.
Instead of a branch performing closed-set classiﬁcation, wepropose to add a feature branch, whose output is a feature vector.
This branch models a feature ﬁeld 161 describing some kind of characteristics of each spatial point.
We supervise the feature ﬁeld by apretrained pixel-level image encoder f img as a teacher network.
Speciﬁcally, given a 3D coordinate x , 163 the new NeRF outputs a feature vector f ( x ) in addition to density
ˆ F ( r ) =
K
X k
= ˆ T ( t k ) ↵ (  ( x k )
k )
f
( x k ) .
optimize f minimizing f
( I,
network via the volume rendering
We call
Neural
) .1 https://huggingface.co/sentence-transformers/clip-ViT-B-32-multilingual-v
Li et al
[2022]’s transformers
[Ranftl 130 et al., 2021]
Radford et al.,
I
132 p ( l | I r ) predicted by dot product of pixel feature f
( I,r ) text label 133 feature f q ( l ) by softmax:p ( l | I,r ) =
exp( f img ( I,r ) f q ( l ) T ) P l 0 2 L exp( f img ( I,r ) f
q ( l 0 ) T ) , (3) where L is a set of possible labels.
We omit the temperature parameter ⌧ in softmax.
During 135 training, LSeg optimizes only the image encoder f img (
154 et al., 2021a, Fu et al., 2022] adds a branch, whose output is a probability distribution of closed-setsemantic labels.
They train the branch by supervision from images with ground-truth semantic labels 156 and use the model to produce more labeled images from novel views for augmenting the labeleddataset.
We further extend such ideas and enable NeRF to perform 3D zero-shot segmentation using open-set 159 text labels or other feature queries.
Instead of a branch performing closed-set classiﬁcation, wepropose to add a feature branch, whose output is a feature vector.
This branch models a feature ﬁeld 161 describing some kind of characteristics of each spatial point.
We supervise the feature ﬁeld by apretrained pixel-level image encoder f img as a teacher network.
Speciﬁcally, given a 3D coordinate x , 163 the new NeRF outputs a feature vector f ( x ) in addition to density  ( x ) and color c ( x , d ) .
Volumerendering is also performed in the feature ﬁeld as followsˆ F
[Equation]
k
[Equation]
We can optimize f through SGD on minimizing the difference between rendered features
ˆ F ( r ) and 166 theteacher’soutputs f img ( I,r ) .
Thiscanbeseenasdistillationfrom2Dteachernetworkto3Dstudentnetwork via the volume rendering trick.
We call this model Neural Perceptual Fields (NePeRF) .
https://huggingface.co/sentence-transformers/clip-ViT-B-32-multilingual-v  p ( l | I,r ) =
exp( f img ( I
[Table]
r ) f q ( l ) T ) P l 0 2 L exp( f img ( I
[Table]
r ) f q ( l 0 ) T ) , (
[Table]
train the branch by supervision from images with ground-truth semantic labels 156 and use the model to produce more labeled images from novel views for augmenting the labeled 157 dataset.
158 We further extend such ideas and enable NeRF to perform 3D zero-shot segmentation using open-set 159 text labels or other feature queries.
Instead of a branch performing closed-set classiﬁcation, we 160 propose to add a feature branch, whose output is a feature vector.
This branch models a feature ﬁeld 161 describing some kind of characteristics of each spatial point.
We supervise the feature ﬁeld by a 162 pretrained pixel-level image encoder f img as a teacher network.
Speciﬁcally, given a 3D coordinate x ,the new NeRF outputs a feature vector f ( x ) in addition to density  ( x ) and color c ( x , d ) .
Volumerendering is also performed in the feature ﬁeld as follows
[Equation]
We can optimize f through SGD on minimizing the difference between rendered features
ˆ F ( r ) andthe teacher’s outputs f img ( I,r ) .
This can be seen as distillation from 2D teacher network to 3D studentnetwork via the volume rendering trick.
We call this model Neural Perceptual Fields (NePeRF) .1 https://huggingface.co/sentence-transformers/clip-ViT-B-32-multilingual-v
3. Pre-
Models and Zero-shot Segmentation of Image
Zero-shot semantic segmentation is a task of predicting target regions after training without information about what semantic concepts are targeted during test time.
A typical approach is trainingencoders on large-scale datasets for obtaining generalizable features.
Recent development of selfsupervised learning improves the performance of such a feature-based approach.
Li et al.
[2022]’s LSeg employs an image feature encoder using dense prediction transformers [Ranftlet al., 2021] and a text label feature encoder using CLIP
[Radford et al., 2021], which are trained onlarge-scale language-image contrastive learning.
Probability of a label l of a pixel r in an image I ,p ( l | I, r ) , are predicted by dot product of pixel-level image feature f img ( I, r ) and queried text labelfeature f q ( l ) followed by softmax:p ( l | I, r ) =
exp( f img ( I, r ) f q ( l ) T ) P l 0 2 L exp( f img ( I, r ) f
q ( l 0 ) T ) , (3) where L is a set of possible labels.
We omit the temperature parameter ⌧ in softmax.
Duringtraining, LSeg optimizes only the image encoder f img ( I, r ) through SGD on minimizing cross entropyon supervised semantic segmentation image datasets.
The text encoder f q ( l ) is frozen from thepre-trained CLIP checkpoint [Radford et al., 2021].
The pre-trained CLIP becomes a platformand has been extended with further training another modules sharing the same latent space.
Forexample, Reimers and Gurevych [2019, 2020] trains a multi-lingual (more than 50+ languages) textencoder, which enables CLIP and CLIP-inspired variants to use non-English queries like Japanese 1 .
Because our work uses the same latent space, we can decompose NeRF with such non-English queriessimilarly.
Segmentation can be performed with other types of query.
For example, we can use image or pixelfeature as a query f q in the same or similar formulation of Equation 3.
For example, DINO [Caronet al., 2021], a self-supervised vision model, unsupervisedly solves video instance segmentation(tracking) by calculating similarity among the features in adjacent frames.
Amir et al.
[2021] alsoinvestigates co-segmentation and point correspondence by similarity and clustering of DINO features.
In our experiments, we use these two publicly available models, LSeg and DINO, for producingimage or text features of observed view images or users’ queries for 3D decomposition.
Neural Perceptual
Fields
4. Learning Feature via Volume Rendering
The basic NeRF learns a ﬁeld to compute the density and the view-dependent color of a point,  ( x ) andc ( x , d ) .
We can extend NeRF by additionally modeling quantities.
For example, SemanticNeRF
[Zhiet al., 2021a, Fu et al., 2022] adds a branch, whose output is a probability distribution of closed-setsemantic labels.
They train the branch by supervision from images with ground-truth semantic labelsand use the model to produce more labeled images from novel views for augmenting the labeleddataset.
We further extend such ideas and enable NeRF to perform 3D zero-shot segmentation using open-settext labels or other feature queries.
Instead of a branch performing closed-set classiﬁcation, wepropose to add a feature branch, whose output is a feature vector.
This branch models a feature ﬁelddescribing some kind of characteristics of each spatial point.
We supervise the feature ﬁeld by apretrained pixel-level image encoder f img as a teacher network.
Speciﬁcally, given a 3D coordinate x ,the new NeRF outputs a feature vector f ( x ) in addition to density  ( x ) and color c ( x , d ) .
Volumerendering is also performed in the feature ﬁeld as follows
[Equation]
We can optimize f through SGD on minimizing the difference between rendered features
ˆ F ( r ) andthe teacher’s outputs f img ( I, r ) .
This can be seen as distillation from 2D teacher network to 3D studentnetwork via the volume rendering trick.
We call this model Neural Perceptual Fields (NePeRF) .
https://huggingface.co/sentence-transformers/clip-ViT-B-32-multilingual-v  supervision could be sparse or noisy.
Yang et al.
[2021] demonstrated that, given view-consistent 80 ground-truth instance segmentation masks during training, NeRF could decompose each object [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
3 Preliminaries
[Figure]
[Equation] [Equation]
[Equation] [Equation]
↵ (  ( x k )  k ) c ( x k , d ) , (1)
ˆ T ( t k ) =
exp
k
X k
=
( x k 0 )  k 0 !
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
We further extend such ideas and enable NeRF to perform 3D zero-shot segmentation using open-set 159 text labels or other feature queries.
Instead of a branch performing closed-set classiﬁcation, wepropose to add a feature branch, whose output is a feature vector.
This branch models a feature ﬁeld 161 describing some kind of characteristics of each spatial point.
We supervise the feature by
[Table]
pretrained pixel
[Table]
level image encoder img as
[Table]
teacher network.
Speciﬁcally, given a 3D coordinate x , 163 the new NeRF outputs a feature vector f ( x ) in addition to density  ( x ) and color c ( x , d ) .
Volumerendering is also performed in the feature ﬁeld as follows
[Equation]
We can optimize f through SGD on minimizing the difference between rendered features
ˆ F ( r ) and 166 the teacher’s outputs f img ( I,r ) .
This can be seen as distillation from 2D teacher network to 3D studentnetwork via the volume rendering trick.
We call this model Neural Perceptual Fields (NePeRF) .
[Figure]
We follow the training objective of the original NeRF [Mildenhall et al., 2020] and add a new objectivefor minimizing minimizing the difference between rendered features
ˆ F ( r ) and the teacher’s outputsf img ( I,r ) .
We use two networks for volume rendering with coarse-and-ﬁne hierarchical sampling as 171 well as the original NeRF does.
We simultaneously train each network from scratch by minimizingphotometric loss L c and feature loss L f :
[Equation]
where R are sampled rays, and C ( r ) is the ground truth color of pixel ray r .
Total training loss L is
[Equation]
where  is the weight of the feature loss and is set to 0.04 to balance the losses [Zhi et al., 2021a].
We apply stop-gradient to density during rendering features ˆ F ( r ) in Equation 4, because multi-view 176 inconsistent supervision with noise could harm reconstruction quality of geometry, although theeffect seems negligible in preliminary experiments.
4.2 Query-based Decomposition and Editing
After training a NePeRF model, we can perform 3D zero-shot segmentation by directly using the 180 feature ﬁeld f and the pretrained text encoder f q .
Speciﬁcally, probability of a label l of a point x inthe 3D space, p ( l | x ) , are predicted by dot product of the 3D feature f ( x ) and text label feature f q ( l ) 182 followed by softmax:
[Equation]
When we use multiple queries, L = { l 1 ,... } , for selecting a group of regions, we use p ( L| x k ) =
P l 0 2L p ( l 0 | x k ) .
This segmentation information is the fruits of the proposed PeNeRF framework.
185 It is calculated at any 3D points without resolution limitation, and naturally used together with aradiance ﬁeld and volume rendering.
Note that the segmentation depends on only the 3D coordinate 187 and the query 2 , so
it and view synthesis with it are 3D consistent as well as the original NeRF.
Unlikethe proposed method, editing multiple synthesized images by image-based postprocessing breaks 189 3D consistency.
In addition, we can change the segmentation by changing only the query withoutretraining, which cannot be realized by existing methods using closed-set semantic segmentation [Zhi 191 et al., 2021a] or instance segmentation annotation [Yang et al., 2021], but important for user-friendlyinteractive [-editing.
-]editing.2 It is an interesting direction to introduce view dependency to the segmentation for discriminating view- dependent query like referring expressions (e.g., “the chair left to the table” where
R are sampled rays and C ( r ) is the ground truth color of pixel ray r ,  is the weight of thefeature loss and is set to 0.04 to balance the losses [Zhi et al., 2021a].
We apply stop-gradient todensity during rendering features ˆ F ( r ) in Equation 3, because teacher’s features f img ( I, r ) are notcompletely multi-view consistent, which could harm reconstruction quality of geometry.
4.2 Query-based Decomposition and Editing
After training a NePeRF model we can perform 3D zero-shot segmentation by directly using thefeature ﬁeld f and another query encoder f q .
Speciﬁcally, probability of a label l of a point x in the3D space, p ( l | x ) , are predicted by dot product of the 3D feature f ( x ) and text label feature f q ( l )followed by softmax:
[Equation]
When we use multiple queries, M = { l 1 , l 2 , ... } , for selecting a group of regions, we use p ( M| x k ) =
P l 0 2M p ( l 0 | x k ) .
This segmentation information is the fruits of the proposed DFF framework.
It 209 can be calculated at any 3D point without limiting resolution, so naturally used together with aradiance ﬁeld and volume rendering.
Note that the segmentation depends on only the 3D coordinateand the query 1 , so it and view synthesis with it are 3D consistent as well as the original NeRF.
Unlike the proposed method, editing multiple synthesized images by image-based postprocessingbreaks 3D consistency.
In addition, importantly for user-friendly interactive editing, we can change1 It is an interesting direction to introduce view dependency to the segmentation for discriminating viewdependent query like referring expressions (e.g., “the chair left to the table”)
[Chen et al., 2020, Achlioptas al., 2020, Liu
., 2021a, Azuma et
., 2022], but future work.
of color/feature and ground-truth color/teacher’s feature, respectively.
We can optimize f through SGD on minimizing the difference between rendered features
ˆ F ( r ) andthe teacher’s outputs f img ( I, r ) .
This can be seen as distillation from 2D teacher network to 3Dstudent network via the volume rendering trick.
We call this model distilled feature ﬁeld (DFF).
We follow the original NeRF [Mildenhall et al., 2020] for the training objective and the volumerendering strategy.
In addition to the photometric loss, we add a new objective for minimizing thedifference between rendered features ˆ F ( r ) and the teacher’s outputs f img ( I, r ) .
For volume rendering,we use two networks for volume rendering with coarse-and-ﬁne hierarchical sampling as well as theoriginal NeRF does.
We simultaneously train each network from scratch by minimizing photometricloss L p and feature loss L f , in total, L :
L =
L p + L f ,
L p =
X r 2R  ˆ C ( r )  C ( r )  2 2 ,
L f = X r 2R  ˆ F ( r )  f img ( I, r )  1 , (4) where R are sampled rays, and C ( r ) is the ground truth color of pixel ray r ,  is the weight of thefeature loss and is set to 0.04 to balance the losses [Zhi et al., 2021a].
We apply stop-gradient todensity during rendering features ˆ F ( r ) in Equation 3, because teacher’s features f img ( I, r ) are not
After training a NePeRF model, we can perform 3D zero-shot segmentation by directly using the feature ﬁeld f and another query encoder f q .
Speciﬁcally, probability of a label l of a point x in the3D space, p ( l | x ) , are predicted by dot product of the 3D feature f ( x ) and text label feature f q ( l )followed by softmax:
[Equation]
When we use multiple queries,
M = { l 1 , l 2 , ... } , for selecting a group of regions, we use p ( M| x k ) =
P l 0 2M p ( l 0 | x k ) .
This segmentation information is the fruits of the proposed DFF framework.
Itcan be calculated at any 3D point without limiting resolution, so naturally used together with aradiance ﬁeld and volume rendering.
Note that the segmentation depends on only the 3D coordinateand the query 1 , so it and view synthesis with it are 3D consistent as well as the original NeRF.
Unlike the proposed method, editing multiple synthesized images by image-based postprocessingbreaks 3D consistency.
In addition, importantly for user-friendly interactive editing, we can change1 It is an interesting direction to introduce view dependency to the segmentation for discriminating viewdependent query like referring expressions (e.g., “the chair left to the table”)
[Chen et al., 2020, Achlioptas et al., 2020, Liu et al., 2021a, Azuma et al., 2022], but left for future work.
[Table]
p ( l | I
[Table]
[Table]
exp( f img
[Table]
I, r ) [Table] q ( l ) T ) P l 0 2 L exp( f img ( I, r ) f q ( l 0 ) T ) , (3) where L is a set of possible labels.
We omit the temperature parameter ⌧ in softmax.
During 135 training, LSeg optimizes only the image encoder f img ( I, r ) through SGD on minimizing cross entropy 136 on supervised semantic segmentation image datasets.
The text encoder f q ( l ) is frozen from the 137 pre-trained CLIP checkpoint [Radford et al., 2021].
The pre-trained CLIP becomes a platform 138 and has been extended with further training another modules sharing the same latent space.
For 139 example, Reimers and Gurevych [2019, 2020] trains a multi-lingual (more than 50+ languages) text 140 encoder, which enables CLIP and CLIP-inspired variants to use non-English queries like Japanese 1 .
Because our work uses the same latent space, we can decompose NeRF with such non-English queries 142 similarly.
143 Segmentation can be performed with other types of query.
For example, we can use image or pixel 144 feature as a query f q in the same or similar formulation of Equation 3.
For example, DINO [Caron 145 et al., 2021], a self-supervised vision model, unsupervisedly solves video instance segmentation 146 (tracking) by calculating similarity among the features in adjacent frames.
Amir et al.
[2021] also 147 investigates co-segmentation and point correspondence by similarity and clustering of DINO features.
In our experiments, we use these two publicly available models, LSeg and DINO, for producing 149 image or text features of observed view images or users’ queries for 3D decomposition.
Neural Perceptual Fields Learning
Volume Rendering
The basic NeRF learns a ﬁeld to compute the density and the view-dependent color of a point,  ( x ) and 153 c ( x , d ) .
We can extend NeRF by additionally modeling quantities.
For example, SemanticNeRF
[Zhi 154 et al., 2021a, Fu et al., 2022] adds a branch, whose output is a probability distribution of closed-setsemantic labels.
They train the branch by supervision from images with ground-truth semantic labels
We further extend such ideas and enable NeRF to perform 3D zero-shot segmentation using open-settext labels or other feature queries.
Instead of a branch performing closed-set classiﬁcation, wepropose to add a feature branch, whose output is a feature vector.
This branch models a feature ﬁelddescribing some kind of characteristics of each spatial point.
We supervise the feature ﬁeld by apretrained pixel-level image encoder f img as a teacher network.
Speciﬁcally, given a 3D coordinate x ,the new NeRF outputs a feature vector f ( x ) in addition to density  ( x ) and color c ( x , d ) .
Volumerendering is also performed in the feature ﬁeld as follows
[Equation]
We can optimize f through SGD on minimizing the difference between rendered features
ˆ F ( r ) andthe teacher’s outputs f img ( I, r ) .
This can be seen as distillation from 2D teacher network to 3D studentnetwork via the volume rendering trick.
We call this model Neural Perceptual Fields (NePeRF) .
[Figure]
feature ﬁeld
192 We follow the original NeRF [Mildenhall et al., 2020] for the training objective and the volumerendering strategy.
In addition to the photometric loss, we add a new objective for minimizing thedifference between rendered features ˆ F ( r ) and the teacher’s outputs f img ( I, r ) .
For volume rendering,we use two networks for volume rendering with coarse-and-ﬁne hierarchical sampling as well as theoriginal NeRF does.
We simultaneously train each network from scratch by minimizing photometricloss L p and feature loss L f , in total, L :
L =
L p + L f , L p =
X r 2R  ˆ C ( r )  C ( r )  2 2 , L f = X r 2R  ˆ F ( r )  f img ( I, r )  1 , (4) where R are sampled rays, and C ( r ) is the ground truth color of pixel ray r ,  is the weight of thefeature loss and is set to 0.04 to balance the losses [Zhi et al., 2021a].
We apply stop-gradient to Equation 3, teacher’s features f ( I, r ) are notcompletely multi-view consistent, which could harm reconstruction quality
After training a NePeRF model, we can perform 3D zero-shot segmentation by directly using thefeature ﬁeld f and another query encoder f q .
Speciﬁcally, probability of a label l of a point x in the3D space, p ( l | x ) , are predicted by dot product of the 3D feature f ( x ) and text label feature f q ( l )followed by softmax:
[Equation]
When we use multiple queries, M = { l 1 , l 2 , ... } , for selecting a group of regions, we use p ( M| x k ) =
[Equation]
k ) .
This segmentation information is the fruits of the proposed DFF framework.
Itcan be calculated at any 3D point without limiting resolution, so naturally used together with aradiance ﬁeld and volume rendering.
Note that the segmentation depends on only the 3D coordinateand the query 1 , so it and view synthesis with it are 3D consistent as well as the original NeRF.
Unlike the proposed method, editing multiple synthesized images by image-based postprocessingbreaks 3D consistency.
In addition, importantly for user-friendly interactive editing, we can change view-
2D via volume
We follow the original NeRF [Mildenhall et al., 2020] for the training objective and the volumerendering strategy.
In addition to the photometric loss, we add a new objective for minimizing thedifference between rendered features ˆ F ( r ) and the teacher’s outputs f img ( I, r ) .
For volume rendering,we use two networks for volume rendering with coarse-and-ﬁne hierarchical sampling as well as theoriginal NeRF does.
We simultaneously train each network from scratch by minimizing photometricloss L p and feature loss L in total,
L =
L p + L f ,
L p =
X r 2R  ˆ C ( r )  C ( r )  2 2 ,
L f = X r 2R  ˆ F ( r )  f img ( I, r )  1 , (4) where R are sampled rays, and C ( r ) is the ground truth color of pixel ray r ,  is the weight of thefeature loss and is set to 0.04 to balance the losses [Zhi et al., 2021a].
We apply stop-gradient todensity during rendering features ˆ F ( r ) in Equation 3, because teacher’s features f img ( I, r ) are notcompletely multi-view consistent, which could harm reconstruction quality of geometry.4.
Query-based Decomposition and Editing
After training a NePeRF model, we can perform 3D zero-shot segmentation by directly using thefeature ﬁeld f and another query encoder f q .
Speciﬁcally, probability of a label l of a point x in the3D space, p ( l | x ) , are predicted by dot product of the 3D feature f ( x ) and text label feature f q ( l )followed by softmax:
[Equation]
When we use multiple queries, M = { l 1 , l 2 , ... } , for selecting a group of regions, we use p ( M| x k ) =
P l 0 2M p ( l 0 | x k ) .
This segmentation information is the fruits of the proposed DFF framework.
Itcan be calculated at any 3D point without limiting resolution, so naturally used together with aradiance ﬁeld and volume rendering.
Note that the segmentation depends on only the 3D coordinateand the query 1 , so it and view synthesis with it are 3D consistent as well as the original NeRF.
Unlike the proposed method, editing multiple synthesized images by image-based postprocessingbreaks 3D consistency.
In addition, importantly for user-friendly interactive editing, we can change1 It is an interesting direction to introduce view dependency to the segmentation for discriminating viewdependent query like referring expressions (e
[Table]
g., “the chair left to the table”)
[Chen et al., 2020, Achlioptas et al., 2020, Liu et al., 2021a, Azuma et al., 2022], but left for future work.
[Table]
[Table]
[Table]
a neural ﬁeld to compute the density and the view-dependent color of a point,  ( x ) 159 and c ( x , d ) .
We may extend NeRF by adding decoders for other quantities of interest.
For example,
SemanticNeRF [86] adds a branch outputting a probability distribution of closed-set semantic labels, 161 trained with supervision via images with ground-truth semantic labels.
This enables prediction ofsemantic segmentation masks from novel views and use the model to produce more labeled images 163 from novel views for augmenting the labeled dataset.
Because ground-truth annotation is costly, themethod is practically inefﬁcient as a means of scene editing [81].
For speciﬁc domains like trafﬁc 165 scenes [20, 32], instead of ground truth, we could train a closed-set segmentation model and use itsprediction for training object-aware neural ﬁelds.
However, the method is possible only if types of 167 objects are limited and the domain-speciﬁc supervised dataset is available; it limits the application ofscene editing in terms of domain and ﬂexibility of decomposition.
We further extend such ideas and enable NeRF to perform 3D zero-shot segmentation using open-set 170 text labels or other feature queries.
Instead of a branch performing closed-set classiﬁcation, wepropose to add a feature branch outputting a feature vector itself.
This branch models a 3D feature 172 ﬁeld describing semantics of each spatial point.
We supervise the feature ﬁeld by a pretrained pixellevel image encoder f img as a teacher network.
Speciﬁcally, given a 3D coordinate x , the new NeRF 174 outputs a feature vector f ( x ) in addition to density  ( x ) and color c ( x , d ) , as shown in Figure 1.
Volume rendering is performed similarly in the feature ﬁeld as follows
[Equation]
We can optimize f through SGD on minimizing the difference between rendered features
ˆ F ( r ) and 177 the teacher’s outputs f img ( I,r ) .
This can be seen as distillation from 2D teacher network to 3Dstudent network via the volume rendering trick.
We call this model distilled feature ﬁeld (DFF).
We follow the original NeRF [46] for the training objective and the volume rendering strategy.
In 180 addition to the photometric loss, we add a new objective for minimizing the difference betweenrendered features ˆ F ( r ) and the teacher’s outputs f img ( I,r ) .
For volume rendering, we use two 182 networks for volume rendering with coarse-and-ﬁne hierarchical sampling as well as the original
NeRF does.
We simultaneously train each network from scratch by minimizing photometric loss L p 184 and feature loss L f , in total, L :
[Equation]
where L is a set of possible labels.
If negative labels are not available, we may use other scores like 139 thresholded cosine similarity to directly compute the probability of a label.
During training, LSeg 140 optimizes only the image encoder f img ( I,r ) through SGD on minimizing cross entropy on supervised 141 semantic segmentation image datasets.
The text encoder f q ( l ) is obtained from a pre-trained CLIP 142 model [56].
Recently, pre-trained CLIP has been leveraged as a backbone for a variety of tasks and 143 has been extended with further training modules sharing the same latent space.
For example, Reimers 144 and Gurevych [61, 62] trains a multi-lingual (more than 50+ languages) text encoder, which enables 145 CLIP and CLIP-inspired variants to use non-English queries like Japanese.
In this work, we similarly 146 use the latent space of a pre-trained CLIP, enabling decomposition of NeRFs with both English and 147 non-English queries.
148 Segmentation can further be performed with other types of queries.
For example, we can use 149 image, patch or pixel features as a query f
q using a similar dot-product similarity formulation as 150 in Eq.
Notably, DINO [7], a self-supervised vision model, unsupervisedly solves video instance 151 segmentation (tracking) well by calculating similarity among the features in adjacent frames.
Amir 152 etal.[3]alsodemonstratethatDINOfeaturesworkwellonco-segmentationandpointcorrespondence 153 by similarity and clustering.
In our experiments, we use these two publicly available models, LSeg and DINO, for producing 155 image or text features of observed view images or users’ queries for 3D decomposition.
156 4 Neural Perceptual Fields 157 4.1 Distilling Foundation Modules into 3D Feature Fields via Volume Rendering 158 NeRF learns a neural ﬁeld to compute the density and the view-dependent color of a point,  ( x ) 159 and c ( x , d ) .
We may extend NeRF by adding decoders for other quantities of interest.
For example, 160 SemanticNeRF [86] adds a branch outputting a probability distribution of closed-set semantic labels, 161 trained with supervision via images with ground-truth semantic labels.
This enables prediction ofsemantic segmentation masks from novel views and use the model to produce more labeled images 163 from novel views for augmenting the labeled dataset.
Because ground-truth is , themethod is practically inefﬁcient as means editing [81].
For like 165 scenes [20, 32], instead of ground truth, we could train a closed-set segmentation model and use itsprediction for training object-aware neural ﬁelds.
However, the method is possible only if types of 167 objects are limited and the domain-speciﬁc supervised dataset is available; it limits the application ofscene editing in terms of domain and ﬂexibility of decomposition.
We further extend such ideas and enable NeRF to perform 3D zero-shot segmentation using open-set 170 text labels or other feature queries.
Instead of a branch performing closed-set classiﬁcation, wepropose to add a feature branch outputting a feature vector itself.
This branch models a 3D feature 172 ﬁeld describing semantics of each spatial point.
We supervise the feature ﬁeld by a pretrained pixellevel image encoder f img as a teacher network.
Speciﬁcally, given a 3D coordinate x , the new NeRF 174 outputs a feature vector f ( x ) in addition to density  ( x ) and color c ( x , d ) , as shown in Figure 1.
Volume rendering is performed similarly in the feature ﬁeld as follows
[Equation]
We can optimize f through SGD on minimizing the difference between rendered features
ˆ F ( r ) and 177 the teacher’s outputs f img ( I,r ) .
This can be seen as distillation from 2D teacher network to 3Dstudent network via the volume rendering trick.
We call this model distilled feature ﬁeld (DFF).
We follow the original NeRF [46] for the training objective and the volume rendering strategy.
In 180 addition to the photometric loss, we add a new objective for minimizing the difference betweenrendered features ˆ F ( r ) and the teacher’s outputs f img ( I,r ) .
For volume rendering, we use two 182 networks for volume rendering with coarse-and-ﬁne hierarchical sampling as well as the original
NeRF does.
We simultaneously train each network from scratch by minimizing photometric loss L p 184 and feature loss L f , in total, L :
[Equation]
where L is a set of possible labels.
If negative labels are not available, we may use other scores like 139 thresholded cosine similarity to directly compute the probability of a label.
During training, LSeg 140 optimizes only the image encoder f img ( I,r ) through SGD on minimizing cross entropy on supervised 141 semantic segmentation image datasets.
The text encoder f q ( l ) is obtained from a pre-trained CLIP 142 model [56].
Recently, pre-trained CLIP has been leveraged as a backbone for a variety of tasks and 143 has been extended with further training modules sharing the same latent space.
For example, Reimers 144 and Gurevych [61, 62] trains a multi-lingual (more than 50+ languages) text encoder, which enables 145 CLIP and CLIP-inspired variants to use non-English queries like Japanese.
In this work, we similarly 146 use the latent space of a pre-trained CLIP, enabling decomposition of NeRFs with both English and 147 non-English queries. 148 Segmentation can further be performed with other types of queries.
For example, we can use 149 image, patch or pixel features as a query f
q using a similar dot-product similarity formulation as 150 in Eq. Notably, DINO [7], a self-supervised vision model, unsupervisedly solves video instance 151 segmentation (tracking) well by calculating similarity among the features in adjacent frames.
Amir 152 etal.[3]alsodemonstratethatDINOfeaturesworkwellonco-segmentationandpointcorrespondence 153 by similarity and clustering. In our experiments, we use these two publicly available models, LSeg and DINO, for producing 155 image or text features of observed view images or users’ queries for 3D decomposition.
4 Neural Perceptual Fields
NeRF learns a neural ﬁeld to compute the density and the view-dependent color of a point,  ( x ) 159 and c ( x , d ) .
We may extend NeRF by adding decoders for other quantities of interest.
For example,
SemanticNeRF [86] adds branch outputting probability trained with supervision via images with ground-truth semantic labels.
This enables prediction ofsemantic segmentation masks from novel views and use the model to produce more labeled images 163 from novel views for augmenting the labeled dataset.
Because ground-truth annotation is costly, themethod is practically inefﬁcient as a means of scene editing [81].
For speciﬁc domains like trafﬁc 165 scenes [20, 32], instead of ground truth, we could train a closed-set segmentation model and use itsprediction for training object-aware neural ﬁelds.
However, the method is possible only if types of 167 objects are limited and the domain-speciﬁc supervised dataset is available; it limits the application ofscene editing in terms of domain and ﬂexibility of decomposition.
We further extend such ideas and enable NeRF to perform 3D zero-shot segmentation using open-set 170 text labels or other feature queries.
Instead of a branch performing closed-set classiﬁcation, wepropose to add a feature branch outputting a feature vector itself.
This branch models a 3D feature 172 ﬁeld describing semantics of each spatial point.
We supervise the feature ﬁeld by a pretrained pixellevel image encoder f img as a teacher network.
Speciﬁcally, given a 3D coordinate x , the new NeRF 174 outputs a feature vector f ( x ) in addition to density  ( x ) and color c ( x , d ) , as shown in Figure 1.
Volume rendering is performed similarly in the feature ﬁeld as follows
[Equation]
We can optimize f through SGD on minimizing the difference between rendered features
ˆ F ( r ) and 177 the teacher’s outputs f img ( I,r ) .
This can be seen as distillation from 2D teacher network to 3Dstudent network via the volume rendering trick.
We call this model distilled feature ﬁeld (DFF).
We follow the original NeRF [46] for the training objective and the volume rendering strategy.
In 180 addition to the photometric loss, we add a new objective for minimizing the difference betweenrendered features ˆ F ( r ) and the teacher’s outputs f img ( I,r ) .
For volume rendering, we use two 182 networks for volume rendering with coarse-and-ﬁne hierarchical sampling as well as the original
NeRF does.
We simultaneously train each network from scratch by minimizing photometric loss L p 184 and feature loss L f , in total, L :
[Equation]
[Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
of query.
For example, we can use image or pixel 144 feature as a query f q in the same or similar formulation of Equation 3.
For example, DINO [Caron 145 et al., 2021], a self-supervised vision model, unsupervisedly solves video instance segmentation 146 (tracking) by calculating similarity among the features in adjacent frames.
Amir et al.
[2021] also 147 investigates co-segmentation and point correspondence by similarity and clustering of DINO features.
In our experiments, we use these two publicly available models, LSeg and DINO, for producing 149 image or text features of observed view images or users’ queries for 3D decomposition.
Neural Perceptual Fields 151 4.1 Learning Feature via Volume Rendering
The basic NeRF learns a ﬁeld to compute the density and the view-dependent color of a point,  ( x ) and 153 c ( x , d ) .
We can extend NeRF by additionally modeling quantities.
For example, SemanticNeRF
[Zhiet al., 2021a, Fu et al., 2022] adds a branch, whose output is a probability distribution of closed-setsemantic labels.
They train the branch by supervision from images with ground- semantic labelsand use the model to produce more labeled images from novel views for augmenting the labeleddataset.
We further extend such ideas and enable NeRF to perform 3D zero-shot segmentation using open-settext labels or other feature queries.
Instead of a branch performing closed-set classiﬁcation, wepropose to add a feature branch, whose output is a feature vector.
This branch models a feature ﬁelddescribing some kind of characteristics of each spatial point.
We supervise the feature ﬁeld by apretrained pixel-level image encoder f img as a teacher network.
Speciﬁcally, given a 3D coordinate x ,the new NeRF outputs a feature vector f ( x ) in addition to density  ( x ) and color c ( x , d ) .
Volumerendering is also performed in the feature ﬁeld as follows
[Equation]
We can optimize f through SGD on minimizing the difference between rendered features
ˆ F ( r ) and [Table] the teacher’s outputs f img ( I, r ) .
This can be seen as distillation from 2D teacher network to 3D student [Table] network via the volume rendering trick.
We call this model Neural Perceptual Fields (NePeRF) . [Table]
https://huggingface.co/sentence-transformers/clip-ViT-B-32-multilingual-v
[Table]
and Gurevych [2019, 2020] trains a multi-lingual (more than 50+ languages) text 140 encoder, which enables CLIP and CLIP-inspired variants to use non-English queries {+like Japanese 1 . 
+}
Because our work uses the same latent space, we can decompose NeRF with such non-English queries 142 similarly.
143 Segmentation can be performed with other types of query.
For example, we can use image or pixel 144 feature as a query f q in the same or similar formulation of Equation 3.
For example, DINO [Caron 145 et al., 2021], a self-supervised vision model, unsupervisedly solves video instance segmentation 146 (tracking) by calculating similarity among the features in adjacent frames.
Amir et al.
[2021] also 147 investigates co-segmentation and point correspondence by similarity and clustering of DINO features.
In our experiments, we use these two publicly available models, LSeg and DINO, for producing 149 image or text features of observed view images or users’ queries for 3D decomposition.
Neural Perceptual Fields 151 4.1 Learning Feature via Volume Rendering
The basic NeRF learns a ﬁeld to compute the density and the view-dependent color of a point,  ( x ) and 153 c ( x , d ) . We can extend NeRF by additionally modeling quantities. For example, SemanticNeRF
[Zhiet al., 2021a, Fu et al., 2022] adds a branch, whose output is a probability distribution of closed-setsemantic labels.
They train the branch by supervision from images with ground-truth semantic labels 156 and use the model to produce more labeled images from novel views for augmenting the labeleddataset.
We further extend such ideas and enable NeRF to perform 3D zero-shot segmentation using open-set 159 text labels or other feature queries.
Instead of a branch performing closed-set classiﬁcation, wepropose to add a feature branch, whose output is a feature vector.
This branch models a feature ﬁeld 161 describing some kind of characteristics of each spatial point.
We supervise the feature ﬁeld by apretrained pixel-level image encoder f img as a teacher network.
Speciﬁcally, given a 3D coordinate x , 163 the new NeRF outputs a feature vector f ( x ) in addition to density  ( x ) and color c ( x , d ) .
Volumerendering is also performed in the feature ﬁeld as follows
[Equation]
We can optimize f through SGD on minimizing the difference between rendered features
ˆ F ( r ) andthe teacher’s outputs f img ( I,r ) .
This can be seen as distillation from 2D teacher network to 3D student 167 network via the volume rendering trick.
We call this model Neural Perceptual Fields (NePeRF) .
https://huggingface.co/sentence-transformers/clip-ViT-B-32-multilingual-v  where L is a set of possible labels.
If negative labels are not available, we may use other scores likethresholded cosine similarity to directly compute the probability of a label.
During training, LSegoptimizes only the image encoder f img ( I, r ) through SGD on minimizing cross entropy on supervisedsemantic segmentation image datasets.
The text encoder f q ( l ) is obtained from a pre-trained CLIPmodel [56].
Recently, pre-trained CLIP has been leveraged as a backbone for a variety of tasks andhas been extended with further training modules sharing the same latent space.
For example, Reimersand Gurevych [61, 62] trains a multi-lingual (more than 50+ languages) text encoder, which enables
CLIP and CLIP-inspired variants to use non-English queries like Japanese.
In this work, we similarlyuse the latent space of a pre-trained CLIP, enabling decomposition of NeRFs with both English andnon-English queries.
Segmentation can further be performed with other types of queries.
For example, we can useimage, patch or pixel features as a query
f q using a similar dot-product similarity formulation asin Eq. 2.
Notably, DINO [7], a self-supervised vision model, unsupervisedly solves video instancesegmentation (tracking) well by calculating similarity among the features in adjacent frames.
Amiret al.
[3] also demonstrate that DINO features work well on co-segmentation and point correspondenceby similarity and clustering.
In our experiments, we use these two publicly available models, LSeg and DINO, for producingimage or text features of observed view images or users’ queries for 3D decomposition.
Neural Perceptual Fields
Foundation 3D Feature Fields Volume Rendering
NeRF learns a neural ﬁeld to compute the density and the view-dependent color of a point,  ( x )and c ( x , d ) .
We may extend NeRF by adding decoders for other quantities of interest.
For example,
SemanticNeRF [86] adds a branch outputting a probability distribution of closed-set semantic labels,trained with supervision via images with ground-truth semantic labels.
This enables prediction ofsemantic segmentation masks from novel views and use the model to produce more labeled imagesfrom novel views for augmenting the labeled dataset.
Because ground-truth annotation is costly, themethod is practically inefﬁcient as a means of scene editing [81].
For speciﬁc domains like trafﬁcscenes [20, 32], instead of ground truth, we could train a closed-set segmentation model and use itsprediction for training object-aware neural ﬁelds.
However, the method is possible only if types ofobjects are limited and the domain-speciﬁc supervised dataset is available; it limits the application ofscene editing in terms of domain and ﬂexibility of decomposition.
We further extend such ideas and enable NeRF to perform 3D zero-shot segmentation using open-settext labels or performing
, wepropose to add a feature branch outputting a feature vector itself.
This branch models a 3D featureﬁeld describing semantics of each spatial point.
We supervise the feature ﬁeld by a pretrained pixellevel image encoder f img as a teacher network.
Speciﬁcally, given a 3D coordinate x , the new NeRFoutputs a feature vector f ( x ) in addition to density  ( x ) and color c ( x , d ) , as shown in Figure 1.
Volume rendering is performed similarly in the feature ﬁeld as follows
[Equation]
We can optimize f through SGD on minimizing the difference between rendered features
ˆ F ( r ) andthe teacher’s outputs f img ( I, r ) .
This can be seen as distillation from 2D teacher network to 3Dstudent network via the volume rendering trick.
We call this model distilled feature ﬁeld (DFF).
We follow the original NeRF [46] for the training objective and the volume rendering strategy.
Inaddition to the photometric loss, we add a new objective for minimizing the difference betweenrendered features ˆ F ( r ) and the teacher’s outputs f img ( I, r ) .
For volume rendering, we use twonetworks for volume rendering with coarse-and-ﬁne hierarchical sampling as well as the original
NeRF does.
We simultaneously train each network from scratch by minimizing photometric loss L pand feature loss L f , in total, L : [Equation]
label feature encoder [69], trained via large-scale language-image contrastive learning.
The probability of a text label l given a pixel r in an image I , p ( l | I, r ) , is then calculated via dot product of pixel-level image feature f img ( I, r ) and queried text feature f q ( l ) followed by a softmax:
[Equation]
where L is a set of possible labels.
If negative labels are not available, we may use other scores like thresholded cosine similarity to directly compute the probability of a label.
During training, LSeg optimizes only the image encoder f img ( I, r ) by minimizing cross-entropy on supervised semantic segmentation datasets.
The text encoder f q ( l ) is obtained from a pre-trained CLIP model [69].
Recently, pre-trained CLIP has been leveraged as the backbone for a variety of tasks and has been extended with additional modules sharing the same latent space.
For example, Reimers and Gurevych [74, 75] trains a multi-lingual (more than 50+ languages) text encoder, which enables CLIP and
CLIP-inspired variants to use non-English queries like Japanese.
We similarly use the latent space of a pre-trained CLIP for LSeg via distillation, enabling the decomposition of NeRFs with both English and non-English queries.
Segmentation can further be performed with other modalities such as image, patch or pixel query features f q using a similar dot-product similarity formulation as in Eq. 2.
Notably, DINO [12], a self-supervised vision model, solves video instance segmentation and tracking by calculating similarity among features in adjacent frames.
Amir et al.
[3] also demonstrate that
DINO features work well on co-segmentation and point correspondence by similarity and clustering.
In our experiments, we use these two publicly available models, LSeg and DINO, to obtain features of images and texts for 3D decomposition.
Distilled Feature Fields 4.1 Distilling Foundation Modules into 3D Feature Fields via Volume Rendering
NeRF learns a neural field to compute the density and view-dependent color, σ ( x ) and c ( x , d ) .
We may extend NeRF by adding decoders for other quantities of interest.
For example, SemanticNeRF [112] adds a branch outputting a probability distribution of closed-set semantic labels, trained with supervision via images with ground-truth semantic labels.
This enables the prediction of pairs of RGB and semantic segmentation masks from novel views, useful for data augmentation.
However, because ground-truth annotation is costly, the method is inefficient as a means of scene editing [104].
For specific domains like traffic scenes [25, 41], we may instead train a closed-set segmentation model and use its prediction for training object-aware neural fields.
However, this approach is possible only if the types of objects are limited and the domain-specific supervised dataset is available; limiting the application of scene editing in terms of domain and flexibility of decomposition.
We build on top of these ideas and perform 3D zero-shot segmentation of NeRFs using open-set text labels or other feature queries.
Instead of a branch performing closed-set classification, we propose to add a feature branch outputting a feature vector itself.
This branch models a 3D feature field describing the semantics of each spatial point.
We supervise the feature field by a pretrained pixel-level image encoder f img as a teacher network.
Given a 3D coordinate x , the feature field outputs a feature vector f ( x ) in addition to density σ ( x ) and color c ( x , d ) , as shown in Fig.
Volume rendering of the feature field is similarly performed via
[Equation]
We can optimize f by minimizing the difference between rendered features ˆ F ( r ) and the teacher’s features f img ( I, r ) .
Effectively, we are distilling [34] the 2D teacher network into our 3D student network via differentiable rendering, and thus dub this model a distilled feature field (DFF).
We add a feature objective L f penalizing the difference between rendered features
ˆ F ( r ) and the teacher’s outputs f img ( I, r ) to the photometric loss of the original NeRF.
We use two networks for volume rendering with coarse-and-fine hierarchical sampling.
We thus minimize the sum of photometric loss
L p and feature loss L f , in total, L :
[Equation]
where R are sampled rays, C ( r ) is the ground truth pixel color of ray r , λ is the weight of the feature loss and is set to 0.04 to balance the losses [112].
We apply stop-gradient to density in rendering of features ˆ F ( r ) in Equation 3 as the teacher’s features f img ( I, r ) are not fully multi-view consistent, which could harm the quality of reconstructed geometry.
4.2 Query-based Decomposition and Editing
A trained DFF model can perform 3D zero-shot segmentation by its feature field f and a query encoder f q .
Probability of a label l of a point x in the 3D space, p ( l | x ) , is calculated by dot product of the 3D feature f ( x ) and text label feature f q ( l ) followed by a softmax:
[Equation]
This query-based segmentation field is at the core of the proposed method.
It can be calculated at any 3D point without limiting resolution, naturally used in tandem with a radiance field and volume rendering.
Note that the segmentation depends on only the 3D coordinate and the query 1 .
As the original NeRF, it is thus multi-view consistent.
In addition and important for interactive editing, we can change the segmentation via queries without re-training, which cannot be realized by closed-set methods using semantic [112] or instance segmentation annotation [104].
We may now use this query-conditional segmentation to identify a specific 3D region for editing.
Various edits can be generalized to the merging of two NeRF scenes σ 1 ( x ) , c 1 ( x , d ) and σ 2 ( x ) , c 2 ( x , d ) , where we use the segmentation field p for blending.
In the experiments section, we simply modify Eq.
1 as a blend of two scenes based on the ratio of α :
[Equation]
c 2 ( x k , d )(1 − ρ k )) , (6) where ρ k = α ( σ 1 ( x k ) δ k ) α ( σ 1 ( x k ) δ k ) + α ( σ 2 ( x k ) δ k ) , ˆ T ( t k ) =
k −(cid:89) k ′ = α ( σ 1 ( x k ′ )
δ k ′ ) + α ( σ 2 ( x k ′ ) δ k ′ ) . For example, if we want to apply a geometric transformation g to a region of a query l in a NeRF scene ( σ, c ) , we can render the transformed scene via Eqs.
6 and 7 by setting α ( σ 1 ( x k ) δ k ) = (1 − p ( l | x k ))
α ( σ ( x k ) δ k ) , α ( σ 2 ( x k ) δ k ) =
p ( l | g − 1 ( x k ))
α ( σ ( g − 1 ( x k ))
δ k ) , c 1 ( x k , d ) =
(1 − p ( l | x k ))
c ( x k , d ) , and c 2 ( x k , d ) =
p ( l | g − 1 ( x k ))
c ( g − 1 ( x k ) , g − 1 ( d )) .
More details of editing for
G r o und t r u t h f i n e
M L P c o a r
s e M L P [Table] colorization, translation, and deletion are shown in Appendix B. We can combine this with more complex edits, including optimization-based methods like CLIPNeRF [97].
While CLIPNeRF itself cannot selectively edit specific regions in multi-object scenes, our decomposition method enables it to update only desired objects without breaking unintended areas.
Experiments
We first conduct a quantitative evaluation of the decomposition achieved by DFF.
We demonstrate that DFF enables 3D semantic segmentation in a benchmark dataset using scanned point clouds with human-annotated semantic segmentation labels.
We then investigate the capabilities of DFF for editing and subsequent novel-view synthesis on real-world datasets.
We use two teacher networks,
LSeg [44] and DINO [12], which are pre-trained and publicly available.
Each training image is encoded by the image encoders of the networks and used as target feature maps, f img ( I, r ) , defined in Equation 4.
Because the feature maps are of reduced sizes due to the limitation of the networks, we first resize them to the original image size.
The implementation and settings of NeRF, unless otherwise stated, follow Zhi et al.
During the training of 200K iterations, the loss L in Equation 4 is minimized by Adam with a linearly decaying learning rate (5e-4 to 8e-5). During training, Gaussian noise for density is also applied. The number of coarse and fine samplings isand 128, respectively.
The MLP of the neural radiance field consists of eight ReLU layers withdimensions, followed by a linear layer for density, three layers for color, and three layers for feature, as shown in Fig.
Positional encoding of length 10 is used for the input coordinate and its skip connection, and that of length 4 is for viewing direction.
If an independent MLP is prepared for the feature field, it consists of four layers (with a skip connection at the third layer if the positional encoding is used).
The size of a training image is 320 × 240 for the Replica dataset and 1008 ×for the other datasets.
The batchsize of training rays is 1024 for Replica and 2048 for the others. During finetuning of feature fields or radiance fields, Gaussian noise is removed, and the learning rate is set to 1e-4.
See appendix A and C for further training details.
5.1 3D Semantic Segmentation
We construct a 3D semantic segmentation benchmark from four scenes in the Replica dataset [86] with data split and posed images provided by [112].
See appendix D for further details of the dataset.
We train DFF to reconstruct each scene with radiance and feature fields from training images and evaluate the quality of novel view synthesis and 3D segmentation of the annotated point clouds.
We use LSeg as a teacher network.
The LSeg text encoder encodes each label, and the probability of each point is calculated by Equation 2 2 .
Note that the training uses only the photometric and feature losses (Equation 4) and does not access any supervision via semantic labels.
Semantic Segmentation Results.
First, we show evaluation metrics of 3D semantic segmentation, mean intersection-over-union (mIoU) and accuracy in Table 1.
For comparison, we also experiment with a sparse 3D convolution-based segmentation model, MinkowskiNet42 [18] taking a colored [Table]
point cloud as input. It has a standard state-of-the-art architecture for point cloud segmentation and is trained on the ScanNet dataset [20], the largest annotated training dataset of 3D semantic segmentation 3 .
Results demonstrate that DFF, taught by LSeg, achieves promising performance, even better than the supervised model.
This indicates that DFF succeeds at distilling 3D semantic segmentation from the 2D teacher network.
Impact of Sampling on Semantic Segmentation.
NeRF employs two MLPs for hierarchical sampling, where the coarse MLP performs volume rendering with fewer points (64) using stratified sampling, and the fine MLP works with importance sampling (192 in total).
So, we have two sampling options to train a feature field.
Although fine sampling is critical for training accurate radiance fields, segmentation is of significantly lower spatial frequency than texture.
We thus analyze the impact of coarse and fine training in Fig.
As expected, the coarse model produces smooth segmentations, while the fine version introduces high-frequency artifacts.
This smoothness property is important for natural editable novel view synthesis and is discussed again later.
Compatibility with View Synthesis.
We also check and compare the quality of novel view synthesis with NeRF, which does not learn feature fields.
Because the feature branch partially shares the layers with the radiance field (as shown in Fig. 1), learning feature fields could possibly harm the radiance field.
Despite this concern, as shown in Tab. 2, the performance of view synthesis is not degraded.
Thus, we can train and use the branch-based DFF with small computational and parameter overhead compared to the original NeRF.
If we excessively increased the weight of the feature loss, λ × 10 , it hurt view synthesis while not improving segmentation performance further.
We further confirm that training independent, light-weight feature-field MLP, instead of a branch of the radiance-field MLP, achieves semantic segmentation results competitive with the branch-based approach (see appendix Tab. 3 for the result of all variants) 4 .
This option is useful especially when we want to introduce DFF decomposition into arbitrary 3D scene representations, including off-the-shelf NeRF models, dynamic NeRFs [26, 66, 45], or meshes, without re-training of the radiance field.
5.2 Editable Novel View Synthesis
In the previous section, we quantitatively validated the ability of DFF to perform semantic decomposition.
We now discuss the capability for editable view synthesis on real-world scenes, including the LLFF dataset [55] and our own dataset.
Our method can be used even for LLFF scenes based on normalized device coordinates.
Please see the supplemental web page for further results, including videos.
In addition to LSeg using a text query, we also experiment with self-supervised DINO [12] as another teacher network to enable query-based decomposition using image patch queries.
Here, we use thresholded cosine similarity to directly compute the probability of a query instead of softmax with negative queries in Eq. 5 and set p = 1 if the similarity exceeds the threshold 5 , and p =otherwise for hard decomposition.
We first train NeRFs without a feature branch for each scene for 200K iterations ( L p ) and then finetune them with a feature branch via distillation for 5K iterations ( L p + λL f ) since we found that the feature loss converged significantly faster than the photometric loss and short training was thus sufficient.
We use coarse sampling for training feature branches and use it for edited rendering with fine sampling.
See appendix A for the details.
interested in the feature-field MLP, it is important to simultaneously prepare the radiance-field MLP for reconstructing geometry well and training the feature field via geometrically plausible volume rendering
[Figure]
[Figure]
The edited views consistent,
Appearance Editing, Deletion, Extraction.
We show qualitative evaluations of novel view synthesis in Fig.
3 and Fig.
Specific 3D regions in these scenes are identified and locally edited via decomposition depending on various query modalities.
In these experiments, we use a text query for
LSeg-DFF as in Section 5.1 and use an image patch query for DINO-DFF.
Because DINO features capture the similarity and correspondences of regions well thanks to self-supervised learning [12, 3], image patch queries help select all semantically similar areas at once.
The patch feature is then calculated by averaging the features of all pixels in the patch.
In Fig. 3, we demonstrate that the DFF enables convincing selective appearance edits.
Because our focus is region selection via decomposition, we use simple color transformation for clarity here (e.g., flip RGB to BGR, blend colors).
One might think that the MLP of a radiance field by the original NeRF also has hidden layers, and their features could possibly be used for decomposition.
We confirm that the naive usage of NeRF features is not robust to decomposition, as shown in Fig.
5, especially in a complex multi-object scene.
We use the 8th hidden layer of the fine radiance field network (i.e., the layer just before branching in Fig. 1) 6 .
NeRF features cannot clearly decompose even objects with simple shapes and colors.
The region selections are leaked to other parts with similar colors, geometry, or positions while they do not entirely cover the targets. For example, floor qualitatively. [Figure]
↑ ↑ ↓ ↓ ↓ ↓ ↑ ↑ ↓ ↓ ↓↓ ↓ ↓ ↓ ↓ ↓ ↓
[Figure]
↑ ↑
↓ [Figure] [Figure] selection is leaked to walls, a table, bins, or ceilings.
Chair selection is leaked to irrelevant black parts like television, cables, lighting equipment, or shadows.
This indicates that the feature space of the original NeRF does not learn semantic similarity well and is entangled with unpredictable and more low-level factors like color or spatial adjacency.
In Fig.
4, we demonstrate that the DFF also works well on deletion or extraction of objects, using two patch queries (query- 1 ⃝ for leaves and ground, query- 2 ⃝ for flowers) and a text
query- 3 ⃝ “flower”.
For comparison with a baseline editing method, we show the results by a state-of-the-art image inpainting model, LaMa [87].
Because the model requires masks for inpainting regions, we manually annotate the views for evaluation.
As shown in the figure, the image inpainting model cannot generate clear and realistic images, and the different views are inconsistent.
On the other hand, DFF produces multi-view consistent plausible results, especially succeeding at extracting foreground objects.
Although the performance on deleting foreground objects is high, a remaining shortcoming is the existence of floating artifacts and blurred volumes in the far distance behind the deleted object.
Priors for Smooth Decomposition We can organize the challenges of editable NeRFs into several categories: surface decomposition, volume decomposition, lighting decomposition, and estimation of less or never observed parts.
If we edit appearances only, it practically requires decomposing regions only near the surface of objects, i.e., surface decomposition, because the color of a ray is determined mostly in a condensed interval around the surface.
On the other hand, geometric transformations often require a higher level of decomposition.
As shown in the deletion examples, geometric transformation may move or remove some surfaces and expose the space behind them.
This forces models to render unknown regions less or never observed due to occlusions, including even the inside of objects.
Thus, it is desirable to decompose volumes smoothly while synthesizing their inside and back 7 .
Although these include the same challenges as novel view synthesis tackles, editability further highlights their importance.
Apart from lighting decomposition discussed in prior work [8, 9, 111], we further investigate the new challenge of smooth volume decomposition by experimenting with different DFF setups.
As discussed in Section 5.1, DFF has two sampling options to train feature fields.
The coarse training may introduce smoothness regularization and help cohesive decomposition and smoother in-painting of unobserved regions.
Another reasonable smoothness regularizer is to eliminate the high-frequency positional encoding (PE).
We thus train an independent MLP network for a feature field without
PE.
We compare four combinations of renderings in Fig.
To better understand their behavior, we use the DINO-DFF, show k-means clusters of the rendered feature map, and delete the head of the
Triceratops by a query choosing its corresponding clusters.
As expected, coarsely trained models and no-PE models succeed in smoother volume decomposition, and this combination can minimize high-frequency floating artifacts.
A side effect is the lack of high-frequency representation power, which sometimes deletes disparate background regions and misses to represent features of complex structures (e.g., see the cluster visualization of the thin frames of the window).
Towards the best of both worlds, developing proper priors or inductive biases is an important direction for future work [70].
Otherwise, surface-aware representations like IDR [106, 98] could avoid problems with floating artifacts.
Note that not all geometric edits suffer from these problems.
For example, it is often less problematic to move objects closer to the camera, enlarge them, or warp them to other scenes, as shown in Fig.
Localizing Optimization-based Editing.
Finally, we show a combination with an optimizationbased editing method.
CLIPNeRF [97] optimizes the parameters of a radiance field so that its rendered images match with a text prompt via CLIP.
While it is mainly designed for a single-object scene of specific categories, it is possible to apply to other real-world NeRFs.
However, because it cannot control the scope of editing, a prompt like “white flower” may change the color of unintentional [Figure] targets like leaves.
Our DFF-based decomposition can upgrade such an optimization-based method to render a scene via the composition of a CLIP-optimized NeRF scene and the original NeRF scene.
We show the results in Fig.
Although the naive CLIPNeRF edits unintentional parts, our method helps it to locally edit intentional parts only.
In addition to switching rendering, we can also use the decomposition for controlling training signals during backpropagation.
The additional experiment is shown in Appendix F. These extensions broaden the application of CLIPNeRF or other optimization-based editing methods to complex scenes.
Discussion, Limitations, and Conclusions
In this work, we propose distilled feature field (DFF), a novel method of NeRF scene decomposition for selective editing.
We present quantitative evaluations of segmentation and extensive qualitative evaluations of editable novel view synthesis.
In addition to these promising results, DFF-based models will benefit from future improvements to self-supervised 2D foundation models.
We also clarify future directions on editable view synthesis through our experiments, especially for smoothness priors and estimation of unobserved regions.
Furthermore, while this work focuses on editable view synthesis, it is also intriguing to transfer DFF to other applications, including 3D registration of text queries [16, 1, 47, 4] or robot teaching [32, 80]. The limitations of the DFF framework are two-fold.
The first one is the upper bduround of the performance due to distillation.
The student model of distillation cannot largely outperform the teacher model 9 .
If the resolution of teacher encoders is low, the corresponding DFFs also becomes coarse-grained.
If the LSeg cannot understand a text query, the LSeg-DFF also cannot.
Secondly, the DFF uses volume rendering depending on the 3D reconstruction by NeRF. A NeRF model is sometimes optimized to geometrically wrong solutions (e.g., floaters). Such geometry errors of the radiance fields would make supervision to DFFs noisy. As a possible negative societal impact one might use our method for making realistic but fake content by editing NeRFs as desired.
Automatic fake detection methods may help in preventing such misuse.
NeRFs are further computation-intense, leading to high electricity usage.
Recent work on efficient
NeRFs [23, 57, 15] may alleviate this concern.
We thank Tsukasa Takagi, Toshiki Nakanishi, Hiroharu Kato and Masaaki Fukuda for their helpful feedback.
B Editing Procedure
Editing for colorization, translation, and deletion proceed as follows: (1) Sample points [ ..., x i , ... ] on a ray as usual in NeRF.
For each point, we query the DFF. We can now calculate the probability of the coordinate being matched with a set of queries as p ∈ [0 , 1] . We now define “the coordinate is selected by the query” if p is above a user-defined threshold, otherwise “not selected”. We can also use the mixture of selected and not-selected results in the proportion of p without the threshold.
If not selected , we calculate density σ ( x ) and color c ( x ) at x via the
vanilla NeRF If selected, we may apply the following transforms: (4-A) Deletion (Fig. 4):
We set the density σ ( x ) of the point to zero. (4-B) Color editing: We query the NeRF for density σ ( x ) and color c ( x ) by querying the NeRF.
The color is then edited by a colorization function b , i e., it is transformed to b ( c ( x )) (4’-C) Translation / rescaling: Geometric transformation needs another step before performing (2). We first compute a deformed point coordinate x ′ : x ′ is computed by applying the inverse of the editing transformation; that is, x ′ =
g − 1 ( x ) .
For translation, g would be a simple addition with a vector. If x ′ is selected by the query, x ′ is used instead of x for calculating color and density. If both x and x ′ are selected and have non-zero density (e.g., the boundary between the deformed apple and others in Fig. 7), we mix their colors c ( x ) and c ( x ′ ) in the ratio of their alphas at the point for simplicity.
(5)Finally, as usual, we perform volume rendering with the series of (density, color) tuples.
We investigate two teacher networks, LSeg and DINO, which are pre-trained and publicly available 10 . Each training image is encoded by the encoders of the networks and used as target feature maps, f img ( I, r ) , defined in Equation 4. Because the feature maps are of reduced sizes, we use them after resizing to the original image size via interpolation.
For LSeg, we use the official demo model, which has the ViT-L/16 image encoder and CLIP’s ViT-B/32 text encoder.
Inference follows the official script and uses multi-scale inference (scales = [0.75, 0.83, 0.92, 1.0, 1.08, 1.17, 1.25] for Replica, [0.75, 1.0, 1.25, 1.5, 1.75, 2.0, 2.25] for the others) 11 . The model is trained on seven different datasets [42], including ADE20K [115], BDD [108], Inference with large scale means inference with a zoomed and cropped image. Although it may increase the effective resolution of feature maps, it loses context information and may produce noisy features with wrong semantic understanding.
Cityscapes [19], COCO-Panoptic [46, 11], IDD [95], Mapillary Vistas [58], and SUN RGBD [84].
Possibly because the mixed dataset is biased, especially to traffic scenes, we experimentally confirmed that LSeg does not work well in out-of-distribution regions and queries. For example, small objects or parts are often not discriminative by the network. More careful training will improve its zero-shot ability in various domains and that of student neural feature fields.
For DINO, we use the extended implementation by Amir et al. It uses overlapping patches with stride 4 to enlarge the feature map’s size.
We use the feature at the 11th layer of the dino_vits8 model taking a 448x448-resized image as input. We average it with the features of the horizontally flipped image. While DINO cannot accept text queries, its feature captures more fine-grained information than LSeg’s. We visualize features of teacher networks, LSeg and DINO, in Fig.
We also visualize features of NeRF in Fig.
For visualizing feature spaces, we use scikit-learn’s sklearn.decomposition. PCA [67]. We calculate 3-dimensional PCA components using the teacher’s feature map of the first frame of the training view images, and visualize features by the components as RGB, normalized with min-max values with outlier removal.
Although the 3-dimensional
PCA cannot perfectly visualize the feature space, we show the results for reference and understanding.
As explained, although the features of both models are sufficiently view-consistent, their semantic resolution seems different. LSeg tends to show boundaries with coarse-grained categories, possibly due to the bias in the training datasets. The result will be changed if we train LSeg with other datasets focusing on fine-grained parts or regions. In contrast to LSeg, DINO produces fine-grained features where we can feel most of the original edges.
This enables a wide range of decomposition based on various queries as we confirmed in the experiments.
However, even with DINO, the finest-grained details in high resolution are still challenging to capture, e.g., the ribs of T-rex. The improvement of feature encoders will push the boundary of the decomposition quality of complex objects. D Replica Dataset Experiment We experiment distilled feature field on 3D semantic segmentation in Section 5.1 for demonstrating basic ability of segmentation of a 3D space and text queries.
Because 3D zero-shot semantic segmentation is a novel task except for some studies Michele et al.
[54], no standard benchmark datasets exist.
Furthermore, even for 3D semantic segmentation with closed label sets, there is no dataset with high-quality images, accurate point clouds 13 , and reasonable annotations.
For proof-of-concept of our work, we created a dataset from the existing Replica dataset [86].
It is a moderate-quality room reconstruction dataset with semantic segmentation labels. We use four scenes, room_0, room_1, office_3, and office_4 with posed images rendered by Zhi et al.
Because the pose trajectory was randomly generated, some images are unrealistically rendered (e.g., rendered from a camera inside furniture). We filtered such images from training and evaluation.
Near and far of NeRF follow their setting, (0.1, 10.0), except for office_3 (0.1, 15.0). Although the quality of reconstructions is better than other datasets like ScanNet [20], it still suffers from collapsed geometry, less photo-realistic appearances, and defective annotations. Some objects are difficult to predict their labels due to bad appearance, label ambiguity, or both.
Because this evaluation is not intended to measure the ability to overcome such unrealistic biases or artifacts in a dataset, we semi-automatically fix the label set of the dataset.
We first apply LSeg to predict the semantic label map of each RGB image and evaluate the accuracy against Replica’s original label map. In the evaluation, the accuracy of some labels is almost or exactly zero. We ignore points with such labels from the evaluation. In addition, noticeable label ambiguity is also fixed manually by ignoring the label or re-labeling (e.g., ‘rug’ and ‘floor’ are merged as they are intrinsically nonexclusive and often indistinguishable in the dataset).
For reference, we show a visualization of predictions by LSeg and Replica’s ground truth in
Fig. 10, and the confusion matrix in Fig.
Note that it is not guaranteed that training images cover all the regions of the scene and its point cloud.
Hence, the evaluation also measures the ability of generalization to unobserved regions via propagation.
E Ablation Experiments of Variants
We experiment with some variants of DFF architectures and show the result in Table 3.
While they could indicate different behavior for each scene, the average performance difference was marginal.
In total, using MLP trained with volume rendering of coarse sampling performs better than MLP with fine sampling based on hierarchical sampling.
The coarse sampling might implicitly regularize MLP
[Table]
and help to obtain smoothness properties. Even if increasing the weight of feature loss, λ , it did not improve the performance and hurt the quality of view synthesis [Table] F Implementation of CLIPNeRF Experiment
Because the official implementation of CLIPNeRF [97] is not available, we implement it from the description by ourselves.
In addition, the experiment for Figure 14 in their paper (i
[Table]
e., editing of the LLFF scene) unfortunately lacks significant descriptions to be required for reproduction.
Thus, there might be nuanced differences while we confirmed that our result shows similar behaviors.
We follow the description in the paper as much as possible.
For 500 iterations, we optimize the parameters of the
NeRF while freezing the sub-parameters related to density.
It minimizes negative cosine similarity between a text prompt and a rendered image from a randomly sampled pose in the training dataset, calculated by CLIP ViT-B/32, with Adam of learning rate 0.0001.
Because a naive differentiable rendering of high-resolution images is intractable due to memory constraints, we use an efficient method.
We render a patch of 128 × 128 size, followed by bilinear interpolation for resizing and feed it to CLIP.
For making the patch cover the view widely, rays in 128 × 128 patch are sampled with stride interval 5; thus, CLIP’s receptive field is 640 × 640 of the original image.
For memory efficiency, during the rendering of a patch, we reduce the number of importance sampling to 16; fine
MLP uses 64 + 16 = 80 in total.
While it is out of the scope of this paper, it is possible to further increase the patch resolution if we use a recomputation technique, where NeRF first computes depth without the need for gradient, and then, renders a patch with a few samples nearby the depth surface per ray in a differentiable manner.
For demonstrating that distilled feature field can be used with other 3D scene representations, we additionally experimented with the InstantNGP [57] and the LSeg-DFF, followed by CLIPNeRF editing. The implementation is derived from https://github.com/kwea123/ngp_pl .
For computational efficiency, we do not perform the complete process of volume rendering for feature rendering.
We first compute the pseudo depth of the surface as a weighted average of the depth of each sampled point in volume rendering. We simply compute the feature in five points around the pseudo surface point by adding small perturbations during training and inference. This method is very efficient but works for training the distilled feature field.
When optimizing the scene with CLIPNeRF, instead of backpropagating gradients to all the rays, we backpropagate the gradients to only the rays selected by “apple”.
The result of the “rainbow apple”-edit is shown in Fig.
We use 2D vision(-and-language) models as teacher networks for distillation.
In particular, for processing text queries, we use LSeg [44] as a teacher network.
We can use other models for teacher networks.
As introduced in Section 2, within a few months 14 , many studies [44, 52, 99, 103, 72] concurrently tackled image semantic segmentation tasks with similar approaches using CLIP [69]. They use the pretrained CLIP models as a text encoder and an image feature encoder in freezing or finetuning manners and train zero-shot image semantic segmentation models. Because they were concurrent, differences in various choices and performances are missing so far.
While we use LSeg by Li et al.
[44] for simplicity of implementation, it is possible to use other models, especially text-unconditional encoders, where the image encoder produces a feature map without using text labels and calculates a score map by pixel-level calculations with text labels. Although the publicly available LSeg model can accept any text queries thanks to the CLIP’s text encoder, it is not so robust to out-of-distribution data (e.g., “fossils”, “T-Rex”, “Triceratops”) due to their training strategy overfitting to the bias in its training datasets (e.g., traffic scenes). Using other concurrent models or the ongoing improvement of training and architectures [50] will alleviate such issues and improve distilled feature field.