Decomposing NeRF for Editing via Feature Field Distillation
research distilled-feature-fields
Abstract
Emerging neural radiance ﬁelds (NeRF) are a promising scene representation for computer graphics, enabling high-quality 3D reconstruction and novel view synthesis from image observations.
However, editing a scene represented by a
NeRF is challenging, as the underlying connectionist representations such as MLPs or voxel grids are not object-centric or compositional.
In particular, it has been difﬁcult to selectively edit speciﬁc regions or objects.
In this work, we tackle the problem of semantic scene decomposition of NeRFs to enable query-based local editing of the represented 3D scenes.
We propose to distill the knowledge of off-the-shelf, supervised and self-supervised 2D image feature extractors such as
CLIP-LSeg or DINO into a 3D feature ﬁeld optimized in parallel to the radiance ﬁeld.
Given a user-speciﬁed query of various modalities such as text, an image patch, or a point-and-click selection, 3D feature ﬁelds semantically decompose 3D space without the need for re-training and enable us to semantically select and edit regions in the radiance ﬁeld.
Our experiments validate that the distilled feature ﬁelds can transfer recent progress in 2D vision and language foundation models to 3D scene representations, enabling convincing 3D segmentation and selective editing of emerging neural graphics representations.
Introduction
Emerging neural implicit representations or neural ﬁelds have been shown to be a promising approach for representing a variety of signals [82, 53, 65, 106, 56].
In particular, they play an important role in 3D scene reconstruction and novel view synthesis from a limited number of context images.
Neural radiance ﬁelds (NeRF) [56] enabled the recovery of a continuous volume density and radiance ﬁeld from a limited number of observations, producing high-quality images from arbitrary views via volume rendering with promising applications in computer graphics.
However, editing a scene reconstructed by NeRF is non-obvious because the scene is not object-centric and is implicitly encoded in the weights of a connectionist representation such as an MLP [56] or a voxelgrid [23].
Although we can transform the scene in input or output space or via optimization-based editing [37, 97], this does not enable selective object-centric or semantic , local edits, such as moving a single object.
Prior work has addressed this challenge via coordinate-level, semantic decompositions which allow to selectively move, deform, paint, or optimize parts of a NeRF, but relies on costly annotation of instance segmentations and training of instance-speciﬁc networks [104].
While this can be alleviated with pre-trained segmentation models [25, 41], such models require pre-deﬁned closed label sets and 36th Conference on Neural Information Processing Systems (NeurIPS 2022). domains (e.g., trafﬁc scenes), limiting decomposition and editing.
Local editing of NeRFs ideally requires an efﬁcient, open-set method for coordinate-level decomposition.
In this work, we present distilled feature ﬁelds (DFFs), a novel approach to query-based scene decomposition for local, interactive editing of NeRFs.
We focus on 3D neural feature ﬁelds, which map every 3D coordinate to a semantic feature descriptor of that coordinate.
Conditioned on a user query such as a text or image patch, this 3D feature ﬁeld can compute a decomposition of a scene without re-training.
We train a scene-speciﬁc DFF via teacher-student distillation [34], using supervision from feature encoders pre-trained on the image domain.
Unlike the domain of 3D scenes, the image domain boasts massive high-quality datasets and abundant prior work on self-supervised and supervised training of effective feature extraction models.
Notably, recently proposed transformer-based models [96, 22] have demonstrated impressive capabilities across various vision- and text-based tasks (e.g., CLIP [69], LSeg [44], DINO [12]).
Such feature spaces capture the semantic properties of regions and make it possible to correspond and segment them well by text, image queries, or clustering.
We employ these models as teacher networks and distill them into 3D feature ﬁelds via volume rendering.
The trained feature ﬁeld enables us to semantically select and edit speciﬁc regions in 3D NeRF scenes and render multi-view consistent images from the locally edited scenes.
In extensive experiments, we investigate the applications of neural feature ﬁelds with two different pre-trained teacher networks, (1) LSeg [44], a CLIP-inspired language-driven semantic segmentation network, and (2) DINO [12, 3], a self-supervised network aware of various object boundaries and correspondences.
LSeg and DINO features allow us to select 3D regions by a simple text query or an image patch, respectively.
We ﬁrst quantitatively demonstrate that LSeg-based DFFs with label queries can have high 3D segmentation performance compared with an existing point-cloud based 3D segmentation baseline trained on ScanNet [20], a supervised point-cloud dataset.
We then demonstrate a variety of 3D appearance and geometry edits across real-world NeRF scenes with no annotations of segmentation; and show that we may edit regions with a single query of text, image, pixel, or cluster choice.
Related Work
Neural Implicit Representations.
Neural implicit representations or neural ﬁelds have recently advanced neural processing for 3D data and multi-view 2D images [82, 53, 65, 106, 56].
For a review of this emerging space we point the reader to the reports by Kato et al.
[39], Tewari et al.
[90], and Xie et al.
In particular, a neural radiance ﬁeld (NeRF) can be ﬁtted to a set of posed 2D images and maps a 3D point coordinate and a view direction to RGB color and density.
When observations are limited, NeRF often overﬁts and fails to synthesize novel views with correct geometry and appearance.
Pre-trained vision models have been used for regularizing NeRF via ﬂows [62], multi-view consistency [35], perceptual loss [110], or depth estimation [100, 77].
Some pre-trained models operate not only in the visual world but also in other modalities such as language.
The recently proposed CLIP model [69] has demonstrated impressive performance in image-and-text alignment, with strong generalization to various textual and visual concepts.
Wang et al.
[97], Jain et al.
[36], and Poole et al.
[68] use CLIP or Imagen [79] to edit or generate a single-object NeRF with a text prompt query by optimizing the NeRF parameters to generate images matched with the text.
While such methods are promising, they do not enable accurate selective editing of speciﬁc scene regions.
For example, the prompt “yellow ﬂowers” may affect unintended scene regions, such as the leaves of a plant.
Our proposed decomposition method leverages pre-trained foundation models to enable selective editing of real-world NeRF scenes.
Neural descriptor ﬁelds [80] use intermediate features that emerge in a 3D occupancy ﬁeld network [53] for efﬁciently teaching robots object grasping.
Instead of a pre-trained object-centric 3D model, we use 2D vision models as teacher networks via distillation, exploiting recent progress in pre-trained foundation models [7].
Geometric Decomposition of Neural Scene Representations Kohli et al.
[40] and Zhi et al.
[112] show that neural implicit representations can be combined with the supervision of semantic labels.
Yang et al.
[104] demonstrate that given view-consistent ground-truth instance segmentation masks during training, NeRF can be trained to represent each object as different volumes, although such an annotation is expensive in practice.
Concurrently, Benaim et al.
[6] also experiment with the different parametarization.
Conditional [49, 37, 21, 63] and generative models [60, 61, 31] enable a degree of category-speciﬁc decomposition (e.g., human bodyparts) and editing on constrained domains with large datasets.
Regular structures such as voxelgrids or octrees [13, 48, 14, 88, 94, 89, 43, 81, 107, 59, 60] or unsupervised decomposition [73, 85, 109, 83] enable editability via manipulation of localized parameters.
However, the decomposition is limited due to the inﬂexibly structured boundaries or strong assumptions about scenes; self-supervised object-centric learning is a difﬁcult task.
Other studies also explored reconstruction with more structured hybrid representations via pipelines specialized to a domain (e.g., trafﬁc scene)
[64, 25, 41] or situation (e.g., each object data is independently accessible) [28, 27, 105].
Note that this line of work deﬁnes and constrains domains or the types of segmentation during or before training and thus limits the degrees of freedom for editable scenes and objects.
In contrast, our method can decompose scene-speciﬁc NeRFs into arbitrary semantic units via text and image queries, enabling versatile scene edits without retraining.
A concurrent paper by Tschernezki et al.
[93] also explores the same training framework and, in particular, investigates how fused features are improved from 2D teacher networks.
It also complementarily shows the results with other teacher models (MoCo-v3 [17] and DeiT [91]), dimension reduction via PCA, and NeuralDiff [92]-based neural ﬁelds.
Other concurrent studies explore decomposition through training scene-speciﬁc segmentation ﬁeld [113] or 3DCNN [76] supervised by click or scribble annotations.
Lastly, in a different but related task, video editing, Kasten et al.
[38] use foreground-background decomposition and atlas representation for time-consistent, local editing; Loeschcke et al.
[51] and Bar-Tal et al.
[5] further use CLIP for editing.
Zero-shot Semantic Segmentation.
Zero-shot semantic segmentation is a challenging task [24, 2, 10] where a model has to predict semantic labels of pixels in images without a-priori information of the categories.
A typical solution is to use vision-and-language cross-modal encoders.
They are trained to encode images (pixels) and text labels into the same semantic space and perform zero-shot prediction based on the similarity or alignments of the two inputs.
Recent development of image encoder architectures [96, 22, 71] and large-scale training [69, 12] have improved the ability and generalization of vision models, including zero-shot models [44, 52, 99, 103, 114, 72].
On the other hand, ongoing studies on zero-shot perception in 3D still suffer from the lack of effective, efﬁcient, and high-resolution architectures and large-scale annotated datasets [54, 33, 29, 101, 78, 30].
Our method is a new approach to perform zero-shot semantic segmentation on scene-speciﬁc 3D ﬁelds by exploiting progress in the image domain without semantic 3D supervision.
We note that the goal of this paper is not to achieve state-of-the-art performance on 3D semantic segmentation tasks.
Instead, our goal is the decomposition of neural scene representations for editing, which requires smooth segmentation results on continuous 3D space rather than segmentation of discrete point clouds or voxelgrids.
Preliminaries 3.1 Neural Radiance Fields (NeRF)
NeRF [56] uses MLPs to output density  and color c given a point coordinate x
= ( x, y, z ) in a 3D scene.
This simple scene representation can be rendered and optimized via volume rendering.
Given a pixel’s camera ray r ( t ) =
o + t d , depth t with bounds [ t near , t far ] , camera position o , and its view direction d , NeRF calculates the color of a ray using quadrature of K sampled points { x k } Kk =1 with depths { t k } Kk =1 as
[Equation]
, (1) where ↵ ( x ) = 1  exp(  x ) , and  k = t k +1  t k is the distance between adjacent point samples.
NeRFs are optimized solely on a dataset of images and their camera poses by minimizing a rerendering loss.
3.2 Pre-trained Models and Zero-shot Segmentation of Images
Most semantic segmentation models pre-deﬁne a closed set of labels, and cannot ﬂexibly change the segmentation categories or boundaries without supervised training.
In contrast, zero-shot semantic segmentation predicts target regions given open-set queries.
Li et al.
[44] proposes LSeg, a model to
[Table]
example, we can use image or pixel 144 feature as a query f q in the same or similar formulation of Equation 3.
For example, DINO [Caron 145 et al., 2021], a self-supervised vision model, unsupervisedly solves video instance segmentation 146 (tracking) by calculating similarity among the features in adjacent frames.
Amir et al.
[2021] also 147 investigates co-segmentation and point correspondence by similarity and clustering of DINO features.
In our experiments, we use these two publicly available models, LSeg and DINO, for producing 149 image or text features of observed view images or users’ queries for 3D decomposition.
150 4 Neural Perceptual Fields 151 4.1 Learning Feature via Volume Rendering 152 ThebasicNeRFlearnsaﬁeldtocomputethedensityandtheview-dependentcolorofapoint,  ( x ) and 153 c ( x , d ) .
We can extend NeRF by additionally modeling quantities.
For example, SemanticNeRF
[Zhiet al., 2021a, Fu et al., 2022] adds a branch, whose output is a probability distribution of closed-set 155 semantic labels.
They train the branch by supervision from images with ground- semantic labels and use the model to produce more labeled images from novel views for augmenting the labeled 157 dataset.
We further extend such ideas and enable NeRF to perform 3D zero-shot segmentation using open-set 159 text labels or other feature queries.
Instead of a branch performing closed-set classiﬁcation, wepropose to add a feature branch, whose output is a feature vector.
This branch models a feature ﬁeld 161 describing some kind of characteristics of each spatial point.
We supervise the feature ﬁeld by apretrained pixel-level image encoder f img as a teacher network.
Speciﬁcally, given a 3D coordinate x , 163 the new NeRF outputs a feature vector f ( x ) in addition to density  ( x ) and color c ( x , d ) .
Volumerendering is also performed in the feature ﬁeld as follows
[Equation]
We can optimize f through SGD on minimizing the difference between rendered features
3.2 Pre-trained Models and Zero-shot Segmentation of Image 125 Zero-shot semantic segmentation is a task of predicting target regions after training without infor- 126 mation about what semantic concepts are targeted during test time.
A typical approach is training 127 encoders on large-scale datasets for obtaining generalizable features.
Recent development of self- 128 supervised learning improves the performance of such a feature-based approach.
129 Li et al.
[2022]’s LSeg employs an image feature encoder using dense prediction transformers [Ranftl 130 et al., 2021] and a text label feature encoder using CLIP [Radford et al., 2021], which are trained on 131 large-scale language-image contrastive learning.
Probability of a label l of a pixel r in an image I , 132 p ( l | I,r ) , are predicted by dot product of pixel-level image feature f img ( I,r ) and queried text label 133 feature f q ( l ) followed by softmax:p ( l | I,r ) =
exp( f img ( I,r ) f q ( l ) T ) P l 0 2 L exp( f img ( I,r ) f
q ( l 0 ) T ) , (3) where L is a set of possible labels.
We omit the temperature parameter  in softmax.
During 135 training, LSeg optimizes only the image encoder f img ( I,r ) through SGD on minimizing cross entropy 136 on supervised semantic segmentation image datasets.
The text encoder f q ( l ) is frozen from the 137 pre-trained CLIP checkpoint [Radford et al., 2021].
The pre-trained CLIP becomes a platform 138 and has been extended with further training another modules sharing the same latent space.
For 139 example, Reimers and Gurevych [2019, 2020] trains a multi-lingual (more than 50+ languages) text 140 encoder, which enables CLIP and CLIP-inspired variants to use non-English queries like Japanese 1 . 141 Becauseourworkusesthesamelatentspace, wecandecomposeNeRFwithsuchnon-Englishqueries 142 similarly.
143 Segmentation can be performed with other types of query.
For example, we can use image or pixel 144 feature as a query f q in the same or similar formulation of Equation 3.
For example, DINO [Caron 145 et al., 2021], a self-supervised vision model, unsupervisedly solves video instance segmentation tracking calculating
147 investigates co
In our experiments, we use these two publicly available models, and DINO, for producing 149 image or text observed view images or
3D
4 Neural Perceptual Fields4.1 Learning Feature via Volume Rendering
ThebasicNeRFlearnsaﬁeldtocomputethedensityandtheview-dependentcolorofapoint,  ( x ) and 153 c ( x , d ) .
We can extend NeRF by additionally modeling quantities.
For example, SemanticNeRF
[Zhiet al., 2021a, Fu et al., 2022] adds a branch, whose output is a probability distribution of closed-set 155 semantic labels.
They train the branch by supervision from images with ground-truth semantic labelsand use the model to produce more labeled images from novel views for augmenting the labeled 157 dataset.
We further extend such ideas and enable NeRF to perform 3D zero-shot segmentation using open-set 159 text labels or other feature queries.
Instead of a branch performing closed-set classiﬁcation, wepropose to add a feature branch, whose output is a feature vector.
This branch models a feature ﬁeld 161 describing some kind of characteristics of each spatial point.
We supervise the feature ﬁeld by apretrained pixel-level image encoder f img as a teacher network.
Speciﬁcally, given a 3D coordinate x , 163 the new NeRF outputs a feature vector f ( x ) in addition to density
ˆ F ( r ) =
K
X k
= ˆ T ( t k ) ↵ (  ( x k )
k )
f
( x k ) .
optimize f minimizing f
( I,
network via the volume rendering
We call
Neural
) .1 https://huggingface.co/sentence-transformers/clip-ViT-B-32-multilingual-v
Li et al
[2022]’s transformers
[Ranftl 130 et al., 2021]
Radford et al.,
I
132 p ( l | I r ) predicted by dot product of pixel feature f
( I,r ) text label 133 feature f q ( l ) by softmax:p ( l | I,r ) =
exp( f img ( I,r ) f q ( l ) T ) P l 0 2 L exp( f img ( I,r ) f
q ( l 0 ) T ) , (3) where L is a set of possible labels.
We omit the temperature parameter  in softmax.
During 135 training, LSeg optimizes only the image encoder f img (
154 et al., 2021a, Fu et al., 2022] adds a branch, whose output is a probability distribution of closed-setsemantic labels.
They train the branch by supervision from images with ground-truth semantic labels 156 and use the model to produce more labeled images from novel views for augmenting the labeleddataset.
We further extend such ideas and enable NeRF to perform 3D zero-shot segmentation using open-set 159 text labels or other feature queries.
Instead of a branch performing closed-set classiﬁcation, wepropose to add a feature branch, whose output is a feature vector.
This branch models a feature ﬁeld 161 describing some kind of characteristics of each spatial point.
We supervise the feature ﬁeld by apretrained pixel-level image encoder f img as a teacher network.
Speciﬁcally, given a 3D coordinate x , 163 the new NeRF outputs a feature vector f ( x ) in addition to density  ( x ) and color c ( x , d ) .
Volumerendering is also performed in the feature ﬁeld as followsˆ F
[Equation]
k
[Equation]
We can optimize f through SGD on minimizing the difference between rendered features
ˆ F ( r ) and 166 theteacher’soutputs f img ( I,r ) .
Thiscanbeseenasdistillationfrom2Dteachernetworkto3Dstudentnetwork via the volume rendering trick.
We call this model Neural Perceptual Fields (NePeRF) .
https://huggingface.co/sentence-transformers/clip-ViT-B-32-multilingual-v  p ( l | I,r ) =
exp( f img ( I
[Table]
r ) f q ( l ) T ) P l 0 2 L exp( f img ( I
[Table]
r ) f q ( l 0 ) T ) , (
[Table]
train the branch by supervision from images with ground-truth semantic labels 156 and use the model to produce more labeled images from novel views for augmenting the labeled 157 dataset.
158 We further extend such ideas and enable NeRF to perform 3D zero-shot segmentation using open-set 159 text labels or other feature queries.
Instead of a branch performing closed-set classiﬁcation, we 160 propose to add a feature branch, whose output is a feature vector.
This branch models a feature ﬁeld 161 describing some kind of characteristics of each spatial point.
We supervise the feature ﬁeld by a 162 pretrained pixel-level image encoder f img as a teacher network.
Speciﬁcally, given a 3D coordinate x ,the new NeRF outputs a feature vector f ( x ) in addition to density  ( x ) and color c ( x , d ) .
Volumerendering is also performed in the feature ﬁeld as follows
[Equation]
We can optimize f through SGD on minimizing the difference between rendered features
ˆ F ( r ) andthe teacher’s outputs f img ( I,r ) .
This can be seen as distillation from 2D teacher network to 3D studentnetwork via the volume rendering trick.
We call this model Neural Perceptual Fields (NePeRF) .1 https://huggingface.co/sentence-transformers/clip-ViT-B-32-multilingual-v
3. Pre-
Models and Zero-shot Segmentation of Image
Zero-shot semantic segmentation is a task of predicting target regions after training without information about what semantic concepts are targeted during test time.
A typical approach is trainingencoders on large-scale datasets for obtaining generalizable features.
Recent development of selfsupervised learning improves the performance of such a feature-based approach.
Li et al.
[2022]’s LSeg employs an image feature encoder using dense prediction transformers [Ranftlet al., 2021] and a text label feature encoder using CLIP
[Radford et al., 2021], which are trained onlarge-scale language-image contrastive learning.
Probability of a label l of a pixel r in an image I ,p ( l | I, r ) , are predicted by dot product of pixel-level image feature f img ( I, r ) and queried text labelfeature f q ( l ) followed by softmax:p ( l | I, r ) =
exp( f img ( I, r ) f q ( l ) T ) P l 0 2 L exp( f img ( I, r ) f
q ( l 0 ) T ) , (3) where L is a set of possible labels.
We omit the temperature parameter  in softmax.
Duringtraining, LSeg optimizes only the image encoder f img ( I, r ) through SGD on minimizing cross entropyon supervised semantic segmentation image datasets.
The text encoder f q ( l ) is frozen from thepre-trained CLIP checkpoint [Radford et al., 2021].
The pre-trained CLIP becomes a platformand has been extended with further training another modules sharing the same latent space.
Forexample, Reimers and Gurevych [2019, 2020] trains a multi-lingual (more than 50+ languages) textencoder, which enables CLIP and CLIP-inspired variants to use non-English queries like Japanese 1 .
Because our work uses the same latent space, we can decompose NeRF with such non-English queriessimilarly.
Segmentation can be performed with other types of query.
For example, we can use image or pixelfeature as a query f q in the same or similar formulation of Equation 3.
For example, DINO [Caronet al., 2021], a self-supervised vision model, unsupervisedly solves video instance segmentation(tracking) by calculating similarity among the features in adjacent frames.
Amir et al.
[2021] alsoinvestigates co-segmentation and point correspondence by similarity and clustering of DINO features.
In our experiments, we use these two publicly available models, LSeg and DINO, for producingimage or text features of observed view images or users’ queries for 3D decomposition.
Neural Perceptual
Fields
4. Learning Feature via Volume Rendering
The basic NeRF learns a ﬁeld to compute the density and the view-dependent color of a point,  ( x ) andc ( x , d ) .
We can extend NeRF by additionally modeling quantities.
For example, SemanticNeRF
[Zhiet al., 2021a, Fu et al., 2022] adds a branch, whose output is a probability distribution of closed-setsemantic labels.
They train the branch by supervision from images with ground-truth semantic labelsand use the model to produce more labeled images from novel views for augmenting the labeleddataset.
We further extend such ideas and enable NeRF to perform 3D zero-shot segmentation using open-settext labels or other feature queries.
Instead of a branch performing closed-set classiﬁcation, wepropose to add a feature branch, whose output is a feature vector.
This branch models a feature ﬁelddescribing some kind of characteristics of each spatial point.
We supervise the feature ﬁeld by apretrained pixel-level image encoder f img as a teacher network.
Speciﬁcally, given a 3D coordinate x ,the new NeRF outputs a feature vector f ( x ) in addition to density  ( x ) and color c ( x , d ) .
Volumerendering is also performed in the feature ﬁeld as follows
[Equation]
We can optimize f through SGD on minimizing the difference between rendered features
ˆ F ( r ) andthe teacher’s outputs f img ( I, r ) .
This can be seen as distillation from 2D teacher network to 3D studentnetwork via the volume rendering trick.
We call this model Neural Perceptual Fields (NePeRF) .
https://huggingface.co/sentence-transformers/clip-ViT-B-32-multilingual-v  supervision could be sparse or noisy.
Yang et al.
[2021] demonstrated that, given view-consistent 80 ground-truth instance segmentation masks during training, NeRF could decompose each object [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
3 Preliminaries
[Figure]
[Equation] [Equation]
[Equation] [Equation]
↵ (  ( x k )  k ) c ( x k , d ) , (1)
ˆ T ( t k ) =
exp
k
X k
=
( x k 0 )  k 0 !
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
We further extend such ideas and enable NeRF to perform 3D zero-shot segmentation using open-set 159 text labels or other feature queries.
Instead of a branch performing closed-set classiﬁcation, wepropose to add a feature branch, whose output is a feature vector.
This branch models a feature ﬁeld 161 describing some kind of characteristics of each spatial point.
We supervise the feature by
[Table]
pretrained pixel
[Table]
level image encoder img as
[Table]
teacher network.
Speciﬁcally, given a 3D coordinate x , 163 the new NeRF outputs a feature vector f ( x ) in addition to density  ( x ) and color c ( x , d ) .
Volumerendering is also performed in the feature ﬁeld as follows
[Equation]
We can optimize f through SGD on minimizing the difference between rendered features
ˆ F ( r ) and 166 the teacher’s outputs f img ( I,r ) .
This can be seen as distillation from 2D teacher network to 3D studentnetwork via the volume rendering trick.
We call this model Neural Perceptual Fields (NePeRF) .
[Figure]
We follow the training objective of the original NeRF [Mildenhall et al., 2020] and add a new objectivefor minimizing minimizing the difference between rendered features
ˆ F ( r ) and the teacher’s outputsf img ( I,r ) .
We use two networks for volume rendering with coarse-and-ﬁne hierarchical sampling as 171 well as the original NeRF does.
We simultaneously train each network from scratch by minimizingphotometric loss L c and feature loss L f :
[Equation]
where R are sampled rays, and C ( r ) is the ground truth color of pixel ray r .
Total training loss L is
[Equation]
6) where  is the weight of the feature loss and is set to 0.04 to balance the losses [Zhi et al., 2021a].
We apply stop-gradient to density during rendering features ˆ F ( r ) in Equation 4, because multi-view 176 inconsistent supervision with noise could harm reconstruction quality of geometry, although theeffect seems negligible in preliminary experiments.
4.2 Query-based Decomposition and Editing
After training a NePeRF model, we can perform 3D zero-shot segmentation by directly using the 180 feature ﬁeld f and the pretrained text encoder f q .
Speciﬁcally, probability of a label l of a point x inthe 3D space, p ( l | x ) , are predicted by dot product of the 3D feature f ( x ) and text label feature f q ( l ) 182 followed by softmax:
[Equation]
When we use multiple queries, L = { l 1 ,... } , for selecting a group of regions, we use p ( L| x k ) =
P l 0 2 L p ( l 0 | x k ) .
This segmentation information is the fruits of the proposed PeNeRF framework.
185 It is calculated at any 3D points without resolution limitation, and naturally used together with aradiance ﬁeld and volume rendering.
Note that the segmentation depends on only the 3D coordinate 187 and the query 2 , so
it and view synthesis with it are 3D consistent as well as the original NeRF.
Unlikethe proposed method, editing multiple synthesized images by image-based postprocessing breaks 189 3D consistency.
In addition, we can change the segmentation by changing only the query withoutretraining, which cannot be realized by existing methods using closed-set semantic segmentation [Zhi 191 et al., 2021a] or instance segmentation annotation [Yang et al., 2021], but important for user-friendlyinteractive {+editing.
+} It is (e.g., “the chair left to the where
R are sampled rays and C ( r ) is the ground truth color of pixel ray r ,  is the weight of thefeature loss and is set to 0.04 to balance the losses [Zhi et al., 2021a].
We apply stop-gradient todensity during rendering features ˆ F ( r ) in Equation 3, because teacher’s features f img ( I, r ) are notcompletely multi-view consistent, which could harm reconstruction quality of geometry.
4.2 Query-based Decomposition and Editing
After training a NePeRF model we can perform 3D zero-shot segmentation by directly using thefeature ﬁeld f and another query encoder f q .
Speciﬁcally, probability of a label l of a point x in the3D space, p ( l | x ) , are predicted by dot product of the 3D feature f ( x ) and text label feature f q ( l )followed by softmax:
[Equation]
When we use multiple queries, M = { l 1 , l 2 , ... } , for selecting a group of regions, we use p ( M| x k ) =
P l 0 2 M p ( l 0 | x k ) .
This segmentation information is the fruits of the proposed DFF framework.
It 209 can be calculated at any 3D point without limiting resolution, so naturally used together with aradiance ﬁeld and volume rendering.
Note that the segmentation depends on only the 3D coordinateand the query 1 , so it and view synthesis with it are 3D consistent as well as the original NeRF.
Unlike the proposed method, editing multiple synthesized images by image-based postprocessingbreaks 3D consistency.
In addition, importantly for user-friendly interactive editing, we can change1 It is an interesting direction to introduce view dependency to the segmentation for discriminating viewdependent query like referring expressions (e.g., “the chair left to the table”)
[Chen al., 2020, Achlioptas al., 2020, Liu
., 2021a, Azuma et
., 2022], but future work.
of x a color/feature and ground-truth color/teacher’s feature, respectively.
We can optimize f through SGD on minimizing the difference between rendered features
ˆ F ( r ) andthe teacher’s outputs f img ( I, r ) .
This can be seen as distillation from 2D teacher network to 3Dstudent network via the volume rendering trick.
We call this model distilled feature ﬁeld (DFF).
We follow the original NeRF [Mildenhall et al., 2020] for the training objective and the volumerendering strategy.
In addition to the photometric loss, we add a new objective for minimizing thedifference between rendered features ˆ F ( r ) and the teacher’s outputs f img ( I, r ) .
For volume rendering,we use two networks for volume rendering with coarse-and-ﬁne hierarchical sampling as well as theoriginal NeRF does.
We simultaneously train each network from scratch by minimizing photometricloss L p and feature loss L f , in total, L :
L =
L p +  L f ,
L p =
X r 2 R  ˆ C ( r )  C ( r )  2 2 ,
L f = X r 2 R  ˆ F ( r )  f img ( I, r )  1 , (4) where R are sampled rays, and C ( r ) is the ground truth color of pixel ray r ,  is the weight of thefeature loss and is set to 0.04 to balance the losses [Zhi et al., 2021a].
We apply stop-gradient todensity during rendering features ˆ F ( r ) in Equation 3, because teacher’s features f img ( I, r ) are not
After training a NePeRF model, we can perform 3D zero-shot segmentation by directly using the feature ﬁeld f and another query encoder f q .
Speciﬁcally, probability of a label l of a point x in the3D space, p ( l | x ) , are predicted by dot product of the 3D feature f ( x ) and text label feature f q ( l )followed by softmax:
[Equation]
When we use multiple queries,
M = { l 1 , l 2 , ... } , for selecting a group of regions, we use p ( M| x k ) =
P l 0 2 M p ( l 0 | x k ) .
This segmentation information is the fruits of the proposed DFF framework.
Itcan be calculated at any 3D point without limiting resolution, so naturally used together with aradiance ﬁeld and volume rendering.
Note that the segmentation depends on only the 3D coordinateand the query 1 , so it and view synthesis with it are 3D consistent as well as the original NeRF.
Unlike the proposed method, editing multiple synthesized images by image-based postprocessingbreaks 3D consistency.
In addition, importantly for user-friendly interactive editing, we can change1 It is an interesting view the segmentation for discriminating view-
.g. [Chen et al , 2020, Achlioptas et al., 2020, Liu et al., 2021a, Azuma et al., 2022],
[Table]
[Table]
p ( l | I, r ) = exp( f img ( I, r ) f q ( l ) T ) P l 0 2 L exp( f img ( I, r ) f q ( l 0 ) T ) , (3) where L is a set of possible labels.
We omit the temperature parameter  in softmax.
During 135 training, LSeg optimizes only the image encoder f img ( I, r ) through SGD on minimizing cross entropy 136 on supervised semantic segmentation image datasets.
The text encoder f q ( l ) is frozen from the 137 pre-trained CLIP checkpoint [Radford et al., 2021].
The pre-trained CLIP becomes a platform 138 and has been extended with further training another modules sharing the same latent space.
For 139 example, Reimers and Gurevych [2019, 2020] trains a multi-lingual (more than 50+ languages) text 140 encoder, which enables CLIP and CLIP-inspired variants to use non-English queries like Japanese 1 .
Because our work uses the same latent space, we can decompose NeRF with such non-English queries 142 similarly.
143 Segmentation can be performed with other types of query.
For example, we can use image or pixel 144 feature as a query f q in the same or similar formulation of Equation 3.
For example, DINO [Caron 145 et al., 2021], a self-supervised vision model, unsupervisedly solves video instance segmentation 146 (tracking) by calculating similarity among the features in adjacent frames.
Amir et al.
[2021] also 147 investigates co-segmentation and point correspondence by similarity and clustering of DINO features.
In our experiments, we use these two publicly available models, LSeg and DINO, for producing 149 image or text features of observed view images or users’ queries for 3D decomposition.
Neural Perceptual Fields 151 4.1 Learning
Volume Rendering
The basic NeRF learns a ﬁeld to compute the density and the view-dependent color of a point,  ( x ) and 153 c ( x , d ) .
We can extend NeRF by additionally modeling quantities.
For example, SemanticNeRF
[Zhi 154 et al., 2021a, Fu et al., 2022] adds a branch, whose output is a probability distribution of closed-setsemantic labels.
They train the branch by supervision from images with ground-truth semantic
We further extend such ideas and enable NeRF to perform 3D zero-shot segmentation using open-settext labels or other feature queries.
Instead of a branch performing closed-set classiﬁcation, wepropose to add a feature branch, whose output is a feature vector.
This branch models a feature ﬁelddescribing some kind of characteristics of each spatial point.
We supervise the feature ﬁeld by apretrained pixel-level image encoder f img as a teacher network.
Speciﬁcally, given a 3D coordinate x ,the new NeRF outputs a feature vector f ( x ) in addition to density  ( x ) and color c ( x , d ) .
Volumerendering is also performed in the feature ﬁeld as follows
[Equation]
We can optimize f through SGD on minimizing the difference between rendered features
ˆ F ( r ) andthe teacher’s outputs f img ( I, r ) .
This can be seen as distillation from 2D teacher network to 3D studentnetwork via the volume rendering trick.
We call this model Neural Perceptual Fields (NePeRF) .
[Figure]
outputs f img distillation from 2D teacher network to 3Dstudent network via the volume rendering trick. We call this model distilled feature ﬁeld (DFF).
We follow the original NeRF [Mildenhall et al., 2020] for the training objective and the volumerendering strategy.
In addition to the photometric loss, we add a new objective for minimizing thedifference between rendered features ˆ F ( r ) and the teacher’s outputs f img ( I, r ) .
For volume rendering,we use two networks for volume rendering with coarse-and-ﬁne hierarchical sampling as well as theoriginal NeRF does.
We simultaneously train each network from scratch by minimizing photometricloss L p and feature loss L f , in total, L :
L =
L p +  L f , L p =
X r 2 R  ˆ C ( r )  C ( r )  2 2 , L f = X r 2 R  ˆ F ( r )  f img ( I, r )  1 , (4) where R are sampled rays, and C ( r ) is the ground truth color of pixel ray r ,  is the weight of thefeature loss and is set to 0.04 to balance the losses [Zhi et al., 2021a].
We apply stop-gradient to in Equation 3, because teacher’s features f img ( I, r ) are notcompletely multi-view consistent, which could harm reconstruction quality
Query-based Decomposition and Editing
After training a NePeRF model, we can perform 3D zero-shot segmentation by directly using thefeature ﬁeld f and another query encoder f q .
Speciﬁcally, probability of a label l of a point x in the3D space, p ( l | x ) , are predicted by dot product of the 3D feature f ( x ) and text label feature f q ( l )followed by softmax:
[Equation]
When we use multiple queries, M = { l 1 , l 2 , ... } , for selecting a group of regions, we use p ( M| x k ) =
[Equation]
p ( l 0 | x k ) .
This segmentation information is the fruits of the proposed DFF framework.
Itcan be calculated at any 3D point without limiting resolution, so naturally used together with aradiance ﬁeld and volume rendering.
Note that the segmentation depends on only the 3D coordinateand the query 1 , so it and view synthesis with it are 3D consistent as well as the original NeRF.
Unlike the proposed method, editing multiple synthesized images by image-based postprocessingbreaks 3D consistency.
In addition, importantly for user-friendly interactive editing, we can change1 It is an interesting direction to introduce view dependency to the segmentation for discriminating viewdependent query like referring expressions (e.g., “the chair left to the table”)
[Chen et al., 2020, Achlioptas et al., 2020, Liu et al., 2021a, Azuma et al., 2022], but left for future work.
We follow the original NeRF [Mildenhall et al., 2020] for the and rendering strategy.
In addition to the photometric loss, we add a new objective for minimizing thedifference between rendered features ˆ F ( r ) and the teacher’s outputs f img ( I, r ) .
For volume rendering,we use two networks for volume rendering with coarse-and-ﬁne hierarchical sampling as well as theoriginal NeRF does.
We simultaneously train each network from scratch by minimizing photometricloss L p and loss L f , in total, L :
L =
L p +  L f ,
L p =
X r 2 R  ˆ C ( r )  C ( r )  2 2 ,
L f = X r 2 R  ˆ F ( r )  f img ( I, r )  1 , (4) where R are sampled rays, and C ( r ) is the ground truth color of pixel ray r ,  is the weight of thefeature loss and is set to 0.04 to balance the losses [Zhi et al., 2021a].
We apply stop-gradient todensity during rendering features ˆ F ( r ) in Equation 3, because teacher’s features f img ( I, r ) are notcompletely multi-view consistent, which could harm reconstruction quality of geometry.
Query-based Decomposition and Editing
After training a NePeRF model, we can perform 3D zero-shot segmentation by directly using thefeature ﬁeld f and another query encoder f q .
Speciﬁcally, probability of a label l of a point x in the3D space, p ( l | x ) , are predicted by dot product of the 3D feature f ( x ) and text label feature f q ( l )followed by softmax:
[Equation]
When we use multiple queries, M = { l 1 , l 2 , ... } , for selecting a group of regions, we use p ( M| x k ) =
P l 0 2 M p ( l 0 | x k ) .
This segmentation information is the fruits of the proposed DFF framework.
Itcan be calculated at any 3D point without limiting resolution, so naturally used together with aradiance ﬁeld and volume rendering.
Note that the segmentation depends on only the 3D coordinateand the query 1 , so it and view synthesis with it are 3D consistent as well as the original NeRF.
Unlike the proposed method, editing multiple synthesized images by image-based postprocessingbreaks 3D consistency.
In addition, importantly for user-friendly interactive editing, we can change
[Table]
1 It is an interesting direction to introduce view dependency to the segmentation for discriminating viewdependent query like referring expressions (e.g., “the chair left to the table”)
[Chen et al., 2020, Achlioptas et al., 2020, Liu et al., 2021a, Azuma et al., 2022], but left for future work. [Table]
In our experiments,
[Table]
use
[Table]
publicly
[Table]
models LSeg and DINO, for producing NeRF learns a neural ﬁeld to compute the density and the view-dependent color of a point,  ( x ) 159 and c ( x , d ) .
We may extend NeRF by adding decoders for other quantities of interest.
For example,
SemanticNeRF [86] adds a branch outputting a probability distribution of closed-set semantic labels, 161 trained with supervision via images with ground-truth semantic labels.
This enables prediction ofsemantic segmentation masks from novel views and use the model to produce more labeled images 163 from novel views for augmenting the labeled dataset.
Because ground-truth annotation is costly, themethod is practically inefﬁcient as a means of scene editing [81].
For speciﬁc domains like trafﬁc 165 scenes [20, 32], instead of ground truth, we could train a closed-set segmentation model and use itsprediction for training object-aware neural ﬁelds.
However, the method is possible only if types of 167 objects are limited and the domain-speciﬁc supervised dataset is available; it limits the application ofscene editing in terms of domain and ﬂexibility of decomposition.
We further extend such ideas and enable NeRF to perform 3D zero-shot segmentation using open-set 170 text labels or other feature queries.
Instead of a branch performing closed-set classiﬁcation, wepropose to add a feature branch outputting a feature vector itself.
This branch models a 3D feature 172 ﬁeld describing semantics of each spatial point.
We supervise the feature ﬁeld by a pretrained pixellevel image encoder f img as a teacher network.
Speciﬁcally, given a 3D coordinate x , the new NeRF 174 outputs a feature vector f ( x ) in addition to density  ( x ) and color c ( x , d ) , as shown in Figure 1.
Volume rendering is performed similarly in the feature ﬁeld as follows
[Equation]
We can optimize f through SGD on minimizing the difference between rendered features
ˆ F ( r ) and 177 the teacher’s outputs f img ( I,r ) .
This can be seen as distillation from 2D teacher network to 3Dstudent network via the volume rendering trick.
We call this model distilled feature ﬁeld (DFF).
We follow the original NeRF [46] for the training objective and the volume rendering strategy.
In 180 addition to the photometric loss, we add a new objective for minimizing the difference betweenrendered features ˆ F ( r ) and the teacher’s outputs f img ( I,r ) .
For volume rendering, we use two 182 networks for volume rendering with coarse-and-ﬁne hierarchical sampling as well as the original
NeRF does.
We simultaneously train each network from scratch by minimizing photometric loss L p 184 and feature loss L f , in total, L :
[Equation]
where L is a set of possible labels.
If negative labels are not available, we may use other scores like 139 thresholded cosine similarity to directly compute the probability of a label.
During training, LSeg 140 optimizes only the image encoder f img ( I,r ) through SGD on minimizing cross entropy on supervised 141 semantic segmentation image datasets.
The text encoder f q ( l ) is obtained from a pre-trained CLIP 142 model [56].
Recently, pre-trained CLIP has been leveraged as a backbone for a variety of tasks and 143 has been extended with further training modules sharing the same latent space.
For example, Reimers 144 and Gurevych [61, 62] trains a multi-lingual (more than 50+ languages) text encoder, which enables 145 CLIP and CLIP-inspired variants to use non-English queries like Japanese.
In this work, we similarly 146 use the latent space of a pre-trained CLIP, enabling decomposition of NeRFs with both English and 147 non-English queries.
148 Segmentation can further be performed with other types of queries.
For example, we can use 149 image, patch or pixel features as a query f
q using a similar dot-product similarity formulation as 150 in Eq.
Notably, DINO [7], a self-supervised vision model, unsupervisedly solves video instance 151 segmentation (tracking) well by calculating similarity among the features in adjacent frames.
Amir 152 etal.[3]alsodemonstratethatDINOfeaturesworkwellonco-segmentationandpointcorrespondence 153 by similarity and clustering.
In our experiments, we use these two publicly available models, LSeg and DINO, for producing 155 image or text features of observed view images or users’ queries for 3D decomposition.
156 4 Neural Perceptual Fields 157 4.1 Distilling Foundation Modules into 3D Feature Fields via Volume Rendering 158 NeRF learns a neural ﬁeld to compute the density and the view-dependent color of a point,  ( x ) 159 and c ( x , d ) .
We may extend NeRF by adding decoders for other quantities of interest.
For example, 160 SemanticNeRF [86] adds a branch outputting a probability distribution of closed-set semantic labels, 161 trained with supervision via images with ground-truth semantic labels.
This enables prediction ofsemantic segmentation masks from novel views and use the model to produce more labeled images 163 from novel views for augmenting the labeled dataset.
Because ground-truth annotation is costly, themethod is practically inefﬁcient as a means of scene editing [81].
For speciﬁc domains like trafﬁc 165 scenes [20, 32], instead of ground truth, we could train a closed-set segmentation model and use itsprediction for training object-aware neural ﬁelds.
However, the method is possible only if types of 167 objects are limited and the domain-speciﬁc supervised dataset is available; it limits the application ofscene editing in terms of domain and ﬂexibility of decomposition.
We further extend such ideas and enable NeRF to perform 3D zero-shot segmentation using open-set 170 text labels or other feature queries.
Instead of a branch performing closed-set classiﬁcation, wepropose to add a feature branch outputting a feature vector itself.
This branch models a 3D feature 172 ﬁeld describing semantics of each spatial point.
We supervise the feature ﬁeld by a pretrained pixellevel image encoder f img as a teacher network.
Speciﬁcally, given a 3D coordinate x , the new NeRF 174 outputs a feature vector f ( x ) in addition to density  ( x ) and color c ( x , d ) , as shown in Figure 1.
Volume rendering is performed similarly in the feature ﬁeld as follows
[Equation]
We can optimize f through SGD on minimizing the difference between rendered features
ˆ F ( r ) and 177 the teacher’s outputs f img ( I,r ) .
This can be seen as distillation from 2D teacher network to 3Dstudent network via the volume rendering trick.
We call this model distilled feature ﬁeld (DFF).
We follow the original NeRF [46] for the training objective and the volume rendering strategy.
In 180 addition to the photometric loss, we add a new objective for minimizing the difference betweenrendered features ˆ F ( r ) and the teacher’s outputs f img ( I,r ) .
For volume rendering, we use two 182 networks for volume rendering with coarse-and-ﬁne hierarchical sampling as well as the original
NeRF does.
We simultaneously train each network from scratch by minimizing photometric loss L p 184 and feature loss L f , in total, L :
[Equation]
where L is a set of possible labels.
If negative labels are not available, we may use other scores like 139 thresholded cosine similarity to directly compute the probability of a label.
During training, LSeg 140 optimizes only the image encoder f img ( I,r ) through SGD on minimizing cross entropy on supervised 141 semantic segmentation image datasets.
The text encoder f q ( l ) is obtained from a pre-trained CLIP 142 model [56].
Recently, pre-trained CLIP has been leveraged as a backbone for a variety of tasks and 143 has been extended with further training modules sharing the same latent space.
For example, Reimers 144 and Gurevych [61, 62] trains multi lingual (more than languages text
, which enables CLIP to work, we similarly 146 use further be
For example, we can use , patch or pixel dot as
, DINO
Amir 152 etal similarity our use these two publicly models, producing
4 Neural Perceptual Fields
NeRF learns a neural ﬁeld to compute the density and the view-dependent color of a point,  ( x ) 159 and c ( x , d ) .
We may extend NeRF by adding decoders for other quantities of interest.
For example,
SemanticNeRF [86] adds a branch outputting a probability distribution of closed-set semantic labels with supervision via images with ground-truth semantic labels.
This enables prediction ofsemantic masks from novel views and use the model to produce more labeled images 163 from novel views for augmenting the labeled
Because ground-truth annotation is costly, themethod is practically as a means of scene editing
For domains like 165 scenes [20, 32], instead of ground truth, we could train a closed-set segmentation model and use itsprediction for training object-aware neural ﬁelds.
However, the method is possible only if types of 167 objects are limited and the domain-speciﬁc supervised dataset is available; it limits the application ofscene editing in terms of domain and ﬂexibility of decomposition.
We further extend such ideas and enable NeRF to perform 3D zero-shot segmentation using open-set 170 text labels or other feature queries.
Instead of a branch performing closed-set classiﬁcation, wepropose to add a feature branch outputting a feature vector itself.
This branch models a 3D feature 172 ﬁeld describing semantics of each spatial point.
We supervise the feature ﬁeld by a pretrained pixellevel image encoder f img as a teacher network.
Speciﬁcally, given a 3D coordinate x , the new NeRF 174 outputs a feature vector f ( x ) in addition to density  ( x ) and color c ( x , d ) , as shown in Figure 1.
Volume rendering is performed similarly in the feature ﬁeld as follows
[Equation]
We can optimize f through SGD on minimizing the difference between rendered features
ˆ F ( r ) and 177 the teacher’s outputs f img ( I,r ) .
This can be seen as distillation from 2D teacher network to 3Dstudent network via the volume rendering trick.
We call this model distilled feature ﬁeld (DFF).
We follow the original NeRF [46] for the training objective and the volume rendering strategy.
In 180 addition to the photometric loss, we add a new objective for minimizing the difference betweenrendered features ˆ F ( r ) and the teacher’s outputs f img ( I,r ) .
For volume rendering, we use two 182 networks for volume rendering with coarse-and-ﬁne hierarchical sampling as well as the original
NeRF does.
We simultaneously train each network from scratch by minimizing photometric loss L p 184 and feature loss L f , in total, L :
[Equation]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
143 Segmentation can be performed with other types of query.
For example, we can use image or pixel 144 feature as a query f q in the same or similar formulation of Equation 3.
For example, DINO [Caron 145 et al., 2021], a self-supervised vision model, unsupervisedly solves video instance segmentation 146 (tracking) by calculating similarity among the features in adjacent frames.
Amir et al.
[2021] also 147 investigates co-segmentation and point correspondence by similarity and clustering of DINO features.
In our experiments, we use these two publicly available models, LSeg and DINO, for producing 149 image or text features of observed view images or users’ queries for 3D decomposition.
Neural Perceptual Fields 151 4.1 Learning Feature via Volume Rendering
The basic NeRF learns a ﬁeld to compute the density and the view-dependent color of a point,  ( x ) and 153 c ( x , d ) .
We can extend NeRF by additionally modeling quantities.
For example, SemanticNeRF
[Zhiet al., 2021a, Fu et al., 2022] adds a branch, whose output is a probability distribution of closed-setsemantic labels.
They train the branch by supervision from images with ground- labelsand use the model to produce more labeled images from novel views for augmenting the labeleddataset.
We further extend such ideas and enable NeRF to perform 3D zero-shot segmentation using open-settext labels or other feature queries.
Instead of a branch performing closed-set classiﬁcation, wepropose to add a feature branch, whose output is a feature vector.
This branch models a feature ﬁelddescribing some kind of characteristics of each spatial point.
We supervise the feature ﬁeld by apretrained pixel-level image encoder f img as a teacher network.
Speciﬁcally, given a 3D coordinate x ,the new NeRF outputs a feature vector f ( x ) in addition to density  ( x ) and color c ( x , d ) .
Volumerendering is also performed in the feature ﬁeld as follows
[Equation]
We can optimize f through SGD on minimizing the difference between rendered features
ˆ F ( r ) andthe teacher’s outputs f img ( I, r ) .
This can be seen as distillation from 2D teacher network to 3D studentnetwork via the volume rendering trick.
We call this model Neural Perceptual Fields (NePeRF) .
-v
[Table]
latent space. For 139 example, Reimers Gurevych [2019, 2020] trains multi-lingual more than 50+ languages) text 140 encoder, enables CLIP and CLIP-inspired variants to use non-English queries[-like Japanese 1 . 
-]
Japanese 1 . 141 Because our work uses the same latent space, we can decompose NeRF with such non-English queries 142 similarly.
can be performed with other types of query.
For example, we can use image or pixel a query f q in the same or similar formulation of Equation 3.
For example, DINO [Caron 145 et al., 2021], a self-supervised vision model, unsupervisedly solves video instance segmentation ) by calculating similarity among features in adjacent frames.
Amir et al.
[2021] also 147 investigates co- and correspondence similarity and clustering of DINO features.
In our experiments, we use these two publicly available models, LSeg and DINO, for producing 149 image or text features of observed view images or users’ queries for 3D decomposition.
Fields Volume Rendering
The basic NeRF learns a to compute the density and -dependent of a point,  x ) and
et al., 2021a, Fu et al., 2022] adds a branch, whose output is a probability distribution of closed-setsemantic labels.
They train the branch by supervision from images with ground-truth semantic labels 156 and use the model to produce more labeled images from novel views for augmenting the labeleddataset.
We further extend such ideas and enable NeRF to perform 3D zero-shot segmentation using open-set 159 text labels or other feature queries.
Instead of a branch performing closed-set classiﬁcation, wepropose to add a feature branch, whose output is a feature vector.
This branch models a feature ﬁeld 161 describing some kind of characteristics of each spatial point.
We supervise the feature ﬁeld by apretrained pixel-level image encoder f img as a teacher network.
Speciﬁcally, given a 3D coordinate x , 163 the new NeRF outputs a feature vector f ( x ) in addition to density  ( x ) and color c ( x , d ) .
Volumerendering is also performed in the feature ﬁeld as follows
[Equation]
We can optimize f through SGD on minimizing the difference between rendered features
ˆ F ( r ) andthe teacher’s outputs f img ( I,r ) .
This can be seen as distillation from 2D teacher network to 3D student 167 network via the volume rendering trick.
We call this model Neural Perceptual Fields (NePeRF) .
https://huggingface.co/sentence-transformers/clip-ViT-B-32-multilingual-v  where L is a set of possible labels.
If negative labels are not available, we may use other scores likethresholded cosine similarity to directly compute the probability of a label.
During training, LSegoptimizes only the image encoder f img ( I, r ) through SGD on minimizing cross entropy on supervisedsemantic segmentation image datasets.
The text encoder f q ( l ) is obtained from a pre-trained CLIPmodel [56].
Recently, pre-trained CLIP has been leveraged as a backbone for a variety of tasks andhas been extended with further training modules sharing the same latent space.
For example, Reimersand Gurevych [61, 62] trains a multi-lingual (more than 50+ languages) text encoder, which enables
CLIP and CLIP-inspired variants to use non-English queries like Japanese.
In this work, we similarlyuse the latent space of a pre-trained CLIP, enabling decomposition of NeRFs with both English andnon-English queries.
Segmentation can further be performed with other types of queries.
For example, we can useimage, patch or pixel features as a query
f q using a similar dot-product similarity formulation asin Eq. 2.
Notably, DINO [7], a self-supervised vision model, unsupervisedly solves video instancesegmentation (tracking) well by calculating similarity among the features in adjacent frames.
Amiret al.
[3] also demonstrate that DINO features work well on co-segmentation and point correspondenceby similarity and clustering.
In our experiments, we use these two publicly available models, LSeg and DINO, for producingimage or text features of observed view images or users’ queries for 3D decomposition.
Neural Perceptual Fields
Distilling Foundation Modules into 3D Feature Fields via Volume Rendering
NeRF learns a neural ﬁeld to compute the density and the view-dependent color of a point,  ( x )and c ( x , d ) .
We may extend NeRF by adding decoders for other quantities of interest.
For example,
SemanticNeRF [86] adds a branch outputting a probability distribution of closed-set semantic labels,trained with supervision via images with ground-truth semantic labels.
This enables prediction ofsemantic segmentation masks from novel views and use the model to produce more labeled imagesfrom novel views for augmenting the labeled dataset.
Because ground-truth annotation is costly, themethod is practically inefﬁcient as a means of scene editing [81].
For speciﬁc domains like trafﬁcscenes [20, 32], instead of ground truth, we could train a closed-set segmentation model and use itsprediction for training object-aware neural ﬁelds.
However, the method is possible only if types ofobjects are limited and the domain-speciﬁc supervised dataset is available; it limits the application ofscene editing in terms of domain and ﬂexibility of decomposition.
We further extend such ideas and enable NeRF to perform 3D zero-shot segmentation using open-settext labels or other feature queries.
Instead of a branch performing closed-set classiﬁcation, wepropose to add a feature branch outputting a feature vector itself.
This branch models a 3D featureﬁeld describing semantics of each spatial point.
We supervise the feature ﬁeld by a pretrained pixellevel image encoder f img as a teacher network.
Speciﬁcally, given a 3D coordinate x , the new NeRFoutputs a feature vector f ( x ) in addition to density  ( x ) and color c ( x , d ) , as shown in Figure 1.
Volume rendering is performed similarly in the feature ﬁeld as follows
[Equation]
We can optimize f through SGD on minimizing the difference between rendered features
ˆ F ( r ) andthe teacher’s outputs f img ( I, r ) .
This can be seen as distillation from 2D teacher network to 3Dstudent network via the volume rendering trick.
We call this model distilled feature ﬁeld (DFF).
We follow the original NeRF [46] for the training objective and the volume rendering strategy.
Inaddition to the photometric loss, we add a new objective for minimizing the difference betweenrendered features ˆ F ( r ) and the teacher’s outputs f img ( I, r ) .
For volume rendering, we use twonetworks for volume rendering with coarse-and-ﬁne hierarchical sampling as well as the original
NeRF does.
We simultaneously train each network from scratch by minimizing photometric loss L pand feature loss L f , in total, L :
[Equation]
perform zero-shot semantic segmentation by aligning pixel-level features and a text query feature. LSeg employs an image feature encoder with the DPT architecture [71] and a CLIP-based text label feature encoder [69], trained via large-scale language-image contrastive learning.
The probability of a text label l given a pixel r in an image I , p ( l | I, r ) , is then calculated via dot product of pixel-level image feature f img ( I, r ) and queried text feature f q ( l ) followed by a softmax:
[Equation]
where L is a set of possible labels.
If negative labels are not available, we may use other scores like thresholded cosine similarity to directly compute the probability of a label.
During training, LSeg optimizes only the image encoder f img ( I, r ) by minimizing cross-entropy on supervised semantic segmentation datasets.
The text encoder f q ( l ) is obtained from a pre-trained CLIP model [69].
Recently, pre-trained CLIP has been leveraged as the backbone for a variety of tasks and has been extended with additional modules sharing the same latent space.
For example, Reimers and Gurevych [74, 75] trains a multi-lingual (more than 50+ languages) text encoder, which enables CLIP and
CLIP-inspired variants to use non-English queries like Japanese.
We similarly use the latent space of a pre-trained CLIP for LSeg via distillation, enabling the decomposition of NeRFs with both English and non-English queries.
Segmentation can further be performed with other modalities such as image, patch or pixel query features f q using a similar dot-product similarity formulation as in Eq. 2.
Notably, DINO [12], a self-supervised vision model, solves video instance segmentation and tracking by calculating similarity among features in adjacent frames.
Amir et al.
[3] also demonstrate that
DINO features work well on co-segmentation and point correspondence by similarity and clustering.
In our experiments, we use these two publicly available models, LSeg and DINO, to obtain features of images and texts for 3D decomposition.
Distilled Feature Fields 4.1 Distilling Foundation Modules into 3D Feature Fields via Volume Rendering
NeRF learns a neural ﬁeld to compute the density and view-dependent color,  ( x ) and c ( x , d ) .
We may extend NeRF by adding decoders for other quantities of interest.
For example, SemanticNeRF
[112] adds a branch outputting a probability distribution of closed-set semantic labels, trained with supervision via images with ground-truth semantic labels.
This enables the prediction of pairs of
RGB and semantic segmentation masks from novel views, useful for data augmentation.
However, because ground-truth annotation is costly, the method is inefﬁcient as a means of scene editing [104].
For speciﬁc domains like trafﬁc scenes [25, 41], we may instead train a closed-set segmentation model and use its prediction for training object-aware neural ﬁelds.
However, this approach is possible only if the types of objects are limited and the domain-speciﬁc supervised dataset is available; limiting the application of scene editing in terms of domain and ﬂexibility of decomposition.
We build on top of these ideas and perform 3D zero-shot segmentation of NeRFs using open-set text labels or other feature queries.
Instead of a branch performing closed-set classiﬁcation, we propose to add a feature branch outputting a feature vector itself.
This branch models a 3D feature ﬁeld describing the semantics of each spatial point.
We supervise the feature ﬁeld by a pretrained pixel-level image encoder f img as a teacher network.
Given a 3D coordinate x , the feature ﬁeld outputs a feature vector f ( x ) in addition to density  ( x ) and color c ( x , d ) , as shown in Fig.
Volume rendering of the feature ﬁeld is similarly performed via
[Equation]
We can optimize f by minimizing the difference between rendered features
ˆ F ( r ) and the teacher’s features f img ( I, r ) .
Effectively, we are distilling [34] the 2D teacher network into our 3D student network via differentiable rendering, and thus dub this model a distilled feature ﬁeld (DFF).
We add a feature objective L f penalizing the difference between rendered features
ˆ F ( r ) and the teacher’s outputs f img ( I, r ) to the photometric loss of the original NeRF.
We use two networks for volume rendering with coarse-and-ﬁne hierarchical sampling.
We thus minimize the sum of photometric loss
L p and feature loss L f , in total, L :
[Equation]
where R are sampled rays, C ( r ) is the ground truth pixel color of ray r ,  is the weight of the feature loss and is set to 0.04 to balance the losses [112].
We apply stop-gradient to density in rendering of features ˆ F ( r ) in Equation 3 as the teacher’s features f img ( I, r ) are not fully multi-view consistent, which could harm the quality of reconstructed geometry.
4.2 Query-based Decomposition and Editing
A trained DFF model can perform 3D zero-shot segmentation by its feature ﬁeld f and a query encoder f q .
Probability of a label l of a point x in the 3D space, p ( l | x ) , is calculated by dot product of the 3D feature f ( x ) and text label feature f q ( l ) followed by a softmax:
[Equation]
This query-based segmentation ﬁeld is at the core of the proposed method.
It can be calculated at any 3D point without limiting resolution, naturally used in tandem with a radiance ﬁeld and volume rendering.
Note that the segmentation depends on only the 3D coordinate and the query 1 .
As the original NeRF, it is thus multi-view consistent.
In addition and important for interactive editing, we can change the segmentation via queries without re-training, which cannot be realized by closed-set methods using semantic [112] or instance segmentation annotation [104].
We may now use this query-conditional segmentation to identify a speciﬁc 3D region for editing.
Various edits can be generalized to the merging of two NeRF scenes  1 ( x ) , c 1 ( x , d ) and  2 ( x ) , c 2 ( x , d ) , where we use the segmentation ﬁeld p for blending.
In the experiments section, we simply modify Eq.
1 as a blend of two scenes based on the ratio of ↵ :
[Equation]
where ⇢ k = [Equation]
[Equation]
Y
[Equation]
For example, if we want to apply a geometric transformation g to a region of a query l in a NeRF scene (  , c ) , we can render the transformed scene via Eqs.
6 and 7 by setting ↵ (  1 ( x k )  k )
= (1  p ( l | x k ))
↵ (  ( x k )  k ) , ↵ (  2 ( x k )  k ) =
p ( l | g  1 ( x k ))
↵ (  ( g  1 ( x k ))
k ) , c 1 ( x k , d ) =
(1  p ( l | x k ))
c ( x k , d ) , and c 2 ( x k , d ) =
p ( l | g  1 ( x k ))
c ( g  1 ( x k ) , g  1 ( d )) .
More details of editing for
G r o und t r u t h f i n e
M L P c o a r
s e
M L P
[Table]
colorization, translation, and deletion are shown in Appendix B. We can combine this with more complex edits, including optimization-based methods like CLIPNeRF [97].
While CLIPNeRF itself cannot selectively edit speciﬁc regions in multi-object scenes, our decomposition method enables it to update only desired objects without breaking unintended areas.
Experiments
We ﬁrst conduct a quantitative evaluation of the decomposition achieved by DFF.
We demonstrate that DFF enables 3D semantic segmentation in a benchmark dataset using scanned point clouds with human-annotated semantic segmentation labels.
We then investigate the capabilities of DFF for editing and subsequent novel-view synthesis on real-world datasets.
We use two teacher networks,
LSeg [44] and DINO [12], which are pre-trained and publicly available.
Each training image is encoded by the image encoders of the networks and used as target feature maps, f img ( I, r ) , deﬁned in Equation 4.
Because the feature maps are of reduced sizes due to the limitation of the networks, we ﬁrst resize them to the original image size.
The implementation and settings of NeRF, unless otherwise stated, follow Zhi et al.
During the training of 200K iterations, the loss L in
Equation 4 is minimized by Adam with a linearly decaying learning rate (5e-4 to 8e-5).
During training, Gaussian noise for density is also applied.
The number of coarse and ﬁne samplings isand 128, respectively.
The MLP of the neural radiance ﬁeld consists of eight ReLU layers withdimensions, followed by a linear layer for density, three layers for color, and three layers for feature, as shown in Fig.
Positional encoding of length 10 is used for the input coordinate and its skip connection, and that of length 4 is for viewing direction.
If an independent MLP is prepared for the feature ﬁeld, it consists of four layers (with a skip connection at the third layer if the positional encoding is used).
The size of a training image is 320 ⇥ 240 for the Replica dataset and 1008 ⇥for the other datasets.
The batchsize of training rays is 1024 for Replica and 2048 for the others.
During ﬁnetuning of feature ﬁelds or radiance ﬁelds, Gaussian noise is removed, and the learning rate is set to 1e-4.
See appendix A and C for further training details.
5.1 3D Semantic Segmentation
We construct a 3D semantic segmentation benchmark from four scenes in the Replica dataset [86] with data split and posed images provided by [112].
See appendix D for further details of the dataset.
We train DFF to reconstruct each scene with radiance and feature ﬁelds from training images and evaluate the quality of novel view synthesis and 3D segmentation of the annotated point clouds.
We use LSeg as a teacher network.
The LSeg text encoder encodes each label, and the probability of each point is calculated by Equation 2 2 .
Note that the training uses only the photometric and feature losses (Equation 4) and does not access any supervision via semantic labels.
Semantic Segmentation Results.
First, we show evaluation metrics of 3D semantic segmentation, mean intersection-over-union (mIoU) and accuracy in Table 1.
For comparison, we also experiment with a sparse 3D convolution-based segmentation model, MinkowskiNet42 [18] taking a colored
[Table]
point cloud as input.
It has a standard state-of-the-art architecture for point cloud segmentation and is trained on the ScanNet dataset [20], the largest annotated training dataset of 3D semantic segmentation 3 .
Results demonstrate that DFF, taught by LSeg, achieves promising performance, even better than the supervised model.
This indicates that DFF succeeds at distilling 3D semantic segmentation from the 2D teacher network.
Impact of Sampling on Semantic Segmentation.
NeRF employs two MLPs for hierarchical sampling, where the coarse MLP performs volume rendering with fewer points (64) using stratiﬁed sampling, and the ﬁne MLP works with importance sampling (192 in total).
So, we have two sampling options to train a feature ﬁeld.
Although ﬁne sampling is critical for training accurate radiance ﬁelds, segmentation is of signiﬁcantly lower spatial frequency than texture.
We thus analyze the impact of coarse and ﬁne training in Fig.
As expected, the coarse model produces smooth segmentations, while the ﬁne version introduces high-frequency artifacts.
This smoothness property is important for natural editable novel view synthesis and is discussed again later.
Compatibility with View Synthesis.
We also check and compare the quality of novel view synthesis with NeRF, which does not learn feature ﬁelds.
Because the feature branch partially shares the layers with the radiance ﬁeld (as shown in Fig. 1), learning feature ﬁelds could possibly harm the radiance ﬁeld.
Despite this concern, as shown in Tab. 2, the performance of view synthesis is not degraded.
Thus, we can train and use the branch-based DFF with small computational and parameter overhead compared to the original NeRF.
If we excessively increased the weight of the feature loss,  ⇥ 10 , it hurt view synthesis while not improving segmentation performance further.
We further conﬁrm that training independent, light-weight feature-ﬁeld MLP, instead of a branch of the radiance-ﬁeld MLP, achieves semantic segmentation results competitive with the branch-based approach (see appendix
Tab. 3 for the result of all variants) 4 .
This option is useful especially when we want to introduce
DFF decomposition into arbitrary 3D scene representations, including off-the-shelf NeRF models, dynamic NeRFs [26, 66, 45], or meshes, without re-training of the radiance ﬁeld.
5.2 Editable Novel View Synthesis
In the previous section, we quantitatively validated the ability of DFF to perform semantic decomposition.
We now discuss the capability for editable view synthesis on real-world scenes, including the LLFF dataset [55] and our own dataset.
Our method can be used even for LLFF scenes based on normalized device coordinates.
Please see the supplemental web page for further results, including videos.
In addition to LSeg using a text query, we also experiment with self-supervised DINO [12] as another teacher network to enable query-based decomposition using image patch queries.
Here, we use thresholded cosine similarity to directly compute the probability of a query instead of softmax with negative queries in Eq. 5 and set p = 1 if the similarity exceeds the threshold 5 , and p
=otherwise for hard decomposition.
We ﬁrst train NeRFs without a feature branch for each scene for 200K iterations ( L p ) and then ﬁnetune them with a feature branch via distillation for 5K iterations ( L p +  L f ) since we found that the feature loss converged signiﬁcantly faster than the photometric loss and short training was thus sufﬁcient.
We use coarse sampling for training feature branches and use it for edited rendering with ﬁne sampling.
See appendix A for the details.
tune the range and scale of input point clouds to maximize the performance of MinkowskiNet42 on the Replica dataset.
Note that, even if we are interested in the feature-ﬁeld MLP, it is important to simultaneously prepare the radiance-ﬁeld MLP for reconstructing geometry well and training the feature ﬁeld via geometrically plausible volume rendering room
[Figure]
[Figure]
Appearance Editing, Deletion, Extraction.
We show qualitative evaluations of novel view synthesis in Fig.
3 and Fig.
Speciﬁc 3D regions in these scenes are identiﬁed and locally edited via decomposition depending on various query modalities.
In these experiments, we use a text query for
LSeg-DFF as in Section 5.1 and use an image patch query for DINO-DFF.
Because DINO features capture the similarity and correspondences of regions well thanks to self-supervised learning [12, 3], image patch queries help select all semantically similar areas at once.
The patch feature is then calculated by averaging the features of all pixels in the patch.
In Fig.
3, we demonstrate that the DFF enables convincing selective appearance edits.
Because our focus is region selection via decomposition, we use simple color transformation for clarity here (e.g., ﬂip RGB to BGR, blend colors).
One might think that the MLP of a radiance ﬁeld by the original NeRF also has hidden layers, and their features could possibly be used for decomposition.
We conﬁrm that the naive usage of NeRF features is not robust to decomposition, as shown in Fig.
5, especially in a complex multi-object scene.
We use the 8th hidden layer of the ﬁne radiance ﬁeld network (i.e., the layer just before branching in Fig. 1) 6 .
NeRF features cannot clearly decompose even objects with simple shapes and colors.
The region selections are leaked to other parts with similar colors, geometry, or positions while they do not entirely cover the targets.
For example, ﬂoor selection is leaked to walls, a table, bins, or ceilings.
Chair selection is leaked to irrelevant black parts like television, cables, lighting equipment, or shadows.
This indicates that the feature space of the original NeRF does not learn semantic similarity well and is entangled with unpredictable and more low-level factors like color or spatial adjacency.
In Fig.
4, we demonstrate that the DFF also works well on deletion or extraction of objects, using two patch queries (query- 1  for leaves and ground, query- 2  for ﬂowers) and a text
query- 3  “ﬂower”.
For comparison with a baseline editing method, we show the results by a state-of-the-art image inpainting model, LaMa [87].
Because the model requires masks for inpainting regions, we manually annotate the views for evaluation.
As shown in the ﬁgure, the image inpainting model cannot generate clear and realistic images, and the different views are inconsistent.
On the other hand, DFF produces multi-view consistent plausible results, especially succeeding at extracting foreground objects.
Although the performance on deleting foreground objects is high, a remaining shortcoming is the existence of ﬂoating artifacts and blurred volumes in the far distance behind the deleted object.
Priors for Smooth Decomposition We can organize the challenges of editable NeRFs into several categories: surface decomposition, volume decomposition, lighting decomposition, and estimation of less or never observed parts.
If we edit appearances only, it practically requires decomposing regions only near the surface of objects, i.e., surface decomposition, because the color of a ray is determined mostly in a condensed interval around the surface.
On the other hand, geometric transformations often
“ f l oo r ”
Our method Using NeRF feature high threshold “c h a i r ” ↓ ↓ ↓ ↑ ↑ ↓ ↓ ↓ ↓ ↑ ↑ ↓ ↓ ↓↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓↓ ↑ ↑ ↓ low threshold
F e a t u re s ② ① ① pa t c h ② pa t c h
Query patches:
Delete
N o P
E
B r a n c h - b a s e d fine coarse Features -clusters RGB , require a higher level of decomposition.
As shown in the deletion examples, geometric transformation may move or remove some surfaces and expose the space behind them.
This forces models to render unknown regions less or never observed due to occlusions, including even the inside of objects.
Thus, it is desirable to decompose volumes smoothly while synthesizing their inside and back 7 .
Although these include the same challenges as novel view synthesis tackles, editability further highlights their importance.
Apart from lighting decomposition discussed in prior work [8, 9, 111], we further investigate the new challenge of smooth volume decomposition by experimenting with different DFF setups.
As discussed in
Section 5.1, DFF has two sampling options to train feature ﬁelds.
The coarse training may introduce smoothness regularization and help cohesive decomposition and smoother in-painting of unobserved regions.
Another reasonable smoothness regularizer is to eliminate the high-frequency positional encoding (PE).
We thus train an independent MLP network for a feature ﬁeld without PE.
We compare four combinations of renderings in Fig.
To better understand their behavior, we use the DINO-DFF, show k-means clusters of the rendered feature map, and delete the head of the Triceratops by a query choosing its corresponding clusters.
As expected, coarsely trained models and no-PE models succeed in smoother volume decomposition, and this combination can minimize high-frequency ﬂoating artifacts.
A side effect is the lack of high-frequency representation power, which sometimes deletes disparate background regions and misses to represent features of complex structures (e.g., see the cluster visualization of the thin frames of the window).
Towards the best of both worlds, developing proper priors or inductive biases is an important direction for future work [70].
Otherwise, surface-aware representations like IDR [106, 98] could avoid problems with ﬂoating artifacts.
Note that not all geometric edits suffer from these problems.
For example, it is often less problematic to move objects closer to the camera, enlarge them, or warp them to other scenes, as shown in Fig.
Localizing Optimization-based Editing.
Finally, we show a combination with an optimizationbased editing method.
CLIPNeRF [97] optimizes the parameters of a radiance ﬁeld so that its rendered images match with a text prompt via CLIP.
While it is mainly designed for a single-object scene of speciﬁc categories, it is possible to apply to other real-world NeRFs.
However, because it cannot control the scope of editing, a prompt like “white ﬂower” may change the color of unintentional targets like leaves.
Our DFF-based decomposition can upgrade such an optimization-based method
[Figure]
to render a scene via the composition of a CLIP-optimized NeRF scene and the original NeRF scene.
We show the results in Fig.
Although the naive CLIPNeRF edits unintentional parts, our method helps it to locally edit intentional parts only.
In addition to switching rendering, we can also use the decomposition for controlling training signals during backpropagation.
The additional experiment is shown in Appendix F. These extensions broaden the application of CLIPNeRF or other optimization-based editing methods to complex scenes.
Discussion, Limitations, and Conclusions
In this work, we propose distilled feature ﬁeld (DFF), a novel method of NeRF scene decomposition for selective editing.
We present quantitative evaluations of segmentation and extensive qualitative evaluations of editable novel view synthesis.
In addition to these promising results, DFF-based models will beneﬁt from future improvements to self-supervised 2D foundation models.
We also clarify future directions on editable view synthesis through our experiments, especially for smoothness priors and estimation of unobserved regions.
Furthermore, while this work focuses on editable view synthesis, it is also intriguing to transfer DFF to other applications, including 3D registration of text queries [16, 1, 47, 4] or robot teaching [32, 80].
The limitations of the DFF framework are two-fold.
The ﬁrst one is the upper bduround of the performance due to distillation.
The student model of distillation cannot largely outperform the teacher model 9 .
If the resolution of teacher encoders is low, the corresponding DFFs also becomes coarse-grained.
If the LSeg cannot understand a text query, the LSeg-DFF also cannot.
Secondly, the DFF uses volume rendering depending on the 3D reconstruction by NeRF.
A NeRF model is sometimes optimized to geometrically wrong solutions (e.g., ﬂoaters).
Such geometry errors of the radiance ﬁelds would make supervision to DFFs noisy.
As a possible negative societal impact, one might use our method for making realistic but fake content by editing NeRFs as desired.
Automatic fake detection methods may help in preventing such misuse.
NeRFs are further computation-intense, leading to high electricity usage.
Recent work on efﬁcient
NeRFs [23, 57, 15] may alleviate this concern.
Acknowledgements
We thank Tsukasa Takagi, Toshiki Nakanishi, Hiroharu Kato and Masaaki Fukuda for their helpful feedback.