Unsupervised Multi-View Object Segmentation Using Radiance Field Propagation
Abstract
We present radiance field propagation (RFP), a novel approach to segmentingobjects in 3D during reconstruction given only unlabeled multi-view images ofa scene.
RFP is derived from emerging neural radiance field-based techniques,which jointly encodes semantics with appearance and geometry.
The core of ourmethod is a novel propagation strategy for individual objects’ radiance fields witha bidirectional photometric loss, enabling an unsupervised partitioning of a sceneinto salient or meaningful regions corresponding to different object instances.
Tobetter handle complex scenes with multiple objects and occlusions, we furtherpropose an iterative expectation-maximization algorithm to refine object masks.
Tothe best of our knowledge, RFP is the first unsupervised approach for tackling 3Dscene object segmentation for neural radiance field (NeRF) without any supervision,annotations, or other cues such as 3D bounding boxes and prior knowledge ofobject class.
Experiments demonstrate that RFP achieves feasible segmentationresults that are more accurate than previous unsupervised image/scene segmentationapproaches, and are comparable to existing supervised NeRF-based methods.
Thesegmented object representations enable individual 3D object editing operations.
Codes and datasets will be made publicly available.
Introduction
Despite deep learning’s remarkable success in object detection and segmentation [11, 30, 33] overthe past decade, several issues remain relevant.
First, modern neural network approaches requirea massive amount of labeled, heavily curated data [9, 20] to achieve superior performance, or theymay have difficulty dealing with problems in which labeled data are lacking.
In addition, mostsegmentation methods regard images as flat 2D arrays of pixels where pixel groupings form objectinstances.
Natural images are 2D projections of the underlying 3D world.
Thus, discarding one of thedimensions gives rise to fundamental ambiguities and challenges to the task resulting from partial ortotal occlusion, especially for single-image methods.
The above fundamental issues motivate us to reconsider the problem of object detection and segmentation from a 3D perspective.
In this paper, we study the problem of automatically partitioning a 3Dscene into an arbitrary number of salient or meaningful regions, given only 2D images of the scenefrom multiple viewpoints.
Estimating semantic labels of a 3D scene is closely related to predicting its geometry and appearance.
Therefore, recent success in novel view synthesis achieved by Neural Radiance Field (NeRF) [24]and its many follow-up work [1, 4, 25, 41] will likely make a long-lasting impact for semantic sceneunderstanding.
For example, SemanticNeRF
[45] extends radiance fields to jointly encode semanticswith appearance and geometry; its multi-task learning setting enables smooth and coherent semanticsprediction.
Nonetheless, as a supervised approach, SemanticNeRF requires semantic labels for full
[Figure]
supervision.
When labels are incomplete or noisy, this approach still relies on a cross-entropy lossbetween the prediction and the target label to supervise its semantic branch.
Previous work [43] andconcurrent work [35] studied unsupervised discovery of object field to enable 3D scene segmentationand editing.
But their system needs pre-training on datasets with specific object categories, such as
CLEVR [15] and ShapeNet [38] objects, thus not capable of segmenting arbitrary objects in generalscenes.
To explore fully self-supervised image segmentation in the multi-view setting, we study the task ofpartitioning a scene into multiple object instances with only multi-view images and calibrated cameraposes.
In this setting, annotated labels in any form are not available.
Furthermore, there are no othercues, such as 3D bounding boxes for the objects or prior knowledge about object classes.
To tackle this challenging task, we propose radiance field propagation (RFP) based on the emergingradiance field-based scene representation.
Fig.
1 shows an overview of our method.
We use acontinuous 3D semantic field like [45] to model semantic information in space.
Note that we do notmodel object geometry and appearance by a single unified radiance field, but rather by a layered scenerepresentation similarly adopted in [13] and [40], representing each entity (each foreground objectand the background) as an independent radiance field.
Based on the layered scene representation, weintroduce a propagation strategy on each radiance field.
The input to these continuous radiance fieldfunctions can be (a) a coordinate occupied by the corresponding object, or (b) a coordinate occupiedby other objects or no object.
For (a) , we expect the function’s output to be an accurate model of theobject’s geometry and appearance.
We propagate the statistical bias of appearance from (a) to (b) .(Note
that the density function or geometry is not propagated.)
Inspired from the single image unsupervised segmentation approach [32], we assume that an accuratepartitioning of the scene should make the estimation of one part from others difficult.
If we usethe semantic field to distinguish (a) and (b) , then an accurate semantic field should maximizethe error between the propagated appearance from (a) to (b) and the ground truth appearance.
Abidirectional photometric loss term is thus proposed that maximizes the error of erroneously renderedimage from propagated appearance, while minimizing the error of correctly rendered image, so thatself-supervision of the learning of the semantic fields can be achieved.
While the above techniques can segment well scenes with a single salient object, to handle morecomplex scenes with multiple objects and occlusions, we design a post-processing procedure torefine object masks based on iterative expectation-maximization.
These procedures improve the 3Dsegmentation of each object in scenes with multiple objects.
With individually segmented object
NeRFs, we can render individual objects and even enable various 3D scene editing, including objectinsertion, duplication, translation, and rotation, which is achieved by the previous works [13, 40] onlywhen using manual annotation.
As the first pertinent unsupervised multi-image method, the experiments show our system performsbetter than the SoTA methods for unsupervised single image segmentation or 3D scene segmentation,and is comparable with supervised NeRF-based novel view semantic synthesis approaches.
In summary, the contributions of this paper are as follows.
• We are the first to study fully unsupervised multi-view image segmentation with NeRFwithout any annotations such as 3D bounding boxes, or prior knowledge of object class tothe best of our knowledge.
• We exploit layered radiance fields and propose Radiance Field Propagation with a bidirectional photometric loss to guide the training of the semantic field.
• We design an iterative expectation-maximization algorithm to refine object masks forhandling complex scenes.
• Extensive experimental validation and extensive ablation studies justify the design of eachcomponent, and demonstrate the effectiveness of our system on applications such as individual object rendering and editing.
Related Work
Unsupervised Single Image Segmentation.
Despite the success of supervised deep learning basedmethods [2, 5, 6, 21, 31, 44], much effort has been made on unsupervised single image segmentation.
Different from supervised image segmentation, where pixel-level semantic labels such as floor ortable, unsupervised image segmentation aims to predict more general labels, such as foregroundand background.
To solve unsupervised category-agnostic image segmentation, which could beregarded as primarily a grouping or clustering problem, as opposed to a labeling task, using color,contrast, and hand-crafted features to cluster pixels has been investigated [8, 14, 16, 18].
More recentapproaches use generative models to partition an image [3, 7].
In [32] image segmentation froman information-theoretic perspective was studied, where unsupervised segmentation was achievedby partitioning an image into maximally independent sets.
Nevertheless, all these works performsegmentation on a single image basis without considering the 3D information of the scene underlyingthe 2D image.
In addition, the class-agnostic segmentation approaches [3, 7, 32] can only tackleimages with single salient object such as those from [27, 37, 42] while we consider scenes with anarbitrary number of salient objects.
Radiance field-based scene representations.
Our unsupervised segmentation work builds on Neural
Radiance Fields (NeRF) [24], which represents a scene using a multi-layer perceptron (MLP) thatmaps positions and directions to densities and radiances.
Following work [1, 4, 41] improve NeRFfor faster training and inference and more realistic rendering.
Using MLP [24] or explicit featuregrids [1, 4], these radiance field-based scene representations achieve unprecedented novel viewsynthesis effects.
Our system can be built upon any radiance field-based scene representation.
NeRF with semantics and object decompositions.
Previous works have investigated inferringsemantics and achieving object-level scene understanding using NeRF [45].
In particular, SemanticNeRF [45] adds an extra head to NeRF to predict semantic labels at any 3D position.
Recent work
PNF [19] can estimate a panoptic radiance field representation of any scene from just color images.
However, SemanticNeRF [45] takes 2D semantic label maps as input, while PNF [19] is prior-basedand trained with external annotated data.
In [10, 13, 28, 40], a scene was decomposed into a setof NeRFs associated with foreground objects separated from the background.
However, [13, 40]cannot produce 3D segmentation of objects to enable rendering and editing of individual objects.
ST-NeRF[13] and ObjectNeRF [40] represent each entity with separated NeRFs.
While enablingrendering and editing of a single object, they require semantic information provided by the label mapfor supervision.
GIRAFFE [26] represents scenes as compositional generative neural feature fields,enabling disentanglement of individual objects for image synthesis without studying obtaining 3Dscene segmentation.
Previous [43] and concurrent work [35] studied unsupervised discovery of objectfields which enables scene segmentation and editing in 3D.
But their systems are prior-based andneed pre-training on datasets with specific object categories, such as CLEVR [15] and ShapeNet [38]objects, thus not capable of segmenting arbitrary objects in general scenes.
Unlike all these methods,our method can segment objects in 3D scenes and perform individual object rendering and editingwithout any semantic annotation and knowledge of object class.
Method
With a set of posed images of a 3D scene as input, our goal is to reconstruct the scene and segmenteach of the object instances.
We view each salient object in the foreground as an object instance, andthe background an extra separate instance.
We assume there are in total K salient foreground objectinstances in the scene (we denote foreground objects as instance 1 to K and use instance 0 to denotethe background as instance
in addition).
3.1 Propagation of Layered Radiance Field
Extented from NeRF [24], we represent the continuous volume density, emitted color, and semantics [45] as functions with a spatial coordinate x = ( x, y, z ) and viewing direction
d = ( θ, ϕ ) as input.
Specifically, we represent volume density as a function that maps a world coordinate x = ( x, y, z ) tothe continuous 3D scene density σ :
[Equation]
Following [45], we formalize semantic information as an inherently view-invariant function thatmaps only a world coordinate x to a distribution over K + 1 labels corresponding to K + 1 objectinstances via pre-softmax semantic logits s ( x ) =
( s 0 , . . .
, s K ) :
[Equation]
Here we only consider K + 1 labels for all practical instances, without an additional label for nonobject coordinates of which the contribution would be eliminated automatically by a value close to 0of F σ ( x ) .
With semantic logits of each coordinate, we compute the label to assign a coordinate to byargmax over all the objects.
[Equation]
Since the operation of argmax is not differentiable, we use the straight through trick [12, 22, 36, 39]and compute the assignment as
[Equation]
where sg is the stop gradient operator.
ˆ l ( x ) has the one-hot value of assignment to an object, whosegradient is equal to the gradient of s ( x ) , thus making the system differentiable and allowing gradientback propagation.
We obtain the 3D mask for object k as
[Equation]
We further define m = 1 − m as the inversion operation on masks.
Instead of a unified function torepresent the continuous field of the emitted color of the whole scene, we use K + 1 functions tomodel the appearance of K + 1 instances.
Such individual modeling of each object with a differentfunction is similar to some previous work [13, 40].
For object k , we represent its color as a functionthat maps a world coordinate x = ( x, y, z ) as well as view direction
d = ( θ, ϕ ) to the continuous 3Dscene color c f = ( r, g, b ) :
[Equation]
Starting here we use c k ( x ) instead of c k ( x , d ) for notation simplicity.
The input world coordinatex to this function can be an arbitrary coordinate in the scene, i.e., either occupied by object k orotherwise.
The 3D object mask in Eqn.
5 naturally distinguishes these two cases.
For the former case,m k ( x ) = 1 , and we expect c k ( x ) to be the precise color of the foreground object to achieve realisticrendering results.
In the latter case, m k ( x ) = 0 , and we encourage c k ( x ) here to be propagatedfrom coordinates in the former case and have a value of color close to object k .
This 3D propagationprocedure is enforced by the propagation regularizer
[Equation]
where µ is the average value of m k ( x ) c k ( x ) (not counting masked out values).
The average operationis performed on the training batch.
At coordinates not occupied by object k , the propagated value is
[Figure]
[Figure]
3.2 Bidirectional Photometric Loss
Since an accurate segmentation of object k should maximize the error of this appearance prediction,a loss term Eqn.
12 encouraging such maximization of error would guide the optimization of thesemantic field without supervision from any ground-truth annotations.
For each coordinate, we calculate the value of all K + 1 color fields at it.
The correct assignment forthis coordinate color is given by
[Equation]
[Equation]
If we artificially color object i with c j ( x ) , and color objects other than object i with the color ofobject c i ( x ) , we get a deliberately erroneous color field˜c i,j ( x )
= m i ( x )
c j ( x ) + m i ( x ) c i ( x ) .
To compute the color of a single pixel, with either the correct or an erroneous radiance field, weapproximate volume rendering by numerical quadrature, the same way as [24].
Let r ( t )
=
o + t dbe the ray emitted from the centre of projection of camera space through a given pixel, traversingbetween near and far bounds ( t n and t f ), then for selected P random quadrature points { t k } Pp =1between t n and t f , the approximated expected color using c ( x ) and the erroneous color using ˜c i,j ( x )are given by:
[Equation]
where ˆ T ( t p ) = exp (cid:16) − (cid:80) p − 1 p ′ =1 σ ( t p ) δ p (cid:17) , α ( x ) = 1 − exp( − x ) , and δ p =
t p +1 − t p is the distancebetween two adjacent quadrature sample points.
We show examples of images using different rendering scheme in Fig.
2 and Fig. To train our system, we minimize the error of the correctlyrendered colors (Fig. 2(a)), and at the same time maximize the error of the erroneously renderedcolors (Fig. 2(d)).
This yield to the bidirectional photometric loss function to train our system.
[Equation]
where R are the sampled rays within a training batch.
We compute the expected semantic logits of asingle pixel in an image the same way as [45]:
[Equation]
which can then be transformed into multi-class probabilities through a softmax normalization layer.
The 2D label map and object mask for object k are
[Equation] [Equation]
We also found that estimating a coarse 2D label map for each input image using an unsupervisedsingle image-based approach [7, 32] and then computing a cross-entropy loss between the map andthe estimated semantic logits can improve the results of our system:
[Equation]
(16) where S init is the initial coarse label map.
Since the approach we use is unsupervised and does notinvolve external data, so this initial stage does not undermine this property of our method.
Hence thetotal training loss L combining the bidirectional photometric loss Eqn.
12, the propagation regularizer
Eqn. 7 and the initial semantic estimation loss Eqn.
16 is:
[Equation]
where λ prop and λ init are weights of corresponding loss terms.
3.3 EM-based Mask Refinement for Complex Scenes
In the previous subsection we have described the core components necessary for performing 3D segmentation, and our experiments have shown the effectiveness of the proposed system on segmentingscenes with one salient object.
To extend the segmentation on more complex scenes with multipleforeground objects and occlusion among them, we further propose an EM algorithm-based procedureto refine the object mask as post-processing.
For each input image, we first feed it into a deep feature extractor, and cluster all the pixels intotwo classes using k -means clustering.
In this way we get initial 2D image masks for the foreground (consisting of K objects) S foreground and the background S background .
We apply differentiablefeature clustering (DFC) [18] to further partition foreground masked by S foreground into K parts, 205 (cid:8)
S k init (cid:9) k
=1 ,...,K and let S 0 init =
S background , for computing the initial semantic estimation loss Eqn.16.
Note that here we introduce a deep feature extractor [34] trained on large-scale data, which isthe only introduction of external data to our method.
But since no annotated data is involved, theunsupervised nature of our method is not undermined.
DFC results in a response vector map v ( x ) foreach image, where the x is a pixel index.
Pixels belonging to the same object tend to have similarresponse vectors.
After the training of our system, expectation-maximization (EM) is employed torefine the output mask obtained from Eqn.
15 with these response vector maps.
Let µ k be a meanvector of object k , and we initialize µ k as
[Equation]
[Figure] N [Figure] N
[Figure] N/A N/A x x x
IEM [32] and ReDO [7] are SoTA unsupervised single image-based methods. There are no ground truth labels for the LLFF dataset [23], and thus the supervised approach SemanticNeRF [45] is not applicable. Images have been cropped due to typography.
We formulate the posterior probability of v given µ k and the latent variable z k at t -th EM iteration as
[Equation]
At each EM iteration, we update the µ as
[Equation]
[Equation]
The final semantic logits are given a weighted sum of Z concatenated from (cid:8)
z T k (cid:9) k
=0 ,...,K at T -thiteration and estimated logits after training in Eqn. 13:
[Equation]
22) where w is a weight.
Therefore, the refined mask can be computed similarly to Eqn.
14 and Eqn. 15.
With scene segmentation obtained using our system, applications such as single object rendering andediting can be achieved.
We defer the description of the relevant procedure in the supplementalmaterial.
Experiments
We evaluate our method on three datasets with different complexities.
We report qualitative andquantitative results on scene segmentation with comparison to previous work.
We also show theapplication of our method on individual object rendering and editing.
We conduct ablation studiesto justify each component of our method.
We gently urge readers to check the supplementary [Figure] material for more qualitative results in the form of pictures and videos, as well as the settings ofour experiments in detail.
Datasets.
We first test RFP on scenes with a single foreground object using two real datasets, Local
Light Field Fusion (LLFF) [23] and Common Objects in 3D (CO3D) [29].
As LLFF does notprovide ground-truth label maps, we only show qualitative results on the LLFF dataset.
To test RFPon scenes with multiple objects, we build a synthetic dataset upon ClevrTex [17].
We split all thedatasets into training views and testing views with a ratio of around 9 to 1.
Metrics.
We adopt the widely-used pixel classification accuracy (Acc.) and mean intersection overunion (mIoU) as our metric.
To evaluate scene segmentation in 3D, we report the metrics computedfor both training and novel views.
The former enables direct comparison to 2D methods, and thelatter reflects the 3D nature.
We denote Acc.
and mIoU of novel views as N-Acc.
and N-mIoU. Notethat for our method, the difference between the training and the novel view is only related to whetherthe color image of the view is provided.
The semantic label of any view is not input.
Differently,for the supervised methods [40, 45], both the color image of the training view and the ground truthsemantic label are provided as inputs.
4.1 3D Segmentation of Scenes with a Single Foreground Object
Because there is no previous work with the same unsupervised multi-view setting as our work, wecompare with SoTA unsupervised class-agnostic segmentation methods, ReDO [7] and IEM [32],which ensembles a grouping or clustering problem with single images without considering 3D nature.
We also compare RFP with supervised novel view semantic synthesis method SemanticNeRF [45].
We show quantitative results in Tab. 1 and qualitative results in Fig. 4.
Quantitatively, RFPoutperforms all single 2D image-based methods on all metrics.
Qualitatively, compared to single 2Dimage-based methods [7, 32], our method’s results have sharper and smoother edges, with no holesin the interior of objects due to the 3D nature of the scene.
Our results are even comparable to thesupervised method SemanticNeRF [45] on the CO3D dataset [29], which may suffer from the noisyground truth masks of the CO3D. [Table]
[Table]
Scenes with Multiple Objects
When tackling scenes with multiple objects and occlusion, we further perform the EM-based refinement introduced in Sec. 3.3.
While using a deep feature extractor trained on large-scale data meansintroducing external data to our system here for the only time, since no annotated data is involved,the unsupervised nature of our method is not undermined.
We compare RFP with the unsupervised segmentation method DFC [18].
Since DFC relies on aninitial separation of the foreground and the background, we perform k -means clustering beforehand.
Such k -means clustering followed by DFC segmentation is also the pre-processing procedurementioned in Sec.
We also compare RFP with unsupervised object radiance field discovery(uORF)
[43], as well as supervised NeRF-based methods: ObjectNeRF
[40] and SemanticNeRF [45].
We show quantitative results in Tab. 1 and qualitative results in Fig.
Quantitatively, RFPoutperforms all unsupervised image/scene segmentation methods on all metrics.
Qualitatively, single2D image-based methods [18] miss more valid detection than ours.
uORF can hardly give goodresults, probably because the method was pre-trained on a dataset consisting of naive scenes and thuswith a large domain gap with our evaluation data.
4.3 Individual Object Rendering and Scene Editing
Following the procedure detailed in the supplemental material , our approach achieves depthaware and photo-realistic free-viewpoint single object rendering and editing results.
Fig.
6 andsupplemental matierials show examples of such applications.
Fig.
6 (a) & (d) are color imagesrendered from novel views.
Fig.
6 (b) & (e) show single object rendering.
Fig.
6 (c) shows the editingeffect of duplicating and repositioning of the duplicated blue chairs.
Fig.
6 (f) shows the editingeffect of rotating the helicopter.
4.4 Ablation Study
Here, we evaluate the performance of different components and loss terms in our approach.
Let W / O
, W / O INIT .
EST .
and W / O EM REFINEMENT denote the respective variations of ourapproach without the radiance field propagation and using vanilla photometric loss instead of thebidirectional loss, without the initial semantic estimation loss, and without the EM refinement.
Asshown in Tab.
1 quantitatively and in the supplementary material qualitatively, our complete approach achieves more accurate results than all the above variations.
The radiance field propagation and thebidirectional photometric loss ensure good utilization of the 3D nature of scenes, which helps toavoid ragged edges, holes in the object’s interior, and false-positive estimates outside of the object.
Although unsupervised single image segmentation approaches are shown to be not robust in most ofthe cases, it is still essential for our system as a valid initialization.
The EM-based refinement furtherbrings good edges when tackling complex scenes with multiple objects.
Conclusion
In this work we present the first approach in NeRF for tackling unsupervised multi-view imagesegmentation using radiance field propagation with a bidirectional photometric loss to guide thereconstruction of semantic field.
We design an EM-based mask refinement procedure to handlecomplex scenes.
Our experiments show the effectiveness of our approach, with useful applicationson NeRF such as individual object rendering and editing
Checklist
For all
authors
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See the supplemental material.(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] See the supplemental material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] Not yet, but wewill make the code and data publicly available upon acceptance. (b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See the supplemental material.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A](c)
Did you include any new assets either in the supplemental material or as a URL?
[No]Not yet, but we will make the code and data publicly available upon acceptance.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full
[Table]