Unsupervised Multi-View Object Segmentation Using Radiance Field Propagation
Abstract
We present radiance ﬁeld propagation (RFP), a novel approach to segmenting objects in 3D during reconstruction given only unlabeled multi-view images of a scene.
RFP is derived from emerging neural radiance ﬁeld-based techniques, which jointly encodes semantics with appearance and geometry.
The core of our method is a novel propagation strategy for individual objects’ radiance ﬁelds with a bidirectional photometric loss, enabling an unsupervised partitioning of a scene into salient or meaningful regions corresponding to different object instances.
To better handle complex scenes with multiple objects and occlusions, we further propose an iterative expectation-maximization algorithm to reﬁne object masks.
To the best of our knowledge, RFP is the ﬁrst unsupervised approach for tackling 3D scene object segmentation for neural radiance ﬁeld (NeRF) without any supervision, annotations, or other cues such as 3D bounding boxes and prior knowledge of object class.
Experiments demonstrate that RFP achieves feasible segmentation results that are more accurate than previous unsupervised image/scene segmentation approaches, and are comparable to existing supervised NeRF-based methods.
The segmented object representations enable individual 3D object editing operations.
Introduction
Despite deep learning’s remarkable success in object detection and segmentation [35, 32, 12] over the past decade, several issues remain relevant.
First, modern neural network approaches require a massive amount of labeled, heavily curated data [9, 20] to achieve superior performance, or they may have difﬁculty dealing with problems in which labeled data are lacking.
In addition, most segmentation methods regard images as ﬂat 2D arrays of pixels where pixel groupings form object instances.
Natural images are 2D projections of the underlying 3D world.
Thus, discarding one of the dimensions gives rise to fundamental ambiguities and challenges to the task resulting from partial or total occlusion, especially for single-image methods.
The above fundamental issues motivate us to reconsider the problem of object detection and segmentation from a 3D perspective.
In this paper, we study the problem of automatically partitioning a 3D scene into an arbitrary number of salient or meaningful regions, given only 2D images of the scene from multiple viewpoints.
Estimating semantic labels of a 3D scene is closely related to predicting its geometry and appearance.
Therefore, recent success in novel view synthesis achieved by Neural Radiance Field (NeRF) [24] and its many follow-up work [25, 46, 3, 10] will likely make a long-lasting impact for semantic scene understanding.
For example, SemanticNeRF
[51] extends radiance ﬁelds to jointly encode semantics with appearance and geometry; its multi-task learning setting enables smooth and coherent semantics prediction.
Nonetheless, as a supervised approach, SemanticNeRF requires semantic labels for full supervision.
When labels are incomplete or noisy, this approach still relies on a cross-entropy loss 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
[Figure]
between the prediction and the target label to supervise its semantic branch.
Previous work [48] and concurrent work [37] studied unsupervised discovery of object ﬁeld to enable 3D scene segmentation and editing.
But their system needs pre-training on datasets with speciﬁc object categories, such as
CLEVR [15] and ShapeNet [41] objects, thus not capable of segmenting arbitrary objects in general scenes.
To explore fully self-supervised image segmentation in the multi-view setting, we study the task of partitioning a scene into multiple object instances with only multi-view images and calibrated camera poses.
In this setting, annotated labels in any form are not available.
Furthermore, there are no other cues, such as 3D bounding boxes for the objects or prior knowledge about object classes.
To tackle this challenging task, we propose radiance ﬁeld propagation (RFP) based on the emerging radiance ﬁeld-based scene representation.
Fig.
1 shows an overview of our method.
We use a continuous 3D semantic ﬁeld like [51] to model semantic information in space.
Note that we do not model object geometry and appearance by a single uniﬁed radiance ﬁeld, but rather by a layered scene representation similarly adopted in [49] and [43], representing each entity (each foreground object and the background) as an independent radiance ﬁeld.
Based on the layered scene representation, we introduce a propagation strategy on each radiance ﬁeld.
The input to these continuous radiance ﬁeld functions can be (a) a coordinate occupied by the corresponding object, or (b) a coordinate occupied by other objects or no object.
For (a) , we expect the function’s output to be an accurate model of the object’s geometry and appearance.
We propagate the statistical bias of appearance from (a) to (b) .
(Note that the density function or geometry is not propagated.)
Inspired from the unsupervised object detection and segmentation approaches [45, 44, 34], we assume that an accurate partitioning of the scene should make the estimation of one part from others difﬁcult.
If we use the semantic ﬁeld to distinguish (a) and (b) , then an accurate semantic ﬁeld should maximize the error between the propagated appearance from (a) to (b) and the ground truth appearance.
A bidirectional photometric loss term is thus proposed that maximizes the error of erroneously rendered image from propagated appearance, while minimizing the error of correctly rendered image, so that self-supervision of the learning of the semantic ﬁelds can be achieved.
While the above techniques can segment well scenes with a single salient object, to handle more complex scenes with multiple objects and occlusions, we design a post-processing procedure to reﬁne object masks based on iterative expectation-maximization.
These procedures improve the 3D segmentation of each object in scenes with multiple objects.
With individually segmented object
NeRFs, we can render individual objects and even enable various 3D scene editing, including object insertion, duplication, translation, and rotation, which is achieved by the previous works [49, 43] only when using manual annotation.
As the ﬁrst pertinent unsupervised multi-image method, the experiments show our system performs better than the SoTA methods for unsupervised single image segmentation or 3D scene segmentation, and is comparable with supervised NeRF-based novel view semantic synthesis approaches.
In summary, the contributions of this paper are as follows.
• We are the ﬁrst to study fully unsupervised multi-view image segmentation with NeRF without any annotations such as 3D bounding boxes, or prior knowledge of object class to the best of our knowledge.
• We exploit layered radiance ﬁelds and propose Radiance Field Propagation with a bidirectional photometric loss to guide the training of the semantic ﬁeld.
• We design an iterative expectation-maximization algorithm to reﬁne object masks for handling complex scenes.
• Extensive experimental validation and extensive ablation studies justify the design of each component, and demonstrate the effectiveness of our system on applications such as individual object rendering and editing.
Related Work
Unsupervised Single Image Segmentation.
Despite the success of supervised deep learning based methods [21, 33, 1, 50, 4, 5], much effort has been made on unsupervised single image segmentation.
Different from supervised image segmentation, where pixel-level semantic labels such as ﬂoor or table, unsupervised image segmentation aims to predict more general labels, such as foreground and background.
To solve unsupervised category-agnostic image segmentation, which could be regarded as primarily a grouping or clustering problem, as opposed to a labeling task, using color, contrast, and hand-crafted features to cluster pixels has been investigated [7, 14, 16, 18].
More recent approaches use generative models to partition an image [6, 2].
Notably, in Contextual Information Separation (CIS) [45], object detection and segmentation from an information-theoretic perspective were ﬁrst studied.
To detect moving objects in images, CIS uses a segmentation network to minimize the mutual information between the inside and the outside of putative motion regions.
This procedure is however sensitive to motion errors.
To further enable stationary objects detection and segmentation, DyStaB [44] partitions the motion ﬁeld by minimizing the mutual information between segments which are then used to learn object models.
While in
CIS [45] inpainting is discussed on any function of the image (color histogram or optical ﬂow), later in [34] (with somewhat similar technicalities to [45]) the authors consider inpainting on just the image itself and does not involve training a deep network with external data.
Nevertheless, all these works perform segmentation on a single image basis without considering the 3D information of the scene underlying the 2D image.
In addition, the class-agnostic segmentation approaches can only tackle images with single salient object such as those from [39, 47, 27] while we consider scenes with an arbitrary number of salient objects.
Radiance ﬁeld-based scene representations.
Our unsupervised segmentation work builds on Neural
Radiance Fields (NeRF) [24], which represents a scene using a multi-layer perceptron (MLP) that maps positions and directions to densities and radiances.
Following work [46, 10, 3] improve NeRF for faster training and inference and more realistic rendering.
Dynamic neural representations enable 4D reconstruction and rendering [30, 49, 29, 40].
Using MLP [24] or explicit feature grids [10, 3], these radiance ﬁeld-based scene representations achieve unprecedented novel view synthesis effects.
Our system can be built upon any radiance ﬁeld-based scene representation.
NeRF with semantics and object decompositions.
Previous works have investigated inferring semantics and achieving object-level scene understanding using NeRF [51].
In particular, SemanticNeRF [51] adds an extra head to NeRF to predict semantic labels at any 3D position.
Recent work
PNF [19] can estimate a panoptic radiance ﬁeld representation of any scene from just color images.
However, SemanticNeRF [51] takes 2D semantic label maps as input, while PNF [19] is prior-based and trained with external annotated data.
In [49, 43, 28, 11], a scene was decomposed into a set of NeRFs associated with foreground objects separated from the background.
However, [49, 43] cannot produce 3D segmentation of objects to enable rendering and editing of individual objects.
ST-NeRF[49] and ObjectNeRF [43] represent each entity with separated NeRFs.
While enabling rendering and editing of a single object, they require semantic information provided by the label map for supervision.
GIRAFFE [26] represents scenes as compositional generative neural feature ﬁelds, enabling disentanglement of individual objects for image synthesis without studying obtaining 3D scene segmentation.
Previous [48] and concurrent work [37] studied unsupervised discovery of object ﬁelds which enables scene segmentation and editing in 3D.
But their systems are prior-based and need pre-training on datasets with speciﬁc object categories, such as CLEVR [15] and ShapeNet [41] objects, thus not capable of segmenting arbitrary objects in general scenes.
Unlike all these methods, our method can segment objects in 3D scenes and perform individual object rendering and editing without any semantic annotation and knowledge of object class.
Method
With a set of posed images of a 3D scene as input, our goal is to reconstruct the scene and segment each of the object instances.
We view each salient object in the foreground as an object instance, and the background an extra separate instance.
We assume there are in total K salient foreground object instances in the scene (we denote foreground objects as instance 1 to K and use instance 0 to denote the background as instance 0 in addition).
3.1 Propagation of Layered Radiance Field
Extented from NeRF [24], we represent the continuous volume density, emitted color, and semantics [51] as functions with a spatial coordinate x
= ( x, y, z ) and viewing direction d
= ( ✓ ,  ) as input.
Speciﬁcally, we represent volume density as a function that maps a world coordinate x = ( x, y, z ) to the continuous 3D scene density  :
[Equation]
Following [51], we formalize semantic information as an inherently view-invariant function that maps only a world coordinate x to a distribution over K + 1 labels corresponding to K + 1 object instances via pre-softmax semantic logits s ( x ) =
( s 0 , . . .
, s K ) :
[Equation]
Here we only consider K + 1 labels for all practical instances, without an additional label for nonobject coordinates of which the contribution would be eliminated automatically by a value close toof F  ( x ) .
With semantic logits of each coordinate, we compute the label to assign a coordinate to by argmax over all the objects.
[Equation]
Since the operation of argmax is not differentiable, we use the straight through trick [13, 22, 38, 42] and compute the assignment as
[Equation]
where sg is the stop gradient operator.
ˆ l ( x ) has the one-hot value of assignment to an object, whose gradient is equal to the gradient of s ( x ) , thus making the system differentiable and allowing gradient back propagation.
We obtain the 3D mask for object k as
[Equation]
We further deﬁne m = 1  m as the inversion operation on masks.
Instead of a uniﬁed function to represent the continuous ﬁeld of the emitted color of the whole scene, we use K + 1 functions to model the appearance of K + 1 instances.
Such individual modeling of each object with a different function is similar to some previous work [49, 43].
For object k , we represent its color as a function that maps a world coordinate x = ( x, y, z ) as well as view direction d
= ( ✓ ,  ) to the continuous 3D scene color c f = ( r, g, b ) :
[Equation]
Starting here we use c k ( x ) instead of c k ( x , d ) for notation simplicity.
The input world coordinate x to this function can be an arbitrary coordinate in the scene, i.e., either occupied by object k or otherwise.
The 3D object mask in Eqn.
5 naturally distinguishes these two cases.
For the former case,
[Figure] [Figure]
m k ( x ) = 1 , and we expect c k ( x ) to be the precise color of the foreground object to achieve realistic rendering results.
In the latter case, m k ( x ) = 0 , and we encourage c k ( x ) here to be propagated from coordinates in the former case and have a value of color close to object k .
This 3D propagation procedure is enforced by the propagation regularizer
[Equation]
(7) where µ is the average value of m k ( x ) c k ( x ) (not counting masked out values).
The average operation is performed on the training batch.
At coordinates not occupied by object k , the propagated value is similar to one at a coordinate that belongs to object k , which can be regarded as a prediction of the radiance of a partition other than object k knowing object k ’s radiance ﬁeld.
3.2 Bidirectional Photometric Loss
Since an accurate segmentation of object k should maximize the error of this appearance prediction, a loss term Eqn.
12 encouraging such maximization of error would guide the optimization of the semantic ﬁeld without supervision from any ground-truth annotations.
For each coordinate, we calculate the value of all K + 1 color ﬁelds at it.
The correct assignment for this coordinate color is given by
[Equation]
If we artiﬁcially color object i with c j ( x ) , and color objects other than object i with the color of
object c i ( x ) , we get a deliberately erroneous color ﬁeld
[Equation]
To compute the color of a single pixel, with either the correct or an erroneous radiance ﬁeld, we approximate volume rendering by numerical quadrature, the same way as [24].
Let r ( t )
=
o + t d be the ray emitted from the centre of projection of camera space through a given pixel, traversing between near and far bounds ( t n and t f ), then for selected P random quadrature points { t k } Pp = between t n and t f , the approximated expected color using c ( x ) and the erroneous color using ˜c i,j ( x ) are given by:
[Equation]
where ˆ
[Equation]
=
[Equation]
, ↵ ( x ) = 1  exp(  x ) , and  p = {+t p +
+}
t p is the distance between two adjacent quadrature sample points.
We show examples of images using different rendering scheme in Fig.
2 and Fig.
To train our system, we minimize the error of the correctly rendered colors (Fig. 2(a)), and at the same time maximize the error of the erroneously rendered colors (Fig. 2(d)).
This yield to the bidirectional photometric loss function to train our system.
[Equation]
where R are the sampled rays within a training batch.
We compute the expected semantic logits of a single pixel in an image the same way as [51]:
[Equation]
which can then be transformed into multi-class probabilities through a softmax normalization layer.
The 2D label map and object mask for object k are
[Equation]
We also found that estimating a coarse 2D label map for each input image using an unsupervised single image-based approach and then computing a cross-entropy loss between the map and the estimated semantic logits can improve the results of our system:
[Equation]
where S init is the initial coarse label map.
Since the approach we use is unsupervised and does not involve external data, so this initial stage does not undermine this property of our method.
Hence the total training loss L combining the bidirectional photometric loss Eqn.
12, the propagation regularizer
Eqn. 7 and the initial semantic estimation loss Eqn.
16 is:
[Equation]
where  prop and  init are weights of corresponding loss terms.
3.3 EM-based Mask Reﬁnement for Complex Scenes
In the previous subsection we have described the core components necessary for performing 3D segmentation, and our experiments have shown the effectiveness of the proposed system on segmenting scenes with one salient object.
To extend the segmentation on more complex scenes with multiple foreground objects and occlusion among them, we further propose an EM algorithm-based procedure to reﬁne the object mask as post-processing.
For each input image, we ﬁrst feed it into a deep feature extractor, and cluster all the pixels into two classes using k -means clustering.
In this way we get initial 2D image masks for the foreground (consisting of K objects) S foreground and the background S background .
We apply differentiable feature clustering (DFC) [18] to further partition foreground masked by S foreground into K parts,
S k init k
=1 ,...,K and let S 0 init =
S background , for computing the initial semantic estimation loss Eqn.
Note that here we introduce a deep feature extractor [36] trained on large-scale data, which is the only introduction of external data to our method.
But since no annotated data is involved, the unsupervised nature of our method is not undermined.
DFC results in a response vector map v ( x ) for each image, where the x is a pixel index.
Pixels belonging to the same object tend to have similar response vectors.
After the training of our system, expectation-maximization (EM) is employed to reﬁne the output mask obtained from Eqn.
15 with these response vector maps.
Let µ k be a mean vector of object k , and we initialize µ k as
[Equation]
We formulate the posterior probability of v given µ k and the latent variable z k at t -th EM iteration as
[Equation]
At each EM iteration, we update the µ as
[Equation] [Equation]
The ﬁnal semantic logits are given a weighted sum of Z concatenated from
[Equation] .,K
at T -th iteration and estimated logits after training in Eqn. 13:
[Equation]
22) where w is a weight.
Therefore, the reﬁned mask can be computed similarly to Eqn.
14 and Eqn. 15.
With scene segmentation obtained using our system, applications such as single object rendering and editing can be achieved.
We defer the description of the relevant procedure in the supplemental material.
Experiments
We evaluate our method on three datasets with different complexities.
We report qualitative and quantitative results on scene segmentation with comparison to previous work.
We also show the application of our method on individual object rendering and editing.
We conduct ablation studies to justify each component of our method.
We gently urge readers to check the supplementary material for more qualitative results in the form of pictures and videos, as well as the settings of our experiments in detail.
Datasets.
We ﬁrst test RFP on scenes with a single foreground object using two real datasets, Local
Light Field Fusion (LLFF) [23] and Common Objects in 3D (CO3D) [31].
As LLFF does not provide ground-truth label maps, we only show qualitative results on the LLFF dataset.
To test RFP on scenes with multiple objects, we build a synthetic dataset upon ClevrTex [17].
We split all the datasets into training views and testing views with a ratio of around 9 to 1.
Metrics.
We adopt the widely-used pixel classiﬁcation accuracy (Acc.) and mean intersection over union (mIoU) as our metric.
To evaluate scene segmentation in 3D, we report the metrics computed for both training and novel views.
The former enables direct comparison to 2D methods, and the latter reﬂects the 3D nature.
We denote Acc.
and mIoU of novel views as N-Acc.
and N-mIoU. Note that for our method, the difference between the training and the novel view is only related to whether the color image of the view is provided.
The semantic label of any view is not input.
Differently, for the supervised methods [51, 43], both the color image of the training view and the ground truth semantic label are provided as inputs. [Figure] IEM
[Figure]
N/A N/A
N/A N/A x x x
[Figure]
Acc. "
mIoU " N-Acc. " N-mIoU " Acc. "
mIoU " N-Acc. "
N-mIoU " ReDO [6] 86.
- - - - - - IEM [34] 95.
- - - - - - DFC [18]
[Table]
- uORF [48] -
[Table]
SemanticNeRF [51] [Table] [Table] [Table]
[Table]
RFP
[Table]
W / [Table] PROPAGATION ) 96.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
best results supervision
4.1 3D Segmentation of Scenes with a Single Foreground Object
Because there is no previous work with the same unsupervised multi-view setting as our work, we compare with SoTA unsupervised class-agnostic segmentation methods, ReDO [6] and IEM [34], which ensembles a grouping or clustering problem with single images without considering 3D nature.
We also compare RFP with supervised novel view semantic synthesis method SemanticNeRF [51].
We show quantitative results in Tab. 1 and qualitative results in Fig. 4.
Quantitatively, RFP outperforms all single 2D image-based methods on all metrics.
Qualitatively, compared to single 2D image-based methods [34, 6], our method’s results have sharper and smoother edges, with no holes in the interior of objects due to the 3D nature of the scene.
Our results are even comparable to the supervised method SemanticNeRF [51] on the CO3D dataset [31], which may suffer from the noisy ground truth masks of the CO3D.
Inputview
[Figure]
4.2 3D Segmentation of Scenes with Multiple Objects
When tackling scenes with multiple objects and occlusion, we further perform the EM-based reﬁnement introduced in Sec. 3.3.
While using a deep feature extractor trained on large-scale data means introducing external data to our system here for the only time, since no annotated data is involved, the unsupervised nature of our method is not undermined.
We compare RFP with the unsupervised segmentation method DFC [18].
Since DFC relies on an initial separation of the foreground and the background, we perform k -means clustering beforehand.
Such k -means clustering followed by DFC segmentation is also the pre-processing procedure mentioned in Sec.
We also compare RFP with unsupervised object radiance ﬁeld discovery (uORF) [48], as well as supervised NeRF-based methods: ObjectNeRF
[43] and SemanticNeRF [51].
We show quantitative results in Tab. 1 and qualitative results in Fig.
Quantitatively, RFP outperforms all unsupervised image/scene segmentation methods on all metrics.
Qualitatively, single 2D image-based methods [18] miss more valid detection than ours.
uORF can hardly give good results, probably because the method was pre-trained on a dataset consisting of naive scenes and thus with a large domain gap with our evaluation data.
4.3 Individual Object Rendering and Scene Editing
Following the procedure detailed in the supplemental material , our approach achieves depthaware and photo-realistic free-viewpoint single object rendering and editing results.
Fig.
6 and supplemental matierials show examples of such applications.
Fig.
6 (a) & (d) are color images rendered from novel views.
Fig.
6 (b) & (e) show single object rendering.
Fig.
6 (c) shows the editing effect of duplicating and repositioning of the duplicated blue chairs.
Fig.
6 (f) shows the editing effect of rotating the helicopter.
（ a ） （ b ） （ c ） （ d ） （ e ） （ f ）
4.4 Ablation Study
Here, we evaluate the performance of different components and loss terms in our approach.
Let W / O
PROPAGATION , W / O INIT .
EST .
and W / O EM REFINEMENT denote the respective variations of our approach without the radiance ﬁeld propagation and using vanilla photometric loss instead of the bidirectional loss, without the initial semantic estimation loss, and without the EM reﬁnement.
As shown in Tab. 1 quantitatively and in the supplementary material qualitatively, our complete approach achieves more accurate results than all the above variations.
The radiance ﬁeld propagation and the bidirectional photometric loss ensure good utilization of the 3D nature of scenes, which helps to avoid ragged edges, holes in the object’s interior, and false-positive estimates outside of the object.
Although unsupervised single image segmentation approaches are shown to be not robust in most of the cases, it is still essential for our system as a valid initialization.
The EM-based reﬁnement further brings good edges when tackling complex scenes with multiple objects.
Limitation
As the ﬁrst approach dealing with unsupervised multi-view segmentation, our method may have several limitations.
To begin with, our method may have difﬁculty when the object of interest is severely occluded, textured or with similar appearance to the background.
In addition, our work mainly takes advantage of appearance instead geometric cues.
Thus using generative method to perform geometry inpainting may lead to a fruitful direction for future exploration.
Third, the dependence on a pretrained feature extractor should be alleviated.
Conclusion
In this work, we present the ﬁrst approach in NeRF for tackling unsupervised multi-view image segmentation using radiance ﬁeld propagation with a bidirectional photometric loss to guide the reconstruction of semantic ﬁeld.
We design an EM-based mask reﬁnement procedure to handle complex scenes.
Our experiments show the effectiveness of our approach, with useful applications on NeRF such as individual object rendering and editing
Acknowledgements
We would like to thank Yichen Liu and Shengnan Liang for fruitful discussion at the inception stage of the project.
This research is supported in part by the Research Grant Council of the Hong Kong
SAR under grant no.