SCONE: Surface Coverage Optimization in Unknown Environments by Volumetric Integration
Abstract
Next Best View computation (NBV) is a long-standing problem in robotics, and consists in identifying the next most informative sensor position(s) for reconstructing a 3D object or scene efficiently and accurately.
Like most current methods, we consider NBV prediction from a depth sensor like Lidar systems.
Learningbased methods relying on a volumetric representation of the scene are suitable for path planning, but have lower accuracy than methods using a surface-based representation.
However, the latter do not scale well with the size of the scene and constrain the camera to a small number of poses.
To obtain the advantages of both representations, we show that we can maximize surface metrics by Monte
Carlo integration over a volumetric representation.
In particular, we propose an approach, SCONE, that relies on two neural modules: The first module predicts occupancy probability in the entire volume of the scene.
Given any new camera pose, the second module samples points in the scene based on their occupancy probability and leverages a self-attention mechanism to predict the visibility of the samples.
Finally, we integrate the visibility to evaluate the gain in surface coverage for the new camera pose.
NBV is selected as the pose that maximizes the gain in total surface coverage.
Our method scales to large scenes and handles free camera motion: It takes as input an arbitrarily large point cloud gathered by a depth sensor as well as camera poses to predict NBV.
We demonstrate our approach on a novel dataset made of large and complex 3D scenes.
Introduction
Next Best View computation (NBV) is a long-standing problem in robotics [6, 29], which consists in identifying the next most informative sensor position(s) for reconstructing a 3D object or scene efficiently and accurately.
Typically, a position is evaluated on how much it can increase the total coverage of the scene surface.
Few methods have relied on Deep Learning (DL) for the NBV problem, even though DL can provide useful geometric prior to obtain a better prediction of the surface coverage [33, 16, 25].
Like most current methods, we consider NBV prediction from a depth sensor.
Existing methods based on a depth sensor rely either on a volumetric or on a surface-based representation of the scene geometry.
Volumetric mapping-based methods can compute collision efficiently, which is practical for path planning in real case scenarios [20, 23, 24, 14, 1, 7].
However, they typically rely on voxels or a global embedding [12, 3, 21, 22, 27, 5] for the scene, which results in poor accuracy in reconstruction and poor performance in NBV selection for complex 3D objects.
On the contrary, surface mapping-based methods that process directly a dense point cloud of the surface as gathered by the depth sensor are efficient for NBV prediction with high-detailed geometry.
They are however limited to very specific cases, generally a single, small-scale, isolated object with the camera constrained to stay on a sphere centered on the object [15, 24, 8, 4, 13, 14, 14, 33, 16, 25].
Thus, they cannot be applied to the exploration of 3D scenes.
As shown in Figure 1, we introduce a volumetric DL method that efficiently identifies NBVs for unknown large-scale 3D scenes in which the camera can move freely.
Instead of representing the scene with a single global embedding, we choose to use a grid of local point clouds, which scales much better to large and complex 3D scenes.
We show how to learn to predict the visibility of unseen 3D points in all directions given an estimate of the 3D scene geometry.
We can then integrate these visibilities in the direction of any camera by using a Monte Carlo integration approach, which allows us to optimize the camera pose to find the next most informative views.
We call our method SCONE, for Surface Coverage Optimization in uNknown Environments.
In this respect, we introduce a theoretical framework to translate the optimization of surface coverage gain, a surface metric on manifolds that represents the ability of a camera to increase the visible area of the surface, into an optimization problem on volumetric integrals.
Such a formalism allows us to use a volumetric mapping of geometry, which is convenient not only to scale the model to exploration tasks and scene reconstruction, but also to make probabilistic predictions about geometry.
In particular, given a partial point cloud gathered by a depth sensor, our model learns to make predictions with virtually infinite resolution about the occupancy probability in the scene volume by learning a deep implicit function [17, 28, 18, 30, 31, 19].
Such predictions scale to very large point clouds since they depend only on neighborhood geometry.
Then, our model leverages a self-attention mechanism [26] to predict occlusions and compute informative functions mapped on a continuous sphere that represent visibility likelihood of points in all directions.
The occupancy probability field is finally used as a basis to sample points and compute Monte Carlo integrals of visibility scores.
Since NBV learning-based methods are mostly limited to single, small-scale, centered object reconstruction in literature, we first compare the performance of our model to the state of the art on the
ShapeNet dataset [2], following the protocol introduced in [33].
While our method was designed to handle more general frameworks such as 3D scene reconstruction and continuous cameras poses in the scene, it outperforms the state of the art for dense reconstruction of objects when the camera is constrained to stay on a sphere centered on the object.
We then conduct experiments in large 3D environments using a simple planning algorithm that builds a camera trajectory online by iteratively selecting NBVs with SCONE.
Since, to the best of our knowledge, we propose the first supervised
Deep Learning method for such free 6D motion of the camera, we created a dataset made of several large-scale scenes under the CC License for quantitative evaluation.
We made our code and this dataset available for allowing comparison of future methods with SCONE on our project webpage: https://github.com/Anttwo/SCONE .
Approach
Let us consider a depth sensor exploring a 3D scene, at time step t ≥ 0 .
Using its observations at discrete time steps j with 0 ≤ j ≤ t , the sensor has gathered a cloud of points distributed on the surface of the scene.
We refer to this cloud as the partial surface point cloud , as it describes the part of the surface seen –or covered – by the sensor in the scene.
To solve the NBV problem, we want to identify a camera pose that maximizes the coverage of previously unseen surface.
To this end, our method takes as input the partial surface point cloud as well as the history of 6D camera poses at time steps j ≤ t ( i.e. all previous positions and orientations of the sensor).
Our integrate visibility over approach is built around two successive steps, each relying on a dedicated neural module as shown in figure 2: First, we make a prediction about the geometry of the scene, to estimate where the uncovered points could be.
Then, we predict the visibility gain of uncovered points from any new camera pose;
The NBV is finally selected as the camera with the most new visible points in its field of view.
Although we seek to maximize a surface metric such as surface coverage gain, our method relies on a volumetric representation of the object or scene to reconstruct.
In this regard, we show that we can maximize a surface metric by integrating over a volumetric representation with virtually infinite resolution.
As we argue below, such a representation is not only useful for collision-free robot navigation but is also much more efficient for optimizing surface coverage gain than the alternative of identifying the 3D points lying on the surface, which is difficult in an unknown and occluded environment.
More exactly, we derive a volumetric integral that is asymptotically proportional to the surface coverage gain metric, which is enough for our maximization problem.
In the following subsection, we first present this derivation considering volume represented as a perfect binary occupancy map.
We then present the two neural modules of SCONE and explain how we use them to predict all terms involved in the volumetric integral, by leveraging neural models and self-attention mechanisms to predict occupancy and occlusions.
2.1 Maximizing Surface Coverage Gain on a Binary Occupancy Map
Here, we consider a binary occupancy map σ : R 3 → { 0 , 1 } representing the volume of the target object or scene.
We will relax our derivations to a probabilistic occupancy map when looking for the next best view in the next subsections.
From the binary map σ , we can define the set χ of occupied points, i.e. , the set of points x verifying σ ( x ) = 1 , its surface as the boundary ∂χ , and the surface coverage C ( c ) achieved by a camera pose c
= ( c pos , c rot ) ∈ C :
= R 3 × SO (3) as the following surface integral:
[Equation] [Equation]
where | ∂χ | S :
= (cid:82) ∂χ
d x is the area of surface ∂χ .
χ c ⊂ χ is the subset of occupied points contained in the field of view of camera c , and v c ( x ) is the visibility of point x from camera c , i.e. , v c ( x ) =
χ c ( x ) · 1 ( σ ( { (1 − λ ) c pos + λx such that λ ∈ [0 , 1) } )
= { 0 } ) .
Since we want to maximize the total coverage of the surface by all cameras during reconstruction, we are actually interested in maximizing the coverage of previously unobserved points rather than the absolute coverage.
Given a set of previous camera poses, which we call the camera history
H ⊂ C , and a 3D point x , we introduce the knowledge indicator γ H : R 3 → { 0 , 1 } such that γ H ( x ) = max { v c ( x ) : c ∈ H } .
We then define the coverage gain G H ( c ) of camera pose c as:
[Equation]
where ν Hc ( x ) =
(1 − γ H ( x )) · v c ( x ) is the visibility gain of x in χ c , for camera history H .
This function is equal to 1 iff x is visible at pose c but was not observed by any camera pose in H .
Given a camera history H , our goal is to identify a pose c that maximizes G H ( c ) .
Given an occupancy map σ , we could evaluate the integral in Eq.
(2) by simply sampling points p on surface ∂χ .
However, in practice we will estimate the occupancy map iteratively in an unknown environment, and we will only have access to an occupancy probability distribution.
Extracting surface points from such a probabilistic occupancy map gives results that can differ a lot from the true surface: Indeed, in 3D, a surface acts as a very concentrated set with zero-measure, and requires high confidence to give meaningful results.
Instead of extracting surface points, we extend the properties of such points to a small spherical neighborhood of the surface.
This will allow us to replace the maximization of a surface metric by the maximization of a volumetric integral, which is much easier to compute from our volumetric representation.
More exactly, we assume there exists a quantity µ 0 > 0 such that any volume point in the spherical neighborhood T ( ∂χ, µ 0 ) :
= (cid:8) p ∈ R 3 | ∃ x ∈ ∂χ, ∥ x − p ∥ 2 < µ 0 (cid:9) keeps the same visibility property as its neighboring surface points.
With such a hypothesis, we give a thickness to the surface, which makes sense when working with discrete points sampled in space to approximate a volume.
To this end, we introduce a new visibility gain function g Hc to adapt the definition of the former visibility gain ν Hc .
For any 0 < µ < µ 0 :
[Equation]
where N is the inward normal vector field.
With further regularity assumptions about the surface that are detailed in the appendix, such quantities are well defined.
Assuming µ 0 is small enough, the following explicit formula translates the surface approach into a volume integral for any camera pose c ∈ C and µ < µ 0 :
[Equation]
with W x 0 the Weingarten map at x 0 , that is, the Hessian of the signed distance function on the boundary of χ , which is continuous on the scene surface, assumed to be compact [10].
By developing the determinant, we find that det( I − λW x 0 )
=
1 + λb ( λ, x 0 ) where b is a bounded function on the compact space [ − µ, µ ] × ∂χ .
Moreover, for all x 0 ∈ ∂χ , we have by definition g H c ( µ ; x 0 + λN ( x 0 ))
=
g H c ( µ ; x 0 ) =
ν H c ( x 0 )
when 0 ≤ λ < µ , and g H c ( µ ; x 0 + λN ( x 0 ))
=when − µ < λ < 0 .
It follows that, for every 0 < µ < µ 0 :
[Equation]
The complete derivations are given in the appendix.
Function g Hc ( µ ; · ) is naturally equal to 0 for every point outside T ( ∂χ, µ ) .
Moreover, considering the regularity assumptions we made on the compact surface, if µ 0 is chosen small enough then for all x 0 ∈ ∂χ, µ < µ 0 , the point x 0 + µN ( x 0 ) is located inside the volume, such that (cid:82) T ( ∂χ,µ ) g
Hc ( µ ; x ) d x = (cid:82) χ g Hc ( µ ; x ) d x .
Since | g Hc ( µ ; · ) | ≤ 1 for all c ∈ C and µ > 0 , we deduce the following theorem by bounding | b | on [ − µ, µ ] × ∂χ :
Theorem 1.
Under the previous regularity assumptions on the volume χ of the scene and its surface ∂χ , there exist µ 0 > 0 and M > 0 such that for all µ < µ 0 , and any camera c ∈ C :
[Equation]
[Equation]
where | χ | V is the volume of χ .
This theorem states that, asymptotically for small values of µ , the volume integral (cid:82) χ g Hc ( µ ; x ) d x gets proportional to the surface coverage gain values G H ( c ) that we want to maximize.
This result is convenient since a volume integral can be easily approximated with Monte-Carlo integration on the volume and a uniform dense sampling based on the occupancy function σ .
Consequently, the more points we sample in the volume, the smaller µ we can choose, and the closer maximizing the volume integral of spherical neighborhood visibility gain gets to maximizing the surface coverage gain.
2.2 Architecture
To approximate the volumetric integral in Equation 6 for any camera pose c , we need to compute χ c as well as function g Hc .
In this regard, we need to compute both the occupancy map and the visibility gains of points for any camera pose.
Since the environment is not perfectly known, we predict each one of these functions with a dedicated neural module.
The first module takes as input the partial point cloud gathered by the depth sensor to predict the occupancy probability distribution.
The second module takes as input a camera pose, a feature representing camera history as well as a predicted sequence of occupied points located in the camera field of view to predict visibility gains.
Predicting the occupancy probability field ˆ σ .
The occupancy function σ is not known perfectly in practice.
To represent occupancy, most volumetric NBV methods rely on memory-heavy representations (like an occupancy 3D-grid or a volumetric voxelization), that are generally less efficient for encoding fine details and optimizing dense reconstructions, and will necessarily downgrade the resolution compared to a point cloud directly sampled on the surface.
To address this issue while still working with a volumetric representation of the scene, we use a deep implicit function to encode the 3D mapping of occupancy efficiently.
Such a function has a virtually infinite resolution, and prevents us from saving a large 3D grid in memory.
We thus approximate σ with the first module of our model, which consists of a neural network ˆ σ : P ( R 3 ) × R 3 → [0 , 1] that takes as inputs a partial surface point cloud P H ⊂ R 3 and query points x ∈ R 3 , and outputs the occupancy probability ˆ σ ( P H ; x ) of x .
P H is obtained by merging together all depth maps previously captured from cameras in H .
As shown in Figure 3, rather than using a direct encoding of the global shape of P H as input to ˆ σ , we take inspiration from [9] to achieve scalability and encode P H using features computed from the points’ neighborhoods.
The difference with [9] is that we rely on a multiscale approach: For a given query 3D point x , these features are computed from the k nearest neighbors of x computed at different scales.
For each scale s , we downsample point cloud P H around x into a sparser point cloud P ( s ) H before recomputing the nearest neighbors p ( s ) i ( x ) , i = 1 , ..., k of x : In this way, the size of the neighborhood increases with scale.
Next, for each value of s , we use small attention units [26, 11] on the sequence of centered neighborhood points ( p ( s ) 1 ( x ) − x, ..., p ( s ) k
( x ) − x ) and apply pooling operations to encode each sequence of k neighbors into a single feature that describes the local geometry for the corresponding scale.
We finally concatenate these different scale features with another uncentered global feature as well as the query point x , and feed them to an MLP to predict the occupancy probability.
The last global feature aims to provide really coarse information about the geometry and the location of x in the scene.
This model scales well to large scenes: Adding points from distant views to the current partial point cloud does not change the local state of the point cloud.
To avoid computing neighborhoods on the entire point cloud when reconstructing large scenes, we partition the space into cells in which we store the points in P H .
Given a query point x , we only use the neighboring cells to compute p ( s ) i ( x ) .
Predicting the visibility gain g
Hc
To maximize surface coverage gain, we need to compute the volumetric integral of visibility gain functions g Hc .
We do this again by Monte Carlo sampling, however, in unknown environments we cannot compute explicitly occlusions to derive visibility gain functions g
Hc since the geometry, represented as a point cloud, is partially unknown and sparser than a true surface.
We thus train the second module of our model to predict visibility gain functions by leveraging a self-attention mechanism that helps to estimate occlusion effects in the point cloud P H .
In particular, for any camera pose c ∈ C and 3D point x ∈ χ c , the second module derives its prediction of visibility gains from three core features: (i)
The predicted probability ˆ σ ( P ; x ) of x to be occupied, (ii) the occlusions on x by the subvolume χ c and (iii) the camera history H .
To feed all this information to our model in an efficient way, we follow the pipeline presented in Figure 4.
The model starts by using the predicted occupancy probability function ˆ σ to sample 3D points in the volume χ .
These samples will be used for Monte Carlo integration.
We refer to these points as proxy points as we use them to encode the volume in the camera field of view, i.e. , in a pyramidal frustum view.
We write ˆ χ as the discrete set of sampled proxy points, and ˆ χ c as the set of proxy points located in the field of view of the camera c .
We first encode these proxy points individually by applying a small MLP on their 3D coordinates and their occupancy probability value concatenated together.
Then, our model processes the sequence of these encodings with a self-attention unit to handle occlusion effects of subvolume χ c on every individual point.
Note there is no pooling operation on the output of this unit: The network predicts per-point features and does not aggregate predictions, since we do it ourselves with Monte Carlo integration.
Next, for each proxy point x ∈ ˆ χ , we compute an additional feature h H ( x ) that encodes the history of camera positions H with respect to this point as a spherical mapping: It consists in the projection on a sphere centered on x of all camera positions for which x was in the field of view.
These features are concatenated to the outputs of the self-attention unit.
Our model finally uses an MLP on these features to predict the entire visibility gain functions of every point x as a vector of coordinates in the orthonormal basis of spherical harmonics.
With such a formalism, the model is able to compute visibility gains for points inside a subvolume in all directions with a single forward pass.
In this regard, we avoid unnecessary computation and are able to process a large number of cameras in the same time when they share the same proxy points in their field of view ( e.g. , reconstruction of a single object centered in the scene, where ˆ χ c
= ˆ χ for all c , or when several cameras observe the same part of the 3D scene, i.e. , ˆ χ c = ˆ χ ′ ⊂ ˆ χ for several c ).
Formally, if we denote by Y ml : S 2 → R the real spherical harmonic of rank ( l, m ) and ϕ ml (ˆ χ c ; x, h H ( x ))
the predicted coordinate of rank ( l, m ) for proxy point x ∈ ˆ χ c with attention to subset ˆ χ c and camera history feature h H ( x ) , the visibility gain of point x in direction d ∈ S 2 is defined as
[Equation]
so that the coverage gain G H ( c ) of a camera pose c ∈ C is proportional to
[Equation]
The next best view among several camera positions is finally defined as the camera pose c ∗ with the highest value for I H ( c ∗ ) .
Equation 8 is a Monte Carlo approximation of the volumetric integral in
Equation 6, where the occupancy map and the visibility gains are predicted with neural networks.
We choose to use a Monte-Carlo integral rather than a neural aggregator because this approach is simple, fast, makes training more stable, has good performance, better interpretability, and can handle sequences of arbitrary size.
In particular, it implicitly encourages our model to compute meaningful visibility gains for each point since there is no asymmetry between the points.
We also use spherical harmonics to encode the camera history encoding h H ( x ) of each point x , which makes it a homogeneous input to the predicted output.
Consequently, this input comes at the end of the architecture, and aims to adapt the visibility gain according to previous camera positions.
This convenient representation, inspired by [32], allows us to handle free camera motion; On the contrary, several models in the literature encode the camera directions on a discrete sphere [33, 16, 25].
Training.
We train the occupancy probability module alone with a Mean Squared Error loss with the ground truth occupancy map.
We do not compute ground-truth visibility gains to train the second module since it would make computation more difficult and require further assumptions: We supervise directly on I H ( c ) by comparing it to the ground-truth surface coverage gain for multiple cameras, with softmax normalization and Kullback-Leibler divergence loss.
Extensive details about the training of our modules and the choices we made are given in the appendix.
Experiments
As discussed in the introduction, deep learning-based NBV models for dense reconstruction are currently limited to single, small-scale, centered object reconstruction.
To compare SCONE to these previous methods in this context, we first constrain the camera pose to lie on a sphere centered on an object.
We then introduce our dataset made of 13 large-scale 3D models that we created to evaluate
SCONE on free camera motions (3D models courtesy of Brian Trepanier, Andrea Spognetta, and 3D
Interiors, under CC License; all models were downloaded from the website Sketchfab).
3.1 Next Best View for Single Object Reconstruction
We first compare the performance of our model to the state of the art on a subset of the ShapeNet dataset [2] introduced in [33] and following the protocol of [33]: We sample 4,000 training meshes from 8 specific categories of objects, 400 validation meshes and 400 test meshes from the same categories, and 400 additional test meshes from 8 categories unseen during training.
The evaluation on the test datasets consists of 10-view reconstructions of single objects.
Given a mesh in the dataset, camera positions are discretized on a sphere.
We start the reconstruction process by selecting a random camera pose, then we iterate NBV selection 9 times in order to maximize coverage with a sequence of 10 views in total.
The evaluation metric is the area under the curve (AUC) of surface coverage throughout the reconstruction.
This criterion not only evaluates the quality of final surface coverage, but also the convergence speed toward a satisfying coverage.
Results are presented in Table 1.
Further details about the evaluation, the estimation of ground truth surface coverage gains, and the metric computation are available in the appendix.
[Table]
[Figure]
3.2 Active View Planning in a 3D Scene
To evaluate the scalability of our model to large environments as well as free camera motion in 3D space, we also conducted experiments using a naive planning algorithm that incrementally builds a path in the scene: We first discretize the camera poses in the scene on a 5D grid, corresponding to coordinates c pos =
( x c , y c , z c ) of the camera as well as the elevation and azimuth to encode rotation c rot .
The number of poses depends on the dimensions of the bounding box of the scene.
This box is an input to the algorithm, as a way for the user to tell which part of the scene should be reconstructed.
In our experiments, the number of different poses is around 10,000.
Note we did not retrain our model for such scenes and use the previous model trained on ShapeNet as explained in Section 3.1.
The depth sensor starts from a random pose (from which, at least, a part of the main structure to reconstruct is visible).
Then, at each iteration, our method estimates the coverage gain of direct neighboring poses in the 5D grid, and selects the one with the highest score.
The sensor moves to this position, captures a new partial point cloud and concatenates it to the previous one.
Since we focus on online path planning for dense reconstruction optimization and designed our model to be scalable
[Figure] [Figure]
with local neighborhood features, the size of the global point cloud can become very large.
We iterate the process either 100 times to build a full path around the object in around 5 minutes on a single
Nvidia GTX1080 GPU, and recovered a partial point cloud with up to 100,000 points.
Contrary to single, small-scale object reconstruction where the object is always entirely included in the field of view, we simulated a limited range in terms of field of view and depth for the depth sensor, so that the extent of the entire scene cannot be covered in a single view.
Thus, taking pictures at long range and going around the scene randomly is not an efficient strategy; the sensor must find a path around the surface to optimize the full coverage.
We use a ray-casting renderer to approximate a real Lidar.
More details about the experiments can be found in the appendix.
We compared the performance of SCONE with simple baselines:
First, a random walk, which chooses neighboring poses randomly with uniform probabilities.
Then, we evaluated an alternate version of our model, which we call SCONE-Entropy, which leverages the first module of our full model to compute occupancy probability in the scene, then selects the next best view as the position that maximizes the Shannon entropy in its field of view.
The comparison is interesting, since SCONEEntropy adapts classic NBV approaches based on information theory to a deep learning framework.
Figures 5 and 6 provide qualitative results.
Figure 7 shows the convergence speed of covered surface by SCONE and our two baselines, averaged on all scenes of the dataset.
Despite being trained only on ShapeNet 3D models, our method is able to compute meaningful paths around the structures and consistently reach satisfying global coverage.
Since we focused on building a metric for NBV computation, this experiment is simple and does not implement any further prior from common path planning strategies: For instance, we do not compute distant NBV, nor optimal paths to move from a position to a distant NBV with respect to our volumetric mapping.
There is no doubt the model could benefit from further strategies inspired by the path planning literature.
[Figure]
3.3 Ablation Study
We now provide an ablation study for both modules of our full model:
The prediction of occupancy probability, and the computation of coverage gain from predictions about visibility gains.
To evaluate both modules separately, we compared their performance on their training critera: MSE for occupancy probability, and softmax followed by KL-Div on a dense set of cameras for coverage gain.
The values are reported in Figure 8.
We provide extensive details and analysis in the appendix.
Occupancy probability.
Apart from the base architecture presented in Figure 3, we trained the first module under two additional settings.
First, we removed the multi-scale neighborhood features computed by downsampling the partial point cloud P H , and trained the module to predict an occupancy probability value ˆ σ only from global feature g
( P H ) and encoding f ( x ) .
As anticipated, the lack of neighborhood features not only prevents the model from an efficient scaling to large scenes, but also causes a huge loss in performance.
We also trained a slightly more complex version of the module by feeding it the camera history harmonics h H ( x ) as an additional feature.
It appears this helps the model to increase its confidence and make better predictions, but the gains are quite marginal.
Visibility gain.
We trained two variations of our second module.
First, we completely removed the geometric prediction: We use directly the surface points as proxy points, mapped with an occupancy probability equal to 1.
As a consequence, the model suffers from a significant loss in performance to compute coverage gain from its predicted visibility gain functions.
Thus, we can confirm that the volumetric framework improves the performance of SCONE.
This result is remarkable, since surface representations usually make better NBV predictions for dense reconstruction of detailed objects in literature.
On the contrary, we show that our formalism is able to achieve higher performance by leveraging a volumetric representation with a high resolution deep implicit function ˆ σ .
We trained a second variation without the spherical mappings h H of camera history.
We verified that such additional features increase performance, especially at the end of a 3D reconstruction.
Limitations and Conclusion
Beyond the prediction of the Next Best View, our method SCONE is able to evaluate the value of any camera pose given a current estimate of the scene volume.
We demonstrated this ability with a simple path planning algorithm that looks for the next pose around the current pose iteratively.
However:
• Like current methods, SCONE relies on a depth sensor and assumes that the captured depth maps are perfect.
In practice, such a sensor is not necessarily available and can also be noisy.
It would be very interesting to extend the approach to color cameras.
• Also like current methods, it is limited to the prediction of the camera pose for the next step.
This is greedy, non-optimal as multiple poses are required anyway for a complete 3D reconstruction.
Thanks to its scalability, we believe that our method could be extended to the prediction of the “Next Best Path”, where future camera poses would be predicted jointly for their complementarity.
Acknowledgements
This work was granted access to the HPC resources of IDRIS under the allocation 2022-AD made by GENCI.
We thank Renaud Marlet, Elliot Vincent, Romain Loiseau, and Tom Monnier for inspiring discussions and valuable feedback.
[-= 
-]
[-Let p be a point belonging to a new depth map computed by the sensor; then, we identify the cell in which p is located, and add p to the point cloud P H if and only if for all points p 0 in the cell, ∥ p − p 
-]