V IEW S YNTHESIS WITH S CULPTED N EURAL P OINTS
A BSTRACT
We address the task of view synthesis, generating novel views of a scene given a set of images as input.
In many recent works such as NeRF (Mildenhall et al., 2020), the scene geometry is parameterized using neural implicit representations ( i.e. , MLPs).
Implicit neural representations have achieved impressive visual quality but have drawbacks in computational efficiency.
In this work, we propose a new approach that performs view synthesis using point clouds.
It is the first pointbased method that achieves better visual quality than NeRF while being 100 × faster in rendering speed.
Our approach builds on existing works on differentiable point-based rendering but introduces a novel technique we call “Sculpted Neural Points (SNP)”, which significantly improves the robustness to errors and holes in the reconstructed point cloud.
We further propose to use view-dependent point features based on spherical harmonics to capture non-Lambertian surfaces, and new designs in the point-based rendering pipeline that further boost the performance.
Finally, we show that our system supports fine-grained scene editing.
Code is available at https://github.com/princeton-vl/SNP .
I NTRODUCTION
We address the task of view synthesis: generating novel views of a scene given a set of images as input.
It has important applications including augmented and virtual reality.
View synthesis can be posed as the task of recovering from existing images a rendering function that maps an arbitrary viewpoint into an image.
In many recent works, this rendering function is parameterized using neural implicit representations of scene geometry (Mildenhall et al., 2020; Yu et al., 2021c; Park et al., 2021; Garbin et al., 2021; Niemeyer et al., 2021).
In particular, NeRF (Mildenhall et al., 2020) represents 3D geometry as a neural network that maps a 3D coordinate to a scalar indicating occupancy.
Implicit neural representations have achieved impressive visual quality but are typically computationally inefficient.
To render a single pixel, NeRF needs to evaluate the neural network at hundreds of 3D points along the [Figure] ray, which is wasteful because most of the 3D spaces are unoccupied.
NeRF’s implicit representation also makes it inflexible for scene editing operations such as deformation, which is important for downstream applications including augmented reality and video games.
Several works enable
NeRF to do scene editing (Lombardi et al., 2019; Liu et al., 2021; Yang et al., 2021a; Pumarola et al., 2021), but either the way of editing is highly constrained, or images captured under all desired object poses are required.
On the other hand, this limitation is easily overcome by explicit representations such as meshes or point clouds.
To rasterize a mesh or a point cloud, no computation is wasted on unoccupied 3D spaces.
Scene editing operations such as composition and deformation is also straightforward.
Moreover, rasterizing meshes or point clouds is a mature technology already widely deployed in the industry for movies and video games, capable of producing real-time performance and high realism.
An intriguing question is whether we can achieve state-of-the-art visual quality by using explicit representations such as point clouds.
The basic framework of point-based neural rendering is to represent the scene as a featurized point cloud, which is reconstructed through a multiview stereo (MVS) system.
The features are learned by maximizing photoconsistency on the input images via differentiable rendering.
Although this framework has been studied in several recent works (Aliev et al., 2020; Wiles et al., 2020; Lassner & Zollhofer, 2021), the overall rendering quality still lags behind NeRF, mainly due to the ghosting effects and blurriness caused by the errors in geometry.
Our approach adopts this basic framework but introduces a novel technique we call “Sculpted Neural Points (SNP)”, which significantly improves the robustness to the errors and holes in the reconstructed point cloud.
The idea is to “sculpt” the initial point cloud reconstructed by the MVS system.
In particular, we remove existing points and add additional points to improve the photo-consistency of the renders against input images.
These sculpting decisions are discrete in nature, but are tightly coupled with gradient-based optimization of the continuous per-point features.
We further propose a few novel designs in the point-based rendering pipeline that boost the performance.
We use spherical harmonics (SH) in high-dimensional point feature space to capture non-Lambertian visual effects, which is faster and better than using MLPs.
Inspired by Dropout (Srivastava et al., 2014), we propose a point dropout layer that significantly improves the generalization to novel views.
Last but not least, we find that it is essential to not use any normalization layers in the U-Net.
Compared to previous works that use point cloud-based representation, ours is the first model that achieves better rendering quality than NeRF, while being 100 × faster in rendering, and reducing the training time by 66%.
We evaluate our method on common benchmarks including DTU (Jensen et al., 2014), LLFF (Mildenhall et al., 2019), NeRF-Synthetic (Mildenhall et al., 2020), and Tanks&Temples (Knapitsch et al., 2017), and our method shows better or comparable performance against all baselines.
Finally, we show that our model allows fine-grained scene editing in a user-friendly way.
Compared to previous works that can only do object-level composition (Lombardi et al., 2019; Yu et al., 2021b; Yang et al., 2021b) or require a special user interface (Liu et al., 2021), our point-based representation inherently supports editing at a finer resolution, and users can use existing graphics toolbox to edit the point cloud without any custom changes.
The main contributions of this paper are three-fold: 1) We propose a novel point-based approach to view synthesis, “Sculpted Neural Points”, a technique that is key to achieving high quality and viewconsistent output; 2) We demonstrate, for the first time, that a point-based method can achieve better visual quality than NeRF while being 100 × faster in rendering.
We propose several improvements to the point-based rendering pipeline that significantly boost the visual quality.
R ELATED W ORK
Methods for view synthesis can be categorized based on how they represent the scene geometry.
View Synthesis with Implicit Representations NeRF (Mildenhall et al., 2020) uses a neural network to map a 3D spatial location to volume density.
To render a pixel, the neural network needs to be repeatedly evaluated along the ray, making rendering computationally expensive.
Followup works on NeRF (Yu et al. , 2021c;b; Park et al., 2021) focus on improving the speed or the general- ization ability to new scenes or with a limited number of reference views.
Our method does not use an implicit representation; instead, we explicitly represent the scene geometry using a point cloud, which allows much faster rendering, as well as easy and flexible scene editing.
View Synthesis with Point Clouds To use point clouds for view synthesis, existing approaches typically use an external multiview stereo system to reconstruct a point cloud from the input images, and then optimize the features and 3D positions of each point through differentiable rendering.
NPBG (Aliev et al., 2020) is the first work to combine neural rendering with point clouds; it uses a featurized point cloud to represent the scene, and rasterizes with one-pixel point splats at multiple scales followed by a post-processing U-Net to fill the holes.
SynSin (Wiles et al., 2020) proposes a soft rasterization pipeline that allows better gradient flow and produces smoother results.
Our method achieves significantly better visual quality compared to them.
Pulsar (Lassner & Zollhofer, 2021) uses featurized spheres to represent the scene, and proposes a very efficient soft rasterizer that can rasterize millions of spheres in less than 100 milliseconds.
Pulsar qualitatively shows that geometry reconstruction can be done through its differentiable rendering system, but has shown no quantitative results on view synthesis.
We adopt Pulsar as our backbone.
There are a few concurrent works using point cloud representations.
NPBG++ (Rakhimov et al., 2022) focuses on lifting the requirement of per-scene optimization.
It does not revise the geometry and is thus more sensitive to the point cloud quality compared to ours.
It also proposes to use viewdependent point features, but parameterized as MLP rather than SH as we do.
ADOP (R ¨ uckert et al., 2022) mainly focuses on unbounded outdoor scenes with large exposure changes among views.
Point-NeRF (Xu et al., 2022) uses a featurized point cloud to represent the scene geometry, but renders with volume rendering instead of rasterization.
To revise the initial point cloud provided by an MVS system, existing differential renderers compute gradients with respect to the 3D coordinates of each point.
Pulsar (Lassner & Zollhofer, 2021) approximates the gradient by modeling points as spheres with a certain radius, with the color blending weights changing smoothly with respect to the distance of the camera ray to the sphere center.
ADOP (R ¨ uckert et al., 2022) instead computes the partial derivatives of the photometric loss with respect to the point positions by taking the finite difference in the pixel space.
While Point-NeRF (Xu et al., 2022) and our method both refine the point cloud by deleting and adding points, the difference are that: 1) their pruning is based on the volume density optimized for photo-consistency, while our point pruning is based on multiview consistency and doesn’t require test-time training; 2) their point growing progressively adds points near existing points, while our point adding has only one round and can add new points in any location.
Our method builds upon existing techniques of differentiable point-based rendering but differs substantially in how we revise the initial point cloud given by an MVS system.
Although we find prior methods capable of local revision around the existing points by adjusting their locations using the gradients, such local operations, however, cannot fill large holes or add new points in empty spaces far away.
In contrast, our sculpting technique is global.
It does not use gradients and can add new points in locations arbitrarily far away from existing points.
A PPROACH O VERVIEW
An overview of our approach is illustrated in Fig.
The input is a set of H × W RGB images { I 1 , . . .
I m } of m reference views and their corresponding intrinsic and extrinsic camera parameters, { C 1 , . . .
C m } .
We define the camera projection function Π and its inverse Π − 1 as follows:
[Equation]
(1) where K C , R C and t C are the intrinsics, rotation, and translation of camera C .
P ∈ R 3 is a 3D point and p ∈ R 2 is its 2D projection in the pixel space with depth d p .
Further, ([ X, Y, Z ] T ) ↓ is defined as ([ X/Z, Y/Z ] T ) .
Our approach consists of three main components: point cloud reconstruction, point cloud sculpting, and differentiable rendering.
In this section, we describe the backbone of our approach with only point cloud reconstruction and differentiable rendering, and leave point cloud sculpting to Sec. 4.
3. P OINT C LOUD R ECONSTRUCTION
We use an MVS network (Ma et al., 2022) to extract a dense depth map { D 1 , . . .
D m } ∈ R H 4 × Wfor each of the reference views.
Each depth map is un-projected into a set of 3D points by applying the inverse projection in Eqn.
The points from all depth maps are combined, without any filtering, to form a larger set of original 3D points P o = { p 1 , ..., p N } .
We associate point p i ∈ R 3 with a learnable K -dimensional feature vector f i ∈ R K and a scalar o i ∈ [0 , 1] representing its opacity .
D IFFERENTIABLE R ENDERING
Given a featurized 3D point cloud and a target view, we use a differentiable rendering function with learnable parameters to map the point cloud into an RGB image.
For each scene individually, we learn the parameters of this rendering function together with the point features through gradient descent to minimize photometric errors against the input images.
Spherical Harmonic Point Feature
We use the spherical harmonics (SH) functions to model the view-dependent effects.
Recently Yu et al.
(2021b;a) brings up the attention of using SH in neural rendering.
Unlike previous works that use SH directly in RGB space, we propose to use SH in highdimensional feature space, where each element of a vector is modulated by a set of SH coefficients.
We use the SH basis up to degree 2 , which yields 9 basis in total.
This choice follows Yu et al. (2021b) and we find it sufficient to capture highly non-Lambertian surfaces in our experiments.
For a 3D point p i , the SH layer takes its feature vector f i ∈ R K and a target view direction v as input, and outputs a modulated feature vector s
i ∈
R K 9 .
Specifically, we first compute the SH basis according to v , yielding a basis vector b v ∈ R 9 × 1 .
We then reshape f i into f ′ i ∈
R K 9 × 9 , and finally compute
s i with a dot product s i = f ′
i · b v .
Note that evaluating SH functions is cheap as it avoids complex matrix multiplication operations.
We find that it leads to better performance and faster rendering speed compared to the MLP parameterization used in NeRF, as shown in Sec. 5.2.
Differentiable Soft Rasterization We use soft rasterization proposed in Liu et al. ; Lassner & Zollhofer (2021) to convert the view-dependent features into a 2D feature map F given a target view.
Soft rasterization blends the features of multiple points hit by a camera ray with weights depending on their depth and opacities.
We refer the readers to Pulsar (Lassner & Zollhofer, 2021) for details.
Note that in addition to updating f i , we also compute the gradient of the photometric loss w.r.t.
the point positions p i and opacity o i , and optimize them through gradient descent, following Lassner &
Zollhofer (2021), which we show helpful in improving fine geometric details in our experiments.
Point Dropout Layer We find that the existing point rendering pipeline is prone to over-fitting, i.e. , the image quality on test views is much worse than on training views.
The reasons are two-fold: 1.
The “training set” for view synthesis consists of only tens of images, and there are barely any data augmentation techniques that can be applied except random cropping.
Some points are covered by their neighbors in training views, but get visible in test views.
The features for these points are not well-optimized.
The blending weights are very low for these points when the rasterizer is “soft”.
To resolve the above issue, we propose to use a “Point Dropout Layer” before rasterization.
In each forward pass, we randomly select a subset of points to feed into the rasterizer, whose size depends on the dropout rate p d .
Note that at inference time, we cannot simply rasterize all points and multiply the output by p d as in the neural network (NN) case, because the rasterization operation is non-linear in contrast to the matrix multiplication in NN.
Since it is impossible to traverse all subsets, we simply rasterize L multiple random subsets and average the output feature maps at inference time.
When rendering videos, we find that sampling independently for each frame causes obvious flickering artifacts.
Therefore, we use the same subsets across all frames, which leads to a better consistency.
Intuitively, the point dropout layer allows us to train on an ensemble of point clouds, and give all points a chance to get optimized even if they are covered, and thus alleviating the over-fitting problem.
As a side effect, we also gain a speed-up because fewer points get rasterized.
Although the design is simple, this idea has not been explored in previous works, and our experiments show that it significantly improves the image quality on test views.
2-D Rendering without BatchNorm Given a target view, we convert the 2D feature map F into the RGB image I t with a 2D ConvNet.
We use a U-Net (Ronneberger et al., 2015) with two
[Figure]
downsampling layers and two upsampling layers, and optionally one more upsampling layer to produce high-resolution outputs.
The intuition behind using a U-Net is that it can remove noise in the feature map.
We use a dropout layer and relatively small point radius, leaving the rasterized feature map with tiny holes, which makes such denoising necessary.
The large receptive field of U-Net is favorable for denoising.
Previous works (Aliev et al., 2020; Rakhimov et al., 2022) directly use the original U-Net design with BatchNorm (Ioffe & Szegedy, 2015) layers, which we find unsuitable for the view synthesis task for two reasons.
First, the small training set size makes the estimation of the moving average in BatchNorm unstable.
Second, the benefit of accelerated training is minimal since the network is shallow.
Therefore, we use no normalization layer in our U-Net.
P OINT S CULPTING
The point clouds reconstructed from MVS usually contain many errors, even with the state-of-the-art
MVS systems.
The errors typically take the form of distorted or incomplete geometry.
If we directly use such point clouds, the synthesized views will have poor visual quality with salient artifacts.
To address this issue, we introduce a new technique we call “point sculpting”.
It has two steps, Point
Pruning and Point Adding .
The sculpting procedure and outputs are illustrated in Fig.
4. P OINT P RUNING
The MVS system we use produces a dense depth map for each input image.
Like other depth-based systems (Yao et al., 2018; Yan et al., 2020; Chen et al., 2020), it adopts a fusion step that merges the depth maps from different views into a final point cloud.
A geometry consistency check is often used to remove outlier points.
Using the depth maps, the consistency check projects a pixel into another view, reprojects the corresponding point back, and sees if the original pixel is recovered up to a threshold.
For example, COLMAP (Sch¨onberger & Frahm, 2016) defines the consistency error ψ i,j p between view i and j for pixel p as:
[Equation]
where q is p ’s corresponding point in view j .
Yan et al.
(2020) further propose an improved version, Dynamic Consistency Checking (DCC), which achieved the state-of-the-art filtering results.
The main problem of this type of forward-backward consistency check is that it tends to be overaggressive in filtering out points, resulting in highly incomplete geometry that is detrimental to view synthesis.
In datasets such as DTU and LLFF, many areas are only visible in a small number of views.
Those areas can easily be filtered out by this check as no confident match could be found.
Therefore, we take the raw depth maps from the multiview stereo system and propose a new technique for our own consistency checking geared toward view synthesis.
We check only the forward consistency to maximize completeness while still removing outliers.
Formally, a pixel p in view i passes the check if and only if m (cid:84)
[Equation]
(cid:3) , where q is p ’s corresponding point in view j (same as in Eqn. 2), d j q is the predicted depth of q in view j , D j ( · ) is the depth of a point in view j (the z value of a 3D point in camera j ’s coordinate), and δ d is a hyperparameter controlling the relative tolerance.
Intuitively, our point pruning method keeps a point as long as it is not significantly closer than the original surface to any reference view camera.
It filters out the points that are floating in the free space between the actual surface and the camera, which are likely to be outliers.
It also keeps all points that are only visible in a small number of views.
Although the position of such points may not be accurate, it is useful to keep them as candidates for further optimization.
4. P OINT A DDING
As Fig.
2 shows, after pruning, the point cloud can have holes, either due to points being pruned or incorrect depth estimates (e.g. depth estimates that are close to infinity or zero).
Previous works (Lassner & Zollhofer, 2021; Yifan et al., 2019) tackled this problem by performing gradient-based updates to the point locations.
However, such updates are limited to local changes of existing points and are unable to recover large areas of missing geometry.
We thus introduce a technique to add new points to the pruned point cloud.
The basic idea is to find a set of 3D points that, if added to the point cloud, could help minimize the photometric error after optimization of the point features.
Note that these new points do not need to be perfect; they just need to be a superset of the ground truth geometry, because the extraneous points can get optimized through the subsequent gradient-based optimization.
On the other hand, an excessive number of new points can lead to overfitting and slower rendering, so a good balance is needed.
Our point adding algorithm consists of two steps:
• Optimizing with existing points: We optimize the features and opacity of the current points through gradient descent until convergence.
For the i -th input image, we extract an error map between the rendered and ground-truth image: E i =
||
I gti −
I renderi || 1 .
Note that we use f i ∈ R 27 , which is converted to s i ∈ R 3 by the SH layer.
s i is directly treated as RGB values during rasterization, and we use no U-Net in this step, as the U-Net hallucination makes E i less informative.
Proposing new points: For a pixel ( u, v ) in an input view i , we check if its rendering error E i ( u, v ) is bigger than a pre-defined threshold δ e .
If so, we sample 3D points uniformly along the ray of the pixel within the bounds of the scene, and search for points that do not occlude any of the existing points in any of the input views.
If multiple such points exist, we propose the closest M points, where M is a hyperparameter.
We go through all pixels in all input views, collect all the proposed points, and add them to the existing point cloud.
The design of this algorithm builds upon the assumption that our rendering pipeline can approximate the radiance of each point arbitrarily well on the input images and that any high rendering error can only be caused by incomplete geometry, as those areas having no points covered can only take the default background color.
Based on this assumption, we propose new points for pixels with high rendering errors, but exclude points that occlude the existing surface in other views.
We propose up to M closest points and choose M = 5 in our experiments to strike a good balance between geometry coverage and rendering speed.
We can alternate between gradient-based optimization and point adding for multiple rounds, but in practice we find one round of point adding to be sufficient for good results.
We present the full details of the point adding algorithm in Appendix C.
E XPERIMENTS
Datasets We evaluate our method on DTU (Jensen et al., 2014), LLFF (Mildenhall et al., 2019; 2020), NeRF’s Realistic Synthetic 360 ◦ (Mildenhall et al., 2020), and Tanks&Temples (Knapitsch et al., 2017).
The datasets we choose provide good coverage of both forward-facing and 360 ◦ scenes.
We evaluate using the standard PSNR, SSIM, and LPIPS (Zhang et al., 2018) metrics.
Baselines We compare our model with NeRF (Mildenhall et al., 2020).
On DTU and LLFF, we run two point-based methods NPBG (Aliev et al., 2020) and SynSin (Wiles et al., 2020) using the same external MVS system to reconstruct point clouds, with no pruning or adding.
We additionally compare against two point-based methods NPBG++ (Rakhimov et al., 2022) and Point-NeRF (Xu et al., 2022), and two voxel-based methods NV Lombardi et al. and NSVF Liu et al.
Implementation Details We implement our method with PyTorch (Paszke et al., 2019) and PyTorch3D (Ravi et al., 2020).
We experiment on a single RTX 3090 GPU, optimizing for 50, steps on each scene with a batch size of 1.
We initialize all SH coefficients for each point as 0s and the point opacity as 1.
We initialize the U-Net parameters randomly.
We set the point radii to be a dataset-specific hyperparameter, which is the same for all points and fixed.
See Appendix B for details on the MVS network and other implementations.
5. P RIMARY R ESULTS
[Figure]
Results on DTU The quantitative comparison is presented in Tab. 1.
Results show that our model has better SSIM and LPIPS, and slightly worse PSNR compared to NeRF.
We present the visualizations in Fig.
13, Appendix F. We claim to use LPIPS as the major quality metric, as we find that PSNR and SSIM may not reflect actual visual quality because they are highly sensitive to small pixel shifts (See Appendix.
A).
[Figure]
[Table]
Results on LLFF The quantitative results are shown in Tab. 1.
Similar to DTU, our method achieves consistently better SSIM and LPIPS, while being slightly worse in PSNR.
Qualitative comparisons are shown in Fig.
3 and Fig. 12, Appendix F. Compared to NeRF, our model can reconstruct very fine details.
Our method also has significantly better visual quality compared NPBG and SynSin.
[Table]
Results on Tanks&Temples We present the numbers in Tab. 2.
All baselines numbers are from Point-NeRF (Xu et al., 2022).
Our method achieves comparable quality as
Point-NeRF, while being significantly better than other baselines.
We present qualitative comparisons in Fig.
16, Appendix F.
Results on NeRF-Synthetic Results are shown in Tab. 3.
Our method achieves comparable performance to NeRF while being worse than Point-NeRF, which is also reflected in Fig.
Our method is better at capturing the reflective drum surfaces, while struggles with the microphone which has fine geometry.
Our explanation is that our view-dependent point features are very expressive in modeling high-frequency textures, while our point cloud is not accurate enough in cases of fine geometries.
[Table] [Table]
5. A BLATION S TUDIES
[Table]
[Figure]
We conduct ablation studies of our proposed designs.
We show results in Tab. 4.
In the 1 st block, We compare with several baselines on point cloud refinement, including 1) the filtering algorithm DCC
(Yan et al., 2020) which achieves SOTA performance on MVS, 2) using the raw MVS point cloud without any pruning or adding, 3) pruning with the proposed point pruning but no point adding , 4) using the same point cloud as the complete model, while keeping the point positions and opacity values fixed during gradient updates.
Also see Fig.
5 for qualitative comparisons.
Results show that all proposed geometry refinement components contribute to the final model.
While point pruning contributes to sharper object boundaries near the head of the plush, point adding is especially helpful for filling large holes on the table in the rabbit scene.
See Fig.
11, Appendix C for visualizations of the point cloud generated by each method.
The 2 nd block shows that using SH reduces the layer latency by 82% and improves the PSNR by 0 .
38 dB compared to MLP.
For the MLP baseline, we use a 2-layer MLP with 256 hidden units, which takes as input the concatenation of the point feature and the positional-encoded view direction, following NeRF.
See also Fig.
14, Appendix F for visualization of the non-Lambertian effect learned by our model.
The 3 rd block shows that using dropout layer improves the PSNR by 1 . 28 dB, and the model is not sensitive to the dropout rate.
The 4 th block shows that not using any normalization layers improves the PSNR by 1 . 49 dB compared to using BatchNorm, and by 0 .
60 dB compared to
InstanceNorm.
Replacing U-Net with a 2-layer 1 × 1 Conv network gives significantly worse results.
5. S CENE E DITING
[Table]
We show that our system supports, with high fidelity, scene editing operations such as scene composition, object deformation, and erasing.
Results are shown in the inset figure on the right.
Composition is achieved by first co-training two scenes with separate point features and a shared U-Net, then putting the points into a single scene at inference time.
For deformation, we export the sculpted point cloud into MeshLab (Cignoni et al., 2008), where we manually select the moving part and its axis of rotation.
For erasing, we filter out points based on their z coordinates.
Compared to existing neural rendering pipelines that support scene editing, our system has two main advantages: 1.
Fine-grained editing : Previous works (Lombardi et al., 2019; Yu et al., 2021b; Yang et al., 2021b) use explicit representations like voxel grids, which are typically limited in resolution.
Therefore, they can only achieve object-level operations such as composition.
In comparison, we represent object surfaces densely with millions of points, so we can do fine-grained editing such as object deformation.
Ease of use : Previous works doing scene editing with NeRF either require a special interface to take user inputs (Liu et al., 2021), or a complex pipeline that uses meshes as an intermediate representation (Yuan et al., 2022).
In contrast, our point cloud representation is directly supported by nearly all graphics toolboxes such as MeshLab or Blender, which allows users to edit the scene intuitively without any specialized tool.
5. I NFERENCE S PEED , T RAINING T IME , AND M ODEL S IZE
We compare our model’s inference speed, training time, and model size with a few baselines on the
NeRF-Synthetic dataset, shown in Tab.
All speeds are benchmarked using an RTX 3090 GPU.
Compared to NeRF, our model is more than 100 × faster in inference and requires only 33% training time.
PlenOctrees (Yu et al., 2021b) bakes the radiance field into a voxel-based cache, resulting in faster rendering speed but also a significantly larger model size and longer training time.
NPBG (Aliev et al., 2020) achieves faster inference speed with their one-pixel point splats, but at the cost of worse visual quality.
Finally, we are about 25 × faster than Point-NeRF (Xu et al., 2022) in rendering while other metrics are roughly the same.
[Table]
D ISCUSSIONS AND L IMITATIONS
There are a few limitations that need to be addressed in future work: 1) MVS dependency .
Although the proposed point sculpting can partly solve this problem, the performance of the system still depends heavily on the MVS quality.
That said, as MVS systems continue to improve, we do not see this as a fundamental limitation in the long run.
2) View Consistency .
Our system has a 2D U-Net and is thus only approximately 3D consistent.
Especially when viewed in videos, some background areas have flickering effects due to the hallucination of U-Net.
Doing away with a 2D post-processing network is a future direction.
3) Far-away background .
Our current system cannot deal with outdoor scenes with arbitrarily far-away objects ( e.g. sky or clouds).
Using a spherical environment map as in Zhang et al. ; R ¨ uckert et al. (2022) could resolve this problem.
This work was partially supported by the National Science Foundation under Award IIS-1942981.
We thank Jing Wen, Zeyu Ma, and Lahav Lipson for their insightful discussions. We thank Artem Sevastopolsky for generously sharing the NPBG data and clarifying the paper details
A PPENDIX A L IMITATION OF PSNR AND SSIM AS V IEW S YNTHESIS M ETRICS
PSNR, SSIM, and LPIPS are commonly used metrics to measure the similarity between two images.
The LPIPS paper (Zhang et al., 2018) found that LPIPS is significantly more robust than PSNR or SSIM under distortions such as random noise, blurring, and spatial shifts.
Although the evidence is pretty strong for the 2D cases, the robustness of the three metrics is seldom discussed in the context of view synthesis, where 3D geometry plays a key role.
We here provide a detailed case study to show that LPIPS is more robust and consistent with human perception than PSNR and SSIM for the novel view synthesis task, and is more suitable as an evaluation metric.
A. M ISALIGNMENT IN N OVEL V IEW S YNTHESIS
Our key observation is that spatial misalignment is a very common type of error in the view synthesis task.
The misalignment could be caused by inaccurate camera intrinsics and extrinsics (e.g., the
COLMAP camera pose used in LLFF).
Even in the DTU dataset, where the cameras are carefully calibrated with a robotic arm, there is misalignment caused by ambiguity in geometry.
For example, many background objects are only visible in one view, which makes it impossible to synthesize pixels that are identical to the ground truth.
Such errors in the camera poses of existing views will cause small shifts in the rendering of a new target view.
We find that PSNR and SSIM, especially PSNR, can drop significantly in the presence of such pixel shifts, even though they have no detectable impact on the perceived visual quality.
In contrast, LPIPS is relatively more robust.
In Fig.
6, we show by a synthetic experiment that a slight shift could cause a huge drop in PSNR, while LPIPS is robust to such a shift, behaving more similarly to the human perceptual system, which is very insensitive to a such shift.
See the figure caption for details.
(a) 7.17/0.095/ 0.
(b)
[Figure]
Clearly (a) is visually more similar to the reference image, while only LPIPS agrees with the fact.
We also conduct an experiment on the NeRF-Synthetic dataset to demonstrate our findings, where the ground-truth camera poses are perfect.
We apply a tiny noise ∼ N (0 , 0 .
01 I ) in the tangent space to the camera rotation.
As shown in Tab. 6, under such a small perturbation, the PSNR and SSIM of our method and NeRF become even lower than the NPBG baseline, which has significantly worse visual quality, while our LPIPS score is consistently better.
In Fig.
7, we show the visual effect of perturbation.
[Table]
[Figure]
A. Q UALITATIVE E VALUATION
We show qualitative results on the LLFF dataset in Fig.
Our rendering results have better visual quality and contain sharper details compared to NeRF
[Figure]
while having lower PSNR due to the misalignment caused by imperfect camera parameters.
We further apply a 3 × 3 Gaussian blurring to our results
[Figure]
which leads to worse visual quality but higher PSNR.
It indicates that blurred images could have advantages in PSNR under misalignment
[Figure]
whereas LPIPS always prefers our sharper results
[Figure]
B I MPLEMENTATION D ETAILS B. D ATASETS
DTU We select 10 scenes from the DTU test set.
We only use scenes from the test set because the training data of CER-MVS (Ma et al., 2022) includes the DTU training scenes.
The render resolution is 400 × 300 , following PixelNeRF (Yu et al., 2021c).
We reserve 1 in every 7 images for testing, resulting in 42 training views and 7 test views.
LLFF LLFF contains 8 forward-facing scenes.
Following NeRF, we render under the resolution of 1008 × 720 and reserve 1 / 8 of the views for testing.
NeRF-Synthetic
We use the same setting as NeRF (Mildenhall et al., 2020).
The training set and test set for each scene contain 100 and 200 images respectively, and the resolution is 800 × 800.
Tanks& Temples We test on 5 scenes using the split and masks provided by NSVF (Liu et al., 2020).
The render resolution is 1088 × 640 , following Point-NeRF (Xu et al., 2022).
B. MVS R ECONSTRUCTION
We use the CER-MVS (Ma et al., 2022) network to extract depth maps for all scenes.
We scale the scenes so that the median depth is about 600, which is the depth scale that CER-MVS was trained on.
For the DTU dataset, we use the CER-MVS trained on the DTU training set, which has no overlap with our test scenes.
For the NeRF-Synthetic and the Tanks&Temples datasets, we use the
CER-MVS trained on BlendedMVS (Yao et al., 2020).
For LLFF, since the domain gap is large, we take the CER-MVS pre-trained on DTU and finetune the model individually for each scene with the dense depth map provided by COLMAP (Sch ¨ onberger & Frahm, 2016;
Sch ¨ onberger et al., 2016). Specifically, we use COLMAP customized by the authors of NerfingMVS (Wei et al., 2021), which additionally generates a confidence mask based on geometric consistency.
We apply loss only to the areas with a positive mask.
The intuition for finetuning is that we leverage COLMAP at geometric consistency areas, and we rely on the learning-based priors at the areas where COLMAP fails, thus taking advantage of both methods.
To justify the proposed point cloud generation pipeline, we compare our method to directly using the COLMAP point cloud, as shown in Tab. 7 and Fig.
The “w/ COLMAP” baseline uses the fused point cloud from COLMAP, with our differentiable rendering but no point sculpting.
Results show that our method achieves better performance, especially in texture-less areas such as the ceiling, where the COLMAP point cloud is incomplete.
Running the COLMAP MVS takes 0.3 hours.
We finetune for 5k steps on each scene, which takes about 1 hour.
The 1.3 hours additional overhead is insignificant compared to the 6-8 hours training time of our model.
Taking this overhead into account, our method is still more efficient in training than NeRF (about 21 hours on LLFF).
All numbers are reported on a single RTX 3090 GPU.
After the point sculpting step, we downsample the point cloud sizes to 500K for LLFF and NeRFSynthetic, and 1M for DTU and Tanks&Temples, to make the training and inference speed roughly the same for all scenes.
[Figure]
[Table]
B. M ODEL D ETAILS
The feature vector f i attached to each point is 288-dim, and the modulated feature outputted by the SH layer is 32-dim.
Unlike using the tiny 8-dim feature vector as in NPBG (Aliev et al., 2020), we find that increasing the feature dimension would monotonically give better results.
We choose the dimension by balancing the performance and speed/memory cost.
For the point dropout layer, we use a dropout rate p d = 0 .
At inference time we rasterize L =random subsets for all experiments.
We empirically find that L = 2 gives a significant improvement in performance over L = 1 , while the advantage is minimal for averaging more subsets.
See
Appendix E for details.
The U-Net we use has two down-sampling layers and two up-sampling layers with skip connections between the feature maps with the same resolution.
Empirically, we find that limiting the capacity of the U-Net could reduce artifacts and leads to better generalization to novel views.
Therefore, we use a shallower network compared to the 5-layer U-Net used in NPBG (Aliev et al., 2020).
For the
LLFF, NeRF-Synthetic, and Tanks&Temples datasets which have higher resolution, we rasterize at half resolution and add an additional up-sampling layer to output the high-resolution images.
The U-Net is randomly initialized and trained individually for each scene.
We use an L 1 loss between the rendered image and the target.
We also add a total variation loss on the 2D feature map to improve the smoothness of the output:
[Equation]
The final loss can be written as L =
L 1 + λ TV · L TV , where λ TV is set to 0 .
01 in all experiments.
The learning rates for the U-Net, the feature vectors f , the point position p , and the opacity o are set to 10 − 4 , 10 − 2 , 10 − 4 , 10 − 4 , respectively.
The rasterization “softness” hyper-parameter γ is selected to be 10 − 3 .
The opacity is passed through a sigmoid layer to map its range into [0 , 1] .
We use the Adam optimizer (Kingma & Ba, 2014) with the OneCycleLR learning rate scheduler (Smith & Topin, 2019), and train the model with a batch size of 1 for 50,000 steps for all experiments.
No augmentation including random cropping is applied.
We find using a larger batch size and data augmentations are not helpful.
We set the point radius to be 1 .
5 × 10 − 3 , 1 .
× 10 − 3 , 7 . 5 × 10 − 3 for all scenes in DTU, LLFF and NeRF-Synthetic, respectively.
For Tanks&Temples, since there is a large scale variance among scenes, we use different radius for each scene.
Specifically, radius are 1 .
× 10 − 3 for “Ignatius” and “Family”, 1 .
× 10 − 2 for “Truck” and “Caterpillar”, and 5 .
× 10 − 3 for “Barn”.
C T HE P OINT S CULPTING A LGORITHM
For point pruning, we set the depth tolerance threshold to δ d = 0 .
8 in all experiments.
The formal description of the point adding algorithm is presented in Alg.
The definition of Π , Π − 1 , R C and t C are the same as Eqn.
1 in the main body of the paper.
For DTU, we set z near and z far to be 800 and 1400 , respectively.
For LLFF, since those are unbounded scenes, we borrow the idea of using inverse depth as the NDC parameterization in NeRF.
We sample uniformly in the inverse depth space, and the equivalent z near and z far is 800 and + ∞ .
For NeRF-Synthetic, we use z near = 2 .
and z far = 6 . 0 , and for Tanks&Temples we find the depth range for each scene using the bounding provided by the NSVF (Liu et al., 2020) authors.
z step is set so that there are 100 depth bins.
The hyperparameters δ e is set to 5 · Avg ( E ) , as we find that using δ e relative to the average error improves the robustness.
We keep the closest M = points for each ray.
Fig.
11 shows the visualization of the point cloud generated by DCC (Yan et al., 2020), the initial MVS point cloud (“No Adding; No Pruning”), the point cloud filtered by our pruning only (“No Adding”), and our point sculpting method.
We see that most of the outlier points floating in free spaces are pruned by our method (blue points in “Pruning Only”).
We also show how many points get pruned and added in Tab. 8.
Note that the initial numbers of points for each scene are slightly different because we filter out the points based on the scene depth range z near and z far .
More points are added for scan110 , scan114 , and scan118 (corresponding to the last 3 rows in Fig.
11), where the tables in the scene contain large holes in the initial point cloud.
[Table]
D C OMPARISON WITH P ULSAR
Pulsar (Lassner & Zollhofer, 2021) is a highly related work to our method.
Unfortunately, we are unable to do a quantitative comparison with Pulsar, because the code for the view synthesis part of Pulsar is not available.
We show qualitatively that our method is better than Pulsar in Fig.
10, where the Pulsar figure is directly copied from the paper.
Conceptually, while using the same rasterization backbone, our method is better because 1) we use MVS to initialize point positions, whereas they start from random positions; 2) we do point adding to improve the completeness, whereas they don’t; 3) we use SH features and a point dropout layer to boost the performance, whereas they don’t have such designs.
[Figure]
E A NALYSIS OF THE P OINT D ROPOUT L AYER
We do an analysis of the visual quality v.s. the number of subsets used in the point dropout layer.
We do experiments on the DTU dataset and the results are shown in Tab. 9.
Results show that averaging two subsets gives a significant improvement over using only one subset, but using more subsets doesn’t help.
On the other hand, the latency grows almost linearly as we rasterize more subsets.
Therefore, one would prefer using a small number of subsets such as 2.
[Table]
9: Candidates = ∅ 10: for i = {+0 : m do ▷ 11: for u = 0 : H v = 
+}
: W do ▷
12: if E i ( u, v ) ≥ δ e then ▷ propose in the regions with large error 13:
Candidates = S AMPLE C ANDIDATES ( C i , ( u, v ) ) 14: counter =15: for c in Candidates do 16: if E VALUATE C ANDIDATES ( c , C 1: m , D 1: m ) =
= 1 then 17:
P o ← P o ∪ c ▷ add to the point cloud 18: counter = counter
+19: end if 20: if counter ≥ M then 21: break ▷ break once we have M candidates 22: end if 23: end for 24: end if 25: end for 26: end for 27: 28: function S AMPLE C ANDIDATES ( C , ( u, v ) ) 29:
Candidates = ∅ 30: for z = z near : z step : z far do ▷ linearly sample the depth 31:
Candidates ← Candidates ∪ Π − 1 ([ u, v ] T , z, C )
▷ add the 3D point 32: end for 33: return Candidates 34: end function 35: 36: function E VALUATE C ANDIDATES ( c , C 1: m , D 1: m ) 37: no conflict =38: for i = 0 : m do ▷ loop over the views 39: ( u i , v i ) =
Π( c, C i )
▷ the corresponding 2D point 40: [ x i , y i , z i ] T
=
R
C
i c
+ t C
i ▷ c in the i -th camera coordinates 41: if 0 ≤ u i ≤ H − 1 , 0 ≤ v i ≤ W − 1 and z i < D i ( u i , v i ) then 42: no conflict =
▷ conflict if c occludes the existing surface 43: end if 44: end for 45: return no conflict 46: end function
[Figure]
F M ORE V ISUALIZATIONS
Due to the limited space in the main body of the paper, we present more visualizations of our model in this section, to help the readers better compare our methods with the baselines.
Results are presented in Fig.
12, 13, 14, 15, and 16.
See figure captions for detailed comparisons.
NeRF
Ours
Reference NPBG
[Figure] [Figure]
[Figure]
Our method is especially good at capturing surfaces with complex textures and strong nonLambertian effects, such as the glossy ball and the transparent drum surfaces.
For tiny structures such as the microphone and the ficus leaves, our rendering tends to be over-smoothed compared to
Point-NeRF.
[Figure]
G P ER - SCENE BREAKDOWN
We show the per-scene PSNR, SSIM and LPIPS in Tab. 10, 11, 12, 13, and 14.
PSNR ↑
NPBG 16.
SynSin 19.
NeRF 28.
Ours 24.
SSIM ↑
NPBG 0.
SynSin 0.
NeRF 0.
Ours 0.
LPIPS ↓
NPBG 0.
SynSin 0.
[Table] NeRF 0.
Ours 0.
[Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table]
[Table]
SynSin 0.
[Table]
[Table]
[Table]
NeRF 0.
[Table]
[Table]
[Table]
[Table] [Table]
[Table] [Table]