Compressible-composable NeRF via Rank-residual Decomposition
Abstract
Neural Radiance Field (NeRF) has emerged as a compelling method to represent 3D objects and scenes for photo-realistic rendering.
However, its implicit representation causes difficulty in manipulating the models like the explicit mesh representation.
Several recent advances in NeRF manipulation are usually restricted by a shared renderer network, or suffer from large model size.
To circumvent the hurdle, in this paper, we present a neural field representation that enables efficient and convenient manipulation of models.
To achieve this goal, we learn a hybrid tensor rank decomposition of the scene without neural networks.
Motivated by the low-rank approximation property of the SVD algorithm, we propose a rank-residual learning strategy to encourage the preservation of primary information in lower ranks.
The model size can then be dynamically adjusted by rank truncation to control the levels of detail, achieving near-optimal compression without extra optimization.
Furthermore, different models can be arbitrarily transformed and composed into one scene by concatenating along the rank dimension.
The growth of storage cost can also be mitigated by compressing the unimportant objects in the composed scene.
We demonstrate that our method is able to achieve comparable rendering quality to state-of-the-art methods, while enabling extra capability of compression and composition.
Code is available at https://github.com/ashawkey/CCNeRF .
Introduction
Photo-realistic rendering and manipulation of 3D scenes have been long standing problems with numerous real-world applications, such as VR/AR, computer games, and video creation.
Recently, the volumetric Neural Radiance Field (NeRF) representations [26, 1, 8, 27] show impressive progress in rendering photo-realistic images with rich details.
However, due to this implicit representation of geometry and appearance, manipulating the underlying scenes encoded by NeRF still remains a challenging problem.
To solve this problem, some works [21, 45, 19] introduce scene-specific features and scene agnostic rendering network, so that scenes trained with a shared rendering network can be composed together.
However, the constrained and biased capability of these rendering networks causes difficulty in extending to various objects or scenes.
New objects have to be trained with a fixed rendering network to be compatible with the old objects.
Other works [36] discard the rendering network and adopt an no-neural-network NeRF representation, which is more convenient to manipulate the reconstructed scenes and is still able to render high-quality images.
Nevertheless, the large storage requirement for each single model is detrimental to composing complex scenes with lots of objects.
We present a novel approach that allows efficient and convenient manipulation of scenes represented with our model.
Two aspects should be fulfilled to achieve this goal.
The first is that we can dynamically adjust the model size to support different levels of detail (LOD) in different scenarios.
compression
PSNR: 36.
PSNR: 33.
…… 79MB 1.6MB composability
For the compressibility we are motivated by the properties of Singular Value Decomposition (SVD) and High
SVD (HOSVD) [10].
Our is to learn of a
3D from only 2D observations like TensoRF [8] and further preserve the near-optimal low-rank approximation property.
We propose a simple and flexible tensor rank decomposition based neural radiance field and residual strategy.
Each
3D is which can
The are to the CANDECOMP decomposition and a less compact triple plane variant
We introduce a rank- residual learning strategy to encourage the lower ranks to preserve more important information of the whole scene
Combined with an empirical sort
-and-truncate strategy the proposed method achieves near optimal low-rank approximation at any targeted rank.
Different are represented with different low-rank truncations of the model, allowing dynamic trade-off between model size and rendering quality without retraining.
Besides
, our model contains no neural networks and thus naturally supports composability .
Since there are no MLP renderers in our model we can compose different objects by simply concatenating their rank components.
A transformation matrix is recorded for each object to control its position and orientation in the scene.
As demonstrated in Figure 1, we are able to control each model’s LOD and size in a flexible range, and perform arbitrary transformation and composition of different models.
Furthermore, these two properties are connected together through the underlying concept of rank, and can be combined in practical use.
For example, we can mitigate the growth of model size of a complex scene composed of multiple objects, by compressing the less important objects.
Our contributions can be summarized as follows
: (1) We propose a simple radiance field representation based on two
types of tensor rank decomposition, which allows flexible control of model size and naturally supports transformation
and composition
of different models
(2) We design a rank-residual learning strategy to enable
near optimal low rank
approximation.
After training, our
model can be dynamically
adjusted
to trade off
between performance and
model size
without
retraining
The proposed method
reaches comparable rendering quality with state-of-the arts,
while additionally enabling both
compressibility
and composability
Related Work 2.1 Scene Representation with NeRF
3D scenes can be represented with various forms, including volumes, point clouds, meshes, and implicit representations [33, 34, 7, 30, 38, 25, 23, 28].
NeRF [26] proposes to use a 5D function to represent the scene and applies volumetric rendering for novel view synthesis, achieving photorealistic results and detailed geometry reconstruction.
This powerful representation quickly receives attention and is extensively studied and applied in various fields [50, 24], such as generative settings [6, 37, 29, 5], dynamic scenes [20, 31], and texture mapping [44].
In particular, we categorize recent progress by the design of the underlying functions into three classes: neural network-based, hybrid and no-neural-network.
neural network-based representations typically apply an MLP, as the implicit function to encode 3D scenes.
The original NeRF [26] and most following works [1, 2, 51, 43, 46, 35] choose this representation for its simplicity.
However, the training and inference speed of such a network is generally slow due to the relatively expensive MLP computation.
Therefore, hybrid representations try to reduce the size of the MLP, by storing the 3D features in an explicit data structure.
Since a dense 3D representation is unaffordable, different methods are explored.
For example, NSVF [21] adopts sparse voxel grids, PlenOctrees [49] adopts octrees, instant-ngp [27] adopts a multi-scale hashmap, and TensoRF [8] factorizes the scene into lower-rank components.
Querying such hybrid representation is much faster, thus reducing training and inference time and even reaching interactive FPS.
Lastly, no-neural-network representations attempt to model the 3D scene without neural networks.
Plenoxels [36] shows that only the explicit sparse voxels representation is enough to model complex 3D scenes.
Our method also belongs to this representation, sharing the similar tensor rank decomposition idea to TensoRF [8], but we focus on two additional capabilities, i.e. , compressibility and composability, which are important yet usually absent in previous work.
2.2 Tensor Decomposition and Low-rank Approximation
Decomposition of high-order tensors [18] can be considered as the generalizations of matrix singular value decomposition.
The Tucker decomposition [40] decomposes a tensor into a core tensor multiplied by a matrix along each mode.
The CANDECOMP/PARAFAC (CP) decomposition [14, 4] factorizes a tensor into a sum of component rank-one tensors, and can be viewed as a special case of
Tucker where the core tensor is superdiagonal.
The high-order singular value decomposition [10] provides a method to compute a specific Tucker decomposition with an all-orthogonal core tensor.
Low-rank approximation is a common problem that applies tensor decomposition, and has found various applications such as image compression.
Although the truncated HOSVD does not hold the optimal property contrary to the truncated SVD, it still results in a quasi-optimal solution [10, 41, 12], which is enough to yield a sufficiently good solution in practical uses.
Tensor rank decomposition and its variants [10, 11] has been used in various vision and learning tasks [47, 8, 48].
Specifically,
TensoRF [8] first leverages the CP decomposition and a Vertex-Matrix (VM) decomposition to factorize neural radiance fields, but its other designs ( e.g. , use of MLP) disturbs the property of tensor rank decomposition and prevents it from achieving compression or composition.
Instead, we focus on modeling neural radiance fields only with tensor rank decomposition, and aim to preserve the low-rank approximation property, enabling the compression of a learned neural radiance field similar to the SVD compression of an image.
2.3 Manipulation and Composition of NeRF
Manipulation and Composition are important for a 3D representation’s practical usage.
Explicit 3D representations, e.g. , meshes, are natively editable and composable.
However, neural network-based implicit representations like a vanilla NeRF is difficult to perform such operations.
NSVF [21] can composite separate objects together, but these objects have to be trained together using a shared MLP, which limits its flexibility and potential usage.
Later works [45, 29, 13, 19] learn object-compositional
NeRF, but are usually scene-specific and do not allow cross-scene composition without retraining.
Geometry and appearance editing [22, 42] of neural fields also requires an extra optimization step to modify the neural network-based representation.
With the explicit sparse voxel representation,
Plenoxels [36] naturally supports direct composition of different objects, but suffers from the large storage on the dense index matrix.
Our method also supports arbitrary affine transformations and compositions without extra optimization.
Further, we can efficiently mitigate the model size growth due to the compact tensor rank decomposition and the compressibility.
⋅ (
[Equation]
𝑆𝑆 𝜎𝜎 𝑟𝑟 𝑔𝑔 𝑏𝑏
[Figure]
Methodology 3.1 Preliminaries on Neural Radiance Fields
Neural Radiance Fields (NeRF) [26] represents a 3D volumetric scene with a 5D function f Θ that maps a 3D coordinate x = ( x, y, z ) and a 2D viewing direction d
= ( θ, ϕ ) into a volume density σ and an emitted color c = ( r, g, b ) .
Given a ray r originating at o with direction d , we query f Θ at points x i
= o
+ t i d sequentially sampled along the ray to get densities { σ i } and colors { c i } .
The color of the pixel corresponding to the ray is then estimated by numerical quadrature:
[Equation]
where δ i is the step size, α i is the opacity, and T i is the transmittance.
Since this volume rendering process is differentiable, NeRF can be optimized only from 2D image supervision by minimizing the
L2 difference between each pixel’s predicted color ˆC ( r ) and the ground truth color C ( r ) from the image:
[Equation]
3.2 Preliminaries on Tensor Decomposition
For a 3D tensor T ∈ R H
× W × D , each element T i,j,k ∈ R can be represented via the Tucker decomposition [40] by:
[Equation]
where S ∈ R P × Q × R is the core tensor, P, Q, R are the number of components along each axis, and U x ∈ R H × P , U y ∈ R W × Q , U z ∈ R D
× R are the factor matrices.
The CP decomposition can be viewed as a special case of Tucker when P = Q = R and S is superdiagonal [18] ( i.e. ,
S i,j,k ̸ =
⇐⇒ i
=
j = k ):
[Equation]
where s = diag ( S ) ∈ R R is the reduced core tensor (or rank weights).
Although s is usually absorbed into the factor matrices, we write it out for the convenience of later discussion.
3.3 Decompose NeRF without MLP
Hybrid Feature Volume Decomposition.
We are interested in multi-feature volumetric encoded as a 4D tensor T ∈ R C × H × W × D , where C is the feature dimension ( e.g. , density, RGB values, or other features), and ( H, W, D ) is the spatial resolution (usually C ≪ min( H, W, D ) ).
A straightforward way is to perform C independent decompositions for each channel.
However, different feature channels such as the RGB values are highly correlated in real 3D scenes.
A more compact way is to share the factorized matrices like TensoRF [8], and only use different rank weights s for different channels.
Therefore, we propose to represent T through the CP decomposition by:
[Equation]
where S ∈ R C × R is the matrix of rank weights for C channels, and ∗ denotes the Hadamard product.
Since the above decomposition relies on rank-one vectors (1D tensors), it may require very high ranks to represent complex 3D scenes, which leads to expensive computation at each location.
A less compact but more computation-friendly alternative is to adopt matrices (2D tensors) to factorize the 3D scene.
This variant in the form of CP decomposition is given by:
[Equation]
(6) where U x,y ∈
R H
× W × R , U y,z ∈ R W × D × R , U x,z ∈ R H
× D × R are the factorized matrices along three planes, each containing R components.
This variant can be comprehended by first slicing and tiling the original 3D space along each axis, and then learning a CP decomposition on R HW × WD × HD .
Although this representation is less compact and takes more storage, recent works [8, 5] have shown that it is able to represent scenes with smaller R and better quality.
We denote this variant as the
Triple Plane (TP) decomposition.
Further, we notice that for each individual rank, the underlying vector- or matrix-based decomposition can be selected independently.
Therefore, a hybrid variant (HY) that combines the above CP and TP decomposition is proposed.
We can flexibly adjust the ratio of two decompositions by R = R vec + R mat to trade off between model size and performance.
The model structure is illustrated in Figure 2.
Learning the Decomposition via Differentiable Rendering.
For simplicity, we take the CP decomposition as an example.
Our model only consists of four tensors to optimize, i.e. , S , U x , U y , U z .
To represent scenes through neural radiance fields, we need to learn the volume density σ and color c at each location.
As the volume density is only dependent on the 3D coordinate x , we can represent it with one feature channel T density i,j,k ∈ R .
However, the color is dependent on both the 3D location x and the 2D viewing direction d , which is a 5D function in total.
To represent it within the 3D feature volumes, we adopt the spherical harmonics (SH) functions to approximate the additional 2D viewing directions dependency [49, 36].
In particular, for spherical harmonics of maximum degree ℓ max , it takes ( ℓ max + 1) 2 SH coefficients to model the view-dependent color per channel.
We use T κi,j,k ∈ R ( ℓ max +1) 2 , κ ∈ { r, g, b } to represent these coefficients.
The density and color at each location (coordinate indices are omitted for simplicity) can then be represented with:
[Equation]
where ϕ ( · ) is the density activation, ψ ( · ) is the color activation, d is the viewing direction, and Y mℓ are the SH functions.
Our model can then be optimized through the standard RGB loss in Equation 2.
3.4 Rank-residual Learning for Compressibility
The idea of low-rank approximation is to only keep the most important rank components, where the importance of each rank component can be represented by the singular values in the SVD algorithm.
Similarly, we can define the importance of each rank component in our decomposition by the rank weights S averaged on all feature channels, and multiplied with the magnitude of three factor matrices along the rank dimension.
However, the CP decomposition doesn’t hold this low-rank approximation property [18].
If we directly sort the rank components by the rank importance and truncate the model for compression, the rendering quality drops sharply compared to the optimal model ( i.e. , model retrained with the same parameters), as illustrated by the baseline method in Figure 3.
We propose a rank-residual learning strategy to close this performance gap and achieve near-optimal compression results.
This strategy aims to simulate SVD’s low-rank approximation property, where
[Figure] [Figure]
the lower rank components contain more information and contribute more to the approximation.
Since the rank components in our method can be flexibly adjusted, a direct solution is to train in a progressive way.
Suppose the total number of ranks is R , and the number of training stages is M .
The total R rank components can be sequentially divided into M non-empty groups, and we denote the accumulated number of ranks for each group by R m where m ∈ { 1 , 2 , · · · , M } .
We start from training the first R 1 rank components, and after its convergence, we fix them and append the next
R 2 − R 1 rank components to train in a new stage.
In the last stage, all R ranks are involved.
However, this is inefficient since we need to make sure each stage is fully converged before increasing the number of ranks.
A more efficient way is to train all stages in parallel.
In particular, we simultaneously supervise the sequentially accumulated outputs from all M groups.
This can be viewed as supervising the outputs from M truncations of the decomposition with a rank-residual loss:
[Equation]
where ˆC m ( r ) is RGB values calculated from the truncated decomposition S m , U xm , U ym , U zm which only keeps the first R m rank components (taking the CP decomposition as an example).
During training, each group is able to learn the residual error from previous groups, and eventually leads to the desired low-rank approximation property.
Ideally, choosing M = R groups assures this low-rank approximation property to hold at any targeted rank, but usually computationally unaffordable with a large R .
In practice we choose M ≪ R , so the property only holds at those dividing ranks { R m } .
For any other rank R ′ , assuming R m < R ′ <
R m +1 , we first keep the fully covered R m rank components, and then use the empirical sort-and-truncate strategy to select the remaining R
′ − R m ranks in the last group { R m +1 , R m +2 · · · , R m +1 } .
By selecting the top rank components with the largest average importance, it is enough to keep the near-optimal low-rank approximation property at any targeted ranks, as shown by the proposed method in Figure 3.
The decomposition model trained with our rank-residual learning allows dynamic adjustment of model size and rendering quality with no extra optimization.
In practice, we usually need different
LODs to adapt to different cases, such as the texture mipmaps.
While the other NeRF representations need to retrain for different LODs and store every model separately, we just train once and get a unified model for all LODs.
Given a targeted storage upper bound or performance lower bound, we only need to select the targeted rank and truncate the decomposition with the simple slicing operation.
3.5 Composability without Constraints
As illustrated in Figure 5, any 3D object or scene represented by our model can be composed without the constraints in previous work [21, 45].
Since our model describes each 3D scene with a set of rank components, composability is naturally accomplished by concatenating along the rank dimension and summing up the number of ranks R = (cid:80)
Nn =1 R n , where n denotes the object index.
In practice, this is implemented by appending new rank components to a parameter list, so that models with different
[Figure]
resolution and decomposition forms can be composed together.
For each object, we record its number of ranks so we can still distinguish it from the whole scene.
Arbitrary affine transformations of individual objects are supported by recording a transformation matrix T n ∈ R 4 × 4 for each object, which includes the translation t n ∈ R 3 , rotation R n ∈ SO(3) and scale s n ∈ R 3 .
We warp the ray into each object’s coordinate system before querying the density { σ i,n } and color { c i,n } at the sampled points:
[Equation]
Here we assume the coordinate x i is homogeneous, and the viewing direction d is represented by a unit vector.
To correctly handle the occlusion between different objects, we still sample one ray per pixel, and perform the composition at each sample point by:
[Equation]
We sum up the density from all objects to calculate opacity, and weight the color by the density after the softmax function φ N .
The rendering formula in Equation 1 can then be applied to calculate the pixel color.
Although the model size and rendering time grows linearly with the total number of ranks (complexity of the scene), we show in experiments that the compression property can be applied to mitigate the growth and improve efficiency.
Experiments 4.1 Implementation Details
The model is implemented with the PyTorch framework [32].
The degree for SH coefficients is 3, which equals 48 channels for the color feature.
We use the Adam optimizer [16] with an initial learning rate of 0.02 for the factorized matrices, and 0.001 for the singular values.
All the experiments are performed on one NVIDIA V100 GPU.
The resolution of the feature grid is determined by the total number of voxels N and the bounding box, where N is increased from 128 3 to 300 3 for HY models and 500 3 for CP models in early training steps.
To accelerate rendering, we adopt the binary occupancy mask pruning technique as in [8, 27], and use separate rank components for density and color to avoid unnecessary querying in empty space.
This occupancy mask is also used to shrink the initial bounding box for more precise modeling.
We mainly carry out experiments on the NeRF-synthetic dataset [26] (CC BY 3.0 license) and the Tanks and Temples dataset [17] (CC
BY-NC-SA 3.0 license).
Please check the supplementary materials for more details.
4.2 Compression Results
Firstly, We evaluate the compressibility of our model.
Since the color components cost most of the total storage, we mainly focus on compressing the color components, and keep the density components fixed.
Given the number of ranks R , we first train two models with the original loss
B a s e li n e
P r opo s e d compress
B a s e li n e
P r opo s e d
The baseline method deteriorates significantly, while our proposed method remains high rendering quality.
We show composition between different models with our method.
Our method can successfully handle occlusion and remain high-quality rendering.
[Table]
L NeRF in Equation 2 and our rank-residual loss L residual in Equation 8.
We denote them as M R and
M ours R , respectively.
At any targeted rank r ≤ R to compress, we design three strategies to verify whether the proposed compression is near-optimal: (1) Retrain a model M r with L NeRF at the given rank.
This requires a retraining from scratch, and can be viewed as the optimal compression result at the given rank.
(2) Sort and truncate the rank of the original model M R to M base r , which can be viewed as the baseline compression result.
(3) Sort and truncate the rank of the proposed model
M ours R to M ours r .
We denote these three settings as ‘Optimal’, ‘Baseline’, and ‘Proposed’ respectively.
The quantitative results are listed in Table 1.
We find that the performance of the baseline method degrades significantly compared to the optimal method, whereas our proposed method is comparable to the optimal method at all targeted ranks.
The visualization in Figure 6 demonstrates how the baseline model gradually deteriorates compared to the proposed model.
Note that at the the right-most column where the baseline model is not compressed and the same as the optimal model, the rendering quality of the proposed model is hard to discern from the optimal model.
In Figure 3, we show that even though our model is only supervised at 5 discrete ranks, it can remain good compression quality at any other ranks.
Figure 3 provides an explanation for the compressibility of our model.
With the rank-residual learning, the rank importance is more concentrated to the lower ranks, which benefits the low-rank approximation.
[Table]
4.3 Composition Results
In Figure
[Table]
we demonstrate the composability of our method.
Without extra optimization, we are able to perform affine transformation and composition of different models like composing meshes in a 3D editor.
Note that our model can correctly handle the occlusion and collision between different objects.
In Figure 5, we combine the compression capability of our model in scene composition.
In a complex scene composed of lots of objects, we can compress the less important objects to make a trade-off between the model size and rendering quality.
4.4 Comparisons and Discussion
Rendering Quality.
We compare our method with some recent works in Table 2.
We focus on the extra capabilities to facilitate practical applications, but not boosting the rendering quality over the previous state-of-the-arts, since the vanilla NeRF [26] already reaches photo-realistic rendering in most cases.
Although the performance of the proposed method is not the best, the simple model design with the extra compressibility and composability is unique and enables various applications.
MLP Renderers.
As discussed in [8, 27], the absence of a small MLP renderer network generally leads to worse performance, especially for the specular details.
However, these renderers are trained separately and cannot be shared across scenes unless explicitly restricted, which causes inconvenience or limits the potency of composition.
We therefore discard the MLP at the cost of a slightly worse rendering quality, but facilitates the composition and compression.
Limitations.
Although our method can correctly compose the geometry of multiple objects, we don’t consider the lighting conditions.
Our model bakes lighting conditions into the color like the vanilla
NeRF, and cannot perform re-lighting to achieve consistent lighting effect after composition.
A future direction is to integrate the reflectance models [3, 52].
Besides, we only support modeling bounded objects for now.
A background model [51] can be combined to simulate unbounded scenes.
Conclusion
In this work, we present a novel compressible and composable neural radiance field representation.
Our model is designed to be simple and flexible, yet still being effective enough to render photorealistic images.
A rank-residual learning strategy is proposed to enable near-optimal low-rank approximation, which allows dynamic adjustment of the model size to support different LODs in different scenarios.
All models represented with our method can be arbitrarily transformed and composed together like in a 3D editor.
Powered by these properties, we are able to efficiently and conveniently manipulate complex scenes with multiple objects.
We believe our method will further facilitate the NeRF-based scene representation in real-world applications.
Acknowledgements.
This work is supported by the National Key Research and Development
Program of China (2020YFB1708002), National Natural Science Foundation of China (61632003, 61375022, 61403005), Grant SCITLAB-20017 of Intelligent Terminal Key Laboratory of SiChuan
Province, Beijing Advanced Innovation Center for Intelligent Robots and Systems (2018IRS11) and
PEK-SenseTime Joint Laboratory of Machine Vision.