Compressible-composable NeRF via Rank-residual Decomposition
Anonymous Author(s)
AffiliationAddress email
Abstract
Neural Radiance Field (NeRF) has emerged as a compelling method to represent 3D objects and scenes for photo-realistic rendering.
However, its implicitrepresentation causes difficulty in manipulating the models like the explicit meshrepresentation.
Several recent advances in NeRF manipulation are usually restrictedby a shared renderer network, or suffer from large model size.
To circumvent thehurdle, in this paper, we present a neural field representation that enables efficientand convenient manipulation of models.
To achieve this goal, we learn a hybridtensor rank decomposition of the scene without neural networks.
Motivated bythe low-rank approximation property of the SVD algorithm, we propose a rankresidual learning strategy to encourage the preservation of primary information inlower ranks.
The model size can then be dynamically adjusted by rank truncationto control the levels of detail, achieving near-optimal compression without extraoptimization.
Furthermore, different models can be arbitrarily transformed andcomposed into one scene by concatenating along the rank dimension.
The growthof storage cost can also be mitigated by compressing the unimportant objects in thecomposed scene.
We demonstrate that our method is able to achieve comparablerendering quality to state-of-the-art methods, while enabling extra capability ofcompression and composition.
Code will be made publicly available.
Introduction
Photo-realistic rendering and manipulation of 3D scenes have been long standing problems withnumerous real-world applications, such as VR/AR, computer games, and video creation.
Recently,the volumetric Neural Radiance Field (NeRF) representations [26, 1, 8, 27] show impressive progressin rendering photo-realistic images with rich details.
However, due to this implicit representationof geometry and appearance, manipulating the underlying scenes encoded by NeRF still remains achallenging problem.
To solve this problem, some works [21, 45, 19] introduce scene-specific featuresand scene agnostic rendering network, so that scenes trained with a shared rendering network canbe composed together.
However, the constrained and biased capability of these rendering networkscauses difficulty in extending to various objects or scenes.
New objects have to be trained witha fixed rendering network to be compatible with the old objects.
Other works [36] discard therendering network and adopt an no-neural-network NeRF representation, which is more convenientto manipulate the reconstructed scenes and is still able to render high-quality images.
Nevertheless,the large storage requirement for each single model is detrimental to composing complex scenes withlots of objects.
We present a novel approach that allows efficient and convenient manipulation of scenes representedwith our model.
Two aspects should be fulfilled to achieve this goal.
The first is that we can dynamically adjust the model size to support different levels of detail (LOD) in different scenarios.
compression
PSNR: 36.
PSNR: 33.
…… 79MB 1.6MB
Combined with an empirical sort-and-truncate strategy the proposed method achieves near-optimal low-rank approximation at any targeted rank.
Different LODs are represented with different low-rank truncations of the model, allowing dynamic trade-off between model size and rendering qualitywithout retraining.
Besides, our model contains no neural networks and thus naturally supportscomposability .
Since there are no MLP renderers in our model, we can compose different objects bysimply concatenating their rank components.
A transformation matrix is recorded for each object tocontrol its position and orientation in the scene.
As demonstrated in Figure 1, we are able to control each model’s LOD and size in a flexible range,and perform arbitrary transformation and composition of different models.
Furthermore, these twoproperties are connected together through the underlying concept of rank, and can be combined inpractical use.
For example, we can mitigate the growth of model size of a complex scene composedof multiple objects, by compressing the less important objects.
Our contributions can be summarizedas follows: (1) We propose a simple radiance field representation based on two types of tensor rankdecomposition, which allows flexible control of model size and naturally supports transformationand composition of different models.
(2) We design a rank-residual learning strategy to enablenear-optimal low-rank approximation.
After training, our model can be dynamically adjusted totrade off between performance and model size without retraining.
The proposed method reachescomparable rendering quality with state-of-the-arts, while additionally enabling both compressibilityand composability.
Related Work2.1 Scene Representation with NeRF
3D scenes can be represented with various forms, including volumes, point clouds, meshes, andimplicit representations [33, 34, 7, 30, 38, 25, 23, 28].
NeRF [26] proposes to use a 5D function to represent the scene and applies volumetric rendering for novel view synthesis, achieving photorealistic results and detailed geometry reconstruction.
This powerful representation quickly receivesattention and is extensively studied and applied in various fields [50, 24], such as generative settings [6,37, 29, 5], dynamic scenes [20, 31], and texture mapping [44].
In particular, we categorize recentprogress by the design of the underlying functions into three classes: neural network-based, hybridand no-neural-network.
neural network-based representations typically apply an MLP, as the implicitfunction to encode 3D scenes.
The original NeRF [26] and most following works [1, 2, 51, 43, 46, 35]choose this representation for its simplicity.
However, the training and inference speed of such anetwork is generally slow due to the relatively expensive MLP computation.
Therefore, hybridrepresentations try to reduce the size of the MLP, by storing the 3D features in an explicit datastructure.
Since a dense 3D representation is unaffordable, different methods are explored.
Forexample, NSVF [21] adopts sparse voxel grids, PlenOctrees [49] adopts octrees, instant-ngp [27]adopts a multi-scale hashmap, and TensoRF [8] factorizes the scene into lower-rank components.
Querying such hybrid representation is much faster, thus reducing training and inference time and evenreaching interactive FPS.
Lastly, no-neural-network representations attempt to model the 3D scenewithout neural networks.
Plenoxels [36] shows that only the explicit sparse voxels representation isenough to model complex 3D scenes.
Our method also belongs to this representation, sharing thesimilar tensor rank decomposition idea to TensoRF [8], but we focus on two additional capabilities,i.e. , compressibility and composability, which are important yet usually absent in previous work.
2.2 Tensor Decomposition and Low-rank Approximation
Decomposition of high-order tensors [18] can be considered as the generalizations of matrix singularvalue decomposition.
The Tucker decomposition [40] decomposes a tensor into a core tensormultiplied by a matrix along each mode.
The CANDECOMP/PARAFAC (CP) decomposition [14, 4]factorizes a tensor into a sum of component rank-one tensors, and can be viewed as a special case of
Tucker where the core tensor is superdiagonal.
The high-order singular value decomposition [10]provides a method to compute a specific Tucker decomposition with an all-orthogonal core tensor.
Low-rank approximation is a common problem that applies tensor decomposition, and has foundvarious applications such as image compression.
Although the truncated HOSVD does not hold theoptimal property contrary to the truncated SVD, it still results in a quasi-optimal solution [10, 41, 12],which is enough to yield a sufficiently good solution in practical uses.
Tensor rank decompositionand its variants [10, 11] has been used in various vision and learning tasks [47, 8, 48].
Specifically,
TensoRF [8] first leverages the CP decomposition and a Vertex-Matrix (VM) decomposition tofactorize neural radiance fields, but its other designs ( e.g. , use of MLP) disturbs the property of tensorrank decomposition and prevents it from achieving compression or composition.
Instead, we focuson modeling neural radiance fields only with tensor rank decomposition, and aim to preserve thelow-rank approximation property, enabling the compression of a learned neural radiance field similarto the SVD compression of an image.
2.3 Manipulation and Composition of NeRF
Manipulation and Composition are important for a 3D representation’s practical usage.
Explicit 3Drepresentations, e.g. , meshes, are natively editable and composable.
However, neural network-basedimplicit representations like a vanilla NeRF is difficult to perform such operations.
NSVF [21] cancomposite separate objects together, but these objects have to be trained together using a shared MLP,which limits its flexibility and potential usage.
Later works [45, 29, 13, 19] learn object-compositional NeRF, but are usually scene-specific and do not allow cross-scene composition without retraining.
Geometry and appearance editing [22, 42] of neural fields also requires an extra optimization stepto modify the neural network-based representation.
With the explicit sparse voxel representation, Plenoxels [36] naturally supports direct composition of different objects, but suffers from the largestorage on the dense index matrix.
Our method also supports arbitrary affine transformations andcompositions without extra optimization.
Further, we can efficiently mitigate the model size growthdue to the compact tensor rank decomposition and the compressibility.
⋅ ( [Equation]
[Figure]
Methodology
3.1 Preliminaries on Neural Radiance Fields
Neural Radiance Fields (NeRF) [26] represents a 3D volumetric scene with a 5D function f Θ thatmaps a 3D coordinate x = ( x, y, z ) and a 2D viewing direction d
= ( θ, ϕ ) into a volume density σand an emitted color c = ( r, g, b ) .
Given a ray r originating at o with direction d , we query f Θ atpoints x i
= o
+ t i d sequentially sampled along the ray to get densities { σ i } and colors { c i } .
Thecolor of the pixel corresponding to the ray is then estimated by numerical quadrature:
[Equation]
where δ i is the step size, α i is the opacity, and T i is the transmittance.
Since this volume renderingprocess is differentiable, NeRF can be optimized only from 2D image supervision by minimizing the
L2 difference between each pixel’s predicted color ˆC ( r ) and the ground truth color C ( r ) from theimage:
L NeRF = (cid:88)
[Equation]
3.2 Preliminaries on Tensor Decomposition
For a 3D tensor T ∈ R H
× W × D , each element T i,j,k ∈ R can be represented via the Tuckerdecomposition [40] by:
[Equation]
where S ∈ R P × Q × R is the core tensor, P, Q, R are the number of components along each axis,and U x ∈ R H × P , U y ∈ R W × Q , U z ∈ R D
× R are the factor matrices.
The CP decompositioncan be viewed as a special case of Tucker when P = Q = R and S is superdiagonal [18] ( i.e. ,
S i,j,k ̸ =
⇐⇒ i
=
j = k ):
[Equation]
yj
[Equation]
where s = diag ( S ) ∈ R R is the reduced core tensor (or rank weights).
Although s is usually absorbedinto the factor matrices, we write it out for the convenience of later discussion.
3.3 Decompose NeRF without MLP
Hybrid Feature Volume Decomposition.
We are interested in multi-feature volumetric encoded as a4D tensor T ∈ R C × H × W × D , where C is the feature dimension ( e.g. , density, RGB values, or otherfeatures), and ( H, W, D ) is the spatial resolution (usually C ≪ min( H, W, D ) ).
A straightforwardway is to perform C independent decompositions for each channel.
However, we notice that in real3D scenes, different feature channels such as the RGB values are highly correlated.
A more compactway is to share the factorized matrices, and only use different rank weights s for different channels.
Therefore, we propose to represent T through the CP decomposition by:
[Equation]
where S ∈ R C × R is the matrix of rank weights for C channels, and ∗ denotes the Hadamard product.
Since the above decomposition relies on rank-one vectors (1D tensors), it may require very highranks to represent complex 3D scenes, which leads to expensive computation at each location.
A lesscompact but more computation-friendly alternative is to adopt matrices (2D tensors) to factorize the3D scene.
This variant in the form of CP decomposition is given by:
[Equation]
) where U x,y ∈ R H
× W × R , U y,z ∈ R W × D × R , U x,z ∈ R H
× D × R are the factorized matrices alongthree planes, each containing R components.
This variant can be comprehended by first slicing andtiling the original 3D space along each axis, and then learning a CP decomposition on R HW × WD × HD .
Although this representation is less compact and takes more storage, recent works [8, 5] have shownthat it is able to represent scenes with smaller R and better quality.
We denote this variant as the
Triple Plane (TP) decomposition.
Further, we notice that for each individual rank, the underlyingvector- or matrix-based decomposition can be selected independently.
Therefore, a hybrid variant(HY) that combines the above CP and TP decomposition is proposed.
We can flexibly adjust the ratioof two decompositions by R = R vec + R mat to trade off between model size and performance.
Themodel structure is illustrated in Figure 2.
Learning the Decomposition via Differentiable Rendering.
For simplicity, we take the CP decomposition as an example.
Our model only consists of four tensors to optimize, i.e. , S , U x , U y , U z .
Torepresent scenes through neural radiance fields, we need to learn the volume density σ and color c ateach location.
As the volume density is only dependent on the 3D coordinate x , we can representit with one feature channel T density i,j,k ∈ R .
However, the color is dependent on both the 3D locationx and the 2D viewing direction d , which is a 5D function in total.
To represent it within the 3Dfeature volumes, we adopt the spherical harmonics (SH) functions to approximate the additional2D viewing directions dependency [49, 36].
In particular, for spherical harmonics of maximumdegree ℓ max , it takes ( ℓ max + 1) 2 SH coefficients to model the view-dependent color per channel.
Weuse T κi,j,k ∈ R ( ℓ max +1) 2 , κ ∈ { r, g, b } to represent these coefficients.
The density and color at eachlocation (coordinate indices are omitted for simplicity) can then be represented with:
[Equation]
where ϕ ( · ) is the density activation, ψ ( · ) is the color activation, d is the viewing direction, and Y mℓare the SH functions.
Our model can then be optimized through the standard RGB loss in Equation 2.
3.4 Rank-residual Learning for Compressibility
The idea of low-rank approximation is to only keep the most important rank components, where theimportance of each rank component can be represented by the singular values in the SVD algorithm.
Similarly, we can define the importance of each rank component in our decomposition by the rankweights S averaged on all feature channels, and multiplied with the magnitude of three factor matricesalong the rank dimension.
However, the CP decomposition doesn’t hold this low-rank approximationproperty [18].
If we directly sort the rank components by the rank importance and truncate the modelfor compression, the rendering quality drops sharply compared to the optimal model ( i.e. , modelretrained with the same parameters), as illustrated by the baseline method in Figure 3.
We propose a rank-residual learning strategy to close this performance gap and achieve near-optimalcompression results.
This strategy aims to simulate SVD’s low-rank approximation property, where
[Figure]
[Figure]
the lower rank components contain more information and contribute more to the approximation.
Since the rank components in our method can be flexibly adjusted, a direct solution is to train in aprogressive way.
Suppose the total number of ranks is R , and the number of training stages is M .
The total R rank components can be sequentially divided into M non-empty groups, and we denotethe accumulated number of ranks for each group by R m where m ∈ { 1 , 2 , · · · , M } .
We start fromtraining the first R 1 rank components, and after its convergence, we fix them and append the next
R 2 − R 1 rank components to train in a new stage.
In the last stage, all R ranks are involved.
However,this is inefficient since we need to make sure each stage is fully converged before increasing thenumber of ranks.
A more efficient way is to train all stages in parallel.
In particular, we simultaneouslysupervise the sequentially accumulated outputs from all M groups.
This can be viewed as supervisingthe outputs from M truncations of the decomposition with a rank-residual loss:
[Equation]
where ˆC m ( r ) is RGB values calculated from the truncated decomposition S m , U xm , U ym , U zm whichonly keeps the first R m rank components (taking the CP decomposition as an example).
Duringtraining, each group is able to learn the residual error from previous groups, and eventually leads tothe desired low-rank approximation property.
Ideally, choosing M = R groups assures this low-rankapproximation property to hold at any targeted rank, but usually computationally unaffordable with alarge R .
In practice we choose M ≪ R , so the property only holds at those dividing ranks { R m } .
For any other rank R ′ , assuming R m < R ′ <
R m +1 , we first keep the fully covered R m rankcomponents, and then use the empirical sort-and-truncate strategy to select the remaining R
′ − R mranks in the last group { R m +1 , R m +2 · · · , R m +1 } .
By selecting the top rank components with thelargest average importance, it is enough to keep the near-optimal low-rank approximation property atany targeted ranks, as shown by the proposed method in Figure 3.
The decomposition model trained with our rank-residual learning allows dynamic adjustment ofmodel size and rendering quality with no extra optimization.
In practice, we usually need different
LODs to adapt to different cases, such as the texture mipmaps.
While the other NeRF representationsneed to retrain for different LODs and store every model separately, we just train once and get aunified model for all LODs.
Given a targeted storage upper bound or performance lower bound, weonly need to select the targeted rank and truncate the decomposition with the simple slicing operation.
3.5 Composability without Constraints
As illustrated in Figure 5, any 3D object or scene represented by our model can be composed withoutthe constraints in previous work [21, 45].
Since our model describes each 3D scene with a set of rankcomponents, composability is naturally accomplished by concatenating along the rank dimension andsumming up the number of ranks R = (cid:80)
Nn =1 R n , where n denotes the object index.
In practice, thisis implemented by appending new rank components to a parameter list, so that models with different
[Figure]
resolution and decomposition forms can be composed together.
For each object, we record its numberof ranks so we can still distinguish it from the whole scene.
Arbitrary affine transformations ofindividual objects are supported by recording a transformation matrix T n ∈ R 4 × 4 for each object,which includes the translation t n ∈ R 3 , rotation R n ∈ SO(3) and scale s n ∈ R 3 .
We warp theray into each object’s coordinate system before querying the density { σ i,n } and color { c i,n } at thesampled points:
[Equation]
Here we assume the coordinate x i is homogeneous, and the viewing direction d is represented by aunit vector.
To correctly handle the occlusion between different objects, we still sample one ray perpixel, and perform the composition at each sample point by:
[Equation]
We sum up the density from all objects to calculate opacity, and weight the color by the density afterthe softmax function φ N .
The rendering formula in Equation 1 can then be applied to calculate thepixel color.
Although the model size and rendering time grows linearly with the total number of ranks(complexity of the scene), we show in experiments that the compression property can be applied tomitigate the growth and improve efficiency.
Experiments
4.1 Implementation Details
The model is implemented with the PyTorch framework [32].
The degree for SH coefficients is4, which equals 48 channels for the color feature.
We use the Adam optimizer [16] with an initiallearning rate of 0.02 for the factorized matrices, and 0.001 for the singular values.
All the experimentsare performed on one NVIDIA V100 GPU.
The resolution of the feature grid is determined by thetotal number of voxels N and the bounding box, where N is increased from 128 3 to 300 3 for HYmodels and 500 3 for CP models in early training steps.
To accelerate rendering, we adopt the binaryoccupancy mask pruning technique as in [8, 27], and use separate rank components for densityand color to avoid unnecessary querying in empty space.
This occupancy mask is also used toshrink the initial bounding box for more precise modeling.
We mainly carry out experiments onthe NeRF-synthetic dataset [26] (CC BY 3.0 license) and the Tanks and Temples dataset [17] (CC
BY-NC-SA 3.0 license).
Please check the supplementary materials for more details.
4.2 Compression Results
Firstly, We evaluate the compressibility of our model.
Since the color components cost most ofthe total storage, we mainly focus on compressing the color components, and keep the densitycomponents fixed.
Given the number of ranks R , we first train two models with the original loss
B a s e li n e
P r opo s e d compress
B a s e li n e
P r opo s e d
The baseline method deteriorates significantly, while our proposed method remains high rendering quality.
We show composition between different models with our method.
Our method can successfully handle occlusion and remain high-quality rendering.
[Table]
L NeRF in Equation 2 and our rank-residual loss L residual in Equation 8.
We denote them as M R and
M ours R , respectively.
At any targeted rank r ≤ R to compress, we design three strategies to verifywhether the proposed compression is near-optimal: (1) Retrain a model M r with L NeRF at the givenrank.
This requires a retraining from scratch, and can be viewed as the optimal compression resultat the given rank.
(2) Sort and truncate the rank of the original model M R to M base r , which can beviewed as the baseline compression result.
(3) Sort and truncate the rank of the proposed model
M ours R to M ours r .
We denote these three settings as ‘Optimal’, ‘Baseline’, and ‘Proposed’ respectively.
The quantitative results are listed in Table 1.
We find that the performance of the baseline methoddegrades significantly compared to the optimal method, whereas our proposed method is comparableto the optimal method at all targeted ranks.
The visualization in Figure 6 demonstrates how thebaseline model gradually deteriorates compared to the proposed model.
Note that at the the right-mostcolumn where the baseline model is not compressed and the same as the optimal model, the renderingquality of the proposed model is hard to discern from the optimal model.
In Figure 3, we show thateven though our model is only supervised at 5 discrete ranks, it can remain good compression qualityat any other ranks.
Figure 3 provides an explanation for the compressibility of our model.
With therank-residual learning, the rank importance is more concentrated to the lower ranks, which benefitsthe low-rank approximation.
[Table]
4.3 Composition Results
In Figure
[Table]
we demonstrate the composability of our method.
Without extra optimization, we areable to perform affine transformation and composition of different models like composing meshes ina 3D editor.
Note that our model can correctly handle the occlusion and collision between differentobjects.
In Figure 5, we combine the compression capability of our model in scene composition.
In acomplex scene composed of lots of objects, we can compress the less important objects to make atrade-off between the model size and rendering quality.
4.4 Comparisons and Discussion
Rendering Quality.
We compare our method with some recent works in Table 2.
We focus on theextra capabilities to facilitate practical applications, but not boosting the rendering quality over theprevious state-of-the-arts, since the vanilla NeRF [26] already reaches photo-realistic rendering inmost cases.
Although the performance of the proposed method is not the best, the simple modeldesign with the extra compressibility and composability is unique and enables various applications.
MLP Renderers.
As discussed in [8, 27], the absence of a small MLP renderer network generallyleads to worse performance, especially for the specular details.
However, these renderers are trainedseparately and cannot be shared across scenes unless explicitly restricted, which causes inconvenienceor limits the potency of composition.
We therefore discard the MLP at the cost of a slightly worserendering quality, but facilitates the composition and compression.
Limitations.
Although our method can correctly compose the geometry of multiple objects, we don’tconsider the lighting conditions.
Our model bakes lighting conditions into the color like the vanilla
NeRF, and cannot perform re-lighting to achieve consistent lighting effect after composition.
A futuredirection is to integrate the reflectance models [3, 52].
Besides, we only support modeling boundedobjects for now.
A background model [51] can be combined to simulate unbounded scenes.
Conclusion
In this work, we present a novel compressible and composable neural radiance field representation.
Our model is designed to be simple and flexible, yet still being effective enough to render photorealistic images.
A rank-residual learning strategy is proposed to enable near-optimal low-rankapproximation, which allows dynamic adjustment of the model size to support different LODs indifferent scenarios.
All models represented with our method can be arbitrarily transformed andcomposed together like in a 3D editor.
Powered by these properties, we are able to efficiently andconveniently manipulate complex scenes with multiple objects.
We believe our method will furtherfacilitate the NeRF-based scene representation in real-world applications.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] Please see Section 4.4( c)
Did you discuss any potential negative societal impacts of your work?
[No](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...(
a) Did you state the full set of assumptions of all theoretical results?
N/A](b) Did you include
complete
proofs of all theoretical results
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] Please see Section 4.1 and the supplementary materials.(c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Please see Section 4.1 and the
supplementary materials.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes](b) Did you mention the license of the assets?
[Yes] Please see Section 4.1 and thesupplementary materials.(c)
Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]