CageNeRF: Cage-based Neural Radiance Field for Generalized 3D Deformation and Animation
Abstract
While implicit representations have achieved high-fidelity results on 3D rendering,deforming and animating the learned implicit field remain a challenging task.
Existing works typically leverage specific 3D model as deformation prior, such as SMPLfor human animation.
However, the category-specific prior dependency limits themto generalize to other objects.
In this work, we propose a novel framework fordeforming and animating the neural radiance field learned on arbitrary objects.
Thekey insight is that we introduce a cage-based representation as deformation prior,which is category-agnostic.
Specifically, the deformation is performed based onan enclosing cage with sparsely defined vertices inside the rendering space, whereeach point is projected into a novel position based on the barycentric interpolationof the deformed cage vertices via weight functions.
In this way, we transformcage into a generalized constraint, which is able to deform and animate arbitrarytarget while preserving geometry details.
Based on extensive experiments, wedemonstrate the effectiveness of our framework in the task of geometry editing,object animation and deformation transfer.
Introduction
Editing and manipulating 3D object is a fundamental task for generating immerse contents, whichenables numerous applications in VR/AR.
Although recent advances in implicit neural rendering,such as neural radiance field (NeRF) [30], are able to produce photo-realistic modeling of staticobjects, how to edit the 3D content remains a challenging task.
In this paper, we focus on learning ageneral deformation field for NeRF, enabling the deformation and animation of general objects.
Editing explicit 3D representation, such as point cloud and mesh, has been well studied.
Point cloudand deformable mesh [7, 3, 14] store the geometry information in 3D space, which are directlyeditable and are widely utilized in computer graphics for modeling dynamic objects [31].
Cage-baseddeformation [21, 22, 26], for example, has been used to animate polygon mesh, where per-vertexdeformation can be generated by translations prescribed on a coarse cage.
While mesh and point cloud representations are relatively flexible to edit and animate, it is difficult to produce geometricallyaccurate and visually-faithful complex objects and scenes.
Additional data such as normal maps andtextures are usually required to achieve high-quality rendering results.
Recent works on implicit neural rendering, such as NeRF, employ multi-layer perceptron (MLP)as a universal approximator [16, 37], allowing them to approximate a continuous volumetric scenefunction to reconstruct high-fidelity results with only sparsely sampled images.
Therefore, it ishighly desirable if the neural rendering models can be also applied to model dynamic objects.
However, optimizing a deformable radiance field end-to-end for dynamic objects like human body isa highly under-constrained problem.
To address this issue, when modeling dynamic objects, mostworks optimize inside a deformation-invariant canonical space with data-dependent priors, such as
[Figure]
3DMM [4] for face deformation [1, 15], and SMPL [27] for human animation [35, 33].
While usingdata-dependent priors when modeling specific dynamic objects proved to be effective in providingthe stable correspondence between the object structure and the implicit field, it also limits the model’scapability to generalize for broader applications.
To address this issue, we propose a novel framework which focuses on deforming and animatinggeneral objects represented by neural radiance fields, as shown in Figure 1.
We take inspiration fromthe vanilla cage deforming approaches [21, 45] and extend them to work with implicit neural radiancefields.
However, directly applying cage deformation into implicit field is non-trivial, due to thefollowing issues.
i) Traditional cage deformation methods work on polygon mesh with finite verticesand faces, while implicit radiance field is a continuous 3D volume.
ii) Cage-based deformationtransfer requires explicit geometry of the target, which is not directly accessible in the implicit field.iii) It is computationally expensive to densely sample and discretize the implicit field.
To addressthese issues, we propose to estimate a coarse geometry of the target and generate a discrete fieldwhich supports bi-directional mapping between the canonical space and the deformed space, whilekeeping the radiance field continuous.
Moreover, we further design a novel sampling strategy toreduce the computational cost while maintaining the render quality.
The advantages of our framework are three-fold.
First, by utilizing category-agnostic cage-baseddeformation, our framework does not rely on data-dependent priors and can be generalized to workwith a wide rage of objects.
Second, since cage-based deformation is a self-contained method, oncethe model is trained on the static scene, no additional training is required to apply deformation.
Third, both cage deformation and volumetric rendering operate on the Euclidean space, we candesign our framework in a decoupled manner as long as we keep the underlying coordinate systemconsistent between the two modules.
This increases the flexibility and versatility of our framework.
Moreover, as a well-developed method in computer graphics, cage-based deformation can be used invarious tasks, which further expands the application scenario of our framework.
We demonstrate theeffectiveness of our framework in geometry editing, object animation, and deformation transfer.
Related Work
Novel View Synthesis.
Traditional methods for novel view synthesis rely on explicit representationsof 3D scenes.
In these methods, 3D objects are modeled by meshes, point clouds [47, 38], etc,and then novel views are generated by shading and rendering.
However, high-fidelity textures andwell-designed 3D models are required for photo-realistic rendering, which makes the procedurehighly expensive and inefficient.
Recently, NeRF [30] has achieved impressive performance onnovel view synthesis, where 3D objects are represented by implicit functions which map positionsto densities and colors.
A number of follow-up works have made further improvements to NeRFto yield synthesis results with higher quality [50, 2, 49, 43].
Although these NeRF-based implicitmethods are good at representing static scenes, it is still challenging to model dynamic scenes.
Explicit Geometry Deformation.
Modeling deformable 3D objects is a fundamental task in computergraphics and computer vision, and triangle meshes are most commonly utilized due to their flexibilityin manipulation.
Traditional methods achieve this by deforming a fixed set of vertex points and then
[Figure]
∑𝜙
[Figure]
[Figure]
[Figure]
[Figure]
Cage ∑𝜙 𝑗𝐶 ′ (𝑝;𝑣 𝑐′ )
Deformed Radiance Field
Canonical Radiance Field
MLP
Novel View (Canonical) Novel Pose (Deformed) C a g e D e f o r m e r
Neural Renderer
Cage Deformer (𝑥,𝑦,𝑧) (𝑥 ′ ,𝑦 ′ ,𝑧′)
MLP deforms the implicit interpolating values among them [23, 44], such as skinning on skeletons [18, 29] or cages-basedmethods [22, 39, 52].
Along with the advances of large-scale datasets [6, 8, 48], learning-basedmethods are proposed by analyzing the prior principle of mesh deformation [10, 45, 11, 20].
Wang etal.
[45] proposed a differentiable cage-based deformation module to synthesize shape variations andperform deformation transfer.
However, these deformation methods can be only applied to explicitmodels, e.g., meshes.
It is expensive to generate high-fidelity textures for photo-realistic rendering,and these methods cannot be directly applied to implicit models.
Implicit Geometry Deformation.
Compared to explicit representations that only discretely controlthe vertex of meshes, the deformation of implicit geometry is more challenging, since the space isrepresented with a continuous function.
Recently, various approaches build different deformationfields on implicit representations for animation [32, 19, 40, 41].
However, the generalization abilitiesof these methods are limited with respect to novel poses and different objects.
Referring to the widelyconcerned digital humans, a vast range of methods are proposed for the deformation of the implicitface [13, 53, 15] and body [35, 9, 36, 34].
However, these methods heavily rely on parametric modelsas priors, such as 3DMM [5, 25] and SMPL [27], which are difficult to transfer to non-humanoidobjects.
In this paper, we propose a cage-based implicit geometry deformation method, which utilizesa sparse neural cage as guidance to build the deformation field for NeRF.
The proposed CageNeRF isa general algorithm which is applicable to a wide variety of objects.
Methodology
We propose a novel framework, dubbed CageNeRF, for editing and animating arbitrary 3D objects byseamlessly collaborating the cage-based deformation with neural rendering.
Cage-based deformationoperates on explicit 3D geometry representations such as polygon mesh, where each vertex is given amean value coordinate (MVC), which is calculated through barycentric interpolation. MVC encodesthe relative position of a given point w.r.t.
the vertices on the cage. Deformation on the mesh iscarried out by first translating the cage vertices and then using the relative position encoded by MVCto calculate a new position for each mesh vertex.
As shown in Figure 2, our framework consists oftwo modules: a neural rendering module and a cage deformer module. We first optimize the radiancefield of a given static object inside canonical space using multiview images with known cameraparameters.
Then the cage deformer module warps the cage and reconstructs the implicit radiancefield by translating the points inside the canonical space, based on the deformed cage vertices.
3.1 Canonical Space Neural Rendering
We first train our neural renderer module to acquire the NeRF volume [30], which acts as a referencecanonical space.
NeRF models the scene along with its geometry, lighting, and material as acontinuous volume and approximates it as a 5D radiance field using an MLP network.
Given a 3D position of a point p =
( x, y, z ) , the network learns a mapping function of its density σ and colorc =
( r, g, b ) variations with respect to the view direction d
= ( θ, ϕ ) :
[Equation]
where γ ( · ) is a positional embedding function of sinusoidal basis and Θ is the trainable parameters.
Then, volumetric rendering is performed to synthesize images by marching camera rays r
= o + t dfrom viewpoint o , with direction d , through the radiance field.
The pixel color ˆ C is estimated usingdiscrete integration on points sampled along each ray:
[Equation]
where δ i =
t i +1 − t
i is the interval between adjacent sample points.
We then optimize our neuralrender module inside this canonical space using L 2 loss calculated between the estimated color ˆ C ( r )and ground truth pixel value C ( r ) of the input images.
[Equation]
where R denotes the set of rays that contributes to sampled pixel.
3.2 Cage Optimization
Since it is non-trivial to establish the correspondence between the deformation target (often in explicitform) and the deformation source (i.e., our implicit radiance field), explicit transformation cannot bedirectly applied to the canonical space. In contrast, cage-based deformation can be applied universallyto any point inside the canonical space via barycentric interpolation using the vertices of a given cage.
We notice that this space is sparsely occupied by our model, it is impractical and inefficient to denselysample the entire space for deformation. Therefore, we take the explicit geometry of the object as aproxy for our cage deformer module. Specifically, the geometry G can can be extracted with eithermarching cubes algorithm [28] or by optimizing a neural SDF representation [43] using the sametraining dataset as our neural renderer module: [Equation]
where f s ( x ; θ ) is the SDF function with learnable parameters θ , and the geometry of the object is thezero-level set of f s ( x ; θ ) . We then obtain the geometry proxy G p ( v , f ) with vertices v ∈ R N × 3 andfaces f by uniformly sampling vertices points v from G can , and building faces f using the nearestneighbor algorithm.
Since cage-based deformation in our framework warps the entire neural volume that the cage encloses,the object must reside inside the warping space. Considering that G p ( v , f ) can be of arbitrary shape,we adopt an optimization process that translates a template cage C t ( v , f ) into an optimized cage C can ( v , f ) , which geometrically aligns and encloses this object. As a starting point, we provideour cage deformer module with the template cage C t ( v , f ) .
Then, we calculate the MVC of pointsuniformly sampled from the geometry proxy G p .
Since negative in MVC occurs when 1) the pointlies outside the cage and 2) the cage itself is self-overlapping or highly concave, by penalizingthe negative MVC values of the points sampled from the extracted object geometry, our networkautomatically adapts to the template cage:
[Equation]
5) where ϕ i is the calculated MVC weight w.r.t.
vertex v i of cage C t ( v , f ) .
3.3 Cage-based Deformation
After acquiring a fitting cage C can ( v , f ) from the optimization process, we are able to performdeformation based on the target novel pose/shape and the current cage shape.
We take inspiration from
Neural Cage [45] and adopt an encoder-decoder structure to calculate the displacement prescribed oneach vertex of the optimized cage C can , as shown in Figure 3. [Figure] 𝐸 𝐶 𝐷 𝐶
+
Feature Extraction Novel Pose 𝒞 𝑑 Deformed Cage 𝒞 𝑐𝑎𝑛 𝑓 𝐶 𝑓 𝑁 We pretrain our cage deformer in two differentdatasets: 1) ShapeNet [8] for deformation transfer and geometry editing on general objects; 2)an extended version of Surreal dataset [12] forbetter generalization ability on novel pose synthesis and animation.
Since the optimized cagein the canonical space remains invariant during deformation, we extract the cage feature f Cfrom C can using E C , and concatenate it to thefeature extracted from novel pose geometry f N .
The output of our decoder is the displacement∆ v i for each vertex on the optimized cage C can ( v , f ) : [Equation]
[Equation]
where G is the geometry representing the target pose and ⊕ refers to vector concatenation.
Since thecage deformation only changes vertex position while preserving its topology, the deformed cage C dcan be obtained by adding the displacement ∆ v :
[Equation]
The deformed cage C d is later used to generate an MVC field for editing the radiance field .
3.4 Radiance Field Editing
Once the deformed cage C d is obtained by the cage deformer module, we are able to edit the canonicalradiance field by building a dense MVC field, which is generated by sampling the deformed spaceusing both C d and deformed geometry G n .
Geometry Proxy as 3D Mask.
Since our radiance field is sparsely occupied by the render target. Inorder to increase the efficiency of our framework while maintaining render quality, we employ thedeformed geometry proxy as a 3D mask and only points within the vicinity of this mesh are sampledduring volumetric rendering:
[Equation]
where P render is the set of points sampled during volumetric rendering, o is the camera origin
d isthe direction of camera ray, p n is the closest point on G n to p and ϵ is the vicinity threshold, which isset to 0 . 03 in our experiments. To obtain this deformed geometry proxy G n , given the optimized cage C can , and the geometry proxy G p ( v , f ) , where every point p ∈
G p has its MVC determined through barycentric interpolation.
The MVC coordinate of a point p with vertices v j ∈ C can as its basis is denoted as
[Equation]
where ϕ C j ( · ) is a series of weight function depending on the relative position between the cage verticesv c and the given point p .
The detailed derivation of the 3D barycentric interpolation is given in thesupplementary material. G n can be obtained by taking the same MVC calculated under C can andapplying it with vertices v ′ j ∈ C d as its new basis: [Equation]
Anti-aliasing MVC Sampling.
Traditional cage-based deformation only considers points resideon a mesh surface S , which essentially generates an MVC mesh M under a given cage C 0 with Nvertices, where every point p ∈ M is represented via an N -dimensional MVC coordinate:
[Equation]
As shown in Equations 11,12, while the coordinate representation of point changed in Euclidean space,the MVC representation remains invariant.
We extend this property on meshes into 3D volumes,which yields our editable radiance field.
By utilizing this invariant correspondence, we are able toeffectively edit the radiance field via volumetric cage-based deformation:
[Equation]
where F d is the deformable radiance field represented using MVC coordinate and F can is the learnedcanonical radiance field.
We can fully express the deformation by keeping track of the point’s MVCcoordinates inside the canonical space and replacing the original basis using vertices v ′ d ∈ C d .
To reduce computation cost, we build a pre-sampled MVC field based on the deformed geometry proxy G n , as is shown in Figure 4.
Specifically, we apply a grid sampling strategy to the deformed space, and the MVC of the grid point p g is equal to the coordinates of p n ∈ G n , which is the nearest point on the deformed geometry proxy G n to p g .
[Figure]
While this sampling strategy is efficient, it causesthe aliasing issue as shown in Figure 4.
The points{ p 0 , p 1 , p 2 } are sampled grid points, since they allproject to the same closest point p n on the deformedobject G n , i.e., they share the same MVC value.
Sincethe point position in the cage space is only determinedby its MVC value, when transformed back to canonicalspace, these points are “squeezed” into the same pointon G p .
However, these points are sampled with different
Euclidean coordinates during volumetric rendering, this“squeezing” effect in canonical space in turn yields an“expanding” effect in the deformation space. This causesaliased and inflated render results, which cannot correctly reflect the implicit geometry learned by the neuralrenderer module.
To mitigate this issue, we additionallydefine a shell G s , which closely bounds the surface of G n by a small gap.
When assigning MVC value to gridpoints, we first compare its distance to both G s and G dand determine the MVC value according to:
[Equation]
where Φ grid is the MVC of the sampled grid point, Φ g isthe MVC of the closest point p gn on G p , Φ s is the MVCof the closest point p sn on G s .
After acquiring this pre-sampled MVC field, we use thevolumetric rendering technique to synthesize images ofthe deformed radiance field using Equation 1, with acage deformation function C ( · ) in Equation 10 applied:
[Equation]
Experiments
To verify the generalization of cage deformation, weevaluate CageNeRF on three tasks:
Geometry editing via deformation transfer (Section 4.1).
Deformation transfer is a task thattakes two objects with different geometry properties (e.g., shape, structure and scale), and modifiesthese properties on the source to make it resemble the target.
In this task, CageNeRF is trained on asynthetic benchmark, in which a chair is rendered around 360 ◦ using 3D software.
Neural animation (Section 4.2).
In this task, we train and evaluate CageNeRF on both synthesizedobjects and human videos.
The synthesized 3D objects contain two characters: ‘Hulk’ and a blue [Figure] [Table] i ff
[Table]
r e n t V i e w s
[Table] whale. Furthermore, we evaluate CageNeRF on real humans using Human 3.6M [17].
The animationsin the benchmarks are driven by Linear Blend Skinning [24].
Animation reenactment (Section 4.3).
In this task, we select a benchmark synthesized by ahumanoid robot.
This dataset contains the rendered 360 ◦ multi-view images with paired cameraparameters.
We randomly sample posing SMPL models from the Surreal dataset [42] as the target.
We further provide the ablation study w.r.t.
the MVC sampling strategy.
Following the majority ofprevious literature [30, 33, 35, 43], we use PSNR/SSIM [46] and LPIPS [51] as evaluation metrics.
We refer readers to the supplementary material for more details and results.
4.1 Geometry Editing via Deformation Transfer
In our framework, deformation transfer can be applied to edit the geometric properties of the implicit3D objects.
Our framework is first trained on an office chair in the canonical space, as a deformationsource.
We randomly select objects from the ShapeNet dataset [8] as the deformation targets, andthen apply our cage deformer to edit the implicit geometry stored in the canonical radiance field.
As shown in Figure 5, our framework is capable of editing the implicit geometry of object in thecanonical space by deforming the radiance field learned by our neural renderer module.
The deformedresults resemble the selected target in aspects of size, shape and scale.
These results demonstrate thatour framework can edit the radiance field such that the deformed geometry resembles the target.
4.2 Neural Animation
Synthesized Object Animation.
To quantitatively determine the animation precision, we analyzethe rendering quality of an animated sequence and compare it to the traditional linear blend skinning(LBS) animation method which operates on the explicit deformable mesh.
We train our neuralrenderer using multiview images of the object rendered at the starting frame.
To animate the object,we extract the animated mesh from the animation sequence and use it as the deformed geometry G n .
Our cage deformer then builds the MVC field using Equation 14.
We animate our object by applyingthe extracted meshes G n sequentially to our framework.
O u r s
L B S D r i v e n
L B S D r i v e n
O u r s
A n i N e R F
O u r s
G r o un d T r u t h
O u r s
L B S D r i v e n L B S D r i v e n
O u r s
A n i N e R F
O u r s
G r o un d T r u t h
As is shown in Figure 6, our framework is capable of deforming the radiance field and generatingcontinuous animation of objects. CageNeRF also achieves nice render quality which is comparable tothat driven by explicit animation methods. As shown in Table 1, our framework achieves promisingresults in animating the neural radiance field when compared to explicit mesh deforming methods.
Human Animation.
We also evaluate on a real-world dataset (i.e., Human 3.6M), and compare withthe state-of-the-art method AniNeRF [33].
The Human 3.6M dataset is captured by 4 synchronizedcameras and has 2D and 3D joint annotations.
We first estimate the geometry proxy of the posinghuman using the SMPL model by optimizing the 3D joint error and 2D re-projection joint error.
Then,we utilize the geometry proxy as the “novel pose” and apply the operation in Figure 3 to generatea deformed cage with the matching pose.
Then we build the MVC field using this deformed cageand posing geometry proxy.
Since our model only stores the information inside the canonical space,
CageNeRF can generalize to novel poses without additional training.
This is in contrast to AniNeRF,which needs to optimize an additional blend weight network using the estimated geometry in order tosynthesize novel pose results.
As shown in Figure 7 and Table 2, the performance of our framework in synthesizing dynamic humanbody is comparable to AniNeRF.
CageNeRF achieves slightly lower PSNR due to the inaccurateestimation of the geometry used in our cage deformer module.
Moreover, we employ SMPL modelas the geometry proxy during cage deformation, where SMPL only models human body, whilethe performer in Human 3.6M dataset wears loosely fitted clothes which has complex nonrigiddeformations during motion.
The inaccuracy in the geometry proxy further induces samplingdisplacement during volumetric rendering and thus impacts the performance of our framework.
Despite the fact that our framework only takes a coarse estimation of the object, it is still able tocorrectly deform and synthesize complex dynamic objects such as the human body.
[Table] [Table]
R ee n a c t m e n t
T a r g e t s
D e f o r m a t
i o n
R e s u l t s Deformation Source [Figure] 4.3 Pose Reenactment
To analyze the generalization ability of CageNeRF, we apply our framework to tasks of pose reenactment and motion retargeting from humans to humanoid objects.
We keep the same training processas Section 4.1.
Afterward, we randomly sample posed SMPL models from the Surreal dataset asthe target for deformation transfer.
Both the posed SMPL model and the optimized robot cage arefed into our cage deformer to calculate the cage deformation.
Finally, the barycentric interpolationis applied to deform the static robot model using Equation 11.
As is shown in Figure 8, althoughthe structure of the source robot is notably different from the target human, the deformed pose andgeometric structure correspond well to the given target, while keeping the source identity and texture.
4.4 Imapct of the MVC Sampling Strategy
To evaluate the effectiveness of our anti-aliasing MVC sampling strategy, we generate an MVC fieldusing different sampling strategies and qualitatively and quantitatively compare the rendering results.
As shown in Figure 9, the rendering result generated with grid-sampled MVC field displays inflatedgeometry and blurred texture, due to the inaccurate mapping between the canonical and the deformedspace.
In contrast, the result rendered using our anti-aliasing sampled MVC field has notably sharpertextures and a more accurate geometry structure.
As also shown in Table 3, the anti-aliasing samplingstrategy significantly increases the render quality on all three evaluation metrics.
Conclusion
We propose an versatile framework for deforming and animating implicit field.
Comparing to previousmethods, our framework excels in aspects of generalization, structural flexibility and editing capability.
This is achieved by adopting a deformable cage as the universal constraint for our radiance field editingwhich does not require any data-dependent prior knowledge.
We demonstrate the effectiveness of ourframework in the tasks of object animation, geometry editing and deformation transfer.
Experimentalresults show that our framework is capable of animating and deforming arbitrary objects representedby implicit field and produces results comparable to other state-of-the-art methods.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope
[Yes](b) Did you describe the limitations of your work?
[Yes](c)
Did you discuss any potential negative societal impacts of your work?
[Yes] , our workfocuses on providing editing method for implicit 3D representations in a generalizedmanner, it is possible that someone utilizes our method to produce fake videos.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A] We do not
include theoretical results.
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We describe
the implementation details in the supplementary material.
The data utilized is publicly available, and we will make our code publicly available.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] We discuss the training details in supplementary material.
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See supplementary for detaileddescriptions.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We use existingdatasets and cited them in our paper.
(b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] We discuss this in the supplementary material.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content
[No] The data used in our work does not containsuch information.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable
No Our work does not involve crowdsourcing as such, same as follows
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[No]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[No]