CageNeRF: Cage-based Neural Radiance Field for Generalized 3D Deformation and Animation
Abstract
While implicit representations have achieved high-fidelity results on 3D rendering,deforming and animating the learned implicit field remain a challenging task.
Existing works typically leverage specific 3D model as deformation prior, such as SMPLfor human animation.
However, the category-specific prior dependency limits themto generalize to other objects.
In this work, we propose a novel framework fordeforming and animating the neural radiance field learned on arbitrary objects.
Thekey insight is that we introduce a cage-based representation as deformation prior,which is category-agnostic.
Specifically, the deformation is performed based onan enclosing cage with sparsely defined vertices inside the rendering space, whereeach point is projected into a novel position based on the barycentric interpolationof the deformed cage vertices via weight functions.
In this way, we transformcage into a generalized constraint, which is able to deform and animate arbitrarytarget while preserving geometry details.
Based on extensive experiments, wedemonstrate the effectiveness of our framework in the task of geometry editing,object animation and deformation transfer.
Introduction
Editing and manipulating 3D object is a fundamental task for generating immerse contents, whichenables numerous applications in VR/AR.
Although recent advances in implicit neural rendering,such as neural radiance field (NeRF) [29] are able to produce photo-realistic modeling of staticobjects, how to edit the 3D content remains a challenging task.
In this paper, we focus on learning ageneral deformation field for NeRF, enabling the deformation and animation of general objects.
Editing explicit 3D representation, such as point cloud and mesh, has been well studied.
Point cloudand deformable mesh [7, 3, 14] store the geometry information in 3D space, which are directlyeditable and are widely utilized in computer graphics for modeling dynamic objects [31].
Cage-baseddeformation [21, 22, 26], for example, has been used to animate polygon mesh, where per-vertexdeformation can be generated by translations prescribed on a coarse cage.
While mesh and point cloud representations are relatively flexible to edit and animate, it is difficult to produce geometricallyaccurate and visually-faithful complex objects and scenes.
Additional data such as normal maps andtextures are usually required to achieve high-quality rendering results.
Recent works on implicit neural rendering, such as NeRF, employ multi-layer perceptron (MLP)as a universal approximator [16, 38], allowing them to approximate a continuous volumetric scenefunction to reconstruct high-fidelity results with only sparsely sampled images.
Therefore, it ishighly desirable if the neural rendering models can be also applied to model dynamic objects.
However, optimizing a deformable radiance field end-to-end for dynamic objects like human body isa highly under-constrained problem.
To address this issue, when modeling dynamic objects, mostworks optimize inside a deformation-invariant canonical space with data-dependent priors, such as
[Figure]
3DMM [4] for face deformation [1, 15], and SMPL [27] for human animation [35, 33].
While usingdata-dependent priors when modeling specific dynamic objects proved to be effective in providingthe stable correspondence between the object structure and the implicit field, it also limits the model’scapability to generalize for broader applications.
To address this issue, we propose a novel framework which focuses on deforming and animatinggeneral objects represented by neural radiance fields, as shown in Figure 1.
We take inspiration fromthe vanilla cage deforming approaches [21, 46] and extend them to work with implicit neural radiancefields.
However, directly applying cage deformation into implicit field is non-trivial, due to thefollowing issues.
i) Traditional cage deformation methods work on polygon mesh with finite verticesand faces, while implicit radiance field is a continuous 3D volume.
ii) Cage-based deformationtransfer requires explicit geometry of the target, which is not directly accessible in the implicit field.iii) It is computationally expensive to densely sample and discretize the implicit field.
To addressthese issues, we propose to estimate a coarse geometry of the target and generate a discrete fieldwhich supports bi-directional mapping between the canonical space and the deformed space, whilekeeping the radiance field continuous.
Moreover, we further design a novel sampling strategy toreduce the computational cost while maintaining the render quality.
The advantages of our framework are three-fold.
First, by utilizing category-agnostic cage-baseddeformation, our framework does not rely on data-dependent priors and can be generalized to workwith a wide rage of objects.
Second, since cage-based deformation is a self-contained method, oncethe model is trained on the static scene, no additional training is required to apply deformation.
Third, both cage deformation and volumetric rendering operate on the Euclidean space, we candesign our framework in a decoupled manner as long as we keep the underlying coordinate systemconsistent between the two modules.
This increases the flexibility and versatility of our framework.
Moreover, as a well-developed method in computer graphics, cage-based deformation can be used invarious tasks, which further expands the application scenario of our framework.
We demonstrate theeffectiveness of our framework in geometry editing, object animation, and deformation transfer.
Related Work
Novel View Synthesis.
Traditional methods for novel view synthesis rely on explicit representationsof 3D scenes.
In these methods, 3D objects are modeled by meshes, point clouds [48, 39], etc,and then novel views are generated by shading and rendering.
However, high-fidelity textures andwell-designed 3D models are required for photo-realistic rendering, which makes the procedurehighly expensive and inefficient.
Recently, NeRF [29] has achieved impressive performance onnovel view synthesis, where 3D objects are represented by implicit functions which map positionsto densities and colors.
A number of follow-up works have made further improvements to NeRFto yield synthesis results with higher quality [51, 2, 50, 44].
Although these NeRF-based implicitmethods are good at representing static scenes, it is still challenging to model dynamic scenes.
Explicit Geometry Deformation.
Modeling deformable 3D objects is a fundamental task in computergraphics and computer vision, and triangle meshes are most commonly utilized due to their flexibilityin manipulation.
Traditional methods achieve this by deforming a fixed set of vertex points and then
[Figure]
MLP
[Figure]
Neural Renderer
[Figure]
𝑥𝑥
[Figure]
𝑦𝑦
[Figure]
[Figure]
interpolating values among them [23, 45], such as skinning on skeletons [18, 28] or cages-basedmethods [22, 40, 53].
Along with the advances of large-scale datasets [6, 8, 49], learning-basedmethods are proposed by analyzing the prior principle of mesh deformation [10, 46, 11, 20].
Wang etal.
[46] proposed a differentiable cage-based deformation module to synthesize shape variations andperform deformation transfer.
However, these deformation methods can be only applied to explicitmodels, e.g., meshes.
It is expensive to generate high-fidelity textures for photo-realistic rendering,and these methods cannot be directly applied to implicit models.
Implicit Geometry Deformation.
Compared to explicit representations that only discretely controlthe vertex of meshes, the deformation of implicit geometry is more challenging, since the space isrepresented with a continuous function.
Recently, various approaches build different deformationfields on implicit representations for animation [32, 19, 41, 42].
However, the generalization abilitiesof these methods are limited with respect to novel poses and different objects.
Referring to the widelyconcerned digital humans, a vast range of methods are proposed for the deformation of the implicitface [13, 54, 15] and body [35, 9, 37, 34].
However, these methods heavily rely on parametric modelsas priors, such as 3DMM [5, 25] and SMPL [27], which are difficult to transfer to non-humanoidobjects.
In this paper, we propose a cage-based implicit geometry deformation method, which utilizesa sparse neural cage as guidance to build the deformation field for NeRF.
The proposed CageNeRF isa general algorithm which is applicable to a wide variety of objects.
Methodology
We propose a novel framework, dubbed CageNeRF , for editing and animating arbitrary 3D objectsrepresented by a neural radiance field.
While radiance fields achieves photo-realistic rendering results,one remaining challenge is how to deform the objects represented by a a continuous implicit volume.
To tackle this challenge, CageNeRF We address this challenge by employing the cage-based deformation, which offers an elegant way ofmanipulating points inside a continuous space through a predefined coarse triangle mesh called cage.
As shown in Figure 2, our framework consists of two modules: the neural rendering module optimizesthe radiance field of a given static object inside canonical space using multiview images with knowncamera parameters.
Then the cage deformer module deforms the cage inside this canonical space andreconstructs the implicit radiance field by translating points to novel positions.
3.1 Canonical Space Neural Rendering
We first train our neural renderer module to acquire the NeRF volume [29], which acts as a referencecanonical space.
NeRF models the scene along with its geometry, lighting, and material as acontinuous volume and approximates it as a 5D radiance field using an MLP network.
Given a 3D position of a point p =
( x, y, z ) , the network learns a mapping function of its density σ and colorc =
( r [Equation] g, b ) variations with respect to the view direction d
= ( θ, ϕ ) :
[Equation]
where γ ( · ) is a positional embedding function of sinusoidal basis and Θ is the trainable parameters.
Then, volumetric rendering is performed to synthesize images by marching camera rays r
= o + t dfrom viewpoint o , with direction d , through the radiance field.
The pixel color ˆ C is estimated usingdiscrete integration on points sampled along each ray:
[Equation]
where δ i =
t i +1 − t
i is the interval between adjacent sample points.
We then optimize our neuralrender module inside this canonical space using L 2 loss calculated between the estimated color ˆ C ( r )and ground truth pixel value C ( r ) of the input images.
[Equation]
where R denotes the set of rays that contributes to sampled pixel.
3.2 Cage Optimization
To deform the neural radiance field, we need to establish a consistent correspondence between thecanonical and the deformed fields. To achieve this goal, a cage, i.e., a coarse polygon mesh inside thecanonical space, is optimized as the structural constraint of the object. Then, the deformation fieldcan be interpreted as the deformation based on this cage.
To achieve accurate deformation, the cageneeds to be optimized to enclose the objects of arbitrary shape and size.
The geometry of the object inside the canonical space is represented by a polygon mesh G = { v , f } ,where v and f represent the vertex and face, respectively. When an explicit surface is not available, Gcan be obtained by applying the marching cube algorithm [30] on a radiance field or SDF [44].
The template cage C is a polygon mesh containing N vertices, which is optimized w.r.t.
the meanvalue coordinates (MVC) of sampled vertices of G .
Since negative MVC occurs when 1) the pointlies outside the cage and 2) the cage itself is self-overlapping or highly concave, by penalizing thenegative values, this optimization process automatically drives the vertices of the template cage toenclose the object inside the canonical space:
[Equation]
where ϕ i ( · ) is the MVC weight function w.r.t.
sampled vertex v from polygon mesh G and the i -thvertex on cage C .
3.3 Cage-based Deformation
[Figure] After the cage C is optimized w.r.t. an object,we can perform deformation based on the target novel pose/shape and the current cage shape.
While it is possible to perform the cage-based deformation by explicitly editing the cage vertices,we take inspiration from Neural Cage [46] anddesign a cage deformer module, which generates a deformed cage-based on a novel polygonmesh G n as the deformation target.
This cagedeformer module makes our framework applicable to tasks like deformation transfer and objectreenactment. Similar to the network proposed in
Neural Cages, this module takes an encoder-decoder structure with PointNet++ [36] as its backboneto automatically calculate the displacement prescribed on each vertex of the optimized cage C , asshown in Figure 3.
We pretrain our cage deformer in two different datasets: 1) ShapeNet [8] for deformation transferand geometry editing on general objects; 2) an extended version of Surreal dataset [12] for bettergeneralization ability on novel pose synthesis and animation.
When performing the cage deformation, the encoder E extracts geometry features from the optimizedcage C and the given deformation target G t , and outputs f c and f t , respectively.
The decoder D takesin the concatenated features and outputs the displacement ∆ v i for each vertex on cage C :
[Equation]
where ⊕ refers to vector concatenation and ∆ v is the per-vertex displacement w.r.t cage C .
Since thecage deformation only changes vertex position while preserving its topology, the deformed cage C ′can be obtained by adding the displacement ∆ v :
[Equation]
The deformed cage C ′ is later used to generate an MVC field for radiance field editing.
3.4 Radiance Field Editing
Once the deformed cage C ′ is acquired, we can edit the radiance field with the following two steps.1) Deforming the original geometry G acquired in Section 3.2 into a novel mesh G ′ using traditionalcage-based deformation.
This newly generated mesh is used as the 3D mask to filter out redundantsample points and speed up the rendering process.
2) Building a dense MVC field, which acts as theconsistent correspondence during deformation, based on the deformed cage C ′ through grid sampling.
Deformed mask as 3D Mask.
The deformed mesh G ′ can be obtained by performing cage-baseddeformation directly on the extracted mesh G of the object inside the canonical space.
For everyvertex v i ∈ G , we first express its position using MVC coordinates:
[Equation]
where ϕ C j ( · ) is a series of weight function depending on the relative position between the cage verticesv j and the given mesh vertex v i .
The detailed derivation of the 3D barycentric interpolation is givenin the supplementary material.
Then, the vertices of the deformed mesh G ′ can be obtained by:
[Equation]
where every vertex v ′ i on G ′ has its novel position determined by the deformed cage vertices v ′
j ∈ C ′as its new basis.
Since our radiance field is sparsely occupied by the render target, in order to increase the renderingspeed while maintaining visual quality, we use the deformed mesh G ′ as a 3D mask, where only thepoints within the vicinity threshold ϵ of the mesh surface are sampled during volumetric rendering.
Anti-aliasing MVC Sampling.
As shown in Equations 7,8, while the coordinate representationof point changed in Euclidean space, the MVC representation remains invariant.
We extend thisproperty on meshes into 3D volumes, which yields our editable radiance field.
By utilizing thisinvariant correspondence, we are able to effectively edit the radiance field via volumetric cage-baseddeformation:
[Equation]
(9) where every point p in this deformable radiance field F d has its MVC coordinates calculated undera given cage C . F is the canonical radiance field learned by our neural renderer module.
And thedeformation in this neural volume can be carried out by replacing the original cage C with a novelone C ′ .
Since the MVC values remain invariant once the cage is fixed, to reduce computation cost, we build apre-sampled MVC field based on the deformed mesh G ′ , as is shown in Figure 4.
Specifically, weapply a grid sampling strategy to the deformed space, and the MVC of the sampled grid point p gis equal to the barycentric coordinates of the closest point on the deformed mesh p n ∈ G ′ to p g . Source Deformation Targets D i ff e r e n t V i e w s c a nn o n i c a l s c a l e r o t a t e
[Figure]
𝜖𝜖 [Figure] 𝑟𝑟 sampling method projects grid points onto the geometry target w.r.t. the surface normal ⃗n i .
Grid points outside the sample range ϵ will be filtered out. Points p 0 , 1 ,inside the sample range have the same nearest point p n on deformed mesh G ′ and thus share the same MVC value.
This causes aliasing when rendering the deformed radiance field.
We devise a shell G s (dotted blue circle) which closely bounds the render geometry by a small margin and assign the MVC value by comparing the distance from the grid point to both the deformed mesh G ′ and the shell G s to negate this effect.
While this sampling strategy is efficient, it causesthe aliasing issue as shown in Figure 4. The points{ p 0 , p 1 , p 2 } are sampled grid points, since they allproject to the same closest point p n on the deformed mesh G ′ , i.e., they share the same MVC value. When rendering the deformed object, sampled points{ p 0 , p 1 , p 2 } are projected to the same position insidethe canonical space and thus have the same color anddensity value.
Aliased sampled points such as p 2 hasthe same color and density as p n instead of the correcttransparent color. This causes the ”expanding” effectin the deformation space, e.g., the deformed object inthe figure will be rendered as a circle with radius r + ϵ ,which should be r instead.
To mitigate this issue, weadditionally define a shell G s , which closely bounds thesurface of G ′ by a small gap.
When assigning MVCvalue to grid points, we compare its distance to both G sand G ′ and determine the MVC value according to:
[Equation]
where ϕ grid is the MVC of the sampled grid point, ϕ gis the MVC of the closest point p g on mesh G ′ , ϕ s isthe MVC of the closest point p s on shell G s .
Whenrendering the deformed neural volume, points such asp 1 , 2 with MVC of ϕ s are projected outside the object incanonical space and thus have correct transparent color.
After acquiring this pre-sampled MVC field, we use thevolumetric rendering technique to synthesize images ofthe deformed radiance field using Equation 1, with acage deformation function C ( · ) in Equation 7 applied:
[Equation]
[Table]
Experiments
To verify the generalization of cage deformation, we evaluate CageNeRF on three tasks:
Geometry editing(Section 4.1).
In this task, we edit the geometry of the implicit object by deformingthe radiance field.
This is achieved by either explicitly edit the cage shape or use the cage deformermodule to generate novel cage via deformation transfer.
Deformation transfer is a task that takestwo objects with different geometry properties (e.g., shape, structure and scale), and modifies theseproperties on the source to make it resemble the target.
In this task, CageNeRF is trained on the legodataset used in NeRF [29] and a chair dataset generated in the same manner.
Neural animation (Section 4.2).
In this task, we train and evaluate CageNeRF on both synthesizedobjects and human videos.
The synthesized 3D objects contain two characters: ‘Hulk’ and a bluewhale.
Furthermore, we evaluate CageNeRF on real humans using Human 3.6M [17].
The animationsin the benchmarks are driven by Linear Blend Skinning [24].
Animation reenactment (Section 4.3).
In this task, we select a benchmark synthesized by ahumanoid robot.
This dataset contains the rendered 360 ◦ multi-view images with paired cameraparameters.
We randomly sample posing SMPL models from the Surreal dataset [43] as the target.
We further provide the ablation study w.r.t.
the MVC sampling strategy.
Following the majority ofprevious literature [29, 33, 35, 44], we use PSNR/SSIM [47] and LPIPS [52] as evaluation metrics.
We refer readers to the supplementary material for more details and results.
4.1 Geometry Editing
We demonstrate the effectiveness of deforming raidance field via explicit cage editing on the legodataset as shown in Figure 6.
We edit the cage geometry to achieve the effect of stretching androtating the bucket.
In our framework, deformation transfer can be applied to edit the geometricproperties of the implicit 3D objects.
Our framework is first trained on an office chair in the canonicalspace, as a deformation source.
We randomly select objects from the ShapeNet dataset [8] as thedeformation targets, and then apply our cage deformer to edit the implicit geometry stored in thecanonical radiance field.
As shown in Figure 5, our framework is capable of editing the implicitgeometry of object in the canonical space by deforming the radiance field learned by our neuralrenderer module.
The deformed results resemble the selected target in aspects of size, shape andscale.
These results demonstrate that our framework can edit the radiance field such that the deformedgeometry resembles the target.
4.2 Neural Animation
Synthesized Object Animation.
To quantitatively determine the animation precision, we analyzethe rendering quality of an animated sequence and compare it to the traditional linear blend skinning(LBS) animation method which operates on the explicit deformable mesh.
We train our neuralrenderer using multiview images of the object rendered at the starting frame.
To animate the object,we extract the animated mesh from the animation sequence and use it as the deformed geometry G n .
Our cage deformer then builds the MVC field using Equation 10.
We animate our object by applyingthe extracted meshes G n sequentially to our framework.
As is shown in Figure 7, our framework is capable of deforming the radiance field and generatingcontinuous animation of objects. CageNeRF also achieves nice render quality which is comparable tothat driven by explicit animation methods. As shown in Table 1, our framework achieves promisingresults in animating the neural radiance field when compared to explicit mesh deforming methods.
[Figure] i v e n [Figure]
i [Figure] n
O u r
[Figure]
A n i N e R F
O u r s
G r o un d T r u t h
[Figure] L B [Figure]
D r i v e n L B S D r i v e n
[Figure]
N e R F
O u r s
G r o un d T r u t h
[Figure]
Human Animation.
We also evaluate on a real-world dataset (i.e., Human 3.6M), and compare withthe state-of-the-art method AniNeRF [33].
The Human 3.6M dataset is captured by 4 synchronizedcameras and has 2D and 3D joint annotations.
We first estimate the geometry proxy of the posinghuman using the SMPL model by optimizing the 3D joint error and 2D re-projection joint error.
Then,we utilize the geometry proxy as the “novel pose” and apply the operation in Figure 3 to generatea deformed cage with the matching pose.
Then we build the MVC field using this deformed cageand posing geometry proxy.
Since our model only stores the information inside the canonical space,
CageNeRF can generalize to novel poses without additional training.
This is in contrast to AniNeRF,which needs to optimize an additional blend weight network using the estimated geometry in order tosynthesize novel pose results.
As shown in Figure 8 and Table 2, the performance of our framework in synthesizing dynamic humanbody is comparable to AniNeRF.
CageNeRF achieves slightly lower PSNR due to the inaccurateestimation of the geometry used in our cage deformer module.
Moreover, we employ SMPL modelas the geometry proxy during cage deformation, where SMPL only models human body, whilethe performer in Human 3.6M dataset wears loosely fitted clothes which has complex nonrigiddeformations during motion.
The inaccuracy in the geometry proxy further induces samplingdisplacement during volumetric rendering and thus impacts the performance of our framework.
Despite the fact that our framework only takes a coarse estimation of the object, it is still able tocorrectly deform and synthesize complex dynamic objects such as the human body.
[Table]
[Table]
4.3 Pose Reenactment
To analyze the generalization ability of CageNeRF, we apply our framework to tasks of pose reenactment and motion retargeting from humans to humanoid objects.
We keep the same training processas Section 4.1.
Afterward, we randomly sample posed SMPL models from the Surreal dataset asthe target for deformation transfer.
Both the posed SMPL model and the optimized robot cage arefed into our cage deformer to calculate the cage deformation.
Finally, the barycentric interpolationis applied to deform the static robot model using Equation 8.
As is shown in Figure 9, althoughthe structure of the source robot is notably different from the target human, the deformed pose andgeometric structure correspond well to the given target, while keeping the source identity and texture.
4.4 Impact of the MVC Sampling Strategy
To evaluate the effectiveness of our anti-aliasing MVC sampling strategy, we generate an MVC fieldusing different sampling strategies and qualitatively and quantitatively compare the rendering results.
As shown in Figure 10, the rendering result generated with grid-sampled MVC field displays inflatedgeometry and blurred texture, due to the inaccurate mapping between the canonical and the deformedspace.
In contrast, the result rendered using our anti-aliasing sampled MVC field has notably sharpertextures and a more accurate geometry structure.
As also shown in Table 3, the anti-aliasing samplingstrategy significantly increases the render quality on all three evaluation metrics.
Conclusion
We propose an versatile framework for deforming and animating implicit field.
Comparing to previousmethods, our framework excels in aspects of generalization, structural flexibility and editing capability.
This is achieved by adopting a deformable cage as the universal constraint for our radiance field editingwhich does not require any data-dependent prior knowledge.
We demonstrate the effectiveness of ourframework in the tasks of object animation, geometry editing and deformation transfer.
Experimentalresults show that our framework is capable of animating and deforming arbitrary objects representedby implicit field and produces results comparable to other state-of-the-art methods.
Checklist
For all authors..
(a) Do the main claims made in the abstract and introduction accurately
reflect
the paper’scontributions and scope?
[Yes] (b) Did you describe the limitations of your
work
[Yes (c)
Did you discuss any potential negative societal impacts of your work?
Yes ] , our workfocuses on providing editing method for implicit 3D representations in a generalizedmanner, it is possible that someone utilizes our method to produce fake videos.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A] We do not
include theoretical results.
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[Yes] We describe
the implementation details in the supplementary material.
The data utilized is publiclyavailable, and we will make our code publicly available.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] We discuss the training details in supplementary material.
(c) Did you report error bars (e
g., with respect to the random seed after running experi- ments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See supplementary for detaileddescriptions.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We use existing datasets and cited them in our paper.
(b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] We discuss this in the supplementary material.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No] The data used in our work does not contain such information.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[No] Our work does not involve crowdsourcing as such, same as follows.
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[No]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[No]