CageNeRF: Cage-based Neural Radiance Field for Generalized 3D Deformation and Animation
Abstract
While implicit representations have achieved high-fidelity results in 3D rendering, deforming and animating the implicit field remains challenging.
Existing works typically leverage data-dependent models as deformation priors, such as SMPL for human body animation.
However, this dependency on category-specific priors limits them to generalize to other objects.
To solve this problem, we propose a novel framework for deforming and animating the neural radiance field learned on arbitrary objects.
The key insight is that we introduce a cage-based representation as deformation prior, which is category-agnostic.
Specifically, the deformation is performed based on an enclosing polygon mesh with sparsely defined vertices called cage inside the rendering space, where each point is projected into a novel position based on the barycentric interpolation of the deformed cage vertices.
In this way, we transform the cage into a generalized constraint, which is able to deform and animate arbitrary target objects while preserving geometry details.
Based on extensive experiments, we demonstrate the effectiveness of our framework in the task of geometry editing, object animation and deformation transfer.
The code and supplementary materials are available at https://pengyicong.github.io/CageNeRF/.
Introduction
Editing and manipulating 3D object is a fundamental task for generating immerse contents, which enables numerous applications in VR/AR.
Although recent advances in implicit neural rendering, such as neural radiance field (NeRF) [33], are able to produce photo-realistic modeling of static objects, how to edit the 3D content remains a challenging task.
In this paper, we focus on learning a general deformation field for NeRF, enabling the deformation and animation of general objects.
Editing explicit 3D representations, such as point cloud and mesh, has been well studied.
Point cloud and deformable mesh [6, 3, 16] store the geometry information in 3D space, which are directly editable and are widely utilized in computer graphics for modeling dynamic objects [34].
Cage-based deformation [23, 24, 29], for example, has been used to animate polygon mesh, where per-vertex deformation can be generated by translations prescribed on a coarse cage.
While mesh and point cloud representations are relatively flexible to edit and animate, it is difficult to produce geometrically accurate and visually-faithful complex objects and scenes.
Additional data such as normal maps and textures are usually required to achieve high-quality rendering results.
Recent works on implicit neural rendering, such as NeRF, employ multi-layer perceptron (MLP) as a universal approximator [18, 42], allowing them to approximate a continuous volumetric scene
[Figure]
function to reconstruct high-fidelity results with only sparsely sampled images.
Therefore, it is highly desirable if the neural rendering models can be also applied to model dynamic objects.
However, optimizing a deformable radiance field end-to-end for dynamic objects like human body is a highly under-constrained problem.
To address this issue, when modeling dynamic objects, most works optimize inside a deformation-invariant canonical space with data-dependent priors, such as 3DMM [4] for face deformation [1, 17], and SMPL [30] for human animation [39, 37].
While using data-dependent priors when modeling specific dynamic objects proved to be effective in providing stable correspondence between the object structure and the implicit field, it also limits the model’s capability to generalize for broader applications.
To address this issue, we propose a novel framework which focuses on deforming and animating general objects represented by neural radiance fields, as shown in Figure 1.
We take inspiration from the vanilla cage deforming approaches [23, 52] and extend them to work with implicit neural radiance fields.
However, directly applying cage deformation into implicit field is non-trivial, due to the following issues.
i) Traditional cage deformation methods work on polygon mesh with finite vertices and faces, while implicit radiance field is a continuous 3D volume.
ii) Cage-based deformation transfer requires explicit geometry of the target, which is not directly accessible in the implicit field.
iii)
It is computationally expensive to densely sample and discretize the implicit field.
To address these issues, we propose to estimate a coarse geometry of the target and generate a discrete field which supports bi-directional mapping between the canonical space and the deformed space, while keeping the radiance field continuous.
Moreover, we further design a novel sampling strategy to reduce the computational cost while maintaining the render quality.
The advantages of our framework are three-fold.
First, by utilizing category-agnostic cage-based deformation, our framework does not rely on data-dependent priors and can be generalized to work with a wide range of objects.
Second, since cage-based deformation is a self-contained method, once the model is trained on the static scene, no additional training is required to apply deformation.
Third, both cage deformation and volumetric rendering operate on the Euclidean space, we can design our framework in a decoupled manner as long as we keep the underlying coordinate system consistent between the two modules.
This increases the flexibility and versatility of our framework.
Moreover, as a well-developed method in computer graphics, cage-based deformation can be used in various tasks, which further expands the application scenario of our framework.
We demonstrate the effectiveness of our framework in geometry editing, object animation, and deformation transfer.
Related Work
Novel View Synthesis.
Traditional methods for novel view synthesis rely on explicit representations of 3D scenes.
In these methods, 3D objects are modeled by meshes, point clouds [55, 43], etc, and then novel views are generated by shading and rendering.
However, high-fidelity textures and well-designed 3D models are required for photo-realistic rendering, which makes the procedure highly expensive and inefficient.
Recently, NeRF [33] has achieved impressive performance on novel view synthesis, where 3D objects are represented by implicit functions which map positions to densities and colors.
A number of follow-up works have made further improvements to NeRF
[Figure]
MLP
[Figure]
Neural Renderer
[Figure]
𝑥𝑥
[Figure]
𝑦𝑦
[Figure]
[Figure]
to yield synthesis results with higher quality [58, 2, 57, 50].
Although these NeRF-based implicit methods are good at representing static scenes, it is still challenging to model dynamic scenes.
Explicit Geometry Deformation.
Modeling deformable 3D objects is a fundamental task in computer graphics and computer vision, and triangle meshes are most commonly utilized due to their flexibility in manipulation.
Traditional methods achieve this by deforming a fixed set of vertex points and then interpolating values among them [25, 51], such as skinning on skeletons [20, 32] or cages-based methods [24, 45, 61].
Along with the advances of large-scale datasets [5, 7, 56], learning-based methods are proposed by analyzing the prior principle of mesh deformation [11, 52, 12, 22].
Wang et al.
[52] proposed a differentiable cage-based deformation module to synthesize shape variations and perform deformation transfer.
However, these deformation methods can be only applied to explicit models, e.g., meshes.
It is expensive to generate high-fidelity textures for photo-realistic rendering, and these methods cannot be directly applied to implicit models.
Implicit Geometry Deformation.
Compared to explicit representations that only discretely control the vertex of meshes, the deformation of implicit geometry is more challenging, since the space is represented with a continuous function.
Recently, various approaches build different deformation fields on implicit representations for animation [35, 21, 47, 48].
However, the generalization abilities of these methods are limited with respect to novel poses and different objects.
Referring to the widely concerned digital humans, a vast range of methods are proposed for the deformation of the implicit face [15, 62, 17] and body [39, 8, 41, 38].
However, these methods heavily rely on parametric models as priors, such as 3DMM [4, 28] and SMPL [30], which are difficult to transfer to non-humanoid objects.
In this paper, we propose a cage-based implicit geometry deformation method, which utilizes a sparse neural cage as guidance to build the deformation field for NeRF.
The proposed CageNeRF is a general algorithm which is applicable to a wide variety of objects.
Methodology
We propose a novel framework, dubbed CageNeRF , for editing and animating arbitrary 3D objects represented by a neural radiance field.
Radiance fields can render photo-realistic content.
However, it remains challenging to perform deformation on radiance fields, since it’s non-trivial to learn correspondence between implicit representations.
We address this challenge by employing the cage-based deformation, which offers an elegant way of manipulating points inside a continuous space through a predefined coarse triangle mesh called cage.
As shown in Figure 2, our framework consists of two modules: the neural rendering module optimizes the radiance field of a given static object inside canonical space using multiview images with known camera parameters.
Then the cage deformer module deforms the cage inside this canonical space and reconstructs the implicit radiance field by translating points to novel positions.
3.1 Canonical Space Neural Rendering
We first train our neural renderer module to acquire the NeRF volume [33], which acts as a reference canonical space.
NeRF models the scene along with its geometry, lighting, and material as a continuous volume and approximates it as a 5D radiance field using an MLP network.
Given a 3D position of a point p =
( x, y, z ) , the network learns a mapping function of its density σ and color c
=
( r, g, b ) variations with respect to the view direction d
= ( θ, ϕ ) :
[Equation]
where γ ( · ) is a positional embedding function of sinusoidal basis and Θ is the trainable parameters.
Then, volumetric rendering is performed to synthesize images by marching camera rays r
= o + t d from viewpoint o , with direction d , through the radiance field.
The pixel color ˆ C is estimated using discrete integration on points sampled along each ray:
[Equation]
where δ i =
t i +1 − t
i is the interval between adjacent sample points.
We then optimize our neural render module inside this canonical space using L 2 loss calculated between the estimated color ˆ C ( r ) and ground truth pixel value C ( r ) of the input images:
[Equation]
where R denotes the set of rays that contributes to sampled pixel.
3.2 Cage Optimization
To deform the neural radiance field, we need to establish a consistent correspondence between the canonical and the deformed fields.
To achieve this goal, a cage, i.e., a coarse polygon mesh inside the canonical space is first optimized to be geometrically aligned and enclose the object.
Then the deformation field can be interpreted as the deformation based on the structural constraint of this optimized cage.
Similar to Neural Cages, we provide our network with predefined polygon meshes as the starting point of this optimization process.
The geometry of the object inside the canonical space is represented by a polygon mesh G = { v , f } , where v and f represent the vertex and face, respectively.
When an explicit surface is not available, G can be obtained by applying the marching cube algorithm [31] on a radiance field or SDF [50].
The template cage C is a polygon mesh containing N vertices, which is optimized w.r.t.
the mean value coordinates (MVC) of sampled vertices of G .
Since negative MVC occurs when 1) the point lies outside the cage and 2) the cage itself is self-overlapping or highly concave.
By penalizing the negative values, this optimization process automatically drives the vertices of the template cage to enclose the object inside the canonical space:
[Equation]
where ϕ i ( · ) is the MVC weight function w.r.t.
sampled vertex v from polygon mesh G and the i -th vertex on cage C .
3.3 Cage-based Deformation
After the cage C is optimized w.r.t.
an object, we can perform deformation based on the target novel pose/shape and the current cage shape.
While it is possible to perform the cage-based deformation by explicitly editing the cage vertices, we take inspiration from Neural Cage [52] and design a cage deformer module, which generates a deformed cage based on a novel polygon mesh G n given as the deformation target.
This cage deformer module makes our framework applicable to tasks like deformation transfer and object reenactment.
Similar to the network proposed in Neural Cages, this module takes an encoder-decoder structure with PointNet++ [40] as its backbone to automatically calculate the displacement prescribed on each vertex of the optimized cage C , as shown in Figure 3.
[Figure]
[Equation]
[Figure]
𝐸𝐸
We pretrain our cage deformer in two different datasets: 1) ShapeNet [7] for deformation transfer and geometry editing on general objects; 2) an extended version of Surreal dataset [13] for better generalization ability on novel pose synthesis and animation.
When performing the cage deformation, the encoder E extracts geometry features from the optimized cage C and the given deformation target G t , and outputs f c and f t , respectively.
The decoder D takes in the concatenated features and outputs the displacement ∆ v i for each vertex on cage C :
[Equation]
where ⊕ refers to vector concatenation and ∆ v is the per-vertex displacement w.r.t cage C .
Since the cage deformation only changes vertex position while preserving its topology, the deformed cage C ′ can be obtained by adding the displacement ∆ v :
[Equation]
The deformed cage C ′ is later used to generate an MVC field for radiance field editing.
3.4 Radiance Field Editing
Once the deformed cage C ′ is acquired, we can edit the radiance field with the following two steps.
1) Deforming the original geometry G acquired in Section 3.2 into a novel mesh G ′ using traditional cage-based deformation.
This newly generated mesh is used as the 3D mask to filter out redundant sample points and speed up the rendering process.
2) Building a dense MVC field, which acts as the consistent correspondence during deformation, based on the deformed cage C ′ through grid sampling.
Deformed mesh as 3D Mask.
The deformed mesh G ′ can be obtained by performing cage-based deformation directly on the extracted mesh G of the object inside the canonical space.
For every vertex v i ∈ G , we first express its position using MVC coordinates:
[Equation]
where ϕ C j ( · ) is a series of weight functions depending on the relative position between the cage vertices v j and the given mesh vertex v i .
The detailed derivation of the 3D barycentric interpolation is given in Appendix A.
Then, the vertices of the deformed mesh G ′ can be obtained by:
[Equation]
where every vertex v ′ i on G ′ has its novel position determined by the deformed cage vertices v ′ j ∈ C ′ as its new basis.
Since our radiance field is sparsely occupied by the render target, in order to increase the rendering speed while maintaining visual quality, we use the deformed mesh G ′ as a 3D mask, where only the points within the vicinity threshold ϵ of the mesh surface are sampled during volumetric rendering.
Moreover, this explicit mesh enables us to robustly establish the backward deformation, i.e., from the deformed space to canonical space, when canonical information is not present in datasets like Human 3.6M [19].
See appendix for a detailed discussion.
Anti-aliasing MVC Sampling.
As shown in Equations 7,8, while the coordinate representation of point changed in Euclidean space, the MVC representation remains invariant.
We extend this property on meshes into 3D volumes, which yields our editable radiance field.
By utilizing this invariant correspondence, we are able to effectively edit the radiance field via volumetric cage-based deformation:
[Equation]
[Equation]
where every point p in this deformable radiance field F d has its MVC coordinates calculated under a given cage C .
F is the canonical radiance field learned by our neural renderer.
And the deformation in this neural volume can be carried out by replacing the original cage C with a novel one C ′ .
[Figure]
Note that Grid points outside ϵ are filtered out.
Considering this problem, we devise a shell
G s (dotted blue circle) which closely bounds the render geometry by a small margin.
Then, we assign the MVC value by comparing the distance from the grid point to both the deformed mesh G ′ and the shell G s to negate this effect.
Since the MVC values remain invariant once the cage is fixed, to reduce computation cost, we build a presampled MVC field based on the deformed mesh G ′ , as is shown in Figure 4.
Specifically, we apply a grid sampling strategy to the deformed space, and the MVC of the sampled grid point p g is equal to the barycentric coordinates of the closest point on the deformed mesh p n ∈ G ′ to p g .
While this sampling strategy is efficient, it causes the aliasing issue as shown in Figure 4.
The points { p 0 , p 1 , p 2 } are sampled grid points, since they all project to the same closest point p n on the deformed mesh G ′ , i.e., they share the same MVC value.
When rendering the deformed object, sampled points { p 0 , p 1 , p 2 } are projected to the same position inside the canonical space and thus have the same color and density value.
Aliased sampled points such as p 2 have the same color and density as p n instead of the correct transparent color.
This causes the “expanding” effect in the deformation space, e.g., the deformed object in the figure will be rendered as a circle with radius r + ϵ , which should be r instead.
One naive solution would be directly reducing the vicinity threshold ϵ .
This approach imposes a higher demand on the accuracy of both the implicit representation and the extracted polygon mesh, which is non-trivial.
To mitigate this issue while maintaining a relatively large ϵ , we additionally define a shell G s , which is also a polygon mesh that closely bounds the surface of G ′ by a small gap.
This shell can be obtained by translating vertices on G ′ along their surface normal while maintaining the topology.
When assigning MVC value to grid points, we compare its distance to both G s and G ′ and determine the MVC value according to:
[Equation]
where ϕ grid is the MVC of the sampled grid point, ϕ g is the MVC of the closest point p g on mesh
G ′ , ϕ s is the MVC of the closest point p
s on shell G s .
When rendering the deformed neural volume, points such as p 1 , 2 with MVC of ϕ s are projected outside the object in canonical space and thus have correct transparent color.
Deformed Rendering.
After acquiring this MVC field, we can render the deformed radiance field through volumetric accumulation.
1) Sample points along the camera ray in render space using the deformed mesh G ′ as a 3D mask.
2) Calculate the MVC value of filtered points using the MVC field via trilinear interpolation based on their coordinates.
3) Apply the cage deformation using the sampled MVC value to project points back to canonical space.
4) Infer the color and density of sampled points inside canonical space and apply volumetric rendering:
[Equation]
D i ff e r e n t V i e w s c a nn o n i c a l s c a l e r o t a t e
[Table]
where C ( · ) is the cage deformation that translates sampled points back to canonical space.
Experiments
To verify the generalization of cage deformation
[Table]
we evaluate CageNeRF on three tasks:
Geometry editing (Section 4.1).
In this task, we demonstrate the capability of CageNeRF in editing the geometry of the implicit object by deforming the radiance field.
This is achieved by either explicitly editing the cage shape or using the cage deformer module to generate novel cage via deformation transfer.
Deformation transfer is a task that takes two objects with different geometry properties (e.g., shape, structure and scale), and modifies these properties on the source to make it resemble the target.
In this task, CageNeRF is trained on the lego dataset used in NeRF [33] and a chair dataset generated in the same manner.
Neural animation (Section 4.2).
In this task, we evaluate the continuous deformation quality of
CageNeRF by comparing the results with targets animated using Linear Blend Skinning [26].
We use both synthesized objects and human videos as our benchmarks.
The synthesized 3D objects contain two animated characters: ‘Hulk’ and a blue whale.
Furthermore, we evaluate CageNeRF on real humans using Human 3.6M [19].
Pose reenactment (Section 4.3).
In this task, we generate a synthetic dataset using a static model of a humanoid robot, which only contains polygon mesh and texture without rigged animation.
This dataset contains the rendered 360 ◦ multi-view images with paired camera parameters.
We randomly sample posing SMPL models from the Surreal dataset [49] as the reenactment target.
We further provide the ablation study w.r.t.
the MVC sampling strategy.
Following the majority of previous literature [33, 37, 39, 50], we use PSNR/SSIM [53] and LPIPS [59] as evaluation metrics.
We refer readers to Appendix C for more details and results.
O u r s
L B S D r i v e n L B S D r i v e n
O u r s
A n i N e R F
O u r s
G r o un d T r u t h
A n i N e R F
O u r s
G r o un d
T r u t h
4.1 Geometry Editing
We demonstrate the effectiveness of deforming raidance field via explicit cage editing on the lego dataset as shown in Figure 6.
We edit the cage geometry to achieve the effect of stretching and rotating the bucket.
In our framework, deformation transfer can be applied to edit the geometric properties of the implicit 3D objects.
Our framework is first trained on an office chair in the canonical space, as a deformation source.
We randomly select objects from the ShapeNet dataset [7] as the deformation targets and then apply our cage deformer to edit the implicit geometry stored in the canonical radiance field.
As shown in Figure 5, our framework is capable of editing the implicit geometry of object in the canonical space by deforming the radiance field learned by our neural renderer module.
The deformed results resemble the selected target in aspects of size, shape, and scale.
These results demonstrate that our framework can edit the radiance field such that the deformed geometry resembles the target.
4.2 Neural Animation
Synthesized Object Animation.
To quantitatively determine the animation precision, we analyze the rendering quality of an animated sequence and compare it to the traditional linear blend skinning (LBS) animation method which operates on the explicit deformable mesh.
We train our neural renderer using multiview images of the object rendered at the starting frame.
To animate the object, we extract the animated mesh from the animation sequence and use it as the deformed geometry G ′ .
Our cage deformer then builds the MVC field using Equation 10.
We animate our object by applying the extracted meshes G ′ sequentially to our framework.
As is shown in Figure 7, our framework is capable of deforming the radiance field and generating continuous animation of objects.
CageNeRF also achieves nice render quality which is comparable to that driven by explicit animation methods.
As shown in Table 1, our framework achieves promising results in animating the neural radiance field when compared to explicit mesh deforming methods.
Human Animation.
We also evaluate on a real-world dataset (i.e., Human 3.6M), and compare with the state-of-the-art method AniNeRF [37] as well as other human sythesis methods [46, 54].
The
Human 3.6M dataset is captured by 4 synchronized cameras and has 2D and 3D joint annotations.
We first estimate the geometry proxy of the posing human using the SMPL model by optimizing the 3D joint error and 2D re-projection joint error.
Then, we utilize the geometry proxy as the “novel pose” and apply the operation in Figure 3 to generate a deformed cage with the matching pose.
Then we build the MVC field using this deformed cage and posing geometry proxy.
Since our model only stores the information inside the canonical space, CageNeRF can generalize to novel poses without
R ee n a c t m e n t
T a r g e t s
D e f o r m a t
i o n
R e s u l t s
[Table]
additional training.
This is in contrast to AniNeRF, which needs to optimize an additional blend weight network using the estimated geometry in order to synthesize novel pose results.
As shown in Figure 8 and Table
[Table]
the performance of our framework in synthesizing dynamic human body is comparable to AniNeRF across different subjects.
CageNeRF achieves slightly lower PSNR due to the inaccurate estimation of the geometry used in our cage deformer module.
Moreover, we employ SMPL model as the geometry proxy during cage deformation, where SMPL only models human body, while the performer in Human 3.6M dataset wears loosely fitted clothes which has complex nonrigid deformations during motion.
The inaccuracy in the geometry proxy further induces sampling displacement during volumetric rendering and thus impacts the performance of our framework.
Despite the fact that our framework only takes a coarse estimation of the object, it is still able to correctly deform and synthesize complex dynamic objects such as the human body.
4.3 Pose Reenactment
To analyze the generalization ability of CageNeRF, we apply our framework to tasks of pose reenactment and motion retargeting from humans to humanoid objects.
We keep the same training process as Section 4.1.
Afterward, we randomly sample posed SMPL models from the Surreal dataset as the target for deformation transfer.
Both the posed SMPL model and the optimized robot cage are fed into our cage deformer to calculate the cage deformation.
Finally, the barycentric interpolation is applied to deform the static robot model using Equation 8.
As is shown in Figure 9, although the structure of the source robot is notably different from the target human, the deformed pose and geometric structure correspond well to the given target, while keeping the source identity and texture.
4.4 Impact of the MVC Sampling Strategy
To evaluate the effectiveness of our anti-aliasing MVC sampling strategy, we generate an MVC field using different sampling strategies and qualitatively and quantitatively compare the rendering results.
As shown in Figure 10, the rendering result generated with grid-sampled MVC field displays inflated geometry and blurred texture, due to the inaccurate mapping between the canonical and the deformed space.
In contrast, the result rendered using our anti-aliasing sampled MVC field has notably sharper
[Figure]
[Table]
textures and a more accurate geometry structure.
As also shown in Table 3, the anti-aliasing sampling strategy significantly increases the render quality on all three evaluation metrics.
Limitations
Our approach achieves high quality results in deforming and animating implicit objects by combining cage-based deformation with neural radiance field.
However, there remains a few limitations of our approach we would like to address.
While the cage-based deformation is category-agnostic and easy to generalize under various settings, it is not suited for modeling complex and subtle deformations such as facial expressions.
Our work is a proof of the concept that explicit deformation can be applied to implicit neural representations.
We believe our framework is also compatible with other explicit deformation schemes, which may provide the extra capability in adapting to complex movements.
Since the color and density of the canonical radiance field are relatively stable, our framework is not suited for modeling dynamic objects with changing color or rendering objects under varying lighting conditions.
Recent works [44, 60] have developed novel techniques on factorizing basic color and density information into several decoupling elements such as surface normal, albedo, and light visibility, this further enables material editing and neural scene relighting.
It would be interesting to further improve and augment our framework to have editable material, color, and lighting by incorporating novel neural rendering algorithms and structures.
Conclusion
We propose a versatile framework for deforming and animating implicit field.
Comparing to previous methods, our framework excels in aspects of generalization, structural flexibility, and editing capability.
This is achieved by adopting a deformable cage as the universal constraint for our radiance field editing which does not require any data-dependent prior knowledge.
We demonstrate the effectiveness of our framework in the tasks of object animation, geometry editing, and deformation transfer.
Experimental results show that our framework is capable of animating and deforming arbitrary objects represented by implicit field, and produces results comparable to other state-of-the-art methods.
Acknowledgments and Disclosure of Funding
This work is supported by NSFC (62225112, 62101325, 61831015, 62201342, U19B2035), the
National Key Research and Development Program of China (2021YFE0206700), Shanghai Municipal
Science and Technology Major Project (2021SHZDZX0102), and CCF-Alibaba Innovative Research Fund For Young Scholars.
Authors would like to appreciate the Student Innovation Center of SJTU for the GPU support.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[Yes] , our work focuses on providing editing method for implicit 3D representations in a generalized manner, it is possible that someone utilizes our method to produce fake videos.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] We do not include theoretical results.
(b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We describe the implementation details in the supplementary material.
The data utilized is publicly available, and we will make our code publicly available.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] We discuss the training details in supplementary material.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See supplementary for detailed descriptions.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] We use existing datasets and cited them in our paper.
(b) Did you mention the license of the assets?
[No] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[Yes] We discuss this in the supplementary material.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[No] The data used in our work does not contain such information.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[No] Our work does not involve crowdsourcing as such, same as follows.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[No] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[No]
[Figure]
MVC defined by basis vertices v 1 , 2 , 3 are visualized through color.
By translating the basis vertices to novel positions point p is shifted to p ′ while maintaining its MVC.
[Figure]
Mean value coordinates (MVC) were first introduced in [9] as a way of representing points using convex combinations of vertices from a polygon.
It was later extended to work with 3D closed meshes [10, 24].
Mean value coordinates encode the position of a given point with respect to a set of predefined vertices.
The relationship between the MVC of a given point and the predefined vertices is analogous to that between a vector and its corresponding basis in linear algebra, i.e., the predefined vertices act as the basis in MVC space.
Figure 11 illustrates the cage-based deformation using MVC coordinates in the 2D scenario.
Given the predefined vertices { v 1 , v 2 , v 3 } , which act as the basis in the MVC space, we can express arbitrary points inside the enclosed triangle as linear combinations of these vertices.
For example, the midpoint of line segment v 1 v 2 can be expressed as 12 · v 1 + 12 · vand hence having the MVC of (0 . 5 , 0 . 5 , 0) .
In Figure 11, we visualize the MVC of points inside the triangle by using different colors, and provide the Cartesian plane with origin Ω as the reference coordinate system.
Compared to the left, basis vertices are translated to novel positions.
All points inside the enclosed triangle are shifted and thus have different coordinates in the Cartesian coordinate system.
Notice that while the position of an arbitrary point p is shifted to p ′ after the deformation,
its
MVC remain unchanged, i.e., p and p ′ share the same color.
In general, given an arbitrary point p inside the convex kernel K of a polygon Ω with n vertices, we can calculate the MVC by constructing a series of non-negative functions ϕ 1 , ϕ 2 , · · · , ϕ n : K → R , such that:
[Equation]
where v i is the i -th ordered vertex of Ω .
The MVC is defined by:
[Equation]
where α i is the angle ∠ v
i
pv i +1 , and r i ( p ) =
∥ v i − p ∥ .
To extend this case to the 3D scenario, given a closed polygon mesh Ω , its vertices ˆ v i ∈ Ω are first projected to a unit sphere with p at its center.
As shown in Figure 12, { ⃗ e i ,⃗ e j ,⃗ e k } are unit vectors from p to the projected vertices { v i , v j , v k } on the unit sphere.
The MVC of the point p takes the same form as Equation 12, except that ϕ
i is defined by:
[Equation]
where T is the spherical triangle enclosed by { v i , v j , v k } and µ i, T is given by:
[Equation] [Equation]
B Datasets
We train our framework on different datasets and evaluate CageNeRF on several tasks.
In the following section, we provide a detailed description of the employed datasets.
We will make the synthetic data publicly available.
For the task of Geometric Editing via Deformation Transfer , we use a synthetic dataset of an office chair, which act as the static object inside the canonical space, to train our neural renderer.
We also employ the sampled objects from ShapeNet [7] as deformation targets and comparison baseline to evaluate the editing results.
Our synthetic dataset of the office chair is created from a polygon mesh with colored texture using 3D render software.
The dataset contains around 1.2K 360 ◦ inward-facing multiview images with corresponding masks and camera parameters.
Some samples of the multiview images and masks are shown in Figure 13.
As for the camera parameters, we store the camera rotation
R ∈ R 3 × 3 , camera translation T ∈ R 1 × 3 , and the camera intrinsic parameters K ∈ R 3 × 3 .
Since we use the same camera for the entire render process, the intrinsic parameters are shared across all the images, while the rotation and translation vary with the rendering view.
[Figure]
In the experiment of Neural Animation , we use two types of datasets: 1) a synthetic dataset of animated objects driven by linear blend skinning, and 2) a real-world dataset, i.e., Human3.6M [19], which is often utilized as a benchmark for human body synthesis.
The synthetic datasets contain two characters, one contains a comic character Hulk, and the other contains a swimming blue whale.
The deformable meshes of both objects are fully rigged with corresponding skeletons and blend weights.
Both objects are capable of producing short animation clips using liner blend skinning as driving method.
Similar to the synthetic dataset used in Geometric Editing , we render 1.2K 360 ◦ inward-facing multiview images of the animatable objects at the starting frame.
The deformed polygon meshes of the following frames are also extracted as the animation target for our cage deformer module to generate new deformation fields.
The Human3.6M dataset was captured bysynchronized inward-facing cameras from different viewpoints.
Every section of this dataset captures a short clip of a posing performer with 2D and 3D joint annotations and per-frame masks.
For each frame in this clip, a SMPL [30] human model is estimated based on the annotations and masks for its position, rotation, and joint locations.
This estimated SMPL model, similar to the animated mesh in our synthetic dataset, is used in our cage deformer to generate deformations on the radiance field.
We provide additional results on synthesizing human body on different subjects in Figure 14.
A n i
N e R
F
O u r s
G r o un d
T r u t h
As for Pose Reenactment , we also create a synthetic dataset in the same manner as the office chair dataset used in Geometry Editing .
We use a static robot model with colored texture, instead of an office chair, to create this dataset.
We sample our reenactment targets from the Surreal [13] dataset.
C Implementation Details C.1 Training Configuration
We train our network on a single Nvidia RTX3090 GPU for 500 epochs, depending on the complexity of the geometry and texture.
The overall training process takes around 22 hours for a dataset with 1K multi-view images.
We take A DAM optimizer [27] with an exponential learning rate scheduler.
We set our initial learning rate to 5 e − 4 and our learning rate decays by a factor of 10 for everyepochs.
We implement our entire framework using P Y T ORCH [36] deep learning library.
It roughly takes 12G GPU memory to train our framework.
Our framework achieves the average test-time rendering speed of 2.071 seconds per frame with the resolution of 800 × 800 .
C.2 Network Structure
Different from the original NeRF [33], which optimizes two radiance fields, i.e., a coarse and a fine model, the neural render in our framework only optimizes a single multi-layer perceptron (MLP) network.
We sample 128 points along each camera ray during volumetric rendering.
We also provide a trainable latent code to compensate for the texture and normal variances of the object inside the canonical space.
The structure of the neural renderer is shown in Figure 15.
Our neural renderer has a similar structure to the original NeRF network except for the additional latent code input.
We take inspiration from [52, 14] and design our cage deformer in a similar encoder-decoder manner.
We use the PointNet++ [40] with multi-scale grouping strategy to extract the feature from 3D objects in our encoder.
For the decoder, we adopt an MLP network to predict the vertex-wise offset for the input cage base on the extracted feature.
D Additional Experiment Analysis D.1 Impact of the Network Structure
𝛾 𝒙 𝒙 ℓ 𝑐 𝑎 𝑛
𝛾 𝒙 𝒙ℓ 𝑐𝑎𝑛𝜎 𝛾 𝒅 (𝒅) መ 𝐶 ( 𝒙 , 𝒅 )
[Table]
[Table] [Table]
Hulk
Whale
H36M
NeRF
[Table]
As mentioned in our paper, we design our framework in a decoupled manner, the underlying structure of our neural renderer module, which learns the implicit representation of a static object in canonical space, can be replaced with other designs without affecting the integrity of our framework.
To validate this point, we prepare two neural renderers with different structures and carry out experiments for the task of neural animation.
Our neural renderer adopts a modified NeRF structure, as described in
Section C.2, with additional latent code to represent the appearance of the object inside the canonical space, while the other uses the vanilla version of NeRF.
Table 4 shows the impact of different network structures on the rendering quality across three datasets.
When using the vanilla NeRF structure as the neural renderer, the rendering quality slightly drops w.r.t.
PSNR.
The introduced additional latent code is effective in improving the rendering quality for both the synthetic and real-world datasets.
D.2 Impact of Mesh Accuracy
[Figure]
SDF Output
Distorted
Reference
Our cage-based deformation method utilizes a geometry proxy extracted from the learned radiance field as a 3D mask.
The quality of this geometry proxy affects the point sampling process during volumetric rendering.
To examine its influence on our framework, we apply deformation transfer to our canonical radiance field using the MVC field generated from the ground truth mesh and the mesh reconstructed by the signed distance function [50].
As shown in
Figure 16, while the reconstructed mesh has minor artifacts in terms of smoothness and geometric detail, the overall shape and structure match the ground truth.
The artifacts and dilation in certain areas such as the armrest causes blurred and distorted texture.
This result shows the accuracy of geometry proxy has a noticeable effect in rendering quality.
In the future, one possible direction to further improve the flexibility of CageNeRF is to learn geometry directly from multi-view images.
D.3 Functionality of Explicit Mesh
The explicit mesh extracted in our method has the following functionalities:
Enhance Deformation Capability.
While our framework supports explicit deformation by manually editing the shape of the canonical cage, we want to enhance the deformation capability further.
Neural
Cage [52] offers impressive performance with explicit mesh in tasks like deformation transfer and pose reenactment.
By incorporating an explicit mesh into our framework, we are able to transfer the capability of Neural Cage, i.e., a mesh-based method, to neural radiance field.
Increase Sampling Efficiency.
As mentioned in our paper Section 3.4, the canonical radiance field is sparsely occupied by the render target.
Restricting the sample points inside the vicinity space of the render target can effectively reduce the sample points down to 10% of the original number required in volume rendering.
Provide Robust Correspondence.
Our framework handles deformation by first learning a static radiance field inside the canonical space.
However, this canonical space is not always directly accessible.
For example, the Human 3.6M dataset only contains posing human subjects captured by multi-view cameras, the canonical space, i.e., “T-pose” images of the target is not given.
This requires first estimating the deformed cages of the targets in various poses and performing cage-based deformation backward to transform posing targets to canonical “T-pose”.
An explicit mesh can be used to generate correct MVC field which enables us to robustly establish this backward deformation.
Cage-based deformation can express the forward deformation correctly, i.e., from the canonical space to the deformed space, but not always vice versa.
In some cases, both Neural Cages and our cage deformer module will output a deformed cage with overlapping faces or collapsed vertices.
Naively taking the MVC value calculated in such deformed cages will induce artifacts, since points that reside in such regions cannot be correctly projected into the canonical space.
This property can be interpreted as: CBD using a cage with a self-overlapping structure is similar to applying a matrix that is not full rank, and hence not invertible.
We provide backward deformation results on SMPL model using cage with and without self-overlapping structures in.
Additionally, if the cage vertices are manually edited without overlapping faces and collapsed vertices, we can generate MVC field directly based on this deformed cage, i.e., without the need of an explicit mesh, since this cage deformation is invertible from deformed space to canonical space.
[Figure]
E Broader Impacts
As shown in our experiments, our work is capable of producing high-fidelity results of edited and animated 3D objects with 2D images as its input.
Our framework may find application in 3D animation, digital human creation, high fidelity 3D object editing.
Especially, the deformation method used in our framework is category-agnostic, our framework can be applied in cross-domain animation retargeting and reenactment.