CASA: Category-agnostic Skeletal Animal Reconstruction
[Figure]
Abstract
Recovering the skeletal shape of an animal from a monocular video is a longstanding challenge.
Prevailing animal reconstruction methods often adopt a control-point driven animation model and optimize bone transforms individually without considering skeletal topology, yielding unsatisfactory shape and articulation.
In contrast, humans can easily infer the articulation structure of an unknown animal by associating it with a seen articulated character in their memory.
Inspired by this fact, we present CASA , a novel C ategory-
A gnostic S keletal A nimal reconstruction method consisting of two major components: a video-to-shape retrieval process and a neural inverse graphics framework.
During inference, CASA first retrieves an articulated shape from a 3D character assets bank so that the input video scores highly with the rendered image, according to a pretrained language-vision model.
CASA then integrates the retrieved character into an inverse graphics framework and jointly infers the shape deformation, skeleton structure, and skinning weights through optimization.
Experiments validate the efficacy of CASA regarding shape reconstruction and articulation.
We further demonstrate that the resulting skeletalanimated characters can be used for re-animation.
Introduction
Recovering the shape, articulation, and dynamics of animals from images and videos is a longstanding task in computer vision and graphics.
Achieving this goal will enable numerous future applications for 3D modeling and reanimation of animals.
Nevertheless, accurately reasoning about the geometry and kinematics of animals in the wild remains an ambitious problem for three reasons: 1) partial visibility of the captured animal, 2) variability of shape across different categories, and 3) ambiguity of unknown kinematics.
Take the camel in Fig.
1 as an example – to faithfully model it requires: 1) hallucinating the unobserved ( e.g ., occluded, back-side) region, 2) recovering its unique shape and scale, and 3) predicting its kinematic structure.
Remarkable progress has been made in addressing the aforementioned challenges by either exploiting richer sensor configurations ( e.g ., multi-view cameras [11, 17] or depth sensors [41, 42, 50]), or by making strong class-specific assumptions ( e.g ., humans [34, 55–57, 78, 80] or quadruped animals [83– 85]).
However, these assumptions greatly limit the applicability of reconstruction systems, which fail to generalize to animals in the wild.
We aim to reconstruct arbitrary articulated animals using monocular videos casually captured in the wild .
Recent works [74–76, 70, 45] demonstrate promising results.
However, they often impose non-realistic assumptions on articulation, such as control point driven deformation [74–76, 70] or freeform deformation [45, 50].
As a result, they fall short of the goal of modeling skeletal characters that can be realistically re-animated in downstream applications.
Furthermore, there remains significant improvement space for the quality of the inferred animal shape.
In this work, we propose CASA , a novel solution for C ategory- A gnostic S keletal A nimal reconstruction in the wild.
CASA jointly estimates an arbitrary animal’s 3D shape, kinematic structure, rigging weight, and articulated poses of each frame from a monocular video (Fig. 1).
Unlike existing nonrigid reconstruction works [74, 75, 70], we exploit a skeleton-based articulation model as well as forward kinematics, ensuring the realism of the resulting skeletal shape (§ 3.1).
Specifically, we propose two novel components: a video-to-shape retrieval process and a skeleton-based neural inverse graphics framework.
Given an input video, CASA first finds a template shape from a 3D character assets bank so that the input video scores highly with the rendered image, according to a pretrained language-vision model [47] (§ 3.2).
Using the retrieved character as initialization, we jointly optimize bone length, joint angle, 3D shape, and blend skinning weight so that the final outputs are consistent with visual evidence, i.e ., input video (§ 3.3).
Another issue that hinders the study of animal reconstruction is that the existing datasets [4, 13, 31] lack realistic video footage and ground-truth labels across different dynamic animals.
To address it, we introduce a photo-realistic synthetic dataset PlanetZoo , which is generated using the physicalbased rendering [7] and rich simulated 3D assets [35].
PlanetZoo is large-scale and consists ofdifferent articulated animals.
Importantly, PlanetZoo provides ground-truth 3D shapes, skeletons, joint angles, and rigging, allowing evaluation of category-agnostic 4D reconstruction holistically (§ 4).
We evaluate CASA on both PlanetZoo and the real-world dataset DAVIS [46].
Experiments demonstrate that CASA recovers fine shape and realistic skeleton topology, handles a wide variety of animals, and adapts well to unseen categories. Additionally, we showed that CASA reconstructs a skeletal-animatable character readily compatible with downstream re-animation and simulation tasks.
In summary, we make the following contributions: 1) We propose a simple, effective, and generalizable video-to-shape retrieval algorithm based on a pretrained CLIP model. We introduce a novel neural inverse graphics optimization framework that incorporates stretchable skeletal models for category-agnostic articulated shape reconstruction. 3) We present a large-scale yet diverse skeletal shape dataset PlanetZoo.
Related work
Category-specific nonrigid reconstruction.
Parametric model (template) is a classic approach for category-specific nonrigid reconstruction[40, 18, 2, 78, 77, 36, 32, 54] . Templates are built for various objects including human body [34, 1], hand [53], face [29, 3], and animals [85, 2].
Template-free methods [19, 30, 25, 64, 57, 70, 23, 44, 69] study the problem of predicting nonrigid objects directly for a specific category. However, these methods heavily rely on strong category priors such as key-points annotations [19], canonical shapes [10], temporal consistency constraints [30], or canonical surface mapping [23], making them hard to generalize to category-agnostic setting.
Category-agnostic nonrigid reconstruction.
Classic approaches [5, 24, 60, 43] utilize Non-Rigid
Structure from Motion (NRSfM) for non-rigid reconstruction, which requires point correspondences and strong priors on shape or motion.
However, NRSfM does not work well for monocular videos in the wild, in which obtaining long-range correspondences is hard.
Recent works adopt learning-based approaches via either dynamic reconstruction [27, 50] or inverse graphics optimization [74–76]. However, these methods do not reason skeletal kinematics and suffer from incorrect shape prediction when certain parts are occluded or invisible. Animatable shape.
Humans and animals undergo skeleton-driven deformation. A skeletal shape often consists of two parts: a surface mesh representation used to draw the character and a skeleton [Figure] structure to control the motion.
Each skeleton is a hierarchical set of bones.
Each bone has associated with a portion of vertices (skinning).
The bone’s transformation is determined through a forward kinematics process.
As the character is animated, the bones change their transformation over time.
Linear Blend Skinning (LBS) [37, 28] is a standard way for modeling skeletal deformation, which deforms each vertex of the shape based on a linear combination of bone transformations. For improvement, multi-weight enveloping [68, 38] is used to overcome the issue of shape collapse near joints when related bones are rotated or moved. Dual-quaternion blending (DQB) [20] adopts quaternion rotation representation to solve the artifacts in blending bone rotations, and STBS [15] extends LBS to include the stretching and twisting of bones. Recent works [74–76] employs LBS for modeling the motion of shapes recovered from video clips.
However, these methods do not enforce a skeleton-based forward kinematic structure. Hence their recovered animated shapes are not interpretable and cannot be directly used in skeletal animation and simulation pipeline. Language-vision approaches. Self-supervised language-vision models have gone through rapid advances in recent years [62, 66, 48] due to their impressive generalizability.
The seminal work CLIP [66] learns a joint language-vision embedding using more than 400 billion text-image pairs. The learned representation is semantically meaningful and expressive, thus has been adapted to various downstream tasks [82, 71, 49, 79, 65]. In this work, we adopt CLIP in the retrieval process.
3D reconstruction dataset. A plethora of synthetic 3D datasets [61, 81, 16, 52] and interactive simulated 3D environments [58, 59, 9, 67] have been proposed in recent years. ShapeNet [8] provides a benchmark for common static 3D object shape reconstruction.
Among all of these datasets, only a few aim for dynamic object reconstruction [31].
Given that the synthetic dataset has a large domain gap to real-world settings, photo-realistic rendered samples are strongly preferred.
To this end, we propose a photo-realistic synthetic dataset PlanetZoo to study the dynamic animal reconstruction problem.
PlanetZoo contains high-fidelity assets and covers a wide range of animal categories.
Category-agnostic skeletal animal reconstruction
CASA aims to reconstruct various dynamic animals in-the-wild from monocular videos.
It takes as input the RGB frames { I t } 1 ...T from a monocular video, object masks { M t } 1 ...T , and optical flow maps { F t } 1 ...T computed from consecutive frames.
From these inputs, we aim to recover an animal shape s 0 and its articulated deformed shape s t at each time t . Fig.
2 demonstrates an overview of our approach. Our method exploits the skeletal articulation model and forward kinematics to ensure the realism of the resulting skeletal shape (§ 3.1).
Our method consists of two phases.
In the retrieval phase (§ 3.2), CASA finds a template character from an existing asset bank based on the similarity to the input video in the deep embedding space using a pre-trained encoder.
The retrieved template is fed into the neural inverse graphics phase as initialization(§ 3.3).
Finally, we jointly reason the final shape, skeleton, rigging and the articulated pose of each frame through an energy minimization framework.
[Figure] 3.1 Articulation model
Skeletal parameters.
We exploit a stretchable bone-based skeletal model for our deformable shape parameterization ,as shown in Fig.
This articulated model consists of three components: 1) a triangular mesh consisting of a set of vertices and shapes in the canonical pose, describing the object’s shape; 2) a set of bones connected by a kinematic tree.
Each one has a bone length parameter and associated rigging weight over each vertex; 3) a joint angle describing the relative transformation between each adjacent bone.
To summarize, a deformed shape s t at time t can be represented as:
[Equation]
(1) where v i is the vertex position, f j represents a triangle face parameterized as a tuple of three vertex indices, w i is the rigging weight constrained in a K-dimensional simplex: ∆ K = (cid:110) ( w 0 , . . .
, w K ) ∈ R K | (cid:80) Kk =
w k = 1 and w k ≥
(cid:111) ;
b k is the scalar bone length for each bone and q tk is the joint angle for each bone at a time t , represented as a unit quaternion in the SO (3) space.
Note that all the variables except joint angles are shared across time.
Our skeletal model is grounded by the nature of many articulated objects.
Unlike the commonly used control-point based articulation [41, 75, 74], our model reflects the constraints imposed by bones and joints, leading to more natural articulated deformation.
Compared to category-specific parametric models [34, 85], it is more flexible and generalizable.
Forward kinematics.
We use the forward kinematic model [26] to compute the transformation of each bone along the kinematic tree.
Specifically, the rigid transformation of a bone is uniquely defined by the joint angles and bone length scales of the bone itself and its ancestors along the kinematic tree.
Given a bone k , the rigid transform between its own frame and the root can be computed by recursively applying the relative rigid transformation along the chain:
[Equation]
where T tk , T t pa ( k ) is the transformation of bone k and its parent node at frame t .
q ans(k) , b ans(k) are the joint angles and bone lengths from all the ancestors of target bone k .
T tk, pa ( k ) ( q tk , b k ) is the relative rigid transform between the bone and its parent frame, consists of a joint center translation along z axis decided by the bone length
b k e z = [0 , 0 , b k ] and a rotation around the joint center R ( q tk ) .
Linear Blend Skinning (LBS).
Given the rigid transformation of each bone, we adopt LBS to compute the transform of each point.
Specifically, the deformation of each point v is decided by a linear composing the rigid transforms of bones through its skinning weight: v ti = (cid:80) Kk =
w i,k T tk v i where w i,k is the rigging weight.
T tk is the transformation of bone k at frame t as defined in Eq. 2.
For simplicity, we omit its dependent variables.
Stretchable bones.
The aforementioned nonrigid deformation assumes independent shape and bone lengths.
Naturally, the shape should deform accordingly as bone length changes.
To model such a relationship, we exploit a stretchable bone deformation model [15].
For each bone, we compute an additional scaling transform along the bone vector direction based on its bone length change and apply this scaling transform to any points associated with this bone.
Fig.
3 provides an illustration of stretching.
For mathematical details on stretching, we refer readers to our supplementary document.
Stretchable bones bring a two-fold advantage: 1) it allows us to model shape variations within the same topological structures; 2) it makes the topological structure adjustable by “shrinking” the bones, e.g., a quadrupedal animal can be evolved into a seal-like skeletal model through optimization. Reparameterization. Optimizing each vertex position offers flexibility yet might lead to undesirable mesh due to the lack of regularization.
We use a neural displacement field to re-parameterize the vertex deformation to incorporate regularization, such as smooth and structured vertex deformation. We use a coordinate-based multi-layer perceptron (MLP) to define this displacement field V . In addition, we incorporate a global scale scalar u in our framework to handle the shape misalignment between our initialization and the target. The position of each vertex is defined as:
[Equation]
where u v i is the scaled position at the canonical pose, V θ ( u p i ) represents the vertex offset parameterized by θ , and
v ′
i is the updated position for vertex i .
During inference, instead of directly optimizing v i , we will only optimize the parameters of the displacement network.
This displacement field reparameterization allow us to smoothly deform canonical shape during inference.
In practice we find this implicit regularization compares favorably over explicit smoothness terms such as as-rigid-as-possible and laplacian smoothness.
3.2 Video-to-shape retrieval
Directly optimizing all skeletal parameters defined in Eq.
1 is challenging due to the variability and highly structured skeletal parameterization.
Many deformable objects share similar topology structures, albeit with significant shape differences.
To address it, we propose initializing skeletal shapes in a data-driven manner by choosing the best matching template from an asset bank.
Videos and skeletal shapes are in two different modalities; hence establishing a similarity measure is hard.
Inspired by the recent success of language-vision pretraining models for 3D [66, 39], we utilize realistic rendering and pretraining image embedding models [47] to bridge this gap.
Specifically, we first pre-render video footage for each character in the asset bank through a physically based renderer [7].
The environment lighting, background, articulated poses, and camera pose for each video are randomized to gain diversity and robustness.
We then extract the image embedding features for each video using CLIP [47], which is a language-vision embedding model that is pretrained on a large-scale image-caption dataset.
It captures the underlying semantic similarity between images well despite the large appearance and viewpoint difference.
This is particularly suitable for retrieving shapes with similar kinematic structures, as the kinematics of animals is often related to semantics.
During inference, we extract embedding features from a given input video and measure the L distance between the input video and the rendered video of each object in the embedding space.
We then select the highest-scoring articulated shape as the retrieved character. Please refer to the supplementary material for implementation details.
3.3 Neural inverse graphics via energy minimization
We expect our final output skeletal shape to 1) be consistent with the input video observation; 2) reflect prior knowledge about articulated shapes ,such as symmetry and smoothness.
Inspired by this, we formulate ours as an energy minimization problem.
Energy formulation.
We exploit visual cues from each frame including segmentation mask { M t } and optical flow { F t } through off-the-shelf neural networks [63, 74].
We also incorporate two types of prior knowledge, including the motion smoothness for joint angles q tk of bones across frames, as well as the symmetry constraint for the per-vertex offset v i at the reference frame.
In particular, let
I =
( { I t } , { M t } , { F t } ) be the input video frames and corresponding visual cues, and { s t } be the predicted shapes of all frames, we formulate the energy for minimization as:
[Equation]
where E cue measures the consistency between the articulated shape and the visual cues; E smooth promotes smooth transitions overtime; E symm encodes the fact that many articulated objects are symmetric.
The three energy terms complement each other, helping our model capture motions of the object according to visual observations, as well as constraining the deformed object to be natural and temporally consistent.
We describe details of each term below. [Table]
Visual cue consistency.
The visual cue consistency energy measures the agreement between the rendered maps of the articulated shape and 2D evidence (flow and silhouettes) from videos.
We use a differentiable renderer [33] to generate projected object silhouettes { M t } .
Additionally, we project vertices of predicted shapes at two consecutive frames to the camera view and compute the projected 2D displacement to render optical flow, following the prior work [74].
We leverage PointRend
[22] for object segmentation and volumetric correspondence net [73] for flow prediction.
The energy measures the difference between the rendered and inferred cues in ℓ 2 distance:
[Equation]
5) where β is the trade-off weight; π flow ( s t ) is the rendered the 2D flow map for each pixel given the deformed shape s t ; π seg ( s t ) is the rendered object mask.
Similar to previous inverse rendering work [75], the object-camera transform is encoded as the root node transform.
Motion smoothness.
This energy term encodes that motions of animals should be smooth and continuous across frames.
We impose a constraint to ensure that there is little difference in joint angles of one bone from two consecutive frames should be slight.
We implement this by computing the multiplication of the joint quaternion at the current frame and the inverse of the joint quaternion at the next frame, which should be close to an identity quaternion q = (0 , 0 , 0 , 1) :
[Equation]
where ◦ is the quaternion composition operator.
Symmetry offset.
This term encourages the resulting shape in canonical space to be symmetric at the reference frame.
It is inspired by the fact that most animals in the real world are symmetric if putting them into a certain canonical pose (e.g., ‘T-pose‘ for bipedal animals).
Following previous works [74], we enforce this property at the canonical shape (where the joint angles are all zero).
Specifically, we calculate the chamfer distance for measuring the similarity between the set of vertices { v i } under the canonical shape and its reflection:
[Equation] [Equation]
where H is the Householder reflection matrix.
3.4 Inference
We reason the skeletal shape by minimizing the energy defined in Eq. 4.
Our optimization variables include the vertex position v i at the canonical pose, the rigging weight w i ,
bone length
b k as well as the joint angles q tk .
All the variables except joint angles are shared across time, and joint angles are optimized per frame.
Initialization.
We initialize the vertex position, bone length, and the rigging weight v i , b k , w i of canonical shape using the retrieved template character.
The skeleton tree structure of this character is also taken as the basis of our skeletal parameterization.
We initialize all the joint rotations as a unit quaternion.
Optimization.
The energy function is fully differentiable and can be optimized end-to-end.
We use
Adam optimizer [21] to learn all the optimization variables.
We adopt a scheduling strategy to avoid getting stuck at a local minimum.
Specifically, we first optimize the mesh scaling factor based on silhouette correspondences.
We then jointly update the bone length scale, joint angles, and the neural offset field by minimizing the energy function defined in Eq. 4. [Figure]
PlanetZoo dataset
Benchmarking 4D animal reconstruction requires a large amount of nonrigid action sequences with ground truth articulated 3D shapes.
Towards this goal, we construct a synthetic dataset PlanetZoo consisting of hundreds of animated animals with textures and skeletons from different categories. Appendix Fig.
12 depicts a snapshot of the assets and rendering images from our dataset, demonstrating the diversity and quality.
Data generation.
We extracted animal meshes from the zoo simulator Planetzoo [35].
Cobratools [12] are used to extract those meshes along with their skeleton.
With the extracted mesh and skeleton, we further render RGB maps, segmentation masks and optical flow for each frame using
Blender.
Assets.
To set a diverse environment, we set the background with random HDRI pictures for environmental lighting, and set floor textures with random materials from ambientCG 2 .
To reduce the gap between synthetic and real, we set the location of the light source along with its strength to simulate different environments in real-world situations.
Camera.
To generate realistic action sequences, we randomly change camera locations between every 12 frames, resulting in a constant view-point change following the animal.
The camera is allowed to rotate at a certain angle, ranging from 15 ◦ to 22 .
5 ◦ .
Articulation. In order to obtain animated animal sequences, for every 12 frames, 8 bones are selected from the skeleton tree with a transformation attached.
The angle value of rotation for each bone is sampled from a uniform distribution.
By doing this, we are able to cover the whole action space, providing diverse action sequences.
Rendering.
We generate silhouettes, optical flow, depth map, camera parameters, and RGB images using Vision Blender [7].
The physically based renderer is capable of showing fine details such as fur, making the rendering results more realistic.
For each animal in the dataset, 180 frames are rendered.
Experiments
In this section, we first introduce our experimental setting (§ 5.1).
We then compare CASA against a comprehensive set of articulated baselines in various reconstruction and articulation metrics on both simulation (§ 5.2) and real-world datasets (§ 5.3).
Finally, we demonstrate our inferred shape can be used for downstream reanimation tasks (§ 5.4).
5.1 Experimental setup
Benchmarks.
We validate our proposed method on two datasets.
Our proposed photorealistic rendering dataset PlanetZoo as well as a real-world animal video dataset DAVIS[46], containing multiple real animal videos with mask annotations.
For PlanetZoo, we choose 24 out of 249 total animals for testing and use the rest for validation and training.
The testing dataset includes diverse articulated animals from unseen categories, including multiple quadrupeds, bipeds, birds categories, as well as unseen articulated topology such as pinnipeds.
Metrics.
We measure the reconstruction quality by Intersection over Union (IOU) and Chamfer distance, as well as skinning distance, joint distance and re-animation quality on PlanetZoo. [Figure] [Figure]
1) mean Intersection Over Union(mIOU) measures the volumetric similarity between two shapes.
We voxelized the reference and the predicted shape into occupancy grids and calculate the IOU ratio.
2) mean Chamfer Distance(mCham) computes the bidirectional vertex-to-vertex distances between the reference and the predicted shape.
3) Joint is the symmetric Chamfer Distance between joints.
We evaluate CD-J2J following [72]. Given a
predicted shape, we compute the Euclidean distance between each joint and its nearest joint in the reference shape, then divide it by the total joint number.
4) Skinning distance measures the similarity between the skinning weights.
We first extract vertices associated with each joint using skinning weight.
For each pair of joints from GT and prediction, we calculate the Chamfer distance between their associated vertices.
Finally, we exploit the JonkerVolgenant algorithm to find the minimum distance matching between prediction and reference.
5) Re-animation measures how well we can re-pose an articulated shape to a target shape.
Specifically, we minimize the Chamfer Distance between the reference shape and the predicted shape by optimizing the joint angles of each bone for skeletal shape, or the rigid transform for control point-based shape.
We consider this a holistic metric that jointly reflects the quality of shapes, skinning, and skeleton.
Baselines.
We compare with state-of-the-art approaches for monocular-video articulated shape reconstruction ,including LASR [74], ViSER [75], and BANMo [76] 3 .
Similar to our method, they utilize 2D supervision of videos for training including segmentation masks and optical flows.
We download the open-source version of them from GitHub.
For the input data, we use our ground truth silhouette either in PlanetZoo dataset or DAVIS[46] dataset.
We follow the optimization scripts in their code and get the baseline results.
3D reconstruction from a monocular video inevitably brings scale ambiguities.
To alleviate the issue of scale differences, for every predicted shape among all the competing baselines, we conduct a line search to find the optimal scale that maximizes the IOU between the reference and the predicted shape. [Figure]
5.2 Results on PlanetZoo
We present quantitative results on PlanetZoo in Tab. 1.
CASA achieves higher performance on metrics mIOU and mCham, demonstrating a shape with higher fidelity.
Our approach also outperforms other competing methods in reanimation, demonstrating the superior performance of holistic articulated shape reasoning and the potential for downstream reanimation tasks.
CASA compares less favorably to LASR in the skinning quality.
We conjecture this is due to additional degrees of freedom benefits brought by joint control point articulation.
Fig. 4 and Fig.
5 depict a few qualitative examples of all competing algorithms on PlanetZoo.
We show the reconstructed mesh from both the camera view and the opposite view.
The figure shows that
CASA can recover accurate shape across various view angles under partial observation.
In contrast, baselines fail to reconstruct reliable 3D mesh when the objects are partially observed.
In addition,
CASA also produces meshes with higher fidelity and local details in the visible region.
5.3 Real-world reconstruction
We demonstrate qualitative results of the competing algorithms on the real-world animal video dataset
DAVIS [46] in Fig 6 and Fig. 4.
This figure shows that both our method and baselines can get good reconstruction results from the camera perspective.
But both two baselines fail to reconstruct those unseen parts.
In contrast, CASA can reliably reason the shape and articulation in the unseen regions, thanks to its symmetry constraints, skeletal parameterization, and 3D template retrieval.
5.4 Reanimation
We now show how the reconstructed articulated 3D shape can be retargeted to new poses.
Given an inferred skeletal shape and the target GT shape at a different articulated pose, we apply an inverse kinematic to compute the articulated pose to reanimate the inferred shape.
Specifically, we optimize the articulated transforms such that the reanimated mesh is as close to the target in Chamfer distance.
Joint quaternions are optimized for skeletal mesh, and the articulation transforms are optimized for the control-point-based method [74].
Fig.
7 shows a comparison between the retargeted meshes of our methods and LASR on PlanetZoo, with a GT target mesh as a reference.
We observe that our retargeted meshes look realistic and accurate and preserve geometric details.
Despite having more degrees of freedom in reanimation, LASR fails to produce realistic retarget results.
Tab.reports a quantitative comparison in chamfer distance between the retargeted and the reference mesh, demonstrating our approach outperforms competing methods by a large margin. [Table] [Table]
[Table]
[Table] 5.5 Ablation study
We provide an ablation study to demonstrate the efficacy of each design choice in our framework.
Energy terms.
In Tab. 2, we ablate different terms of our energy function.
Specifically, we consider the mask consistency, flow consistency, symmetry and smoothness regularization separately.
We find that the mask part is crucial for the performance, while the optical flow part also improves the framework results.
Removing the symmetry offset term results in performance degradation since this term plays a vital role in regularizing the neural offset, which can alleviate issues brought by the ambiguity of single-view monocular video.
The smooth term leads to better qualitative results.
Optimization.
We compare different optimization settings in Tab. 3.
The results show that: 1) using neural offset field is superior to per-vertex offset or not deforming shape (CASA vs. w/o offset vs. w/o disp.
field); stretchable skeleton helps (CASA vs. w/o skeleton); removing shape scaling step leads to performance degradation (CASA vs. w/o scaling).
Retrieval strategies.
In Tab. 4, we test different retrieval strategies.
Our result demonstrates that
CLIP is the preferred backbone for retrieval, most likely due to training with significantly richer semantic information than ImageNet pretrained models.
Initialization.
We test different initialization strategies for skinning weights in Tab. 5.
We replace the rigging from the retrieved animal by using k-means for initializing weights.
The comparison in the table shows that high-quality rigging weight initialization is essential for good shape predictions.
The results of sphere initialization also confirm the necessity of the proposed retrieval phase.
Stretchable bone.
Fig.
8 shows qualitative results with/without stretchable bone parameterization.
Compared to merely deforming each vertex, stretchable bone deformation ensures global consistency and smoothness.
As noted in the figure, without stretchable bones, we see discontinuity at the nose of the animal and a mismatch between the lower and upper mouth.
5.6 Limitations
[Figure] While demonstrating superior performance in animal reconstruction, our method has a few remaining drawbacks.
Firstly, the source asset bank restricts the diversity of the retrieved articulation topology.
Bone length optimization partially alleviates such limitations through “shrinking” bones.
Still, it cannot add new bones to the kinematic tree, e.g., we cannot create a spider from a quadrupedal template.
Secondly, our method so far does not impose the constraint that bones are inside the mesh.
We plan to tackle such challenges in the future.
Conclusion
In this paper, we propose CASA, a novel category-agnostic animation reconstruction algorithm.
Our method can take a monocular RGB video and predict a 4D skeletal mesh, including the surface, skeleton structure, skinning weight, and the joint angle at each frame.
Our experimental results show that the proposed method achieves state-of-the-art performances in two challenging datasets.
Importantly, we demonstrate that we could retarget our reconstructed 3D skeletal character and generate new animated sequences.
Acknowledgements.
The authors thank Vlas Zyrianov and Albert Zhai for their feedback on the writing. The project is partially funded by the Illinois Smart Transportation Initiative STII-21-07. We also thank Nvidia for the Academic Hardware Grant. A Additional Results
We provide additional qualitative results on animals from PlanetZoo in Fig. As shown in the figure, CASA well recovers the shape and topology of target animals from input videos. The sample of ostrich in this figure illustrates the limitations of CASA: 1) partial observation from monocular video leads to ambiguity on animal poses.
The predicted shape fits well with the target from the camera view. However, from an alternative view, we can find that the wings of the prediction are of a wrong pose; 2) we do not impose a constraint on bones that they should be inside the mesh, resulting in joints in the head becoming outside the shape. In order to demonstrate that CASA is able to adjust the topology of the initial shape, we provide the evolution of predictions for a seal in Fig. Although the retrieved animal is a quadrupedal animal (otter), CASA gradually warps the predicted shapes to the target seal, whose topology is significantly different from quadrupeds.
[Figure]
[Figure]
We minimize the energy function using Adam optimizer [21] with β 1 = 0 . 9 and β 2 = 0 . We optimize the framework for 200 epochs in total.
A scheduling strategy is adopted. We optimize a scaling factor for the first 60 epochs by minimizing the mask term. This factor is initialized by aligning the bounding boxes between the rendering mask of the initial shape and the ground-truth mask. For the rest 140 epochs, we jointly optimize the bone length scale, joint angles, and the neural offset field.
The learning rate is set to 5 e − 2 for the first stage and 4 e − 3 for all parameters in the second stage, except for the mesh scale and neural offset, which are set to 1 e − 3 . The trade-off weights for the mask term, flow term, smoothing term, and symmetry term are empirically set to 1 e 4 , 1 e 6 , 1 e 6 , and 1 e 4 , respectively. The neural offset field is parameterized by a four-layer MLP. C Retrieval Result
Fig.
11 shows some retrieval results. Input animals are of various colors, skinning textures, and poses, together with different backgrounds and lighting conditions in their corresponding videos. Still, our method can retrieve animals from the database with a similar topology to the input, as CLIP [66] captures high-level semantic information and avoids distractions brought by different appearances and background environments. The sample of the seal at the bottom right is a failure case, as the two animals have different typologies.
However, as shown in Fig. 10, CASA is able to bridge the gap between retrieved and target animals. D Stretchable Bone Formulation To incorporate stretchable bone parameterization into our linear blend skinning model, we have to re-formulate the skinning equation. The original formulation for linear blend skinning is defined as:
[Equation]
(8) where v ti and v i are the position of vertex i at frame t and the rest pose, respectively. w i,k is the rigging weight. T tk is the transformation of bone k at frame t .
Since the bone transformation T tk is rigid, we can replace it by a translation and a rotation.
In particular, given the head positions of each bone before and after deformation (which can be calculated by performing transformations according to forward kinematics), we can define the skinning model as:
[Equation]
where c k and c tk are the head positions of bone k before and after deformation. R tk is the rotation matrix decomposed from the rigid transformation T t k . Simply adding a scaling term into the skinning model can result in shape explosion, especially for vertices located near tails of bones, as illustrated in [15]. Following the solution proposed in [15], we define endpoint weight functions
e k ( p ) for all bones, to control shape deformation in scaling.
With the functions, the new skinning model is defined as:
[Equation]
where s k = ( b k − 1)
( d i − c i ) is the full stretching vector at the tail of bone d i , b k is the bone scaling factor.
Each vertex p i should have a unique stretching vector corresponding to this bone, which is decided by the endpoint weight function e k
( p i ) . Typically the value of endpoint weight functions increases from 0 to 1 along the rest vector ( d k − c k ) of each bone. We define these functions by projecting each vertex v i onto the nearest point of each bone, calculating the fraction of where it falls between the head and tail of the bone as the weight:
[Equation]
[Figure]
where c k and d k are the head and tail positions of bone k ,
proj k
( v i ) is the projection of v i onto the bone. E Template-based Baseline We compare CASA with an additional template-based baseline in Tab. 6.
In order to ensure a fair comparison, we only incorporate quadruped animals in the PlanetZoo testset in experiments since ACFM is a category-specific method.
The results show that CASA outperforms ACFM without pre-training on a large-scale database. [Table] F Optimization Framework Comparison We compare CASA-retrieval + LASR with the entire CASA pipeline.
For LASR, rather than using the whole coarse-to-fine optimization steps, we utilized only the final step.
The quantitative comparison can be found in table 7.
[Table] G Efficiency We conduct a comparison on the efficiency between CASA and the other two state-of-the-art methods, LASR [74] and ViSER [75].
We test their average running time for animals in the testing set of
PlanetZoo, using one RTX 2080Ti.
The results are shown in Tab. 8.
Our method is approximately 3 × faster than LASR and 5 × faster than ViSER, demonstrating that our method is highly efficient.
This is natural as CASA do not need to train any complex neural networks.
[Table]
H Detailed Implementation of Retrieval
We denote the input monocular video containing an animal to be reconstructed as { I t } t =1 ,...,T 1 . The 3D asset bank for retrieving includes N categories of animals. For each animal i , the asset bank includes a 3D mesh M i , a skeleton S i and a realistic rendering video { I t } t =1 ,...,T 2 . In our retrieval stage, our goal is to search the animal with the best similarity to the input animal in the asset bank, and the output should be its 3D shape s j .
In short, our retrieval stage learns a mapping function from 2D video to 3D mesh.
Since the input and output of retrieval are in different modalities, the target mapping function is not easy to learn.
We propose to measure the similarity of the input video { I t } and all rendering videos in the asset bank using CLIP, obtain the most similar rendering video { I rt } , and take the corresponding animal mesh M r as the retrieval result.
To be specific, given the input video and any animal shape in the asset bank, we utilize pre-trained
CLIP to encode all frames of the input video and the photo-realistic rendering :
[Equation]
where { I t }} t =1 ,...,T 1 is the input video, s j is the j -th animal shape, g CLIP is the image embedding network of the CLIP model and π ( s j , q v ) is the photo-realistic rendering of the articulated shape s j at a randomized skeletal poses q v .
T 1 is the length of the input video.
V is the number of randomized skeletal pose.
Note that we render only one frame for each pose.
With the corresponding encoding features { F input t } and { F jv } , we first calculate the similarity D between the t -th input frame and the v -th skeletal pose for the j -th shape by:
[Equation]
With the similarity defined above, We are able to retrieve the animal category r in the asset bank that is the most similar with the input animal, which is formulated as:
[Equation]
We then obtain the 3D animal shape s r together with its corresponding skeleton and skinning weight from the asset bank, which is used for initializing the following optimization process.
I Initialization Strategy of CASA
We provide details of our initialization strategy for CASA.
In the skeletal optimization stage of CASA, optimizing parameters include a displacement field, the skinning weight, joint rotation angles for each bone, a root transformation, and bone length scales.
During initialization, the displacement field is defined by a Multi-Layer Perception (MLP) with random network parameters. The skinning weight is set according to the retrieval results. Bone length scales are all set to 1. The joint angles are set to T-pose.
The root transformation is represented as a combination of rotation and translation.
For the convergence of skeletal optimization, the root transformation should ensure the consistency of the mesh in the camera coordinate with the ground-truth.
Instead of estimating such transformation in world coordinate, we propose to predict camera parameters for each video, and fix the root transformation as an identical one.
In practice, for synthetic videos in which the camera parameters are available, we directly use them as initialization.
For videos where camera parameters are not available, we adopt a naive strategy to predict the camera parameters.
Specifically, we first optimize the camera intrinsic and focus point offset from the origin by minimizing the mask loss between rendering and ground-truth.
In each iteration, we randomly sample a point as the location of the camera from a sphere with its center at the origin and radius R set according to the animal scale.
When meeting the convergence criteria, we fix the camera intrinsic and focus point offset.
We then pick camera locations sampled before with the Top-10 lowest mask loss values.
For each location, we randomly sample points in its nearby regions as new camera locations, calculate the corresponding mask loss values, and select the location with the lowest value as our final camera location. J More information on PlanetZoo We provide the names of all categories in PlanetZoo in Tab. 9. We also show an overview figure of
PlanetZoo in Fig. [Table] female [Table]
male [Table] female [Table]
juvenile
[Table] male amazonian giant centipede [Table]
female american [Table] juvenile american bison male arctic [Table]
female arctic wolf juvenile arctic wolf male [Table] female [Table]
juvenile [Table]
female
[Table]
juvenile bactrian camel male bairds tapir female bairds tapir juvenile bairds
[Table]
male bengal [Table] female bengal tiger juvenile bengal tiger male binturong female binturong juvenile binturong male black wildebeest female black wildebeest juvenile black wildebeest male boa constrictor bongo female bongo male bonobo female bonobo juvenile bonobo male bornean orangutan female bornean orangutan juvenile bornean orangutan male brazilian salmon pink tarantula brazilian wandering spider capuchin monkey female capuchin monkey juvenile capuchin monkey male cassowary female cassowary juvenile cassowary male cheetah female cheetah juvenile cheetah male chinese pangolin female chinese pangolin juvenile chinese pangolin male clouded leopard female clouded leopard juvenile clouded leopard male common ostrich female common ostrich
juvenile common ostrich male common warthog female common warthog juvenile common warthog male cuviers dwarf caiman female cuviers dwarf caiman juvenile cuviers dwarf caiman male dall sheep female dall sheep juvenile dall sheep male death adder dhole female dhole juvenile dhole male diamondback terrapin dingo female dingo juvenile dingo male eastern blue tongued lizard eastern brown snake formosan black bear female formosan black bear juvenile formosan black bear male galapagos giant tortoise female galapagos giant tortoise juvenile galapagos giant tortoise male gemsbok female gemsbok juvenile gemsbok male gharial female gharial juvenile gharial male giant anteater female giant anteater juvenile giant anteater male giant burrowing cockroach giant desert hairy scorpion giant forest scorpion giant leaf insect giant otter female giant otter juvenile giant otter male giant panda female giant panda juvenile giant panda male gila monster golden poison frog goliath beetle goliath birdeater goliath frog gray wolf female gray wolf juvenile gray wolf male greater flamingo female greater flamingo juvenile greater flamingo male green iguana grey seal female grey seal juvenile grey seal male grizzly bear female grizzly bear juvenile grizzly bear male himalayan brown bear female himalayan brown bear juvenile himalayan brown bear male hippopotamus female hippopotamus juvenile hippopotamus male indian elephant female indian elephant juvenile indian elephant male indian peafowl juvenile indian peafowl male indian rhinoceros female indian rhinoceros juvenile indian rhinoceros male jaguar female jaguar juvenile jaguar male japanese macaque female japanese macaque juvenile japanese macaque male king penguin female king penguin juvenile king penguin male koala female koala juvenile koala male komodo dragon female komodo dragon juvenile komodo dragon male lehmanns poison frog lesser antillean iguana llama female llama juvenile llama male malayan tapir female malayan tapir juvenile malayan tapir male mandrill female mandrill juvenile mandrill male mexican redknee tarantula nile monitor juvenile nile monitor male nyala female nyala juvenile nyala male okapi female okapi juvenile okapi male plains zebra female plains zebra juvenile plains zebra male polar bear female polar bear juvenile polar bear male proboscis monkey female proboscis monkey juvenile proboscis monkey male pronghorn antelope female pronghorn antelope juvenile pronghorn antelope male puff adder pygmy hippo female pygmy hippo juvenile pygmy hippo male red eyed tree frog red kangaroo female red kangaroo juvenile red kangaroo male red panda juvenile red panda male red ruffed lemur female red ruffed lemur juvenile red ruffed lemur male reindeer female reindeer juvenile reindeer male reticulated giraffe female reticulated giraffe juvenile reticulated giraffe male ring tailed lemur juvenile ring tailed lemur male sable antelope female sable antelope juvenile sable antelope male saltwater crocodile female saltwater crocodile juvenile saltwater crocodile male siberian tiger female siberian tiger juvenile siberian tiger male snow leopard female snow leopard juvenile snow leopard male spotted hyena female spotted hyena juvenile spotted hyena male springbok female springbok juvenile springbok male sun bear female sun bear juvenile sun bear male thomsons gazelle female thomsons gazelle juvenile thomsons gazelle male titan beetle west african lion female west african lion juvenile west african lion male western chimpanzee female western chimpanzee juvenile western chimpanzee male western diamondback rattlesnake western lowland gorilla female western lowland gorilla juvenile western lowland gorilla male yellow anaconda
[Table]