CASA: Category-agnostic Skeletal Animal Reconstruction
[Figure]
Abstract
Recovering a skeletal shape from a monocular video is a longstanding challenge.
Prevailing nonrigid animal reconstruction methods often adopt a control-pointdriven animation model and optimize bone transforms individually without considering skeletal topology, yielding unsatisfactory shape and articulation.
In contrast,humans can easily infer the articulation structure of an unknown character byassociating it with a seen articulated object in their memory.
Inspired by this fact,we present CASA, a novel category-agnostic articulated animal reconstructionmethod.
Our method consists of two components, a video-to-shape retrieval process and a neural inverse graphics framework.
During inference, CASA first findsa matched articulated shape from a 3D character assets bank so that the input videoscores highly with the rendered image, according to a pretrained image-languagemodel.
It then integrates the retrieved character into an inverse graphics frameworkand jointly infers the shape deformation, skeleton structure, and skinning weightsthrough optimization.
Experiments validate the efficacy of our method in shapereconstruction and articulation.
We further show that we can use the resultingskeletal-animated character for re-animation.
Introduction
Recovering the shape, articulation and dynamics of animals from images and videos is a long-standingtask in computer vision and graphics.
Achieving this goal will enable new future applications for3D modeling and reanimation of characters captured with mobile devices.
Nevertheless, accuratelyreasoning about the geometry and kinematics of an animal in the wild remains an ambitious problemfor several reasons: (1) partial visibility of the captured animal; (2) variability of shape acrossdifferent animal categories; (3) ambiguity of unknown kinematics.
Take the elephant in Fig.
1 as anexample – to faithfully modeling it requires: 1) hallucinate the unobserved ( e.g ., occluded, back-side)region, 2) recovering the unique shape and scale, and 3) predicting its kinematic structure.
Remarkable progress has been made to address the aforementioned challenges by either exploitingricher sensor configurations, such as multi-camera cage [10, 16] and depth sensors [33, 41, 34], oralternatively by making strong assumptions specific to a particular object category like humans [29,
[Figure]
46, 45, 63, 47] and quadruped animals [66, 67, 68].
However, these assumptions greatly limit theapplicability of reconstruction systems, which fail to generalize to animals in the wild.
We aim to reconstruct an arbitrary articulated animal using a monocular video casually captured inthe wild.
A few recent works so far demonstrate promising results toward tackling this challengingproblem [60, 61, 56, 62, 37].
However, they often impose non-realistic assumptions on articulation,such as control point driven deformation [60, 61, 56, 62] or free-form deformation [37, 41].
As aresult, they fall short of the goal of modeling skeletal characters that can be realistically re-animatedin downstream applications.
Furthermore, there remains significant improvement space for the qualityof the inferred animal shape.
In this work, we propose CASA , a novel solution for category-agnostic animal reconstructionin the wild.
CASA jointly estimates an arbitrary animal’s 3D shape, kinematic structure, riggingweight, and articulated poses of each frame from a monocular video (see Fig. 1).
Unlike previousnonrigid reconstruction works [60, 61, 56], we exploit a skeleton-based articulation model as wellas forward kinematics, ensuring the realism of the resulting skeletal shape (§ 3.1).
Specifically, wepropose two novel components: 1) a video-to-shape retrieval process, as well as 2) a neural inversegraphics framework.
Given an input video, CASA first finds a template shape from a 3D characterassets bank so that the input video scores highly with the rendered image, according to a pretrainedimage-language model [40] (§ 3.2).
Using the retrieved character as initialization, we jointly optimizebone length, joint angle, 3D shape, and blend skinning weight so that the final outputs are consistentwith visual evidence, i.e ., input video (§ 3.3).
Another issue that hinders the study of articulated 3D reconstruction is that the existing datasets [12,27, 3] lack realistic video footage and ground-truth labels across dynamic objects of different classes.
To overcome this challenge, we introduce a photorealistic synthetic dataset PlanetZoo, which isgenerated using the physical-based rendering [7] and rich 3D assets from a zoo simulator [30].
PlanetZoo consists of a realistic rendering of 251 categories of articulated animals.
Importantly,
PlanetZoo also provides ground-truth shape, skeleton, joint angles, and rigging, allowing us toevaluate category-agnostic 4D reconstruction holistically (§ 4).
We validate our proposed approach on both PlanetZoo and the real-world DAVIS dataset.
Ourexperiments demonstrate that CASA can recover fine shape details and realistic skeleton topology,handles a wide variety of shapes, and adapts well to unseen categories.
In addition, CASA enjoys greatgeneralizability since all the evidence neural networks are pre-trained from external datasets withoutfine-tuning.
Finally, CASA recovered a skeletal-animatable character that is readily compatible fordownstream tasks such as re-animation, rendering and simulation.
Related Work
Parametric shape reconstruction.
Parameteric models have been proposed to efficiently modelbody [29], hand [44], face [25] and animal [68] 3D-shapes and body poses.
Recent works use suchmodels on human body capture [65, 32, 2, 6, 57, 38, 36] and animal reconstruction [68, 1, 67].
Whilemodel-based approaches show great results in reconstructing objects of a certain category with fixed
[Figure]
topology, it is challenging to apply those models to categories that are hard to distinguish or haveinsufficient 3D data, especially for animals.
Category-specific reconstruction.
Many recent works [32, 17, 26, 22, 53, 47, 56, 26, 20] study theproblem of predicting deformable objects from image collections or videos within a specific category.
These methods heavily rely on strong category priors.
CMR [17] uses keypoint annotations and masksto predict the shape deformation.
U-CMR [56] replaces the keypoints in CMR with category-specificcanonical shapes.
VMR [26] extends the input of CMR from images to videos, as well as adopts anas-rigid-as-possible (ARAP) constraint for per-frame shape reconstructions based on video inputs.
Besides the strict requirement on the category of input objects, most of these methods are trained onsingle-view images, limiting their ability to recover high-fidelity mesh details.
[Rebuttal: ACSM [20]infers the articulation and pose of template models corresponding to input videos by predicting acanonical surface mapping.]
Category-agnostic nonrigid shape reconstruction.
Classic approaches use Non-Rigid Structurefrom Motion (NRSfM) [4, 21, 50] for category-agnostic nonrigid shape reconstruction.
They usea collection of images capturing the same object from multiple views with 2D correspondences.
Given that the long-range correspondences are hard to obtain, they often require manual annotations.[Rebuttal: WIM [35] learns re-poseable shape representation for objects of different categories, butdepending on multi-view observations.]Recent works adopts learning-based approaches by eitherdynamic reconstruction [24, 41] or inverse graphics optimization [60, 61], LASR [60] and ViSER [61]optimize articulated shapes from monocular videos and achieve promising reconstruction results.
However, these methods suffer from incorrect shape prediction when certain parts are occluded orinvisible in the input video sequence.3D reconstruction dataset A plethora of synthetic datasets [51, 64, 15, 43] and interactive simulatedenvironment [48, 49, 9, 55] aimed for 3D reconstruction have been proposed in recent years.
The
ShapeNet [8] dataset provides a benchmark for common static 3D object shape reconstruction.
Among all of these datasets, only a few aim for dynamic object reconstruction [27].
Given that thesynthetic dataset has a large domain gap to the real-world settings, photo-realistic rendered samplesare strongly preferred.
To this end, we propose a photorealistic synthetic dataset PlanetZoo to studythe dynamic animal reconstruction problem.
Our dataset contains high-fidelity rendered images andcovers a wide range of animal categories.
Category-agnostic Skeletal Animal Reconstruction
We present CASA, a novel category-agnostic skeletal animal reconstruction framework.
The input toour method are RGB frames from a monocular video { I t } 1 ...T , object segmentation masks { M t } 1 ... Tand optical flow maps { F t } 1 ...T computed from each pair of consecutive frames.
From these inputs,we aim to recover an animal shape s 0 and its articulated deformed shape s t at each time t .
Our method exploits the skeletal articulation model and forward kinematics to ensure the realism ofthe resulting skeletal shape (§ 3.1).
Our method consists of two phases.
In the retrieval phase (§ 3.2),
CASA finds a template character from an existing asset bank based on the similarity to the inputvideo in the deep embedding space using a pre-trained encoder.
The retrieved template is fed intothe neural inverse graphics phase as initialization(§ 3.3).
Finally, we jointly reason the final shape,skeleton, rigging and the articulated pose of each frame through an energy minimization framework.
Fig
[Figure]
[Figure]
Skeletal parameters.
We exploit a stretchable bone-based skeletal model for our deformable shapeparameterization.
This articulated model consists of three components: 1) a triangular mesh consistingof a set of vertices and shapes in the canonical pose, describing the object’s shape; 2) a set of bonesconnected by a kinematic tree.
Each one has a bone length parameter and associated rigging weightover each vertex; 3) a joint angle describing the relative transformation between each adjacent bone.
To summarize, a deformed shape s t at time t can be represented as:
[Equation]
(1) where v i is the vertex position, f j represents a triangle face parameterized as a tuple of threevertex indices, w i is the rigging weight constrained in a K-dimensional simplex: ∆ K = 117 (cid:110) ( w 0 , . . .
, w K ) ∈ R K | (cid:80) Kk =
w k = 1 and w k ≥
(cid:111) ;
b k is the scalar bone length for each boneand q tk is the joint angle for each bone at a time t , represented as an unit quaternion in the SO (3)space.
Note that all the variables except joint angles are shared across time.
Fig.
3 gives an overview of our skeletal model.
It is grounded by the nature of many articulatedobjects.
Unlike the commonly used control-point based articulation [33, 61, 60], our model reflects theconstraints imposed by bones and joints, leading to more natural articulated deformation.
Comparedto category-specific parametric models [29, 68], it is more flexible and generalizable.
Forward kinematics.
We use the forward kinematic model [23] to compute the transformation ofeach bone along the kinematic tree.
Specifically, the rigid transformation of a bone is uniquely definedby the joint angles and bone length scales of the bone itself and its ancestors along the kinematic tree.
Given a bone k , the rigid transform between its own frame and the root can be computed throughrecursively applying the relative rigid transformation along the chain:
[Equation]
where T tk , T t pa ( k ) is the transformation of bone k and its parent node at frame t .
q ans(k) , b ans(k) are thejoint angles and bone lengths from all the ancestors of target bone k .
T tk, pa ( k ) ( q tk , b k ) is the relativerigid transform between the bone and its parent frame, consists of a joint center translation along zaxis decided by the bone length
b k e z = [0 , 0 , b k ] and a rotation around the joint center R ( q tk ) .
Linear blend skinning.
Given the rigid transformation of each bone, we adopt linear blend skinning(LBS) to compute the transform of each point.
Specifically, the deformation of each point v isdecided by a linear composing the rigid transforms of bones through its skinning weight: v ti = 136 (cid:80)
Kk =
w i,k T tk v i where w i,k is the rigging weight.
T tk is the transformation of bone k at frame t asdefined in Eq. 2.
For simplicity, we omit its dependent variables.
Stretchable bones.
The aforementioned nonrigid deformation assumes the independent shape andbone lengths.
Naturally, the shape should deform accordingly as bone length changes.
To model sucha relationship, we exploit a stretchable bone deformation model [14].
For each bone, we computean additional scaling transform along the bone vector direction based on its bone length change andapply this scaling transform to any points associated with this bone.
Fig.
3 provides an illustration ofstretching.
For mathematical details on stretching, we refer readers to our supplementary document.
Stretchable bones bring a two-fold advantage: (a) it allows us to model shape variations within thesame topological structures; (b) it makes the topological structure adjustable by “shrinking” the bones,e.g. a quadrupedal animal can be evolved into a seal-like skeletal model through optimization.
3.2 Video-to-shape retrieval
Directly optimizing all skeletal parameters defined in Eq.
1 is challenging due to the variabilityand highly structured skeletal parameterization.
Many deformable objects share similar topologystructures, albeit with significant shape differences.
Motivated by this, we propose initializing skeletalshapes in a data-driven manner by choosing the best matching template from an asset bank.
Videos and skeletal shapes are in two different modalities; hence establishing a similarity measure ishard.
Inspired by the recent success of image-language pretraining models for 3D [54, 31], we utilizerealistic rendering and pretraining image embedding models [40] to bridge this gap.
Specifically,we first pre-render video footages for each character in the asset bank through physical-basedrendering [7].
The environment lighting, background, articulated poses, and camera pose for eachvideo are randomized to gain diversity and robustness.
We then extract the image embedding featuresfor each video using CLIP [40], which is a language-image embedding model that is pretrained ona large-scale image-caption dataset.
It captures the underlying semantic similarity between imageswell despite the large appearance and viewpoint difference.
This is particularly suitable for retrievingshapes with similar kinematic structures, as the kinematics of animals is often related to semantics.
During inference, we extract embedding features from a given input video and measure the cosinesimilarity between the input video and the rendered video of each object in the embedding space.
Wethen select the highest-scoring articulated shape as the retrieved character.
3.3 Neural inverse graphics via energy minimization
We expect our final output skeletal shape to 1) be consistent with the input video observation; 2)reflect prior knowledge about articulated shape, such as symmetry and smoothness.
Inspired by this,we formulate our as an energy minimization problem.
Energy formulation.
We exploit visual cues from each frame including segmentation mask { M t }and optical flow { F t } through off-the-shelf neural networks [52, 60].
We also incorporate two typesof prior knowledge, including the motion smoothness for joint angles q tk of bones across frames, aswell as the symmetry constraint for the per-vertex offset v i at the reference frame.
In particular, let
I =
( { I t } , { M t } , { F t } ) be the input video frames and corresponding visual cues, and { s t } be thepredicted shapes of all frames, we formulate the energy for minimization as:
[Equation]
where E cue measures the consistency between the articulated shape and the visual cues; E smoothpromotes smooth transitions overtime; E symm encodes the fact that many articulated objects aresymmetric.
The three energy terms complement each other, helping our model capture motions of theobject according to visual observations, as well as constraining the deformed object to be natural andtemporally consistent.
We describe details of each term below.
Visual cue consistency.
The visual cue consistency energy measures the agreement between therendered maps of the articulated shape and 2D evidence (flow and silhouettes) from videos.
We use adifferentiable renderer [28] to generate projected object silhouettes { M t } .
Additionally, we projectvertices of predicted shapes at two consecutive frames to the camera view and compute the projected2D displacement to render optical flow, following the prior work [60].
We leverage PointRend
[Figure]
[Table] [Figure] [Table] for object segmentation and volumetric correspondence net [59] for flow prediction.
The energymeasures the difference between the rendered and inferred cues in ℓ 2 distance:
[Equation]
[Equation]
where β is the trade-off weight; π flow ( s t ) is the rendered the 2D flow map for each pixel given thedeformed shape s t ; π seg ( s t ) is the rendered the object mask.
Similar to previous inverse renderingwork [61], the object-camera transform is encoded as the root node transform.
Motion smoothness.
This energy term encodes that motions of animals should be smooth andcontinuous across frames.
We impose a constraint to ensure that there is little difference in jointangles of one bone from two consecutive frames should be slight.
We implement this by computingthe multiplication of the joint quaternion at the current frame and the inverse of the joint quaternionat the next frame, which should be close to an identity quaternion q = (0 , 0 , 0 , 1) :
[Equation]
Symmetry offset.
This term encourages the resulting shape in canonical space to be symmetric at thereference frame.
It is inspired by the fact that most animals in the real world are symmetric if puttingthem into a certain canonical pose (e.g. ‘T-pose‘ for bipedal animal).
In practice, we enforce thisproperty at the canonical shape (where the joint angles are all zero).
Specifically, we calculate thechamfer distance for measuring the similarity between the set of vertices { v i } under the canonicalshape and its reflection:
[Equation]
where H is the Householder reflection matrix.
3.4 Inference
We reason the skeletal shape by minimizing the energy defined in Eq.
Our optimization variablesinclude the vertex position v i at the canonical pose, the rigging weight w i ,
bone length
b k as well asthe joint angles q t k .
All the variables except joint angles are shared across time, and joint angles areoptimized per frame.
Initialization.
We initialize the vertex position, bone length, and the rigging weight v i , b k , w i ofcanonical shape using the retrieved template character.
The skeleton tree structure of this character isalso taken as the basis of our skeletal parameterization.
We initialize all the joint rotations as a unitquaternion. [Figure]
Reparameterization.
Optimizing each vertex position offers flexibility yet might lead to undesirablemesh due to the lack of regularization.
We use a neural displacement field to re-parameterize thevertex deformation to incorporate regularization, such as smooth and structured vertex deformation.
We use a coordinate-based multi-layer perceptron (MLP) to define this displacement field V .
Inaddition, we incorporate a global scale scalar u in our framework to handle the shape misalignmentbetween our initialization and the target.
The position of each vertex is defined as:
[Equation]
7) where u v i is the scaled position at the canonical pose, V θ ( u p i ) represents the vertex offset parameterized by θ , and p ′
i is updated position for vertex i .
During inference, instead of directly optimizingv i , we will only optimize the parameters of the displacement network.
This displacement field reparameterization allow us to smoothly deform canonical shape during inference.
In practice we find thisimplicit regularization compares favorably over explicit smoothness terms such as as-rigid-as-possibleand laplacian smoothness.
Optimization.
The energy function is fully differentiable and can be optimized end-to-end.
We use
Adam optimizer [18] to learn all the optimization variables.
We adopt a scheduling strategy to avoidgetting stuck at a local minimum.
Specifically, we firstly optimize the mesh scaling factor based onsilhouette correspondences.
We then jointly update the bone length scale, joint angles, and the neuraloffset field by minimizing the energy function defined in Eq. 3.
PlanetZoo Dataset
Training and validating animatable 3D shape reconstruction require to have a large amount of nonrigid action sequences with ground truth 3D shapes and articulation structures.
To achieve this goal,we construct a synthetic non-rigid dataset PlanetZoo, which consists of a large number of animatedanimals with textures and skeletons.
We extracted those characters from the zoo simulator game
Planetzoo [30] where hundreds of different animals are available.
Data Generation.
In Planetzoo, the shapes are represented as a mesh.
We use Cobra-tools [11] toextract those meshes along with their skeleton.
With the extracted mesh and skeleton, we generate RGB maps, segmentation masks and optical flow for each frame.
All data is obtained using Blender.
Assets.
To set a diverse environment, we randomly set the background with public released HDRIpictures, providing a background light, and set floor textures with materials got from ambientCG 1 .
Toreduce the gap between synthetic data and real-world data, we randomly set the location of the lightsource along with its strength to simulate different environments in real-world situations.
Articulation.
To generate realistic action sequences, we randomly change camera locations betweenevery 12 frames, resulting in a constant view-point change following the animal.
The camera isallowed to rotate at a certain angle, ranging from 15 ◦ to 22 .
5 ◦ .
In order to obtain animated animalsequences, for every 12 frames, 8 bones are selected from the skeleton tree with a transformationattached.
The angle value of rotation for each bone is sampled from a uniform distribution.
By doingthis, we are able to cover the whole action space, providing diverse action sequences.
Rendering.
We generate segmentation masks, optical flow, depth map, camera parameters, andphoto-realistic RGB images using Vision Blender [7].
The physical-based renderer provided by
Blender is capable of showing details such as fur, making the rendering results more realistic.
For
//ambientcg.
[Figure]
each animal in the dataset, 180 frames are rendered.
Fig.
5 depicts a snapshot of the assets andrendering images from our dataset, demonstrating the diversity and quality.
Experiments
In this section, we first introduce our experimental setting (§ 5.1).
We then compare CASA against acomprehensive set of articulated baselines in various reconstruction and articulation metrics on bothsimulation (§ 5.2) and real-world datasets (§ 5.3).
Finally, we demonstrate our inferred shape can beused for downstream reanimation tasks (§ 5.4).
5.1 Experimental Setup
Benchmarks.
We validate our proposed method on two datasets.
Our proposed photorealisticrendering dataset PlanetZoo as well as a real-world animal video dataset DAVIS[39], containingmultiple real animal videos with mask annotations.
For dataset PlanetZoo, we choose 24 animals in249 total animals as our testing and the rest as validation and training.
The testing dataset includesdiverse articulated animals from unseen categories, including multiple quadrupeds, bipeds, birdscategories, as well as unseen articulated topology such as pinnipeds.
Metrics.
We measure the reconstruction quality by Intersection over Union (IOU) and Chamferdistance, as well as skinning distance, joint distance and re-animation quality on PlanetZoo.
1) IOU measures the volumetric similarity between two shapes.
We first voxelized reference shapeand predicted shape into occupancy grids and then calculate the IOU ratio.2)
Chamfer Distance computes the bidirectional vertex-to-vertex distances between the referenceshape and the generated shape.
3) Joint is the symmetric Chamfer Distance between joints.
We evaluate CD-J2J following [58].
Given a test shape, we compute the Euclidean distance between each joint and its nearest joint in thereference shape, then divide it by the total joint number.
4) Skinning distance measures the similarity between the skinning weights.
We first extract verticesassociated with each joint using skinning weight.
For each pair of joints from GT and prediction,we calculate the Chamfer distance between their associated vertices.
Finally, we exploit the Jonker-
Volgenant algorithm to find the minimum distance matching between prediction and reference.
5) Re-animation measures how well we can re-pose an articulated shape to a target shape.
Specifically,
we minimize the Chamfer Distance between the reference shape and the predicted shape by optimizingthe joint angles of each bone for skeletal shape, or the rigid transform for control point-based shape.
We consider this a holistic metric that jointly reflects the quality of shapes, skinning, and skeleton.
Baselines.
We compare with state-of-the-art approaches for single monocular-video articulatedshape reconstruction including LASR [60], ViSER [61], and BANMo [62], which are methodsthat reconstruct articulated shapes on monocular videos.
Similar to our method, they utilize 2Dsupervision of videos for training including segmentation masks and optical flows.
We downloadthe open-source version of them from github.
For the input data, we use our ground truth silhouetteeither in PlanetZoo dataset or DAVIS[39] dataset.
We follow the optimization scripts in their codeand get the baseline results.
3D reconstruction from a monocular video inevitably brings scale ambiguities.
To alleviate the issueof scale differences, for every predicted shape among all the competing baselines, we conduct a linesearch to find the optimal scale that maximizes the IOU between the reference and predicted shape.
5.2 Results on PlanetZoo
We present quantitative results on PlanetZoo in Tab. 1.
CASA achieves higher performance on metricsmIOU and mCham, demonstrating a shape with higher fidelity.
Our approach also outperforms othercompeting methods in reanimation, demonstrating the superior performance of holistic articulatedshape reasoning and potential for downstream reanimation tasks.
CASA compares less favorably to
LASR in the joint distance and skinning quality.
We conjecture this is due to additional degrees offreedom benefits brought by joint control point articulation.
Fig. 4 and Fig.
6 depicts a few qualitative examples of all competing algorithms on PlanetZoo.
Weshow the reconstructed mesh from both the camera view and the opposite view.
The figure shows that
CASA can recover accurate shape across various view angles under partial observation.
In contrast,baselines fail to reconstruct reliable 3D mesh when the objects are partially observed.
In addition,
CASA also produces meshes with higher fidelity and local details in the visible region.
5.3 Real-world Reconstruction
We demonstrate qualitative results of the competing algorithms on the real-world animal video dataset
DAVIS [39] in Fig 7 and Fig. 4.
This figure shows that both our method and baselines can get goodreconstruction results from the camera perspective.
But both two baselines fail to reconstruct thoseunseen parts.
In contrast, CASA can reliably reason the shape and articulation in the unseen regions,thanks to its symmetry constraints, skeletal parameterization, and 3D template retrieval.
5.4 Reanimation
We now show how the reconstructed articulated 3D shape can be retargeted to new poses.
Given aninferred skeletal shape and the target GT shape at a different articulated pose, we apply an inversekinematic to compute the articulated pose to reanimate the inferred shape.
Specifically, we optimizethe articulated transforms such that the reanimated mesh is as close to the target in Chamfer distance.
Joint quaternions are optimized for skeletal mesh, and the articulation transforms are optimized forthe control-point-based method [60].
Fig.
8 shows a comparison between the retargeted meshesof our methods and LASR on PlanetZoo, with a GT target mesh as a reference.
We observe thatour retargeted meshes look realistic and accurate and preserve geometric details.
Despite havingmore degree-of-freedoms in reanimation, LASR fails to produce realistic retarget results.
Tab.
1reports a quantitative comparison in chamfer distance between the retargeted and the reference mesh,demonstrating our approach outperforms competing methods by a large margin.
5.5 Limitations
While demonstrating superior performance on animal reconstruction, our method has a few remainingdrawbacks.
Firstly, the source asset bank restricts the diversity of the retrieved articulation topology.
Bone length optimization partially alleviates such limitations through “shrinking” bones.
Still, itcannot add new bones to the kinematic tree, e.g., we cannot create a spider from a quadrupedaltemplate.
Secondly, our method so far does not impose the constraint that bones are inside the mesh.
We plan to tackle such challenges in the future.
Conclusion
In this paper, we propose CASA, a novel category-agnostic animation reconstruction algorithm.
Ourmethod can take a monocular RGB video and predict a 4D skeletal mesh, including the surface,skeleton structure, skinning weight, and the joint angle at each frame.
Our experimental resultsshow that the proposed method achieves state-of-the-art performances in two challenging datasets.
Importantly, we demonstrate that we could retarget our reconstructed 3D skeletal character andgenerate new animated sequences. Checklist For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No] we will release
the code and data upon acceptance
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No](d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] We use eight GTX 2080 Ti andRTX A6000 GPUs.4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]upon acceptance
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using
curating
[Yes] BADJA
is
open
sourced
; PlanetZoo has commercial license,following prior practices in our community [42, 5, 13], we will ask users to purchasethe game before accessing the dataset.
(e) Did you discuss whether the data you are using/curating contains personally identi-
fiable information or offensive content?
[Yes] Both datasets do not contain personal
identifiable information.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]