P ROTO R ES : P ROTO -R ESIDUAL N ETWORK FOR P OSE A UTHORING VIA L EARNED I NVERSE K INEMATICS
Anonymous authors
Paper under double-blind review
A BSTRACT
Our work focuses on the development of a learnable neural representation of human pose for advanced AI assisted animation tooling.
Speciﬁcally, we tackle the problem of constructing a full static human pose based on sparse and variable user inputs ( e.g. locations and/or orientations of a subset of body joints).
To solve this problem, we propose a novel neural architecture that combines residual connections with prototype encoding of a partially speciﬁed pose to create a new complete pose from the learned latent space.
We show that our architecture outperforms a baseline based on Transformer, both in terms of accuracy and computational efﬁciency.
Additionally, we develop a user interface to integrate our neural model in Unity, a real-time 3D development platform.
Furthermore, we introduce two new datasets representing the static human pose modeling problem, based on high-quality human motion capture data, which will be released publicly along with model code.
I NTRODUCTION
Modeling human pose and learning pose representations have received increasing attention recently due to their prominence in applications, such as computer graphics and animation (Harvey et al., 2020;
Xu et al., 2020); immersive augmented reality (Facebook Reality Labs, 2021;
Capece et al., 2018;
Lin & O’Brien, 2019; Yang et al., 2021); entertainment (McDonald, 2018; Xpire, 2019); sports and wellness (Rosenhahn et al., 2008; Kim et al., 2021) as well as human machine interaction (Heindl et al., 2019; Casillas-Perez et al., 2016; Schwarz et al., 2014) and autonomous driving (Kumar et al., 2021).
In the gaming industry, state-of-the-art real-time pose manipulation tools, such as CCD (Kenwright, 2012), FABRIK (Aristidou & Lasenby, 2011) or FinalIK (RootMotion, 2020), are popular for rapid execution and rely on forward and inverse kinematics models deﬁned via non-learnable kinematic equations.
Inverse kinematics (IK) is the process of computing the internal geometric parameters of a kinematic system resulting in the desired conﬁguration (e.g. global positions) of system’s joints (Paul, 1992).
Forward kinematics (FK) refers to the use of the kinematic equations to compute the positions of joints from speciﬁed values of internal geometric parameters.
While mathematically accurate, non-learnable IK models do not guarantee that the underconstrained solutions derived from sparse constraints (e.g. positions of a small subset of joints) result in plausible human poses.
In this paper we develop a neural modeling approach to reconstruct full human pose from a sparse set of constraints supplied by a user, in the context of pose authoring and game development.
We bridge the gap between skeleton-aware human pose representation based on IK/FK ideas and the neural embedding of human pose.
Our approach effectively implements a learnable model for skeleton IK, mapping desired joint conﬁguration into predictions of skeleton internal parameters (local rotations), learning the statistics of natural poses using datasets derived from high-quality motion capture (MOCAP) sequences.
The approach, which we call ProtoRes, models the semantics of joints and their interactions using a novel prototypical residual neural network architecture.
Inspired by prototypical networks, which showed that one semantic class can be represented by the prototype (mean) of a few examples (Snell et al., 2017), we extend it using a multi-block residual approach: the ﬁnal pose embedding is a mean across embeddings of sparse constraints and across partial pose predictions produced in each block.
We show that in terms of the pose reconstruction accuracy,
ProtoRes outperforms existing gaming industry tools such as FinalIK, as well as out-of-the-box machine-learning solution based on Transformer (Vaswani et al., 2017), which also happens to betimes less effective in terms of training speed than the proposed architecture.
Finally, we develop user-facing tools that integrate learned ProtoRes pose representation in the Unity game engine, providing convincing qualitative examples of solutions to the problem of the AI assisted human pose authoring.
The examples reveal that at the qualitative level, getting traditional workﬂows to behave the way ProtoRes does would require one to use many techniques in tandem, including IK,
FK, layered animation pose libraries, along with procedural rigs encoding explicit heuristics.
The process would be highly labor-intensive even for an experienced user while the results would still be of variable ﬁdelity depending on the skill of the user.
This is because traditional rigs have no inductive bias towards realistic poses and only allow the user to explore a limited linear latent space deﬁned by uniform interpolation of a heuristic constraint system.
ProtoRes forms a foundation that allows any junior or indie/studio user to bypass these existing complexities and create entirely new workﬂows for meaningfully exploring learned latent space using a familiar yet far more powerful way.
We believe that our model and tools will help speed up the animation process and alleviate game artist animation skill requirements thus simplifying and democratizing game development.
1. B ACKGROUND
We consider the full-body pose authoring animation task depicted in Fig.
The animator provides a few inputs, which we call effectors , that the target pose has to respect.
For example, in Fig. 1, the look-at effector speciﬁes that the head should be facing the orange dot, the positional effectors constrain the right foot and the right hand to be pinned to the pink dots and the rotational effector, shown in cyan, constrains the world-space rotation of the pelvis.
We assume that the animator can generate arbitrary number of such effectors placed on any skeletal joint (one joint can be driven by more than one effector).
The task of the model is to combine all the information provided via effectors and generate a plausible full-body pose respecting provided effector constraints.
We deﬁne the full-body pose as the set of all kinematic parameters necessary to recreate the appearance of the body in 3D.
We assume that each effector can be represented in the space
R d eff , where d eff is taken to be maximum over all effector types.
Suppose we have 3D position and 6D rotation ( e.g. two-row rotation matrix representation (Zhou et al., 2019) ) effectors : d eff is 6.
In position effectors, 3 extra values are 0.
We formulate the pose authoring problem as learning the mapping ϒ θ : R N × d eff → R d kin with learnable parameters θ ∈ Θ .
ϒ θ maps the input space R N × d eff of variable dimensionality N (the number of effectors is not known apriori) to the space R d kin , containing all kinematic parameters to reconstruct full-body pose.
For example, a body with J joints can be fully deﬁned using a tree model with 6D local rotation per joint and 3D coordinate for the root joint, in which case d kin = 6 J + 3, assuming ﬁxed bone lengths.
Given a dataset D = { x i , y i } M i = 1 of poses containing pairs of inputs x i ∈ R N × d eff and outputs y
i ∈
R d kin it is viable to deﬁne the empirical risk minimization problem to learn ϒ θ :
[Equation]
1. R ELATED W ORK
Joint representations.
3D position joint representations can be used to specify a pose (Cheng et al., 2021;
Cai et al., 2019; Khapugin & Grishanin, 2019).
However, this approach is sub-optimal as it does not enforce ﬁxed-length bones, nor speciﬁes joint rotations.
Bone length constraints play important role in modeling realistic poses (Pavllo et al., 2018).
Joint rotations are crucial in downstream applications, such as deforming a 3D mesh on top of the skeleton, to avoid unrealistic twisting.
A practical solution is to predict joint rotations, automatically satisfying bone lengths and adequately modelling rotations (Pavllo et al., 2018).
This is viable via skeleton representations based on Euler angles (Han et al., 2017), rotation matrices (Zhang et al., 2018) and quaternions (Pavllo et al., 2018).
In this work, we use the two-row 6D rotation matrix representation that addresses the continuity issues reminiscent of quaternion and Euler representations (Zhou et al., 2019).
Pose modeling architectures.
Multi-Layer Perceptrons (MLPs) (Cho & Chen, 2014; Khapugin &
Grishanin, 2019; Mirzaei et al., 2020) and kernel methods (Grochow et al., 2004; Holden et al., 2015) have been used to learn single pose representations.
Beyond single pose, skeleton moving through time can be modeled as a spatio-temporal graph (Jain et al., 2016) or as a graph convolution (Yan et al., 2018; Mirzaei et al., 2020).
A common limitation of these approaches is their reliance on a ﬁxed set of inputs, whereas our architecture is speciﬁcally designed to handle sparse variable inputs.
Pose prediction from sparse constraints.
Real-time methods based on nearest-neighbor search, local dynamics and motion matching have been used on sparse marker position and accelerometer data (Tautges et al., 2011; Riaz et al., 2015; Chai & Hodgins, 2005; B¨uttner & Clavet, 2015).
MLPs and RNNs have been used for real-time processing of sparse signals such as accelerometers (Huang et al., 2018; Holden et al., 2017; Starke et al., 2020; Lee et al., 2018) and VR constraints (Lin &
O’Brien, 2019; Yang et al., 2021).
These approaches rely on the past pose information to disambiguate next frame prediction and as such are not applicable to our problem, in which only current pose constraints are available.
Iterative IK algorithms such as FinalIK (RootMotion, 2020) have been popular in real-time applications.
FinalIK works by setting up multiple IK chains for each limb of the body of a predeﬁned human skeleton and for a ﬁxed set of effectors.
Several iterations are executed to solve each of these chains using a conventional bone chain IK method, e.g. CCD.
In FinalIK, the end effector (hands and feet) can be positioned and rotated, while mid-effectors (shoulders and thighs) can only be positioned.
Effectors can have a widespread effect on the body via a hand-crafted pulling mechanism that gives a different weight to each chain.
This and similar tools suffer from limited realism when used for human full-body IK, as they are not data-driven.
Learning-based methods strive to alleviate this by providing learned model of human pose.
Grochow et al.
(2004) proposed a kernel based method for learning a pose latent space in order to produce the most likely pose satisfying sparse effector constraints via online constrained optimization.
The more recent commercial tool
Cascadeur uses a cascade of several MLPs (each dealing with ﬁxed set of positional effectors: 6, 16, 28) to progressively produce all joint positions without respecting bone constraints (Khapugin &
Grishanin, 2019; Cascadeur, 2019).
Unlike our approach, Cascadeur cannot handle arbitrary effector combinations, rotation or look-at constraints and requires post processing to respect bone constraints.
Permutation invariant architectures.
Models for encoding unstructured variable inputs have been proposed in various contexts.
Attention models (Bahdanau et al., 2015) and Transformer (Vaswani et al., 2017) have been proposed in the context of natural language processing.
Prototypical networks (Snell et al., 2017) used average pooled embedding to encode semantic classes via a few support images in the context of few-shot image classiﬁcation.
Maxpool representations over variable input dimension were proposed by Qi et al.
(2017) as PointNet and Zaheer et al.
(2017) as DeepSets for segmentation and classiﬁcation of 3D point clouds, image tagging, set anomaly detection and text concept retrieval.
Niemeyer et al.
(2019) further generalized the PointNet by chaining the basic maxpool/concat PointNet blocks resulting in ResPointNet architecture.
1. S UMMARY OF C ONTRIBUTIONS
The contributions of our paper can be summarized as follows.
• We deﬁne the 3D character posing task and publicly release two associated benchmarks.
• We show that a learned inverse kinematics solution can construct better poses, qualitatively and quantitatively, compared to a non-learned approach.
• We extend existing architectures with (i) semantic conditioning of joint ID and type at the input, (ii) novel residual scheme involving prototype subtraction and accumulation across blocks, as opposed to maxpool/concat daisy chain of ResPointNet, (iii) two-stage architecture with computationally efﬁcient residual decoder that improves accuracy at smaller computational cost, as opposed to the naive ﬁnal linear projection approach of PointNet and ResPointNet, and (iv) two-stage decoder design.
• We propose a novel look-at loss function and a novel randomized weighting scheme combining randomly generated effector tolerance levels and effector noise to increase the effectiveness of multi-task training.
[Figure]
c a t
B l o ck
[Figure]
l o ck
[Figure]
o ck
K
[Figure]
P
[Figure]
N x E i n
[Figure]
P r o t o
[Figure]
P ROTO R ES
ProtoRes, shown in Fig. 2, follows the encoder-decoder pattern, unlike PointNet and ResPointNet that use a linear layer as a decoding mechanism (Qi et al., 2017; Niemeyer et al., 2019).
The encoder has to deal with N effectors, whereas the decoder processes the collapsed representation of the pose and is therefore N times more compute efﬁcient.
Adding more decoder blocks thus results in accuracy gains at a fraction of compute cost.
Below, we describe the rest of architecture in more detail.
User inputs.
ProtoRes accepts position (3D coordinates), rotation (6D representation of Zhou et al.
and look-at (3D target position and 3D local facing direction) effectors.
All positions are re-referenced relative to the centroid of the positional effectors to achieve translation invariance.
Translation invariance simpliﬁes the handling of poses in global space removing the need in universal reference frame, which is tricky to deﬁne.
Each effector is further characterized by a positive tolerance value.
Smaller tolerance implies that the effector has to be more strictly respected in the reconstructed pose.
Moreover, the input includes effector semantics encoded via joint ID, an integer in [ 0 , J ) , and effector type, indicating a positional (0), rotational (1) or look-at (2) effector.
Unlike e.g. PointNet (Qi et al., 2017) acting on dense extensive point clouds, ProtoRes acts on sparse inputs that barely provide enough information about the full pose.
Therefore, it is critical to provide the semantic information on whether the given effector affects hands or feet and whether this is a positional or a rotational effector, for example.
Type and joint ID variables are embedded into continuous vectors and concatenated with effector data, resulting in the encoder input, a matrix x in ∈ R N × E in with E in corresponding to the combined dimension of all embeddings plus 6D effector data and a scalar for tolerance.
Pose Encoder is a two-loop residual network.
The ﬁrst residual loop is implemented inside each block depicted in Fig.
2 (bottom right).
The second residual loop shown in Fig.
2 (left) implements the proposed Prototype-Subtract-Accumulate (PSA) residual stacking principle, which we empirically found to outperform ResPointNet
’s Maxpool-Concat daisy chain proposed by Niemeyer et al.
Next, we ﬁrst lay out the encoder equations and then describe the motivation behind them in detail.
We assume the encoder input to be x 1 = x in ∈ R N × E in , omitting the batch dimension for brevity,
in which case the fully-connected layer FC r ,(cid:96) , with (cid:96) = 1 ... L , in the residual block r , r = 1 ... R , with weights
W r ,(cid:96) and biases a r ,(cid:96) can be conveniently described as FC r ,(cid:96) ( h r ,(cid:96) − 1 ) ≡ R E LU
( W r ,(cid:96) h r ,(cid:96) − 1 + a r ,(cid:96) ) .
The prototype layer is deﬁned as P ROTOTYPE ( x ) ≡ 1 N ∑ N i = 1 x [ i , : ] .
The pose encoder is then described as:
[Equation]
Equations (3) and (4) implement the MLP and the ﬁrst residual loop.
The proposed PSA residual mechanism, described in equations (2) and (5), is motivated by the following.
First, it implements the inductive bias that the information in individual effectors is only valuable when it is different from what is already stored in the embedding of entire pose.
Equation (2) implements this logic by forcing delta-mode in effectors w.r.t.
to the pose embedding, p r − 1 , from the previous block, which additionally creates another residual loop that should facilitate gradient ﬂow.
Second, equation (5) collapses the forward encoding of individual effectors into the representation of the entire pose via prototype, which is known to be very effective at representing information from sparse examples (see e.g. Snell et al.
Finally, the representation of pose is accumulated across residual blocks in (5), effectively implementing skip connections from very early layers.
Distant skip connections implemented via concatenation were shown to be effective in DenseNet (Huang et al., 2017).
Concatenation based skipping requires additional computation and is most efﬁcient with convolutional networks, in which kernel size can be traded for feature width while implementing distant skip connections, whereas accumulation is more compute efﬁcient in our context.
Pose Decoder has two blocks: global position decoder (GPD) and inverse kinematics decoder (IKD).
Both rely on the fully-connected residual (FCR) architecture depicted in Fig.
2 (right).
GPD unrolls the pose embedding generated by encoder into the unconstrained predictions of 3D joint positions.
IKD generates the internal geometric parameters (joint rotations) of the skeleton that are guaranteed to generate feasible joint positions after forward kinematics pass.
GPD accepts the encoded pose embedding, (cid:101) b 0 = p R ∈ R E , and produces 3D position predictions (cid:101)
f R ∈ R 3 J of all skeletal joints using the FCR whose r -th block is described as follows:
[Equation]
Since GPD produces predictions with no regard to skeleton constraints, its predictions do not respect bone lengths.
For the IKD to provide correct rotations, the origin of the kinematic chain in world space must be given, and GPD conveniently provides the prediction of the reference (root) joint.
IKD accepts input (cid:98) b 0 ∈ R E + 3 J , consisting of the concatenation of the encoder-generated pose embedding, p R ∈ R E , and the output of GPD, (cid:101)
f R ∈ R 3 J .
Effectively, the draft pose generated by GPD, is used to condition IKD.
We show in Section 3 that this additional conditioning improves accuracy.
IKD predicts the 6DoF angle for each joint, (cid:98)
f R ∈ R 6 J , and its r -th block operates as follows:
[Equation]
Forward Kinematics (FK) pass, described in detail in Appendix A, applies skeleton kinematic equations to the local joint rotations and global root position produced by IKD.
For each joint j , it produces the global transform matrix (cid:98) G j containing the global rotation matrix, (cid:98) G 13 j ≡ (cid:98) G j [ 1 : 3 , 1 : 3 ] , and the 3D global position, (cid:98) g
j = (cid:98) G j
[ 1 : 3 , 4 ] , of the joint.
2. L OSSES
We use three losses to train the architecture in a multi-task fashion.
The total loss combines loss terms additively with weights chosen to equalize their magnitude orders.
L2 loss penalizes the mean squared error between the prediction (cid:98) y and the ground truth y :
[Equation]
L2 loss is used to supervise the GPD as well as the IKD output after FK pass.
In the latter case it drives IKD to learn to predict rotations that lead to small position errors after FK.
Geodesic loss penalizes the errors of the IKD’s rotational outputs.
It represents the smallest arc (in radians) to go from one rotation to another over the surface of a sphere.
The geodesic loss is deﬁned for the ground truth rotation matrix R and its prediction (cid:98) R as (see e.g. Salehi et al.
[Equation] [Equation]
We believe that using a combination of L2 and Geodesic losses is necessary to learn a high-quality pose representation.
This is especially important when the task is to reconstruct a sparsely speciﬁed pose, giving rise to multiple plausible reconstructions.
We argue that a model trained to reconstruct both plausible joint positions and rotations is better equipped to solve the task accurately.
Empirical evidence presented in Section 3.4 supports this intuition: a model trained on both L2 and Geodesic generalizes better on both losses than models trained only on one of those terms.
Look-at loss , proposed in this paper, enables the “look-at” feature, i.e. the ability to orient a joint to face a particular global position (e.g. having the head looking at a given object).
It allows the model to align any direction vector d j ∈ R 3 of a joint, expressed in its local frame of reference, towards a global target location t .
Given the predicted global transform matrix (cid:98) G j , look-at loss is deﬁned as:
[Equation]
(10) −−−−→ ( t − (cid:98) g j ) is a unit-length vector pointing at the target object in world space.
(cid:98) G 13 j , when multiplied by d j , represents the global predicted look-at direction.
The look-at loss trains the IKD to produce (cid:98) G 13 j consistent with the look-at direction deﬁned by t and d j , both provided as network inputs.
2. T RAINING M ETHODOLOGY
The training methodology involves techniques to (i) regularize model via rotation and mirror augmentations, (ii) learn handling of sparse inputs and (iii) effectively combine multi-task loss terms.
Sparse inputs modeling relies on effector sampling.
First, the total number of effectors is sampled uniformly at random in the range [3, 16].
Given the total number of effectors, the effector IDs (one of 64 joints) and types (position, rotation, or look-at) are sampled from the Multinomial without replacement.
This induces exponentially large number of effector type and joint permutations, resulting in strong regularizing effects and teaching the network to deal with variable inputs.
Effector tolerance and randomized loss weighting.
The motivation behind randomized loss weighting is two-fold.
First, we empirically ﬁnd that when a random weight is multiplicatively applied to the respective loss term and its reciprocal is used as one of the network inputs for the corresponding effector, the network learns to respect the tolerance level.
When exposed as a user interface feature, it lets the user control the degree of responsiveness of the model to different effectors.
We also discovered that this only works when noise is added to the effector value and the standard deviation of the noise is appropriately modulated by tolerance.
For example, the noise teaches the model to disregard the effector completely if the tolerance input value corresponds to the high noise variance regime.
Second, we notice that the randomized weighting improves multi-task training and generalization performance.
In particular, we observe signiﬁcant competition between rotation and position losses on our task.
The introduction of the randomized loss weighting seems to turn the competition into cooperation as our empirical results suggest in Section 3.
We implement the randomized loss weighting scheme via the following steps.
For each sampled effector, we uniformly sample Λ ∈ [ 0 , 1 ] treated as the effector tolerance.
Given an effector tolerance Λ , noise with the maximum standard deviation σ M modulated by Λ (noise models used for different effector types are described in detail in Appendix B) is added to effector data before feeding them to the neural network:
[Equation]
We use η > 10 to shape the distribution of σ to smaller values.
Furthermore, each effector is attached with a randomized loss weight reciprocal to σ ( Λ ) , capped at W M if σ ( Λ ) < 1 / W M :
[Equation]
(12) Λ drives network input and W ( Λ ) weighs the loss term affected by the effector.
The detailed procedure to compute the ProtoRes loss based on one batch item is presented in
Algorithm 1 of Appendix C and the summary is provided below.
First, we sample (i) the number of effectors and (ii) their associated types and IDs.
For each effector, we randomly sample the tolerance level and compute the associated noise std and loss weight.
Given noise std, an appropriate noise model is applied to generate input data based on effector type as described in Appendix B.
Then
ProtoRes predicts draft joint positions (cid:101) f R , j , local joint rotations (cid:98) R j , as well as world-space rotations and positions (cid:98) G j for all joints j ∈ [ 0 , J ) .
We conclude by calculating the individual deterministic and randomized loss terms, whose weighted sum is used for backpropagation.
[Table]
E MPIRICAL R ESULTS
Our results demonstrate that (i) ProtoRes reconstructs sparsely deﬁned pose more accurately than existing non-ML IK solution, (ii) ProtoRes is more accurate than two ML baselines, (iii) the proposed encoder-decoder design is accurate and efﬁcient, (iv) our Prototype-Subtract-Accumulate residual scheme is more effective than the Maxpool-Concat daisy chain of Niemeyer et al.
(2019), (v) twostage GPD+IKD decoding is more effective than IKD-only decoding, (vi) the proposed randomized loss weighting improves multi-task training, (vii) joint Geodesic/L2 loss training is synergetic.
3. D ATASETS
miniMixamo We use the following procedure to create our ﬁrst dataset from the publicly available
MOCAP data available from mixamo.com , generously provided by Adobe Inc. (2020).
We download a total of 1598 clips and retarget them on our custom 64-joint skeleton using the Mixamo online tool.
This skeleton deﬁnition is used in Unity to extract the global positions as well as global and local rotations of each joint at the rate of 60 frames per second (total 356,545 frames).
The resulting dataset is partitioned at the clip level into train/validation/test splits (with proportion 0.8/0.1/0.1, respectively) by sampling clip IDs uniformly at random.
Splitting by clip makes the evaluation framework more realistic and less prone to overﬁtting: frames belonging to the same clip are often similar.
At last, the ﬁnal splits retain only 10% of randomly sampled frames (miniMixamo has 33,676 frames total after subsampling) and all the clip identiﬁcation information (clip ID, meta-data/description, character information, etc.) is discarded.
This anonymization guarantees that the original sequences from mixamo.com cannot be reconstructed from our dataset, allowing us to release the dataset for reproducibility purposes without violating the original dataset license (Adobe Inc., 2020).
miniAnonymous To collect our second dataset we predeﬁne a wide range of human motion scenarios and hire a qualiﬁed MOCAP studio to record 1776 clips (967,258 total frames @ fps).
Then we create a dataset of a total of 96,666 subsampled frames following exactly the same methodology that was employed for miniMixamo.
3. T RAINING AND E VALUATION S ETUP
We use Algorithm 1 of Appendix C to sample batches of size 2048 from the training subset, hyperparameters are adjusted on the validation set.
The number of effectors is sampled once per batch and is ﬁxed for all batch items to maximize data throughput.
We report metrics L detgpd − L 2 , L detikd − L 2 , L detloc − geo , deﬁned in Appendix E and calculated on the test set, using models trained on the training set.
L detgpd − L 2 is computed only on the root joint.
These metrics characterise both the 3D position accuracy and the bone rotation accuracy.
The evaluation framework tests model performance on a pre-generated set of seven ﬁles containing 6 , 7 ... 12 effectors respectively.
Metrics are averaged over all ﬁles, assessing the overall quality of pose reconstruction in scenario with sparse variable inputs.
Tables present results averaged over 4 random seed retries and metrics computed every 10 epochs over last 1000 epochs.
Additional details and hyperparameter settings appear in Appendix E.
K EY R ESULTS
To demonstrate the advantage of the proposed architecture, we perform two evaluations.
First, we compare ProtoRes against two ML baselines in the random effector evaluation setup described in
Section 3.2.
The ﬁrst baseline, Masked-FCR, is a brute-force unstructured baseline that uses a very wide J · 3 · 7 input layer ( J joints, 3 effector types, 6D effector data and 1D tolerance) handling all effector permutations.
Missing effectors are masked with 3 · J learnable 7D placeholders.
MaskedFCR has 3 encoder and 6 decoder blocks to match ProtoRes.
The second baseline is based on the
Transformer encoder (Vaswani et al.
(2017), see Appendix G for architecture and hyperparameter settings).
The bottom of Table 1, summarizing this study, shows clear advantage of ProtoRes w.r.t.
both baselines.
Additionally, training Transformer on NVIDIA M40 24GB GPU for 40k epochs of miniMixamo takes 1055 hours (batch size 1024 to ﬁt Transformer in GPU memory), whereas training
ProtoRes takes 106 hours.
ProtoRes is clearly more compute efﬁcient.
Second, Table 1 (top) compares ProtoRes against a non-ML IK solution FinalIK (RootMotion, 2020), as well as Transformer and Masked-FCR, on a 5-point evaluation benchmark.
The 5-point benchmark tests the reconstruction of the full pose from ﬁve position effectors: chest, left and right hands, left and right feet.
It is chosen, because generating the exponentially large number of FinalIK conﬁgurations to process all heterogeneous effector combinations in the random benchmark is not feasible.
Note that the 5-point benchmark and random benchmark results are not directly comparable.
We can see that all ML methods signiﬁcantly outperform FinalIK in reconstruction accuracy, ProtoRes being the best overall (please also refer to qualitative analysis results in Appendices J, K.4, K.8).
Clearly, ML methods learn the right inductive biases from the data to solve the ill-posed sparse input pose reconstruction problem, unlike the pure non-learnable IK method FinalIK.
Third, qualitative posing results are shown in Fig.
3, demonstrating that visually appealing poses can be obtained with small number of effectors (4 and 7 effectors for the two poses on the right).
The left 4 poses demonstrate how pose can be reﬁned successively by adding more effectors.
Please refer to
Appendix K and supplementary videos for more demonstration examples.
3. A BLATION S TUDIES
Decoder ablation is shown in Table 2 (top).
We keep all hyperparameters at defaults described in
Section 3.2 and vary the number of decoder blocks (0 blocks corresponds to a simple linear projection) and compare to the ProtoRes baseline with 3 decoder blocks.
We see consistent gain adding more decoder blocks at small compute cost (91, 95, 102, 106 hours of train time on miniMixamo dataset and NVIDIA M40 GPU for 0,1,2,3 blocks).
Please see more detailed results in Appendix H.
Prototype-Subtract-Accumulate ablation is presented in Appendix I, in which we compare it against the ResPointNet stacking scheme (Maxpool-Concat daisy chain by Niemeyer et al.
We show that our stacking scheme is more accurate in a computationally efﬁcient conﬁguration with 3 encoder blocks and allows stacking deeper networks gaining more accuracy.
GPD ablation is shown in Table 2 (middle).
We remove GPD and increase IKD depth to 6 blocks to match the capacity of IKD+GPD.
Comparing to the baseline, we see that GPD creates consistent gain across metrics and datasets by conditioning IKD with a draft pose.
[Table]
Ablation of loss terms, shown in Table 2 (middle), studies the effect of (i) removing all L2 loss terms from the output of the FK pass and (ii) removing all Geodesic loss terms from the rotation output of IKD.
Interestingly, removing either of the loss terms results in the degradation of all monitored metrics on both datasets.
We conclude that jointly penalizing positions with L2 and rotations with
Geodesic results in positive synergetic effects and improves the overall quality of pose model.
Randomized loss weighting ablation is shown in Table 2 (bottom).
The randomized loss weighting scheme (see Algorithm 1 in Appendix C) is disabled by replacing all randomized loss terms with their deterministic counterparts.
For example, L rndikd − L 2 is replaced with ∑ Jj = 1 MSE ( g j , (cid:98) g j ) .
The inclusion of randomized weighting signiﬁcantly improves generalization performance on all datasets and metrics.
Additionally, when L2 weight W pos increases with disabled randomized weighting, position L metrics improve, but at the expense of declining rotation metrics.
Therefore, randomized weighting scheme contributes positive effect that cannot be achieved by tweaking the deterministic loss weights.
The limitations of the current work , discussed in detail in Appendix L, include (i) the lack of temporal consistency as we focus on the problem of authoring a discrete pose, (ii) constraints are satisﬁed approximately, as opposed to the more conventional systems, (iii) exotic poses signiﬁcantly deviating from the training data distribution may be hard to achieve.
C ONCLUSIONS
We deﬁne and solve the discrete full-body pose authoring task using sparse and variable user inputs.
We deﬁne and release two datasets to support the development of ML models for discrete pose authoring and animation.
We propose ProtoRes, a novel ML architecture which processes a variable number of heterogeneous user inputs (position, angle, direction) to reconstruct a full-body pose.
We compare ProtoRes against two strong ML baselines, Masked-FCR and Transformer, showing superior results for ProtoRes, both in terms of accuracy and computational efﬁciency.
We also show that ML models reconstruct full-body poses from sparse user inputs more accurately than existing non-learnable inverse kinematics models.
We develop a suite of UI tools for the integration of our model in Unity and provide demos showing how our model can be used effectively to solve the discrete pose authoring problem by the end user.
Our results have a few implications.
First, our
ML based tools will have positive impacts on the simpliﬁcation and democratization of the game development process by helping a wide audience materialize their creative animation ideas.
Second, our novel approach to neural pose representation could be applied in a variety of tasks where efﬁcient and accurate reconstruction of full-body poses from noisy intermittent measurements is important. L Limitations Block Block...
Block R + FC layer FC layer L Linear Linear ...
+
ReLu
Residual
Block
Output
[Figure]
Input
Output
A A RCHITECTURE D ETAILS
Residual block depicted in Fig.
4 (left) is used as the basic building block of the ProtoRes encoder and decoder.
Decoders The block diagram of the global position and the inverse kinematics decoders used in the main architecture (see Fig. 2) is presented in Fig.
The architecture has fully connected residual topology consisting of multiple fully connected blocks connected using residual connections.
Each block has residual and forward outputs.
The forward output contributes to the ﬁnal output of the decoder.
The residual connection sums the hidden state of the block with the linear projection of the input and applies a ReLU non-linearity.
In the main text we use a convention that the number of layers and blocks in the encoder, as well as in GPD and IKD decoders is the same and is given by L and R respectively.
Obviously, using a different number of layers and residual blocks in each of the blocks might be more optimal.
Forward Kinematics pass is applied to the output of the IKD, transforming local joint rotations and global root position into the global joint rotations and positions using skeleton kinematic equations.
The FK pass relies on the offset vector o j
=
[ o x , j , o y , j , o z , j ] (cid:124) and the rotation matrix R j for each joint j .
The offset vector is a ﬁxed non-learnable vector representing bone length constraint for joint j .
It provides the displacement of this joint with respect to its parent joint when joint j rotation is zero.
R j can be na¨ıvely represented using local Euler rotation angles α j , β j , γ j : o j = (cid:34) o x
[Equation]
o
[Equation]
(cid:34)
[Equation] [Equation]
(cid:35)(cid:34)
cos
[Equation]
sin β
[Equation]
− sin
[Equation]
0 cos
[Equation]
(cid:35)(cid:34)
[Equation]
0 cos γ j − sin γ j sin γ
[Equation]
cos γ j (cid:35) .
However, we use a more robust representation proposed by (Zhou et al., 2019), relying on vector norm −→ u ≡ u / (cid:107) u (cid:107) 2 and vector cross product u × v =
(cid:107) u (cid:107)(cid:107) v (cid:107) cos ( γ )
−→ n ( γ is the angle between u and v in the plane containing them and −→ n is the normal to the plane):
[Equation]
Provided with the local offset vectors and rotation matrices of all joints, the global rigid transform of any joint j is predicted following the tree recursion from the parent joint p ( j ) of joint j :
[Equation]
The global transform matrix (cid:98) G j of joint j contains its global rotation matrix, (cid:98) G 13 j ≡ (cid:98) G j [ 1 : 3 , 1 : 3 ] , and its 3D global position, (cid:98) g j = (cid:98) G j [ 1 : 3 , 4 ] .
B E FFECTOR NOISE MODEL
This section describes the details of the N OISE M ODEL that is used in Algorithm 1 to corrupt model effector input x
[ i , : ] based on appropriate noise level σ ( Λ i ) .
B. P OSITION EFFECTOR NOISE MODEL
If effector type is positional ( T i = 0), i.e. effector i is a coordinate in 3D space, typically corresponding to the desired position of joint I i in 3D space, we employ Gaussian white noise model:
[Equation]
[Equation]
Here x [ i , : ] is the i -th model input, g I i is the ground truth location of joint
I i , σ ( Λ i ) is the noise standard deviation computed based on eq.
(20) and ε i is a 3D vector sampled from the zero-mean
Normal distribution N ( 0 , I ) .
B. R OTATION EFFECTOR NOISE MODEL
If effector type is angular ( T i = 1), i.e. effector i is a 6DoF rotation matrix representation, we employ random rotation model that is implemented in the following stages.
First, suppose f I i is the ground truth 6DoF representation of the global rotation of joint I i corresponding to effector i .
We transform it to the rotation matrix representation G
I i using equation (14).
Second, we generate the random 3D
Euler angles vector ε i from the zero-mean Gaussian distribution N ( 0 , σ ( Λ i ) I ) 1 and convert it to the random rotation matrix Ψ
i using eq.
[Equation]
=
[Equation]
[Equation] [Equation]
[Equation] [Equation]
[ 3 ] (cid:35) .
Third, we apply random rotation to the ground truth matrix, G 13 (cid:48)
I i =
Ψ i G 13 I i .
Finally, we convert the randomly perturbed rotation matrix back to the 6DoF representation:
[Equation]
B. L OOK - AT EFFECTOR NOISE MODEL
If effector type is look-at ( T i = 2), i.e. effector i is a position of the target at which a given joint is supposed to look, we employ random sampling of the target point along the ray cast in the direction formed by the global rotation of a given joint.
First, we sample the local direction vector d i from the zero-mean normal 3D distribution N ( 0 , I ) and normalize it to unit length.
Second, we sample the distance between the joint and the target object, d t , from the normal distribution N ( 0 , 5 ) folded over at 0 by taking the absolute value.
The location of the target object is then determined as t
i
=
g
I
i + d t d i + σ ( Λ i ) ε i .
Finally, the output is constructed as follows
[Equation] [Equation]
As previously, ε i is a 3D vector sampled from the zero-mean Normal distribution N ( 0 , I ) .
Require: R j , G j ; N ∼ U NIFORM [ 3 , 16 ]
(cid:46) Ground truth for all joints j ∈ [ 0 , J ) ; number of effectors
Ensure: x (cid:46)
Sample inputs
I 1 ,..., I N ← M ULTINOMIAL ( { 0 ,..., J − 1 } , N ) (cid:46) Effector IDs
T 1 ,..., T N ← M ULTINOMIAL ( { 0 , 1 , 2 } , N ) (cid:46) Effector type for i in 1 ...
N do Λ i ← U NIFORM [ 0 , 1 ] (cid:46) Effector tolerance σ ( Λ i ) ; W ( Λ i ) ← σ M Λ η i ; min ( W M , 1 / σ ( Λ i ))
(cid:46)
Effector noise std and weight x
[ i , : ] ← N OISE M ODEL ( G I i , σ ( Λ i ) , T i ) (cid:46) Generate noisy effector end for
Predict: (cid:101) f R , j , (cid:98) R j , (cid:98) G j ∀ j based on x
L rndgpd − L 2 ← 1 ∑ Ni = 1 1 Ti = 0 W ( Λ i )
∑ Ni = 1 1 T i = 0 W ( Λ i ) MSE ( g I i , (cid:101)
f R , I i ) (cid:46)
Randomized GPD position loss
L rndikd − L 2 ← 1 ∑ Ni = 1 1 Ti = 0 W ( Λ i )
∑ Ni = 1 1 T i = 0 W ( Λ i ) MSE ( g I i , (cid:98) g
I i ) (cid:46)
Randomized IKD position loss
L detgpd − L 2 ← ∑ Jj = 1 MSE ( g j , (cid:101) f R , j ) (cid:46)
Deterministic GPD position loss
L detikd − L 2 ← ∑ Jj = 1 MSE ( g j , (cid:98)
[Table]
IKD
[Table]
C T RAINING AND E VALUATION M ETHODOLOGY : D ETAILS
The training methodology involves techniques targeting to (i) regularize model via data augmentation, (ii) learn handling of sparse inputs and (iii) effectively combine multi-task loss terms.
Data augmentation is based on the rotation and mirror augmentations.
The former rotates the skeleton around the vertical Y axis by a random angle in [ 0 , 2 π ] .
Rotation w.r.t.
ground XZ plane is not applied to avoid creating poses implausible according to the gravity direction.
Mirror augmentation removes any implicit left- or right-handedness biases by ﬂipping the skeleton w.r.t.
the YZ plane.
Sparse inputs modeling relies on effector sampling.
First, the total number of effectors is sampled uniformly at random in the range [3, 16].
Given the total number of effectors, the effector IDs (one of 64 joints) and types (one of 3 types: position, rotation, or look-at) are sampled from the Multinomial without replacement.
This sampling scheme produces an exponentially large number of different permutations of effector types and joints, resulting in strong regularizing effects.
Effector tolerance and randomized loss weighting.
The motivation behind the randomized loss weighting is two-fold.
First, the randomized loss weighting was originally introduced as a binary indicator to force the model to better respect constraints provided as effectors, compared to the joints predicted by the model.
Afterwards, we realized that this can be made more ﬂexible by generating a continuous variable representing the tolerance level.
This variable can be provided as an input to the network and it can be exposed as a user interface feature to let the user control the degree of responsiveness of the model to different effectors.
We also discovered that the latter feature only works when a noise is added to effector value and the standard deviation of the noise is appropriately synchronised with the tolerance.
The noise teaches the model to disregard the effector completely if the tolerance input value corresponds to the high noise variance regime.
Second, we observed that the use of the randomized weighting improves multi-task training and generalization performance.
Initially, we noticed that increasing the weight of position loss would drive the generalization on the position metric to a better spot, while the rotation metric generalization would be compromised, which is not surprising.
This was especially evident when the position loss weight was increased by one or two orders of magnitude.
This is a well-known phenomenon when dealing with multiple loss terms, which we informally call “ﬁghting” between losses (related to the Pareto front, more formally).
This effect can be observed when comparing two bottom rows in
Table 2.
Introducing the randomized loss weighting scheme we observed two things.
“Fighting” disappeared, i.e. the randomly generated weights of position effectors varied in a wide range between 1e-1 and 1e5 within a batch, but the fact that some of the weight values are one or two orders of magnitude greater than the baseline position weight of 100, did not lead to the deterioration of the rotation loss.
Moreover, the introduction of the randomized loss weighting positively affected the generalization on both position and rotation metrics, which can be assessed by comparing the ﬁrst row of Table 2 with its bottom rows.
This leads us to believe that the randomized loss weighting introduces a sinergy in the multi-task training that is not achievable by simple adjustment of static loss weights.
We believe this technique could be more generally applicable to multi-task training, but a more detailed investigation of this is outside of the current scope.
We now describe the technical details behind randomized loss weighting implementation.
For each sampled effector, we further uniformly sample Λ ∈ [ 0 , 1 ] treated as effector tolerance.
Given an effector tolerance Λ , noise (noise models used for different effector types are described in detail in
Appendix B) with variance proportional to Λ is added to effector data before feeding them to the neural network:
[Equation]
We use η > 10 to shape the distribution of σ to smaller values.
Furthermore, to each effector is attached a randomized loss weight reciprocal to σ ( Λ ) , capped at W M if σ ( Λ ) < 1 / W M :
[Equation]
Λ drives network inputs and is simultaneously used to weigh losses by W ( Λ ) .
Thus ProtoRes learns to respect effector tolerance, leading to two positive outcomes.
First, ProtoRes provides a tool allowing one to emphasize small tolerance effectors ( Λ ≈ 0) and relax the large tolerance ones ( Λ ≈ 1).
Second, randomized loss weighting improves the overall accuracy in the multi-task training scenario.
The detailed procedure to compute the ProtoRes loss based on one batch item is presented in
Algorithm 1 and the summary is provided below.
First, we sample (i) the number of effectors and (ii) their associated type and ID.
For each effector, we randomly sample the tolerance level and compute the associated noise std and loss weight.
Given noise std, an appropriate noise model is applied to generate input data based on effector type as described in Appendix B. Then ProtoRes predicts draft joint positions (cid:101) f R , j , local joint rotations (cid:98) R j , as well as world-space rotations and positions (cid:98) G j for all joints j ∈ [ 0 , J ) .
We conclude by calculating the individual deterministic and randomized loss terms, whose weighted sum is used for backpropagation.
D D ATASETS : D ETAILS
miniMixamo We use the following procedure to create our ﬁrst dataset from the publicly available
MOCAP data available from mixamo.com , generously provided by Adobe Inc. (2020).
We download a total of 1598 clips and retarget them on our custom 64-joint skeleton using the Mixamo online tool.
This skeleton deﬁnition is used in Unity to extract the global positions as well as global and local rotations of each joint at the rate of 60 frames per second (total 356,545 frames).
The resulting dataset is partitioned at the clip level into train/validation/test splits (with proportion 0.8/0.1/0.1, respectively) by sampling clip IDs uniformly at random.
Splitting by clip makes the evaluation framework more realistic and less prone to overﬁtting: frames belonging to the same clip are often similar.
At last, the ﬁnal splits retain only 10% of randomly sampled frames (miniMixamo has 33,676 frames total after subsampling) and all the clip identiﬁcation information (clip ID, meta-data/description, character information, etc.) is discarded.
This anonymization guarantees that the original sequences from mixamo.com cannot be reconstructed from our dataset, allowing us to release the dataset for reproducibility purposes without violating the original dataset license (Adobe Inc., 2020).
For miniMixamo our contribution is as follows.
Mixamo data is not available as a single ﬁle.
Therefore, anyone who wants to use the data for academic purposes needs to go through a lengthy process of downloading individual ﬁles.
Importantly, this step creates additional risks for the reproducibility of results.
We have gone through this step and assembled all ﬁles in one place.
Furthermore, Mixamo data cannot be redistributed, according to Adobe licensing, which is again a reproducibility risk.
[Table]
However
[Table]
we do not need the entire dataset for benchmarking on the task we deﬁned.
Therefore, we deﬁned a suitable subsampling and anonymization procedure that allowed us to obtain (i) a high quality reproducible benchmark dataset for our task and (ii) a legal permission from Mixamo/Adobe to redistribute this benchmark for academic research purposes.
We are extremely grateful to the representatives from Mixamo and Adobe who approved it to facilitate the democratization of character animation.
The entire process of creating the benchmark took us a few months of work, which we consider a signiﬁcant contribution to the research community.
miniAnonymous To collect our second dataset we predeﬁne a wide range of human motion scenarios and hire a qualiﬁed MOCAP studio to record 1776 clips (967,258 total frames @ fps).
Then we create a dataset of a total of 96,666 subsampled frames following exactly the same methodology that was employed for miniMixamo.
The key differentiators of the datasets that we release that make them signiﬁcant contributions toward
AI driven artistic pose development are as follows:
• Both miniMixamo (derived from the Mixamo, which is generously provided by Adobe) and miniAnonymous are collected by professional studio contractors relying on the service of professional actors using high-end MOCAP studio equipment.
• Both datasets are clean and contain data of very high quality.
For our dataset, we speciﬁcally had to go through multiple cleaning iterations to make sure all the data collection and conversion artifacts are removed.
We are very grateful to our contractor for being diligent, detail oriented, and determined to provide the high quality data.
• Both datasets provide data in the industry standard skeleton format compatible with multiple existing animation rigs and therefore making it easy to experiment with the ML assisted pose authoring results in 3D development environments such as Unity.
This is in contrast to CMU and AMASS datasets that are collected in heterogeneous environments using non-standard sensor placements.
• Both our datasets provide 64 joint skeletons and contain ﬁne grain hands and feet data, unlike other publicly available datasets.
E T RAINING AND E VALUATION S ETUP : D ETAILS
We use Algorithm 1 of Appendix C to sample batches of size 2048 from the training subset.
The number of effectors is sampled once per batch and is ﬁxed for all batch items to maximize data throughput.
The training loop is implemented in PyTorch (Paszke et al., 2019) using Adam optimizer (Kingma &
Ba, 2015) with a learning rate of 0.0002. Hyperparameter values are adjusted on the validation set (see Appendix E for hyperparameter settings). We report L detgpd − L 2 , L detikd − L 2 , L detloc − geo metrics calculated on the test set, using models trained on the training set. L detgpd − L 2 is computed only on the root joint.
These metrics characterise both the 3D position accuracy ( L detgpd − L 2 , L detikd − L 2 ) and the bone rotation accuracy ( L detloc − geo ).
They are deﬁned as follows:
[Equation]
[Equation]
[Equation]
Here (cid:101) f R , 0 and g 0 are ground truth global location of the root joint and its prediction from the GPD, respectively; g j and (cid:98) g j ground truth location of joint j obtained by subjecting the rotation prediction from IKD to the forward kinematics process; R j and (cid:98) R j are ground truth local rotation matrix of joint j and its prediction obtained from the IKD, respectively.
The evaluation framework tests model performance on a pre-generated set of seven ﬁles containing 6, 7, .. .
, 12 effectors respectively.
Skeleton is split in six zones, with four main zones including each limb, the hip zone and the head zone.
In each ﬁle, we ﬁrst sample one positional effector from each main zone.
Remaining effectors are sampled randomly from all zones and effector types, mimicking pose authoring scenarios observed in practice.
Metrics are averaged over all samples in all ﬁles, assessing the overall quality of pose reconstruction in scenario with sparse and variable inputs.
All tables present results averaged over 4 random seed retries and metric values computed everyepochs over last 1000 epochs, rounded to the last statistically signiﬁcant digit.
Hyperparameter settings The training loop is implemented in PyTorch (Paszke et al., 2019) using
Adam optimizer (Kingma & Ba, 2015) with a learning rate of 0.0002.
We tried to use SGD optimizer to train the architecture, but it was very difﬁcult to obtain stable results with it.
Adam optimizer turned out to be much more suitable for our problem.
The learning rate was selected to be 0.0002, which is lower than Adam’s default.
Obtaining stable training results with higher learning rates was not feasible.
Batch size is selected to be 2048 to accelerate training speed.
In practice we observed slightly better generalization results with smaller batch size (1024 and 512).
The detailed settings of
ProtoRes hyperparameters are presented in Table 3.
F M ASKED -FCR BASELINE ARCHITECTURE
Masked-FCR is a brute-force unstructured baseline that uses a very wide J · 3 · 7 input layer ( J joints, 3 effector types, 6D effector plus one tolerance value) to handle all possible effector permutations.
Each missing effector is masked with one of 3 · J learnable 7D placeholders.
Masked-FCR hasencoder and 6 decoder blocks to match ProtoRes.
G T RANSFORMER BASELINE ARCHITECTURE
We implement Transformer baseline using the default Transformer module available from PyTorch https://pytorch.org/docs/stable/generated/torch.nn.Transformer. html .
The block diagram of the Transformer baseline architecture is shown in Fig.
We use a standard transformer application scenario in which the transformer source input is fed with the variable length input and the required outputs are queried via target input.
In our case the variable length input corresponds to the effector data concatenated with embedded effector categorical variables.
The query for the output consists of the embeddings of all joints.
Note that the joint embedding is reused both for source and target inputs and both inputs are projected to the internal d model dimensionality of transformer.
Transformer Baseline
OUTPUTS POSE DECODER
Transformer
TRANSLATION INVARIANCE USER INPUTS
Position, N P x
Effector Joint ID, Nx
Rotation, N R x
Effector Type, NxPadding N P x
Nx
C on c a t
Fully Connected
Residual Decoder - +
Joint Positions, unconstrained (Jx3) reference
Joint Rotations (Jx6) Effector Tolerance, Nx
Joint Positions (Jx3) root position
NxE in
Proto
Concat
Embedding Embedding
FORWARD KINEMATICS
Look-at, N L x
N=N P +N R + N L src
All Joint IDs, Jx
Embedding tgt
Jx
Linear
J x d_model
L i nea r
N x d_model out
Linear Linear
J x d_model
J x
Hyperparameter Value
Grid
FCR decoder parameteres
FCR Blocks ( R )
N/A
FCR Width ( d h )
N/A
Transformer parameteres d model [64, 128, 256] nhead [1, 2, 4, 8] num encoder layers [1,2,3] num decoder layers [1,2,3] dim feedforward [256, 512, 1024] dropout 0.
activation relu relu
[Table]
The embeddings of joint IDs are used to query the Transformer output, producing one encoder embedding for each of J joints.
The J encodings are fed into the 6-block FCR decoder (to match the total number of decoder blocks in ProtoRes) with two heads: one predicting rotation and one predicting unconstrained position.
This is similar to the use of Transformer to predict bounding box class IDs and sizes for object detection (Carion et al., 2020).
Predictions of rotations and of the root joint are used in the forward kinematics pass, just as in ProtoRes.
Internally, Transformer processes both source and target inputs via self-attention ﬁrst and then applies the multi-head attention between source and target after self-attention.
This results in the output embedding for each skeleton joint that depends on all the input information as well as the learned interactions across all output skeleton joints.
The output embedding of the transformer is then decoded to unconstrained position and rotation outputs using a two-headed Fully-Connected residual stack (this is the same architecture as the one used in ProtoRes decoders).
Note that this is a well-known Transformer application scheme that has recently been used to achieve SOTA results in object detection, for example (Carion et al., 2020).
Table 4 lists the hyperparameter settings for the
Transformer baseline.
Note that only hyperparameters that are unique to this baseline or different from the ProtoRes defaults appearing in Table 3 are listed.
H A BLATION OF THE D ECODER : D ETAILS
The detailed results of the decoder ablation are shown in Table 5.
One block of decoder is much less computationally expensive than one block of encoder.
One block of encoder processes N effectors, [Table] whereas the decoder deals with the partially deﬁned pose representation collapsed to a vector.
Hence the decoder block is N times less expensive.
To demonstrate the effectiveness of decoder, we keep all hyperparameters at defaults described in Section 3.2 and vary the number of encoder and decoder blocks in ProtoRes (0 decoder blocks corresponds to a simple linear projection of encoder output).
We measure the train time of each conﬁguration on NVIDIA M40 24GB GPU installed on Dell
PowerEdge R720 server with two Intel Xeon E5-2667 2.90GHz CPU.
The table reveals a few things.
First, adding more decoder blocks signiﬁcantly increases accuracy when the number of encoder blocks is lower (e.g. 3 or 5).
When the number of encoder blocks is high (e.g. 7) linear projection provides similar accuracy.
Second, using non-trivial decoder is computationally more efﬁcient.
For example, the 3 encoder and 3 decoder blocks conﬁguration has comparable accuracy with 5 encoder and 1 decoder blocks, however it is noticeably more compute efﬁcient (5+1 conﬁguration uses 40% more compute time than 3+3).
I A BLATION OF THE P ROTOTYPE -S UBTRACT -A CCUMULATE STACKING PRINCIPLE
Here we compare the proposed Prototype-Subtract-Accumulate (PSA) residual stacking scheme described in equations (2)-(5) against the ResPointNet (Niemeyer et al., 2019) stacking scheme:
Maxpool-Concat Daisy Chain (MCDC).
To implement the MCDC stacking proposed by Niemeyer et al.
we setup the experiment as follows
• Equation (2) is replaced with the concatenation of the output of the previous block, b r , with the maxpool of the previous block output along axis 1 (we use batch, effector, channels convention for tensor axes 0,1,2; respectively)
• In equation (4) we only compute b r , since f r is not used • Equation (5) is removed •
The ﬁnal pose embedding is created using the maxpool along axis 1 of b r at the last encoder block •
We use decoder with 0 blocks, i.e. only the linear projection at the end to derive both PSA and MCDC results.
This is because (i) residual decoder is not part of the original design by Niemeyer et al.
(2019), (ii) in this study we focus exclusively on the effects of stacking within the encoder, equalizing all other experimental conditions.
• Hyperparameters of both architectures are taken from Table 3 in Appendix C [Table]
We show that our stacking scheme is more accurate and allows stacking deeper networks more effectively.
Quantitative results are shown in Table 6.
It is clear that the proposed stacking approach provides gain in setups with varying number of encoder blocks.
For example, in the case of small number of 3 blocks (computationally efﬁcient setup) our PSA approach is clearly more accurate than MCDC approach of (Niemeyer et al., 2019).
Similarly, with 7 blocks (training time is more than two times longer) PSA is again more accurate than MCDC.
Moreover, in the case of PSA we see noticeable accuracy improvement while increasing the number of encoder blocks from 5 to 7, whereas MCDC provides almost no additional gain beyond 5 blocks.
We conclude that the proposed PSA stacking mechanism is better than MCDC as it provides better accuracy in the computationally efﬁcient conﬁguration and it is signiﬁcantly more effective at supporting deeper architectures that provide better accuracy in the case of PSA, whereas MCDC accuracy saturates at 5 blocks providing little to no accuracy gain with deeper architectures. J Q UALITATIVE COMPARISON TO T RANSFORMER AND F INAL IK BASELINES
In this section we provide additional qualitative comparison between ProtoRes and two baselines, the non-learnable baseline FinalIK and the machine learning baseline Transformer.
The FinalIK comparison results appear in the top row of Figure 6.
The Transformer comparison results appear in the bottom row of Figure 6.
The characteristic feature of FinalIK is that it lacks inductive bias towards realistic poses.
Therefore it is very easy to create effector conﬁgurations that result in unnatural poses as can be seen in Figure 6. Conversely, creating realistic poses requires signiﬁcant amount of tuning of the individual joints as most joints of the body act very independently and locally. With FinalIK, it is relatively hard to produce a believable pose with only a few effectors. ProtoRes ﬁlls this gap by providing a learned prior for realistic poses.
As a result, most of the poses created by ProtoRes look natural, no matter how many effectors are used to steer the pose.
In addition, the effector space can be augmented dynamically to capture missing accents, if necessary.
When comparing ProtoRes against Transformer, we can see that Transformer has a less global approach to forming a pose than ProtoRes.
This manifests itself in the Tranformer based model having a tendency to create poses in which limbs penetrate each other or the rest of the body (see
Figure 6 bottom row, pictures 1,2,3 from the left).
Also, Transformer is very good at following the effector inputs and reproducing them in the reconstructed pose — so much that it is willing to sacriﬁce the overall ﬁnal pose plausibility at the expense of sticking to the effector, taking the effector guidance very “literally” and missing its re-interpretation given global context.
A good example of this behaviour is presented in Figure 6 bottom row, rightmost picture.
We see that ProtoRes brings the entire body to the ﬂoor and prepares the arms to touch the ﬂoor to support the body, all in the attempt to produce a plausible pause of someone looking at the look-at target that is placed relatively low.
On the other hand, Transformer takes the look-at effector guidance very literally and locally, willing to break the neck of the pose to look at the provided target and not realizing that the entire body position needs to be changed to accommodate for this rather peculiar conﬁguration of effectors to create a believable pose.
In general, our observation is that the Transformer based baseline response to effector inputs tends to be localized.
As a consequence, its behaviour has a very pronounced pure
IK ﬂavour to it, at times reminding the behaviour of FinalIK, especially when facing difﬁcult effector conﬁgurations.
A lot of the time, however, the outputs of ProtoRes and Transformer are comparable and consistent, since both of them develop strong inductive bias towards realistic poses.
K V IDEOS AND DEMONSTRATIONS
We provide here descriptions related to each video found in the supplementary materials.
Note that most of the demonstrations are done using a ProtoRes model trained on a large internal dataset not evaluated in this work.
One of our demonstrations, described below in Appendix K.5 qualitatively shows some of the most severe impacts of training on ablated datasets.
K. P ROTO R ES D EMO
The video presents an overview of the integration of ProtoRes as a posing tool inside the Unity game engine, showcasing how different effector types can be manipulated and how they can inﬂuence the resulting pose.
In this demo, a user-deﬁned conﬁguration is presented in the UI, allowing one to choose which effectors are enabled within that conﬁguration.
Note that this conﬁguration could contain more or less effectors of each type, and can be built for speciﬁc posing needs.
The most generic conﬁguration would present all possible effectors inside each effector type sub-menu.
K. P OSING FROM I MAGES
This video presents screen recordings of a novice user using ProtoRes to quickly prototype poses taken from 2D silhouette images.
Note that one can reach satisfactory results in less than a minute in each case, with a relatively low number of manipulations.
Note also that ﬁne-tuning the resulting poses can always be achieved by adding more effectors and applying more manipulations.
K. L OSS A BLATION
This recording shows a setup where different models are used with identical effector setup.
Both models use the ProtoRes architecture.
On the left, the model uses the total loss presented in Algorithm 1, whereas the right-hand side model uses positional losses only (GPD and IKD positional losses) and both local and global rotational losses, as well as look-at losses are disabled.
This demonstration clearly shows how positional constraints, even when respected, do not sufﬁce to produce realistic human poses.
Joint rotations have to be modeled as well.
K. F INAL IK C OMPARISON
This recording shows a setup where ProtoRes is compared to a full body biped IK system provided by FinalIK RootMotion (2020), with an identical effector setup.
Note that FinalIK solves constraints by modifying the current pose, often resulting in smaller changes in the output, when compared to
ProtoRes that predicts a full pose at each update.
The lack of a learned model of human poses in
FinalIK becomes quickly noticeable when manipulating effectors signiﬁcantly.
K. D ATASETS C OMPARISON
In these demonstrations, we showcase how training ProtoRes on different datasets can impact the results.
We showcase models trained on the two ablated datasets presented in this work, i.e. miniAnonymous (left) and miniMixamo (right).
We also show performance of a model trained on the full Mixamo Adobe Inc. (2020) dataset (center) to qualitatively show how performance can be improved with more training data.
In all of these recordings, one can notice differences in the resulting poses, emphasizing the fact that human posing from few effectors, when no extra conditioning signals are used, is an ambiguous task that will be inﬂuenced by the training data.
The ﬁrst sequence makes this fact especially obvious on the ﬁnger joints and the head’s look-at direction.
The second sequence shows how good data coverage in the training set can signiﬁcantly impact performance in special or rare effector conﬁgurations.
Finally, the third sequence shows a similar pattern for look-at targets, where the difference in training data can be noticed in the general posture of the character and the varying levels of robustness with respect to those targets.
K. R ETARGETING
This video shows that ProtoRes trained on one dataset can be successfully applied to multiple characters deﬁned on skeletons on which our model was never trained.
Currently, the model is trained on a dataset with a speciﬁc skeleton layout with speciﬁc bone offsets.
In a na¨ıve application scenario, a new skeleton would require retraining the model on a dataset for this speciﬁc skeleton, which could be a prohibitively expensive exercise.
To overcome this limitation, we show in this video that the model learned using our approach can be retargeted to very different skeletons using conventional retargeting methods via a process that is fully automatic, without changing the model in any way.
Thus, our model can be seamlessly used without any retraining with a wide variety of skeletons.
In practice this solution allows the use of the same model on characters having similar skeleton in terms of topology and/or morphology and very different in other aspects such as proportions (bone lengths) or bone count.
For instance, as our demo shows, it can be used with any humanoid character and still produces valuable results.
Transfer of the model across skeletons using more sophisticated ML-based techniques such as ﬁne-tuning, conditioning, domain adaptation is deemed to a be a fruitful future research area, for which our current results form a strong baseline.
K. A PPLICATION TO THE Q UADRUPED S KELETON
In this video we demonstrate that ProtoRes can be applied to a completely different type of skeleton (quadruped) and it does not require any additional coding as opposed to FinalIK (no hand-crafted skeleton, effectors, bone chains, pulling, etc).
It does require data and re-training however, but does not contain anything speciﬁc to humanoids.
In fact, we have successfully trained the proposed model on a quadruped (dog) dataset without changing the model or hyperparameters, which demonstrates the generality of the method.
K. C OMPARISON TO T RANSFORMER
In this video we demonstrate qualitative examples demonstrating the basic difference between
Transformer and ProtoRes.
First, generally speaking, in many cases with challenging effector settings
ProtoRes shows much greater robustness to outliers with respect to pose plausibility, whereas in many cases Transformer’s behaviour feels like that of a pure non-learnable IK model.
Second, Transformer has a tendency to generate self-intersections between the limbs to satisfy some of the constraints more closely at the expense of the overall pose plausibility.
Finally, in response to an extreme look-at constraint Transformer can bend the neck behind the body, which in reality never happens in the data and would break the neck of a real human.
In contrast, ProtoRes keeps the neck naturally oriented when the look-at target is unattainable.
From the theoretical perspective, the fundamental difference between the Transformer framework and the ProtoRes is as follows.
ProtoRes creates one global representation of the partially speciﬁed pose and then reconstructs the full pose in one shot through a global IKD.
On the other hand,
Transformer generates embeddings of individual joints for the full output pose, from which a shared
IKD reconstructs the local rotation angles for each joint.
Transformer has all the ingredients to reproduce processing similar to our approach.
However, in reality the Transformer takes a different learning route and (i) learns more localized joint predictions and (ii) learns to more strictly respect local input constraints, even at the expense of creating poses that are not statistically plausible.
Combining the global (ProtoRes) and local (Transformer) approaches to derive better hybrid models seems like a promising direction for future work.
K. L IMITATIONS
The ﬁnal video shows examples of some speciﬁc limitations of the approach that are listed in
Appendix L. Namely, we ﬁrst expose speciﬁc consequences of the lack of temporal consistency in our problem formulation, where smoothly moving an effector can cause ﬂickering on some joints, such as the ﬁngers.
We also show how between some effector conﬁgurations, the character must be ﬂipped completely to stay in a plausible pose, and how it’s possible to place some effectors to reach an invalid pose coming from that ﬂip region of the latent manifold.
Finally, we showcase some problematic behaviors that can be caused by extreme look-at targets.
In some cases, especially with many other constraints, ProtoRes will tend to produce a plausible pose that will not respect the look-at constraint.
In other cases, the extreme look-at target may cause an unrealistic pose, e.g. by causing the character to have an impossible neck rotation.
It is interesting to note how invalid poses from look-at effectors tend to happen more often than from other effector types with novice users.
We hypothesize that the plausible region of a look-at target, given a current character pose, is less intuitive to grasp than for other effector types.
Indeed, the current pose of the character seems to guide more precisely the placement of positional and rotational effectors than look-at effectors, leading more often to conﬁgurations outside of the training distribution for look-at effectors.
L L IMITATIONS
The limitations of our work can be summarized as follows:
• Constraints are not satisﬁed exactly, as opposed to the conventional systems.
This is the price to pay for the ability of the model to inject the data-driven inductive bias that can be used to reconstruct pose from very sparse inputs.
This could be mitigated using a conventional solver on top of the trained model.
In this case, the model will produce a globally plausible pose, whereas the solver will only do the ﬁnal pass to strictly satisfy certain constraints.
Also, to provide additional ﬂexibility in solving some of the constraints more strictly than the others, our model provides an effector tolerance mechanism that can help the user trade off the strictness of satisfying certain effectors vs. some others.
• Lack of temporal consistency.
Our work solves the problem of creating a discrete pose.
Therefore, it is limited in how it can be applied to modify an underlying smooth animation clip.
For example, we can see ﬂickering of joints (especially ﬁngers) when effectors follow an underlying smooth animation (the ﬁnger embedding space is not smooth and has a high ambiguity).
This happens to a smaller degree with the head when it is not constrained with look-at or rotation inputs.
• Exotic poses signiﬁcantly deviating from the the training data distribution (a common ML/DL problem) may be hard to achieve.
For example, the Lotus yoga pose is very hard to achieve with small number of effectors.
Extreme or rare effector conﬁgurations may not be respected.
Extreme look-at targets may not be followed or can cause artifacts in the resulting pose.
• Some effector displacement can cause a complete ﬂip in the ﬁnal pose as it makes more sense to be e.g. left-oriented or right-oriented to reach a hand position.
This is normal, but we can sometimes reach ”in-between” poses on the boundary of the hand effector that causes the ﬂip, leading to weird poses
We also noted a limitation as ”aiming” poses (holding something in the hands).
For example, Finger poses are generally wrong w.r.t.
to a gun without additional ﬁnger constraints.
It may be cumbersome to place hands + look-at for each aiming pose/angle?
• Runtime.
In its current state, the model allows interactive real-time rate (about 100 FPS).
This is very good for the primary application area of the model in the interactive pose design.
However, the current model cannot be used for runtime applications such as driving
charachters directly in real-time games, because it would consume too high of a time budget (about 10ms, which is too much to be usable in the game runtime context).
• No contextual input is supported (text description or environment awareness), in particular for ﬁnger posing and feet collisions • Good for realism, but might limit creativity.
In particular, no bone stretching support, which is sometimes used by animators to add more expressiveness to non-realistic characters.
• The current model struggles when a large number of ﬁne-grain controls, especially ﬁngers are used simultaneously.
Perhaps, a more structured hierarchical approach can be used to enable this functionality.