Original: D IFF M IMIC : E FFICIENT M OTION M IMICKING WITH D IFFERENTIABLE P HYSICS
Revised: D IFF M IMIC : E FFICIENT M OTION M IMICKING WITH D IFFERENTIABLE P HYSICS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Motion mimicking is a foundational task in physics-based character animation.
Revised: Motion mimicking is a foundational task in physics-based character animation.

Original: However, most existing motion mimicking methods are built upon reinforcement learning (RL) and suffer from heavy reward engineering, high variance, and slow convergence with hard explorations.
Revised: However, most existing motion mimicking methods are built upon reinforcement learning (RL) and suffer from heavy reward engineering, high variance, and slow convergence with hard explorations.

Original: Specifically, they usually take tens of hours or even days of training to mimic a simple motion sequence, resulting in poor scalability.
Revised: Specifically, they usually take tens of hours or even days of training to mimic a simple motion sequence, resulting in poor scalability.

Original: In this work, we leverage differentiable physics simulators (DPS) and propose an efficient motion mimicking method dubbed DiffMimic .
Revised: In this work, we leverage differentiable physics simulators (DPS) and propose an efficient motion mimicking method dubbed DiffMimic .

Original: Our key insight is that DPS casts a complex policy learning task to a much simpler state matching problem.
Revised: Our key insight is that DPS casts a complex policy learning task to a much simpler state matching problem.

Original: In particular, DPS learns a stable policy by analytical gradients with ground-truth physical priors hence leading to significantly faster and stabler convergence than RL-based methods.
Revised: In particular, DPS learns a stable policy by analytical gradients with ground-truth physical priors hence leading to significantly faster and stabler convergence than RL-based methods.

Original: Moreover, to escape from local optima, we utilize an Demonstration Replay mechanism to enable stable gradient backpropagation in a long horizon.
Revised: Moreover, to escape from local optima, we utilize an Demonstration Replay mechanism to enable stable gradient backpropagation in a long horizon.

Original: Extensive experiments on standard benchmarks show that DiffMimic has a better sample efficiency and time efficiency than existing methods ( e.g. , DeepMimic).
Revised: Extensive experiments on standard benchmarks show that DiffMimic has a better sample efficiency and time efficiency than existing methods ( e.g. , DeepMimic).

Original: Notably, DiffMimic allows a physically simulated character to learn back-flip after 10 minutes of training and be able to cycle it after 3 hours of training, while DeepMimic requires about a day of training to cycle back-flip.
Revised: Notably, DiffMimic allows a physically simulated character to learn back-flip after 10 minutes of training and be able to cycle it after 3 hours of training, while DeepMimic requires about a day of training to cycle back-flip.

Original: More importantly, we hope DiffMimic can benefit more differentiable animation systems with techniques like differentiable clothes simulation in future research.
Revised: More importantly, we hope DiffMimic can benefit more differentiable animation systems with techniques like differentiable clothes simulation in future research.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Motion mimicking aims to find a policy to generate control signals for recovering demonstrated motion trajectories, which plays a fundamental role in physics-based character animation, and also serves as a prerequisite for many applications such as control stylization and skill composition.
Revised: Motion mimicking aims to find a policy to generate control signals for recovering demonstrated motion trajectories, which plays a fundamental role in physics-based character animation, and also serves as a prerequisite for many applications such as control stylization and skill composition.

Original: Although tremendous progress in motion mimicking has been witnessed in recent years, existing methods (Peng et al., 2018a; 2021) mostly adopt reinforcement learning (RL) schemes, which require alternatively learning a reward function and a control policy.
Revised: Although tremendous progress in motion mimicking has been witnessed in recent years, existing methods (Peng et al., 2018a; 2021) mostly adopt reinforcement learning (RL) schemes, which require alternatively learning a reward function and a control policy.

Original: Consequently, RL-based methods often take tens of hours or even days to imitate one single motion sequence, making their scalability notoriously challenging.
Revised: Consequently, RL-based methods often take tens of hours or even days to imitate one single motion sequence, making their scalability notoriously challenging.

Original: In addition, RL-based motion mimicking highly relies on the quality of its designed (Peng et al., 2018a) or learned (Peng et al., 2021) reward functions, which further burdens its generalization for complex real-world applications.
Revised: In addition, RL-based motion mimicking highly relies on the quality of its designed (Peng et al., 2018a) or learned (Peng et al., 2021) reward functions, which further burdens its generalization for complex real-world applications.

Original: Recently, differential physics simulator (DPS) has achieved impressive results in many research fields, such as robot control (Xu et al., 2022) and graphics (Li et al., 2022).
Revised: Recently, differential physics simulator (DPS) has achieved impressive results in many research fields, such as robot control (Xu et al., 2022) and graphics (Li et al., 2022).

Original: Specifically, DPS treats physics operators as differentiable computational graphs, and therefore gradients from objectives ( i.e. , rewards) can be directly propagated through the environment dynamics to control policy functions.
Revised: Specifically, DPS treats physics operators as differentiable computational graphs, and therefore gradients from objectives ( i.e. , rewards) can be directly propagated through the environment dynamics to control policy functions.

Original: Instead of alternatively learning between reward functions and control policies, the control policy learning tasks can be resolved in a straightforward and efficient optimization manner with the help of DPS.
Revised: Instead of alternatively learning between reward functions and control policies, the control policy learning tasks can be resolved in a straightforward and efficient optimization manner with the help of DPS.

Original: However, despite their analytical environment gradients, optimization with DPS could easily get into local optima, particularly in contact-rich physical systems that often yield stiff and discontinuous gradients (Freeman et al., 2021; Suh et al., 2022; Zhong et al., 2022).
Revised: However, despite their analytical environment gradients, optimization with DPS could easily get into local optima, particularly in contact-rich physical systems that often yield stiff and discontinuous gradients (Freeman et al., 2021; Suh et al., 2022; Zhong et al., 2022).

Original: Besides, numerical gradients could also vanish/explode along the backward path for long trajectories.
Revised: Besides, numerical gradients could also vanish/explode along the backward path for long trajectories.

Original: 𝑠̂ !"
Revised: 𝑠̂ !"

Original: # 𝑠 !"
Revised: # 𝑠 !"

Original: # 𝑠 !
Revised: # 𝑠 !

Original: 𝑠 !
Revised: 𝑠 !

Original: $# 𝑠̂ !
Revised: $# 𝑠̂ !

Original: 𝑠̂ !
Revised: 𝑠̂ !

Original: $#
Revised: $#

Original: [Figure]
Revised: [Figure]

Original: In this work, we propose DiffMimic, a fast and stable motion mimicking method with the help of DPS.
Revised: In this work, we propose DiffMimic, a fast and stable motion mimicking method with the help of DPS.

Original: Different from RL-based methods that require heave reward engineering and poor sample efficiency, DiffMimic reformulates motion mimicking as a state matching problem, which could directly minimize the distance between a rollout trajectory generated by the current learning policy and the demonstrated trajectory.
Revised: Different from RL-based methods that require heave reward engineering and poor sample efficiency, DiffMimic reformulates motion mimicking as a state matching problem, which could directly minimize the distance between a rollout trajectory generated by the current learning policy and the demonstrated trajectory.

Original: Thanks to the differentiable DPS dynamics, gradients of the trajectory distance can be directly propagated to optimize the control policy.
Revised: Thanks to the differentiable DPS dynamics, gradients of the trajectory distance can be directly propagated to optimize the control policy.

Original: As a result, DiffMimic could significantly improve the sample efficiency with the first-order gradients.
Revised: As a result, DiffMimic could significantly improve the sample efficiency with the first-order gradients.

Original: However, simply utilizing DPS could not guarantee global optimal solutions.
Revised: However, simply utilizing DPS could not guarantee global optimal solutions.

Original: In particular, the rollout trajectory tends to gradually deviate from the expert demonstration and could produce a large accumulative error for long motion sequences, due to the distributional shift between the learning policy and expert policy.
Revised: In particular, the rollout trajectory tends to gradually deviate from the expert demonstration and could produce a large accumulative error for long motion sequences, due to the distributional shift between the learning policy and expert policy.

Original: To address these problems, we introduce the Demonstration Replay training strategy, which randomly inserts reference states into the rollout trajectory as anchor states to guide the exploration of the policy.
Revised: To address these problems, we introduce the Demonstration Replay training strategy, which randomly inserts reference states into the rollout trajectory as anchor states to guide the exploration of the policy.

Original: Empirically, Demonstration Replay gives a smoother gradient estimation, which significantly stabilizes the policy learning of DiffMimic.
Revised: Empirically, Demonstration Replay gives a smoother gradient estimation, which significantly stabilizes the policy learning of DiffMimic.

Original: To the best of our knowledge, DiffMimic is the first to utilize DPS for motion mimicking.
Revised: To the best of our knowledge, DiffMimic is the first to utilize DPS for motion mimicking.

Original: We show that DiffMimic outperforms several commonly used RL-based methods for motion mimicking on a variety of tasks with high accuracy, stability, and efficiency.
Revised: We show that DiffMimic outperforms several commonly used RL-based methods for motion mimicking on a variety of tasks with high accuracy, stability, and efficiency.

Original: In particular, DiffMimic allows learning a challenging back-flip motion in only 10 minutes on a single V100 GPU.
Revised: In particular, DiffMimic allows learning a challenging back-flip motion in only 10 minutes on a single V100 GPU.

Original: In addition, we release the
Revised: In addition, we release the

Original: DiffMimic simulator as a standard benchmark to encourage future research for motion mimicking.
Revised: DiffMimic simulator as a standard benchmark to encourage future research for motion mimicking.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Motion Mimicking.
Revised: Motion Mimicking.

Original: To produce realistic animation, motion mimicking is employed to learn skills from motion captures (Hong et al., 2019; Peng et al., 2018a;b; Lee et al., 2019) for physicsbased characters.
Revised: To produce realistic animation, motion mimicking is employed to learn skills from motion captures (Hong et al., 2019; Peng et al., 2018a;b; Lee et al., 2019) for physicsbased characters.

Original: Motion mimicking enables various downstream tasks in physics-based animation.
Revised: Motion mimicking enables various downstream tasks in physics-based animation.

Original: Learned primitive actions can be re-combined to generate various motions (Peng et al., 2019; Luo et al., 2020), used to achieve a goal (Bergamin et al., 2019;
Revised: Learned primitive actions can be re-combined to generate various motions (Peng et al., 2019; Luo et al., 2020), used to achieve a goal (Bergamin et al., 2019;

Original: Park et al., 2019; Peng et al., 2021), or leveraged as a pre-train model for general-purpose motor skills (Merel et al., 2018; Hasenclever et al., 2020; Peng et al., 2022; Won et al., 2022; ? ).
Revised: Park et al., 2019; Peng et al., 2021), or leveraged as a pre-train model for general-purpose motor skills (Merel et al., 2018; Hasenclever et al., 2020; Peng et al., 2022; Won et al., 2022).

Original: While these downstream tasks look exciting and promising, the scalability can be largely limited by motion mimicking.
Revised: While these downstream tasks look exciting and promising, the scalability can be largely limited by motion mimicking.

Original: In our work, we show the problem can be surprisingly easy with differentiable dynamics.
Revised: In our work, we show the problem can be surprisingly easy with differentiable dynamics.

Original: Speeding Up Motion
Revised: Speeding Up Motion

Original: Mimicking Most motion mimicking works are based on a DRL framework (Peng et al., 2018a; Bergamin et al., 2019), whose optimization is expensive.
Revised: Mimicking Most motion mimicking works are based on a DRL framework (Peng et al., 2018a; Bergamin et al., 2019), whose optimization is expensive.

Original: Several recent works speed up the DRL process by hyper-parameter searching (Yang & Yin, 2021) and constraint relaxation (Ma et al., 2021).
Revised: Several recent works speed up the DRL process by hyper-parameter searching (Yang & Yin, 2021) and constraint relaxation (Ma et al., 2021).

Original: Another line of work learns world models to achieve end-to-end gradient optimization (Won et al., 2022; Fussell et al., 2021).
Revised: Another line of work learns world models to achieve end-to-end gradient optimization (Won et al., 2022; Fussell et al., 2021).

Original: However, learning world models requires extra training and introduces the risk of error accumulation.
Revised: However, learning world models requires extra training and introduces the risk of error accumulation.

Original: We are the first to use the off-the-shelf
Revised: We are the first to use the off-the-shelf

Original: DPS for motion mimicking, which achieves better sample efficiency than optimized DRL frameworks and requires no additional world models.
Revised: DPS for motion mimicking, which achieves better sample efficiency than optimized DRL frameworks and requires no additional world models.

Original: [Figure]
Revised: [Figure]

Original: Differentiable Physics Simulators.
Revised: Differentiable Physics Simulators.

Original: The emerging differentiable physics simulators (DPS) (Hu et al., 2019; 2020; Huang et al., 2021; Qiao et al., 2021) have brought success in many domains and provide a distinctly new approach to control tasks.
Revised: The emerging differentiable physics simulators (DPS) (Hu et al., 2019; 2020; Huang et al., 2021; Qiao et al., 2021) have brought success in many domains and provide a distinctly new approach to control tasks.

Original: Various applications stem from DPS like system identification (Jatavallabhula et al., 2021), continuous controls (Lin et al., 2022; Xu et al., 2022) in the past few years and demonstrate promising results.
Revised: Various applications stem from DPS like system identification (Jatavallabhula et al., 2021), continuous controls (Lin et al., 2022; Xu et al., 2022) in the past few years and demonstrate promising results.

Original: Compared to black-box counterparts which learn the dynamics with neural networks (Fussell et al., 2021; Li et al., 2020), differentiable simulations utilize physical models to provide more reliable gradients with better interpretability.
Revised: Compared to black-box counterparts which learn the dynamics with neural networks (Fussell et al., 2021; Li et al., 2020), differentiable simulations utilize physical models to provide more reliable gradients with better interpretability.

Original: On the other hand, the analytical gradient from DPS suffers from noise or can sometimes be wrong (Zhong et al., 2022; Suh et al., 2022) in a contact-rich environment.
Revised: On the other hand, the analytical gradient from DPS suffers from noise or can sometimes be wrong (Zhong et al., 2022; Suh et al., 2022) in a contact-rich environment.

Original: Non-smoothness or discontinuity in the contact event requires specific techniques to compute the gradient.
Revised: Non-smoothness or discontinuity in the contact event requires specific techniques to compute the gradient.

Original: However, they often introduce noise into the gradient (Zhong et al., 2022).
Revised: However, they often introduce noise into the gradient (Zhong et al., 2022).

Original: Thus, in this work, we empirically show how to handle the noise in the gradient in a contact-rich environment for motion mimicking.
Revised: Thus, in this work, we empirically show how to handle the noise in the gradient in a contact-rich environment for motion mimicking.

Original: Policy Learning with Differentiable Physics Simulators.
Revised: Policy Learning with Differentiable Physics Simulators.

Original: The recent development of differentiable simulators enables the optimization of control policies via analytical gradient (Mozer, 1989;
Revised: The recent development of differentiable simulators enables the optimization of control policies via analytical gradient (Mozer, 1989;

Original: Mora et al., 2021; Xu et al., 2022).
Revised: Mora et al., 2021; Xu et al., 2022).

Original: Concurrent to this work, ILD (Chen et al., 2022) proposes a simple yet effective Imitation Learning framework with DPS for robot control.
Revised: Concurrent to this work, ILD (Chen et al., 2022) proposes a simple yet effective Imitation Learning framework with DPS for robot control.

Original: While the first-order gradient largely enhances the sample efficiency for policy learning, challenges lie ahead.
Revised: While the first-order gradient largely enhances the sample efficiency for policy learning, challenges lie ahead.

Original: The noisy gradient and exploding/vanishing gradients are two crucial issues that hinder the wide deployment of analytical gradient-based reinforcement learning (Degrave et al., 2017).
Revised: The noisy gradient and exploding/vanishing gradients are two crucial issues that hinder the wide deployment of analytical gradient-based reinforcement learning (Degrave et al., 2017).

Original: Due to the local nature of the gradient, the policy could be stuck at sub-optimal points.
Revised: Due to the local nature of the gradient, the policy could be stuck at sub-optimal points.

Original: In particular, we observe that ILD-like optimization fails to let characters explore highly dynamic motions.
Revised: In particular, we observe that ILD-like optimization fails to let characters explore highly dynamic motions.

Original: We introduce a simple replay mechanism to alleviate these issues.
Revised: We introduce a simple replay mechanism to alleviate these issues.

Original:  M ETHODOLOGY 3. A M OTION M IMICKING E NVIRONMENT IN D IFFERENTIABLE P HYSICS E NGINE .
Revised:  M ETHODOLOGY 3. A M OTION M IMICKING E NVIRONMENT IN D IFFERENTIABLE P HYSICS E NGINE .

Original: Environment.
Revised: Environment.

Original: We build our environment in Brax (Freeman et al., 2021).
Revised: We build our environment in Brax (Freeman et al., 2021).

Original: We design our simulated character following DeepMimic (Peng et al., 2018a).
Revised: We design our simulated character following DeepMimic (Peng et al., 2018a).

Original: The humanoid character has 13 links and 34 degrees of freedom.
Revised: The humanoid character has 13 links anddegrees of freedom.

Original: The character has a weight of 45 kg and a height of 1.62m. Contact is applied to all links with the floor.
Revised: The character has a weight of 45 kg and a height of 1.62m. Contact is applied to all links with the floor.

Original: The environment is accelerated by GPU parallelization.
Revised: The environment is accelerated by GPU parallelization.

Original: The physics simulator updates at a rate of 480 FPS.
Revised: The physics simulator updates at a rate of 480 FPS.

Original: The joint limits of the character are relaxed to allow smoother gradient propagation.
Revised: The joint limits of the character are relaxed to allow smoother gradient propagation.

Revised: We keep the system configurations like friction coefficients consistent with

Revised: DeepMimic.

Original: State and Action.
Revised: State and Action.

Original: States include the position p , rotation q , linear velocity ˙ p and angular velocity ˙ q of all links in the local coordinate.
Revised: States include the position p , rotation q , linear velocity ˙ p and angular velocity ˙ q of all links in the local coordinate.

Original: Additionally, following Peng et al.
Revised: Additionally, following Peng et al.

Original: (2018a), a phase variable ϕ in the range [0 , 1] is included in the state to serve as a timestamp.
Revised: (2018a), a phase variable ϕ in the range [0 , 1] is included in the state to serve as a timestamp.

Original: Thus, the state of the environment, s , contains the aforementioned information, s := { p, q, ˙ p, ˙ q, ϕ } .
Revised: Thus, the state of the environment, s , contains the aforementioned information, s := { p, q, ˙ p, ˙ q, ϕ } .

Original: We follow the common practice and use PD controllers to drive the character.
Revised: We follow the common practice and use PD controllers to drive the character.

Original: Given the current joint angle q , angular velocity ˙ q and a target angle ˜ q , the torque on the actuator of the joint will be computed as: 
Revised: Given the current joint angle q , angular velocity ˙ q and a target angle ˜ q , the torque on the actuator of the joint will be computed as: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where ˜˙ q = 0 , k p and k d are manually-specified gains of the PD controller.
Revised: where ˜˙ q = 0 , k p and k d are manually-specified gains of the PD controller.

Original: We keep k p and k d identical to the PD controller used in DeepMimic.
Revised: We keep k p and k d identical to the PD controller used in DeepMimic.

Original: A policy network predicts the target angle for the
Revised: A policy network predicts the target angle for the

Original: PD controller at each joint.
Revised: PD controller at each joint.

Original: The control policy operates at 30 FPS.
Revised: The control policy operates at 30 FPS.

Original: M OTION M IMICKING WITH D IFFERENTIABLE P HYSICS
Revised: M OTION M IMICKING WITH D IFFERENTIABLE P HYSICS

Original: [Table]
Revised: [Table]

Original: Motion Mimicking is about matching the policy rollout with the reference motion.
Revised: Motion Mimicking is about matching the policy rollout with the reference motion.

Original: While it has a straightforward objective, considering human action can be rich and highly flexible, designing a reward to incentivize and guide policy learning is not easy.
Revised: While it has a straightforward objective, considering human action can be rich and highly flexible, designing a reward to incentivize and guide policy learning is not easy.

Original: For example, reward functions that are tailored to guide a policy to learn to walk or learn to back-flip can be different and this makes reward engineering difficult.
Revised: For example, reward functions that are tailored to guide a policy to learn to walk or learn to back-flip can be different and this makes reward engineering difficult.

Original: We reveal that the mimicking task can be surprisingly easy when analytical gradients can be obtained through DPS.
Revised: We reveal that the mimicking task can be surprisingly easy when analytical gradients can be obtained through DPS.

Original: The underlying idea of DiffMimic is to allow the gradient to directly flow back from the distance between reference motion and policy rollout to the policy through an offthe-shelf DPS.
Revised: The underlying idea of DiffMimic is to allow the gradient to directly flow back from the distance between reference motion and policy rollout to the policy through an offthe-shelf DPS.

Original: We show the computation graph of DiffMimic in Fig.
Revised: We show the computation graph of DiffMimic in Fig.

Original: Compared with existing RL-based approaches, DiffMimic directly optimizes the policy with supervised state-matching signals and avoids reward engineering in any form.
Revised: Compared with existing RL-based approaches, DiffMimic directly optimizes the policy with supervised state-matching signals and avoids reward engineering in any form.

Original: Thanks to the analytical gradient, such an approach has a very high sample efficiency.
Revised: Thanks to the analytical gradient, such an approach has a very high sample efficiency.

Original: Prior works (Fussell et al., 2021) explored analytic gradients with a learned differentiable world model which describes the dynamics in latent space.
Revised: Prior works (Fussell et al., 2021) explored analytic gradients with a learned differentiable world model which describes the dynamics in latent space.

Original: However, the learned world model reveals no underlying physics of the system.
Revised: However, the learned world model reveals no underlying physics of the system.

Original: Consequently, they undermine the effectiveness, generality, and interpretability of the learning with physics-based analytic gradients from DPS, as in DiffMimic.
Revised: Consequently, they undermine the effectiveness, generality, and interpretability of the learning with physics-based analytic gradients from DPS, as in DiffMimic.

Original: In addition, online learning and inference with learned world models remain computationally expensive when compared with DPS.
Revised: In addition, online learning and inference with learned world models remain computationally expensive when compared with DPS.

Original: We show in detail how to train the policy with DPS.
Revised: We show in detail how to train the policy with DPS.

Original: For each iteration, we initialize the state to the first reference state and then roll out to the maximum episode length in a batch of parallel environments.
Revised: For each iteration, we initialize the state to the first reference state and then roll out to the maximum episode length in a batch of parallel environments.

Original: Then we compute the step-wise L2 distance between the rollout trajectory and the reference trajectory in terms of positions and rotations:
Revised: Then we compute the step-wise L2 distance between the rollout trajectory and the reference trajectory in terms of positions and rotations:

Original: [Equation]
Revised: [Equation]

Original: where p j and ˆ p j are the global positions of the j -th joint from rollout and reference, q j and ˆ q j are the global rotation of the j -th joint from rollout and reference in 6D rotation representation.
Revised: where p j and ˆ p j are the global positions of the j -th joint from rollout and reference, q j and ˆ q j are the global rotation of the j -th joint from rollout and reference in 6D rotation representation.

Original: Note that we use 6D rotation in loss computation since it is advantageous over quaternions in gradient-based optimization (Zhou et al., 2019).
Revised: Note that we use 6D rotation in loss computation since it is advantageous over quaternions in gradient-based optimization (Zhou et al., 2019).

Original: Without loss of generality, we denote the transition function in the dynamic system as T and the next state of the system can be obtained from the transition function given the current action a t and character state s t , s t +1 =
Revised: Without loss of generality, we denote the transition function in the dynamic system as T and the next state of the system can be obtained from the transition function given the current action a t and character state s t , s t +1 =

Original: T ( s t , a t ) .
Revised: T ( s t , a t ) .

Original: In DiffMimic, DPS serves as a transition function T that is fully differentiable.
Revised: In DiffMimic, DPS serves as a transition function T that is fully differentiable.

Original: Thus, from the loss function, we can directly derive the gradient with respect to both the current action a t and state s t
Revised: Thus, from the loss function, we can directly derive the gradient with respect to both the current action a t and state s t

Original: [Equation]
Revised: [Equation]

Original: By doing this recursively, the gradient can be propagated through the whole trajectory.
Revised: By doing this recursively, the gradient can be propagated through the whole trajectory.

Original: 3. D EMONSTRATION R EPLAY
Revised: 3. D EMONSTRATION R EPLAY

Original: Although mimicking with DPS results in a succinct learning framework, there are three well-known challenges in policy learning with DPS: 1) Exploding/vanishing gradients with the long horizon; 2)
Revised: Although mimicking with DPS results in a succinct learning framework, there are three well-known challenges in policy learning with DPS: 1) Exploding/vanishing gradients with the long horizon; 2)

Original: Local minima may cause gradient-based optimization methods to stall; 3) Noisy or wrong gradients.
Revised: Local minima may cause gradient-based optimization methods to stall; 3) Noisy or wrong gradients.

Original: [Figure]
Revised: [Figure]

Original: We observe that the high non-convexity in the motion mimicking task poses severe challenges to analytical gradient-based optimization.
Revised: We observe that the high non-convexity in the motion mimicking task poses severe challenges to analytical gradient-based optimization.

Original: Due to the local nature of the analytical gradients, the policy can be easily stuck at suboptimal points.
Revised: Due to the local nature of the analytical gradients, the policy can be easily stuck at suboptimal points.

Original: As shown Tab. 6, when trying to mimic the Back-Flip skill, the character learns to use arms to support itself instead of exploring a more dynamic jumping move.
Revised: As shown Tab. 6, when trying to mimic the Back-Flip skill, the character learns to use arms to support itself instead of exploring a more dynamic jumping move.

Original: In practical implementations, dividing the whole trajectory into smaller sub-trajectories for truncated
Revised: In practical implementations, dividing the whole trajectory into smaller sub-trajectories for truncated

Original: Back-Propagation-Through-Time (BPTT) further exacerbates the issue.
Revised: Back-Propagation-Through-Time (BPTT) further exacerbates the issue.

Original: As shown in Tab. 7 (a)-(b), a 10-step gradient truncation leads the policy to a less favorable local optimal.
Revised: As shown in Tab. 7 (a)-(b), a 10-step gradient truncation leads the policy to a less favorable local optimal.

Original: Teacher forcing (Williams & Zipser, 1989) is a common remedy to escape the local minimum in gradient-based sequential modeling.
Revised: Teacher forcing (Williams & Zipser, 1989) is a common remedy to escape the local minimum in gradient-based sequential modeling.

Original: Teacher forcing randomly replace the states in rollout by the reference states.
Revised: Teacher forcing randomly replace the states in rollout by the reference states.

Original: Given a teacher forcing ratio γ :
Revised: Given a teacher forcing ratio γ :

Original: [Equation]
Revised: [Equation]

Original: However, although the mechanism leads to a better global optimal overall, it does not guarantee the character to faithfully mimic the reference motion in each frame.
Revised: However, although the mechanism leads to a better global optimal overall, it does not guarantee the character to faithfully mimic the reference motion in each frame.

Original: As shown in Tab. 6, although the character’s overall trajectory matches the reference, several frames suffer from awkward poses.
Revised: As shown in Tab. 6, although the character’s overall trajectory matches the reference, several frames suffer from awkward poses.

Original: Fig.
Revised: Fig.

Original: 8 shows that a few frames have significantly larger pose errors than other frames.
Revised: 8 shows that a few frames have significantly larger pose errors than other frames.

Original: We propose demonstration-guided exploration to mitigate these challenges in DPS for policy learning to help exploration and encourage faithfully mimicking at the same time.
Revised: We propose demonstration-guided exploration to mitigate these challenges in DPS for policy learning to help exploration and encourage faithfully mimicking at the same time.

Original: The main idea of demonstration-guided exploration is to replace states in the policy rollout with the corresponding demonstration states when the states are too far away from the reference.
Revised: The main idea of demonstration-guided exploration is to replace states in the policy rollout with the corresponding demonstration states when the states are too far away from the reference.

Original: For a threshold ϵ :
Revised: For a threshold ϵ :

Original: [Equation]
Revised: [Equation]

Original: Since the criterion for choosing which states to replace is based on the performance of the current rollout, the replacing frequency dynamic adjusts itself during the model training.
Revised: Since the criterion for choosing which states to replace is based on the performance of the current rollout, the replacing frequency dynamic adjusts itself during the model training.

Original:  E XPERIMENTS 4. E XPERIMENT S ETUP
Revised:  E XPERIMENTS 4. E XPERIMENT S ETUP

Original: For all the experiments, we run the algorithm with one single GPU (NVIDIA Tesla V100) and CPU (Intel Xeon E5-2680).
Revised: For all the experiments, we run the algorithm with one single GPU (NVIDIA Tesla V100) and CPU (Intel Xeon E5-2680).

Original: Following Peng et al.
Revised: Following Peng et al.

Original: (2021), we use the average pose error as the main metric.
Revised: (2021), we use the average pose error as the main metric.

Original: The average pose error over the whole trajectory of length T with J joints is computed between the pose of the simulated character and the reference motion using the relative positions of each joint with respect to the root joint (in units of meters):
Revised: The average pose error over the whole trajectory of length T with J joints is computed between the pose of the simulated character and the reference motion using the relative positions of each joint with respect to the root joint (in units of meters):

Original: [Equation]
Revised: [Equation]

Original: where p jt and ˆ p jt are the position of j -th joint of the simulated motion and reference motion at timestamp t in the 3D cartesian space and root refers to the root joint.
Revised: where p jt and ˆ p jt are the position of j -th joint of the simulated motion and reference motion at timestamp t in the 3D cartesian space and root refers to the root joint.

Original: We mainly compare with
Revised: We mainly compare with

Original: DeepMimic (Peng et al., 2018a), Spacetime Bound (Ma et al., 2021), and Adversarial Motion Prior (AMP)
Revised: DeepMimic (Peng et al., 2018a), Spacetime Bound (Ma et al., 2021), and Adversarial Motion Prior (AMP)

Original: (Peng et al., 2021).
Revised: (Peng et al., 2021).

Original: The Dynamic Time Warping (Sakoe & Chiba, 1978) is applied to synchronize the simulated motion and the reference motion following the convention.
Revised: The Dynamic Time Warping (Sakoe & Chiba, 1978) is applied to synchronize the simulated motion and the reference motion following the convention.

Original: Cyclic Motions.
Revised: Cyclic Motions.

Original: Besides mimicking a single motion clip, a popular benchmark for motion mimicking is to imitate cyclic motions like walking and running, where a single cycle of the motion is provided as the reference motion and the character learns to repeat the motion for 20 seconds.
Revised: Besides mimicking a single motion clip, a popular benchmark for motion mimicking is to imitate cyclic motions like walking and running, where a single cycle of the motion is provided as the reference motion and the character learns to repeat the motion for 20 seconds.

Original: Typically, a curriculum is designed to gradually increase the maximum rollout episode to 20 seconds during training (Yang & Yin, 2021; Peng et al., 2018a).
Revised: Typically, a curriculum is designed to gradually increase the maximum rollout episode to 20 seconds during training (Yang & Yin, 2021; Peng et al., 2018a).

Original: In our experiment, we remove all bells and whistles and fix the maximum rollout episode to 4 seconds during the training.
Revised: In our experiment, we remove all bells and whistles and fix the maximum rollout episode to 4 seconds during the training.

Original: DiffMimic can produce a 20-second long cyclic rollout even though it only sees 4-second rollouts in training.
Revised: DiffMimic can produce a 20-second long cyclic rollout even though it only sees 4-second rollouts in training.

Revised: The motion clips are directly borrowed from AMP Peng et al.

Revised: (2021), which are originally collected from a combination of public mocap libraries, custom recorded mocap clips, and artist-authored keyframe animations.

Original: 4. C OMPARISON ON M OTION M IMICKING
Revised: 4. C OMPARISON ON M OTION M IMICKING

Original: In this section, we aim to understand the 1) efficiency; 2) quality of the learned policy of
Revised: In this section, we aim to understand the 1) efficiency; 2) quality of the learned policy of

Original: DiffMimic.
Revised: DiffMimic.

Original: Following the conventions of previous works (Peng et al., 2018a; 2021; Ma et al., 2021), we count the number of samples required to train the policy to rollout for 20 seconds without falling.
Revised: Following the conventions of previous works (Peng et al., 2018a; 2021; Ma et al., 2021), we count the number of samples required to train the policy to rollout for 20 seconds without falling.

Original: The pose error is calculated over the horizon of 20 seconds.
Revised: The pose error is calculated over the horizon of 20 seconds.

Original: [Figure]
Revised: [Figure]

Original: training time versus pose error for DiffMimic and

Original: AMP.

Original: DiffMimic takes half of training time required by AMP to have a comparable result.

Original: Analytical gradients enhance the sample efficiency in motion mimicking.
Revised: Analytical gradients enhance the sample efficiency in motion mimicking.

Original: We show the comparison between
Revised: We show the comparison between

Original: DiffMimic, DeepMimic, and Spacetime Bound on sample efficiency in Table 2.
Revised: DiffMimic, DeepMimic, and Spacetime Bound on sample efficiency in Table 2.

Original: DeepMimic is an RL-based algorithm with a careful reward design.
Revised: DeepMimic is an RL-based algorithm with a careful reward design.

Original: Spacetime Bound performs hyperparameter searching for DeepMimic to further enhance the sample efficiency.
Revised: Spacetime Bound performs hyperparameter searching for DeepMimic to further enhance the sample efficiency.

Original: Our results show that
Revised: Our results show that

Original: DiffMimic constantly outperforms DeepMimic in terms of sample efficiency.
Revised: DiffMimic constantly outperforms DeepMimic in terms of sample efficiency.

Original: The analytical gradients provided by the differentiable simulation allow us to compute policy gradient with a small number of samples while the RL-based algorithm requires a large batch to have a decent estimate.
Revised: The analytical gradients provided by the differentiable simulation allow us to compute policy gradient with a small number of samples while the RL-based algorithm requires a large batch to have a decent estimate.

Original: Compared with Spacetime Bound, DiffMimic is much more stable and consistent over various tasks.
Revised: Compared with Spacetime Bound, DiffMimic is much more stable and consistent over various tasks.

Original: We notice that
Revised: We notice that

Original: Spacetime Bound may require more samples than DeepMimic even for simple tasks like Jump .
Revised: Spacetime Bound may require more samples than DeepMimic even for simple tasks like Jump .

Original: We show the learning curve of DiffMimic over eight different tasks in Fig. 
Revised: We show the learning curve of DiffMimic over eight different tasks in Fig. It shows that DiffMimic generally learns [Table]

Original: It shows that DiffMimic generally learns high-quality motions (pose error < 0.15) with less than 2e7 samples, even for challenging tasks like
Revised: [Table] [Table] high-quality motions (pose error < 0.15) with less than 2e7 samples, even for challenging tasks like

Original: Back-flip .
Revised: Back-flip .

Original: In terms of the wallclock time, DiffMimic learns to perform Back-flip in 10 minutes and learns to cycle it in 3 hours ( 14 . 88 × 10 6 samples), which can be found in our qualitative results.
Revised: In terms of the wallclock time, DiffMimic learns to perform Back-flip in 10 minutes and learns to cycle it in 3 hours ( 14 . 88 × 10 6 samples)

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: which can be found in our qualitative results.

Original: DiffMimic can learn high-quality motions.
Revised: DiffMimic can learn high-quality motions.

Original: The average pose error for 12 different motion tasks is shown in Table 1.
Revised: The average pose error for 12 different motion tasks is shown in Table 1.

Original: DiffMimic outperforms AMP consistently and has comparable performance to
Revised: DiffMimic outperforms AMP consistently and has comparable performance to

Original: DeepMimic.
Revised: DeepMimic.

Original: Noticeably, DiffMimic only needs to see the demonstrations of 4 seconds to achieve a similar performance of DeepMimic with 20 seconds cyclic rollout [Table] which indicates stable and faithful recovery of the reference motion.
Revised: Noticeably, DiffMimic only needs to see the demonstrations of 4 seconds to achieve a similar performance of DeepMimic with 20 seconds cyclic rollout, which indicates stable and faithful recovery of the reference motion.

Original: This further corroborates the efficacy of DiffMimic.
Revised: This further corroborates the efficacy of DiffMimic.

Original: Analytical gradients accelerate the learning speed in motion mimicking.
Revised: Analytical gradients accelerate the learning speed in motion mimicking.

Original: As it generally takes more time to compute analytical gradients than the estimated one, it is natural to compare the wall-clock time for each algorithm.
Revised: As it generally takes more time to compute analytical gradients than the estimated one, it is natural to compare the wall-clock time for each algorithm.

Original: Since the implementation of DeepMimic does not utilize GPU for parallelization, we choose to compare DiffMimic with AMP.
Revised: Since the implementation of DeepMimic does not utilize GPU for parallelization, we choose to compare DiffMimic with AMP.

Original: The implementation of AMP utilizes the highly parallelized environment, IsaacGym (Makoviychuk et al., 2021).
Revised: The implementation of AMP utilizes the highly parallelized environment, IsaacGym (Makoviychuk et al., 2021).

Original: Both approaches do not require manual reward design and utilize GPU for acceleration.
Revised: Both approaches do not require manual reward design and utilize GPU for acceleration.

Original: We compare DiffMimic and AMP on a task where a humanoid character is wielding a sword and a sheild to perform a 3.2-second attack move.
Revised: We compare DiffMimic and AMP on a task where a humanoid character is wielding a sword and a sheild to perform a 3.2-second attack move.

Original: The hyperparameter of AMP is from its official release of code.
Revised: The hyperparameter of AMP is from its official release of code.

Original: We show the comparison with respect to wallclock time in Fig.
Revised: We show the comparison with respect to wallclock time in Fig.

Original: The simulated motion is demonstrated in Fig.
Revised: The simulated motion is demonstrated in Fig.

Original: DiffMimic takes half of training time required by AMP to have a comparable result
Revised: DiffMimic takes half of training time required by AMP to have a comparable result.

Original: [Table]

Original: 4. A BLATION ON T RUNCATION LENGTH
Revised: 4. A BLATION ON T RUNCATION LENGTH

Original: Training the policy with DPS would suffer from the vanishing/exploding gradients and local optimal.
Revised: Training the policy with DPS would suffer from the vanishing/exploding gradients and local optimal.

Original: Recent research (Xu et al., 2022) points out that such a problem can be mitigated by a truncated learning window, which splits the entire trajectory into segments with shorter horizons.
Revised: Recent research (Xu et al., 2022) points out that such a problem can be mitigated by a truncated learning window, which splits the entire trajectory into segments with shorter horizons.

Original: we carried  out experiments to validate whether such an idea can be directly applied in motion mimicking with
Revised: we carried [Figure] [Figure] out experiments to validate whether such an idea can be directly applied in motion mimicking with

Original: DPS.
Revised: DPS.

Original: Truncation length refers to the horizon over which the gradient is calculated.
Revised: Truncation length refers to the horizon over which the gradient is calculated.

Original: The quantitative results are shown in Fig.
Revised: The quantitative results are shown in Fig.

Original: Indeed, it is difficult to learn a good policy by propagating the gradient through the whole trajectory that is long.
Revised: Indeed, it is difficult to learn a good policy by propagating the gradient through the whole trajectory that is long.

Original: In addition, simply splitting the whole trajectory into seg- [Figure] [Figure] ments would worsen the final performance.
Revised: In addition, simply splitting the whole trajectory into segments would worsen the final performance.

Original: We hypothesize the naive truncation of the trajectory creates discontinuities in the whole trajectory whereas motions in the trajectory are highly interdependent.
Revised: We hypothesize the naive truncation of the trajectory creates discontinuities in the whole trajectory whereas motions in the trajectory are highly interdependent.

Original: For example, how to flip in mid-air closely relates to how the character jumps.
Revised: For example, how to flip in mid-air closely relates to how the character jumps.

Original: We see this as a strong call for a better strategy to handle these two challenges in motion mimicking with DPS.
Revised: We see this as a strong call for a better strategy to handle these two challenges in motion mimicking with DPS.

Original: 4. A BLATION ON D EMONSTRATION R EPLAY
Revised: 4. A BLATION ON D EMONSTRATION R EPLAY

Original: To understand how demonstration replay affects the policy learning of DiffMimic, we compare three different variants of DiffMimic, namely, Demonstration Replay (Random) and Demonstration
Revised: To understand how demonstration replay affects the policy learning of DiffMimic, we compare three different variants of DiffMimic, namely, Demonstration Replay (Random) and Demonstration

Original: Replay (Threshold), and Full Horizon Gradient on Back-flip and Cartwheel .
Revised: Replay (Threshold), and Full Horizon Gradient on Back-flip and Cartwheel .

Original: Demonstration Replay (Random) randomly replaces a state in the rollout of the policy with the demonstration state at the same timestamp similar to the teacher forcing (Williams & Zipser, 1989).
Revised: Demonstration Replay (Random) randomly replaces a state in the rollout of the policy with the demonstration state at the same timestamp similar to the teacher forcing (Williams & Zipser, 1989).

Original: Demonstration Replay (Threshold) inserts the demonstration state based on the pose error.
Revised: Demonstration Replay (Threshold) inserts the demonstration state based on the pose error.

Original: If the pose error between the demonstration state and simulated state exceeds a threshold ϵ , the simulated state will be replaced by the demonstration state.
Revised: If the pose error between the demonstration state and simulated state exceeds a threshold ϵ , the simulated state will be replaced by the demonstration state.

Original: The Full Horizon Gradient variant backpropagates the gradient through the full horizon without any additional operation.
Revised: The Full Horizon Gradient variant backpropagates the gradient through the full horizon without any additional operation.

Original: We show the quantitative result in Fig.
Revised: We show the quantitative result in Fig.

Original: 7, and qualitative result in Fig.
Revised: 7, and qualitative result in Fig.

Original: Demonstration replay helps stabilize policy learning and leads to better performance.
Revised: Demonstration replay helps stabilize policy learning and leads to better performance.

Original: Compared with the Full Horizon Gradient without demonstration replay, the learning curve with demonstration replay is much smoother and finally converges to a lower pose error as shown in Fig.
Revised: Compared with the Full Horizon Gradient without demonstration replay, the learning curve with demonstration replay is much smoother and finally converges to a lower pose error as shown in Fig.

Original: 7 (c)-(d).
Revised: 7 (c)-(d).

Original: We show in Fig.
Revised: We show in Fig.

Original: 6 (b), this vanilla version of DiffMimic can easily get stuck into a local minimum.
Revised: 6 (b), this vanilla version of DiffMimic can easily get stuck into a local minimum.

Original: Instead of learning to back-flip, the policy learns to bend down and use arms to support itself instead of jumping up.
Revised: Instead of learning to back-flip, the policy learns to bend down and use arms to support itself instead of jumping up.

Original: The other two variants, by contrast, both learn to jump and back-flip in mid-air successfully.
Revised: The other two variants, by contrast, both learn to jump and back-flip in mid-air successfully.

Original: [Figure]
Revised: [Figure]

Original:  Replay  duces the overall pose error compared to full-horizon optimization, the per-frame loss remains large in certain time steps due to the lack of a constraint.
Revised: Demonstration Replay Random) duces the overall pose error compared to full-horizon optimization, the per-frame loss remains large in cer tain time steps due to the lack of a constraint.

Original: Demonstration Replay (Threshold) alleviates the issue.
Revised: Demonstration Replay (Threshold) alleviates the issue.

Original: Policy-aware demonstration replay leads to a more faithful recovery of demonstration.
Revised: Policy-aware demonstration replay leads to a more faithful recovery of demonstration.

Original: We compare Demonstration Replay (Threshold) with Demonstration Replay (Random) in
Revised: We compare Demonstration Replay (Threshold) with Demonstration Replay (Random) in

Original: Fig.
Revised: Fig.

Original: 7 (e)-(f).
Revised: 7 (e)-(f).

Original: Quantitatively, both variants yield similar results if the hyperparameter is properly tuned.
Revised: Quantitatively, both variants yield similar results if the hyperparameter is properly tuned.

Original: However, the behavior of policies trained with these two strategies can be significantly different.
Revised: However, the behavior of policies trained with these two strategies can be significantly different.

Original: In Fig.
Revised: In Fig.

Original: 6 (c) and (d), though both policies learn to back-flip successfully, the policy trained with Demonstration Replay (Random) fails to recover the motion frame by frame.
Revised: 6 (c) and (d), though both policies learn to back-flip successfully, the policy trained with Demonstration Replay (Random) fails to recover the motion frame by frame.

Original: We show in Fig.
Revised: We show in Fig.

Original: 8 the per-frame pose error.
Revised: 8 the per-frame pose error.

Original: Even the average error over the whole trajectory for both variant are similar, Demonstration Replay (Threshold) gives a lower maximum per-frame error, which indicates a faithful recovery of the demonstration.
Revised: Even the average error over the whole trajectory for both variant are similar, Demonstration Replay (Threshold) gives a lower maximum per-frame error, which indicates a faithful recovery of the demonstration.

Original: This implies that simply minimizing the pose error may not suffice to learn a policy that tracks the demonstration closely.
Revised: This implies that simply minimizing the pose error may not suffice to learn a policy that tracks the demonstration closely.

Original: Finergrained guidance based on the current performance of the policy is required.
Revised: Finergrained guidance based on the current performance of the policy is required.

Original:  C ONCLUSION & F UTURE WORKS
Revised:  C ONCLUSION & F UTURE WORKS

Original: To conclude, we present DiffMimic that utilize DPS for motion mimicking.
Revised: To conclude, we present DiffMimic that utilize DPS for motion mimicking.

Original: We show that DiffMimic outperforms several commonly used RL-based methods for motion mimicking on a variety of tasks with high accuracy, stability, and efficiency.
Revised: We show that DiffMimic outperforms several commonly used RL-based methods for motion mimicking on a variety of tasks with high accuracy, stability, and efficiency.

Original: We hope DiffMimic can be considered as a starting point for motion mimicking with differentiable physics simulators and our simulation environments bring additional future research opportunities to the community.
Revised: We hope DiffMimic can be considered as a starting point for motion mimicking with differentiable physics simulators and our simulation environments bring additional future research opportunities to the community.

Original: There are enormous exciting future directions to be explored, yet, with various challenges.
Revised: There are enormous exciting future directions to be explored, yet, with various challenges.

Original: Given differentiable physics largely accelerates motion mimicking, it would be much more valuable for downstream tasks if motion mimicking can be carried out within minutes given any arbitrary demonstration.
Revised: Given differentiable physics largely accelerates motion mimicking, it would be much more valuable for downstream tasks if motion mimicking can be carried out within minutes given any arbitrary demonstration.

Original: In this work, we demonstrate that DiffMimic can learn challenging motions in 10 minutes.
Revised: In this work, we demonstrate that DiffMimic can learn challenging motions in 10 minutes.

Original: Yet, the horizon of the task is relatively short and does not require any interaction with other objects.
Revised: Yet, the horizon of the task is relatively short and does not require any interaction with other objects.

Original: The dynamic system would get more complex if multiple objects are involved.
Revised: The dynamic system would get more complex if multiple objects are involved.

Original: We leave them for future work.
Revised: We leave them for future work.

Original: E THICS S TATEMENTS
Revised: E THICS S TATEMENTS

Original: We have read the ICLR Code of Ethics.
Revised: We have read the ICLR Code of Ethics.

Original: While DiffMimic carries out experiments on humanoid characters, no human-subject experiment is involved and all experiments are conducted with opensourced materials with references to them.
Revised: While DiffMimic carries out experiments on humanoid characters, no human-subject experiment is involved and all experiments are conducted with opensourced materials with references to them.

Original: R EPRODUCIBILITY S TATEMENTS
Revised: R EPRODUCIBILITY S TATEMENTS

Original: Our code is available at https://github.com/diffmimic/diffmimic.
Revised: Our code is available at https://github.com/diffmimic/diffmimic.

Original: We promise to clean and release the code upon publication. 
Revised: We promise to clean and release the code upon publication. C ONTENTS 1 Introduction

Revised: 2 Related Work 3 Methodology 3.

Original: A PPENDIX A Q UALITATIVE R ESULTS
Revised: A Motion Mimicking Environment in Differentiable Physics Engine. . . . . . . . . Motion Mimicking with Differentiable Physics . . . . . . . . . . . . . . . . . . . Demonstration Replay . . 4 Experiments 4.

Revised: Experiment Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Revised: Comparison on Motion Mimicking . . . . . . . . . . . . . . . . . . . . . . . . . .

Revised: Ablation on Truncation length . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Revised: Ablation on Demonstration Replay . . . . . . . . . . . . . . . . . . . . . . . . . .

Revised: 5 Conclusion & Future works

Revised: A Qualitative Results

Revised: B Policy Network

Revised: C Hyper-parameters

Revised: D State and Action Space

Revised: E Learning Curves

Revised: F New Character

Original: We present our qualitative results in the following website: https://diffmimic-demo main g7h0i streamlitapp
Revised: G New Skill H New Training Strategy I Analysis on Robustness

Revised: J Analysis on Sensitivity

Revised: K Analysis on Motion Quality

Original: .com/
Revised: L More Discussions on DeepMimic Reward M More Discussions on Related Works A Q UALITATIVE R ESULTS We present our qualitative results in the following website: https://diffmimic-demo-maing7h0i8.streamlitapp.com/.

Original: It contains the following qualitative results:
Revised: It contains the following qualitative results:

Original: • Back-Flip 20 second rollout.
Revised: • Back-Flip 20-second rollout.

Original: • Cartwheel 20 second rollout. • Crawl 20 second rollout.
Revised: • Cartwheel 20-second rollout. • Crawl 20-second rollout.

Original: • Dance 20 second rollout. • Jog 20 second rollout.
Revised: • Dance 20-second rollout. • Jog 20-second rollout.

Original: • Jump 20 second rollout. • Roll 20 second rollout.
Revised: • Jump 20-second rollout. • Roll 20-second rollout.

Original: • Side-Flip 20 second rollout. • Spin-Kick 20 second rollout.
Revised: • Side-Flip 20-second rollout. • Spin-Kick 20-second rollout.

Original: • Walk 20 second rollout.
Revised: • Walk 20-second rollout.

Original: • Zombie 20 second rollout.
Revised: • Zombie 20-second rollout.

Original: • Back-Flip 20 second rollout, trained for 2.5 hours.
Revised: • Back-Flip 20-second rollout, trained for 2.5 hours.

Original: • Back-Flip single cycle rollout, trained for 10 minutes.
Revised: • Back-Flip single cycle rollout, trained for 10 minutes.

Original: B P OLICY N ETWORK
Revised: B P OLICY N ETWORK

Original: We use a MLP for the policy network.
Revised: We use an MLP for the policy network.

Original: The network has two hidden layers with size 512 and 256.
Revised: The network has two hidden layers with sizes 512 and 256.

Original: We use Swish (Ramachandran et al., 2017) for  activation function.
Revised: We use Swish (Ramachandran et al., 2017) for the activation function.

Original: C H YPER - PARAMETERS
Revised: C H YPER - PARAMETERS

Original: We use Adam optimizer (Kingma & Ba, 2014) in training, with a learning rate of 3e-4 that linearly decreases with training iterations.
Revised: We use Adam optimizer (Kingma & Ba, 2014) in training, with a learning rate of 3e-4 that linearly decreases with training iterations.

Original: We apply gradient clipping and set the maximum gradient norm to 0.3.
Revised: We apply gradient clipping and set the maximum gradient norm to 0.3.

Original: We set the batch size to the same as the number of parallel environment.
Revised: We set the batch size to the same as the number of parallel environments.

Original: Cyclic motions.
Revised: Cyclic motions.

Original: We set the maximum iterations to 5000.
Revised: We set the maximum iterations to 5000.

Original: The number of environment is set to 200.
Revised: The number of environments is set to 200.

Original: Acyclic motions.
Revised: Acyclic motions.

Original: We set the maximum iterations to 1000.
Revised: We set the maximum iterations to 1000.

Original: The number of environment is set to 300.
Revised: The number of environments is set to 300.

Original: Expert Replay.
Revised: Expert Replay.

Original: We run experiments with two error thresholds, 0.2 and 0.4 and report the better of the two.
Revised: We run experiments with two error thresholds, 0.2 and 0.4, and report the better of the two.

Original: D S TATE AND A CTION S PACE
Revised: D S TATE AND A CTION S PACE

Original: Humanoid.
Revised: Humanoid.

Original: We use 6d rotation representation (Zhou et al., 2019) in the state feature.
Revised: We use 6d rotation representation (Zhou et al., 2019) in the state feature.

Original: The humanoid has an action size of 28 and a state feature size of 193.
Revised: The humanoid has an action size of 28 and a state feature size of 193.

Original: Humanoid with Sword and Shield.
Revised: Humanoid with Sword and Shield.

Original: We set up the Sword and Shield character following ASE (Peng et al., 2022).
Revised: We set up the Sword and Shield character following ASE (Peng et al., 2022).

Original: Compared to the default humanoid, the new characterhas 3 additional DOF on the right hand
Revised: Compared to the default humanoid, the new character

Original: E L EARNING C URVES
Revised: E L EARNING C URVES

Revised: F N EW C HARACTER

Revised: We include Ant as a new character.

Revised: We use the trajectory in ILD (Chen et al., 2022) as the reference motion, which is obtained from a PPO (Schulman et al., 2017) agent that learns to move forward as fast as possible.

Revised: We achieve a 0.059 meter final pose error.

Revised: We show the training curve in Figure 11.

Revised: and the qualitative result on the demo website under the tab ”New Character”.

Revised: [Figure]

Revised: G N EW S KILL

Revised: We run DiffMimic on a new skill, 360-degree Jump , from the CMU Mocap Dataset (motion sequence CMU 075 09).

Revised: The skill has been reported to be challenging for a RL-based method

Revised: CoMic (Hasenclever et al., 2020) in a recent dataset paper (Wagener et al., 2022).

Revised: DiffMimic is able to spin in the air and achieve a 0.020 meter final pose error.

Revised: We show the training curve in

Revised: Figure 12 and the qualitative result on the demo website under the tab ”New Skill”.

Revised: [Figure]

Revised: H N EW T RAINING S TRATEGY

Revised: We additionally include the Reference State Initialization (RSI) technique (Peng et al., 2018a) into the DiffMimic training.

Revised: RSI starts the training rollout from a random reference frame instead of the first reference frame.

Revised: We observe a substantial empirical improvement after employing the new training strategy.

Revised: The samples required are consistently around 90% less than DeepMimic (Peng et al., 2018a) as shown in Tab. 3.

Revised: The final pose error has also been slightly improved as shown in

Revised: Tab. 4.

Revised: We show the qualitative results on the demo website under the tab ”New Training”.

Revised: Notably, using RSI alone or RSI+Early Termination (ET) (Peng et al., 2018a) does not converge as shown in

Revised: Figure 13.

Revised: [Table]

Revised: [Table]

Revised: [Figure]

Revised: I A NALYSIS ON R OBUSTNESS

Revised: We further evaluate DiffMimic’s robustness to external forces.

Revised: A force is applied to the pelvis of the character halfway through a motion cycle for 0.2 seconds

Revised: [Figure]

Revised: and we measure the maximum forwards and sideways push each policy can tolerate before falling.

Revised: DiffMimic achieves comparable robustness to DeepMimic.

Revised: We show the results in Table 5 and the qualitative results on the demo website under the tab ”Robustness”

Revised: [Figure]

Revised: [Table]

Revised: J A NALYSIS ON S ENSITIVITY

Revised: DiffMimic with inaccurate estimates of physical parameters.

Revised: Briefly speaking, DiffMimic is able to learn the control policy well even though the friction coefficient deviates from the original parameters.

Revised: We show the results in Table 6 and the qualitative results on the demo website under the tab ”Sensitivity”.

Revised: [Table]

Revised: K A NALYSIS ON M OTION Q UALITY

Revised: Motion quality can be severely affected by a small number of large-error poses, which can not be reflected by the existing average pose error metric.

Revised: We propose a Pose Absurdity metric to quantify large-error poses.

Revised: L2@0.01, 0.05, and 0.1 measure the average L2 pose error on the worst 1%, 5%

Revised: [Table]

Revised: and 10% frames respectively.

Revised: Full Horizon Gradient, Demonstration Replay (Random) ratio λ , and Demonstration Replay (Threshold) threshold γ are studied.

Revised: Demonstration Replay (Threshold) outperforms other baselines by a large margin.

Revised: We show the results in Table

Revised: L M ORE D ISCUSSIONS ON D EEP M IMIC R EWARD

Revised: DeepMimic (Peng et al., 2018a) defines the reward r

Revised: It for each frame time as:

Revised: [Equation]

Revised: where w are weighting hyper-parameters, r pt , r vt , r et , r ct are reward functions for joint orientation, joint velocity, end-effector position, and center-of-mass position respectively.

Revised: Concretely, they are

Revised: [Table]

Revised: defined as follows:

Revised: [Equation]

Revised: where − 2 , − 0 .

Revised: 1 , − 40 , − 10 are four more weighting hyper-parameters.

Revised: Note that there are 8 hyper-parameters in the reward function in total.

Revised: Moreover, a selection on joints ( e.g. , only apply position reward to end-effectors and center-of-mass) is also required.

Revised: In comparison, DiffMimic uses a direct L2 distance between states without weighting as defined in

Revised: Eq. 3.2, which is much simpler.

Revised: M M ORE D ISCUSSIONS ON R ELATED W ORKS

Revised: SHAC (Xu et al., 2022) and PODS (Mora et al., 2021) are two differentiable physics-based reinforcement learning algorithms.

Revised: SHAC addresses the gradient exploding and diminishing problem by manually truncating trajectories into small segments.

Revised: It requires careful design of gradient truncation throughout the learning process.

Revised: In contrast, DiffMimic automatically resets the state to the demonstrated state, preventing the state from deviating too much and achieving a similar effect to the trajectory truncation in SHAC.

Revised: In addition, SHAC seeks to maximize the reward, which can often lead to a local optimum when the reward function is ill-defined or difficult to design.

Revised: DiffMimic, on the other hand, benefits from the expert demonstration, which provides explicit guidance to escape from local optimal.

Revised: Using second-order gradients, PODS claims a monotonic policy improvement and a significant speedup in convergence over first-order methods.

Revised: Despite its success, PODS makes strong assumptions about the second-order derivatives of a differentiable simulator and can be sensitive to the accuracy of the second-order derivatives.

Revised: Moreover, second-order derivatives incur more computational demands, which can be potentially costly in tasks with complex dynamics.

Revised: Compared to

Revised: SHAC and DiffMimic, PODS can suffer from gradient exploding and diminishing problems without a proper truncation scheme.

Revised: This problem can even be exaggerated with the use of second-order gradients in longer trajectory rollouts.

Revised: Supertrack (Fussell et al., 2021) adopts the idea of model-based RL and proposes to learn a world model to predict the next simulated character state given the previous simulated character state and the provided PD targets.

Revised: It learns the world model and the control policy in an iterative way: 1) the policy rollouts in a physics simulator, 2) the trajectories are collected to train the world model, 3) the policy is then trained through the differentiable world model.

Revised: However, one major disadvantage of the approach is that its policy horizon, i.e., rollout steps, is small, e.g., 32 steps, during training due to the error accumulation in the world model.

Revised: Training can get unstable for long rollouts, e.g.,steps.

Revised: Therefore, as pointed out by the authors, SuperTrack can be unsuitable when the policy needs a long-term apprehension to mimic some challenging motions.

Revised: DiffMimic uses DPS in replace of the learned world model to avoid error accumulation.

Revised: ScaDiver (Won et al., 2020) presents a scalable approach to learning controllers for a large motion database, where motion mimicking is a key part of the pipeline.

Revised: It proposes to 1) divide the diverse motion database into clusters of similar motions, 2) use DeepMimic to learn a control policy for each motion cluster, and 3) learn a mixture-of-expert controllers to dynamically ensemble the control policies.

Revised: We view ScaDiver as an important downstream application of DiffMimic.

Revised: DiffMimic can serve as a better alternative to DeepMimic in step 2, and contribute to a higher efficiency when learning from large-scale motion databases.