Original: D IFF M IMIC : E FFICIENT M OTION M IMICKING WITH D IFFERENTIABLE P HYSICS
Revised: D IFF M IMIC : E FFICIENT M OTION M IMICKING WITH D IFFERENTIABLE P HYSICS

Original: Anonymous authors Paper under double-blind review
Revised: AI

Original: A BSTRACT
Revised: A BSTRACT

Original: Motion mimicking is a foundational task in physics-based character animation.
Revised: Motion mimicking is a foundational task in physics-based character animation.

Original: However, most existing motion mimicking methods are built upon reinforcement learning (RL) and suffer from heavy reward engineering, high variance, and slow convergence with hard explorations.
Revised: However, most existing motion mimicking methods are built upon reinforcement learning (RL) and suffer from heavy reward engineering, high variance, and slow convergence with hard explorations.

Original: Specifically, they usually take tens of hours or even days of training to mimic a simple motion sequence, resulting in poor scalability.
Revised: Specifically, they usually take tens of hours or even days of training to mimic a simple motion sequence, resulting in poor scalability.

Original: In this work, we leverage differentiable physics simulators (DPS) and propose an efficient motion mimicking method dubbed DiffMimic .
Revised: In this work, we leverage differentiable physics simulators (DPS) and propose an efficient motion mimicking method dubbed DiffMimic .

Original: Our key insight is that DPS casts a complex policy learning task to a much simpler state matching problem.
Revised: Our key insight is that DPS casts a complex policy learning task to a much simpler state matching problem.

Original: In particular, DPS learns a stable policy by analytical gradients with ground-truth physical priors hence leading to significantly faster and stabler convergence than RL-based methods.
Revised: In particular, DPS learns a stable policy by analytical gradients with ground-truth physical priors hence leading to significantly faster and stabler convergence than RL-based methods.

Original: Moreover, to escape from local optima, we utilize an Demonstration Replay mechanism to enable stable gradient backpropagation in a long horizon.
Revised: Moreover, to escape from local optima, we utilize an Demonstration Replay mechanism to enable stable gradient backpropagation in a long horizon.

Original: Extensive experiments on standard benchmarks show that DiffMimic has a better sample efficiency and time efficiency than existing methods ( e.g. , DeepMimic).
Revised: Extensive experiments on standard benchmarks show that DiffMimic has a better sample efficiency and time efficiency than existing methods ( e.g. , DeepMimic).

Original: Notably, DiffMimic allows a physically simulated character to learn back-flip after 10 minutes of training and be able to cycle it after 3 hours of training, while DeepMimic requires about a day of training to cycle back-flip.
Revised: Notably, DiffMimic allows a physically simulated character to learn Backflip after 10 minutes of training and be able to cycle it after 3 hours of training, while the existing approach may require about a day of training to cycle Backflip.

Original: More importantly, we hope DiffMimic can benefit more differentiable animation systems with techniques like differentiable clothes simulation in future research.
Revised: More importantly, we hope DiffMimic can benefit more differentiable animation systems with techniques like differentiable clothes simulation in future research.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Motion mimicking aims to find a policy to generate control signals for recovering demonstrated motion trajectories, which plays a fundamental role in physics-based character animation, and also serves as a prerequisite for many applications such as control stylization and skill composition.
Revised: Motion mimicking aims to find a policy to generate control signals for recovering demonstrated motion trajectories, which plays a fundamental role in physics-based character animation, and also serves as a prerequisite for many applications such as control stylization and skill composition.

Original: Although tremendous progress in motion mimicking has been witnessed in recent years, existing methods (Peng et al., 2018a; 2021) mostly adopt reinforcement learning (RL) schemes, which require alternatively learning a reward function and a control policy.
Revised: Although tremendous progress in motion mimicking has been witnessed in recent years, existing methods (Peng et al., 2018a; 2021) mostly adopt reinforcement learning (RL) schemes, which require alternatively learning a reward function and a control policy.

Original: Consequently, RL-based methods often take tens of hours or even days to imitate one single motion sequence, making their scalability notoriously challenging.
Revised: Consequently, RL-based methods often take tens of hours or even days to imitate one single motion sequence, making their scalability notoriously challenging.

Original: In addition, RL-based motion mimicking highly relies on the quality of its designed (Peng et al., 2018a) or learned (Peng et al., 2021) reward functions, which further burdens its generalization for complex real-world applications.
Revised: In addition, RL-based motion mimicking highly relies on the quality of its designed (Peng et al., 2018a) or learned (Peng et al., 2021) reward functions, which further burdens its generalization for complex real-world applications.

Original: Recently, differential physics simulator (DPS) has achieved impressive results in many research fields, such as robot control (Xu et al., 2022) and graphics (Li et al., 2022).
Revised: Recently, differential physics simulator (DPS) has achieved impressive results in many research fields, such as robot control (Xu et al., 2022) and graphics (Li et al., 2022).

Original: Specifically, DPS treats physics operators as differentiable computational graphs, and therefore gradients from objectives ( i.e. , rewards) can be directly propagated through the environment dynamics to control policy functions.
Revised: Specifically, DPS treats physics operators as differentiable computational graphs, and therefore gradients from objectives ( i.e. , rewards) can be directly propagated through the environment dynamics to control policy functions.

Original: Instead of alternatively learning between reward functions and control policies, the control policy learning tasks can be resolved in a straightforward and efficient optimization manner with the help of DPS.
Revised: Instead of alternatively learning between reward functions and control policies, the control policy learning tasks can be resolved in a straightforward and efficient optimization manner with 𝑠̂ !"

Original: However, despite their analytical environment gradients, optimization with DPS could easily get into local optima, particularly in contact-rich physical systems that often yield stiff and discontinuous gradients (Freeman et al., 2021; Suh et al., 2022; Zhong et al., 2022).

Original: Besides, numerical gradients could also vanish/explode along the backward path for long trajectories.

Original: 𝑠̂ !" # 𝑠 !"
Revised: # 𝑠 !"

Original: # 𝑠 !
Revised: # 𝑠 !

Original: 𝑠 !
Revised: 𝑠 !

Original: $# 𝑠̂ !
Revised: $# 𝑠̂ !

Original: 𝑠̂ !
Revised: 𝑠̂ !

Original: $#
Revised: $#

Original: [Figure]
Revised: [Figure]

Revised: the help of DPS.

Revised: However, despite their analytical environment gradients, optimization with DPS could easily get into local optima, particularly in contact-rich physical systems that often yield stiff and discontinuous gradients (Freeman et al., 2021; Suh et al., 2022; Zhong et al., 2022).

Revised: Besides, numerical gradients could also vanish/explode along the backward path for long trajectories.

Original: In this work, we propose DiffMimic, a fast and stable motion mimicking method with the help of DPS.
Revised: In this work, we propose DiffMimic, a fast and stable motion mimicking method with the help of DPS.

Original: Different from RL-based methods that require heave reward engineering and poor sample efficiency, DiffMimic reformulates motion mimicking as a state matching problem, which could directly minimize the distance between a rollout trajectory generated by the current learning policy and the demonstrated trajectory.
Revised: Different from RL-based methods that require heavy reward engineering and poor sample efficiency, DiffMimic reformulates motion mimicking as a state matching problem, which could directly minimize the distance between a rollout trajectory generated by the current learning policy and the demonstrated trajectory.

Original: Thanks to the differentiable DPS dynamics, gradients of the trajectory distance can be directly propagated to optimize the control policy.
Revised: Thanks to the differentiable DPS dynamics, gradients of the trajectory distance can be directly propagated to optimize the control policy.

Original: As a result, DiffMimic could significantly improve the sample efficiency with the first-order gradients.
Revised: As a result, DiffMimic could significantly improve the sample efficiency with the first-order gradients.

Original: However, simply utilizing DPS could not guarantee global optimal solutions.
Revised: However, simply utilizing DPS could not guarantee global optimal solutions.

Original: In particular, the rollout trajectory tends to gradually deviate from the expert demonstration and could produce a large accumulative error for long motion sequences, due to the distributional shift between the learning policy and expert policy.
Revised: In particular, the rollout trajectory tends to gradually deviate from the expert demonstration and could produce a large accumulative error for long motion sequences, due to the distributional shift between the learning policy and expert policy.

Original: To address these problems, we introduce the Demonstration Replay training strategy, which randomly inserts reference states into the rollout trajectory as anchor states to guide the exploration of the policy.
Revised: To address these problems, we introduce the Demonstration Replay training strategy, which randomly inserts reference states into the rollout trajectory as anchor states to guide the exploration of the policy.

Original: Empirically, Demonstration Replay gives a smoother gradient estimation, which significantly stabilizes the policy learning of DiffMimic.
Revised: Empirically, Demonstration Replay gives a smoother gradient estimation, which significantly stabilizes the policy learning of DiffMimic.

Original: To the best of our knowledge, DiffMimic is the first to utilize DPS for motion mimicking.
Revised: To the best of our knowledge, DiffMimic is the first to utilize DPS for motion mimicking.

Original: We show that DiffMimic outperforms several commonly used RL-based methods for motion mimicking on a variety of tasks with high accuracy, stability, and efficiency.
Revised: We show that DiffMimic outperforms several commonly used RL-based methods for motion mimicking on a variety of tasks with high accuracy, stability, and efficiency.

Original: In particular, DiffMimic allows learning a challenging back-flip motion in only 10 minutes on a single V100 GPU.
Revised: In particular, DiffMimic allows learning a challenging Backflip motion in only 10 minutes on a single V100 GPU.

Original: In addition, we release the
Revised: In addition, we release the

Original: DiffMimic simulator as a standard benchmark to encourage future research for motion mimicking.
Revised: DiffMimic simulator as a standard benchmark to encourage future research for motion mimicking.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Motion Mimicking.
Revised: Motion Mimicking.

Original: To produce realistic animation, motion mimicking is employed to learn skills from motion captures (Hong et al., 2019; Peng et al., 2018a;b; Lee et al., 2019) for physicsbased characters.
Revised: Motion mimicking is a technique used to produce realistic animations in physics-based characters by learning skills from motion captures (Hong et al., 2019; Peng et al., 2018a;b; Lee et al., 2019).

Original: Motion mimicking enables various downstream tasks in physics-based animation. Learned primitive actionscan be re-combined to generate various motions (Peng et al., 2019; Luo et al., 2020), used to achieve a goal (Bergamin et al., 2019; Park et al., 2019; Peng et al., 2021), or leveraged as a pre-train model for general-purpose motor skills (Merel et al., 2018; Hasenclever et al., 2020; Peng et al., 2022; Won et al., 2022).
Revised: This approach has been applied to various downstream tasks in physicsbased animation, including generating new motions by recombining primitive actions (Peng et al., 2019; Luo et al., 2020), achieving specific goals (Bergamin et al., 2019; Park et al., 2019; Peng et al., 2021), and as a pre-training method for general-purpose motor skills (Merel et al., 2018; Hasenclever et al., 2020; Peng et al., 2022; Won et al., 2022).

Original: While thesedownstream taskslook exciting and promising, the scalability can belargely limited by  motion mimicking.
Revised: The scalability of these tasks can be limited by the motion mimicking process, which is a key part of the pipeline in approaches like ScaDiver (Won et al., 2020).

Original: In our work, we show the problem  can be surprisingly easy with differentiable dynamics.
Revised: In this work, we demonstrate that the problem of scalability can be addressed using differentiable dynamics.

Original: Speeding Up Motion
Revised: Speeding Up Motion

Original: Mimicking Most motion mimicking works are based on a DRL framework (Peng et al., 2018a; Bergamin et al., 2019), whose optimization is expensive.
Revised: Mimicking Most motion mimicking works are based on a DRL framework (Peng et al., 2018a; Bergamin et al., 2019), whose optimization is expensive.

Original: Several recent works speed up the DRL process by hyper-parameter searching (Yang & Yin, 2021) and constraint relaxation (Ma et al., 2021).
Revised: Several recent works speed up the DRL process by hyper-parameter searching (Yang & Yin, 2021) and constraint relaxation (Ma et al., 2021).

Original: Another line of work learns world models to achieve end-to-end gradient optimization (Won et al., 2022; Fussell et al., 2021).
Revised: Another line of work learns world models to achieve end-to-end gradi- [Figure] ent optimization (Won et al., 2022; Fussell et al., 2021).

Original: However, learning world models requires extra training and introduces the risk of error accumulation.
Revised: However, learning world models requires extra training and introduces the risk of error accumulation.

Original: We are the first to use the off-the-shelf
Revised: We are the first to use the off-the-shelf

Original: DPS for motion mimicking, which achieves better sample efficiency than optimized DRL frameworks and requires no additional world models.[Figure]
Revised: DPS for motion mimicking, which achieves better sample efficiency than optimized DRL frameworks and requires no additional world models.

Original: Differentiable Physics Simulators.
Revised: Differentiable Physics.

Original: The emerging differentiable physics simulators (DPS) (Hu et al., 2019; 2020; Huang et al., 2021; Qiao et al., 2021) have brought success in many domains and provide a distinctly new approach to control tasks.
Revised: Differentiable physics gains traction recently with the emerging differentiable physics simulators (DPS) (Hu et al., 2019; 2020; Huang et al., 2021; Qiao et al., 2021). It has brought success in many domains and provides a distinctly new approach to control tasks.

Revised: Supertrack (Fussell et al., 2021) proposes to learn a world model to approximate the differentiable physics and has achieved promising results.

Revised: However, its policy horizon is limited during training due to the accumulation of errors in the world model.

Original: Various applications stem from DPS like system identification (Jatavallabhula et al., 2021), continuous controls (Lin et al., 2022; Xu et al., 2022) in the past few years and demonstrate promising results.
Revised: Various applications stem from DPS like system identification (Jatavallabhula et al., 2021), continuous controls (Lin et al., 2022; Xu et al., 2022) in the past few years and demonstrate promising results.

Original: Compared to black-box counterparts which learn the dynamics with neural networks (Fussell et al., 2021; Li et al., 2020), differentiable simulations utilize physical models to provide more reliable gradients with better interpretability.
Revised: Compared to black-box counterparts which learn the dynamics with neural networks (Fussell et al., 2021; Li et al., 2020), differentiable simulations utilize physical models to provide more reliable gradients with better interpretability.

Original: On the other hand, the analytical gradient from DPS suffers from noise or can sometimes be wrong (Zhong et al., 2022; Suh et al., 2022) in a contact-rich environment.
Revised: On the other hand, the analytical gradient from DPS suffers from noise or can sometimes be wrong (Zhong et al., 2022; Suh et al., 2022) in a contact-rich environment.

Original: Non-smoothness or discontinuity in the contact event requires specific techniques to compute the gradient.
Revised: Non-smoothness or discontinuity in the contact event requires specific techniques to compute the gradient.

Original: However, they often introduce noise into the gradient (Zhong et al., 2022).
Revised: However, they often introduce noise into the gradient (Zhong et al., 2022).

Original: Thus, in this work, we empirically show how to handle the noise in the gradient in a contact-rich environment for motion mimicking.
Revised: Thus, in this work, we empirically show how to handle the noise in the gradient in a contact-rich environment for motion mimicking.

Original: Policy Learning with Differentiable Physics Simulators.
Revised: Policy Optimization with Differentiable Physics Simulators.

Original: The recent development of differentiable simulators enables the optimization of control policies via analytical gradient  (Mozer, 1989; Mora et al., 2021; Xu et al., 2022).Concurrent to this work, ILD (Chen et al., 2022) proposes a simple yet effective Imitation Learning framework with DPS for robot control.
Revised: The integration of differentiable simulators in policy optimization has been a significant advancement in reinforcement learning. With analytical gradient calculation, the sample efficiency of policy learning is significantly improved (Mozer, 1989; Mora et al., 2021; Xu et al., 2022).

Original: While the first-order gradient largely enhances the sample efficiency for policy learning, challenges lie ahead.

Original: The noisygradient and exploding/vanishing gradients are two crucial issues that hinder the wide deployment of analytical gradient-based reinforcement learning (Degrave et al., 2017).
Revised: However, the implementation is challenged by the presence of noisy gradients and the risk of exploding or vanishing gradients (Degrave et al., 2017).

Original: Due to the local nature of the gradient, the policy could be stuck at sub-optimal points.
Revised: The policy may also get stuck at sub-optimal points due to the local nature of the gradient.

Revised: To address these issues, various approaches have been proposed.

Revised: SHAC (Xu et al., 2022) truncates trajectories into smaller segments to prevent exploding or vanishing gradients, but this requires careful design.

Revised: On the other hand, PODS (Mora et al., 2021) utilizes second-order gradients, leading to monotonic policy improvement and faster convergence compared to first-order methods.

Original: In particular, we observe that ILD-like optimization fails to let characters explore highly dynamic motions.
Revised: However, it relies on strong assumptions about the second-order derivatives of the differentiable simulator and is sensitive to their accuracy. An alternative approach is the Imitation Learning framework with DPS proposed by ILD (Chen et al., 2022) Although it is simple and effective for robot control, it struggles with the exploration of highly dynamic motions.

Original: We introduce asimple replay mechanism to alleviate these issues.
Revised: To tackle these challenges, we introduce a replay mechanism to enhance its performance.

Original:  M ETHODOLOGY 3. A M OTION M IMICKING E NVIRONMENT IN D IFFERENTIABLE P HYSICS E NGINE .
Revised:  M ETHODOLOGY 3. A M OTION M IMICKING E NVIRONMENT IN D IFFERENTIABLE P HYSICS E NGINE .

Original: Environment.
Revised: Environment.

Original: We build our environment in Brax (Freeman et al., 2021).
Revised: We build our environment in Brax (Freeman et al., 2021).

Original: We design our simulated character following DeepMimic (Peng et al., 2018a).
Revised: We design our simulated character following DeepMimic (Peng et al., 2018a).

Original: The humanoid character has 13 links anddegrees of freedom.
Revised: The humanoid character has 13 links anddegrees of freedom.

Original: The character has a weight of 45 kg and a height of 1.62m. Contact is applied to all links with the floor.
Revised: The character has a weight of 45 kg and a height of 1.62m. Contact is applied to all links with the floor.

Original: The environment is accelerated by GPU parallelization.
Revised: The environment is accelerated by GPU parallelization.

Original: The physics simulator updates at a rate of 480 FPS.
Revised: The physics simulator updates at a rate of 480 FPS.

Original: The joint limits of the character are relaxed to allow smoother gradient propagation.
Revised: The joint limits of the character are relaxed to allow smoother gradient propagation.

Original: We keep the system configurations like friction coefficients consistent with
Revised: We keep the system configurations like friction coefficients consistent with

Original: DeepMimic.
Revised: DeepMimic.

Original: State and Action.
Revised: State and Action.

Original: States include the position p , rotation q , linear velocity ˙ p and angular velocity ˙ q of all links in the local coordinate.
Revised: States include the position p , rotation q , linear velocity ˙ p and angular velocity ˙ q of all links in the local coordinate.

Original: Additionally, following Peng et al.
Revised: Additionally, following Peng et al.

Original: (2018a), a phase variable ϕ in the range [0 , 1] is included in the state to serve as a timestamp.
Revised: (2018a), a phase variable ϕ in the range [0 , 1] is included in the state to serve as a timestamp.

Original: Thus, the state of the environment, s , contains the aforementioned information, s := { p, q, ˙ p, ˙ q, ϕ } .
Revised: Thus, the state of the environment, s , contains the aforementioned information, s := { p, q, ˙ p, ˙ q, ϕ } .

Original: We follow the common practice and use PD controllers to drive the character.
Revised: We follow the common practice and use PD controllers to drive the character.

Original: Given the current joint angle q , angular velocity ˙ q and a target angle ˜ q , the torque on the actuator of the joint will be computed as:[Equation]
Revised: Given the current joint angle q , angular velocity ˙ q and a target angle ˜ q , the torque on the actuator of the joint will be computed as:

Original: [Equation]
Revised: [Equation]

Original: where ˜˙ q = 0 , k p and k d are manually-specified gains of the PD controller.
Revised: where ˜˙ q = 0 , k p and k d are manually-specified gains of the PD controller.

Original: We keep k p and k d identical to the PD controller used in DeepMimic.
Revised: We keep k p and k d identical to the PD controller used in DeepMimic.

Original: A policy network predicts the target angle for the
Revised: A policy network predicts the target angle for the

Original: PD controller at each joint.
Revised: PD controller at each joint.

Original: The control policy operates at 30 FPS.
Revised: The control policy operates at 30 FPS.

Original:  M OTION M IMICKING WITH D IFFERENTIABLE P HYSICS
Revised: 3. M OTION M IMICKING WITH D IFFERENTIABLE P HYSICS

Original: [Table]
Revised: [Table]

Revised: = (cid:40) T ( s t , a t ) , a t ∼ π

Revised: [Table]

Revised: ( a | s

Revised: [Table]

Revised: ) if

Revised: [Table]

Revised: s t − ˆ s t ∥ 22 <

Revised: [Table]

Revised: T (ˆ s t , a t ) , a t ∼ π θ ( a |

Original: Motion Mimicking is about matching the policy rollout with the reference motion.
Revised: Motion Mimicking is about matching the policy rollout with the reference motion.

Original: While it has a straightforward objective, considering human action can be rich and highly flexible, designing a reward to incentivize and guide policy learning is not easy.
Revised: While it has a straightforward objective, considering human action can be rich and highly flexible, designing a reward to incentivize and guide policy learning is not easy.

Original: For example, reward functions that are tailored to guide a policy to learn to walk or learn to back-flip can be different and this makes reward engineering difficult.
Revised: For example, reward functions that are tailored to guide a policy to learn to walk or learn to backflip can be different and this makes reward engineering difficult.

Original: We reveal that the mimicking task can be surprisingly easy when analytical gradients can be obtained through DPS.
Revised: We reveal that the mimicking task can be surprisingly easy when analytical gradients can be obtained through DPS.

Original: The underlying idea of DiffMimic is to allow the gradient to directly flow back from the distance between reference motion and policy rollout to the policy through an offthe-shelf DPS.
Revised: The underlying idea of DiffMimic is to allow the gradient to directly flow back from the distance between reference motion and policy rollout to the policy through an offthe-shelf DPS.

Original: We show the computation graph of DiffMimic in Fig.
Revised: We show the computation graph of DiffMimic in Fig.

Original: Compared with existing RL-based approaches, DiffMimic directly optimizes the policy with supervised state-matching signals and avoids reward engineering in any form.
Revised: Compared with existing RL-based approaches, DiffMimic directly optimizes the policy with supervised state-matching signals and largely eases the reward engineering.

Original: Thanks to the analytical gradient, such an approach has a very high sample efficiency.
Revised: Thanks to the analytical gradient, such an approach has a very high sample efficiency.

Original: Prior works (Fussell et al., 2021) explored analytic gradients with a learned differentiable world model which describes the dynamics in latent space.
Revised: Prior works (Fussell et al., 2021) explored analytic gradients with a learned differentiable world model which describes the dynamics in latent space.

Original: However, the learned world model reveals no underlying physics of the system.
Revised: However, the learned world model reveals no underlying physics of the system.

Original: Consequently, they undermine the effectiveness, generality, and interpretability of the learning with physics-based analytic gradients from DPS, as in DiffMimic.
Revised: Consequently, they undermine the effectiveness, generality, and interpretability of the learning with physics-based analytic gradients from DPS, as in DiffMimic.

Original: In addition, online learning and inference with learned world models remain computationally expensive when compared with DPS.
Revised: In addition, online learning and inference with learned world models remain computationally expensive when compared with DPS.

Original: We show in detail how to train the policy with DPS.
Revised: We show in detail how to train the policy with DPS.

Original: For each iteration, we initialize the state to the first reference state and then roll out to the maximum episode length in a batch of parallel environments.
Revised: For each iteration, we initialize the state to the first reference state and then roll out to the maximum episode length in a batch of parallel environments.

Original: Then we compute the step-wise L2 distance between the rollout trajectory and the reference trajectory in terms of positions and rotations:
Revised: Then we compute the step-wise L2 distance between the rollout trajectory and the reference trajectory in terms of positions and rotations:

Original: [Equation]
Revised: [Equation]

Original: where p j and ˆ p j are the global positions of the j -th joint from rollout and reference, q j and ˆ q j are the global rotation of the j -th joint from rollout and reference in 6D rotation representation. 
Revised: where p j and ˆ p j are the global positions of the j -th joint from rollout and reference, q j and ˆ q j are the global rotation of the j -th joint from rollout and reference in 6D rotation representation. The weights

Revised: [Equation]

Revised: w p , w r , w v , and w a only need to be approximately tuned to roughly equalize their magnitudes.

Original: Note that we use 6D rotation in loss computation since it is advantageous over quaternions in gradient-based optimization (Zhou et al., 2019).
Revised: Note that we use 6D rotation in loss computation since it is advantageous over quaternions in gradientbased optimization (Zhou et al., 2019).

Original: Without loss of generality, we denote the transition function in the dynamic system as T and the next state of the system can be obtained from the transition function given the current action a t and character state s t , s t +1 =
Revised: Without loss of generality, we denote the transition function in the dynamic system as T and the next state of the system can be obtained from the transition function given the current action a t and character state s t , s t +1 =

Original: T ( s t , a t ) .
Revised: T ( s t , a t ) .

Original: In DiffMimic, DPS serves as a transition function T that is fully differentiable.
Revised: In DiffMimic, DPS serves as a transition function T that is fully differentiable.

Original: Thus, from the loss function, we can directly derive the gradient with respect to both the current action a t and state s t
Revised: Thus, from the loss function, we can directly derive the gradient with respect to both the current action a t and state s t

Original: [Equation]
Revised: [Equation]

Original: By doing this recursively, the gradient can be propagated through the whole trajectory.
Revised: By doing this recursively, the gradient can be propagated through the whole trajectory.

Original: 3. D EMONSTRATION R EPLAY
Revised: 3. D EMONSTRATION R EPLAY

Original: Although mimicking with DPS results in a succinct learning framework, there are three well-known challenges in policy learning with DPS: 1) Exploding/vanishing gradients with the long horizon; 2)
Revised: Although mimicking with DPS results in a succinct learning framework, there are three well-known challenges in policy learning with DPS: 1) Exploding/vanishing gradients with the long horizon; 2)

Original: Local minima may cause gradient-based optimization methods to stall; 3) Noisy or wrong gradients.
Revised: Local minima may cause gradient-based optimization methods to stall; 3) Noisy or wrong gradients.

Original: [Figure] We observe that the high non-convexity in the motion mimicking task poses severe challenges to analytical gradient-based optimization.
Revised: We observe that the high non-convexity in the motion mimicking task poses severe challenges to analytical gradient-based optimization.

Original: Due to the local nature of the analytical gradients, the policy can be easily stuck at suboptimal points.
Revised: Due to the local nature of the analytical gradients, the policy can be easily stuck at suboptimal points.

Original: As shown Tab. 6, when trying to mimic the Back-Flip skill, the character learns to use arms to support itself instead of exploring a more dynamic jumping move.
Revised: As shown Tab. 6, when trying to mimic the Backflip skill, the character learns to use arms to support itself instead of exploring a more dynamic jumping move.

Original: In practical implementations, dividing the whole trajectory into smaller sub-trajectories for truncated
Revised: In practical implementations, dividing the whole trajectory into smaller sub-trajectories for truncated

Original: Back-Propagation-Through-Time (BPTT) further exacerbates the issue.
Revised: Back-Propagation-Through-Time (BPTT) further exacerbates the issue.

Original: As shown in Tab. 7 (a)-(b), a 10-step gradient truncation leads the policy to a less favorable local optimal.
Revised: As shown in Tab. 7 (a)-(b), a 10-step gradient truncation leads the policy to a less favorable local optimal.

Original: Teacher forcing (Williams & Zipser, 1989) is a common remedy to escape the local minimum in gradient-based sequential modeling.
Revised: Teacher forcing (Williams & Zipser, 1989) is a common remedy to escape the local minimum in gradient-based sequential modeling.

Original: Teacher forcing randomly replace the states in rollout by the reference states.
Revised: Teacher forcing randomly replace the states in rollout by the reference states.

Original: Given a teacher forcing ratio γ :
Revised: Given a teacher forcing ratio γ :

Original: [Equation]
Revised: [Equation]

Original: However, although the mechanism leads to a better global optimal overall, it does not guarantee the character to faithfully mimic the reference motion in each frame.
Revised: However, although the mechanism leads to a better global optimal overall, it does not guarantee the character to faithfully mimic the reference motion in each frame.

Original: As shown in Tab. 6, although the character’s overall trajectory matches the reference, several frames suffer from awkward poses.
Revised: As shown in Tab. 6, although the character’s overall trajectory matches the reference, several frames suffer from awkward poses.

Original: Fig.
Revised: Fig.

Original: 8 shows that a few frames have significantly larger pose errors than other frames.
Revised: 8 shows that a few frames have significantly larger pose errors than other frames.

Original: We propose demonstration-guided exploration to mitigate these challenges in DPS for policy learning to help exploration and encourage faithfully mimicking at the same time.
Revised: We propose demonstration-guided exploration to mitigate these challenges in DPS for policy learning to help exploration and encourage faithfully mimicking at the same time.

Original: The main idea of demonstration-guided exploration is to replace states in the policy rollout with the corresponding demonstration states when the states are too far away from the reference.
Revised: The main idea of demonstration-guided exploration is to replace states in the policy rollout with the corresponding demonstration states when the states are too far away from the reference.

Original: For a threshold ϵ :
Revised: For a threshold ϵ :

Original: [Equation]
Revised: [Equation]

Original: Since the criterion for choosing which states to replace is based on the performance of the current rollout, the replacing frequency dynamic adjusts itself during the model training.
Revised: Since the criterion for choosing which states to replace is based on the performance of the current rollout, the replacing frequency dynamic adjusts itself during the model training.

Revised: We refer the readers for more results to the appendix.

Original:  E XPERIMENTS 4. E XPERIMENT S ETUP
Revised:  E XPERIMENTS 4. E XPERIMENT S ETUP

Original: For all the experiments, we run the algorithm with one single GPU (NVIDIA Tesla V100) and CPU (Intel Xeon E5-2680).
Revised: For all the experiments, we run the algorithm with one single GPU (NVIDIA Tesla V100) and CPU (Intel Xeon E5-2680).

Original: Following Peng et al.
Revised: Following Peng et al.

Original: (2021), we use the average pose error as the main metric.
Revised: (2021), we use the average pose error as the main metric.

Original: The average pose error over the whole trajectory of length T with J joints is computed between the pose of the simulated character and the reference motion using the relative positions of each joint with respect to the root joint (in units of meters):
Revised: The average pose error over the whole trajectory of length T with J joints is computed between the pose of the simulated character and the reference motion using the relative positions of each joint with respect to the root joint (in units of meters):

Original: [Equation]
Revised: [Equation]

Original: where p jt and ˆ p jt are the position of j -th joint of the simulated motion and reference motion at timestamp t in the 3D cartesian space and root refers to the root joint.
Revised: where p jt and ˆ p jt are the position of j -th joint of the simulated motion and reference motion at timestamp t in the 3D cartesian space and root refers to the root joint.

Original: We mainly compare with
Revised: We mainly compare with

Original: DeepMimic (Peng et al., 2018a), Spacetime Bound (Ma et al., 2021), and Adversarial Motion Prior (AMP)
Revised: DeepMimic (Peng et al., 2018a), Spacetime Bound (Ma et al., 2021), and Adversarial Motion Prior (AMP)

Original: (Peng et al., 2021).
Revised: (Peng et al., 2021).

Original: The Dynamic Time Warping (Sakoe & Chiba, 1978) is applied to synchronize the simulated motion and the reference motion following the convention.
Revised: Dynamic Time Warping (Sakoe & Chiba, 1978) is applied to synchronize the simulated motion and the reference motion following the convention.

Original: Cyclic Motions.
Revised: Cyclic Motions.

Original: Besides mimicking a single motion clip, a popular benchmark for motion mimicking is to imitate cyclic motions like walking and running, where a single cycle of the motion is provided as the reference motion and the character learns to repeat the motion for 20 seconds.
Revised: Besides mimicking a single motion clip, a popular benchmark for motion mimicking is to imitate cyclic motions like walking and running, where a single cycle of the motion is provided as the reference motion and the character learns to repeat the motion for 20 seconds.

Original: Typically, a curriculum is designed to gradually increase the maximum rollout episode to 20 seconds during training (Yang & Yin, 2021; Peng et al., 2018a).
Revised: Typically, a curriculum is designed to gradually increase the maximum rollout episode to 20 seconds during training (Yang & Yin, 2021; Peng et al., 2018a).

Original: In our experiment, we remove all bells and whistles and fix the maximum rollout episode to 4 seconds during the training.
Revised: In our experiment, we remove all bells and whistles and fix the maximum rollout episode to 4 seconds during the training.

Original: DiffMimic can produce a 20-second long cyclic rollout even though it only sees 4-second rollouts in training.
Revised: DiffMimic can produce a 20-second long cyclic rollout even though it only sees 4-second rollouts in training.

Original: The motion clips are directly borrowed from AMP Peng et al. (2021), which are originally collected from a combination of public mocap libraries, custom recorded mocap clips, and artist-authored keyframe animations.
Revised: The motion clips are directly borrowed from AMP (Peng et al., 2021), which are originally collected from a combination of public mocap libraries, custom recorded mocap clips, and artist-authored keyframe animations.

Original: 4. C OMPARISON ON M OTION M IMICKING
Revised: 4. C OMPARISON ON M OTION M IMICKING

Original: In this section, we aim to understandthe 1) efficiency; 2)  quality of the learned policy of DiffMimic.
Revised: In this section, we aim to understand 1) efficiency; 2) the quality of the learned policy of DiffMimic.

Original: Following the conventions of previous works (Peng et al., 2018a; 2021; Ma et al., 2021), we count the number of samples required to train the policy to rollout for 20 seconds without falling.
Revised: Following the conventions of previous works (Peng et al., 2018a; 2021; Ma et al., 2021), we count the number of samples required to train the policy to rollout for 20 seconds without falling.

Original: The pose error is calculated over the horizon of 20 seconds.[Figure]
Revised: The pose error is calculated over the horizon of 20 seconds.

Original: Analytical gradients enhance the sample efficiency in motion mimicking.
Revised: Analytical gradients enhance the sample efficiency in motion mimicking.

Original: We show the comparison between DiffMimic, DeepMimic, and Spacetime Bound on sample efficiency in Table 2.
Revised: We show the comparison between DiffMimic, DeepMimic, and Spacetime Bound on sample efficiency in Table 2.

Original: DeepMimic is an RL-based algorithm with a careful reward design.
Revised: DeepMimic is an RL-based algorithm with a careful reward design.

Original: Spacetime Bound performs hyperparameter searching for DeepMimic to further enhance the sample efficiency.
Revised: Spacetime Bound performs hyperparameter searching for DeepMimic to further enhance the sample efficiency.

Original: Our results show that
Revised: Our results [Table]

Original:  DiffMimic constantly outperforms DeepMimic in terms of sample efficiency.
Revised: show that DiffMimic constantly outperforms DeepMimic in terms of sample efficiency.

Original: The analytical gradients provided by the differentiable simulation allow us to compute policy gradient with a small number of samples while the RL-based algorithm requires a large batch to have a decent estimate.
Revised: The analytical gradients provided by the differentiable simulation allow us to compute policy gradient with a small number of samples while the RL-based algorithm requires a large batch to have a decent estimate.

Original: Compared with Spacetime Bound, DiffMimic is much more stable and consistent over various tasks.
Revised: Compared with Spacetime Bound, DiffMimic is much more stable and consistent over various tasks.

Original: We notice that Spacetime Bound may require more samples than DeepMimic even for simple tasks like Jump .
Revised: We notice that Spacetime Bound may require more samples than DeepMimic even for simple tasks like Jump .

Original: We show the learning curve of DiffMimic over eight different tasks in Fig.It shows that DiffMimic generally learns [Table]
Revised: We show the learning curve of DiffMimic over eight different tasks in Fig.

Original: [Table] [Table] high-quality motions (pose error < 0.15) with less than 2e7 samples, even for challenging tasks like 
Revised: It shows that DiffMimic generally learns high-quality motions (pose error < 0.15) with less than 2e7 samples, even for challenging tasks like Backflip .

Original: Back-flip . In terms of the wallclock time, DiffMimic learns to perform Back-flip in 10 minutes and learns to cycle it in 3 hours ( 14 . 88 × 10 6 samples)
Revised: In terms of the wallclock time, DiffMimic learns to perform Backflip in 10 minutes and learns to cycle it in 3 hours ( 14 . 88 × 10 6 samples), which can be found in our qualitative results.

Original: [Table]
Revised: [Figure]

Original: which can be found in our qualitative results.
Revised: versus pose error for DiffMimic and AMP. DiffMimic takes half of the training time required by AMP to have a comparable result.

Original: DiffMimic can learn high-quality motions.
Revised: DiffMimic can learn high-quality motions.

Original: The average pose error for 12 different motion tasks is shown in Table 1.
Revised: The average pose error for 12 different motion tasks is shown in Table 1.

Original: DiffMimic outperforms AMP consistently and has comparable performance to DeepMimic.
Revised: DiffMimic outperforms AMP consistently and has comparable performance to DeepMimic.

Original: Noticeably, DiffMimic only needs to see the demonstrations of 4 seconds to achieve a similar performance of DeepMimic with 20 seconds cyclic rollout, which indicates stable and faithful recovery of the reference motion.
Revised: Noticeably, DiffMimic only needs to see the demonstrations of 4 seconds to achieve a similar performance of DeepMimic with 20 seconds cyclic rollout, which indicates stable and faithful recovery of the reference motion.

Original: This further corroborates the efficacy of DiffMimic.
Revised: This further corroborates the efficacy of DiffMimic.

Original: Analytical gradients accelerate the learning speed in motion mimicking.
Revised: Analytical gradients accelerate the learning speed in motion mimicking.

Original: As it generally takes more time to compute analytical gradients than the estimated one, it is natural to compare the wall-clock time for each algorithm.
Revised: As it generally takes more time to compute analytical gradients than the estimated one, it is natural to compare the wall-clock time for each algorithm.

Original: Since the implementation of DeepMimic does not utilize GPU for parallelization, we choose to compare DiffMimic with AMP.
Revised: Since the implementation of DeepMimic does not utilize GPU for parallelization, we choose to compare DiffMimic with AMP.

Original: The implementation of AMP utilizes the highly parallelized environment, IsaacGym (Makoviychuk et al., 2021).
Revised: The implementation of AMP utilizes the highly parallelized environment, IsaacGym (Makoviychuk et al., 2021).

Original: Both approaches do not require manual reward design and utilize GPU for acceleration.
Revised: Both approaches do not require manual reward design and utilize GPU for acceleration.

Original: We compare DiffMimic and AMP on a task where a humanoid character is wielding a sword and a sheild to perform a 3.2-second attack move.
Revised: We compare DiffMimic and AMP on a task where a humanoid character is wielding a sword and a shield to perform a 3.2-second attack move (shown in Appendix Fig. 13).

Original: The hyperparameter of AMP is from its official release of code.
Revised: The hyperparameter of AMP is from its official release of code.

Original: We show the comparison with respect to wallclock time in Fig.
Revised: We show the comparison with respect to wallclock time in Fig.

Original: The simulated motion is demonstrated in Fig.

Original: DiffMimic takes half of  training time required by AMP to have a comparable result.
Revised: DiffMimic takes half of the training time required by AMP to have a comparable result.

Original: 4. A BLATION ON T RUNCATION LENGTH
Revised: 4. A BLATION ON T RUNCATION LENGTH

Original: Training the policy with DPS would suffer from the vanishing/exploding gradients and local optimal.
Revised: Training the policy with DPS would suffer from the vanishing/exploding gradients and local optimal.

Original: Recent research (Xu et al., 2022) points out that such a problem can be mitigated by a truncated learning window, which splits the entire trajectory into segments with shorter horizons.
Revised: Recent research (Xu et al., 2022) points out that such a problem can be mitigated by a truncated learning window, which splits the entire trajectory into segments with shorter horizons.

Original: we carried 
Revised: we carried out experiments to validate whether such an idea can be directly applied in motion mimicking with

Original: [Figure] [Figure]
Revised: [Table] [Figure]

Original: out experiments to validate whether such an idea can be directly applied in motion mimicking with

Original: DPS.
Revised: DPS.

Original: Truncation length refers to the horizon over which the gradient is calculated.
Revised: Truncation length refers to the horizon over which the gradient is calculated.

Original: The quantitative results are shown in Fig.
Revised: The quantitative results are shown in Fig.

Original: Indeed, it is difficult to learn a good policy by propagating the gradient through the whole trajectory that is long.
Revised: Indeed, it is difficult to learn a good policy by propagating the gradient through a whole trajectory that is long.

Original: In addition, simply splitting the whole trajectory into segments would worsen the final performance.
Revised: In addition, simply splitting the whole trajectory into segments would worsen the final performance.

Original: We hypothesize the naive truncation of the trajectory creates discontinuities in the whole trajectory whereas motions in the trajectory are highly interdependent.
Revised: We hypothesize the naive truncation of the trajectory creates discontinuities in the whole trajectory whereas motions in the trajectory are highly interdependent.

Original: For example, how to flip in mid-air closely relates to how the character jumps.
Revised: For example, how to flip in mid-air closely relates to how the character jumps.

Original: We see this as a strong call for a better strategy to handle these two challenges in motion mimicking with DPS.
Revised: We see this as a strong call for a better strategy to handle these two challenges in motion mimicking with DPS.

Original: 4. A BLATION ON D EMONSTRATION R EPLAY
Revised: 4. A BLATION ON D EMONSTRATION R EPLAY

Original: To understand how demonstration replay affects the policy learning of DiffMimic, we compare three different variants of DiffMimic, namely, Demonstration Replay (Random) and Demonstration
Revised: To understand how demonstration replay affects the policy learning of DiffMimic, we compare three different variants of DiffMimic, namely, Demonstration Replay (Random) and Demonstration

Original: Replay (Threshold), and Full Horizon Gradient on Back-flip and Cartwheel .
Revised: Replay (Threshold), and Full Horizon Gradient on Backflip and Cartwheel .

Original: Demonstration Replay (Random) randomly replaces a state in the rollout of the policy with the demonstration state at the same timestamp similar to the teacher forcing (Williams & Zipser, 1989).
Revised: Demonstration Replay (Random) randomly replaces a state in the rollout of the policy with the demonstration state at the same timestamp similar to the teacher forcing (Williams & Zipser, 1989).

Original: Demonstration Replay (Threshold) inserts the demonstration state based on the pose error.
Revised: Demonstration Replay (Threshold) inserts the demonstration state based on the pose error.

Original: If the pose error between the demonstration state and  simulated state exceeds a threshold ϵ , the simulated state will be replaced by the demonstration state.
Revised: If the pose error between the demonstration state and the simulated state exceeds a threshold ϵ , the simulated state will be replaced by the demonstration state.

Original: The Full Horizon Gradient variant backpropagates the gradient through the full horizon without any additional operation.
Revised: The Full Horizon Gradient variant backpropagates the gradient through the full horizon without any additional operation.

Original: We show the quantitative result in Fig.
Revised: We show the quantitative result in Fig.

Original: 7, and  qualitative result in Fig.
Revised: 7, and the qualitative result in Fig.

Original: Demonstration replay helps stabilize policy learning and leads to better performance.
Revised: Demonstration replay helps stabilize policy learning and leads to better performance.

Original: Compared with the Full Horizon Gradient without demonstration replay, the learning curve with demonstration replay is much smoother and finally converges to a lower pose error as shown in Fig. 7 (c)-(d).
Revised: Compared with the Full Horizon Gradient without demonstration replay, the learning curve with demonstration replay is much smoother and finally converges to a lower pose error as shown in Fig.(c)-(d).

Original: We show in Fig.
Revised: We show in Fig.

Original: 6 (b), this vanilla version of DiffMimic can easily get stuck into a local minimum.
Revised: 6 (b), this vanilla version of DiffMimic can easily get stuck into a local minimum.

Original: Instead of learning to back-flip, the policy learns to bend down and use arms to support itself instead of jumping up.
Revised: Instead of learning to backflip, the policy learns to bend down and use arms to support itself instead of jumping up.

Original: The other two variants, by contrast, both learn to jump and back-flip in mid-air successfully.
Revised: The other two variants, by contrast, both learn to jump and backflip in mid-air successfully.

Original: [Figure]
Revised: [Figure]

Original: Demonstration Replay Random) duces the overall pose error compared to full-horizon optimization, the per-frame loss remains large in cer tain time steps due to the lack of a constraint.
Revised: [Figure]

Original: Demonstration Replay (Threshold) alleviates the issue.
Revised: horizon issue.

Original: Policy-aware demonstration replay leads to a more faithful recovery of demonstration.
Revised: Policy-aware demonstration replay leads to a more faithful recovery of demonstration.

Original: We compare Demonstration Replay (Threshold) with Demonstration Replay (Random) in Fig.
Revised: We compare Demonstration Replay (Threshold) with Demonstration Replay (Random) in Fig.

Original: 7 (e)-(f).
Revised: 7 (e)-(f).

Original: Quantitatively, both variants yield similar results if the hyperparameter is properly tuned.
Revised: Quantitatively, both variants yield similar results if the hyperparameter is properly tuned.

Original: However, the behavior of policies trained with these two strategies can be significantly different.
Revised: However, the behavior of policies trained with these two strategies can be significantly different.

Original: In Fig.
Revised: In Fig.

Original: 6 (c) and (d), though both policies learn to back-flip successfully, the policy trained with Demonstration Replay (Random) fails to recover the motion frame by frame.
Revised: 6 (c) and (d), though both policies learn to backflip successfully, the policy trained with Demonstration Replay (Random) fails to recover the motion frame by frame.

Original: We show in Fig.
Revised: We show in Fig.

Original: 8 the per-frame pose error.
Revised: 8 the per-frame pose error.

Original: Even  the average error over the whole trajectory for both variant are similar, Demonstration Replay (Threshold) gives a lower maximum per-frame error, which indicates a faithful recovery of the demonstration.
Revised: Even though the average error over the whole trajectory for both variants is similar, Demonstration Replay (Threshold) gives a lower maximum per-frame error, which indicates a faithful recovery of the demonstration.

Original: This implies that simply minimizing the pose error may not suffice to learn a policy that tracks the demonstration closely.
Revised: This implies that simply minimizing the pose error may not suffice to learn a policy that tracks the demonstration closely.

Original: Finergrained guidance based on the current performance of the policy is required.
Revised: Finer-grained guidance based on the current performance of the policy is required.

Original:  C ONCLUSION & F UTURE WORKS
Revised:  C ONCLUSION & F UTURE WORKS

Original: To conclude, we present DiffMimic that utilize DPS for motion mimicking. We show that DiffMimic outperforms several commonly used RL-based methods for motion mimicking on a variety of tasks with high accuracy, stability, and efficiency.
Revised: In summary, we present DiffMimic utilizes differentiable physics simulators (DPS) for motion mimicking and outperforms several commonly used RL-based methods in various tasks with high accuracy, stability, and efficiency.

Original: We hope DiffMimic can be considered as a starting point for motion mimicking with differentiable physics simulators and  our simulation environments bring additional future research opportunities to the community.
Revised: We believe DiffMimic can serve as a starting point for motion mimicking with DPS and that our simulation environments provide exciting opportunities for future research.

Original: There are enormous excitingfuture directions to be explored, yet, with various challenges.
Revised: While there are many exciting directions to explore, there are also various challenges.

Original: Given differentiable physics largely accelerates motion mimicking, it would be much more valuable for downstream tasks if motion mimickingcan be carried out within minutes given any arbitrary demonstration. 
Revised: One such challenge is to enable motion mimicking within minutes given any arbitrary demonstration. With differentiable physics, this would greatly accelerate downstream tasks.

Original: In this work, we demonstrate that DiffMimic can learn challenging motions in  10 minutes.
Revised: In our work, we demonstrate that DiffMimic can learn challenging motions in just 10 minutes.

Original: Yet, the horizon of the task is relatively short and does not require any interaction with other objects.
Revised: However, the tasks we evaluated were relatively short and did not involve any interactions with other objects.

Original: The dynamic system would get more complex if multiple objects are involved. We leave them for future work.
Revised: As the dynamic system becomes more complex with multiple objects, we leave these challenges for future work.

Original: E THICS S TATEMENTS
Revised: E THICS S TATEMENTS

Original: We have read the ICLR Code of Ethics.
Revised: We have carefully reviewed and adhered to the ICLR Code of Ethics.

Original: While DiffMimic carries out experiments  on humanoid characters,no human-subject experiment is involved and  all experiments are conducted with opensourced materials with references to them.
Revised: DiffMimic conducts experiments exclusively on humanoid characters, and no experiments involve human subjects. Additionally, all experiments are performed using open-sourced materials and are properly cited

Original: R EPRODUCIBILITY S TATEMENTS
Revised: R EPRODUCIBILITY S TATEMENTS

Original: Our code is  available  at https://github.com/diffmimic/diffmimic.
Revised: Our code is easily accessible and openly available for review at https://github.com/diffmimic/diffmimic.

Revised: Additionally, the reported results can be easily reproduced using the provided code.

Revised: A CKNOWLEDGMENT

Revised: This research is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award

Original: We promise to clean and release the code upon publication.
Revised: No: AISG2-PhD-2022-01-036T, AISG2-PhD-2021-08-015T and AISG2-PhD-2021-08-018), NTU NAP, MOE AcRF Tier 2 (T2EP20221-0033), and under the RIE2020 Industry Alignment Fund - Industry Collaboration Projects (IAF-ICP)

Revised: Funding Initiative, as well as cash and in-kind contribution from the industry partner(s)

Original: C ONTENTS 1 Introduction
Revised: C ONTENTS 1 Introduction

Original: 2 Related Work 3 Methodology 3.
Revised: 2 Related Work 3 Methodology 3.

Original: A Motion Mimicking Environment in Differentiable Physics Engine. . . . . . . . .
Revised: A Motion Mimicking Environment in Differentiable Physics Engine. . . . . . . . .

Original: Motion Mimicking with Differentiable Physics . . . . . . . . . . . . . . . . . . .
Revised: Motion Mimicking with Differentiable Physics . . . . . . . . . . . . . . . . . . .

Original: Demonstration Replay . .
Revised: Demonstration Replay . .

Original: 4 Experiments 4.
Revised: 4 Experiments 4.

Original: Experiment Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Experiment Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: Comparison on Motion Mimicking . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Comparison on Motion Mimicking . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: Ablation on Truncation length . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Ablation on Truncation length . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: Ablation on Demonstration Replay . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Ablation on Demonstration Replay . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: 5 Conclusion & Future works
Revised: 5 Conclusion & Future works

Original: A Qualitative Results
Revised: A Qualitative Results

Original: B Policy Network
Revised: B Policy Network

Original: C Hyper-parameters
Revised: C Hyper-parameters

Original: D State and Action Space
Revised: D State and Action Space

Original: E Learning Curves
Revised: E Learning Curves

Original: F New Character
Revised: F New Character

Original: G New Skill
Revised: G New Skill

Original: H New Training Strategy
Revised: H New Training Strategy

Original: I Analysis on Robustness
Revised: I Analysis on Robustness

Original: J Analysis on Sensitivity
Revised: J Analysis on Sensitivity

Original: K Analysis on Motion Quality
Revised: K Analysis on Motion Quality

Original: L More Discussions on DeepMimic Reward M More Discussions on Related Works
Revised: L Swording Character

Original: A Q UALITATIVE R ESULTS
Revised: A Q UALITATIVE R ESULTS

Original: We present our qualitative results in the following website: https://diffmimic-demo-maing7h0i8.streamlitapp.com/.
Revised: We present our qualitative results in the following website: https://diffmimic-demo-maing7h0i8.streamlitapp.com/.

Original: It contains the following qualitative results:
Revised: It contains the following qualitative results:

Original: • Back-Flip 20-second rollout.
Revised: • Back-Flip 20-second rollout.

Original: • Cartwheel 20-second rollout. • Crawl 20-second rollout.
Revised: • Cartwheel 20-second rollout. • Crawl 20-second rollout.

Original: • Dance 20-second rollout. • Jog 20-second rollout.
Revised: • Dance 20-second rollout. • Jog 20-second rollout.

Original: • Jump 20-second rollout. • Roll 20-second rollout.
Revised: • Jump 20-second rollout. • Roll 20-second rollout.

Original: • Side-Flip 20-second rollout. • Spin-Kick 20-second rollout.
Revised: • Side-Flip 20-second rollout. • Spin-Kick 20-second rollout.

Original: • Walk 20-second rollout.
Revised: • Walk 20-second rollout.

Original: • Zombie 20-second rollout.
Revised: • Zombie 20-second rollout.

Original: • Back-Flip 20-second rollout, trained for 2.5 hours.
Revised: • Back-Flip 20-second rollout, trained for 2.5 hours.

Original: • Back-Flip single cycle rollout, trained for 10 minutes.
Revised: • Back-Flip single cycle rollout, trained for 10 minutes.

Original: B P OLICY N ETWORK
Revised: B P OLICY N ETWORK

Original: We use an MLP for the policy network.
Revised: We use an MLP for the policy network.

Original: The network has two hidden layers with sizes 512 and 256.
Revised: The network has two hidden layers with sizes 512 and 256.

Original: We use Swish (Ramachandran et al., 2017) for the activation function.
Revised: We use Swish (Ramachandran et al., 2017) for the activation function.

Original: C H YPER - PARAMETERS
Revised: C H YPER - PARAMETERS

Original: We use Adam optimizer (Kingma & Ba, 2014) in training, with a learning rate of 3e-4 that linearly decreases with training iterations.
Revised: We use Adam optimizer (Kingma & Ba, 2014) in training, with a learning rate of 3e-4 that linearly decreases with training iterations.

Original: We apply gradient clipping and set the maximum gradient norm to 0.3.
Revised: We apply gradient clipping and set the maximum gradient norm to 0.3.

Original: We set the batch size to the same as the number of parallel environments.
Revised: We set the batch size to the same as the number of parallel environments.

Original: Cyclic motions.
Revised: Cyclic motions.

Original: We set the maximum iterations to 5000.
Revised: We set the maximum iterations to 5000.

Original: The number of environments is set to 200.
Revised: The number of environments is set to 200.

Original: Acyclic motions.
Revised: Acyclic motions.

Original: We set the maximum iterations to 1000.
Revised: We set the maximum iterations to 1000.

Original: The number of environments is set to 300.
Revised: The number of environments is set to 300.

Original: Expert Replay.
Revised: Expert Replay.

Original: We run experiments with two error thresholds, 0.2 and 0.4, and report the better of the two.
Revised: We run experiments with two error thresholds, 0.2 and 0.4, and report the better of the two.

Original: D S TATE AND A CTION S PACE
Revised: D S TATE AND A CTION S PACE

Original: Humanoid.
Revised: Humanoid.

Original: We use 6d rotation representation (Zhou et al., 2019) in the state feature.
Revised: We use 6d rotation representation (Zhou et al., 2019) in the state feature.

Original: The humanoid has an action size of 28 and a state feature size of 193.
Revised: The humanoid has an action size of 28 and a state feature size of 193.

Original: Humanoid with Sword and Shield.
Revised: Humanoid with Sword and Shield.

Original: We set up the Sword and Shield character following ASE (Peng et al., 2022).
Revised: We set up the Sword and Shield character following ASE (Peng et al., 2022).

Original: Compared to the default humanoid, the new character 
Revised: Compared to the default humanoid, the new character has 3 additional DOF on the right hand.

Original: E L EARNING C URVES
Revised: A sword is attached to its right hand and a shield is attached to its left lower arm. The character has an action size of 31. The state feature size is 208. [Figure]

Original: F N EW C HARACTER
Revised: F N EW C HARACTER

Original: We include Ant as a new character.
Revised: We include Ant as a new character.

Original: We use the trajectory in ILD (Chen et al., 2022) as the reference motion, which is obtained from a PPO (Schulman et al., 2017) agent that learns to move forward as fast as possible.
Revised: We use the trajectory in ILD (Chen et al., 2022) as the reference motion, which is obtained from a PPO (Schulman et al., 2017) agent that learns to move forward as fast as possible.

Original: We achieve a 0.059 meter final pose error.
Revised: We achieve a 0.059 meter final pose error.

Original: We show the training curve in Figure 11.
Revised: We show the training curve in Figure 10.

Original: and the qualitative result on the demo website under the tab ”New Character”.
Revised: and the qualitative result on the demo website under the tab ”New Character”.

Original: [Figure]
Revised: [Figure]

Original: G N EW S KILL
Revised: G N EW S KILL

Original: We run DiffMimic on a new skill, 360-degree Jump , from the CMU Mocap Dataset (motion sequence CMU 075 09).
Revised: We run DiffMimic on a new skill, 360-degree Jump , from the CMU Mocap Dataset (motion sequence CMU 075 09).

Original: The skill has been reported to be challenging for a RL-based method
Revised: The skill has been reported to be challenging for a RL-based method

Original: CoMic (Hasenclever et al., 2020) in a recent dataset paper (Wagener et al., 2022).
Revised: CoMic (Hasenclever et al., 2020) in a recent dataset paper (Wagener et al., 2022).

Original: DiffMimic is able to spin in the air and achieve a 0.020 meter final pose error.
Revised: DiffMimic is able to spin in the air and achieve a 0.020 meter final pose error.

Original: We show the training curve in
Revised: We show the training curve in

Original: Figure 12 and the qualitative result on the demo website under the tab ”New Skill”.
Revised: Figure 11 and the qualitative result on the demo website under the tab ”New Skill”.

Original: [Figure]
Revised: [Figure]

Original: H N EW T RAINING S TRATEGY
Revised: H N EW T RAINING S TRATEGY

Original: We additionally include the Reference State Initialization (RSI) technique (Peng et al., 2018a) into the DiffMimic training.
Revised: We additionally include the Reference State Initialization (RSI) technique (Peng et al., 2018a) into the DiffMimic training.

Original: RSI starts the training rollout from a random reference frame instead of the first reference frame.
Revised: RSI starts the training rollout from a random reference frame instead of the first reference frame.

Original: We observe a substantial empirical improvement after employing the new training strategy.
Revised: We observe a substantial empirical improvement after employing the new training strategy.

Original: The samples required are consistently around 90% less than DeepMimic (Peng et al., 2018a) as shown in Tab. 3.
Revised: The samples required are consistently around 90% less than DeepMimic (Peng et al., 2018a) as shown in Tab. 3.

Original: The final pose error has also been slightly improved as shown in
Revised: The final pose error has also been slightly improved as shown in

Original: Tab. 4.
Revised: Tab. 4.

Original: We show the qualitative results on the demo website under the tab ”New Training”.
Revised: We show the qualitative results on the demo website under the tab ”New Training”.

Original: Notably, using RSI alone or RSI+Early Termination (ET) (Peng et al., 2018a) does not converge as shown in
Revised: Notably, using RSI alone or RSI+Early Termination (ET) (Peng et al., 2018a) does not converge as shown in

Original: Figure 13.
Revised: Figure 12.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: I A NALYSIS ON R OBUSTNESS
Revised: I A NALYSIS ON R OBUSTNESS

Original: We further evaluate DiffMimic’s robustness to external forces.
Revised: We further evaluate DiffMimic’s robustness to external forces.

Original: A force is applied to the pelvis of the character halfway through a motion cycle for 0.2 seconds
Revised: A force is applied to the pelvis of the character halfway through a motion cycle for 0.2 seconds

Original: [Figure]
Revised: [Figure]

Original: and we measure the maximum forwards and sideways push each policy can tolerate before falling.
Revised: and we measure the maximum forwards and sideways push each policy can tolerate before falling.

Original: DiffMimic achieves comparable robustness to DeepMimic.
Revised: DiffMimic achieves comparable robustness to DeepMimic.

Original: We show the results in Table 5 and the qualitative results on the demo website under the tab ”Robustness”
Revised: We show the results in Table 5 and the qualitative results on the demo website under the tab ”Robustness”.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: J A NALYSIS ON S ENSITIVITY
Revised: J A NALYSIS ON S ENSITIVITY

Original: DiffMimic with inaccurate estimates of physical parameters.
Revised: DiffMimic with inaccurate estimates of physical parameters.

Original: Briefly speaking, DiffMimic is able to learn the control policy well even though the friction coefficient deviates from the original parameters.
Revised: Briefly speaking, DiffMimic is able to learn the control policy well even though the friction coefficient deviates from the original parameters.

Original: We show the results in Table 6 and the qualitative results on the demo website under the tab ”Sensitivity”.
Revised: We show the results in Table 6 and the qualitative results on the demo website under the tab ”Sensitivity”.

Original: [Table]
Revised: [Table]

Original: K A NALYSIS ON M OTION Q UALITY
Revised: K A NALYSIS ON M OTION Q UALITY

Original: Motion quality can be severely affected by a small number of large-error poses, which can not be reflected by the existing average pose error metric.
Revised: Motion quality can be severely affected by a small number of large error poses, which can not be reflected by the existing average pose error metric.

Original: We propose a Pose Absurdity metric to quantify large-error poses.
Revised: We propose a Pose Absurdity metric to quantify large-error poses.

Original: L2@0.01, 0.05, and 0.1 measure the average L2 pose error on the worst 1%, 5% [Table] and 10% frames respectively.
Revised: L2@0.01, 0.05, and 0.1 measure the average L2 pose error on the worst 1%, 5%, and 10% frames respectively.

Original: Full Horizon Gradient, Demonstration Replay (Random) ratio λ , and Demonstration Replay (Threshold) threshold γ are studied.
Revised: Full Horizon Gradient, Demonstration Replay (Random) ratio λ , and Demonstration Replay (Threshold) threshold γ are studied.

Original: Demonstration Replay (Threshold) outperforms other baselines by a large margin.
Revised: Demonstration Replay (Threshold) outperforms other baselines by a large margin.

Original: We show the results in Table
Revised: We show the results in Table

Original: L M ORE D ISCUSSIONS ON D EEP M IMIC R EWARD
Revised: L S WORDING C HARACTER

Original: DeepMimic (Peng et al., 2018a) defines the reward r It for each frame time as: [Equation]
Revised: a shield.

Original: where w are weighting hyper-parameters, r pt , r vt , r et , r ct are reward functions for joint orientation, joint velocity, end-effector position, and center-of-mass position respectively. Concretely, they are
Revised: The reference motion clip is from Reallusion (Reallusion, 2022)

Original: [Table]
Revised: [Table]

Original: defined as follows:

Original: [Equation]

Original: where − 2 , − 0 .

Original: 1 , − 40 , − 10 are four more weighting hyper-parameters.

Original: Note that there are 8 hyper-parameters in the reward function in total.

Original: Moreover, a selection on joints ( e.g. , only apply position reward to end-effectors and center-of-mass) is also required.

Original: In comparison, DiffMimic uses a direct L2 distance between states without weighting as defined in

Original: Eq. 3.2, which is much simpler.

Original: M M ORE D ISCUSSIONS ON R ELATED W ORKS

Original: SHAC (Xu et al., 2022) and PODS (Mora et al., 2021) are two differentiable physics-based reinforcement learning algorithms.

Original: SHAC addresses the gradient exploding and diminishing problem by manually truncating trajectories into small segments.

Original: It requires careful design of gradient truncation throughout the learning process.

Original: In contrast, DiffMimic automatically resets the state to the demonstrated state, preventing the state from deviating too much and achieving a similar effect to the trajectory truncation in SHAC.

Original: In addition, SHAC seeks to maximize the reward, which can often lead to a local optimum when the reward function is ill-defined or difficult to design.

Original: DiffMimic, on the other hand, benefits from the expert demonstration, which provides explicit guidance to escape from local optimal.

Original: Using second-order gradients, PODS claims a monotonic policy improvement and a significant speedup in convergence over first-order methods.

Original: Despite its success, PODS makes strong assumptions about the second-order derivatives of a differentiable simulator and can be sensitive to the accuracy of the second-order derivatives.

Original: Moreover, second-order derivatives incur more computational demands, which can be potentially costly in tasks with complex dynamics.

Original: Compared to

Original: SHAC and DiffMimic, PODS can suffer from gradient exploding and diminishing problems without a proper truncation scheme.

Original: This problem can even be exaggerated with the use of second-order gradients in longer trajectory rollouts.

Original: Supertrack (Fussell et al., 2021) adopts the idea of model-based RL and proposes to learn a world model to predict the next simulated character state given the previous simulated character state and the provided PD targets.

Original: It learns the world model and the control policy in an iterative way: 1) the policy rollouts in a physics simulator, 2) the trajectories are collected to train the world model, 3) the policy is then trained through the differentiable world model.

Original: However, one major disadvantage of the approach is that its policy horizon, i.e., rollout steps, is small, e.g., 32 steps, during training due to the error accumulation in the world model.

Original: Training can get unstable for long rollouts, e.g.,steps.

Original: Therefore, as pointed out by the authors, SuperTrack can be unsuitable when the policy needs a long-term apprehension to mimic some challenging motions.

Original: DiffMimic uses DPS in replace of the learned world model to avoid error accumulation.

Original: ScaDiver (Won et al., 2020) presents a scalable approach to learning controllers for a large motion database, where motion mimicking is a key part of the pipeline.

Original: It proposes to 1) divide the diverse motion database into clusters of similar motions, 2) use DeepMimic to learn a control policy for each motion cluster, and 3) learn a mixture-of-expert controllers to dynamically ensemble the control policies.

Original: We view ScaDiver as an important downstream application of DiffMimic.

Original: DiffMimic can serve as a better alternative to DeepMimic in step 2, and contribute to a higher efficiency when learning from large-scale motion databases.