Original: C OMPLEX L OCOMOTION S KILL L EARNING VIA D IFFERENTIABLE P HYSICS
Revised: C OMPLEX L OCOMOTION S KILL L EARNING VIA D IFFERENTIABLE P HYSICS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Differentiable physics enables efﬁcient gradient-based optimizations of neural network (NN) controllers.
Revised: Differentiable physics enables efﬁcient gradient-based optimizations of neural network (NN) controllers.

Original: However, existing work typically only delivers NN controllers with limited capability and generalizability.
Revised: However, existing work typically only delivers NN controllers with limited capability and generalizability.

Original: We present a practical learning framework that outputs uniﬁed NN controllers capable of tasks with signiﬁcantly improved complexity and diversity.
Revised: We present a practical learning framework that outputs uniﬁed NN controllers capable of tasks with signiﬁcantly improved complexity and diversity.

Original: To systematically improve training robustness and efﬁciency, we investigated a suite of improvements over the baseline approach, including periodic activation functions and tailored loss functions.
Revised: To systematically improve training robustness and efﬁciency, we investigated a suite of improvements over the baseline approach, including periodic activation functions and tailored loss functions.

Original: In addition, we ﬁnd our adoption of batching and a modiﬁed Adam optimizer effective in training complex locomotion tasks.
Revised: In addition, we ﬁnd our adoption of batching and a modiﬁed Adam optimizer effective in training complex locomotion tasks.

Original: We evaluate our framework on differentiable mass-spring and material point method (MPM) simulations, with challenging locomotion tasks and multiple robot designs.
Revised: We evaluate our framework on differentiable mass-spring and material point method (MPM) simulations, with challenging locomotion tasks and multiple robot designs.

Original: Experiments show that our learning framework, based on differentiable physics, delivers better results than reinforcement learning and converges much faster.
Revised: Experiments show that our learning framework, based on differentiable physics, delivers better results than reinforcement learning and converges much faster.

Original: We demonstrate that users can interactively control soft robot locomotion and switch among multiple goals with speciﬁed velocity, height, and direction instructions using a uniﬁed NN controller trained in our system.
Revised: We demonstrate that users can interactively control soft robot locomotion and switch among multiple goals with speciﬁed velocity, height, and direction instructions using a uniﬁed NN controller trained in our system.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: [Figure]
Revised: [Figure]

Original: physical simulators deliver accurate analytical physical up efﬁcient
Revised: physical simulators deliver accurate analytical physical up efﬁcient

Original: ) controller training descent
Revised: ) controller training descent

Original: Existing research that on simple tasks, learning systems via differentiable physics can effectively leverage simulation gradient information and converge orders of magnitude faster (see, e.g., de Avila Belbute-Peres et al.
Revised: Existing research that on simple tasks, learning systems via differentiable physics can effectively leverage simulation gradient information and converge orders of magnitude faster (see, e.g., de Avila Belbute-Peres et al.

Original: ; Hu et al.
Revised: ; Hu et al.

Original: than reinforcement learning.
Revised: than reinforcement learning.

Original: However, the capability of existing differentiable physics based learning systems is relatively limited.
Revised: However, the capability of existing differentiable physics based learning systems is relatively limited.

Original: Typically, optimized controllers can only achieve relatively simple single-goal tasks (e.g., moving in one speciﬁc direction, as in ChainQueen Hu et al.
Revised: Typically, optimized controllers can only achieve relatively simple single-goal tasks (e.g., moving in one speciﬁc direction, as in ChainQueen Hu et al.

Original: Those learned controllers often have difﬁculty generalizing the task to a perturbed version.
Revised: Those learned controllers often have difﬁculty generalizing the task to a perturbed version.

Original: In this work, we propose a learning framework for complex locomotion skill learning via differentiable physics.
Revised: In this work, we propose a learning framework for complex locomotion skill learning via differentiable physics.

Original: The complexity of our tasks comes from two aspects: ﬁrst, the degrees of freedom of our soft agents are signiﬁcantly higher compared with the rigid-body ones; second, the agents are expected to learn multiple skills at the same time.
Revised: The complexity of our tasks comes from two aspects: ﬁrst, the degrees of freedom of our soft agents are signiﬁcantly higher compared with the rigid-body ones; second, the agents are expected to learn multiple skills at the same time.

Original: We systematically propose a suite of enhancements (Fig. 4) over existing training approaches (such as Hu et al.
Revised: We systematically propose a suite of enhancements (Fig. 4) over existing training approaches (such as Hu et al.

Original: ; Huang et al.
Revised: ; Huang et al.

Original: to signiﬁcantly improve the efﬁciency, robustness, and generalizability of learning systems based on differentiable physics.
Revised: to signiﬁcantly improve the efﬁciency, robustness, and generalizability of learning systems based on differentiable physics.

Original: In addition, we investigated the contributions of each enhancements to training in detail by a series of ablation studies.
Revised: In addition, we investigated the contributions of each enhancements to training in detail by a series of ablation studies.

Original: We also evaluated our framework through comparisons against Proximal Policy
Revised: We also evaluated our framework through comparisons against Proximal Policy

Original: Optimization (PPO) Schulman et al.
Revised: Optimization (PPO) Schulman et al.

Original: (2017), a state-of-the-art reinforcement learning algorithm.
Revised: (2017), a state-of-the-art reinforcement learning algorithm.

Original: Results show that our system is simple yet effective and has a much-improved convergence rate compared to PPO.
Revised: Results show that our system is simple yet effective and has a much-improved convergence rate compared to PPO.

Original: We demonstrate the versatility of our framework via various physical simulation environments (mass-spring systems and the moving least squares material point method,
Revised: We demonstrate the versatility of our framework via various physical simulation environments (mass-spring systems and the moving least squares material point method,

Original: MLS-MPM Hu et al.
Revised: MLS-MPM Hu et al.

Original: (2018)), robot designs (structured and irregular), locomotion tasks (moving, jumping, turning around, all at a controllable velocity).
Revised: (2018)), robot designs (structured and irregular), locomotion tasks (moving, jumping, turning around, all at a controllable velocity).

Original: To the best of our knowledge, this is the ﬁrst time when a neural network can be trained via differentiable physics to achieve tasks of such complexity (Fig. 1).
Revised: To the best of our knowledge, this is the ﬁrst time when a neural network can be trained via differentiable physics to achieve tasks of such complexity (Fig. 1).

Original: Our agent is simultaneously trained with multiple goals, the learned skills can be continuously interpolated.
Revised: Our agent is simultaneously trained with multiple goals, the learned skills can be continuously interpolated.

Original: For example, we can control both the velocity and height of an agent while it runs.
Revised: For example, we can control both the velocity and height of an agent while it runs.

Original: The trained agent can also be controlled in real-time, enabling direct applications in soft robot control and video games.
Revised: The trained agent can also be controlled in real-time, enabling direct applications in soft robot control and video games.

Original: In summary, our key contributions are listed below:
Revised: In summary, our key contributions are listed below:

Original: • To the best of our knowledge, we proposed the ﬁrst differentiable physics based learning framework for multiple locomotion skills learning on soft agents using a single network.
Revised: • To the best of our knowledge, we proposed the ﬁrst differentiable physics based learning framework for multiple locomotion skills learning on soft agents using a single network.

Original: • We developed an end-to-end differentiable physical simulation environment for deformable robot locomotion skills learning, which supports mass-spring system and material point method as dynamic backends.
Revised: • We developed an end-to-end differentiable physical simulation environment for deformable robot locomotion skills learning, which supports mass-spring system and material point method as dynamic backends.

Original: • We systematically investigated the key factors contributing to the differentiable physics based learning framework.
Revised: • We systematically investigated the key factors contributing to the differentiable physics based learning framework.

Original: We believe our investigation can inspire further researches to explore the possibilities of differentiable physics for more complex learning tasks
Revised: We believe our investigation can inspire further researches to explore the possibilities of differentiable physics for more complex learning tasks

Original:  D IFFERENTIABLE S IMULATION E NVIRONMENTS 2. S IMULATION SETTINGS
Revised:  D IFFERENTIABLE S IMULATION E NVIRONMENTS 2. S IMULATION SETTINGS

Original: Our learning framework supports multiple types of differentiable physically-based deformable body simulators.
Revised: Our learning framework supports multiple types of differentiable physically-based deformable body simulators.

Original: The simulator can be seen as a black box that outputs the states for the next time step given the actuation signal and states from the current time step.
Revised: The simulator can be seen as a black box that outputs the states for the next time step given the actuation signal and states from the current time step.

Original: The NN controller takes the information given by the simulator as part of its input features and decides the actuation signal for the simulator in the next time step.
Revised: The NN controller takes the information given by the simulator as part of its input features and decides the actuation signal for the simulator in the next time step.

Original: With our automatic differentiation system, we can evaluate the gradient information to guide the training of our NN controllers.
Revised: With our automatic differentiation system, we can evaluate the gradient information to guide the training of our NN controllers.

Original: We use the mass-spring systems and the moving least squares material point method (MLS-MPM) as our simulation environments.
Revised: We use the mass-spring systems and the moving least squares material point method (MLS-MPM) as our simulation environments.

Original: The design choices for each environment can be found in the appendix.
Revised: The design choices for each environment can be found in the appendix.

Original: 2. A GENT STRUCTURE AND ACTUATION MODE
Revised: 2. A GENT STRUCTURE AND ACTUATION MODE

Original: Our framework supports a variety of differently shaped agents as shown in Fig.
Revised: Our framework supports a variety of differently shaped agents as shown in Fig.

Original: 2 and 3.
Revised: 2 and 3.

Original: Our actuation signal exerts forces along with the ”muscle” directions of the simulated agents.
Revised: Our actuation signal exerts forces along with the ”muscle” directions of the simulated agents.

Original: This signal falls within the interval [ − 1 , +1] , where its sign determines whether an agent wants to contract (negative sign) or relax (positive sign) a muscle and its absolute value determines the magnitude of the force.
Revised: This signal falls within the interval [ − 1 , +1] , where its sign determines whether an agent wants to contract (negative sign) or relax (positive sign) a muscle and its absolute value determines the magnitude of the force.

Original: In mass-spring systems, the muscle directions are represented by the spring directions, we change the rest-length of springs to generate forces.
Revised: In mass-spring systems, the muscle directions are represented by the spring directions, we change the rest-length of springs to generate forces.

Original: The actuation signal is used to scale the rest-length up
Revised: The actuation signal is used to scale the rest-length up

Original: [Figure]
Revised: [Figure]

Original: to some limit (usually at 20% ).
Revised: to some limit (usually at 20% ).

Original: In addition, only activated springs, i.e. actuators (springs marked red or blue shown in Fig. 1), are able to generate forces according to control signals.
Revised: In addition, only activated springs, i.e. actuators (springs marked red or blue shown in Fig. 1), are able to generate forces according to control signals.

Original: In material point methods, the muscle directions are always along the vertical axis in material space.
Revised: In material point methods, the muscle directions are always along the vertical axis in material space.

Original: In our implementation, we simply modify the Cauchy stress in the vertical direction of material space to apply this force.
Revised: In our implementation, we simply modify the Cauchy stress in the vertical direction of material space to apply this force.

Original: Again, the force is scaled to a user-deﬁned bound which can be adjusted for different simulations
Revised: Again, the force is scaled to a user-deﬁned bound which can be adjusted for different simulations

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: In this section, we describe our training framework based on differentiable physics (Fig. 4).
Revised: In this section, we describe our training framework based on differentiable physics (Fig. 4).

Original: In summary, we develop a differentiable simulator with an embedded NN controller.
Revised: In summary, we develop a differentiable simulator with an embedded NN controller.

Original: In each time step, the program performs an NN controller inference and a differentiable simulator time integration.
Revised: In each time step, the program performs an NN controller inference and a differentiable simulator time integration.

Original: After a few hundred time steps, the program back-propagates gradients end-to-end via reverse-mode automatic differentiation, and updates the weights of the neural network using the
Revised: After a few hundred time steps, the program back-propagates gradients end-to-end via reverse-mode automatic differentiation, and updates the weights of the neural network using the

Original: Adam optimizer Kingma & Ba (2014).
Revised: Adam optimizer Kingma & Ba (2014).

Original: 3. T ASK R EPRESENTATION
Revised: 3. T ASK R EPRESENTATION

Original: Given an agent A , its position and corresponding velocity are denoted as x = ( x, y, z ) and v = ( u, v, w ) .
Revised: Given an agent A , its position and corresponding velocity are denoted as x = ( x, y, z ) and v = ( u, v, w ) .

Original: We presents three tasks (running, jumping and crawling) for 2D agents, two (running and rotating) for 3D.
Revised: We presents three tasks (running, jumping and crawling) for 2D agents, two (running and rotating) for 3D.

Original: In a whole simulation with total steps T , an agent is expected to achieve all goals in a goal sequence G = { G 1 , G 2 ...
Revised: In a whole simulation with total steps T , an agent is expected to achieve all goals in a goal sequence G = { G 1 , G 2 ...

Original: G n | n P = T } , where P is the time period for an agent to achieve a goal, n is the number of goals .
Revised: G n | n P = T } , where P is the time period for an agent to achieve a goal, n is the number of goals .

Original: For each goal G , there are multiple tasks in it.
Revised: For each goal G , there are multiple tasks in it.

Original: Each task is encoded using a target value g , which instructs the controller to drive the agent.
Revised: Each task is encoded using a target value g , which instructs the controller to drive the agent.

Original: For example, G = { g v , g h } represent performing running and jumping simultaneously, the target velocity and height are g v and g h .
Revised: For example, G = { g v , g h } represent performing running and jumping simultaneously, the target velocity and height are g v and g h .

Original: The agent is expected to switch between multiple goals during one simulation.
Revised: The agent is expected to switch between multiple goals during one simulation.

Original: Running.
Revised: Running.

Original: The running task is deﬁned as a velocity tracking problem.
Revised: The running task is deﬁned as a velocity tracking problem.

Original: Given a time step t , the center of mass c is used to represent the position of the agent, its velocity is deﬁned as the averaged velocity ˜ v over a time period P r , ( P r < P ).
Revised: Given a time step t , the center of mass c is used to represent the position of the agent, its velocity is deﬁned as the averaged velocity ˜ v over a time period P r , ( P r < P ).

Original: Then the agent is expected to achieve a velocity ˜ v close to the target velocity g v .
Revised: Then the agent is expected to achieve a velocity ˜ v close to the target velocity g v .

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Jumping.
Revised: Jumping.

Original: The jumping task is deﬁned as reaching a given height g h in a time period P .
Revised: The jumping task is deﬁned as reaching a given height g h in a time period P .

Original: Due to the gravity, it is not possible for a running robot to stay away from the ground.
Revised: Due to the gravity, it is not possible for a running robot to stay away from the ground.

Original: Therefore, the jumping height ˜
Revised: Therefore, the jumping height ˜

Original: h h ( t ) of an agent is deﬁned by the maximum vertical position of its lowest point in a certain period P .
Revised: h h ( t ) of an agent is deﬁned by the maximum vertical position of its lowest point in a certain period P .

Original: To be more formally, given S a set of nodes (or points) that constitute the agent, h is a function that can extract the vertical position given a node (or point), the jumping height over the n -th period is deﬁned as
Revised: To be more formally, given S a set of nodes (or points) that constitute the agent, h is a function that can extract the vertical position given a node (or point), the jumping height over the n -th period is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: ) where n is the index of the period in a whole simulation.
Revised: ) where n is the index of the period in a whole simulation.

Original: Crawling.
Revised: Crawling.

Original: The crawling task is deﬁned as lowering the highest point as much as possible.
Revised: The crawling task is deﬁned as lowering the highest point as much as possible.

Original: The target of crawling g c is deﬁned as an indicator function.
Revised: The target of crawling g c is deﬁned as an indicator function.

Original: The agent is controlled to crawl if g c =
Revised: The agent is controlled to crawl if g c =

Original: otherwise if g c = 0 .
Revised: otherwise if g c = 0 .

Original: Since crawling is a status that needs to be maintained, the crawling height is deﬁned as the vertical position of its highest point at each time step.
Revised: Since crawling is a status that needs to be maintained, the crawling height is deﬁned as the vertical position of its highest point at each time step.

Original: [Equation]
Revised: [Equation]

Original: 3. L OSS FUNCTIONS
Revised: 3. L OSS FUNCTIONS

Original: Consider a locomotion task where an agent is instructed to move at a speciﬁed velocity, we think a proper loss function should satisfy the following requirements:
Revised: Consider a locomotion task where an agent is instructed to move at a speciﬁed velocity, we think a proper loss function should satisfy the following requirements:

Original: Periodicity.
Revised: Periodicity.

Original: The agent is expected to move periodically following a predeﬁned cyclic activation signal.
Revised: The agent is expected to move periodically following a predeﬁned cyclic activation signal.

Original: Therefore, the loss function should encourage periodic motions.
Revised: Therefore, the loss function should encourage periodic motions.

Original: Delayed evaluation.
Revised: Delayed evaluation.

Original: Due to inertia, it takes time for the agent to start running or adjust running velocity.
Revised: Due to inertia, it takes time for the agent to start running or adjust running velocity.

Original: Therefore, an ideal loss function should take this delay into consideration.
Revised: Therefore, an ideal loss function should take this delay into consideration.

Original: Fluctuation tolerance.
Revised: Fluctuation tolerance.

Original: During a single running cycle, requiring the center of mass to move at a constant velocity at each time step may induce highly ﬂuctuated losses.
Revised: During a single running cycle, requiring the center of mass to move at a constant velocity at each time step may induce highly ﬂuctuated losses.

Original: To avoid that, our loss function should be smooth during one motion cycle.
Revised: To avoid that, our loss function should be smooth during one motion cycle.

Original: Task loss.
Revised: Task loss.

Original: For each task, we deﬁned a tailored loss shown in equation 4 according to the requirements above.
Revised: For each task, we deﬁned a tailored loss shown in equation 4 according to the requirements above.

Original: The loss for running L v is deﬁned as the accumulation of the difference between the target and the agent’s velocity from start to current time step.
Revised: The loss for running L v is deﬁned as the accumulation of the difference between the target and the agent’s velocity from start to current time step.

Original: For jumping, we deﬁne a sparse loss
Revised: For jumping, we deﬁne a sparse loss

Original: L h , which only evaluate once for each period.
Revised: L h , which only evaluate once for each period.

Original: The crawling loss L c is deﬁned as the accumulation of the crawling height if the loss is applicable.
Revised: The crawling loss L c is deﬁned as the accumulation of the crawling height if the loss is applicable.

Original: [Equation]
Revised: [Equation]

Original: The λ v , λ h , λ c are weights for losses of dfferent tasks.
Revised: The λ v , λ h , λ c are weights for losses of dfferent tasks.

Original: Our loss function accumulates the contributions of all steps in a “sliding window”.
Revised: Our loss function accumulates the contributions of all steps in a “sliding window”.

Original: This prevents the loss function from being too small as well as the vanishing gradient problem.
Revised: This prevents the loss function from being too small as well as the vanishing gradient problem.

Original: We ﬁnd that having an accumulated loss evaluated at every step works much better when we want to control the speed of the agent explicitly.
Revised: We ﬁnd that having an accumulated loss evaluated at every step works much better when we want to control the speed of the agent explicitly.

Original: Regularization.
Revised: Regularization.

Original: Since the agents may keep shaking when no target velocity or height are given, we introduce an actuation loss as a regularization term.
Revised: Since the agents may keep shaking when no target velocity or height are given, we introduce an actuation loss as a regularization term.

Original: The intuition is to penalize comparatively large actuations when small goals value is given:
Revised: The intuition is to penalize comparatively large actuations when small goals value is given:

Original: [Equation]
Revised: [Equation]

Original: where A ( t ) is the actuation output of NN, µ is a normalizer.
Revised: where A ( t ) is the actuation output of NN, µ is a normalizer.

Original: The actuation loss balances the importance of input channels of the NN, i.e., different input channels produce similar contributions to the ﬁrst layer of the trained NN.
Revised: The actuation loss balances the importance of input channels of the NN, i.e., different input channels produce similar contributions to the ﬁrst layer of the trained NN.

Original: 3. N ETWORK ARCHITECTURE
Revised: 3. N ETWORK ARCHITECTURE

Original: We use two fully connected (FC) layers with sine activation function as the neural network.
Revised: We use two fully connected (FC) layers with sine activation function as the neural network.

Original: The network input vector consists of three parts:
Revised: The network input vector consists of three parts:

Original: Periodic control signal of the same period with different phases, to encourage periodic actuation.
Revised: Periodic control signal of the same period with different phases, to encourage periodic actuation.

Original: In this work, we use sin waves as the periodic control signal.
Revised: In this work, we use sin waves as the periodic control signal.

Original: State feature vector extracted from the current state of the agent.
Revised: State feature vector extracted from the current state of the agent.

Original: Using the positions and velocities of all the vertices of the simulated object as the input feature is not a good idea since these quantities are neither translation-invariant nor rotation-invariant.
Revised: Using the positions and velocities of all the vertices of the simulated object as the input feature is not a good idea since these quantities are neither translation-invariant nor rotation-invariant.

Original: We use a “centralized pose” to remove global translation and rotation information.
Revised: We use a “centralized pose” to remove global translation and rotation information.

Original: To be more speciﬁc, we subtract the position of the center of mass (CoM) in both 2D and 3D cases to remove the global translation.
Revised: To be more speciﬁc, we subtract the position of the center of mass (CoM) in both 2D and 3D cases to remove the global translation.

Original: In 3D cases, we also compute the rotation around the vertical axis of the agent to remove its global rotation.
Revised: In 3D cases, we also compute the rotation around the vertical axis of the agent to remove its global rotation.

Original: Targets that encode the task information such as running velocity, jumping height, and orientation for 3D agents.
Revised: Targets that encode the task information such as running velocity, jumping height, and orientation for 3D agents.

Original: Different tasks use different channels to represents their target values.
Revised: Different tasks use different channels to represents their target values.

Original: During the training
Revised: During the training

Original: we assign random values to these targets.
Revised: we assign random values to these targets.

Original: When validating the model, we always test the agent on a ﬁxed set of target values.
Revised: When validating the model, we always test the agent on a ﬁxed set of target values.

Original: In interactive settings, we assign various target values to control the agent’s motion.
Revised: In interactive settings, we assign various target values to control the agent’s motion.

Original: To amplify the importance of the targets, we duplicate these channels multiple times, try to make them comparably important among the other input features.
Revised: To amplify the importance of the targets, we duplicate these channels multiple times, try to make them comparably important among the other input features.

Original: 3. T RAINING
Revised: 3. T RAINING

Original: In the training process, we run one simulation in one training iteration.
Revised: In the training process, we run one simulation in one training iteration.

Original: The duration of the simulation is divided into several periods P as mentioned in the section 3.1.
Revised: The duration of the simulation is divided into several periods P as mentioned in the section 3.1.

Original: For each P , a goal G contains several tasks is assigned.
Revised: For each P , a goal G contains several tasks is assigned.

Original: For example, G = { g v , g h } represents that the agent is expected to perform both running and jumping in this period.
Revised: For example, G = { g v , g h } represents that the agent is expected to perform both running and jumping in this period.

Original: The goal and its target values are generated randomly in a uniform distribution.
Revised: The goal and its target values are generated randomly in a uniform distribution.

Original: They are kept ﬁxed inside one period P , but varied between different periods.
Revised: They are kept ﬁxed inside one period P , but varied between different periods.

Original:  E XPERIMENTS AND A NALYSIS
Revised:  E XPERIMENTS AND A NALYSIS

Original: In this section, we systematically investigate the key factors that contribute to the training, evaluate our framework through ablation studies and comparisons against PPO Schulman et al.
Revised: In this section, we systematically investigate the key factors that contribute to the training, evaluate our framework through ablation studies and comparisons against PPO Schulman et al.

Original: (2017), a state-of-the-art reinforcement learning algorithm.
Revised: (2017), a state-of-the-art reinforcement learning algorithm.

Original: All experiments in the ablation studies are performed at least three times.
Revised: All experiments in the ablation studies are performed at least three times.

Original: The results of ablation studies are summarized in Table 1 and Fig.
Revised: The results of ablation studies are summarized in Table 1 and Fig.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: 4. V ALIDATION M ETRIC
Revised: 4. V ALIDATION M ETRIC

Original: In validation we test all the trained agents on a ﬁxed set of goals.
Revised: In validation we test all the trained agents on a ﬁxed set of goals.

Original: These goals are represented by combination of targets targets values, which are uniformly sampled by interpolating between the lower and upper bound of all targets appeared in training.
Revised: These goals are represented by combination of targets targets values, which are uniformly sampled by interpolating between the lower and upper bound of all targets appeared in training.

Original: 4. D IFFERENTIABLE P HYSICS G RADIENT A NALYSIS
Revised: 4. D IFFERENTIABLE P HYSICS G RADIENT A NALYSIS

Original: One concern in differentiable physics based learning is the that the gradient may explode during the back-propagation through long simulation steps.
Revised: One concern in differentiable physics based learning is the that the gradient may explode during the back-propagation through long simulation steps.

Original: Here we visualize the distribution of gradients norm of a whole training process in Fig.
Revised: Here we visualize the distribution of gradients norm of a whole training process in Fig.

Original: It can be observed that most values of the gradients are distributed inside the interval [-10, 10] in log scale which indicates that our learning approach provides stable gradients during training.
Revised: It can be observed that most values of the gradients are distributed inside the interval [-10, 10] in log scale which indicates that our learning approach provides stable gradients during training.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Adam vs SGD.
Revised: Adam vs SGD.

Original: Hu et al.
Revised: Hu et al.

Original: (2020) shows that stochastic gradient descent (SGD) with differentiable physics can achieve satisfying results for tasks with single goals, e
Revised: (2020) shows that stochastic gradient descent (SGD) with differentiable physics can achieve satisfying results for tasks with single goals, e

Original: [Figure]
Revised: [Figure]

Original: g. running along one direction.
Revised: g. running along one direction.

Original: However, this no longer holds for tasks with multiple goals.
Revised: However, this no longer holds for tasks with multiple goals.

Original: To investigate the differences, we perform a learning task with multiple goals on a 2D mass spring system.
Revised: To investigate the differences, we perform a learning task with multiple goals on a 2D mass spring system.

Original: In this experiment shown in the left of Fig.
Revised: In this experiment shown in the left of Fig.

Original: 5, SGD (curve Full-OP) makes minor progress while our modiﬁed Adam optimizer with reduced momentum drops the validation obviously within 10000 iterations.
Revised: 5, SGD (curve Full-OP) makes minor progress while our modiﬁed Adam optimizer with reduced momentum drops the validation obviously within 10000 iterations.

Original: One key difference between Adam and SGD is that Adam utilises the momentum while vanilla SGD doesn’t.
Revised: One key difference between Adam and SGD is that Adam utilises the momentum while vanilla SGD doesn’t.

Original: To further
Revised: To further

Original: [Table]
Revised: [Table]

Original: investigate the the effectiveness of the momentum hyperparameters β , we perform a grid search for both β 1 and β 2 whose default values are 0 .
Revised: investigate the the effectiveness of the momentum hyperparameters β , we perform a grid search for both β 1 and β 2 whose default values are 0 .

Original: 9 and 0 .
Revised: 9 and 0 .

Original: 999 respectively.
Revised: 999 respectively.

Original: The results presented in tableshow that reducing β 2 to 0 .
Revised: The results presented in tableshow that reducing β 2 to 0 .

Original: 9 delivers us the best performance.
Revised: 9 delivers us the best performance.

Original: We therefore use this Adam optimizer with reduced momentum in all our experiments.
Revised: We therefore use this Adam optimizer with reduced momentum in all our experiments.

Original: 4. L EARNING TECHNIQUES
Revised: 4. L EARNING TECHNIQUES

Original: Batching We adopt batching into differentiable physics and perform a series of experiments as shown in the right of Fig.
Revised: Batching We adopt batching into differentiable physics and perform a series of experiments as shown in the right of Fig.

Original: 5 to verify its effectiveness, with batch size 1, 8, 32, and 64.
Revised: 5 to verify its effectiveness, with batch size 1, 8, 32, and 64.

Original: It can be observed that without batching, there is a large variance in training loss and it hardly converges.
Revised: It can be observed that without batching, there is a large variance in training loss and it hardly converges.

Original: Batching with a proper size helps reduce the variance and effectively improve the training performance.
Revised: Batching with a proper size helps reduce the variance and effectively improve the training performance.

Original: We also ﬁnd that an overlarge batching size does not help training obviously.
Revised: We also ﬁnd that an overlarge batching size does not help training obviously.

Original: For example, batching with size 64 only improve the performance marginally.
Revised: For example, batching with size 64 only improve the performance marginally.

Original: Loss design We compare our tailored loss function with a naive loss function.
Revised: Loss design We compare our tailored loss function with a naive loss function.

Original: The naive loss function deﬁnes the velocity as the position difference of the center of mass between consecutive time steps, i.e., requiring the agent to move at a constant velocity at each time step.
Revised: The naive loss function deﬁnes the velocity as the position difference of the center of mass between consecutive time steps, i.e., requiring the agent to move at a constant velocity at each time step.

Original: The experiments show that the agent under the naive loss design can not be properly trained to run,
Revised: The experiments show that the agent under the naive loss design can not be properly trained to run,

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: e., there are almost no progress in running loss dropping
Revised: e., there are almost no progress in running loss dropping

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: N o r m li z e d
Revised: N o r m li z e d

Original: [Table]
Revised: [Table]

Original: li d a t i
Revised: li d a t i

Original: [Table]
Revised: [Table]

Original: n
Revised: n

Original: [Table]
Revised: [Table]

Original: a
Revised: a

Original: [Table]
Revised: [Table]

Original: ss
Revised: ss

Original: [Table]
Revised: [Table]

Original: Ablation
Revised: Ablation

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table] [Figure]

Revised: [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table] [Figure]

Revised: [Table]

Revised: [Figure]

Original:  [Figure]
Revised: [Table] [Figure] [Table] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Table] 
Revised: [Figure] [Table] [Figure]

Original: [Figure]

Original: [Table][Figure] [Table] [Figure]
Revised: [Table]

Original: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure]

Original: [Table][Figure] [Table] [Figure]
Revised: [Table]

Original: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure]

Original: [Table][Figure] [Table] [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure]

Original: [Table][Figure] [Table] [Figure]
Revised: [Table]

Original: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure]

Original: [Table][Figure] [Table] [Figure]
Revised: [Table]

Original: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure]

Original: [Table][Figure] [Table] [Figure]
Revised: [Table]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]
Revised: [Figure]

Original: e d V a li d a t i o n J u m p .
Revised: e d V a li d a t i o n J u m p .

Original: [Figure]
Revised: [Figure]

Original: o ss
Revised: o ss

Original: [Figure]
Revised: [Figure]

Original: e d V a li d a t i o n J u m p
Revised: e d V a li d a t i o n J u m p

Original: [Figure]
Revised: [Figure]

Original: N o r m li z e d V a li d a t i n J u
Revised: N o r m li z e d V a li d a t i n J u

Original: [Figure]
Revised: [Figure]

Original: d V a li
Revised: d V a li

Original: [Figure]
Revised: [Figure]

Original: t
Revised: t

Original: [Figure]
Revised: [Figure]

Original: J
Revised: J

Original: [Figure]
Revised: [Figure]

Original: Activation function To validate the effectiveness of our sin activation function, we replaced it for hidden layer with various popular activation functions tanh , relu , gelu and sigmoid .
Revised: Activation function To validate the effectiveness of our sin activation function, we replaced it for hidden layer with various popular activation functions tanh , relu , gelu and sigmoid .

Original: For the output layer, the output actuation is expected to be limited in the range -1 to 1 due to the physical constraints of an actuator, e.g., a spring should not be overly compressed or stretched.
Revised: For the output layer, the output actuation is expected to be limited in the range -1 to 1 due to the physical constraints of an actuator, e.g., a spring should not be overly compressed or stretched.

Original: Therefore we made some modiﬁcations for the activation functions in the output layer.
Revised: Therefore we made some modiﬁcations for the activation functions in the output layer.

Original: To be more speciﬁc, we clamp the values larger than 1 or smaller than -1.
Revised: To be more speciﬁc, we clamp the values larger than 1 or smaller than -1.

Original: In addition, for activation functions whose values are constantly zero (or very small) when input is negative, we modify the relu and gelu by subtracting the value by 1, for sigmoid we map the its value from [0, 1] to [-1, 1].
Revised: In addition, for activation functions whose values are constantly zero (or very small) when input is negative, we modify the relu and gelu by subtracting the value by 1, for sigmoid we map the its value from [0, 1] to [-1, 1].

Original: The results in
Revised: The results in

Original: Fig.7 show that using sin as activation functions gives the best performance among all three losses.
Revised: Fig.7 show that using sin as activation functions gives the best performance among all three losses.

Original: For more results of activation functions ablation studies, please check the appendix
Revised: For more results of activation functions ablation studies, please check the appendix

Original: 4. I MPORTANCE OF INPUT FEATURES
Revised: 4. I MPORTANCE OF INPUT FEATURES

Original: [Figure]
Revised: [Figure]

Original: Ablation on input features The input features consist of three parts: periodic control signal, state vector, and targets, as mentioned in section 3.3.
Revised: Ablation on input features The input features consist of three parts: periodic control signal, state vector, and targets, as mentioned in section 3.3.

Original: We investigate each part of the contribution to training by ablation studies as shown in Fig.8.
Revised: We investigate each part of the contribution to training by ablation studies as shown in Fig.8.

Original: The results show that each of the three parts serves as a necessary component for a converged training.
Revised: The results show that each of the three parts serves as a necessary component for a converged training.

Original: The periodic control signal serves as an important role.
Revised: The periodic control signal serves as an important role.

Original: The agents can hardly move without this signal as the corresponding task loss does not drop along the training (shown by the red curves in Fig.8).
Revised: The agents can hardly move without this signal as the corresponding task loss does not drop along the training (shown by the red curves in Fig.8).

Original: The targets are another essential part, without which the optimizer can hardly work after a minor progress as shown by the yellow curves in Fig.8.
Revised: The targets are another essential part, without which the optimizer can hardly work after a minor progress as shown by the yellow curves in Fig.8.

Original: Although the state vector is less important in terms of reducing the loss, the overall task performance is degraded without the state vector as shown by the green curves in Fig.
Revised: Although the state vector is less important in terms of reducing the loss, the overall task performance is degraded without the state vector as shown by the green curves in Fig.

Original: [Figure]
Revised: [Figure]

Original: 4. B ENCHMARK AGAINST REINFORCEMENT LEARNING
Revised: 4. B ENCHMARK AGAINST REINFORCEMENT LEARNING

Original: [Figure]
Revised: [Figure]

Original: We implement the standard proximal policy optimization (PPO) benchmark in multiple goals settings as shown in Fig. 9.
Revised: We implement the standard proximal policy optimization (PPO) benchmark in multiple goals settings as shown in Fig. 9.

Original: The PPO agent can make progress in single goal learning and learn certain locomotion patterns to move toward the goal.
Revised: The PPO agent can make progress in single goal learning and learn certain locomotion patterns to move toward the goal.

Original: Yet as the task gets complicated, PPO often gets trapped in the local minima.
Revised: Yet as the task gets complicated, PPO often gets trapped in the local minima.

Original: We try our best to ﬁne-tune the PPO hyper-parameters, but still ﬁnd it tends to overﬁt to one goal but fails to ﬁnd a balance between different goals.
Revised: We try our best to ﬁne-tune the PPO hyper-parameters, but still ﬁnd it tends to overﬁt to one goal but fails to ﬁnd a balance between different goals.

Original: It can be observed in
Revised: It can be observed in

Original: Fig.
Revised: Fig.

Original: 9 where the agents trained by PPO can learn to jump but struggle to run.
Revised: 9 where the agents trained by PPO can learn to jump but struggle to run.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Differentiable simulation Differentiable physical simulation is getting increasingly more attention in the learning community.
Revised: Differentiable simulation Differentiable physical simulation is getting increasingly more attention in the learning community.

Original: Two families of methods exist: the ﬁrst family uses neural networks to approximate physical simulators (Battaglia et al., 2016;
Revised: Two families of methods exist: the ﬁrst family uses neural networks to approximate physical simulators (Battaglia et al., 2016;

Original: Chang et al., 2016; Mrowca et al., 2018;
Revised: Chang et al., 2016; Mrowca et al., 2018;

Original: Li et al., 2018).
Revised: Li et al., 2018).

Original: Differentiating the approximating NNs then yields gradients of the (approximate) physical simulation.
Revised: Differentiating the approximating NNs then yields gradients of the (approximate) physical simulation.

Original: The second family is more accurate and direct: many of these methods using differentiable programming (speciﬁcally, reverse-mode automatic differentiation) to implement physical simulators (Degrave et al., 2016; de Avila Belbute-Peres et al., 2018;
Revised: The second family is more accurate and direct: many of these methods using differentiable programming (speciﬁcally, reverse-mode automatic differentiation) to implement physical simulators (Degrave et al., 2016; de Avila Belbute-Peres et al., 2018;

Original: Schenck & Fox, 2018;
Revised: Schenck & Fox, 2018;

Original: Heiden et al., 2019; Hu et al., 2019; 2020; Huang et al., 2021).
Revised: Heiden et al., 2019; Hu et al., 2019; 2020; Huang et al., 2021).

Original: Automatic differentiation works well for explicit time integrators, but when it comes to implicit time integration, people often adopt the adjoint methods (Bern et al., 2019; Geilinger et al., 2020), LCP (de Avila Belbute-Peres et al., 2018) and QR decompositions (Liang et al., 2019; Qiao et al., 2020).
Revised: Automatic differentiation works well for explicit time integrators, but when it comes to implicit time integration, people often adopt the adjoint methods (Bern et al., 2019; Geilinger et al., 2020), LCP (de Avila Belbute-Peres et al., 2018) and QR decompositions (Liang et al., 2019; Qiao et al., 2020).

Original: In this work, we leverage DiffTaichi (Hu et al., 2020), an automatic differentiation system, to create high-performance parallel differentiable simulators and the built-in NN controllers.
Revised: In this work, we leverage DiffTaichi (Hu et al., 2020), an automatic differentiation system, to create high-performance parallel differentiable simulators and the built-in NN controllers.

Original: Locomotion skill learning Producing physically plausible characters with various locomotion skills is a challenging problem.
Revised: Locomotion skill learning Producing physically plausible characters with various locomotion skills is a challenging problem.

Original: One classic approach is to manually design locomotion controllers subject to physics laws of motion in order to generate walking Ye & Liu (2010);
Revised: One classic approach is to manually design locomotion controllers subject to physics laws of motion in order to generate walking Ye & Liu (2010);

Original: Coros et al.
Revised: Coros et al.

Original: (2010) or bicycling Tan et al.
Revised: (2010) or bicycling Tan et al.

Original: (2014) characters.
Revised: (2014) characters.

Original: These physically-based controllers often rely on a complex set of control parameters and are difﬁcult to design and time-consuming to optimize.
Revised: These physically-based controllers often rely on a complex set of control parameters and are difﬁcult to design and time-consuming to optimize.

Original: Another line of research suggests using data to control characters kinematically.
Revised: Another line of research suggests using data to control characters kinematically.

Original: Given a set of data clips, controllers can be made to select the best ﬁt clip to properly react to certain situations Safonova & Hodgins (2007); Lee et al.
Revised: Given a set of data clips, controllers can be made to select the best ﬁt clip to properly react to certain situations Safonova & Hodgins (2007); Lee et al.

Original: ; Liu et al.
Revised: ; Liu et al.

Original: These kinematic models use the motion clips to build a state machine and add transitions between similar frames in adjacent states.
Revised: These kinematic models use the motion clips to build a state machine and add transitions between similar frames in adjacent states.

Original: Although being able to produce higher quality motions than most simulation-based methods, the kinematic methods lack the ability to synthesize behaviors for unseen situations.
Revised: Although being able to produce higher quality motions than most simulation-based methods, the kinematic methods lack the ability to synthesize behaviors for unseen situations.

Original: With the recent advances in deep learning techniques, attempts have been explored to incorporate reinforcement learning (RL) into locomotion skill learning Peng et al.
Revised: With the recent advances in deep learning techniques, attempts have been explored to incorporate reinforcement learning (RL) into locomotion skill learning Peng et al.

Original: ; Liu & Hodgins (2017); Peng et al.
Revised: ; Liu & Hodgins (2017); Peng et al.

Original: ; Park et al.
Revised: ; Park et al.

Original: These modern controllers gain their effectiveness either from tracking high-quality reference motion clips or from cleverly designed rewards to imitate the reference.
Revised: These modern controllers gain their effectiveness either from tracking high-quality reference motion clips or from cleverly designed rewards to imitate the reference.

Original: However, the exploration space for
Revised: However, the exploration space for

Original: RL is usually prohibitively large to achieve complicated target motions.
Revised: RL is usually prohibitively large to achieve complicated target motions.

Original: Carefully designed early termination strategies Ma et al.
Revised: Carefully designed early termination strategies Ma et al.

Original: ; Won et al.
Revised: ; Won et al.

Original: (2020), better optimization methods Yang & Yin (2021), and adversarial RL schemes Peng et al.
Revised: (2020), better optimization methods Yang & Yin (2021), and adversarial RL schemes Peng et al.

Original: (2021) enable these RL-based methods to achieve richer behaviors.
Revised: (2021) enable these RL-based methods to achieve richer behaviors.

Original: However, it is still time-consuming to get a well-trained RL model in complex tasks such as soft robot control.
Revised: However, it is still time-consuming to get a well-trained RL model in complex tasks such as soft robot control.

Original: Our method, on the other hand, leverages the differentiable simulation framework and can achieve much better convergence behavior compared to RL-based methods.
Revised: Our method, on the other hand, leverages the differentiable simulation framework and can achieve much better convergence behavior compared to RL-based methods.

Original:  L IMITATIONS AND C ONCLUSIONS
Revised:  L IMITATIONS AND C ONCLUSIONS

Original: Currently, difﬁculty of the learning tasks depends on robot design and physical parameters.
Revised: Currently, difﬁculty of the learning tasks depends on robot design and physical parameters.

Original: The training performance may be degraded with improper robot designs.
Revised: The training performance may be degraded with improper robot designs.

Original: For instance, in our stool case where unnecessary actuators on its body are allowed, it is more difﬁcult to achieve good results.
Revised: For instance, in our stool case where unnecessary actuators on its body are allowed, it is more difﬁcult to achieve good results.

Original: Ofﬂoading the physical parameter tuning and robot designing to an automatic pipeline will be an interesting future research direction.
Revised: Ofﬂoading the physical parameter tuning and robot designing to an automatic pipeline will be an interesting future research direction.

Original: To summarize, we have presented an effective end-to-end differentiable physics based learning framework for soft robot complex locomotion skills learning.
Revised: To summarize, we have presented an effective end-to-end differentiable physics based learning framework for soft robot complex locomotion skills learning.

Original: We systematically enhance classical differentiable physics learning systems with a suite of techniques and investigated the key factors contributing to the training in detail.
Revised: We systematically enhance classical differentiable physics learning systems with a suite of techniques and investigated the key factors contributing to the training in detail.

Original: We show that our framework can provide stable gradients without explosion during a simulation with hundreds of steps.
Revised: We show that our framework can provide stable gradients without explosion during a simulation with hundreds of steps.

Original: Together with batching, the modiﬁed Adam dramatically outperforms the stochastic gradient descent (SGD) on complex tasks with multiple goals.
Revised: Together with batching, the modiﬁed Adam dramatically outperforms the stochastic gradient descent (SGD) on complex tasks with multiple goals.

Original: Beneﬁted by our tailored loss functions, the network take advantages of the three parts of the input features.
Revised: Beneﬁted by our tailored loss functions, the network take advantages of the three parts of the input features.

Original: We found that the periodic control signal dominates the actuation signal, while state vector and task goals have weaker effects.
Revised: We found that the periodic control signal dominates the actuation signal, while state vector and task goals have weaker effects.

Original: The over-dominated periodic signal may induce high-frequency noises.
Revised: The over-dominated periodic signal may induce high-frequency noises.

Original: To balance the importance between the input features, we additionally pose an actuation loss as a regularization term.
Revised: To balance the importance between the input features, we additionally pose an actuation loss as a regularization term.

Original: It can effectively suppress the noises, which help the agents move more smoothly.
Revised: It can effectively suppress the noises, which help the agents move more smoothly.

Original: In addition, we compare our method with the state-of-the-art reinforcement learning proximal policy optimization.
Revised: In addition, we compare our method with the state-of-the-art reinforcement learning proximal policy optimization.

Original: Our method shows advantages in both training performance and convergence efﬁciency on tasks with multiple goals.
Revised: Our method shows advantages in both training performance and convergence efﬁciency on tasks with multiple goals.

Original: Our framework enables users to ﬂexibly design robots and to teach them locomotion skills.
Revised: Our framework enables users to ﬂexibly design robots and to teach them locomotion skills.

Original: The trained agents can be manipulated to smoothly switch locomotion tasks such as running, jumping, and crawling with different speeds and orientations, interactively.
Revised: The trained agents can be manipulated to smoothly switch locomotion tasks such as running, jumping, and crawling with different speeds and orientations, interactively.

Original: To the best of our knowledge, this is the ﬁrst time a learning system based on differentiable physics can deliver controllers with such robustness, ﬂexibility, and efﬁciency.
Revised: To the best of our knowledge, this is the ﬁrst time a learning system based on differentiable physics can deliver controllers with such robustness, ﬂexibility, and efﬁciency.

Original: A R EINFORCEMENT LEARNING S ETTING
Revised: A R EINFORCEMENT LEARNING S ETTING

Original: Environment.
Revised: Environment.

Original: We use the open-source implementation of PPO (Kostrikov (2018)) in our environments.
Revised: We use the open-source implementation of PPO (Kostrikov (2018)) in our environments.

Original: Part of important hyper-parameters are listed in the table 3.
Revised: Part of important hyper-parameters are listed in the table 3.

Original: [Table]
Revised: [Table]

Original: Reward Design.
Revised: Reward Design.

Original: We design a reward function for PPO based on our loss functions.
Revised: We design a reward function for PPO based on our loss functions.

Original: Recall the velocity part of equation 4.
Revised: Recall the velocity part of equation 4.

Original: We deﬁne the total velocity reward as:
Revised: We deﬁne the total velocity reward as:

Original: [Equation]
Revised: [Equation]

Original: If we directly split the reward into each time steps by t , the rewards of ﬁrst P r time steps would be zero and it is too difﬁcult for PPO to learn the policy.
Revised: If we directly split the reward into each time steps by t , the rewards of ﬁrst P r time steps would be zero and it is too difﬁcult for PPO to learn the policy.

Original: Therefore, we modiﬁed the reward function to tackle this issue.
Revised: Therefore, we modiﬁed the reward function to tackle this issue.

Original: By equation 1, we have
Revised: By equation 1, we have

Original: [Equation]
Revised: [Equation]

Original: We can deﬁne a function f n ( t (cid:48) , t ) =
Revised: We can deﬁne a function f n ( t (cid:48) , t ) =

Original: [ P r g v ( n )] 2 − [ c ( t
Revised: [ P r g v ( n )] 2 − [ c ( t

Original: [Equation]
Revised: [Equation]

Original: ) − ( c ( t − P r )
Revised: ) − ( c ( t − P r )

Original: + P r g v ( n ))] 2 and substitute it back to equation 8.
Revised: + P r g v ( n ))] 2 and substitute it back to equation 8.

Original: The reward can be re-written as
Revised: The reward can be re-written as

Original: [Equation]
Revised: [Equation]

Original: Refer to that for any t ∈ P
Revised: Refer to that for any t ∈ P

Original: [Equation]
Revised: [Equation]

Original: we have f n ( t − P r , t ) =
Revised: we have f n ( t − P r , t ) =

Original: [Equation]
Revised: [Equation]

Original: which means
Revised: which means

Original: [Equation]
Revised: [Equation]

Original: Let t
Revised: Let t

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: So we can split the whole reward into each time step t as:
Revised: So we can split the whole reward into each time step t as:

Original: [Equation]
Revised: [Equation]

Original: r g v ( n ))](13)
Revised: r g v ( n ))](13)

Original: Recall that the goal of a jumping task is to reach a speciﬁed height goal g h , the intuition for jumping reward is to encourage the agent to improve its maximum height until reaching the height.
Revised: Recall that the goal of a jumping task is to reach a speciﬁed height goal g h , the intuition for jumping reward is to encourage the agent to improve its maximum height until reaching the height.

Original: [Equation]
Revised: [Equation]

Original: where the min s ∈ S h ( s, t + n P ) is the agent height at current time step as mentioned in 3.1 and ˜
Revised: where the min s ∈ S h ( s, t + n P ) is the agent height at current time step as mentioned in 3.1 and ˜

Original: h h ( n ) is the max height record during one task period.
Revised: h h ( n ) is the max height record during one task period.

Original: B N ETWORK A RCHITECTURE
Revised: B N ETWORK A RCHITECTURE

Original: [Figure]
Revised: [Figure]

Original: Our network architecture is shown in Fig.
Revised: Our network architecture is shown in Fig.

Original: It is a two layers fully connected network with achannels hidden layer.
Revised: It is a two layers fully connected network with achannels hidden layer.

Original: The number of input and output channels varies according to different agent designs.
Revised: The number of input and output channels varies according to different agent designs.

Original: C S IMULATION S ETTING
Revised: C S IMULATION S ETTING

Original: Mass-spring systems We adopt the classic Hookean spring model to represent the elastic force and use dashpot damping Baraff & Witkin (1998) as the damping force to simulate mass-spring system.
Revised: Mass-spring systems We adopt the classic Hookean spring model to represent the elastic force and use dashpot damping Baraff & Witkin (1998) as the damping force to simulate mass-spring system.

Original: We found that the drag damping model used by Hu et al.
Revised: We found that the drag damping model used by Hu et al.

Original: Hu et al.
Revised: Hu et al.

Original: (2020) damps the gradient of the NN controller while the dashpot damping model is able to generate vividly changing gradients.
Revised: (2020) damps the gradient of the NN controller while the dashpot damping model is able to generate vividly changing gradients.

Original: Adopting dashpot damping model makes our agent more ﬂexible.
Revised: Adopting dashpot damping model makes our agent more ﬂexible.

Original: Material point methods We use MLS-MPM Hu et al.
Revised: Material point methods We use MLS-MPM Hu et al.

Original: as our material point method simulator.
Revised: as our material point method simulator.

Original: We further applied the afﬁne particle-in-cell method Jiang et al.
Revised: We further applied the afﬁne particle-in-cell method Jiang et al.

Original: (2015) to reduce the artiﬁcial damping.
Revised: (2015) to reduce the artiﬁcial damping.

Original: Due to the nature of MPM as a hybrid Lagrangian-Eulerian method, it always requires a background grid during the simulation.
Revised: Due to the nature of MPM as a hybrid Lagrangian-Eulerian method, it always requires a background grid during the simulation.

Original: Naive MPM implementations ﬁx the position of the background grid, hence limit the moving range of the agent.
Revised: Naive MPM implementations ﬁx the position of the background grid, hence limit the moving range of the agent.

Original: We apply a dynamic background grid that follows the agent all the time to overcome this problem.
Revised: We apply a dynamic background grid that follows the agent all the time to overcome this problem.

Original: D I NTERACTIVE CONTROL
Revised: D I NTERACTIVE CONTROL

Original: Smooth interpolation between different tasks is a key advantage of our approach.
Revised: Smooth interpolation between different tasks is a key advantage of our approach.

Original: In interactive settings, users can smoothly control the agent’s motion via input devices such as a keyboard as shown in Fig.
Revised: In interactive settings, users can smoothly control the agent’s motion via input devices such as a keyboard as shown in Fig.

Original: Please refer to our supplemental video for more details.
Revised: Please refer to our supplemental video for more details.

Original: E 3D RESULTS
Revised: E 3D RESULTS

Original: In addition to 2D cases, our framework can be applied to 3D cases seamlessly.
Revised: In addition to 2D cases, our framework can be applied to 3D cases seamlessly.

Original: In a 3D space, an agent is additionally expected to control its orientation on a plane, i.e., rotate.
Revised: In a 3D space, an agent is additionally expected to control its orientation on a plane, i.e., rotate.

Original: We design several 3D agents (shown in Fig. 2) and train them to run and rotate.
Revised: We design several 3D agents (shown in Fig. 2) and train them to run and rotate.

Original: These trained agents can achieve speciﬁed goals given control signals after trained for several hundred to a few thousand iterations.
Revised: These trained agents can achieve speciﬁed goals given control signals after trained for several hundred to a few thousand iterations.

Original: Please check the supplemental videos for more details.
Revised: Please check the supplemental videos for more details.

Original: Loss function in 3D The agents in 3D are trained with the loss function below.
Revised: Loss function in 3D The agents in 3D are trained with the loss function below.

Original: The loss for running L v is inherited from 2D settings.
Revised: The loss for running L v is inherited from 2D settings.

Original: The rotation loss L r is deﬁned as the accumulation of centralized point-wise distance between current position and target rotated position.
Revised: The rotation loss L r is deﬁned as the accumulation of centralized point-wise distance between current position and target rotated position.

Original: [Equation]
Revised: [Equation]

Original: where g ω is the target angular velocity assembled in input features.
Revised: where g ω is the target angular velocity assembled in input features.

Original: The format of R is derived by rotation matrix along XZ-plane.
Revised: The format of R is derived by rotation matrix along XZ-plane.

Original: Note, for solving running tasks, the rotation loss also plays an important role.
Revised: Note, for solving running tasks, the rotation loss also plays an important role.

Original: A signiﬁcant issue of 3D agents running is accumulated orientation error.
Revised: A signiﬁcant issue of 3D agents running is accumulated orientation error.

Original: Utilizing identity rotation matrix R helps adjust agent’s posture.
Revised: Utilizing identity rotation matrix R helps adjust agent’s posture.

Original: In turn’s of rotation tasks, the rotation center should be maintained still which is equivalent to zero velocity.
Revised: In turn’s of rotation tasks, the rotation center should be maintained still which is equivalent to zero velocity.

Original: These two losses complement with each other to achieve the ﬁnal goals.
Revised: These two losses complement with each other to achieve the ﬁnal goals.

Original: Friction model.
Revised: Friction model.

Original: In real world, agents running is driven by static friction between ”feet” and ground.
Revised: In real world, agents running is driven by static friction between ”feet” and ground.

Original: We conﬁgure and train agents with different contact models in our experiments.
Revised: We conﬁgure and train agents with different contact models in our experiments.

Original: With zero friction applied, training loss shows no evidence decreasing and the agent is not able to move no matter what actions patterns it learns.
Revised: With zero friction applied, training loss shows no evidence decreasing and the agent is not able to move no matter what actions patterns it learns.

Original: With a fully sticky surface, where only upwards vertical velocity is kept and all other components of velocities are projected to zero after contact, the 2D agents work well.
Revised: With a fully sticky surface, where only upwards vertical velocity is kept and all other components of velocities are projected to zero after contact, the 2D agents work well.

Original: However, in 3D, it turns out that the agent tends to stuck.
Revised: However, in 3D, it turns out that the agent tends to stuck.

Original: In practice, we adopt the classic slip with friction model from physically-based simulation which involves both kinetic friction and static friction.
Revised: In practice, we adopt the classic slip with friction model from physically-based simulation which involves both kinetic friction and static friction.

Original: These two categories of contact are determined by how much pressure force is applied on the surface and can be treated like a branched function numerically which can be handled by our auto-differential system.
Revised: These two categories of contact are determined by how much pressure force is applied on the surface and can be treated like a branched function numerically which can be handled by our auto-differential system.

Original: Please see the supplemental video for a visual comparison between slip and sticky friction models.
Revised: Please see the supplemental video for a visual comparison between slip and sticky friction models.

Original: F D ETAILED R ESULTS ON F URTHER E XPERIMENTS
Revised: F G ENERALIZATION TO M ANIPULATION T ASKS

Revised: Here we provide results on simple manipulation tasks in additional to locomotion tasks.

Revised: In these tasks, the agent is expected to manipulate a object (marked in purple) to hit a target point (marked in green).

Revised: This task has similar periodicity property like locomotion.

Revised: We designed two scenarios:

Revised: Juggle and Dribbble and Shot shown in Fig.13.

Revised: The visual results are also shown in the video in supplemental materials.

Revised: Juggle In this scenario, the target point appears in the sky above the the agent.

Revised: The agent is expected to ’juggle’ the object to the target point.

Revised: Dribble and Shot

Revised: In this scenario, the target point keeps moving toward left.

Revised: The agent is expected to carry the object and shot it to the target point.

Revised: [Figure]

Revised: [Figure]

Revised: The experiments indicate that our method has the possibility to generalize to tasks beyond locomotion.

Revised: G D ETAILED R ESULTS ON F URTHER E XPERIMENTS

Original: Here we show more detailed results on further experiments, including ablation studies on different agents, validation loss on different tasks for agents and more network weights visualization results.
Revised: Here we show more detailed results on further experiments, including ablation studies on different agents, validation loss on different tasks for agents and more network weights visualization results.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: in the table represent the normalized validation loss for Task . For each setting, the experiments are repeated for multiple times. It can be observed that the model using sin for both hidden and output layer achieves the best results.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]